INFO: rcu detected stall in sys_sendmmsg

Status: fixed on 2019/10/09 10:54
Subsystems: net
[Documentation on labels]
Fix commit: d4d6ec6dac07 sch_hhf: ensure quantum and hhf_non_hh_weight are non-zero
First crash: 1543d, last: 1542d
Similar bugs (6)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in sys_sendmmsg (2) kernel 2 1458d 1459d 0/25 closed as invalid on 2019/12/04 14:14
linux-6.1 BUG: soft lockup in sys_sendmmsg 2 153d 182d 0/3 auto-obsoleted due to no activity on 2023/10/08 22:57
linux-5.15 INFO: rcu detected stall in sys_sendmmsg 1 240d 240d 0/3 auto-obsoleted due to no activity on 2023/08/02 15:37
upstream INFO: rcu detected stall in sys_sendmmsg (3) kernel 3 693d 793d 0/25 closed as invalid on 2022/02/08 10:00
upstream INFO: rcu detected stall in sys_sendmmsg (5) net 3 179d 192d 0/25 auto-obsoleted due to no activity on 2023/09/03 00:09
upstream INFO: rcu detected stall in sys_sendmmsg (4) net 1 492d 492d 0/25 auto-obsoleted due to no activity on 2022/10/25 00:28

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	0-...!: (10499 ticks this GP) idle=c7e/1/0x4000000000000002 softirq=229336/229336 fqs=37 
	(t=10501 jiffies g=307461 q=525)
rcu: rcu_preempt kthread starved for 10428 jiffies! g307461 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
rcu: RCU grace-period kthread stack dump:
rcu_preempt     I29080    10      2 0x80004000
Call Trace:
 context_switch kernel/sched/core.c:3254 [inline]
 __schedule+0x877/0xc50 kernel/sched/core.c:3880
 schedule+0x131/0x1e0 kernel/sched/core.c:3947
 schedule_timeout+0x14f/0x240 kernel/time/timer.c:1807
 rcu_gp_fqs_loop kernel/rcu/tree.c:1611 [inline]
 rcu_gp_kthread+0xef8/0x1790 kernel/rcu/tree.c:1768
 kthread+0x332/0x350 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
NMI backtrace for cpu 0
CPU: 0 PID: 413 Comm: syz-executor.2 Not tainted 5.3.0-rc8+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1d8/0x2f8 lib/dump_stack.c:113
 nmi_cpu_backtrace+0xaf/0x1a0 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x174/0x290 lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x10/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x15a/0x220 kernel/rcu/tree_stall.h:254
 print_cpu_stall kernel/rcu/tree_stall.h:455 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:529 [inline]
 rcu_pending kernel/rcu/tree.c:2736 [inline]
 rcu_sched_clock_irq+0xb95/0x16d0 kernel/rcu/tree.c:2183
 update_process_times+0x134/0x190 kernel/time/timer.c:1639
 tick_sched_handle kernel/time/tick-sched.c:167 [inline]
 tick_sched_timer+0x263/0x420 kernel/time/tick-sched.c:1296
 __run_hrtimer kernel/time/hrtimer.c:1389 [inline]
 __hrtimer_run_queues+0x403/0x850 kernel/time/hrtimer.c:1451
 hrtimer_interrupt+0x38c/0xda0 kernel/time/hrtimer.c:1509
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1106 [inline]
 smp_apic_timer_interrupt+0x109/0x280 arch/x86/kernel/apic/apic.c:1131
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:830
RIP: 0010:__list_del_entry_valid+0x7a/0x100 lib/list_debug.c:51
Code: 00 ad de 49 8b 1e 48 39 d3 74 54 48 83 c2 22 49 39 d7 74 5e 4c 89 f8 48 c1 e8 03 42 80 3c 20 00 74 08 4c 89 ff e8 26 c4 58 fe <49> 8b 17 4c 39 f2 75 53 48 83 c3 08 48 89 d8 48 c1 e8 03 42 80 3c
RSP: 0018:ffff88806a4def98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
RAX: 1ffff1100b0bfb4a RBX: ffff8880585fda50 RCX: 0000000000040000
RDX: dead000000000122 RSI: 000000000003ffff RDI: ffff8880585fd9c0
RBP: ffff88806a4defb8 R08: ffffffff860c5d6a R09: 0000000000000000
R10: fffffbfff117be8d R11: 0000000000000000 R12: dffffc0000000000
R13: ffff8880585fd6c0 R14: ffff8880585fd9b8 R15: ffff8880585fda50
 __list_del_entry include/linux/list.h:131 [inline]
 list_move_tail include/linux/list.h:213 [inline]
 hhf_dequeue+0x487/0xaa0 net/sched/sch_hhf.c:439
 dequeue_skb net/sched/sch_generic.c:258 [inline]
 qdisc_restart net/sched/sch_generic.c:361 [inline]
 __qdisc_run+0x217/0x1b30 net/sched/sch_generic.c:379
 __dev_xmit_skb net/core/dev.c:3533 [inline]
 __dev_queue_xmit+0x1161/0x3020 net/core/dev.c:3838
 dev_queue_xmit+0x17/0x20 net/core/dev.c:3902
 br_dev_queue_push_xmit+0x551/0x650 net/bridge/br_forward.c:52
 NF_HOOK include/linux/netfilter.h:305 [inline]
 br_forward_finish net/bridge/br_forward.c:65 [inline]
 NF_HOOK include/linux/netfilter.h:305 [inline]
 __br_forward+0x439/0x720 net/bridge/br_forward.c:109
 deliver_clone net/bridge/br_forward.c:125 [inline]
 maybe_deliver net/bridge/br_forward.c:181 [inline]
 br_flood+0x52e/0x840 net/bridge/br_forward.c:223
 br_dev_xmit+0xcb6/0x1100 net/bridge/br_device.c:96
 __netdev_start_xmit include/linux/netdevice.h:4406 [inline]
 netdev_start_xmit include/linux/netdevice.h:4420 [inline]
 xmit_one net/core/dev.c:3280 [inline]
 dev_hard_start_xmit+0x1c1/0x3f0 net/core/dev.c:3296
 __dev_queue_xmit+0x1e86/0x3020 net/core/dev.c:3869
 dev_queue_xmit+0x17/0x20 net/core/dev.c:3902
 neigh_resolve_output+0x60c/0x6b0 net/core/neighbour.c:1490
 neigh_output include/net/neighbour.h:511 [inline]
 ip6_finish_output2+0x102c/0x13d0 net/ipv6/ip6_output.c:116
 __ip6_finish_output+0x693/0x910 net/ipv6/ip6_output.c:142
 ip6_finish_output+0x52/0x1e0 net/ipv6/ip6_output.c:152
 NF_HOOK_COND include/linux/netfilter.h:294 [inline]
 ip6_output+0x26f/0x390 net/ipv6/ip6_output.c:175
 dst_output include/net/dst.h:436 [inline]
 NF_HOOK include/linux/netfilter.h:305 [inline]
 ip6_xmit+0xead/0x18c0 net/ipv6/ip6_output.c:279
 inet6_csk_xmit+0x2e5/0x410 net/ipv6/inet6_connection_sock.c:135
 l2tp_xmit_core net/l2tp/l2tp_core.c:1030 [inline]
 l2tp_xmit_skb+0xb71/0x10a0 net/l2tp/l2tp_core.c:1132
 pppol2tp_sendmsg+0x405/0x650 net/l2tp/l2tp_ppp.c:325
 sock_sendmsg_nosec net/socket.c:637 [inline]
 sock_sendmsg net/socket.c:657 [inline]
 ___sys_sendmsg+0x60d/0x910 net/socket.c:2311
 __sys_sendmmsg+0x239/0x470 net/socket.c:2413
 __do_sys_sendmmsg net/socket.c:2442 [inline]
 __se_sys_sendmmsg net/socket.c:2439 [inline]
 __x64_sys_sendmmsg+0xa0/0xb0 net/socket.c:2439
 do_syscall_64+0xfe/0x140 arch/x86/entry/common.c:296
RIP: 0033:0x4598e9
Code: fd b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 cb b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f7431eb6c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00000000004598e9
RDX: 0800000000000059 RSI: 0000000020005fc0 RDI: 0000000000000005
RBP: 000000000075bfc8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f7431eb76d4
R13: 00000000004c70a7 R14: 00000000004dc768 R15: 00000000ffffffff

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/09/10 07:33 upstream 56037cadf604 a60cb4cd .config console log report ci-upstream-kasan-gce-smack-root
2019/09/09 06:33 upstream f74c2bb98776 a60cb4cd .config console log report ci-upstream-kasan-gce-smack-root
* Struck through repros no longer work on HEAD.