INFO: rcu detected stall in wg_packet_tx_worker

Status: fixed on 2020/07/17 17:58
Subsystems: net
[Documentation on labels]
Reported-by: syzbot+0251e883fe39e7a0cb0a@syzkaller.appspotmail.com
Fix commit: df4953e4e997 sch_sfq: validate silly quantum values
First crash: 2303d, last: 2243d
Cause bisection: introduced by (bisect log) :
commit e7096c131e5161fa3b8e52a650d7719d2857adfd
Author: Jason A. Donenfeld <Jason@zx2c4.com>
Date: Sun Dec 8 23:27:34 2019 +0000

  net: WireGuard secure network tunnel

Crash: INFO: rcu detected stall in wg_packet_tx_worker (log)
Repro: C syz .config
  
▶ ▼ Discussions (8)
Title Replies (including bot) Last reply
[PATCH 4.4 00/86] 4.4.224-rc1 review 95 (95) 2020/05/21 07:47
[PATCH 4.14 000/114] 4.14.181-rc1 review 119 (119) 2020/05/19 16:28
[PATCH 4.9 00/90] 4.9.224-rc1 review 95 (95) 2020/05/19 16:27
[PATCH 5.6 000/118] 5.6.13-rc1 review 127 (127) 2020/05/15 08:52
[PATCH 4.19 00/48] 4.19.123-rc1 review 61 (61) 2020/05/14 20:28
[PATCH 5.4 00/90] 5.4.41-rc1 review 95 (95) 2020/05/13 23:01
INFO: rcu detected stall in wg_packet_tx_worker 9 (11) 2020/05/04 11:23
[PATCH net] sch_sfq: validate silly quantum values 2 (2) 2020/04/27 18:50
▶ ▼ Similar bugs (4)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: rcu detected stall in wg_packet_tx_worker (2) net 1 4 2098d 2194d 0/29 auto-closed as invalid on 2021/02/08 20:10
linux-5.15 INFO: rcu detected stall in wg_packet_tx_worker 1 1 794d 794d 0/3 auto-obsoleted due to no activity on 2024/09/14 07:48
upstream INFO: rcu detected stall in wg_packet_tx_worker (4) net 1 1 1718d 1718d 0/29 closed as invalid on 2022/02/08 10:10
upstream INFO: rcu detected stall in wg_packet_tx_worker (3) net 1 syz, C inconclusive 5 1820d 1802d 20/29 fixed on 2021/11/10 00:50

Sample crash report:
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-...!: (10499 ticks this GP) idle=272/1/0x4000000000000002 softirq=53996/53996 fqs=254 
	(t=10500 jiffies g=80873 q=139)
rcu: rcu_preempt kthread starved for 9955 jiffies! g80873 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
rcu: RCU grace-period kthread stack dump:
rcu_preempt     I28848    10      2 0x80004000
Call Trace:
 schedule+0xd0/0x2a0 kernel/sched/core.c:4158
 schedule_timeout+0x35c/0x850 kernel/time/timer.c:1898
 rcu_gp_fqs_loop kernel/rcu/tree.c:1674 [inline]
 rcu_gp_kthread+0x9bf/0x1960 kernel/rcu/tree.c:1836
 kthread+0x388/0x470 kernel/kthread.c:268
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
NMI backtrace for cpu 1
CPU: 1 PID: 3483 Comm: kworker/1:8 Not tainted 5.7.0-rc2-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: wg-crypt-wg0 wg_packet_tx_worker
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x188/0x20d lib/dump_stack.c:118
 nmi_cpu_backtrace.cold+0x70/0xb1 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x231/0x27e lib/nmi_backtrace.c:62
 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline]
 rcu_dump_cpu_stacks+0x19b/0x1e5 kernel/rcu/tree_stall.h:254
 print_cpu_stall kernel/rcu/tree_stall.h:475 [inline]
 check_cpu_stall kernel/rcu/tree_stall.h:549 [inline]
 rcu_pending kernel/rcu/tree.c:3225 [inline]
 rcu_sched_clock_irq.cold+0x55d/0xcfa kernel/rcu/tree.c:2296
 update_process_times+0x25/0x60 kernel/time/timer.c:1727
 tick_sched_handle+0x9b/0x180 kernel/time/tick-sched.c:176
 tick_sched_timer+0x4e/0x140 kernel/time/tick-sched.c:1320
 __run_hrtimer kernel/time/hrtimer.c:1520 [inline]
 __hrtimer_run_queues+0x5ca/0xed0 kernel/time/hrtimer.c:1584
 hrtimer_interrupt+0x312/0x770 kernel/time/hrtimer.c:1646
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1113 [inline]
 smp_apic_timer_interrupt+0x15b/0x600 arch/x86/kernel/apic/apic.c:1138
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:829
 </IRQ>
RIP: 0010:sfq_dequeue+0x19f/0xcc0 net/sched/sch_sfq.c:492
Code: 66 44 03 a5 e2 02 00 00 66 44 89 63 1a 48 8b 9d 28 03 00 00 48 8d 7b 12 48 89 f8 48 c1 e8 03 42 0f b6 14 30 48 89 f8 83 e0 07 <83> c0 01 38 d0 7c 08 84 d2 0f 85 2b 0a 00 00 44 0f b7 63 12 48 8b
RSP: 0018:ffffc9000b69f4e0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000002 RBX: ffff8880986c2370 RCX: ffffffff86466097
RDX: 0000000000000000 RSI: ffffffff86465fd2 RDI: ffff8880986c2382
RBP: ffff888097c2d000 R08: ffff88809b468240 R09: fffff520016d3e7a
R10: 0000000000000003 R11: fffff520016d3e79 R12: 0000000000000000
R13: ffff8880986c238a R14: dffffc0000000000 R15: 0000000000000002
 dequeue_skb net/sched/sch_generic.c:263 [inline]
 qdisc_restart net/sched/sch_generic.c:366 [inline]
 __qdisc_run+0x1ac/0x17b0 net/sched/sch_generic.c:384
 __dev_xmit_skb net/core/dev.c:3704 [inline]
 __dev_queue_xmit+0x1d07/0x30a0 net/core/dev.c:4021
 neigh_hh_output include/net/neighbour.h:499 [inline]
 neigh_output include/net/neighbour.h:508 [inline]
 ip6_finish_output2+0xfb5/0x25b0 net/ipv6/ip6_output.c:117
 __ip6_finish_output+0x442/0xab0 net/ipv6/ip6_output.c:143
 ip6_finish_output+0x34/0x1f0 net/ipv6/ip6_output.c:153
 NF_HOOK_COND include/linux/netfilter.h:296 [inline]
 ip6_output+0x239/0x810 net/ipv6/ip6_output.c:176
 dst_output include/net/dst.h:435 [inline]
 ip6_local_out+0xaf/0x1a0 net/ipv6/output_core.c:179
 ip6tunnel_xmit include/net/ip6_tunnel.h:160 [inline]
 udp_tunnel6_xmit_skb+0x6f4/0xcd0 net/ipv6/ip6_udp_tunnel.c:114
 send6+0x4e3/0xb20 drivers/net/wireguard/socket.c:164
 wg_socket_send_skb_to_peer+0xf5/0x220 drivers/net/wireguard/socket.c:189
 wg_packet_create_data_done drivers/net/wireguard/send.c:250 [inline]
 wg_packet_tx_worker+0x30c/0xc30 drivers/net/wireguard/send.c:278
 process_one_work+0x965/0x16a0 kernel/workqueue.c:2268
 worker_thread+0x96/0xe20 kernel/workqueue.c:2414
 kthread+0x388/0x470 kernel/kthread.c:268
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352

Crashes (24):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/04/26 13:17 upstream b2768df24ec4 99b258dd .config console log report syz C ci-upstream-kasan-gce-386
2020/06/18 19:05 upstream 7ae77150d94d d45a4d69 .config console log report ci-upstream-kasan-gce-smack-root
2020/05/05 15:34 upstream 47cf1b422e60 4b76dd25 .config console log report ci-upstream-kasan-gce-selinux-root
2020/05/05 15:13 upstream 47cf1b422e60 4b76dd25 .config console log report ci-upstream-kasan-gce-root
2020/05/05 07:17 upstream 9851a0dee7c2 9941337c .config console log report ci-upstream-kasan-gce
2020/05/01 22:59 upstream 052c467cb587 bc734e7a .config console log report ci-upstream-kasan-gce-selinux-root
2020/04/29 17:33 upstream 96c9a7802af7 496a08ae .config console log report ci-upstream-kasan-gce
2020/04/27 10:38 upstream 6a8b55ed4056 0ce7569e .config console log report ci-upstream-kasan-gce-root
2020/04/26 15:41 upstream b2768df24ec4 99b258dd .config console log report ci-upstream-kasan-gce
2020/04/26 10:10 upstream b2768df24ec4 99b258dd .config console log report ci-upstream-kasan-gce-root
2020/04/26 08:34 upstream b2768df24ec4 99b258dd .config console log report ci-upstream-kasan-gce-root
2020/04/26 07:15 upstream b2768df24ec4 99b258dd .config console log report ci-upstream-kasan-gce-root
2020/05/01 23:01 upstream 052c467cb587 bc734e7a .config console log report ci-upstream-kasan-gce-386
2020/04/26 22:27 upstream edf17b283844 0ce7569e .config console log report ci-upstream-kasan-gce-386
2020/04/20 07:01 net-old 9bacd256f135 9f7c6d12 .config console log report ci-upstream-net-this-kasan-gce
2020/04/29 17:35 net-next-old 790ab249b55d 496a08ae .config console log report ci-upstream-net-kasan-gce
2020/04/20 06:43 net-next-old 0fde6e3b55a1 9f7c6d12 .config console log report ci-upstream-net-kasan-gce
2020/05/20 09:53 linux-next ac935d227366 67fa1f59 .config console log report ci-upstream-linux-next-kasan-gce-root
2020/05/14 20:20 linux-next ac935d227366 2d572622 .config console log report ci-upstream-linux-next-kasan-gce-root
2020/05/14 09:11 linux-next ac935d227366 a885920d .config console log report ci-upstream-linux-next-kasan-gce-root
2020/05/13 15:51 linux-next ac935d227366 9a6d42fb .config console log report ci-upstream-linux-next-kasan-gce-root
2020/05/11 12:32 linux-next ac935d227366 f8f57555 .config console log report ci-upstream-linux-next-kasan-gce-root
2020/05/09 22:43 linux-next ac935d227366 88cb3e92 .config console log report ci-upstream-linux-next-kasan-gce-root
2020/04/26 07:30 linux-next ac935d227366 99b258dd .config console log report ci-upstream-linux-next-kasan-gce-root
* Struck through repros no longer work on HEAD.