BUG: soft lockup in wg_packet_tx_worker

Status: upstream: reported C repro on 2024/11/16 16:23
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+25c3230da59774581486@syzkaller.appspotmail.com
First crash: 636d, last: 161d
Fix bisection: failed (error log, bisect log)
  
▶ ▼ Bug presence (2)
Date Name Commit Repro Result
2024/11/16 linux-5.15.y (ToT) d98fd109f827 C [report] BUG: soft lockup in wg_packet_tx_worker
2024/11/16 upstream (ToT) e8bdb3c8be08 C Didn't crash
▶ ▼ Similar bugs (11)
Kernel Title Rank 🛈 Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-6.1 BUG: soft lockup in wg_packet_tx_worker 1 2 593d 623d 0/3 auto-obsoleted due to no activity on 2025/04/07 20:48
upstream BUG: soft lockup in wg_packet_tx_worker kvm 1 1 2291d 2291d 15/29 fixed on 2020/07/17 17:58
android-5-10 BUG: soft lockup in wg_packet_tx_worker 1 5 1085d 1135d 0/2 auto-obsoleted due to no activity on 2023/11/22 23:32
upstream BUG: soft lockup in wg_packet_tx_worker (3) net 1 1 499d 494d 0/29 auto-obsoleted due to no activity on 2025/07/10 22:54
android-6-1 BUG: soft lockup in wg_packet_tx_worker 1 3 1000d 1022d 0/2 auto-obsoleted due to no activity on 2024/02/16 07:09
upstream BUG: soft lockup in wg_packet_tx_worker (2) wireguard 1 5 622d 754d 0/29 closed as invalid on 2025/02/06 13:09
android-5-15 BUG: soft lockup in wg_packet_tx_worker 1 1 1094d 1094d 0/2 auto-obsoleted due to no activity on 2023/11/14 01:23
upstream INFO: rcu detected stall in wg_packet_tx_worker (3) net 1 syz, C inconclusive 5 1824d 1806d 20/29 fixed on 2021/11/10 00:50
android-5-15 BUG: soft lockup in wg_packet_tx_worker (2) 1 1 850d 850d 0/2 auto-obsoleted due to no activity on 2024/07/15 09:44
linux-5.15 INFO: rcu detected stall in wg_packet_tx_worker 1 1 799d 799d 0/3 auto-obsoleted due to no activity on 2024/09/14 07:48
upstream INFO: rcu detected stall in wg_packet_tx_worker (4) net 1 1 1722d 1722d 0/29 closed as invalid on 2022/02/08 10:10
▶ ▼ Last patch testing requests (7)
Created Duration User Patch Repo Result
2026/05/17 20:37 20m retest repro linux-5.15.y report log
2026/03/06 08:21 16m retest repro linux-5.15.y report log
2025/12/26 00:49 17m retest repro linux-5.15.y report log
2025/10/15 22:50 46m retest repro linux-5.15.y report log
2025/08/05 08:10 36m retest repro linux-5.15.y report log
2025/05/24 14:11 14m retest repro linux-5.15.y report log
2025/03/13 17:20 17m retest repro linux-5.15.y report log
▶ ▼ Fix bisection attempts (8)
Created Duration User Patch Repo Result
2026/05/10 21:55 16m fix candidate upstream error job log
2026/03/28 09:29 1m fix candidate upstream error job log
2026/02/25 20:08 1m fix candidate upstream error job log
2026/01/26 08:05 1m fix candidate upstream error job log
2025/12/25 19:49 3m fix candidate upstream error job log
2025/10/17 03:38 1m fix candidate upstream error job log
2025/04/01 13:58 3m fix candidate upstream error job log
2025/01/02 00:15 1m bisect fix linux-5.15.y error job log

Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [wg-crypt-wg2:4043]
Modules linked in:
irq event stamp: 6529
hardirqs last  enabled at (6528): [<ffff800011ab9be4>] __exit_to_kernel_mode arch/arm64/kernel/entry-common.c:81 [inline]
hardirqs last  enabled at (6528): [<ffff800011ab9be4>] exit_to_kernel_mode+0x100/0x178 arch/arm64/kernel/entry-common.c:91
hardirqs last disabled at (6529): [<ffff800011ab9e2c>] enter_el1_irq_or_nmi+0x10/0x1c arch/arm64/kernel/entry-common.c:227
softirqs last  enabled at (1822): [<ffff80000c83fe34>] local_bh_enable+0x10/0x34 include/linux/bottom_half.h:31
softirqs last disabled at (1826): [<ffff80000c84af2c>] wg_socket_send_skb_to_peer+0x64/0x1a8 drivers/net/wireguard/socket.c:173
CPU: 0 PID: 4043 Comm: wg-crypt-wg2 Not tainted 5.15.172-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/30/2024
Workqueue: wg-crypt-wg2 wg_packet_tx_worker
pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : fq_dequeue+0x538/0x18fc net/sched/sch_fq.c:549
lr : fq_flow_add_tail net/sched/sch_fq.c:165 [inline]
lr : fq_dequeue+0x700/0x18fc net/sched/sch_fq.c:564
sp : ffff80001ff670c0
x29: ffff80001ff671e0 x28: ffff0000db58b2c0 x27: 000000202179a978
x26: dfff800000000000 x25: ffff0000c8602240 x24: ffff0000db58b2e8
x23: 1fffe000190c0451 x22: ffff0000db58b2d0 x21: ffff0000c8602108
x20: ffff0000db58b2d8 x19: ffff0000c8602288 x18: 0000000000000802
x17: 0000000000000000 x16: ffff8000082ec8b8 x15: 0000000000000001
x14: 1ffff0000295806a x13: dfff800000000000 x12: 0000000000000003
x11: 0000000000000802 x10: 0000000000000000 x9 : ffff0000d5073680
x8 : 0000000000000000 x7 : 0000000000000000 x6 : 0000000000000000
x5 : 0000000000000080 x4 : 0000000000000000 x3 : ffff80000804605c
x2 : 0000000000000002 x1 : 00000000e274b1f4 x0 : 0000000000000001
Call trace:
 fq_dequeue+0x538/0x18fc net/sched/sch_fq.c:549
 dequeue_skb net/sched/sch_generic.c:292 [inline]
 qdisc_restart net/sched/sch_generic.c:397 [inline]
 __qdisc_run+0x1ec/0x1fc0 net/sched/sch_generic.c:415
 __dev_xmit_skb net/core/dev.c:3910 [inline]
 __dev_queue_xmit+0x1038/0x2ac8 net/core/dev.c:4221
 dev_queue_xmit+0x24/0x34 net/core/dev.c:4289
 neigh_hh_output include/net/neighbour.h:493 [inline]
 neigh_output include/net/neighbour.h:507 [inline]
 ip_finish_output2+0xd3c/0x131c net/ipv4/ip_output.c:228
 __ip_finish_output+0x1b0/0x458
 ip_finish_output+0x40/0x218 net/ipv4/ip_output.c:316
 NF_HOOK_COND include/linux/netfilter.h:291 [inline]
 ip_output+0x330/0x49c net/ipv4/ip_output.c:430
 dst_output include/net/dst.h:443 [inline]
 ip_local_out+0x120/0x160 net/ipv4/ip_output.c:126
 iptunnel_xmit+0x40c/0x8fc net/ipv4/ip_tunnel_core.c:82
 udp_tunnel_xmit_skb+0x178/0x284 net/ipv4/udp_tunnel_core.c:175
 send4+0x73c/0xcb0 drivers/net/wireguard/socket.c:85
 wg_socket_send_skb_to_peer+0xc8/0x1a8 drivers/net/wireguard/socket.c:175
 wg_packet_create_data_done drivers/net/wireguard/send.c:251 [inline]
 wg_packet_tx_worker+0x1a8/0x714 drivers/net/wireguard/send.c:276
 process_one_work+0x790/0x11b8 kernel/workqueue.c:2310
 process_scheduled_works kernel/workqueue.c:2373 [inline]
 rescuer_thread+0x5f4/0xe20 kernel/workqueue.c:2564
 kthread+0x37c/0x45c kernel/kthread.c:334
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:870

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/11/16 16:23 linux-5.15.y d98fd109f827 cfe3a04a .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-arm64 BUG: soft lockup in wg_packet_tx_worker
* Struck through repros no longer work on HEAD.