BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 144s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=12 refcnt=13
pending: vmstat_shepherd, 5*ovs_dp_masks_rebalance, 3*nsim_dev_hwstats_traffic_work, psi_avgs_work, delayed_vfree_work, free_obj_work
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=9 refcnt=10
pending: psi_avgs_work, 7*nsim_dev_hwstats_traffic_work, psi_avgs_work
workqueue events_long: flags=0x0
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: 2*defense_work_handler
workqueue events_unbound: flags=0x2
pwq 8: cpus=0-1 flags=0x4 nice=0 active=3 refcnt=4
in-flight: 15524:nsim_dev_trap_report_work ,15564:toggle_allocation_gate ,15561:nsim_dev_trap_report_work
workqueue events_freezable: flags=0x4
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: update_balloon_stats_func
workqueue events_power_efficient: flags=0x80
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8
in-flight: 5953:wg_ratelimiter_gc_entries
pending: neigh_periodic_work, neigh_managed_work, do_cache_clean, 3*check_lifetime
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
in-flight: 6962:gc_worker
workqueue mm_percpu_wq: flags=0x8
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: vmstat_update
workqueue bat_events: flags=0x6000a
pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=25
in-flight: 15565:batadv_dat_purge
inactive: batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, 4*batadv_iv_send_outstanding_bat_ogm_packet, 3*batadv_purge_orig, 3*batadv_iv_send_outstanding_bat_ogm_packet, 3*batadv_nc_worker, 3*batadv_mcast_mla_update, batadv_iv_send_outstanding_bat_ogm_packet
workqueue wg-crypt-wg0: flags=0x28
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-kex-wg0: flags=0x24
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_receive_worker
workqueue wg-crypt-wg0: flags=0x28
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_decrypt_worker
workqueue wg-crypt-wg1: flags=0x28
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg1: flags=0x24
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_receive_worker
workqueue wg-crypt-wg1: flags=0x28
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker
workqueue wg-crypt-wg1: flags=0x28
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg1: flags=0x28
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x28
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
in-flight: 5892:wg_packet_decrypt_worker
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-crypt-wg2: flags=0x28
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_tx_worker
pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=145s workers=6 idle: 10 5906 978 5920
pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=0s workers=6 idle: 6291 5919 2155 5944 43
pool 8: cpus=0-1 flags=0x4 nice=0 hung=0s workers=55 idle: 3517 15559 15566 15519 15572 15569 15521 15570 15563 15560 15554 15558 49 15555 13 2974 2998 59 168 15550 36 5975 2140 15548 15549 15523 15551 15553 15552 15547 15520 15543 15542 15528 15544 15534 15546 15545 15532 15531 15540 15535 15536 15526 15537 15539 15527 15574 15573 15533
Showing backtraces of running workers in stalled CPU-bound worker pools:
pool 2:
task:kworker/0:8 state:R running task stack:23880 pid:5953 tgid:5953 ppid:2 task_flags:0x4208060 flags:0x00004000
Workqueue: events_power_efficient wg_ratelimiter_gc_entries
Call Trace:
context_switch kernel/sched/core.c:5396 [inline]
__schedule+0x16f5/0x4d00 kernel/sched/core.c:6785
preempt_schedule_irq+0xb5/0x150 kernel/sched/core.c:7108
irqentry_exit+0x6f/0x90 kernel/entry/common.c:307
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:lock_is_held_type+0x137/0x190 kernel/locking/lockdep.c:5948
Code: 01 75 44 48 c7 04 24 00 00 00 00 9c 8f 04 24 f7 04 24 00 02 00 00 75 4c 41 f7 c4 00 02 00 00 74 01 fb 65 48 8b 05 89 a3 36 07 <48> 3b 44 24 08 75 43 89 d8 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f
RSP: 0018:ffffc9000459f8b8 EFLAGS: 00000206
RAX: d56c8fdc0a16c100 RBX: 0000000000000000 RCX: d56c8fdc0a16c100
RDX: 0000000000000000 RSI: ffffffff8db6ecbf RDI: ffffffff8be28b80
RBP: 00000000ffffffff R08: ffffffff8ec7e943 R09: 1ffffffff1d8fd28
R10: dffffc0000000000 R11: fffffbfff1d8fd29 R12: 0000000000000246
R13: ffff888026ce5a00 R14: ffffffff8e13ee60 R15: 0000000000000002
lock_is_held include/linux/lockdep.h:249 [inline]
__might_resched+0xf5/0x610 kernel/sched/core.c:8762
wg_ratelimiter_gc_entries+0x3a6/0x450 drivers/net/wireguard/ratelimiter.c:78
process_one_work kernel/workqueue.c:3238 [inline]
process_scheduled_works+0xae1/0x17b0 kernel/workqueue.c:3321
worker_thread+0x8a0/0xda0 kernel/workqueue.c:3402
kthread+0x70e/0x8a0 kernel/kthread.c:464
ret_from_fork+0x3f9/0x770 arch/x86/kernel/process.c:148
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
pool 2:
task:kworker/0:5 state:R running task stack:23824 pid:5892 tgid:5892 ppid:2 task_flags:0x4208060 flags:0x00004000
Workqueue: wg-crypt-wg2 wg_packet_decrypt_worker
Call Trace:
context_switch kernel/sched/core.c:5396 [inline]
__schedule+0x16f5/0x4d00 kernel/sched/core.c:6785
preempt_schedule_common+0x83/0xd0 kernel/sched/core.c:6965
preempt_schedule+0xae/0xc0 kernel/sched/core.c:6989
preempt_schedule_thunk+0x16/0x30 arch/x86/entry/thunk.S:12
__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:160 [inline]
_raw_spin_unlock_irq+0x45/0x50 kernel/locking/spinlock.c:202
process_one_work kernel/workqueue.c:3207 [inline]
process_scheduled_works+0x8ce/0x17b0 kernel/workqueue.c:3321
worker_thread+0x8a0/0xda0 kernel/workqueue.c:3402
kthread+0x70e/0x8a0 kernel/kthread.c:464
ret_from_fork+0x3f9/0x770 arch/x86/kernel/process.c:148
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
----------------
Code disassembly (best guess):
0: 01 75 44 add %esi,0x44(%rbp)
3: 48 c7 04 24 00 00 00 movq $0x0,(%rsp)
a: 00
b: 9c pushf
c: 8f 04 24 pop (%rsp)
f: f7 04 24 00 02 00 00 testl $0x200,(%rsp)
16: 75 4c jne 0x64
18: 41 f7 c4 00 02 00 00 test $0x200,%r12d
1f: 74 01 je 0x22
21: fb sti
22: 65 48 8b 05 89 a3 36 mov %gs:0x736a389(%rip),%rax # 0x736a3b3
29: 07
* 2a: 48 3b 44 24 08 cmp 0x8(%rsp),%rax <-- trapping instruction
2f: 75 43 jne 0x74
31: 89 d8 mov %ebx,%eax
33: 48 83 c4 10 add $0x10,%rsp
37: 5b pop %rbx
38: 41 5c pop %r12
3a: 41 5d pop %r13
3c: 41 5e pop %r14
3e: 41 5f pop %r15