BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 257s!
BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=-20 stuck for 199s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x100
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=25 refcnt=26
    pending: drm_fb_helper_damage_work, 10*nsim_dev_hwstats_traffic_work, 3*psi_avgs_work, send_dm_alert, vmstat_shepherd, 2*psi_avgs_work, 6*ovs_dp_masks_rebalance, rht_deferred_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=9 refcnt=10
    pending: psi_avgs_work, send_dm_alert, switchdev_deferred_process_work, rht_deferred_worker, psi_avgs_work, ovs_dp_masks_rebalance, 3*rht_deferred_worker
workqueue events_highpri: flags=0x110
  pwq 3: cpus=0 node=0 flags=0x0 nice=-20 active=1 refcnt=2
    pending: fill_page_cache_func
  pwq 7: cpus=1 node=0 flags=0x0 nice=-20 active=1 refcnt=2
    pending: fill_page_cache_func
workqueue events_long: flags=0x100
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8
    pending: 4*defense_work_handler, br_multicast_gc_work, 2*br_fdb_cleanup
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=12 refcnt=13
    pending: 3*defense_work_handler, 5*br_multicast_gc_work, 4*br_fdb_cleanup
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=3 refcnt=4
    in-flight: 3465:call_usermodehelper_exec_work
    pending: toggle_allocation_gate, flush_memcg_stats_dwork
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=28 refcnt=29
    in-flight: 4302:bpf_map_free_deferred ,50:nsim_dev_trap_report_work
    pending: 6*cfg80211_wiphy_work, 9*nsim_dev_trap_report_work, 4*macvlan_process_broadcast, cfg80211_wiphy_work, macvlan_process_broadcast, 2*idle_cull_fn, macvlan_process_broadcast, 2*idle_cull_fn
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=12 refcnt=13
    pending: 4*cfg80211_wiphy_work, 5*macvlan_process_broadcast, crng_reseed, idle_cull_fn, macvlan_process_broadcast
workqueue events_power_efficient: flags=0x180
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=5 refcnt=6
    in-flight: 10:gc_worker
    pending: fb_flashcursor, 3*check_lifetime
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=10 refcnt=11
    in-flight: 5929:wg_ratelimiter_gc_entries
    pending: 2*neigh_managed_work, 2*neigh_periodic_work, do_cache_clean, 4*check_lifetime
workqueue kvfree_rcu_reclaim: flags=0xa
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: kfree_rcu_monitor
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: kfree_rcu_monitor
workqueue mm_percpu_wq: flags=0x8
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: vmstat_update
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: vmstat_update
workqueue writeback: flags=0x4a
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wb_workfn
workqueue kblockd: flags=0x18
  pwq 3: cpus=0 node=0 flags=0x0 nice=-20 active=1 refcnt=2
    pending: blk_mq_timeout_work
workqueue ipv6_addrconf: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=10
    pending: addrconf_verify_work
    inactive: 6*addrconf_verify_work
workqueue krxrpcd: flags=0x2001a
  pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=10
    pending: rxrpc_peer_keepalive_worker
    inactive: 6*rxrpc_peer_keepalive_worker
workqueue bat_events: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=53
    in-flight: 6050:batadv_tt_purge
    inactive: 2*batadv_tt_purge, 6*batadv_iv_send_outstanding_bat_ogm_packet, 3*batadv_purge_orig, 5*batadv_iv_send_outstanding_bat_ogm_packet, 3*batadv_purge_orig, 4*batadv_iv_send_outstanding_bat_ogm_packet, 6*batadv_mcast_mla_update, 5*batadv_iv_send_outstanding_bat_ogm_packet, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, 2*batadv_bla_periodic_work, batadv_dat_purge, 3*batadv_tt_purge, batadv_dat_purge, batadv_bla_periodic_work
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_decrypt_worker, wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    in-flight: 12401:wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    in-flight: 6029:wg_packet_decrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=150s workers=18 idle: 11900 9930 12366 6823 11602 88 6064 5914 9816 7007 5906 5811 9 6017 9963
pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=258s workers=16 idle: 24 5928 6711 5963 6018 30 6166 9884 5882 6587 6057 6283 6103 13274 789
pool 8: cpus=0-1 flags=0x4 nice=0 hung=255s workers=33 idle: 6041 10142 6284 60 6157 3489 37 3530 6133 11530 6231 6214 6011 150 5988 3959 6027 6923 6220 12 6016 6992 8934 511 6200 13 9931 3645 6688
Showing backtraces of running workers in stalled CPU-bound worker pools:
pool 6:
task:kworker/1:5     state:R  running task     stack:22696 pid:5929  tgid:5929  ppid:2      task_flags:0x4208060 flags:0x00080000
Workqueue: events_power_efficient wg_ratelimiter_gc_entries
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5256 [inline]
 __schedule+0x1139/0x6150 kernel/sched/core.c:6863
 preempt_schedule_common+0x44/0xc0 kernel/sched/core.c:7047
 preempt_schedule_thunk+0x16/0x30 arch/x86/entry/thunk.S:12
 __raw_spin_unlock include/linux/spinlock_api_smp.h:143 [inline]
 _raw_spin_unlock+0x3e/0x50 kernel/locking/spinlock.c:186
 spin_unlock include/linux/spinlock.h:391 [inline]
 wg_ratelimiter_gc_entries+0x2b0/0x520 drivers/net/wireguard/ratelimiter.c:76
 process_one_work+0x9ba/0x1b20 kernel/workqueue.c:3257
 process_scheduled_works kernel/workqueue.c:3340 [inline]
 worker_thread+0x6c8/0xf10 kernel/workqueue.c:3421
 kthread+0x3c5/0x780 kernel/kthread.c:463
 ret_from_fork+0x983/0xb10 arch/x86/kernel/process.c:158
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
 </TASK>