BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 37s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 867:request_firmware_work_func for 146s pending: psi_avgs_work pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=16 refcnt=17 in-flight: 26:xfrm_state_gc_task for 85s pending: nsim_dev_hwstats_traffic_work, delayed_vfree_work, free_ipc, psi_avgs_work, 3*ovs_dp_masks_rebalance, 3*destroy_super_work, free_obj_work, 4*psi_avgs_work workqueue events_long: flags=0x100 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=4 refcnt=5 pending: 4*defense_work_handler workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=15 refcnt=16 in-flight: 12:cfg80211_wiphy_work for 103s cfg80211_wiphy_work ,189:cfg80211_wiphy_work for 109s cfg80211_wiphy_work ,1398:cfg80211_wiphy_work for 112s cfg80211_wiphy_work ,1478:toggle_allocation_gate for 0s ,1303:cfg80211_wiphy_work for 105s cfg80211_wiphy_work pending: 3*nsim_dev_trap_report_work, flush_memcg_stats_dwork, 2*nsim_dev_trap_report_work pwq 8: cpus=0-1 flags=0x6 nice=0 active=5 refcnt=6 in-flight: 4189:cfg80211_wiphy_work for 93s cfg80211_wiphy_work ,15:cfg80211_wiphy_work for 28s ,4234:cfg80211_wiphy_work for 102s cfg80211_wiphy_work workqueue events_power_efficient: flags=0x182 pwq 8: cpus=0-1 flags=0x6 nice=0 active=5 refcnt=6 in-flight: 5145:gc_worker for 14s ,40:neigh_managed_work for 4s ,39:reg_check_chans_work for 46s pending: fb_flashcursor, wg_ratelimiter_gc_entries workqueue mm_percpu_wq: flags=0x108 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 1360:wb_workfn for 107s workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=6 MAYDAY in-flight: 2822(RESCUER):addrconf_verify_work for 17s workqueue bat_events: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=45 MAYDAY in-flight: 2857(RESCUER):batadv_tt_purge for 12s inactive: 2*batadv_dat_purge, 2*batadv_bla_periodic_work, 2*batadv_dat_purge, 2*batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, 6*batadv_iv_send_outstanding_bat_ogm_packet, 5*batadv_mcast_mla_update, 9*batadv_iv_send_outstanding_bat_ogm_packet, 5*batadv_purge_orig, 4*batadv_tt_purge workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 10:wg_packet_tx_worker for 0s pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_tx_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-kex-wg2: flags=0x124 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_decrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg0: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg1: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker workqueue wg-crypt-wg2: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 4805:wg_packet_handshake_receive_worker for 18s pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_tx_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 4662:wg_packet_handshake_receive_worker for 0s pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg1: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 4819:wg_packet_handshake_receive_worker for 0s pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 1095:wg_packet_handshake_receive_worker for 56s wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg0: flags=0x124 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg1: flags=0x124 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg1: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 4796:wg_packet_handshake_receive_worker for 47s wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 24:wg_packet_handshake_receive_worker for 57s pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_decrypt_worker, wg_packet_tx_worker pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=0s workers=11 idle: 4803 4804 9 4904 4814 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=37s workers=7 idle: 5144 5143 5146 1004 pool 8: cpus=0-1 flags=0x6 nice=0 hung=0s workers=13 manager: 5147 Showing backtraces of busy workers in stalled worker pools: pool 6: task:kworker/1:0 state:R running task stack:0 pid:24 tgid:24 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker Call trace: __switch_to+0x2b0/0x6e0 arch/arm64/kernel/process.c:810 (T) context_switch kernel/sched/core.c:5388 [inline] __schedule+0x1bec/0x2dcc kernel/sched/core.c:7189 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7370 preempt_schedule+0x60/0x78 kernel/sched/core.c:7394 __local_bh_enable_ip+0x20c/0x35c kernel/softirq.c:457 local_bh_enable include/linux/bottom_half.h:33 [inline] rcu_read_unlock_bh include/linux/rcupdate.h:912 [inline] mod_peer_timer+0x21c/0x25c drivers/net/wireguard/timers.c:38 wg_timers_any_authenticated_packet_traversal+0x98/0xc0 drivers/net/wireguard/timers.c:218 wg_receive_handshake_packet drivers/net/wireguard/receive.c:202 [inline] wg_packet_handshake_receive_worker+0x5f0/0xcb8 drivers/net/wireguard/receive.c:213 process_one_work kernel/workqueue.c:3314 [inline] process_scheduled_works+0x79c/0x1098 kernel/workqueue.c:3397 worker_thread+0x754/0xba0 kernel/workqueue.c:3478 kthread+0x2f8/0x3c8 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:842 pool 6: task:kworker/1:3 state:R running task stack:0 pid:1095 tgid:1095 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker Call trace: __switch_to+0x2b0/0x6e0 arch/arm64/kernel/process.c:810 (T) context_switch kernel/sched/core.c:5388 [inline] __schedule+0x1bec/0x2dcc kernel/sched/core.c:7189 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7370 preempt_schedule+0x60/0x78 kernel/sched/core.c:7394 __local_bh_enable_ip+0x20c/0x35c kernel/softirq.c:457 local_bh_enable include/linux/bottom_half.h:33 [inline] __alloc_skb+0x1c8/0x5f8 net/core/skbuff.c:697 alloc_skb include/linux/skbuff.h:1385 [inline] wg_socket_send_buffer_to_peer+0x44/0x168 drivers/net/wireguard/socket.c:191 wg_packet_send_handshake_response+0x168/0x260 drivers/net/wireguard/send.c:103 wg_receive_handshake_packet drivers/net/wireguard/receive.c:154 [inline] wg_packet_handshake_receive_worker+0x550/0xcb8 drivers/net/wireguard/receive.c:213 process_one_work kernel/workqueue.c:3314 [inline] process_scheduled_works+0x79c/0x1098 kernel/workqueue.c:3397 worker_thread+0x754/0xba0 kernel/workqueue.c:3478 kthread+0x2f8/0x3c8 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:842 pool 6: task:kworker/1:1 state:R running task stack:0 pid:26 tgid:26 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: events xfrm_state_gc_task Call trace: __switch_to+0x2b0/0x6e0 arch/arm64/kernel/process.c:810 (T) context_switch kernel/sched/core.c:5388 [inline] __schedule+0x1bec/0x2dcc kernel/sched/core.c:7189 __schedule_loop kernel/sched/core.c:7268 [inline] schedule+0x140/0x218 kernel/sched/core.c:7283 schedule_timeout+0xc0/0x28c kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common+0x204/0x380 kernel/sched/completion.c:132 wait_for_completion+0x18/0x24 kernel/sched/completion.c:153 synchronize_rcu_normal+0x1bc/0x298 kernel/rcu/tree.c:3304 synchronize_rcu+0x1ec/0x250 kernel/rcu/tree.c:3362 xfrm_state_gc_task+0xd8/0x558 net/xfrm/xfrm_state.c:633 process_one_work kernel/workqueue.c:3314 [inline] process_scheduled_works+0x79c/0x1098 kernel/workqueue.c:3397 worker_thread+0x754/0xba0 kernel/workqueue.c:3478 kthread+0x2f8/0x3c8 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:842 bridge0: received packet on veth1_to_bridge with own address as source address (addr:fe:1b:e8:35:52:67, vlan:0) bridge0: received packet on bridge_slave_1 with own address as source address (addr:aa:aa:aa:aa:aa:0c, vlan:0) bridge0: received packet on bridge_slave_1 with own address as source address (addr:aa:aa:aa:aa:aa:0c, vlan:0)