BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 44s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=12 refcnt=13 in-flight: 11:request_firmware_work_func for 116s pending: nsim_fib_event_work, free_obj_work, 2*psi_avgs_work, vmstat_shepherd, drm_fb_helper_damage_work, 5*ovs_dp_masks_rebalance pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=8 refcnt=9 in-flight: 4807:nsim_fib_event_work for 15s ,26:nsim_fib_event_work for 0s pending: rht_deferred_worker, 3*nsim_dev_hwstats_traffic_work, ovs_dp_masks_rebalance, psi_avgs_work workqueue events_long: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5 pending: 4*defense_work_handler pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=6 refcnt=7 pending: 6*defense_work_handler workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=4 refcnt=5 in-flight: 4854:cfg80211_wiphy_work for 81s cfg80211_wiphy_work ,39:cfg80211_wiphy_work for 81s cfg80211_wiphy_work pwq 8: cpus=0-1 flags=0x6 nice=0 active=8 refcnt=9 in-flight: 926:nsim_dev_trap_report_work for 0s ,4862:cfg80211_wiphy_work for 81s cfg80211_wiphy_work ,14:cfg80211_wiphy_work for 81s cfg80211_wiphy_work pending: nsim_dev_trap_report_work, toggle_allocation_gate, nsim_dev_trap_report_work workqueue events_power_efficient: flags=0x182 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 4955:neigh_periodic_work for 37s pwq 8: cpus=0-1 flags=0x6 nice=0 active=4 refcnt=5 in-flight: 1453:gc_worker for 0s ,40:reg_check_chans_work for 15s pending: do_cache_clean, fb_flashcursor workqueue netns: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=4 in-flight: 1184:cleanup_net for 83s workqueue mm_percpu_wq: flags=0x108 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 4956:wb_workfn for 75s workqueue kblockd: flags=0x118 pwq 7: cpus=1 node=0 flags=0x0 nice=-20 active=1 refcnt=2 pending: blk_mq_requeue_work workqueue mld: flags=0x40108 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=32 in-flight: 9:mld_dad_work for 44s inactive: 30*mld_dad_work pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=129 pending: mld_dad_work inactive: mld_ifc_work, mld_dad_work, 2*mld_ifc_work, mld_dad_work, mld_ifc_work, mld_dad_work, mld_ifc_work, 2*mld_dad_work, mld_ifc_work, mld_dad_work, 3*mld_ifc_work, 2*mld_dad_work, 5*mld_ifc_work, 4*mld_dad_work, 2*mld_ifc_work, mld_dad_work, 8*mld_ifc_work, mld_dad_work, mld_ifc_work, mld_dad_work, 5*mld_ifc_work, 2*mld_dad_work, mld_ifc_work, mld_dad_work, 2*mld_ifc_work, 2*mld_dad_work, mld_ifc_work, 3*mld_dad_work, mld_ifc_work, 2*mld_dad_work, 2*mld_ifc_work, 2*mld_dad_work, 3*mld_ifc_work, 3*mld_dad_work, 4*mld_ifc_work, 4*mld_dad_work, 4*mld_ifc_work, 45*mld_dad_work workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=88 MAYDAY in-flight: 2819(RESCUER):addrconf_dad_work for 20s inactive: 82*addrconf_dad_work workqueue bat_events: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=50 MAYDAY in-flight: 2854(RESCUER):batadv_tt_purge for 14s inactive: 2*batadv_tt_purge, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, 6*batadv_mcast_mla_update, 5*batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, 10*batadv_iv_send_outstanding_bat_ogm_packet, 5*batadv_purge_orig, 3*batadv_tt_purge workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 in-flight: 4957:wg_packet_encrypt_worker for 45s pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 866:wg_packet_handshake_receive_worker for 44s pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_tx_worker workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 1094:wg_packet_handshake_receive_worker for 1s workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 12:wg_packet_handshake_send_worker for 2s workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 24:wg_packet_handshake_receive_worker for 1s workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_tx_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_tx_worker pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=44s workers=7 idle: 4644 4806 4752 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=0s workers=6 idle: 4924 4818 pool 8: cpus=0-1 flags=0x6 nice=0 hung=0s workers=12 manager: 4958 Showing backtraces of busy workers in stalled worker pools: pool 2: task:kworker/0:1 state:R running task stack:0 pid:11 tgid:11 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: events request_firmware_work_func Call trace: __switch_to+0x2b0/0x6e0 arch/arm64/kernel/process.c:810 (T) context_switch kernel/sched/core.c:5388 [inline] __schedule+0x1c04/0x2db8 kernel/sched/core.c:7189 __schedule_loop kernel/sched/core.c:7268 [inline] schedule+0x13c/0x20c kernel/sched/core.c:7283 schedule_timeout+0x13c/0x28c kernel/time/sleep_timeout.c:99 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common+0x220/0x394 kernel/sched/completion.c:132 wait_for_completion_killable_timeout+0x14/0x20 kernel/sched/completion.c:293 __fw_state_wait_common drivers/base/firmware_loader/firmware.h:104 [inline] fw_sysfs_wait_timeout drivers/base/firmware_loader/fallback.c:44 [inline] fw_load_sysfs_fallback drivers/base/firmware_loader/fallback.c:110 [inline] fw_load_from_user_helper drivers/base/firmware_loader/fallback.c:162 [inline] firmware_fallback_sysfs+0x4d4/0x918 drivers/base/firmware_loader/fallback.c:238 _request_firmware+0xb34/0xf5c drivers/base/firmware_loader/main.c:898 request_firmware_work_func+0xa4/0x19c drivers/base/firmware_loader/main.c:1150 process_one_work kernel/workqueue.c:3314 [inline] process_scheduled_works+0x788/0x10b8 kernel/workqueue.c:3397 worker_thread+0x798/0xbd0 kernel/workqueue.c:3478 kthread+0x304/0x3d4 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:842 pool 2: task:kworker/0:2 state:R running task stack:0 pid:866 tgid:866 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker Call trace: __switch_to+0x2b0/0x6e0 arch/arm64/kernel/process.c:810 (T) context_switch kernel/sched/core.c:5388 [inline] __schedule+0x1c04/0x2db8 kernel/sched/core.c:7189 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7370 preempt_schedule+0x60/0x78 kernel/sched/core.c:7394 __local_bh_enable_ip+0x20c/0x35c kernel/softirq.c:457 local_bh_enable include/linux/bottom_half.h:33 [inline] rcu_read_unlock_bh include/linux/rcupdate.h:912 [inline] mod_peer_timer+0x21c/0x25c drivers/net/wireguard/timers.c:38 wg_timers_any_authenticated_packet_traversal+0x98/0xc0 drivers/net/wireguard/timers.c:218 wg_packet_send_handshake_response+0x120/0x264 drivers/net/wireguard/send.c:99 wg_receive_handshake_packet drivers/net/wireguard/receive.c:154 [inline] wg_packet_handshake_receive_worker+0x570/0xcf8 drivers/net/wireguard/receive.c:213 process_one_work kernel/workqueue.c:3314 [inline] process_scheduled_works+0x788/0x10b8 kernel/workqueue.c:3397 worker_thread+0x798/0xbd0 kernel/workqueue.c:3478 kthread+0x304/0x3d4 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:842 pool 2: task:kworker/0:6 state:R running task stack:0 pid:4957 tgid:4957 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-crypt-wg0 wg_packet_encrypt_worker Call trace: __switch_to+0x2b0/0x6e0 arch/arm64/kernel/process.c:810 (T) context_switch kernel/sched/core.c:5388 [inline] __schedule+0x1c04/0x2db8 kernel/sched/core.c:7189 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7370 preempt_schedule+0x60/0x78 kernel/sched/core.c:7394 __kunmap_atomic include/linux/highmem-internal.h:247 [inline] sg_miter_stop+0x220/0x2cc lib/scatterlist.c:941 chacha20poly1305_crypt_sg_inplace+0x75c/0xc30 lib/crypto/chacha20poly1305.c:319 chacha20poly1305_encrypt_sg_inplace+0x54/0x70 lib/crypto/chacha20poly1305.c:340 encrypt_packet drivers/net/wireguard/send.c:216 [inline] wg_packet_encrypt_worker+0x74c/0x1290 drivers/net/wireguard/send.c:297 process_one_work kernel/workqueue.c:3314 [inline] process_scheduled_works+0x788/0x10b8 kernel/workqueue.c:3397 worker_thread+0x798/0xbd0 kernel/workqueue.c:3478 kthread+0x304/0x3d4 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:842 pool 2: task:kworker/0:0 state:R running task stack:0 pid:9 tgid:9 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: mld mld_dad_work Call trace: __switch_to+0x2b0/0x6e0 arch/arm64/kernel/process.c:810 (T) context_switch kernel/sched/core.c:5388 [inline] __schedule+0x1c04/0x2db8 kernel/sched/core.c:7189 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7370 preempt_schedule+0x60/0x78 kernel/sched/core.c:7394 __local_bh_enable_ip+0x20c/0x35c kernel/softirq.c:457 __raw_spin_unlock_bh include/linux/spinlock_api_smp.h:196 [inline] _raw_spin_unlock_bh+0x3c/0x4c kernel/locking/spinlock.c:214 spin_unlock_bh include/linux/spinlock.h:396 [inline] rt6_uncached_list_add+0x10c/0x128 net/ipv6/route.c:146 icmp6_dst_alloc+0x1d0/0x260 net/ipv6/route.c:3362 mld_sendpack+0x680/0xffc net/ipv6/mcast.c:1844 mld_send_initial_cr+0x238/0x2fc net/ipv6/mcast.c:2271 mld_dad_work+0x40/0x1d8 net/ipv6/mcast.c:2294 process_one_work kernel/workqueue.c:3314 [inline] process_scheduled_works+0x788/0x10b8 kernel/workqueue.c:3397 worker_thread+0x798/0xbd0 kernel/workqueue.c:3478 kthread+0x304/0x3d4 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:842