sched: DL replenish lagged too much BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 30s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=5 refcnt=6 pending: psi_avgs_work, 2*nsim_dev_hwstats_traffic_work, 2*ovs_dp_masks_rebalance pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=14 refcnt=15 in-flight: 1830:request_firmware_work_func pending: 3*nsim_dev_hwstats_traffic_work, 4*psi_avgs_work, 4*ovs_dp_masks_rebalance, psi_avgs_work, free_obj_work workqueue events_long: flags=0x100 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: 3*defense_work_handler workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=14 refcnt=15 in-flight: 41:cfg80211_wiphy_work ,2072:cfg80211_wiphy_work ,5287:cfg80211_wiphy_work ,15:cfg80211_wiphy_work cfg80211_wiphy_work ,2032:cfg80211_wiphy_work ,258:cfg80211_wiphy_work pending: 2*cfg80211_wiphy_work, flush_memcg_stats_dwork, cfg80211_wiphy_work, toggle_allocation_gate, 2*cfg80211_wiphy_work pwq 8: cpus=0-1 flags=0x6 nice=0 active=5 refcnt=6 in-flight: 862:cfg80211_wiphy_work pending: crng_reseed, 3*cfg80211_wiphy_work workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=9 refcnt=10 in-flight: 42:nsim_dev_trap_report_work ,12:linkwatch_event pending: 2*macvlan_process_broadcast, nsim_dev_trap_report_work, fsnotify_connector_destroy_workfn, fsnotify_mark_destroy_workfn, 2*macvlan_process_broadcast pwq 8: cpus=0-1 flags=0x6 nice=0 active=9 refcnt=10 pending: 4*macvlan_process_broadcast, 3*nsim_dev_trap_report_work, 2*macvlan_process_broadcast workqueue events_power_efficient: flags=0x182 pwq 8: cpus=0-1 flags=0x6 nice=0 active=5 refcnt=6 pending: neigh_managed_work, do_cache_clean, gc_worker, 2*neigh_periodic_work pwq 8: cpus=0-1 flags=0x6 nice=0 active=3 refcnt=4 pending: fb_flashcursor, neigh_managed_work, wg_ratelimiter_gc_entries workqueue kvfree_rcu_reclaim: flags=0xa pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=3 MAYDAY pending: kfree_rcu_monitor workqueue mm_percpu_wq: flags=0x8 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4a pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=5 MAYDAY in-flight: 34(RESCUER):wb_workfn pending: wb_update_bandwidth_workfn workqueue tc_filter_workqueue: flags=0x20002 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=4 pending: flow_destroy_filter_work workqueue bat_events: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=42 in-flight: 4311(RESCUER):batadv_dat_purge inactive: batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, 2*batadv_tt_purge, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, 2*batadv_tt_purge, 5*batadv_mcast_mla_update, 2*batadv_iv_send_outstanding_bat_ogm_packet, 5*batadv_purge_orig, 13*batadv_iv_send_outstanding_bat_ogm_packet, batadv_tt_purge workqueue ext4-rsv-conversion: flags=0xa pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=4 in-flight: 6158(RESCUER):ext4_end_io_rsv_work inactive: ext4_end_io_rsv_work workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_tx_worker workqueue wg-crypt-wg1: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_decrypt_worker, wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x124 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 6630:wg_packet_handshake_receive_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg1: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-crypt-wg1: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-kex-wg2: flags=0x124 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 in-flight: 6608:wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-crypt-wg1: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_decrypt_worker, wg_packet_encrypt_worker workqueue wg-crypt-wg1: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_decrypt_worker, wg_packet_encrypt_worker workqueue wg-crypt-wg2: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=30s workers=10 idle: 26 6680 6615 6555 24 6928 6623 pool 8: cpus=0-1 flags=0x6 nice=0 hung=0s workers=10 manager: 7075 Showing backtraces of running workers in stalled CPU-bound worker pools: pool 6: task:kworker/1:2 state:R running task stack:0 pid:1830 tgid:1830 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: events request_firmware_work_func Call trace: __switch_to+0x418/0x87c arch/arm64/kernel/process.c:741 (T) context_switch kernel/sched/core.c:5325 [inline] __schedule+0x13b0/0x2864 kernel/sched/core.c:6929 preempt_schedule_common+0xe0/0x194 kernel/sched/core.c:7113 preempt_schedule+0x60/0x80 kernel/sched/core.c:7137 irq_work_queue+0xd0/0xfc kernel/irq_work.c:125 nbcon_kthreads_wake+0xe4/0x19c kernel/printk/nbcon.c:1290 vprintk_emit+0x1a4/0x318 kernel/printk/printk.c:2405 vprintk_default+0x54/0x80 kernel/printk/printk.c:2438 vprintk+0x90/0x128 kernel/printk/printk_safe.c:82 _printk+0xd0/0x118 kernel/printk/printk.c:2448 regdb_fw_cb+0x188/0x1a0 net/wireless/reg.c:-1 request_firmware_work_func+0xe8/0x19c drivers/base/firmware_loader/main.c:1161 process_one_work+0x7e8/0x155c kernel/workqueue.c:3263 process_scheduled_works kernel/workqueue.c:3346 [inline] worker_thread+0x958/0xed8 kernel/workqueue.c:3427 kthread+0x5fc/0x75c kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:844 pool 6: task:kworker/1:4 state:R running task stack:0 pid:6608 tgid:6608 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker Call trace: __switch_to+0x418/0x87c arch/arm64/kernel/process.c:741 (T) context_switch kernel/sched/core.c:5325 [inline] __schedule+0x13b0/0x2864 kernel/sched/core.c:6929 preempt_schedule_common+0xe0/0x194 kernel/sched/core.c:7113 preempt_schedule+0x60/0x80 kernel/sched/core.c:7137 __local_bh_enable_ip+0x20c/0x35c kernel/softirq.c:457 local_bh_enable+0x28/0x34 include/linux/bottom_half.h:33 rcu_read_unlock_bh include/linux/rcupdate.h:936 [inline] wg_packet_create_data drivers/net/wireguard/send.c:325 [inline] wg_packet_send_staged_packets+0xdac/0x1288 drivers/net/wireguard/send.c:388 wg_packet_send_keepalive+0x140/0x240 drivers/net/wireguard/send.c:239 wg_receive_handshake_packet drivers/net/wireguard/receive.c:186 [inline] wg_packet_handshake_receive_worker+0x54c/0xbf8 drivers/net/wireguard/receive.c:213 process_one_work+0x7e8/0x155c kernel/workqueue.c:3263 process_scheduled_works kernel/workqueue.c:3346 [inline] worker_thread+0x958/0xed8 kernel/workqueue.c:3427 kthread+0x5fc/0x75c kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:844 pool 6: task:kworker/1:7 state:R running task stack:0 pid:6630 tgid:6630 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker Call trace: __switch_to+0x418/0x87c arch/arm64/kernel/process.c:741 (T) context_switch kernel/sched/core.c:5325 [inline] __schedule+0x13b0/0x2864 kernel/sched/core.c:6929 preempt_schedule_common+0xe0/0x194 kernel/sched/core.c:7113 preempt_schedule+0x60/0x80 kernel/sched/core.c:7137 __local_bh_enable_ip+0x20c/0x35c kernel/softirq.c:457 local_bh_enable+0x28/0x34 include/linux/bottom_half.h:33 rcu_read_unlock_bh include/linux/rcupdate.h:936 [inline] wg_noise_handshake_begin_session+0x6cc/0x9c0 drivers/net/wireguard/noise.c:856 wg_packet_send_handshake_response+0xfc/0x24c drivers/net/wireguard/send.c:96 wg_receive_handshake_packet drivers/net/wireguard/receive.c:154 [inline] wg_packet_handshake_receive_worker+0x4f8/0xbf8 drivers/net/wireguard/receive.c:213 process_one_work+0x7e8/0x155c kernel/workqueue.c:3263 process_scheduled_works kernel/workqueue.c:3346 [inline] worker_thread+0x958/0xed8 kernel/workqueue.c:3427 kthread+0x5fc/0x75c kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:844