BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 51s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=16 refcnt=17 in-flight: 9:request_firmware_work_func for 117s ,4770:nsim_fib_event_work for 72s pending: 3*nsim_dev_hwstats_traffic_work, 2*psi_avgs_work, vmstat_shepherd, 6*ovs_dp_masks_rebalance, drm_fb_helper_damage_work, psi_avgs_work pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=5 refcnt=6 pending: 2*nsim_dev_hwstats_traffic_work, psi_avgs_work, free_obj_work, delayed_vfree_work workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=6 refcnt=7 in-flight: 1402:cfg80211_wiphy_work for 68s cfg80211_wiphy_work ,40:cfg80211_wiphy_work for 70s cfg80211_wiphy_work ,3330:cfg80211_wiphy_work for 72s cfg80211_wiphy_work pwq 8: cpus=0-1 flags=0x6 nice=0 active=14 refcnt=15 in-flight: 183:linkwatch_event for 71s ,5008:nsim_dev_trap_report_work for 0s ,3000:cfg80211_wiphy_work for 72s cfg80211_wiphy_work ,1180:cfg80211_wiphy_work for 73s cfg80211_wiphy_work ,54:cfg80211_wiphy_work for 65s cfg80211_wiphy_work pending: 2*nsim_dev_trap_report_work, toggle_allocation_gate, flush_memcg_stats_dwork, 2*nsim_dev_trap_report_work workqueue events_power_efficient: flags=0x82 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 5009:reg_check_chans_work for 3s pending: neigh_managed_work pwq 8: cpus=0-1 flags=0x6 nice=0 active=6 refcnt=7 in-flight: 1188:neigh_periodic_work for 13s pending: do_cache_clean, fb_flashcursor, neigh_periodic_work, wg_ratelimiter_gc_entries, gc_worker workqueue events_dfl_long: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=8 refcnt=9 pending: 8*defense_work_handler workqueue netns: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=4 in-flight: 58:cleanup_net for 75s workqueue mm_percpu_wq: flags=0x108 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wb_workfn workqueue gid-cache-wq: flags=0x20002 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=5 pending: netdevice_event_work_handler inactive: update_gid_event_work_handler workqueue mld: flags=0x40108 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=66 pending: mld_ifc_work inactive: mld_dad_work, mld_ifc_work, 3*mld_dad_work, 11*mld_ifc_work, mld_dad_work, 2*mld_ifc_work, mld_dad_work, 6*mld_ifc_work, 2*mld_dad_work, 12*mld_ifc_work, mld_dad_work, 2*mld_ifc_work, mld_dad_work, 16*mld_ifc_work, 2*mld_dad_work, 2*mld_ifc_work workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=344 in-flight: 12:addrconf_dad_work for 73s inactive: 340*addrconf_dad_work workqueue bat_events: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=48 MAYDAY in-flight: 2854(RESCUER):batadv_tt_purge for 4s inactive: 2*batadv_tt_purge, batadv_mcast_mla_update, 3*batadv_purge_orig, 2*batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, batadv_iv_send_outstanding_bat_ogm_packet, batadv_mcast_mla_update, 5*batadv_iv_send_outstanding_bat_ogm_packet, 2*batadv_purge_orig, 6*batadv_iv_send_outstanding_bat_ogm_packet, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_mcast_mla_update, 2*batadv_tt_purge, 3*batadv_mcast_mla_update workqueue udp_tunnel_nic: flags=0x20002 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=6 pending: udp_tunnel_nic_device_sync_work inactive: 2*udp_tunnel_nic_device_sync_work workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3 in-flight: 4955:wg_packet_handshake_receive_worker for 15s pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 13:wg_packet_handshake_send_worker for 1s workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=3 in-flight: 4896(RESCUER):wg_packet_decrypt_worker for 0s pending: mayday_cursor_func workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=2 in-flight: 26:wg_packet_handshake_receive_worker for 13s workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3 in-flight: 1092:wg_packet_handshake_receive_worker for 25s pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3 in-flight: 4934:wg_packet_encrypt_worker for 25s wg_packet_encrypt_worker workqueue wg-kex-wg1: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3 in-flight: 4840:wg_packet_handshake_receive_worker for 15s pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3 in-flight: 24:wg_packet_handshake_receive_worker for 15s pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-crypt-wg0: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=4 MAYDAY in-flight: 4914(RESCUER):wg_packet_encrypt_worker for 0s pending: mayday_cursor_func workqueue wg-crypt-wg1: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-crypt-wg2: flags=0x128 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=51s workers=8 idle: 10 1021 4469 4990 4941 4771 pool 6: cpus=1 node=0 flags=0x2 nice=0 hung=0s workers=7 manager: 866 pool 8: cpus=0-1 flags=0x6 nice=0 hung=0s workers=14 manager: 5010 Showing backtraces of busy workers in stalled worker pools: pool 2: task:kworker/0:0 state:R running task stack:0 pid:9 tgid:9 ppid:2 task_flags:0x4288060 flags:0x00000010 Workqueue: events request_firmware_work_func Call trace: __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:775 (T) context_switch kernel/sched/core.c:5520 [inline] __schedule+0x1370/0x2d80 kernel/sched/core.c:7270 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7449 preempt_schedule+0x60/0x78 kernel/sched/core.c:7473 rcu_read_unlock_sched include/linux/rcupdate.h:974 [inline] pfn_valid+0x210/0x214 include/linux/mmzone.h:-1 lowmem_page_address include/linux/mm.h:3111 [inline] __kasan_poison_pages+0x40/0x8c mm/kasan/common.c:151 kasan_poison_pages include/linux/kasan.h:118 [inline] __free_pages_prepare mm/page_alloc.c:1440 [inline] __free_frozen_pages+0x4e0/0xc18 mm/page_alloc.c:2962 free_frozen_pages+0x14/0x20 mm/page_alloc.c:2999 __free_slab+0x11c/0x1b4 mm/slub.c:3528 free_slab+0x2c/0x154 mm/slub.c:3562 discard_slab+0x48/0x90 mm/slub.c:3568 __slab_free+0x1a0/0x1d0 mm/slub.c:5815 ___cache_free+0x80/0x94 mm/slub.c:6571 qlink_free mm/kasan/quarantine.c:163 [inline] qlist_free_all+0xb4/0x120 mm/kasan/quarantine.c:179 kasan_quarantine_reduce+0x118/0x124 mm/kasan/quarantine.c:286 __kasan_slab_alloc+0x2c/0x88 mm/kasan/common.c:350 kasan_slab_alloc include/linux/kasan.h:253 [inline] slab_post_alloc_hook mm/slub.c:4683 [inline] slab_alloc_node mm/slub.c:4996 [inline] __do_kmalloc_node mm/slub.c:5413 [inline] __kmalloc_noprof+0x2e8/0x6c8 mm/slub.c:5439 _kmalloc_noprof include/linux/slab.h:995 [inline] _kzalloc_noprof include/linux/slab.h:1312 [inline] kobject_get_path+0xbc/0x2a4 lib/kobject.c:161 kobject_uevent_env+0x264/0x944 lib/kobject_uevent.c:548 kobject_uevent+0x2c/0x3c lib/kobject_uevent.c:659 device_del+0x5f8/0x71c drivers/base/core.c:3983 fw_load_sysfs_fallback drivers/base/firmware_loader/fallback.c:134 [inline] fw_load_from_user_helper drivers/base/firmware_loader/fallback.c:172 [inline] firmware_fallback_sysfs+0x330/0x930 drivers/base/firmware_loader/fallback.c:248 _request_firmware+0xb34/0xf5c drivers/base/firmware_loader/main.c:898 request_firmware_work_func+0xa4/0x24c drivers/base/firmware_loader/main.c:1162 process_one_work kernel/workqueue.c:3387 [inline] process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3470 worker_thread+0x798/0xbd0 kernel/workqueue.c:3551 kthread+0x304/0x3d4 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:838 pool 2: task:kworker/0:4 state:R running task stack:0 pid:4770 tgid:4770 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:775 (T) context_switch kernel/sched/core.c:5520 [inline] __schedule+0x1370/0x2d80 kernel/sched/core.c:7270 __schedule_loop kernel/sched/core.c:7347 [inline] schedule+0x13c/0x20c kernel/sched/core.c:7362 schedule_timeout+0x13c/0x28c kernel/time/sleep_timeout.c:99 schedule_timeout_uninterruptible+0x78/0xbc kernel/time/sleep_timeout.c:158 msleep+0x3c/0x68 kernel/time/sleep_timeout.c:318 nsim_fib6_rt_add drivers/net/netdevsim/fib.c:693 [inline] nsim_fib6_rt_insert drivers/net/netdevsim/fib.c:759 [inline] nsim_fib6_event drivers/net/netdevsim/fib.c:856 [inline] nsim_fib_event+0x57a0/0x60c4 drivers/net/netdevsim/fib.c:889 nsim_fib_event_work+0x1d8/0x320 drivers/net/netdevsim/fib.c:1493 process_one_work kernel/workqueue.c:3387 [inline] process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3470 worker_thread+0x798/0xbd0 kernel/workqueue.c:3551 kthread+0x304/0x3d4 kernel/kthread.c:436 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:838