BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 30s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=20 refcnt=21 in-flight: 5989:nsim_fib_event_work nsim_fib_event_work ,8:nsim_fib_event_work nsim_fib_event_work ,7:nsim_fib_event_work nsim_fib_event_work ,1631:request_firmware_work_func pending: nsim_dev_trap_report_work, 2*nsim_dev_hwstats_traffic_work, nsim_dev_trap_report_work, vmstat_shepherd, 2*l2cap_info_timeout, 4*ovs_dp_masks_rebalance, kfree_rcu_monitor, psi_avgs_work pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=19 refcnt=20 in-flight: 1201:nsim_fib_event_work nsim_fib_event_work ,5547:nsim_fib_event_work nsim_fib_event_work pending: nsim_dev_trap_report_work, 2*nsim_dev_hwstats_traffic_work, 2*nsim_dev_trap_report_work, nsim_dev_hwstats_traffic_work, 3*l2cap_info_timeout, ovs_dp_masks_rebalance, linkwatch_event, kfree_rcu_monitor, 2*psi_avgs_work, ovs_dp_masks_rebalance workqueue events_long: flags=0x0 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5 pending: 4*defense_work_handler pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: 2*defense_work_handler workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x4 nice=0 active=5 refcnt=6 pending: 4*cfg80211_wiphy_work, flush_memcg_stats_dwork pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: crng_reseed workqueue events_power_efficient: flags=0x82 pwq 8: cpus=0-1 flags=0x4 nice=0 active=2 refcnt=3 pending: wg_ratelimiter_gc_entries, neigh_periodic_work pwq 8: cpus=0-1 flags=0x4 nice=0 active=5 refcnt=6 pending: 2*neigh_managed_work, do_cache_clean, gc_worker, neigh_periodic_work workqueue mm_percpu_wq: flags=0x8 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wb_workfn workqueue dm_bufio_cache: flags=0x8 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: work_fn workqueue mld: flags=0x40008 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=31 pending: mld_ifc_work inactive: 15*mld_ifc_work, 2*mld_dad_work, 2*mld_ifc_work, mld_dad_work, 7*mld_ifc_work, mld_dad_work, mld_ifc_work pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=26 pending: mld_dad_work inactive: 7*mld_ifc_work, mld_dad_work, 7*mld_ifc_work, 3*mld_dad_work, 4*mld_ifc_work, mld_dad_work, mld_ifc_work workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=412 in-flight: 223:addrconf_dad_work inactive: 408*addrconf_dad_work workqueue krxrpcd: flags=0x2001a pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=9 pending: rxrpc_peer_keepalive_worker inactive: 5*rxrpc_peer_keepalive_worker workqueue bat_events: flags=0x6000a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=45 in-flight: 237:batadv_nc_worker inactive: 3*batadv_nc_worker, batadv_purge_orig, batadv_nc_worker, 3*batadv_iv_send_outstanding_bat_ogm_packet, batadv_mcast_mla_update, batadv_purge_orig, 4*batadv_mcast_mla_update, 8*batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, batadv_iv_send_outstanding_bat_ogm_packet, 2*batadv_purge_orig, 5*batadv_tt_purge, 2*batadv_dat_purge, 2*batadv_bla_periodic_work, 2*batadv_dat_purge, 2*batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work workqueue hci0: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci1: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci3: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci2: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci4: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg1: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-crypt-wg2: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-crypt-wg0: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-crypt-wg0: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-crypt-wg0: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-crypt-wg1: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-crypt-wg1: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg1: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x28 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=30s workers=8 idle: 6003 5594 5534 5992 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=30s workers=7 idle: 25 5958 5954 5957 23 pool 8: cpus=0-1 flags=0x4 nice=0 hung=30s workers=8 idle: 42 119 272 28 416 10 Showing backtraces of running workers in stalled CPU-bound worker pools: pool 2: task:kworker/0:5 state:R running task stack:0 pid:5989 tgid:5989 ppid:2 flags:0x00000008 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x314/0x560 arch/arm64/kernel/process.c:553 context_switch kernel/sched/core.c:5409 [inline] __schedule+0x14a4/0x24d4 kernel/sched/core.c:6746 __schedule_loop kernel/sched/core.c:6823 [inline] schedule+0xbc/0x238 kernel/sched/core.c:6838 schedule_timeout+0x1d8/0x348 kernel/time/timer.c:2582 schedule_timeout_uninterruptible+0x74/0xac kernel/time/timer.c:2616 msleep+0x48/0x6c kernel/time/timer.c:2737 nsim_fib6_rt_add drivers/net/netdevsim/fib.c:693 [inline] nsim_fib6_rt_insert drivers/net/netdevsim/fib.c:759 [inline] nsim_fib6_event drivers/net/netdevsim/fib.c:856 [inline] nsim_fib_event drivers/net/netdevsim/fib.c:889 [inline] nsim_fib_event_work+0x247c/0x32bc drivers/net/netdevsim/fib.c:1492 process_one_work+0x7b8/0x15d4 kernel/workqueue.c:3254 process_scheduled_works kernel/workqueue.c:3335 [inline] worker_thread+0x970/0xef4 kernel/workqueue.c:3416 kthread+0x288/0x310 kernel/kthread.c:388 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860 pool 2: task:kworker/0:1 state:R running task stack:0 pid:8 tgid:8 ppid:2 flags:0x00000008 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x314/0x560 arch/arm64/kernel/process.c:553 context_switch kernel/sched/core.c:5409 [inline] __schedule+0x14a4/0x24d4 kernel/sched/core.c:6746 __schedule_loop kernel/sched/core.c:6823 [inline] schedule+0xbc/0x238 kernel/sched/core.c:6838 schedule_timeout+0x1d8/0x348 kernel/time/timer.c:2582 schedule_timeout_uninterruptible+0x74/0xac kernel/time/timer.c:2616 msleep+0x48/0x6c kernel/time/timer.c:2737 nsim_fib4_rt_add drivers/net/netdevsim/fib.c:369 [inline] nsim_fib4_rt_insert drivers/net/netdevsim/fib.c:432 [inline] nsim_fib4_event drivers/net/netdevsim/fib.c:464 [inline] nsim_fib_event drivers/net/netdevsim/fib.c:884 [inline] nsim_fib_event_work+0x18f0/0x32bc drivers/net/netdevsim/fib.c:1492 process_one_work+0x7b8/0x15d4 kernel/workqueue.c:3254 process_scheduled_works kernel/workqueue.c:3335 [inline] worker_thread+0x970/0xef4 kernel/workqueue.c:3416 kthread+0x288/0x310 kernel/kthread.c:388 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860 pool 2: task:kworker/0:0 state:R running task stack:0 pid:7 tgid:7 ppid:2 flags:0x00000008 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x314/0x560 arch/arm64/kernel/process.c:553 context_switch kernel/sched/core.c:5409 [inline] __schedule+0x14a4/0x24d4 kernel/sched/core.c:6746 __schedule_loop kernel/sched/core.c:6823 [inline] schedule+0xbc/0x238 kernel/sched/core.c:6838 schedule_timeout+0x1d8/0x348 kernel/time/timer.c:2582 schedule_timeout_uninterruptible+0x74/0xac kernel/time/timer.c:2616 msleep+0x48/0x6c kernel/time/timer.c:2737 nsim_fib4_rt_add drivers/net/netdevsim/fib.c:369 [inline] nsim_fib4_rt_insert drivers/net/netdevsim/fib.c:432 [inline] nsim_fib4_event drivers/net/netdevsim/fib.c:464 [inline] nsim_fib_event drivers/net/netdevsim/fib.c:884 [inline] nsim_fib_event_work+0x18f0/0x32bc drivers/net/netdevsim/fib.c:1492 process_one_work+0x7b8/0x15d4 kernel/workqueue.c:3254 process_scheduled_works kernel/workqueue.c:3335 [inline] worker_thread+0x970/0xef4 kernel/workqueue.c:3416 kthread+0x288/0x310 kernel/kthread.c:388 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860 pool 2: task:kworker/0:2 state:R running task stack:0 pid:1631 tgid:1631 ppid:2 flags:0x00000008 Workqueue: events request_firmware_work_func Call trace: __switch_to+0x314/0x560 arch/arm64/kernel/process.c:553 context_switch kernel/sched/core.c:5409 [inline] __schedule+0x14a4/0x24d4 kernel/sched/core.c:6746 __schedule_loop kernel/sched/core.c:6823 [inline] schedule+0xbc/0x238 kernel/sched/core.c:6838 schedule_timeout+0x1d8/0x348 kernel/time/timer.c:2582 do_wait_for_common+0x30c/0x468 kernel/sched/completion.c:95 __wait_for_common kernel/sched/completion.c:116 [inline] wait_for_common kernel/sched/completion.c:127 [inline] wait_for_completion_killable_timeout+0x50/0x74 kernel/sched/completion.c:288 __fw_state_wait_common drivers/base/firmware_loader/firmware.h:104 [inline] fw_sysfs_wait_timeout drivers/base/firmware_loader/fallback.c:44 [inline] fw_load_sysfs_fallback drivers/base/firmware_loader/fallback.c:110 [inline] fw_load_from_user_helper drivers/base/firmware_loader/fallback.c:162 [inline] firmware_fallback_sysfs+0x528/0x918 drivers/base/firmware_loader/fallback.c:238 _request_firmware+0x7c8/0xb78 drivers/base/firmware_loader/main.c:914 request_firmware_work_func+0xfc/0x214 drivers/base/firmware_loader/main.c:1165 process_one_work+0x7b8/0x15d4 kernel/workqueue.c:3254 process_scheduled_works kernel/workqueue.c:3335 [inline] worker_thread+0x938/0xef4 kernel/workqueue.c:3416 kthread+0x288/0x310 kernel/kthread.c:388 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860