sched: RT throttling activated
BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 303s!
BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 303s!
BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=-20 stuck for 301s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=25 refcnt=26
    in-flight: 5159:nsim_dev_trap_report_work
    pending: nsim_dev_trap_report_work, ovs_dp_masks_rebalance, 4*nsim_dev_hwstats_traffic_work, 3*nsim_dev_trap_report_work, 4*ovs_dp_masks_rebalance, vmstat_shepherd, ovs_dp_masks_rebalance, psi_avgs_work, 2*ovs_dp_masks_rebalance, kfree_rcu_monitor, switchdev_deferred_process_work, 2*rht_deferred_worker, rt6_probe_deferred, rht_deferred_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=14 refcnt=15
    pending: 2*nsim_dev_hwstats_traffic_work, nsim_dev_trap_report_work, 5*psi_avgs_work, ovs_dp_masks_rebalance, kfree_rcu_monitor, ovs_dp_masks_rebalance, 3*rht_deferred_worker
workqueue events_highpri: flags=0x10
  pwq 3: cpus=0 node=0 flags=0x0 nice=-20 active=1 refcnt=2
    pending: fill_page_cache_func
  pwq 7: cpus=1 node=0 flags=0x0 nice=-20 active=1 refcnt=2
    pending: fill_page_cache_func
workqueue events_long: flags=0x0
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=11 refcnt=12
    pending: 5*defense_work_handler, 2*br_multicast_gc_work, 2*br_fdb_cleanup, br_multicast_gc_work, br_fdb_cleanup
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=14 refcnt=15
    pending: 5*defense_work_handler, 2*br_fdb_cleanup, br_multicast_gc_work, br_fdb_cleanup, 2*br_multicast_gc_work, 3*br_fdb_cleanup
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=4 refcnt=23
    pending: 4*cfg80211_wiphy_work
    inactive: 4*cfg80211_wiphy_work, 5*macvlan_process_broadcast, 2*cfg80211_wiphy_work, 4*macvlan_process_broadcast, toggle_allocation_gate, cfg80211_wiphy_work, idle_cull_fn
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=4 refcnt=18
    in-flight: 11224:cfg80211_wiphy_work ,11260:cfg80211_wiphy_work
    pending: flush_memcg_stats_dwork, crng_reseed
    inactive: 5*macvlan_process_broadcast, idle_cull_fn, 5*cfg80211_wiphy_work, 2*idle_cull_fn
workqueue events_power_efficient: flags=0x80
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
    pending: fb_flashcursor, gc_worker, do_cache_clean
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=7 refcnt=8
    pending: wg_ratelimiter_gc_entries, neigh_periodic_work, 2*neigh_managed_work, neigh_periodic_work, reg_check_chans_work, check_lifetime
workqueue netns: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=4
    pending: cleanup_net
workqueue mm_percpu_wq: flags=0x8
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: vmstat_update
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: vmstat_update
workqueue writeback: flags=0x4a
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=2 refcnt=3
    pending: 2*wb_workfn
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=3 refcnt=4
    pending: 3*wb_workfn
workqueue kblockd: flags=0x18
  pwq 3: cpus=0 node=0 flags=0x0 nice=-20 active=1 refcnt=2
    pending: blk_mq_timeout_work
  pwq 7: cpus=1 node=0 flags=0x0 nice=-20 active=2 refcnt=3
    pending: 2*blk_mq_timeout_work
workqueue dm_bufio_cache: flags=0x8
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: work_fn
workqueue mld: flags=0x40008
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: mld_dad_work
workqueue ipv6_addrconf: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=13
    pending: addrconf_verify_work
    inactive: 9*addrconf_verify_work
workqueue krxrpcd: flags=0x2001a
  pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=13
    pending: rxrpc_peer_keepalive_worker
    inactive: 9*rxrpc_peer_keepalive_worker
workqueue bat_events: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=84
    pending: batadv_nc_worker
    inactive: 7*batadv_nc_worker, 2*batadv_iv_send_outstanding_bat_ogm_packet, batadv_mcast_mla_update, 2*batadv_purge_orig, 2*batadv_mcast_mla_update, 5*batadv_iv_send_outstanding_bat_ogm_packet, 3*batadv_mcast_mla_update, 2*batadv_purge_orig, batadv_dat_purge, batadv_bla_periodic_work, 2*batadv_mcast_mla_update, 3*batadv_iv_send_outstanding_bat_ogm_packet, batadv_tt_purge, batadv_mcast_mla_update, 10*batadv_iv_send_outstanding_bat_ogm_packet, 4*batadv_purge_orig, 3*batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, 4*batadv_iv_send_outstanding_bat_ogm_packet, 2*batadv_tt_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_tt_purge, batadv_dat_purge, 2*batadv_bla_periodic_work, batadv_dat_purge, 5*batadv_tt_purge, batadv_bla_periodic_work, 2*batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, 2*batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_nc_worker
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue hci1: flags=0x20012
  pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4
    pending: hci_conn_timeout
workqueue hci6: flags=0x20012
  pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4
    pending: hci_conn_timeout
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue hci7: flags=0x20012
  pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4
    pending: hci_conn_timeout
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=305s workers=8 idle: 5162 5161 13132 5158 1146 8 14007
pool 8: cpus=0-1 flags=0x4 nice=0 hung=196s workers=13 idle: 61 2436 51 11 2459 10 5668 2447 11222 5622 34
Showing backtraces of running workers in stalled CPU-bound worker pools:
pool 2:
task:kworker/0:4     state:R  running task     stack:24944 pid:5159  tgid:5159  ppid:2      flags:0x00004000
Workqueue: events nsim_dev_trap_report_work
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5409 [inline]
 __schedule+0xf15/0x5d00 kernel/sched/core.c:6746
 preempt_schedule_common+0x44/0xc0 kernel/sched/core.c:6925
 preempt_schedule_thunk+0x1a/0x30 arch/x86/entry/thunk_64.S:12
 __raw_spin_unlock include/linux/spinlock_api_smp.h:143 [inline]
 _raw_spin_unlock+0x3e/0x50 kernel/locking/spinlock.c:186
 spin_unlock include/linux/spinlock.h:391 [inline]
 nsim_dev_trap_report drivers/net/netdevsim/dev.c:823 [inline]
 nsim_dev_trap_report_work+0x8b0/0xc80 drivers/net/netdevsim/dev.c:850
 process_one_work+0x9ac/0x1ac0 kernel/workqueue.c:3254
 process_scheduled_works kernel/workqueue.c:3335 [inline]
 worker_thread+0x6c8/0xf70 kernel/workqueue.c:3416
 kthread+0x2c4/0x3a0 kernel/kthread.c:388
 ret_from_fork+0x48/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>