BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 436s!
BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 436s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 refcnt=4
    in-flight: 25751:nsim_dev_trap_report_work
    pending: nsim_dev_hwstats_traffic_work, psi_avgs_work
  pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=5/256 refcnt=6
    in-flight: 24245:nsim_dev_trap_report_work
    pending: nsim_dev_hwstats_traffic_work, psi_avgs_work, vmstat_shepherd, psi_avgs_work
workqueue events_unbound: flags=0x2
  pwq 6: cpus=0-1 node=0 flags=0x4 nice=0 active=2/512 refcnt=3
    pending: toggle_allocation_gate, idle_cull_fn
  pwq 5: cpus=0-1 node=0 flags=0x4 nice=0 active=5/512 refcnt=6
    pending: flush_memcg_stats_dwork, crng_reseed, 3*idle_cull_fn
workqueue events_power_efficient: flags=0x80
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
    pending: wg_ratelimiter_gc_entries, check_lifetime
  pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=5/256 refcnt=6
    pending: 2*neigh_managed_work, 2*neigh_periodic_work, gc_worker
workqueue mm_percpu_wq: flags=0x8
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: vmstat_update
  pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: vmstat_update
workqueue writeback: flags=0x4a
  pwq 6: cpus=0-1 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
    pending: wb_workfn
workqueue dm_bufio_cache: flags=0x8
  pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: work_fn
workqueue ipv6_addrconf: flags=0x40008
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/1 refcnt=4
    pending: addrconf_verify_work
    inactive: 2*addrconf_verify_work
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 6: cpus=0-1 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
    pending: wg_queued_expired_zero_key_material
workqueue wg-crypt-wg0: flags=0x28
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 6: cpus=0-1 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
    pending: wg_queued_expired_zero_key_material
  pwq 5: cpus=0-1 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
    pending: wg_queued_expired_zero_key_material
workqueue wg-crypt-wg1: flags=0x28
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 6: cpus=0-1 node=0 flags=0x4 nice=0 active=1/256 refcnt=2
    pending: wg_queued_expired_zero_key_material
workqueue wg-crypt-wg2: flags=0x28
  pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
    pending: wg_packet_encrypt_worker
pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=437s workers=5 idle: 3571 32061 21121 29761
pool 2: cpus=1 node=0 flags=0x0 nice=0 hung=436s workers=5 idle: 24599 20891 12468 1802
Showing backtraces of running workers in stalled CPU-bound worker pools:
pool 0:
task:kworker/0:4     state:R  running task     stack:0     pid:24245 ppid:2      flags:0x00000008
Workqueue: events nsim_dev_trap_report_work
Call Trace:
[<ffffffff8000b904>] dump_backtrace+0x2e/0x3c arch/riscv/kernel/stacktrace.c:121
[<ffffffff836750b8>] show_stack+0x34/0x40 arch/riscv/kernel/stacktrace.c:127
[<ffffffff800f612e>] sched_show_task kernel/sched/core.c:9182 [inline]
[<ffffffff800f612e>] sched_show_task+0x2ee/0x414 kernel/sched/core.c:9156
[<ffffffff800cd62e>] show_cpu_pool_hog kernel/workqueue.c:6339 [inline]
[<ffffffff800cd62e>] show_cpu_pools_hogs kernel/workqueue.c:6359 [inline]
[<ffffffff800cd62e>] wq_watchdog_timer_fn+0x854/0x940 kernel/workqueue.c:6436
[<ffffffff801d1a4e>] call_timer_fn+0x15a/0x4f2 kernel/time/timer.c:1700
[<ffffffff801d2364>] expire_timers kernel/time/timer.c:1751 [inline]
[<ffffffff801d2364>] __run_timers+0x57e/0x73c kernel/time/timer.c:2022
[<ffffffff801d2594>] run_timer_softirq+0x72/0x80 kernel/time/timer.c:2037
[<ffffffff836cbb0e>] __do_softirq+0x2ee/0x8a2 kernel/softirq.c:553
[<ffffffff8008ae84>] invoke_softirq kernel/softirq.c:427 [inline]
[<ffffffff8008ae84>] __irq_exit_rcu+0xfa/0x1b0 kernel/softirq.c:632
[<ffffffff8008b11a>] irq_exit_rcu+0x10/0x72 kernel/softirq.c:644
[<ffffffff836b650c>] handle_riscv_irq+0x40/0x4c arch/riscv/kernel/traps.c:357
[<ffffffff836b6bc0>] do_irq+0x5c/0x88 arch/riscv/kernel/traps.c:367
pool 2:
task:kworker/1:6     state:R  running task     stack:0     pid:25751 ppid:2      flags:0x00000008
Workqueue: events nsim_dev_trap_report_work
Call Trace:
[<ffffffff836bdd90>] context_switch kernel/sched/core.c:5382 [inline]
[<ffffffff836bdd90>] __schedule+0x794/0x1884 kernel/sched/core.c:6695