BUG: workqueue lockup - pool cpus=0 node=0 flags=0x2 nice=0 stuck for 40s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x100
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=9 refcnt=10
    in-flight: 3691:free_obj_work
    pending: psi_avgs_work, vmstat_shepherd, 2*psi_avgs_work, rtc_timer_do_work, 2*nsim_dev_hwstats_traffic_work, free_obj_work
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: flush_memcg_stats_dwork
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    in-flight: 161:nsim_dev_trap_report_work
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
    in-flight: 6033:linkwatch_event
    pending: nsim_dev_trap_report_work
workqueue events_power_efficient: flags=0x182
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=4 refcnt=5
    in-flight: 7436:wg_ratelimiter_gc_entries ,6019:gc_worker
    pending: check_lifetime, neigh_managed_work
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=3 refcnt=4
    pending: neigh_managed_work, 2*neigh_periodic_work
workqueue mm_percpu_wq: flags=0x8
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: vmstat_update
workqueue ipv6_addrconf: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=8 MAYDAY
    in-flight: 2070(RESCUER):addrconf_verify_work
    inactive: 2*addrconf_verify_work
workqueue wg-kex-wg0: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=2
    in-flight: 4387:wg_packet_handshake_receive_worker
workqueue wg-kex-wg1: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 3475:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=3 refcnt=4
    in-flight: 38:wg_packet_handshake_send_worker
    pending: 2*wg_packet_handshake_send_worker
workqueue wg-kex-wg2: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=1 refcnt=2
    in-flight: 55:wg_packet_handshake_receive_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    in-flight: 3757:wg_packet_handshake_send_worker
workqueue wg-kex-wg0: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 4533:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 7424:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    in-flight: 3412:wg_packet_handshake_send_worker
workqueue wg-kex-wg1: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 3371:wg_packet_handshake_receive_worker
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 7434:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
workqueue wg-kex-wg2: flags=0x124
  pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_receive_worker
  pwq 6: cpus=1 node=0 flags=0x2 nice=0 active=2 refcnt=3
    in-flight: 5835:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
    in-flight: 26:wg_packet_handshake_send_worker ,3421:wg_packet_handshake_send_worker
pool 2: cpus=0 node=0 flags=0x2 nice=0 hung=40s workers=6 manager: 7435 idle: 5840 6050
pool 6: cpus=1 node=0 flags=0x2 nice=0 hung=0s workers=7 manager: 6675
pool 8: cpus=0-1 flags=0x6 nice=0 hung=3s workers=10 manager: 6024
Showing backtraces of running workers in stalled CPU-bound worker pools:
pool 2:
task:kworker/0:6     state:R  running task     stack:0     pid:3691  tgid:3691  ppid:2      task_flags:0x4208060 flags:0x00000010
Workqueue: events free_obj_work
Call trace:
 __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:742 (T)
 context_switch kernel/sched/core.c:5256 [inline]
 __schedule+0xd1c/0x3278 kernel/sched/core.c:6863
 preempt_schedule_common kernel/sched/core.c:7047 [inline]
 preempt_schedule+0xf4/0x254 kernel/sched/core.c:7071
 rcu_read_unlock_sched include/linux/rcupdate.h:981 [inline]
 pfn_valid include/linux/mmzone.h:2193 [inline]
 lowmem_page_address include/linux/mm.h:2512 [inline]
 folio_address include/linux/mm.h:2523 [inline]
 slab_address mm/slab.h:163 [inline]
 nearest_obj mm/slab.h:295 [inline]
 check_slab_allocation+0x53c/0x5e8 mm/kasan/common.c:229
 __kasan_slab_pre_free+0x10/0x1c mm/kasan/common.c:261
 kasan_slab_pre_free include/linux/kasan.h:199 [inline]
 slab_free_hook mm/slub.c:2485 [inline]
 slab_free mm/slub.c:6670 [inline]
 kmem_cache_free+0x10c/0x774 mm/slub.c:6781
 free_object_list.isra.0+0xe0/0x268 lib/debugobjects.c:326
 free_obj_work+0x1ac/0x360 lib/debugobjects.c:513
 process_one_work+0x7cc/0x18d4 kernel/workqueue.c:3257
 process_scheduled_works kernel/workqueue.c:3340 [inline]
 worker_thread+0x734/0xb84 kernel/workqueue.c:3421
 kthread+0x348/0x5fc kernel/kthread.c:463
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860
pool 2:
task:kworker/0:2     state:R  running task     stack:0     pid:3371  tgid:3371  ppid:2      task_flags:0x4208060 flags:0x00000010
Workqueue: wg-kex-wg1 wg_packet_handshake_receive_worker
Call trace:
 __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:742 (T)
 context_switch kernel/sched/core.c:5256 [inline]
 __schedule+0xd1c/0x3278 kernel/sched/core.c:6863
 preempt_schedule_irq+0x80/0x1a0 kernel/sched/core.c:7190
 raw_irqentry_exit_cond_resched kernel/entry/common.c:173 [inline]
 raw_irqentry_exit_cond_resched+0x40/0x44 kernel/entry/common.c:165
 irqentry_exit+0x18c/0x474 kernel/entry/common.c:216
 exit_to_kernel_mode+0x10/0x1c arch/arm64/kernel/entry-common.c:58
 __el1_irq arch/arm64/kernel/entry-common.c:500 [inline]
 el1_interrupt+0x4c/0x60 arch/arm64/kernel/entry-common.c:510
 el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:515
 el1h_64_irq+0x6c/0x70 arch/arm64/kernel/entry.S:592
 __daif_local_irq_restore arch/arm64/include/asm/irqflags.h:175 [inline] (P)
 arch_local_irq_restore arch/arm64/include/asm/irqflags.h:195 [inline] (P)
 put_cpu_partial+0x1b8/0x364 mm/slub.c:3958 (P)
 __slab_free+0x270/0x348 mm/slub.c:5952
 do_slab_free mm/slub.c:6618 [inline]
 ___cache_free+0xc4/0xd8 mm/slub.c:6737
 qlink_free mm/kasan/quarantine.c:163 [inline]
 qlist_free_all+0x78/0x130 mm/kasan/quarantine.c:179
 kasan_quarantine_reduce+0x148/0x19c mm/kasan/quarantine.c:286
 __kasan_slab_alloc+0x60/0x90 mm/kasan/common.c:350
 kasan_slab_alloc include/linux/kasan.h:253 [inline]
 slab_post_alloc_hook mm/slub.c:4953 [inline]
 slab_alloc_node mm/slub.c:5263 [inline]
 __kmalloc_cache_noprof+0x260/0x710 mm/slub.c:5771
 kmalloc_noprof include/linux/slab.h:957 [inline]
 kzalloc_noprof include/linux/slab.h:1094 [inline]
 keypair_create drivers/net/wireguard/noise.c:100 [inline]
 wg_noise_handshake_begin_session+0xe0/0xa64 drivers/net/wireguard/noise.c:827
 wg_packet_send_handshake_response+0x1a8/0x250 drivers/net/wireguard/send.c:96
 wg_receive_handshake_packet+0x1b8/0x810 drivers/net/wireguard/receive.c:154
 wg_packet_handshake_receive_worker+0xd8/0x314 drivers/net/wireguard/receive.c:213
 process_one_work+0x7cc/0x18d4 kernel/workqueue.c:3257
 process_scheduled_works kernel/workqueue.c:3340 [inline]
 worker_thread+0x734/0xb84 kernel/workqueue.c:3421
 kthread+0x348/0x5fc kernel/kthread.c:463
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860
pool 2:
task:kworker/0:1     state:R  running task     stack:0     pid:4533  tgid:4533  ppid:2      task_flags:0x4208060 flags:0x00000010
Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker
Call trace:
 __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:742 (T)
 context_switch kernel/sched/core.c:5256 [inline]
 __schedule+0xd1c/0x3278 kernel/sched/core.c:6863
 preempt_schedule_common kernel/sched/core.c:7047 [inline]
 preempt_schedule+0xf4/0x254 kernel/sched/core.c:7071
 __local_bh_enable_ip+0x384/0x4a4 kernel/softirq.c:457
 local_bh_enable include/linux/bottom_half.h:33 [inline]
 rcu_read_unlock_bh include/linux/rcupdate.h:936 [inline]
 wg_pubkey_hashtable_lookup+0x138/0x290 drivers/net/wireguard/peerlookup.c:69
 wg_noise_handshake_consume_initiation+0x30c/0x634 drivers/net/wireguard/noise.c:617
 wg_receive_handshake_packet+0x19c/0x810 drivers/net/wireguard/receive.c:144
 wg_packet_handshake_receive_worker+0xd8/0x314 drivers/net/wireguard/receive.c:213
 process_one_work+0x7cc/0x18d4 kernel/workqueue.c:3257
 process_scheduled_works kernel/workqueue.c:3340 [inline]
 worker_thread+0x734/0xb84 kernel/workqueue.c:3421
 kthread+0x348/0x5fc kernel/kthread.c:463
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860