sched: DL replenish lagged too much BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 38s! Showing busy workqueues and worker pools: workqueue events: flags=0x100 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=1 refcnt=2 pending: nsim_dev_hwstats_traffic_work pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=6 refcnt=7 in-flight: 31:nsim_fib_event_work pending: nsim_dev_hwstats_traffic_work, psi_avgs_work, nsim_fib_event_work, psi_avgs_work, drm_fb_helper_damage_work workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 157:linkwatch_event ,1104:nsim_dev_trap_report_work pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 41:nsim_dev_trap_report_work workqueue events_power_efficient: flags=0x182 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 168:wg_ratelimiter_gc_entries pending: gc_worker pwq 8: cpus=0-1 flags=0x6 nice=0 active=3 refcnt=4 in-flight: 12:neigh_managed_work pending: neigh_managed_work, check_lifetime workqueue mm_percpu_wq: flags=0x8 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=6 in-flight: 2072(RESCUER):addrconf_verify_work inactive: addrconf_verify_work workqueue wg-kex-wg0: flags=0x124 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3 in-flight: 3470:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 3695:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker workqueue wg-crypt-wg0: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_tx_worker workqueue wg-kex-wg1: flags=0x124 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg1: flags=0x128 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x124 pwq 2: cpus=0 node=0 flags=0x2 nice=0 active=2 refcnt=3 in-flight: 3699:wg_packet_handshake_receive_worker wg_packet_handshake_receive_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 in-flight: 3425:wg_packet_handshake_receive_worker pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg0: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2 in-flight: 26:wg_packet_handshake_send_worker workqueue wg-kex-wg1: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 in-flight: 63:wg_packet_handshake_send_worker pending: wg_packet_handshake_send_worker workqueue wg-kex-wg2: flags=0x6 pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3 pending: 2*wg_packet_handshake_send_worker pool 2: cpus=0 node=0 flags=0x2 nice=0 hung=0s workers=6 manager: 10 idle: 9 976 3696 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=38s workers=7 idle: 3698 3103 3472 23 pool 8: cpus=0-1 flags=0x6 nice=0 hung=0s workers=8 manager: 3697 Showing backtraces of running workers in stalled CPU-bound worker pools: pool 6: task:kworker/1:5 state:R running task stack:0 pid:3695 tgid:3695 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker Call trace: __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:741 (T) context_switch kernel/sched/core.c:5325 [inline] __schedule+0xd58/0x3180 kernel/sched/core.c:6929 preempt_schedule_common kernel/sched/core.c:7113 [inline] preempt_schedule+0xf4/0x254 kernel/sched/core.c:7137 __local_bh_enable_ip+0x384/0x4a4 kernel/softirq.c:457 local_bh_enable include/linux/bottom_half.h:33 [inline] rcu_read_unlock_bh include/linux/rcupdate.h:936 [inline] mod_peer_timer drivers/net/wireguard/timers.c:38 [inline] wg_timers_session_derived+0x140/0x218 drivers/net/wireguard/timers.c:208 wg_packet_send_handshake_response+0x1b8/0x250 drivers/net/wireguard/send.c:98 wg_receive_handshake_packet+0x1b8/0x810 drivers/net/wireguard/receive.c:154 wg_packet_handshake_receive_worker+0xd8/0x314 drivers/net/wireguard/receive.c:213 process_one_work+0x7cc/0x18d4 kernel/workqueue.c:3263 process_scheduled_works kernel/workqueue.c:3346 [inline] worker_thread+0x734/0xb84 kernel/workqueue.c:3427 kthread+0x348/0x5fc kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860 pool 6: task:kworker/1:3 state:R running task stack:0 pid:3425 tgid:3425 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker Call trace: __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:741 (T) context_switch kernel/sched/core.c:5325 [inline] __schedule+0xd58/0x3180 kernel/sched/core.c:6929 preempt_schedule_irq+0x80/0x19c kernel/sched/core.c:7256 raw_irqentry_exit_cond_resched kernel/entry/common.c:168 [inline] raw_irqentry_exit_cond_resched+0x40/0x44 kernel/entry/common.c:160 irqentry_exit+0x44/0x188 kernel/entry/common.c:211 __exit_to_kernel_mode arch/arm64/kernel/entry-common.c:68 [inline] exit_to_kernel_mode+0x10/0x1c arch/arm64/kernel/entry-common.c:75 __el1_irq arch/arm64/kernel/entry-common.c:522 [inline] el1_interrupt+0x4c/0x60 arch/arm64/kernel/entry-common.c:532 el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:537 el1h_64_irq+0x6c/0x70 arch/arm64/kernel/entry.S:592 __kasan_check_read+0x20/0x2c mm/kasan/shadow.c:31 (P) instrument_atomic_read include/linux/instrumented.h:68 [inline] atomic_read include/linux/atomic/atomic-instrumented.h:32 [inline] __page_table_check_zero+0x188/0x310 mm/page_table_check.c:143 page_table_check_free include/linux/page_table_check.h:43 [inline] free_pages_prepare mm/page_alloc.c:1395 [inline] __free_frozen_pages+0x650/0x1214 mm/page_alloc.c:2906 free_frozen_pages+0x14/0x20 mm/page_alloc.c:2944 __free_slab+0xf0/0x1c8 mm/slub.c:3301 free_slab+0x30/0x104 mm/slub.c:3324 discard_slab mm/slub.c:3330 [inline] __put_partials+0x15c/0x1f0 mm/slub.c:3876 put_cpu_partial+0x1c8/0x364 mm/slub.c:3951 __slab_free+0x1c4/0x36c mm/slub.c:5929 do_slab_free mm/slub.c:6582 [inline] ___cache_free+0xc4/0xd8 mm/slub.c:6708 qlink_free mm/kasan/quarantine.c:163 [inline] qlist_free_all+0x78/0x130 mm/kasan/quarantine.c:179 kasan_quarantine_reduce+0x148/0x19c mm/kasan/quarantine.c:286 __kasan_slab_alloc+0x60/0x90 mm/kasan/common.c:352 kasan_slab_alloc include/linux/kasan.h:252 [inline] slab_post_alloc_hook mm/slub.c:4978 [inline] slab_alloc_node mm/slub.c:5288 [inline] __kmalloc_cache_noprof+0x254/0x6f8 mm/slub.c:5766 kmalloc_noprof include/linux/slab.h:957 [inline] kzalloc_noprof include/linux/slab.h:1094 [inline] keypair_create drivers/net/wireguard/noise.c:100 [inline] wg_noise_handshake_begin_session+0xe0/0xa64 drivers/net/wireguard/noise.c:827 wg_packet_send_handshake_response+0x1a8/0x250 drivers/net/wireguard/send.c:96 wg_receive_handshake_packet+0x1b8/0x810 drivers/net/wireguard/receive.c:154 wg_packet_handshake_receive_worker+0xd8/0x314 drivers/net/wireguard/receive.c:213 process_one_work+0x7cc/0x18d4 kernel/workqueue.c:3263 process_scheduled_works kernel/workqueue.c:3346 [inline] worker_thread+0x734/0xb84 kernel/workqueue.c:3427 kthread+0x348/0x5fc kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860 pool 6: task:kworker/1:1 state:R running task stack:0 pid:31 tgid:31 ppid:2 task_flags:0x4208060 flags:0x00000010 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x208/0x4f0 arch/arm64/kernel/process.c:741 (T) context_switch kernel/sched/core.c:5325 [inline] __schedule+0xd58/0x3180 kernel/sched/core.c:6929 preempt_schedule_common kernel/sched/core.c:7113 [inline] preempt_schedule+0xf4/0x254 kernel/sched/core.c:7137 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline] _raw_spin_unlock_irqrestore+0x94/0xac kernel/locking/spinlock.c:194 __debug_check_no_obj_freed lib/debugobjects.c:1108 [inline] debug_check_no_obj_freed+0x278/0x500 lib/debugobjects.c:1129 free_pages_prepare mm/page_alloc.c:1401 [inline] __free_frozen_pages+0x374/0x1214 mm/page_alloc.c:2906 free_frozen_pages+0x14/0x20 mm/page_alloc.c:2944 __free_slab+0xf0/0x1c8 mm/slub.c:3301 free_slab+0x30/0x104 mm/slub.c:3324 discard_slab mm/slub.c:3330 [inline] __put_partials+0x15c/0x1f0 mm/slub.c:3876 put_cpu_partial+0x1c8/0x364 mm/slub.c:3951 __slab_free+0x1c4/0x36c mm/slub.c:5929 do_slab_free mm/slub.c:6582 [inline] ___cache_free+0xc4/0xd8 mm/slub.c:6708 qlink_free mm/kasan/quarantine.c:163 [inline] qlist_free_all+0x78/0x130 mm/kasan/quarantine.c:179 kasan_quarantine_reduce+0x148/0x19c mm/kasan/quarantine.c:286 __kasan_slab_alloc+0x60/0x90 mm/kasan/common.c:352 kasan_slab_alloc include/linux/kasan.h:252 [inline] slab_post_alloc_hook mm/slub.c:4978 [inline] slab_alloc_node mm/slub.c:5288 [inline] __kmalloc_cache_noprof+0x254/0x6f8 mm/slub.c:5766 kmalloc_noprof include/linux/slab.h:957 [inline] kzalloc_noprof include/linux/slab.h:1094 [inline] nsim_fib6_rt_nh_add+0x4c/0x220 drivers/net/netdevsim/fib.c:500 nsim_fib6_rt_create drivers/net/netdevsim/fib.c:562 [inline] nsim_fib6_rt_insert drivers/net/netdevsim/fib.c:752 [inline] nsim_fib6_event drivers/net/netdevsim/fib.c:856 [inline] nsim_fib_event drivers/net/netdevsim/fib.c:889 [inline] nsim_fib_event_work+0x9f8/0x4038 drivers/net/netdevsim/fib.c:1493 process_one_work+0x7cc/0x18d4 kernel/workqueue.c:3263 process_scheduled_works kernel/workqueue.c:3346 [inline] worker_thread+0x734/0xb84 kernel/workqueue.c:3427 kthread+0x348/0x5fc kernel/kthread.c:463 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860