BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 700s! BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 704s! BUG: workqueue lockup - pool cpus=0-1 flags=0x4 nice=0 stuck for 619s! BUG: workqueue lockup - pool cpus=0-1 node=0 flags=0x4 nice=0 stuck for 702s! BUG: workqueue lockup - pool cpus=0-1 node=0 flags=0x4 nice=0 stuck for 687s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=14 refcnt=15 in-flight: 46:bpf_prog_free_deferred ,3764:bpf_prog_free_deferred ,8:bpf_prog_free_deferred ,3658:bpf_prog_free_deferred ,10:bpf_prog_free_deferred ,3704:bpf_prog_free_deferred ,3703:bpf_prog_free_deferred pending: nsim_dev_hwstats_traffic_work, nsim_dev_trap_report_work, psi_avgs_work, vmstat_shepherd, psi_avgs_work, rht_deferred_worker, kfree_rcu_monitor pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=10 refcnt=11 in-flight: 3736:bpf_prog_free_deferred ,24:bpf_prog_free_deferred ,49:bpf_prog_free_deferred ,97:bpf_prog_free_deferred ,3741:bpf_prog_free_deferred ,100:bpf_prog_free_deferred pending: nsim_dev_hwstats_traffic_work, nsim_dev_trap_report_work, psi_avgs_work, ima_keys_handler workqueue events_unbound: flags=0x2 pwq 9: cpus=0-1 node=0 flags=0x4 nice=0 active=6 refcnt=7 pending: flush_memcg_stats_dwork, macvlan_process_broadcast, crng_reseed, 3*idle_cull_fn pwq 10: cpus=0-1 node=0 flags=0x4 nice=0 active=4 refcnt=5 pending: 3*macvlan_process_broadcast, idle_cull_fn workqueue events_freezable: flags=0x4 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: mmc_rescan workqueue events_power_efficient: flags=0x82 pwq 9: cpus=0-1 node=0 flags=0x4 nice=0 active=7 refcnt=8 in-flight: 30:wg_ratelimiter_gc_entries pending: 2*neigh_managed_work, 2*neigh_periodic_work, gc_worker, check_lifetime workqueue mm_percpu_wq: flags=0x8 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue pm: flags=0x4 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: pm_runtime_work workqueue writeback: flags=0x4a pwq 9: cpus=0-1 node=0 flags=0x4 nice=0 active=1 refcnt=2 pending: wb_workfn workqueue dm_bufio_cache: flags=0x8 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: work_fn workqueue ipv6_addrconf: flags=0x6000a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=6 pending: addrconf_verify_work inactive: 2*addrconf_verify_work workqueue wg-kex-wg0: flags=0x6 pwq 9: cpus=0-1 node=0 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_queued_expired_zero_key_material pwq 10: cpus=0-1 node=0 flags=0x4 nice=0 active=2 refcnt=3 pending: wg_packet_handshake_send_worker, wg_queued_expired_zero_key_material workqueue wg-crypt-wg0: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg1: flags=0x6 pwq 9: cpus=0-1 node=0 flags=0x4 nice=0 active=2 refcnt=3 pending: wg_packet_handshake_send_worker, wg_queued_expired_zero_key_material pwq 10: cpus=0-1 node=0 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_queued_expired_zero_key_material workqueue wg-crypt-wg1: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x6 pwq 9: cpus=0-1 node=0 flags=0x4 nice=0 active=2 refcnt=3 pending: wg_packet_handshake_send_worker, wg_queued_expired_zero_key_material pwq 10: cpus=0-1 node=0 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_queued_expired_zero_key_material workqueue wg-crypt-wg2: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker workqueue wg-kex-wg0: flags=0x6 pwq 9: cpus=0-1 node=0 flags=0x4 nice=0 active=2 refcnt=3 pending: wg_packet_handshake_send_worker, wg_queued_expired_zero_key_material pwq 10: cpus=0-1 node=0 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_queued_expired_zero_key_material workqueue wg-crypt-wg0: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg1: flags=0x6 pwq 9: cpus=0-1 node=0 flags=0x4 nice=0 active=2 refcnt=3 pending: wg_packet_handshake_send_worker, wg_queued_expired_zero_key_material pwq 10: cpus=0-1 node=0 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_queued_expired_zero_key_material workqueue wg-crypt-wg1: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x6 pwq 9: cpus=0-1 node=0 flags=0x4 nice=0 active=2 refcnt=3 pending: wg_packet_handshake_send_worker, wg_queued_expired_zero_key_material pwq 10: cpus=0-1 node=0 flags=0x4 nice=0 active=1 refcnt=2 pending: wg_queued_expired_zero_key_material workqueue wg-crypt-wg2: flags=0x28 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: wg_packet_encrypt_worker pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4 pending: wg_packet_decrypt_worker, wg_packet_tx_worker, wg_packet_encrypt_worker pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=700s workers=11 idle: 3822 3821 3807 3787 pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=704s workers=9 idle: 3819 3749 3768 pool 9: cpus=0-1 node=0 flags=0x4 nice=0 hung=702s workers=9 idle: 60 2485 2177 2470 2220 2205 691 26 Showing backtraces of running workers in stalled CPU-bound worker pools: pool 2: task:kworker/0:3 state:R running task stack:0 pid:3658 tgid:3658 ppid:2 flags:0x00000000 Workqueue: events bpf_prog_free_deferred Call trace: [<818f5b10>] (__schedule) from [<818f6d70>] (preempt_schedule_irq+0x40/0xa8 kernel/sched/core.c:7067) r10:828b0c0c r9:84149800 r8:80200be4 r7:dfd8dd64 r6:ffffffff r5:84149800 r4:00000000 [<818f6d30>] (preempt_schedule_irq) from [<80200c04>] (svc_preempt+0x8/0x18) Exception stack(0xdfd8dd30 to 0xdfd8dd78) dd20: b52ab000 ec093000 00000001 8021b3f8 dd40: 7f003000 00000001 828affe8 7f003000 ec093000 8260cad4 828b0c0c dfd8ddac dd60: dfd8ddb0 dfd8dd80 802109bc 8021b410 80000013 ffffffff r5:80000013 r4:8021b410 [<80210958>] (flush_tlb_kernel_range) from [<804a4ff4>] (__purge_vmap_area_lazy+0x210/0x3d4 mm/vmalloc.c:2269) r4:828b0c2c [<804a4de4>] (__purge_vmap_area_lazy) from [<804a5444>] (_vm_unmap_aliases+0x28c/0x2e8 mm/vmalloc.c:2867) r10:00000000 r9:dfd8de20 r8:00000000 r7:dfd8ddd8 r6:00000008 r5:ddde1c80 r4:dfd8ddd8 [<804a51b8>] (_vm_unmap_aliases) from [<804a9074>] (vm_reset_perms mm/vmalloc.c:3257 [inline]) [<804a51b8>] (_vm_unmap_aliases) from [<804a9074>] (vfree+0x170/0x1e4 mm/vmalloc.c:3336) r10:82c16005 r9:00000001 r8:00000000 r7:ffffffff r6:00000000 r5:84fe6b40 r4:00000000 [<804a8f04>] (vfree) from [<804fbb3c>] (execmem_free+0x30/0x64 mm/execmem.c:69) r9:84149800 r8:00800000 r7:00000000 r6:82c16000 r5:00001000 r4:7f011000 [<804fbb0c>] (execmem_free) from [<8039572c>] (bpf_jit_free_exec+0x10/0x14 kernel/bpf/core.c:1073) r5:00001000 r4:e05b9000 [<8039571c>] (bpf_jit_free_exec) from [<80395b0c>] (bpf_jit_binary_free kernel/bpf/core.c:1119 [inline]) [<8039571c>] (bpf_jit_free_exec) from [<80395b0c>] (bpf_jit_free+0x68/0xe4 kernel/bpf/core.c:1243) [<80395aa4>] (bpf_jit_free) from [<80396c40>] (bpf_prog_free_deferred+0x14c/0x164 kernel/bpf/core.c:2803) r5:84d31b54 r4:84d31800 [<80396af4>] (bpf_prog_free_deferred) from [<802671bc>] (process_one_work+0x1c4/0x510 kernel/workqueue.c:3231) r7:dddd00c0 r6:82c16000 r5:84d31b54 r4:84f4d800 [<80266ff8>] (process_one_work) from [<80267df0>] (process_scheduled_works kernel/workqueue.c:3312 [inline]) [<80266ff8>] (process_one_work) from [<80267df0>] (worker_thread+0x1ec/0x418 kernel/workqueue.c:3393) r10:84149800 r9:84f4d82c r8:61c88647 r7:dddd00e0 r6:82604d40 r5:dddd00c0 r4:84f4d800 [<80267c04>] (worker_thread) from [<80271228>] (kthread+0x104/0x134 kernel/kthread.c:389) r10:00000000 r9:df841e90 r8:84f50f80 r7:84f4d800 r6:80267c04 r5:84149800 r4:84f50d80 [<80271124>] (kthread) from [<80200114>] (ret_from_fork+0x14/0x20 arch/arm/kernel/entry-common.S:134) Exception stack(0xdfd8dfb0 to 0xdfd8dff8) dfa0: 00000000 00000000 00000000 00000000 dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:80271124 r4:84f50d80