BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 162s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x100
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=12 refcnt=13
in-flight: 5748:rht_deferred_worker for 27s
pending: 3*psi_avgs_work, vmstat_shepherd, 3*ovs_dp_masks_rebalance, 4*nsim_dev_hwstats_traffic_work
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=11 refcnt=12
pending: nsim_dev_hwstats_traffic_work, 3*psi_avgs_work, 3*ovs_dp_masks_rebalance, ipvlan_process_multicast, ima_keys_handler, 2*rht_deferred_worker
workqueue events_long: flags=0x100
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=6 refcnt=7
pending: br_multicast_gc_work, 3*defense_work_handler, 2*br_fdb_cleanup
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=7 refcnt=8
pending: 3*defense_work_handler, br_multicast_gc_work, 2*br_fdb_cleanup, br_multicast_gc_work
workqueue events_unbound: flags=0x2
pwq 8: cpus=0-1 flags=0x6 nice=0 active=6 refcnt=7
in-flight: 139:nsim_dev_trap_report_work for 22s ,5990:toggle_allocation_gate for 32s
pending: 2*nsim_dev_trap_report_work, macvlan_process_broadcast, flush_memcg_stats_dwork
pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
in-flight: 1129:nsim_dev_trap_report_work for 65s ,5975:nsim_dev_trap_report_work for 22s
workqueue events_freezable: flags=0x104
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: update_balloon_stats_func
workqueue events_power_efficient: flags=0x180
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=5 refcnt=6
in-flight: 995:neigh_periodic_work for 37s
pending: 2*check_lifetime, neigh_managed_work, gc_worker
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=7 refcnt=8
in-flight: 809:wg_ratelimiter_gc_entries for 162s
pending: neigh_periodic_work, neigh_managed_work, do_cache_clean, 3*check_lifetime
workqueue mm_percpu_wq: flags=0x108
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: vmstat_update
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: vmstat_update
workqueue bat_events: flags=0x6000a
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=37
in-flight: 1122:batadv_tt_purge for 5s
inactive: 4*batadv_mcast_mla_update, batadv_purge_orig, batadv_iv_send_outstanding_bat_ogm_packet, batadv_mcast_mla_update, 4*batadv_iv_send_outstanding_bat_ogm_packet, 4*batadv_purge_orig, 4*batadv_iv_send_outstanding_bat_ogm_packet, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, 2*batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, 4*batadv_tt_purge
workqueue wg-kex-wg0: flags=0x124
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
in-flight: 5727:wg_packet_handshake_receive_worker for 88s wg_packet_handshake_receive_worker
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
in-flight: 24:wg_packet_handshake_receive_worker for 166s
pending: wg_packet_handshake_receive_worker
workqueue wg-crypt-wg0: flags=0x128
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_decrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg1: flags=0x124
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
in-flight: 9:wg_packet_handshake_receive_worker for 143s wg_packet_handshake_receive_worker
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_receive_worker
workqueue wg-kex-wg1: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
in-flight: 35:wg_packet_handshake_send_worker for 7s ,5989:wg_packet_handshake_send_worker for 9s
workqueue wg-crypt-wg1: flags=0x128
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_decrypt_worker
workqueue wg-kex-wg2: flags=0x124
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
in-flight: 10:wg_packet_handshake_receive_worker for 193s wg_packet_handshake_receive_worker
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
in-flight: 5769:wg_packet_handshake_receive_worker for 163s
pending: wg_packet_handshake_receive_worker
workqueue wg-kex-wg2: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
in-flight: 12:wg_packet_handshake_send_worker for 6s
workqueue wg-crypt-wg2: flags=0x128
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_tx_worker, wg_packet_decrypt_worker
workqueue wg-kex-wg0: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
in-flight: 5979:wg_packet_handshake_send_worker for 26s
workqueue wg-crypt-wg0: flags=0x128
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_tx_worker
workqueue wg-kex-wg1: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
in-flight: 5992:wg_packet_handshake_send_worker for 41s
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg2: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_send_worker
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_tx_worker
workqueue wg-kex-wg0: flags=0x124
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
in-flight: 5633:wg_packet_handshake_receive_worker for 194s wg_packet_handshake_receive_worker
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
in-flight: 5762:wg_packet_handshake_receive_worker for 166s
pending: wg_packet_handshake_receive_worker
workqueue wg-kex-wg0: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_tx_worker
workqueue wg-kex-wg1: flags=0x124
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_receive_worker
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
pending: wg_packet_handshake_receive_worker
workqueue wg-kex-wg1: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=2 refcnt=3
in-flight: 13:wg_packet_handshake_send_worker for 27s
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
pending: wg_packet_encrypt_worker, wg_packet_tx_worker
workqueue wg-kex-wg2: flags=0x124
pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
in-flight: 5741:wg_packet_handshake_receive_worker for 162s wg_packet_handshake_receive_worker
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3
in-flight: 1242:wg_packet_handshake_receive_worker for 222s wg_packet_handshake_receive_worker
workqueue wg-kex-wg2: flags=0x6
pwq 8: cpus=0-1 flags=0x6 nice=0 active=4 refcnt=5
in-flight: 36:wg_packet_handshake_send_worker for 51s wg_packet_handshake_send_worker ,68:wg_packet_handshake_send_worker for 12s
pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=3 refcnt=4
pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker, wg_packet_tx_worker
workqueue krds_cp_wq#0/0: flags=0x20002
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=4
in-flight: 760:rds_connect_worker for 40s
workqueue krds_cp_wq#1/0: flags=0x20002
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=4
in-flight: 5991:rds_connect_worker for 46s
workqueue bond1: flags=0x2000a
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=5
pending: bond_alb_monitor
inactive: bond_mii_monitor
workqueue bond4: flags=0x2000a
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=5
pending: bond_mii_monitor
inactive: bond_alb_monitor
workqueue bond2: flags=0x2000a
pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=5
pending: bond_mii_monitor
inactive: bond_alb_monitor
pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=28s workers=9 idle: 9114 5749
pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=163s workers=8 idle: 9113 5734 43
pool 8: cpus=0-1 flags=0x6 nice=0 hung=6s workers=19 manager: 5984 idle: 132 5986 5972
Showing backtraces of busy workers in stalled worker pools:
pool 6:
task:kworker/1:0 state:R running task stack:23848 pid:24 tgid:24 ppid:2 task_flags:0x4208060 flags:0x00080000
Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker
Call Trace:
context_switch kernel/sched/core.c:5388 [inline]
__schedule+0x1840/0x57a0 kernel/sched/core.c:7189
preempt_schedule_irq+0x4d/0xa0 kernel/sched/core.c:7513
irqentry_exit_to_kernel_mode include/linux/irq-entry-common.h:539 [inline]
irqentry_exit+0x14f/0x8e0 kernel/entry/common.c:164
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:should_resched arch/x86/include/asm/preempt.h:104 [inline]
RIP: 0010:__local_bh_enable_ip+0xd8/0x130 kernel/softirq.c:457
Code: 8b e8 9c 00 1b 0a 65 66 8b 05 cc ed ae 11 66 85 c0 75 32 bf 01 00 00 00 e8 75 f3 0b 00 e8 80 ec 45 00 fb 65 8b 05 a8 ed ae 11 <85> c0 75 05 e8 6f e8 ab ff 5b 41 5e e9 07 de 1d 0a cc 90 0f 0b 90
RSP: 0000:ffffc900001e7888 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0000000000000201 RCX: 0000000000000046
RDX: 0000000000000006 RSI: ffffffff8dfa1bbe RDI: ffffffff8c2899a0
RBP: ffffc900001e7af0 R08: ffffffff902f89f7 R09: 1ffffffff205f13e
R10: dffffc0000000000 R11: fffffbfff205f13f R12: dffffc0000000000
R13: 000000000000000c R14: ffffffff867620f6 R15: 0000000000000010
spin_unlock_bh include/linux/spinlock.h:396 [inline]
ptr_ring_consume_bh include/linux/ptr_ring.h:377 [inline]
wg_packet_handshake_receive_worker+0x2e6/0x10c0 drivers/net/wireguard/receive.c:212
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xa8e/0x14e0 kernel/workqueue.c:3397
worker_thread+0xa47/0xfb0 kernel/workqueue.c:3478
kthread+0x389/0x470 kernel/kthread.c:436
ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
pool 6:
task:kworker/1:2 state:R running task stack:24520 pid:809 tgid:809 ppid:2 task_flags:0x4208060 flags:0x00080000
Workqueue: events_power_efficient wg_ratelimiter_gc_entries
Call Trace:
context_switch kernel/sched/core.c:5388 [inline]
__schedule+0x1840/0x57a0 kernel/sched/core.c:7189
preempt_schedule_irq+0x4d/0xa0 kernel/sched/core.c:7513
irqentry_exit_to_kernel_mode include/linux/irq-entry-common.h:539 [inline]
irqentry_exit+0x14f/0x8e0 kernel/entry/common.c:164
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:_raw_spin_lock+0x5/0x40 kernel/locking/spinlock.c:157
Code: e8 a0 6b e2 f5 31 ed 89 e8 5b 41 5e 5d e9 83 4a 00 00 cc 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 53 <48> 89 fb bf 01 00 00 00 e8 0e 5e ee f5 48 8d 7b 18 31 f6 31 d2 31
RSP: 0000:ffffc90004877a98 EFLAGS: 00000293
RAX: ffffffff8677469c RBX: ffffffff8f51b240 RCX: ffff888026621f00
RDX: 0000000000000000 RSI: ffffffff8e213d2e RDI: ffffffff8f51b100
RBP: dffffc0000000000 R08: ffffffff8f51b103 R09: 1ffffffff1ea3620
R10: dffffc0000000000 R11: fffffbfff1ea3621 R12: ffff88805a8945f8
R13: dffffc0000000000 R14: 00000000000008c0 R15: 0000000000000000
spin_lock include/linux/spinlock.h:342 [inline]
wg_ratelimiter_gc_entries+0x5d/0x450 drivers/net/wireguard/ratelimiter.c:63
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xa8e/0x14e0 kernel/workqueue.c:3397
worker_thread+0xa47/0xfb0 kernel/workqueue.c:3478
kthread+0x389/0x470 kernel/kthread.c:436
ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
pool 6:
task:kworker/1:3 state:R running task stack:26368 pid:1242 tgid:1242 ppid:2 task_flags:0x4208060 flags:0x00080000
Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker
Call Trace:
context_switch kernel/sched/core.c:5388 [inline]
__schedule+0x1840/0x57a0 kernel/sched/core.c:7189
__schedule_loop kernel/sched/core.c:7268 [inline]
schedule+0x164/0x360 kernel/sched/core.c:7283
schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:7340
rwsem_down_write_slowpath+0x87d/0x1080 kernel/locking/rwsem.c:1215
__down_write_common kernel/locking/rwsem.c:1347 [inline]
__down_write kernel/locking/rwsem.c:1356 [inline]
down_write+0x1bc/0x200 kernel/locking/rwsem.c:1626
wg_noise_handshake_begin_session+0x36/0xbf0 drivers/net/wireguard/noise.c:822
wg_packet_send_handshake_response+0x137/0x2d0 drivers/net/wireguard/send.c:96
wg_receive_handshake_packet drivers/net/wireguard/receive.c:154 [inline]
wg_packet_handshake_receive_worker+0x6a9/0x10c0 drivers/net/wireguard/receive.c:213
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xa8e/0x14e0 kernel/workqueue.c:3397
worker_thread+0xa47/0xfb0 kernel/workqueue.c:3478
kthread+0x389/0x470 kernel/kthread.c:436
ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
pool 6:
task:kworker/1:6 state:R running task stack:25384 pid:5769 tgid:5769 ppid:2 task_flags:0x4208060 flags:0x00080000
Workqueue: wg-kex-wg2 wg_packet_handshake_receive_worker
Call Trace:
context_switch kernel/sched/core.c:5388 [inline]
__schedule+0x1840/0x57a0 kernel/sched/core.c:7189
preempt_schedule_common+0x82/0xd0 kernel/sched/core.c:7370
preempt_schedule_thunk+0x16/0x30 arch/x86/entry/thunk.S:12
__local_bh_enable_ip+0xe1/0x130 kernel/softirq.c:457
blake2s_compress+0xfa/0x1c00 lib/crypto/x86/blake2s.h:42
blake2s_final+0x10f/0x250 lib/crypto/blake2s.c:142
hmac+0x2dc/0x3b0 drivers/net/wireguard/noise.c:333
kdf drivers/net/wireguard/noise.c:367 [inline]
message_ephemeral+0x255/0x310 drivers/net/wireguard/noise.c:493
wg_noise_handshake_create_response+0x297/0x9f0 drivers/net/wireguard/noise.c:692
wg_packet_send_handshake_response+0x110/0x2d0 drivers/net/wireguard/send.c:94
wg_receive_handshake_packet drivers/net/wireguard/receive.c:154 [inline]
wg_packet_handshake_receive_worker+0x6a9/0x10c0 drivers/net/wireguard/receive.c:213
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xa8e/0x14e0 kernel/workqueue.c:3397
worker_thread+0xa47/0xfb0 kernel/workqueue.c:3478
kthread+0x389/0x470 kernel/kthread.c:436
ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
pool 6:
task:kworker/1:5 state:R running task stack:23976 pid:5762 tgid:5762 ppid:2 task_flags:0x4208060 flags:0x00080000
Workqueue: wg-kex-wg0 wg_packet_handshake_receive_worker
Call Trace:
context_switch kernel/sched/core.c:5388 [inline]
__schedule+0x1840/0x57a0 kernel/sched/core.c:7189
__schedule_loop kernel/sched/core.c:7268 [inline]
schedule+0x164/0x360 kernel/sched/core.c:7283
schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:7340
rwsem_down_write_slowpath+0x87d/0x1080 kernel/locking/rwsem.c:1215
__down_write_common kernel/locking/rwsem.c:1347 [inline]
__down_write kernel/locking/rwsem.c:1356 [inline]
down_write+0x1bc/0x200 kernel/locking/rwsem.c:1626
wg_noise_handshake_begin_session+0x36/0xbf0 drivers/net/wireguard/noise.c:822
wg_packet_send_handshake_response+0x137/0x2d0 drivers/net/wireguard/send.c:96
wg_receive_handshake_packet drivers/net/wireguard/receive.c:154 [inline]
wg_packet_handshake_receive_worker+0x6a9/0x10c0 drivers/net/wireguard/receive.c:213
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xa8e/0x14e0 kernel/workqueue.c:3397
worker_thread+0xa47/0xfb0 kernel/workqueue.c:3478
kthread+0x389/0x470 kernel/kthread.c:436
ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
----------------
Code disassembly (best guess):
0: 8b e8 mov %eax,%ebp
2: 9c pushf
3: 00 1b add %bl,(%rbx)
5: 0a 65 66 or 0x66(%rbp),%ah
8: 8b 05 cc ed ae 11 mov 0x11aeedcc(%rip),%eax # 0x11aeedda
e: 66 85 c0 test %ax,%ax
11: 75 32 jne 0x45
13: bf 01 00 00 00 mov $0x1,%edi
18: e8 75 f3 0b 00 call 0xbf392
1d: e8 80 ec 45 00 call 0x45eca2
22: fb sti
23: 65 8b 05 a8 ed ae 11 mov %gs:0x11aeeda8(%rip),%eax # 0x11aeedd2
* 2a: 85 c0 test %eax,%eax <-- trapping instruction
2c: 75 05 jne 0x33
2e: e8 6f e8 ab ff call 0xffabe8a2
33: 5b pop %rbx
34: 41 5e pop %r14
36: e9 07 de 1d 0a jmp 0xa1dde42
3b: cc int3
3c: 90 nop
3d: 0f 0b ud2
3f: 90 nop