BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 54s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=12/256 refcnt=13 pending: psi_avgs_work, nsim_dev_trap_report_work, nsim_dev_hwstats_traffic_work, 3*nsim_dev_trap_report_work, nsim_dev_hwstats_traffic_work, nsim_dev_trap_report_work, 3*nsim_dev_hwstats_traffic_work, nsim_dev_trap_report_work pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=19/256 refcnt=20 in-flight: 6511:nsim_fib_event_work, 5888:nsim_fib_event_work nsim_fib_event_work, 1554:nsim_fib_event_work nsim_fib_event_work, 6437:nsim_fib_event_work nsim_fib_event_work, 9:nsim_fib_event_work nsim_fib_event_work, 5887:nsim_fib_event_work pending: nsim_fib_event_work, nsim_dev_hwstats_traffic_work, psi_avgs_work, 4*ovs_dp_masks_rebalance, kfree_rcu_monitor, vmstat_shepherd workqueue events_long: flags=0x0 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=5/256 refcnt=6 pending: 5*defense_work_handler workqueue events_unbound: flags=0x2 pwq 4: cpus=0-1 flags=0x4 nice=0 active=5/512 refcnt=7 in-flight: 40:fsnotify_mark_destroy_workfn fsnotify_mark_destroy_workfn, 10:fsnotify_connector_destroy_workfn fsnotify_connector_destroy_workfn pending: toggle_allocation_gate workqueue events_power_efficient: flags=0x82 pwq 4: cpus=0-1 flags=0x4 nice=0 active=2/256 refcnt=4 in-flight: 6275:gc_worker pending: neigh_periodic_work workqueue rcu_gp: flags=0x8 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: srcu_invoke_callbacks workqueue mm_percpu_wq: flags=0x8 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4a pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3 pending: wb_workfn workqueue dm_bufio_cache: flags=0x8 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: work_fn workqueue mld: flags=0x40008 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=18 pending: mld_ifc_work inactive: 12*mld_dad_work, mld_ifc_work, 3*mld_dad_work workqueue ipv6_addrconf: flags=0x40008 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 refcnt=18 pending: addrconf_dad_work inactive: 16*addrconf_dad_work workqueue bat_events: flags=0xe000a pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=51 in-flight: 480:batadv_tt_purge inactive: batadv_tt_purge, batadv_dat_purge, batadv_bla_periodic_work, 6*batadv_nc_worker, 6*batadv_mcast_mla_update, batadv_purge_orig, 7*batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, 2*batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, 2*batadv_iv_send_outstanding_bat_ogm_packet, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, 4*batadv_iv_send_outstanding_bat_ogm_packet, 3*batadv_purge_orig, 3*batadv_iv_send_outstanding_bat_ogm_packet, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_tt_purge workqueue wg-crypt-wg0: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-kex-wg0: flags=0x6 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x28 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_encrypt_worker pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker workqueue wg-crypt-wg1: flags=0x28 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_encrypt_worker pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker workqueue wg-crypt-wg0: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-crypt-wg1: flags=0x28 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_encrypt_worker pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-kex-wg2: flags=0x6 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg2: flags=0x6 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x28 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_encrypt_worker pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg1: flags=0x24 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg1: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker workqueue wg-kex-wg2: flags=0x24 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg2: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-crypt-wg0: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-crypt-wg1: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-kex-wg2: flags=0x6 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x28 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_encrypt_worker pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_encrypt_worker workqueue wg-crypt-wg0: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_decrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg1: flags=0x6 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg1: flags=0x28 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_encrypt_worker pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-crypt-wg2: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 pending: wg_packet_encrypt_worker, wg_packet_tx_worker workqueue wg-kex-wg0: flags=0x24 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-kex-wg0: flags=0x6 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg0: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: wg_packet_encrypt_worker, wg_packet_tx_worker, wg_packet_decrypt_worker workqueue wg-kex-wg1: flags=0x24 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_handshake_receive_worker workqueue wg-crypt-wg1: flags=0x28 pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 pending: wg_packet_tx_worker, wg_packet_encrypt_worker, wg_packet_decrypt_worker workqueue wg-kex-wg2: flags=0x6 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 refcnt=3 pending: wg_packet_handshake_send_worker workqueue wg-crypt-wg2: flags=0x28 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 pending: wg_packet_encrypt_worker pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 pending: wg_packet_tx_worker, wg_packet_encrypt_worker workqueue phy11: flags=0xa0002 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=3 pending: ieee80211_iface_work workqueue phy12: flags=0xa0002 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=3 pending: ieee80211_iface_work workqueue phy13: flags=0xa0002 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=3 pending: ieee80211_iface_work workqueue phy14: flags=0xa0002 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=3 pending: ieee80211_iface_work workqueue phy15: flags=0xa0002 pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 refcnt=3 pending: ieee80211_iface_work pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=54s workers=11 idle: 5886 7 6379 6374 6521 pool 4: cpus=0-1 flags=0x4 nice=0 hung=0s workers=10 idle: 6588 6512 12 6530 39 6387 Showing backtraces of running workers in stalled CPU-bound worker pools: pool 0: task:kworker/0:9 state:R running task stack:0 pid:6511 ppid:2 flags:0x00000008 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x320/0x754 arch/arm64/kernel/process.c:556 context_switch kernel/sched/core.c:5343 [inline] __schedule+0x13e8/0x2450 kernel/sched/core.c:6669 preempt_schedule_irq+0x80/0x188 kernel/sched/core.c:6981 arm64_preempt_schedule_irq arch/arm64/kernel/entry-common.c:265 [inline] __el1_irq arch/arm64/kernel/entry-common.c:474 [inline] el1_interrupt+0x4c/0x68 arch/arm64/kernel/entry-common.c:486 el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:491 el1h_64_irq+0x64/0x68 arch/arm64/kernel/entry.S:587 __daif_local_irq_restore arch/arm64/include/asm/irqflags.h:182 [inline] arch_local_irq_restore arch/arm64/include/asm/irqflags.h:202 [inline] __timer_delete_sync+0xd4/0x1d8 kernel/time/timer.c:1557 timer_delete_sync kernel/time/timer.c:1628 [inline] del_timer_sync include/linux/timer.h:200 [inline] schedule_timeout+0x1e4/0x348 kernel/time/timer.c:2168 schedule_timeout_uninterruptible+0x74/0xac kernel/time/timer.c:2201 msleep+0x48/0x6c kernel/time/timer.c:2322 nsim_fib6_rt_add drivers/net/netdevsim/fib.c:693 [inline] nsim_fib6_rt_insert drivers/net/netdevsim/fib.c:759 [inline] nsim_fib6_event drivers/net/netdevsim/fib.c:856 [inline] nsim_fib_event drivers/net/netdevsim/fib.c:889 [inline] nsim_fib_event_work+0x247c/0x32bc drivers/net/netdevsim/fib.c:1492 process_one_work+0x788/0x12d4 kernel/workqueue.c:2405 process_scheduled_works kernel/workqueue.c:2468 [inline] worker_thread+0xb68/0xfe8 kernel/workqueue.c:2554 kthread+0x288/0x310 kernel/kthread.c:379 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:870 pool 0: task:kworker/0:1 state:R running task stack:0 pid:9 ppid:2 flags:0x00000008 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x320/0x754 arch/arm64/kernel/process.c:556 context_switch kernel/sched/core.c:5343 [inline] __schedule+0x13e8/0x2450 kernel/sched/core.c:6669 preempt_schedule_irq+0x80/0x188 kernel/sched/core.c:6981 arm64_preempt_schedule_irq arch/arm64/kernel/entry-common.c:265 [inline] __el1_irq arch/arm64/kernel/entry-common.c:474 [inline] el1_interrupt+0x4c/0x68 arch/arm64/kernel/entry-common.c:486 el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:491 el1h_64_irq+0x64/0x68 arch/arm64/kernel/entry.S:587 fib_find_node+0x54/0x2a8 net/ipv4/fib_trie.c:943 fib_find_matching_alias net/ipv4/fib_trie.c:1029 [inline] fib_alias_hw_flags_set+0x190/0x934 net/ipv4/fib_trie.c:1052 nsim_fib_event drivers/net/netdevsim/fib.c:884 [inline] nsim_fib_event_work+0xa50/0x32bc drivers/net/netdevsim/fib.c:1492 process_one_work+0x788/0x12d4 kernel/workqueue.c:2405 process_scheduled_works kernel/workqueue.c:2468 [inline] worker_thread+0xb68/0xfe8 kernel/workqueue.c:2554 kthread+0x288/0x310 kernel/kthread.c:379 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:870 pool 0: task:kworker/0:4 state:R running task stack:0 pid:5887 ppid:2 flags:0x00000008 Workqueue: events nsim_fib_event_work Call trace: __switch_to+0x320/0x754 arch/arm64/kernel/process.c:556 context_switch kernel/sched/core.c:5343 [inline] __schedule+0x13e8/0x2450 kernel/sched/core.c:6669 preempt_schedule_irq+0x80/0x188 kernel/sched/core.c:6981 arm64_preempt_schedule_irq arch/arm64/kernel/entry-common.c:265 [inline] __el1_irq arch/arm64/kernel/entry-common.c:474 [inline] el1_interrupt+0x4c/0x68 arch/arm64/kernel/entry-common.c:486 el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:491 el1h_64_irq+0x64/0x68 arch/arm64/kernel/entry.S:587 __daif_local_irq_restore arch/arm64/include/asm/irqflags.h:182 [inline] arch_local_irq_restore arch/arm64/include/asm/irqflags.h:202 [inline] kasan_quarantine_put+0xb8/0x1c8 mm/kasan/quarantine.c:240 ____kasan_slab_free+0x150/0x1c0 mm/kasan/common.c:238 __kasan_slab_free+0x18/0x28 mm/kasan/common.c:244 kasan_slab_free include/linux/kasan.h:162 [inline] slab_free_hook mm/slub.c:1781 [inline] slab_free_freelist_hook mm/slub.c:1807 [inline] slab_free mm/slub.c:3786 [inline] __kmem_cache_free+0x2a8/0x49c mm/slub.c:3799 kfree+0xb8/0x19c mm/slab_common.c:1015 nsim_fib6_event_fini+0x1b8/0x1f4 drivers/net/netdevsim/fib.c:842 nsim_fib_event drivers/net/netdevsim/fib.c:891 [inline] nsim_fib_event_work+0xde0/0x32bc drivers/net/netdevsim/fib.c:1492 process_one_work+0x788/0x12d4 kernel/workqueue.c:2405 process_scheduled_works kernel/workqueue.c:2468 [inline] worker_thread+0xb68/0xfe8 kernel/workqueue.c:2554 kthread+0x288/0x310 kernel/kthread.c:379 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:870