BUG: workqueue lockup in debugfs_reap_work

Status: moderation: reported on 2026/11/19 00:08
Subsystems: wireguard
Labels: prio:normal
[Documentation on labels]
Reported-by: syzbot+0b08997c105ff8744fc3@syzkaller.appspotmail.com
First crash: 4d03h, last: 4d03h
✨ AI-generated root cause summary:

ref_tracker schedules debugfs_reap_worker onto the default per-CPU workqueue (system_wq, events) via schedule_work(). In debugfs_reap_work(), it loops over debugfs_symlinks and debugfs_dentries in a do { ... } while (reaped) loop to remove dead debugfs dentries via debugfs_remove(), which operates in-memory without blocking or calling cond_resched(). During bulk network namespace teardown (cleanup_net), repeated calls to ref_tracker_dir_exit() continuously mark new entries dead, keeping debugfs_reap_work running continuously for tens of seconds in TASK_RUNNING. Because the worker never blocks, workqueue concurrency management treats pool->nr_running as positive and does not wake any idle workers, completely stalling forward progress on the per-CPU worker pool and triggering a workqueue lockup.

▶ ▼ ✨ Full AI root cause analysis
Based on kernel commit ad3fc05c7e65:

Technical Root Cause Analysis

Overview

The reported lockup (BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 30s!) occurs when the per-CPU worker pool for CPU 0 (pool 2) fails to make forward progress for 30 seconds despite having pending work items (active=17, multiple pending works including vmstat_shepherd, psi_avgs_work, netdev_work_proc, delayed_vfree_work).

The in-flight work item on CPU 0 is debugfs_reap_work (kworker/0:2, PID 879) running in the events workqueue (system_wq), which had been executing continuously for 30 seconds (in-flight: 879:debugfs_reap_work for 30s).

Involved Files, Functions, and Structures

  • File: lib/ref_tracker.c
  • Functions: debugfs_reap_work(), ref_tracker_debugfs_mark(), ref_tracker_dir_exit()
  • Structures / Globals: struct xarray debugfs_dentries, struct xarray debugfs_symlinks, struct work_struct debugfs_reap_worker, REF_TRACKER_DIR_DEAD (XA_MARK_0)
  • Workqueue Subsystem: kernel/workqueue.c (process_scheduled_works(), need_more_worker(), keep_working(), worker_thread())

Mechanism of the Failure

  1. Scheduling onto Default Shared Workqueue: When ref_tracker_dir_exit() is invoked (such as during network namespace teardown in cleanup_net, when network interfaces, sockets, and associated tracked objects are destroyed), ref_tracker_debugfs_mark() marks the dir pointer in debugfs_dentries and debugfs_symlinks with the REF_TRACKER_DIR_DEAD mark:

    xa_lock_irqsave(&debugfs_dentries, flags);
    __xa_set_mark(&debugfs_dentries, (unsigned long)dir, REF_TRACKER_DIR_DEAD);
    xa_unlock_irqrestore(&debugfs_dentries, flags);
    ...
    schedule_work(&debugfs_reap_worker);
    

    schedule_work() queues debugfs_reap_worker onto system_wq (events), which is a standard per-CPU workqueue (flags 0x100, bounded to a single CPU, non-unbound, and non-WQ_CPU_INTENSIVE).

  2. Unbounded Non-Sleeping Reap Loop: In lib/ref_tracker.c, debugfs_reap_work() runs:

    static void debugfs_reap_work(struct work_struct *work)
    {
        struct dentry *dentry;
        unsigned long index;
        bool reaped;
    
        do {
            reaped = false;
            xa_for_each_marked(&debugfs_symlinks, index, dentry, REF_TRACKER_DIR_DEAD) {
                xa_erase_irq(&debugfs_symlinks, index);
                debugfs_remove(dentry);
                reaped = true;
            }
            xa_for_each_marked(&debugfs_dentries, index, dentry, REF_TRACKER_DIR_DEAD) {
                xa_erase_irq(&debugfs_dentries, index);
                debugfs_remove(dentry);
                reaped = true;
            }
        } while (reaped);
    }
    
    • debugfs_remove() executes an in-memory deletion (simple_recursive_removal()) which does not sleep or block under uncontended circumstances.
    • There is no cond_resched() anywhere within the loops or the outer do ... while (reaped) construct.
    • During heavy teardown activities (such as cleanup_net draining numerous network namespaces with multiple network devices), calls to ref_tracker_dir_exit() continually mark newly exiting directories as dead, causing the do ... while (reaped) loop in debugfs_reap_work to repeatedly re-traverse the XArrays and reap entries.
  3. Concurrency Management Starvation in Worker Pool: In kernel/workqueue.c, per-CPU worker pool concurrency management controls worker dispatch based on whether running workers are sleeping:

    • A pool checks need_more_worker(pool), which evaluates:
      !list_empty(&pool->worklist) && !pool->nr_running
      
    • Because kworker/0:2 is executing a non-blocking loop, it remains in TASK_RUNNING (state: R). Involuntary preemption by the scheduler (as seen in the trace where it is preempted via preempt_schedule() during xa_erase_irq()) does not call wq_worker_sleeping(), so pool->nr_running remains non-zero (pool->nr_running == 1).
    • Because pool->nr_running is positive, need_more_worker(pool) returns false. As a result, the 6 idle workers in pool 2 (idle: 5565 5476 9 1257 5306 10) are never awakened to handle any of the pending work items in pool->worklist.
    • Furthermore, because debugfs_reap_work does not finish, process_one_work() never completes, and pool->last_progress_ts is not updated.
    • After 30 seconds of no forward progress on pool 2, wq_watchdog_timer_fn() detects time_after(now, ts + thresh) and emits the BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 30s! splat.
▶ ▼ ✨ AI Jobs (2)
ID Workflow Result Correct Bug Created Started Finished Revision
66dbe068 repro-c Reproduced: ✅ ❓ BUG: workqueue lockup in debugfs_reap_work 2026/11/17 06:12 2026/11/17 06:12 2026/11/17 09:08 a3e64cfc
e0d1efd5 assessment-security DenialOfService: ✅ Exploitable: ❌ FilesystemTrigger: ❌ FuzzingArtifact: ❌ MemoryCorruption: ❌ NetworkNeedsSetup: ❌ NetworkTrigger: ❌ PeripheralTrigger: ❌ RemoteTrigger: ❌ Unprivileged: ❌ UserNamespace: ✅ VMGuestTrigger: ❌ VMHostTrigger: ❌ ❓ BUG: workqueue lockup in debugfs_reap_work 2026/11/14 21:18 2026/11/14 21:18 2026/11/14 22:01 819ebe34
▶ ▼ Last patch testing requests (2)
Created Duration User Patch Repo Result
2026/11/18 15:08 11m glider@google.com git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git ad3fc05c7e65009355effeea06aa629a687acef2 error job log
2026/11/17 15:22 22m retest repro git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git ad3fc05c7e65009355effeea06aa629a687acef2 report job log log

Sample crash report:
BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 30s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x100
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=17 refcnt=18
    in-flight: 879:debugfs_reap_work for 30s
    pending: 7*nsim_dev_hwstats_traffic_work, free_obj_work, 2*psi_avgs_work, vmstat_shepherd, 3*psi_avgs_work, netdev_work_proc, delayed_vfree_work
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=7 refcnt=8
    in-flight: 4800:nsim_fib_event_work for 43s ,26:nsim_fib_event_work for 44s nsim_fib_event_work
    pending: nsim_dev_hwstats_traffic_work, psi_avgs_work, reg_todo, delayed_vfree_work
workqueue events_unbound: flags=0x2
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=18 refcnt=19
    in-flight: 1911:cfg80211_wiphy_work for 33s cfg80211_wiphy_work ,1865:cfg80211_wiphy_work for 33s ,40:linkwatch_event for 27s ,577:cfg80211_wiphy_work for 28s ,49:cfg80211_wiphy_work for 32s ,1792:cfg80211_wiphy_work for 32s
    pending: 3*cfg80211_wiphy_work, 2*macvlan_process_broadcast, 2*cfg80211_wiphy_work, nsim_dev_trap_report_work, crng_reseed, 2*macvlan_process_broadcast
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=21 refcnt=22
    in-flight: 12:cfg80211_wiphy_work for 34s cfg80211_wiphy_work ,782:cfg80211_wiphy_work for 34s cfg80211_wiphy_work ,621:cfg80211_wiphy_work for 33s cfg80211_wiphy_work ,553:cfg80211_wiphy_work for 34s cfg80211_wiphy_work
    pending: 2*macvlan_process_broadcast, nsim_dev_trap_report_work, 2*macvlan_process_broadcast, 2*nsim_dev_trap_report_work, flush_memcg_stats_dwork, toggle_allocation_gate, 4*nsim_dev_trap_report_work
workqueue events_power_efficient: flags=0x82
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=12 refcnt=13
    pending: 2*neigh_periodic_work, neigh_managed_work, 2*neigh_periodic_work, gc_worker, 4*neigh_periodic_work, do_cache_clean, neigh_periodic_work
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=8 refcnt=9
    in-flight: 708:neigh_periodic_work for 2s
    pending: 2*neigh_periodic_work, fb_flashcursor, 2*neigh_periodic_work, wg_ratelimiter_gc_entries, neigh_periodic_work
workqueue events_dfl_long: flags=0x2
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=8 refcnt=9
    pending: 8*defense_work_handler
workqueue netns: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=4
    in-flight: 13:cleanup_net for 52s
workqueue mm_percpu_wq: flags=0x108
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: vmstat_update
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: vmstat_update
workqueue gid-cache-wq: flags=0x20002
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=4
    pending: update_gid_event_work_handler
workqueue ipv6_addrconf: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=6 MAYDAY
    in-flight: 59(RESCUER):addrconf_dad_work for 11s
    pending: mayday_cursor_func
workqueue bat_events: flags=0x6000a
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=55 MAYDAY
    in-flight: 2875(RESCUER):batadv_tt_purge for 10s
    pending: mayday_cursor_func
    inactive: 2*batadv_purge_orig, 2*batadv_iv_send_outstanding_bat_ogm_packet, 5*batadv_tt_purge, batadv_bla_periodic_work, batadv_dat_purge, 2*batadv_iv_send_outstanding_bat_ogm_packet, 7*batadv_mcast_mla_update, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_purge_orig, 3*batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, 3*batadv_iv_send_outstanding_bat_ogm_packet, batadv_purge_orig, 2*batadv_iv_send_outstanding_bat_ogm_packet, 2*batadv_purge_orig, 3*batadv_iv_send_outstanding_bat_ogm_packet, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_dat_purge, batadv_bla_periodic_work, batadv_tt_purge
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_encrypt_worker, wg_packet_tx_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg1: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg0: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg0: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=3 refcnt=4
    in-flight: 5388:wg_packet_tx_worker for 31s
    pending: wg_packet_tx_worker, wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-crypt-wg1: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=2 refcnt=3
    pending: wg_packet_decrypt_worker, wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue wg-kex-wg2: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: wg_packet_handshake_send_worker
workqueue wg-crypt-wg2: flags=0x128
  pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
  pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2
    pending: wg_packet_encrypt_worker
workqueue loop6: flags=0x6
  pwq 8: cpus=0-1 flags=0x6 nice=0 active=1 refcnt=2
    pending: loop_rootcg_workfn
pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=30s workers=8 idle: 5565 5476 9 1257 5306 10
pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=26s workers=12 idle: 24 4363 4811 4852 5473 5512 4815 4854 880 4828
pool 8: cpus=0-1 flags=0x6 nice=0 hung=0s workers=13 manager: 5923
Showing backtraces of busy workers in stalled worker pools:
pool 2:
task:kworker/0:2     state:R  running task     stack:0     pid:879   tgid:879   ppid:2      task_flags:0x4208060 flags:0x00000010
Workqueue: events debugfs_reap_work
Call trace:
 __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:792 (T)
 context_switch kernel/sched/core.c:5583 [inline]
 __schedule+0x1368/0x2b50 kernel/sched/core.c:7382
 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7561
 preempt_schedule+0x60/0x78 kernel/sched/core.c:7585
 __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:220 [inline]
 _raw_spin_unlock_irq+0x7c/0x80 kernel/locking/spinlock.c:229
 spin_unlock_irq include/linux/spinlock.h:413 [inline]
 xa_erase_irq include/linux/xarray.h:672 [inline]
 debugfs_reap_work+0x168/0x1e8 lib/ref_tracker.c:511
 process_one_work kernel/workqueue.c:3495 [inline]
 process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3578
 worker_thread+0x798/0xbd0 kernel/workqueue.c:3659
 kthread+0x304/0x3d4 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854
pool 2:
task:kworker/0:5     state:R  running task     stack:0     pid:5388  tgid:5388  ppid:2      task_flags:0x4208060 flags:0x00000010
Workqueue: wg-crypt-wg0 wg_packet_tx_worker
Call trace:
 __switch_to+0x2b4/0x6e4 arch/arm64/kernel/process.c:792 (T)
 context_switch kernel/sched/core.c:5583 [inline]
 __schedule+0x1368/0x2b50 kernel/sched/core.c:7382
 preempt_schedule_common+0xd4/0x190 kernel/sched/core.c:7561
 preempt_schedule+0x60/0x78 kernel/sched/core.c:7585
 __local_bh_enable_ip+0x16c/0x234 kernel/softirq.c:480
 __raw_read_unlock_bh include/linux/rwlock_api_smp.h:283 [inline]
 _raw_read_unlock_bh+0x3c/0x4c kernel/locking/spinlock.c:319
 wg_socket_send_skb_to_peer+0x150/0x19c drivers/net/wireguard/socket.c:183
 wg_packet_create_data_done drivers/net/wireguard/send.c:251 [inline]
 wg_packet_tx_worker+0x188/0x75c drivers/net/wireguard/send.c:276
 process_one_work kernel/workqueue.c:3495 [inline]
 process_scheduled_works+0x91c/0x1250 kernel/workqueue.c:3578
 worker_thread+0x798/0xbd0 kernel/workqueue.c:3659
 kthread+0x304/0x3d4 kernel/kthread.c:436
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:854

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2026/11/14 21:18 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci ad3fc05c7e65 819ebe34 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 BUG: workqueue lockup in debugfs_reap_work
* Struck through repros no longer work on HEAD.