KCSAN: data-race in assign_work / queue_work_on

Status: moderation: reported on 2026/11/20 00:01
Subsystems: block
Labels: prio:high race:benign
[Documentation on labels]
Reported-by: syzbot+eb6c66d8f130ac27fe3a@syzkaller.appspotmail.com
First crash: 2d22h, last: 4h14m
✨ AI-generated root cause summary:

A KCSAN data race occurs between move_linked_works() and queue_work_on() when accessing struct work_struct::data. In move_linked_works(), WORK_STRUCT_LINKED is inspected via an unmarked plain load (*work_data_bits(work) & WORK_STRUCT_LINKED) while holding pool->lock. Concurrently, queue_work_on() locklessly executes test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)), performing an instrumented atomic read-modify-write on the same 8-byte word. In production kernels this race is benign, as WORK_STRUCT_PENDING_BIT is already set and test_and_set_bit() does not alter WORK_STRUCT_LINKED_BIT.

▶ ▼ ✨ Full AI root cause analysis
Based on kernel commit 4405c18c95d2:

Technical Root Cause Analysis

1. Functions, Structs, and Fields Involved

  • struct work_struct (include/linux/workqueue_types.h:16-23): Contains atomic_long_t data (field at offset 0). The data field packs several status flags in its lowest bits (including WORK_STRUCT_PENDING_BIT at bit 0 and WORK_STRUCT_LINKED_BIT at bit 3) alongside pointer / color / pool metadata (include/linux/workqueue.h:26-74).
  • #define work_data_bits(work) (include/linux/workqueue.h:24): Casts (unsigned long *)(&(work)->data).
  • move_linked_works() (kernel/workqueue.c:1200-1222), called by assign_work() (kernel/workqueue.c:1240-1274): Invoked by worker threads (worker_thread() / bh_worker()) under pool->lock to transfer a work item and any associated barrier work items linked behind it from pool->worklist to worker->scheduled.
  • queue_work_on() (kernel/workqueue.c:2589-2605): Invoked locklessly (here from WireGuard's transmit path: wg_xmit() -> wg_packet_create_data() -> wg_queue_enqueue_per_device_and_peer()).

2. Triggering Mechanism & Race Condition

  1. While a work item (here the per-CPU encryption work item of WireGuard's encrypt_queue) is queued on pool->worklist, its WORK_STRUCT_PENDING_BIT is set to 1. This bit remains 1 until execution starts and set_work_pool_and_clear_pending() is invoked in process_one_work() (kernel/workqueue.c:3456).
  2. On CPU 1, a worker thread (PID 3557) picks up this work item from pool->worklist and executes assign_work() -> move_linked_works(). At line 1211 of kernel/workqueue.c:
    list_for_each_entry_safe_from(work, n, NULL, entry) {
        list_move_tail(&work->entry, head);
        if (!(*work_data_bits(work) & WORK_STRUCT_LINKED))
            break;
    }
    
    The condition *work_data_bits(work) & WORK_STRUCT_LINKED performs an uninstrumented, unmarked plain load from work->data.
  3. Concurrently on CPU 0, task 6710 is handling network traffic and attempts to queue new encryption work on the same per-CPU work struct by calling queue_work_on(). At line 2597:
    if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work)) &&
        !clear_pending_if_disabled(work)) {
        __queue_work(cpu, wq, work);
        ret = true;
    }
    
    test_and_set_bit() executes an instrumented atomic read-modify-write on the same 8-byte word work->data (0xffffe8ffffd391c0) without holding pool->lock.
  4. Because one CPU performs an atomic read-modify-write while the other CPU performs a plain, unmarked load to the exact same word, KCSAN detects a data race between assign_work and queue_work_on.

3. Invariant & Lifecycle Analysis

  • Under the Linux Kernel Memory Model, racing a plain load against a concurrent atomic write constitutes a data race.
  • In core workqueue, WORK_STRUCT_LINKED_BIT is modified solely under pool->lock (during insert_wq_barrier() and work processing).
  • The concurrent test_and_set_bit(WORK_STRUCT_PENDING_BIT, ...) sets bit 0, which is already 1, leaving all other bits (including WORK_STRUCT_LINKED_BIT) unchanged.
  • Using an atomic or marked read primitiveβ€”such as test_bit(WORK_STRUCT_LINKED_BIT, work_data_bits(work)) or atomic_long_read(&work->data)β€”in move_linked_works() marks the access appropriately and eliminates the data race.
▶ ▼ ✨ AI Jobs (2)
ID Workflow Result Correct Bug Created Started Finished Revision
979ee466 assessment-kcsan Benign: βœ… ❓ KCSAN: data-race in assign_work / queue_work_on 2026/11/19 11:34 2026/11/19 11:34 2026/11/19 11:49 2006bdcc
f46df8a3 assessment-security DenialOfService: ❌ Exploitable: ❌ FilesystemTrigger: ❌ FuzzingArtifact: ❌ MemoryCorruption: ❌ NetworkNeedsSetup: βœ… NetworkTrigger: βœ… PeripheralTrigger: ❌ RemoteTrigger: βœ… Unprivileged: βœ… UserNamespace: βœ… VMGuestTrigger: ❌ VMHostTrigger: ❌ ❓ KCSAN: data-race in assign_work / queue_work_on 2026/11/19 11:33 2026/11/19 11:33 2026/11/19 11:47 2006bdcc

Sample crash report:
==================================================================
BUG: KCSAN: data-race in assign_work / queue_work_on

read-write (marked) to 0xffff8881024444c0 of 8 bytes by interrupt on cpu 0:
 instrument_atomic_read_write include/linux/instrumented.h:113 [inline]
 test_and_set_bit include/asm-generic/bitops/instrumented-atomic.h:71 [inline]
 queue_work_on+0x42/0x150 kernel/workqueue.c:2597
 queue_work include/linux/workqueue.h:700 [inline]
 tty_buffer_queue_work drivers/tty/tty_buffer.c:66 [inline]
 tty_flip_buffer_push+0x82/0x90 drivers/tty/tty_buffer.c:540
 serial8250_rx_chars drivers/tty/serial/8250/8250_port.c:1704 [inline]
 serial8250_handle_irq_locked+0x40a/0x500 drivers/tty/serial/8250/8250_port.c:1868
 serial8250_handle_irq+0xb5/0x260 drivers/tty/serial/8250/8250_port.c:1889
 serial8250_default_handle_irq+0x65/0xd0 drivers/tty/serial/8250/8250_port.c:1907
 serial8250_interrupt+0x63/0x110 drivers/tty/serial/8250/8250_core.c:87
 __handle_irq_event_percpu+0x92/0x4a0 kernel/irq/handle.c:209
 handle_irq_event_percpu kernel/irq/handle.c:246 [inline]
 handle_irq_event+0x64/0xf0 kernel/irq/handle.c:263
 handle_edge_irq+0x155/0x440 kernel/irq/chip.c:859
 generic_handle_irq_desc include/linux/irqdesc.h:188 [inline]
 handle_irq arch/x86/kernel/irq.c:209 [inline]
 call_irq_handler arch/x86/kernel/irq.c:-1 [inline]
 __common_interrupt+0x60/0xb0 arch/x86/kernel/irq.c:280
 common_interrupt+0x3e/0x90 arch/x86/kernel/irq.c:273
 asm_common_interrupt+0x26/0x40 arch/x86/include/asm/idtentry.h:665
 __sanitizer_cov_trace_cmp8+0x0/0xb0 kernel/kcov.c:303
 pie_calculate_probability+0x2f3/0x4c0 net/sched/sch_pie.c:392
 fq_pie_timer+0xf6/0x1d0 net/sched/sch_fq_pie.c:403
 call_timer_fn+0x36/0x280 kernel/time/timer.c:1749
 expire_timers kernel/time/timer.c:1800 [inline]
 __run_timers kernel/time/timer.c:2375 [inline]
 __run_timer_base+0x40c/0x610 kernel/time/timer.c:2387
 run_timer_base kernel/time/timer.c:2396 [inline]
 run_timer_softirq+0x31/0x70 kernel/time/timer.c:2406
 handle_softirqs+0x109/0x310 kernel/softirq.c:669
 __do_softirq kernel/softirq.c:703 [inline]
 invoke_softirq kernel/softirq.c:542 [inline]
 __irq_exit_rcu+0x40/0xf0 kernel/softirq.c:786
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1062 [inline]
 sysvec_apic_timer_interrupt+0x74/0x80 arch/x86/kernel/apic/apic.c:1062
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:674
 constant_test_bit arch/x86/include/asm/bitops.h:202 [inline]
 arch_test_bit arch/x86/include/asm/bitops.h:232 [inline]
 _test_bit include/asm-generic/bitops/instrumented-non-atomic.h:142 [inline]
 folio_test_unevictable include/linux/page-flags.h:641 [inline]
 lru_gen_inc_refs mm/folio.c:386 [inline]
 folio_mark_accessed+0x93/0x460 mm/folio.c:457
 folio_add_lru+0x14a/0x190 mm/folio.c:515
 folio_add_lru_vma+0x5d/0x80 mm/folio.c:537
 wp_page_copy mm/memory.c:4172 [inline]
 do_wp_page+0x13e8/0x2100 mm/memory.c:4569
 handle_pte_fault mm/memory.c:6716 [inline]
 __handle_mm_fault mm/memory.c:6841 [inline]
 handle_mm_fault+0xe72/0x3880 mm/memory.c:7008
 do_user_addr_fault+0x3ce/0xde0 arch/x86/mm/fault.c:1343
 handle_page_fault arch/x86/mm/fault.c:1483 [inline]
 exc_page_fault+0x62/0xa0 arch/x86/mm/fault.c:1536
 asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:595

read to 0xffff8881024444c0 of 8 bytes by task 421 on cpu 1:
 move_linked_works kernel/workqueue.c:1211 [inline]
 assign_work+0x1c5/0x310 kernel/workqueue.c:1272
 worker_thread+0x55b/0x770 kernel/workqueue.c:3658
 kthread+0x248/0x2a0 kernel/kthread.c:436
 ret_from_fork+0x146/0x330 arch/x86/kernel/process.c:158
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245

Reported by Kernel Concurrency Sanitizer on:
CPU: 1 UID: 0 PID: 421 Comm: kworker/u8:7 Not tainted syzkaller #0 PREEMPT(lazy) 
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 11/06/2026
Workqueue:  0x0 (ttyS-serial_8250)
==================================================================

Crashes (12):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2026/11/19 20:42 upstream 8dac6df92a2e 229865fb .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/19 07:30 upstream 4405c18c95d2 f138241c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/18 20:40 upstream 4405c18c95d2 2006bdcc .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/18 19:36 upstream 20eef29b7c84 2006bdcc .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/18 19:36 upstream 20eef29b7c84 2006bdcc .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/18 07:12 upstream 20eef29b7c84 eb4895d7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/17 21:41 upstream 63fdb511c27a 122f94d5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/17 21:40 upstream 63fdb511c27a 122f94d5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/17 21:40 upstream 63fdb511c27a 122f94d5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/17 17:08 upstream 666539eecdf4 122f94d5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/17 10:40 upstream 666539eecdf4 43af6cac .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
2026/11/17 02:14 upstream 666539eecdf4 a3e64cfc .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-kcsan-gce KCSAN: data-race in assign_work / queue_work_on
* Struck through repros no longer work on HEAD.