| 0/0 |
2026/05/03 02:22 |
flow |
assessment-kcsan |
22m
Results: map[Benign:true Confident:true Explanation:The data race occurs between **`folios_put_refs()`** (the writer) and
**`need_mlock_drain()`** (the reader) on the `nr` field of a `struct
folio_batch`. This batch is part of the per-CPU `mlock_fbatch` structure used to
amortize the cost of `mlock` operations.
### Analysis
**1. Context:**
- **Writer (`mlock_folio` path):** Task 25650 is executing `mlock_folio()`,
which adds a folio to the per-CPU `mlock_fbatch`. If the batch becomes full or
if LRU caching is disabled, it calls `mlock_folio_batch()`, which eventually
calls `folios_put_refs()`. The writer holds the `mlock_fbatch.lock` (a
`local_lock`, which on non-RT kernels is a preemption-disable region).
- **Reader (`lru_add_drain_all` path):** Task 25645 is executing
`lru_add_drain_all()`, which iterates over all online CPUs and checks if their
per-CPU batches need draining. It calls `cpu_needs_drain()`, which in turn calls
`need_mlock_drain()`. This check is performed **locklessly** on remote CPUs to
avoid the overhead of acquiring locks for every CPU.
**2. Data Flow:**
- The reader `need_mlock_drain()` reads `fbatch->nr` to decide whether to
schedule a `lru_add_drain_per_cpu` work item on that CPU.
- If it sees `nr > 0`, it schedules the work. The work item, when it runs on the
target CPU, will acquire the `mlock_fbatch.lock` and re-check `nr` before
processing the batch.
- If it sees `nr == 0` (stale or new), it simply skips that CPU for the current
draining round.
**3. Safety:**
- **Torn Accesses:** The `nr` field is an `unsigned char` (1 byte). Torn
reads/writes are impossible for single-byte accesses on all supported
architectures.
- **Logic Soundness:**
- If the reader sees a non-zero value while the writer is already clearing
the batch (setting `nr = 0`), it will schedule redundant work. This work
will safely see `nr == 0` after acquiring the lock and do nothing.
- If the reader misses a non-zero value because it sees a stale `0`, it
might delay the drain. However, `lru_add_drain_all()` is generally a
best-effort flush. In cases where strict synchronization is required (e.g.,
`lru_cache_disable()`), other mechanisms like `synchronize_rcu()` ensure
that preceding batch updates are visible.
- The `local_lock` (preemption disable) on the writer side ensures that the
scheduled work item cannot start on the same CPU until the writer has
finished its critical section and released the lock, ensuring the work sees
a consistent state.
### Final Classification: **BENIGN**
**Reasoning:**
The race is a classic "lockless-check-before-lock" optimization. The read is
used as a heuristic hint to avoid unnecessary work scheduling. Correctness is
guaranteed because the actual processing of the batch is performed under the
`mlock_fbatch.lock`. The variable is a single byte, precluding torn accesses.
**Recommended Annotations:**
The lockless read in `need_mlock_drain()` should be wrapped in `data_race()` or
`READ_ONCE()`.
```c
// mm/mlock.c
bool need_mlock_drain(int cpu)
{
return data_race(folio_batch_count(&per_cpu(mlock_fbatch.fbatch, cpu)));
}
```]
|
| 1/1 |
2026/05/03 02:22 |
action |
kernel-checkouter |
1m
Results: map[KernelSrc:/app/workdir/cache/src/6aa0292bc8fba0fa86180056e2f57667f77b80e5]
|
| 2/1 |
2026/05/03 02:23 |
action |
kernel-builder |
9m
Results: map[KernelObj:/app/workdir/cache/build/1d7885d2f790864290e12e5ebb3d064401fda5b8]
|
| 3/1 |
2026/05/03 02:32 |
action |
codesearch-prepare |
7m
Results: map[Index:codesearch-index]
|
| 4/1 |
2026/05/03 02:40 |
agent |
expert |
3m
Model:
Results: map[Benign:true Confident:true]
Instruction: # KCSAN Data Race Severity Analysis Protocol
You are an expert Linux kernel concurrency engineer analyzing a Kernel
Concurrency Sanitizer (KCSAN) data race report to determine if it is
**BENIGN** or **HARMFUL**.
## 1. CLASSIFICATIONS
### **BENIGN (Truly Benign)**
The logic is sound and structurally tolerant to compiler optimizations or
stale/torn reads.
- **Diagnostics/Stats:** Reads used only for '/proc', '/sys', counters, or
'pr_info'.
- **Heuristic Hints:** A "hint" flag where an old value only causes a
slightly delayed update or a sub-optimal but safe fast-path.
- **Single-Writer Flag Updates:** A single writer updating flags where the
concurrent read is a simple bitwise check (e.g., 'flags & MASK'). These are
historically tolerated, assuming neither "Fused Accesses" nor "Ordering
Violations" are relevant in this context.
- **Marked Reloads:** A load feeding into a 'cmpxchg()' loop or checked
against a later 'READ_ONCE()' reload.
- **Safe Overwrites:** Writing the same value already present.
### **HARMFUL (Logic Bug or Marking Required)**
The race causes incorrect behavior due to a synchronization failure or
because missing annotations allow the compiler to break the algorithm.
**Marking Required for Correctness:**
The algorithm is logically sound but requires annotations ('READ_ONCE()',
'WRITE_ONCE()', 'smp_load_acquire()', 'smp_store_release()', etc.) to be safe.
- **Fused Accesses:** The compiler might merge accesses or hoist a load out
of a loop, breaking polling/wait loops (livelocks).
- **Torn Accesses:** A large access (e.g., 64-bit on 32-bit arch) might be
split into multiple non-atomic accesses. Note that 'READ_ONCE()' does **not**
guarantee atomicity for 64-bit variables on 32-bit architectures.
- **Ordering Violations:** The race breaks a "happens-before" relationship
(requires primitives with implied or explicit memory barriers).
**Logic Bugs:**
A fundamental synchronization failure. Marking accesses will **not** fix it;
the logic itself must change.
- **Pointers/Lifecycle:** The racing variable is a pointer being dereferenced
or a refcount governing object lifecycle (Use-After-Free risk).
- **Control Flow:** The variable guards a critical section, memory allocation,
or hardware command.
- **Bitfields:** Concurrent writes to different bits in the same word.
Compilers often use non-atomic read-modify-write sequences, meaning a
write to 'bit_A' can "clobber" a concurrent write to 'bit_B'. However,
do not blindly assume all bitfield accesses are harmful; you must prove
that a concurrent write actually clobbers another in a way that breaks
logic.
- **Complex Structures:** Races on shared lists, trees, or hashmaps.
- **Lossy Updates:** Concurrent plain RMW operations (e.g., 'var++') on
non-diagnostic variables where every increment must be preserved.
- **State Machines:** Races allowing a state machine to bypass transitions
or enter an invalid state.
- **Adjacent Unsynchronized Operations:** Consider races happening at the
same time. For example, if both threads execute 'struct->has_elements = true;
list_add(node, &struct->list);', the race on 'has_elements' implies an
adjacent race on 'list_head', which is HARMFUL.
## 2. RESEARCH & ANALYSIS WORKFLOW
1. **Locate the Race:** Find the exact variables and functions in the stack
traces. **Use codesearch tools to read the actual source code and
confirm all assumptions.** Do not speculate about hypothetical compiler
behaviors or theoretical dangers (e.g., dismissing something as
"fundamentally unsafe") without tracing the actual data flow to a crash.
2. **Contextualize:** Identify held locks, RCU sections, or interrupt
contexts. Explain why the race is possible (e.g., "Thread A holds 'lock',
but Thread B is a lockless reader").
3. **Data Flow:** Follow the racing variable into its subsequent uses. If
the reader sees a stale or torn value, what branch is taken? Does it lead
to a pointer dereference or 'BUG_ON()'? **You must prove harm via this
data flow.**
4. **Failure Sequencing (if HARMFUL):** Construct a two-column execution trace
demonstrating the failure.
*Example format:*
CPU0 CPU1
function_a()
// lockless read
ptr = global_ptr
<PREEMPT>
function_b()
lock(&my_lock)
global_ptr = NULL
unlock(&my_lock)
kfree(ptr)
if (ptr)
*ptr = 1; // -> Use-After-Free!
## 3. OUTPUT FORMAT
- **Race Summary:** '[Function A]' vs '[Function B]' on '[Variable/Field]'.
- **Synchronization context:** List held locks and explain the lack of mutual
exclusion.
- **Final Classification:** **BENIGN** or **HARMFUL**.
**If BENIGN:**
- **Reasoning:** Briefly explain why the race is structurally tolerant to stale
or torn reads.
- **Recommended Annotations:** Suggest appropriate annotations (e.g.,
'READ_ONCE()', 'data_race()').
**If HARMFUL:**
- **Failure Reasoning:** Explain and provide a two-column interleaving showing
exactly how the race leads to a failure such as corruption or kernel crash.
- **Recommended Fix (ONLY if fix is trivial):** Suggest a structural fix (e.g.,
"Hold 'mapping->i_pages' lock", "Convert to 'atomic_t'") or required memory
ordering annotations (e.g., "Wrap in 'READ_ONCE()'", "Use
'smp_load_acquire()'").
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt:
The data race report is:
EXT4-fs (loop9): unmounting filesystem 00000000-0000-0000-0000-000000000000.
==================================================================
BUG: KCSAN: data-race in folios_put_refs / need_mlock_drain
write to 0xffff888237d27248 of 1 bytes by task 25650 on cpu 1:
folio_batch_reinit include/linux/folio_batch.h:50 [inline]
folios_put_refs+0x28f/0x2d0 mm/swap.c:1002
folios_put include/linux/mm.h:2149 [inline]
mlock_folio_batch+0x1bae/0x1be0 mm/mlock.c:209
mlock_folio+0x1d1/0x240 mm/mlock.c:259
mlock_vma_folio mm/internal.h:1199 [inline]
__folio_add_anon_rmap mm/rmap.c:1576 [inline]
folio_add_anon_rmap_ptes+0x24e/0x250 mm/rmap.c:1599
remove_migration_pte+0x7d9/0xbf0 mm/migrate.c:432
rmap_walk_anon+0x280/0x430 mm/rmap.c:2999
rmap_walk+0x8b/0xd0 mm/rmap.c:3104
remove_migration_ptes mm/migrate.c:473 [inline]
migrate_folio_move mm/migrate.c:1411 [inline]
migrate_folios_move mm/migrate.c:1740 [inline]
migrate_pages_batch+0x14cc/0x2010 mm/migrate.c:1996
migrate_pages_sync mm/migrate.c:2026 [inline]
migrate_pages+0x102e/0x1860 mm/migrate.c:2135
do_mbind mm/mempolicy.c:1614 [inline]
kernel_mbind mm/mempolicy.c:1757 [inline]
__do_sys_mbind mm/mempolicy.c:1831 [inline]
__se_sys_mbind+0xc98/0xdd0 mm/mempolicy.c:1827
__x64_sys_mbind+0x78/0x90 mm/mempolicy.c:1827
x64_sys_call+0x2b72/0x3020 arch/x86/include/generated/asm/syscalls_64.h:238
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0x12c/0x3b0 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
read to 0xffff888237d27248 of 1 bytes by task 25645 on cpu 0:
folio_batch_count include/linux/folio_batch.h:56 [inline]
need_mlock_drain+0x30/0x50 mm/mlock.c:235
cpu_needs_drain mm/swap.c:792 [inline]
__lru_add_drain_all+0x273/0x450 mm/swap.c:883
lru_add_drain_all+0x10/0x20 mm/swap.c:899
invalidate_bdev+0x47/0x70 block/bdev.c:101
ext4_put_super+0x610/0x7b0 fs/ext4/super.c:1355
generic_shutdown_super+0xee/0x220 fs/super.c:646
kill_block_super+0x2a/0x70 fs/super.c:1725
ext4_kill_sb+0x42/0x80 fs/ext4/super.c:7492
deactivate_locked_super+0x75/0x1c0 fs/super.c:476
deactivate_super+0x97/0xa0 fs/super.c:509
cleanup_mnt+0x2bb/0x330 fs/namespace.c:1312
__cleanup_mnt+0x19/0x20 fs/namespace.c:1319
task_work_run+0x130/0x1a0 kernel/task_work.c:233
resume_user_mode_work include/linux/resume_user_mode.h:50 [inline]
__exit_to_user_mode_loop kernel/entry/common.c:67 [inline]
exit_to_user_mode_loop+0x1f4/0x6f0 kernel/entry/common.c:98
__exit_to_user_mode_prepare include/linux/irq-entry-common.h:207 [inline]
syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:238 [inline]
syscall_exit_to_user_mode include/linux/entry-common.h:318 [inline]
do_syscall_64+0x232/0x3b0 arch/x86/entry/syscall_64.c:100
entry_SYSCALL_64_after_hwframe+0x77/0x7f
value changed: 0x01 -> 0x00
Reported by Kernel Concurrency Sanitizer on:
CPU: 0 UID: 0 PID: 25645 Comm: syz.9.5684 Tainted: G W syzkaller #0 PREEMPT(full)
Tainted: [W]=WARN
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/18/2026
==================================================================
Reply: The data race occurs between **`folios_put_refs()`** (the writer) and **`need_mlock_drain()`** (the reader) on the `nr` field of a `struct folio_batch`. This batch is part of the per-CPU `mlock_fbatch` structure used to amortize the cost of `mlock` operations.
### Analysis
**1. Context:**
- **Writer (`mlock_folio` path):** Task 25650 is executing `mlock_folio()`, which adds a folio to the per-CPU `mlock_fbatch`. If the batch becomes full or if LRU caching is disabled, it calls `mlock_folio_batch()`, which eventually calls `folios_put_refs()`. The writer holds the `mlock_fbatch.lock` (a `local_lock`, which on non-RT kernels is a preemption-disable region).
- **Reader (`lru_add_drain_all` path):** Task 25645 is executing `lru_add_drain_all()`, which iterates over all online CPUs and checks if their per-CPU batches need draining. It calls `cpu_needs_drain()`, which in turn calls `need_mlock_drain()`. This check is performed **locklessly** on remote CPUs to avoid the overhead of acquiring locks for every CPU.
**2. Data Flow:**
- The reader `need_mlock_drain()` reads `fbatch->nr` to decide whether to schedule a `lru_add_drain_per_cpu` work item on that CPU.
- If it sees `nr > 0`, it schedules the work. The work item, when it runs on the target CPU, will acquire the `mlock_fbatch.lock` and re-check `nr` before processing the batch.
- If it sees `nr == 0` (stale or new), it simply skips that CPU for the current draining round.
**3. Safety:**
- **Torn Accesses:** The `nr` field is an `unsigned char` (1 byte). Torn reads/writes are impossible for single-byte accesses on all supported architectures.
- **Logic Soundness:**
- If the reader sees a non-zero value while the writer is already clearing the batch (setting `nr = 0`), it will schedule redundant work. This work will safely see `nr == 0` after acquiring the lock and do nothing.
- If the reader misses a non-zero value because it sees a stale `0`, it might delay the drain. However, `lru_add_drain_all()` is generally a best-effort flush. In cases where strict synchronization is required (e.g., `lru_cache_disable()`), other mechanisms like `synchronize_rcu()` ensure that preceding batch updates are visible.
- The `local_lock` (preemption disable) on the writer side ensures that the scheduled work item cannot start on the same CPU until the writer has finished its critical section and released the lock, ensuring the work sees a consistent state.
### Final Classification: **BENIGN**
**Reasoning:**
The race is a classic "lockless-check-before-lock" optimization. The read is used as a heuristic hint to avoid unnecessary work scheduling. Correctness is guaranteed because the actual processing of the batch is performed under the `mlock_fbatch.lock`. The variable is a single byte, precluding torn accesses.
**Recommended Annotations:**
The lockless read in `need_mlock_drain()` should be wrapped in `data_race()` or `READ_ONCE()`.
```c
// mm/mlock.c
bool need_mlock_drain(int cpu)
{
return data_race(folio_batch_count(&per_cpu(mlock_fbatch.fbatch, cpu)));
}
```
|
| 5/2 |
2026/05/03 02:40 |
llm |
expert |
0m
Model:
Tokens:
input: 5540
output: 128
thoughts: 184
Thoughts: **Analyzing a Data Race in the Folio Batch System**
Okay, so I'm looking at a data race, a classic. It seems the conflict is between `folios_put_refs` (the writer) and `need_mlock_drain` (the reader), specifically on a single-byte field. The address is `0xffff888237d27248`, and the value flipped from `0x01` to `0x00`. Let's dig in.
The writer, `folios_put_refs`, calls `folio_batch_reinit`. The reader, `need_mlock_drain`, calls `folio_batch_count`. Hmm, `folio_batch_reinit` and `folio_batch_count`... those sound like they're working with a `struct folio_batch`. Considering the size of the field and the functions involved, I'd bet the contentious field is `nr` within that `struct folio_batch`. This looks like a classic counter increment/decrement scenario. I'll need to double-check the structure definition and these function implementations to be absolutely sure, but I'm thinking we're on the right track. Time to cross-reference and confirm my initial hypothesis.
|
| 6/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/linux/folio_batch.h IncludeLines:true Name:folio_batch_reinit]
Results: map[SourceCode: 48: static inline void folio_batch_reinit(struct folio_batch *fbatch)
49: {
50: fbatch->nr = 0;
51: fbatch->i = 0;
52: }
SourceFile:include/linux/folio_batch.h]
|
| 7/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/linux/folio_batch.h IncludeLines:true Name:folio_batch_count]
Results: map[SourceCode: 54: static inline unsigned int folio_batch_count(const struct folio_batch *fbatch)
55: {
56: return fbatch->nr;
57: }
SourceFile:include/linux/folio_batch.h]
|
| 8/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/linux/folio_batch.h IncludeLines:true Name:folio_batch]
Results: map[SourceCode: 19: /**
20: * struct folio_batch - A collection of folios.
21: *
22: * The folio_batch is used to amortise the cost of retrieving and
23: * operating on a set of folios. The order of folios in the batch may be
24: * significant (eg delete_from_page_cache_batch()). Some users of the
25: * folio_batch store "exceptional" entries in it which can be removed
26: * by calling folio_batch_remove_exceptionals().
27: */
28: struct folio_batch {
29: unsigned char nr;
30: unsigned char i;
31: bool percpu_pvec_drained;
32: struct folio *folios[FOLIO_BATCH_SIZE];
33: };
SourceFile:include/linux/folio_batch.h]
|
| 9/2 |
2026/05/03 02:40 |
llm |
expert |
0m
Model:
Tokens:
input: 6551
output: 164
thoughts: 214
Thoughts: **Analyzing a Contention Point in Memory Management**
Okay, so I'm tracking down a potential bottleneck, specifically related to folio reference management. The core issue seems to revolve around `fbatch->nr`, the batch counter, and the race condition related to it. My current line of thought points to a conflict between writer and reader operations on these batches.
On the writer side, `folios_put_refs` calls `folio_batch_reinit`, and this is where `fbatch->nr` gets reset to zero. The stack trace clearly indicates this sequence: `folios_put_refs -> mlock_folio_batch`.
Now, the reader side is where things get interesting. The stack trace starts with `need_mlock_drain`, then proceeds to `cpu_needs_drain`, and finally to `__lru_add_drain_all`, which suggests an interaction with the LRU list draining, potentially interacting with the folio batches. This all points to a read operation interacting with the writers.
The immediate next step is to carefully examine the code in both `mm/swap.c` and `mm/mlock.c`. I need to understand how the writer manipulates `fbatch->nr` and how the reader uses it, and then determine how the potential race condition might occur. I need to understand if the writer is interrupting the reader or vice versa. This will involve analyzing the memory barriers, locking mechanisms, and data access patterns within these functions. The ultimate goal is to pinpoint the exact window of opportunity for the race and propose a suitable synchronization mechanism to resolve this contention.
|
| 10/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/swap.c IncludeLines:true Name:folios_put_refs]
Results: map[SourceCode: 942: /**
943: * folios_put_refs - Reduce the reference count on a batch of folios.
944: * @folios: The folios.
945: * @refs: The number of refs to subtract from each folio.
946: *
947: * Like folio_put(), but for a batch of folios. This is more efficient
948: * than writing the loop yourself as it will optimise the locks which need
949: * to be taken if the folios are freed. The folios batch is returned
950: * empty and ready to be reused for another batch; there is no need
951: * to reinitialise it. If @refs is NULL, we subtract one from each
952: * folio refcount.
953: *
954: * Context: May be called in process or interrupt context, but not in NMI
955: * context. May be called while holding a spinlock.
956: */
957: void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
958: {
959: int i, j;
960: struct lruvec *lruvec = NULL;
961: unsigned long flags = 0;
962:
963: for (i = 0, j = 0; i < folios->nr; i++) {
964: struct folio *folio = folios->folios[i];
965: unsigned int nr_refs = refs ? refs[i] : 1;
966:
967: if (is_huge_zero_folio(folio))
968: continue;
969:
970: if (folio_is_zone_device(folio)) {
971: if (lruvec) {
972: lruvec_unlock_irqrestore(lruvec, flags);
973: lruvec = NULL;
974: }
975: if (folio_ref_sub_and_test(folio, nr_refs))
976: free_zone_device_folio(folio);
977: continue;
978: }
979:
980: if (!folio_ref_sub_and_test(folio, nr_refs))
981: continue;
982:
983: /* hugetlb has its own memcg */
984: if (folio_test_hugetlb(folio)) {
985: if (lruvec) {
986: lruvec_unlock_irqrestore(lruvec, flags);
987: lruvec = NULL;
988: }
989: free_huge_folio(folio);
990: continue;
991: }
992: folio_unqueue_deferred_split(folio);
993: __page_cache_release(folio, &lruvec, &flags);
994:
995: if (j != i)
996: folios->folios[j] = folio;
997: j++;
998: }
999: if (lruvec)
1000: lruvec_unlock_irqrestore(lruvec, flags);
1001: if (!j) {
1002: folio_batch_reinit(folios);
1003: return;
1004: }
1005:
1006: folios->nr = j;
1007: mem_cgroup_uncharge_folios(folios);
1008: free_unref_folios(folios);
1009: }
SourceFile:mm/swap.c]
|
| 11/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/mlock.c IncludeLines:true Name:mlock_folio_batch]
Results: map[SourceCode: 179: /*
180: * mlock_folio_batch() is derived from folio_batch_move_lru(): perhaps that can
181: * make use of such folio pointer flags in future, but for now just keep it for
182: * mlock. We could use three separate folio batches instead, but one feels
183: * better (munlocking a full folio batch does not need to drain mlocking folio
184: * batches first).
185: */
186: static void mlock_folio_batch(struct folio_batch *fbatch)
187: {
188: struct lruvec *lruvec = NULL;
189: unsigned long mlock;
190: struct folio *folio;
191: int i;
192:
193: for (i = 0; i < folio_batch_count(fbatch); i++) {
194: folio = fbatch->folios[i];
195: mlock = (unsigned long)folio & (LRU_FOLIO | NEW_FOLIO);
196: folio = (struct folio *)((unsigned long)folio - mlock);
197: fbatch->folios[i] = folio;
198:
199: if (mlock & LRU_FOLIO)
200: lruvec = __mlock_folio(folio, lruvec);
201: else if (mlock & NEW_FOLIO)
202: lruvec = __mlock_new_folio(folio, lruvec);
203: else
204: lruvec = __munlock_folio(folio, lruvec);
205: }
206:
207: if (lruvec)
208: lruvec_unlock_irq(lruvec);
209: folios_put(fbatch);
210: }
SourceFile:mm/mlock.c]
|
| 12/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/mlock.c IncludeLines:true Name:need_mlock_drain]
Results: map[SourceCode: 233: bool need_mlock_drain(int cpu)
234: {
235: return folio_batch_count(&per_cpu(mlock_fbatch.fbatch, cpu));
236: }
SourceFile:mm/mlock.c]
|
| 13/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/swap.c IncludeLines:true Name:__lru_add_drain_all]
Results: map[SourceCode: 796: /*
797: * Doesn't need any cpu hotplug locking because we do rely on per-cpu
798: * kworkers being shut down before our page_alloc_cpu_dead callback is
799: * executed on the offlined cpu.
800: * Calling this function with cpu hotplug locks held can actually lead
801: * to obscure indirect dependencies via WQ context.
802: */
803: static inline void __lru_add_drain_all(bool force_all_cpus)
804: {
805: /*
806: * lru_drain_gen - Global pages generation number
807: *
808: * (A) Definition: global lru_drain_gen = x implies that all generations
809: * 0 < n <= x are already *scheduled* for draining.
810: *
811: * This is an optimization for the highly-contended use case where a
812: * user space workload keeps constantly generating a flow of pages for
813: * each CPU.
814: */
815: static unsigned int lru_drain_gen;
816: static struct cpumask has_work;
817: static DEFINE_MUTEX(lock);
818: unsigned cpu, this_gen;
819:
820: /*
821: * Make sure nobody triggers this path before mm_percpu_wq is fully
822: * initialized.
823: */
824: if (WARN_ON(!mm_percpu_wq))
825: return;
826:
827: /*
828: * Guarantee folio_batch counter stores visible by this CPU
829: * are visible to other CPUs before loading the current drain
830: * generation.
831: */
832: smp_mb();
833:
834: /*
835: * (B) Locally cache global LRU draining generation number
836: *
837: * The read barrier ensures that the counter is loaded before the mutex
838: * is taken. It pairs with smp_mb() inside the mutex critical section
839: * at (D).
840: */
841: this_gen = smp_load_acquire(&lru_drain_gen);
842:
843: /* It helps everyone if we do our own local drain immediately. */
844: lru_add_drain();
845:
846: mutex_lock(&lock);
847:
848: /*
849: * (C) Exit the draining operation if a newer generation, from another
850: * lru_add_drain_all(), was already scheduled for draining. Check (A).
851: */
852: if (unlikely(this_gen != lru_drain_gen && !force_all_cpus))
853: goto done;
854:
855: /*
856: * (D) Increment global generation number
857: *
858: * Pairs with smp_load_acquire() at (B), outside of the critical
859: * section. Use a full memory barrier to guarantee that the
860: * new global drain generation number is stored before loading
861: * folio_batch counters.
862: *
863: * This pairing must be done here, before the for_each_online_cpu loop
864: * below which drains the page vectors.
865: *
866: * Let x, y, and z represent some system CPU numbers, where x < y < z.
867: * Assume CPU #z is in the middle of the for_each_online_cpu loop
868: * below and has already reached CPU #y's per-cpu data. CPU #x comes
869: * along, adds some pages to its per-cpu vectors, then calls
870: * lru_add_drain_all().
871: *
872: * If the paired barrier is done at any later step, e.g. after the
873: * loop, CPU #x will just exit at (C) and miss flushing out all of its
874: * added pages.
875: */
876: WRITE_ONCE(lru_drain_gen, lru_drain_gen + 1);
877: smp_mb();
878:
879: cpumask_clear(&has_work);
880: for_each_online_cpu(cpu) {
881: struct work_struct *work = &per_cpu(lru_add_drain_work, cpu);
882:
883: if (cpu_needs_drain(cpu)) {
884: INIT_WORK(work, lru_add_drain_per_cpu);
885: queue_work_on(cpu, mm_percpu_wq, work);
886: __cpumask_set_cpu(cpu, &has_work);
887: }
888: }
889:
890: for_each_cpu(cpu, &has_work)
891: flush_work(&per_cpu(lru_add_drain_work, cpu));
892:
893: done:
894: mutex_unlock(&lock);
895: }
SourceFile:mm/swap.c]
|
| 14/2 |
2026/05/03 02:40 |
llm |
expert |
0m
Model:
Tokens:
input: 10498
output: 39
thoughts: 79
Thoughts: Here's my analysis of what's going on:
**Deep Dive into Memory Locking: A Code Trace**
Okay, so I'm looking at memory locking and tracing a specific call chain. It seems like the core function here is `folios_put_refs`, and I'm catching on that it's being called by `mlock_folio_batch`. Now, `mlock_folio_batch`, is itself called by `mlock_folio`. Interesting. This `mlock_folio` function also appears to be leveraging a per-CPU `mlock_fbatch` structure. That sounds like an optimization to me β batching operations per CPU to reduce contention, perhaps for atomic reference counting.
To really understand the flow and logic, I need to look at `mlock_folio` in `mm/mlock.c`. That's where the rubber meets the road. It will probably unveil the overall locking and reference management strategies in place and how it interacts with the folio structures. Iβll expect to see details about the `mlock_fbatch` usage in relation to folio management, reference counting, and potential spinlock usage.
|
| 15/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/mlock.c IncludeLines:true Name:mlock_folio]
Results: map[SourceCode: 238: /**
239: * mlock_folio - mlock a folio already on (or temporarily off) LRU
240: * @folio: folio to be mlocked.
241: */
242: void mlock_folio(struct folio *folio)
243: {
244: struct folio_batch *fbatch;
245:
246: local_lock(&mlock_fbatch.lock);
247: fbatch = this_cpu_ptr(&mlock_fbatch.fbatch);
248:
249: if (!folio_test_set_mlocked(folio)) {
250: int nr_pages = folio_nr_pages(folio);
251:
252: zone_stat_mod_folio(folio, NR_MLOCK, nr_pages);
253: __count_vm_events(UNEVICTABLE_PGMLOCKED, nr_pages);
254: }
255:
256: folio_get(folio);
257: if (!folio_batch_add(fbatch, mlock_lru(folio)) ||
258: !folio_may_be_lru_cached(folio) || lru_cache_disabled())
259: mlock_folio_batch(fbatch);
260: local_unlock(&mlock_fbatch.lock);
261: }
SourceFile:mm/mlock.c]
|
| 16/2 |
2026/05/03 02:40 |
llm |
expert |
0m
Model:
Tokens:
input: 11276
output: 39
thoughts: 69
Thoughts: Here's my analysis of what's happening, keeping in mind my expertise:
**Diving into the mlock_folio/lru_add_drain_all Interaction**
Okay, so I'm seeing a potential contention issue here. We've got `mlock_folio` holding a `local_lock` on `mlock_fbatch.lock`. That's standard for managing memory locking operations, nothing unusual there. The interesting bit is that `__lru_add_drain_all`, which I know is heavily involved in page reclaim and LRU management, is calling `cpu_needs_drain(cpu)`. This screams "potential for synchronization issues" to me. Specifically, I need to understand how the locking in `mlock_folio` might interact with the "drain" operation initiated by the LRU code.
My immediate next step is to drill down into the `cpu_needs_drain` function. I *know* that's the key. I need to understand what "draining" actually *means* in this context β is it about flushing writebacks, invalidating TLBs, or something else entirely? And crucially, how does the `cpu_needs_drain` decision impact the `mlock_fbatch.lock` or any shared data structures? Are we seeing lock contention, or perhaps a risk of deadlocks? This warrants a deeper look into `mm/swap.c`. Let's get to it.
|
| 17/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/swap.c IncludeLines:true Name:cpu_needs_drain]
Results: map[SourceCode: 781: static bool cpu_needs_drain(unsigned int cpu)
782: {
783: struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
784:
785: /* Check these in order of likelihood that they're not zero */
786: return folio_batch_count(&fbatches->lru_add) ||
787: folio_batch_count(&fbatches->lru_move_tail) ||
788: folio_batch_count(&fbatches->lru_deactivate_file) ||
789: folio_batch_count(&fbatches->lru_deactivate) ||
790: folio_batch_count(&fbatches->lru_lazyfree) ||
791: folio_batch_count(&fbatches->lru_activate) ||
792: need_mlock_drain(cpu) ||
793: has_bh_in_lru(cpu, NULL);
794: }
SourceFile:mm/swap.c]
|
| 18/2 |
2026/05/03 02:40 |
llm |
expert |
0m
Model:
Tokens:
input: 11987
output: 44
thoughts: 242
Thoughts: **Analyzing the Asynchronous LRU Drain Mechanism**
Okay, so I'm looking at how this LRU drain process works, and I see a very specific asynchronous design. The core of it seems to be centered around `cpu_needs_drain`. I notice that it calls `need_mlock_drain(cpu)` which is where we assess if this particular CPU requires a drain. This call then delves into `per_cpu(mlock_fbatch.fbatch, cpu).nr`. Hmm, so it's checking some sort of count related to a "fbatch" structure associated with memory locking on a per-CPU basis.
What's really interesting is that the code *doesn't* appear to hold the `mlock_fbatch.lock` for the remote CPU when calling `need_mlock_drain`. It's a lockless read, which is a pretty common strategy for performance reasons, but it opens the door to potential race conditions. Itβs making a quick check to see if a drain is *likely* needed, minimizing the impact of contention.
Now, if `cpu_needs_drain` reports a positive value, that's where the magic begins. A work item is queued on the target CPU using `queue_work_on`. Specifically, this code sets up a work item using `INIT_WORK` with the `lru_add_drain_per_cpu` function and then puts it on the `mm_percpu_wq` workqueue. A bitmask, `has_work`, is used to keep track of CPUs that have drain work scheduled.
So the asynchronous part comes from `lru_add_drain_per_cpu`. That's where the actual drain operation is *really* handled. It's safe to assume that `lru_add_drain_per_cpu` will take the necessary locks to ensure that the drain operation is done correctly, since this is a workqueue that runs on the CPU that the work item specifies. Therefore, let's turn our attention to `lru_add_drain_per_cpu` to see what is going on there.
|
| 19/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/swap.c IncludeLines:true Name:lru_add_drain_per_cpu]
Results: map[SourceCode: 776: static void lru_add_drain_per_cpu(struct work_struct *dummy)
777: {
778: lru_add_and_bh_lrus_drain();
779: }
SourceFile:mm/swap.c]
|
| 20/2 |
2026/05/03 02:40 |
llm |
expert |
0m
Model:
Tokens:
input: 12816
output: 47
thoughts: 17
Thoughts: Okay, here's my attempt at summarizing the thought process, as requested, aimed at an expert audience and in the first person:
**Navigating the `lru_add_and_bh_lrus_drain` Function**
Hmm, so we're looking at `lru_add_and_bh_lrus_drain`, are we? Right, this is a critical spot, as I recall. It's the point where we're not just adding to the LRU, but also potentially having to deal with batching and draining. I need to refresh my understanding of the exact synchronization here. What's the lock hierarchy look like around this call? We're likely in a critical section, and I need to make sure I understand the potential for deadlocks or performance bottlenecks. What are the contexts where this function is typically called? Is this from the page fault handler, a reclaim thread, or both? Understanding that will help me reason about priority inversion potential. I want to drill down into the batching logic. Is it simply a size-based batch, or is there more nuanced criteria to determine when to drain? And more crucially, how does the "drain" actually happen? Is it immediate, or is it scheduled, and if the latter, to which workqueue? What are the implications for NUMA locality, especially given the "bh" (bottom half) aspect that suggests an interrupt context could be involved? I need to clarify the relationship between this `lru_add_and_bh_lrus_drain` and other related functions. Is this function atomic with respect to memory order? What is its relationship to the other concurrent LRU access functions. This is where the devil's in the details; I need to make sure I'm not missing any subtleties.
|
| 21/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/swap.c IncludeLines:true Name:lru_add_and_bh_lrus_drain]
Results: map[SourceCode: 748: /*
749: * It's called from per-cpu workqueue context in SMP case so
750: * lru_add_drain_cpu and invalidate_bh_lrus_cpu should run on
751: * the same cpu. It shouldn't be a problem in !SMP case since
752: * the core is only one and the locks will disable preemption.
753: */
754: static void lru_add_and_bh_lrus_drain(void)
755: {
756: local_lock(&cpu_fbatches.lock);
757: lru_add_drain_cpu(smp_processor_id());
758: local_unlock(&cpu_fbatches.lock);
759: invalidate_bh_lrus_cpu();
760: mlock_drain_local();
761: }
SourceFile:mm/swap.c]
|
| 22/2 |
2026/05/03 02:40 |
llm |
expert |
0m
Model:
Tokens:
input: 13523
output: 41
thoughts: 10
Thoughts: Okay, here's my summary, presented as a stream of thought:
**Diving into `mlock_drain_local`**
Alright, let's see... *mlock_drain_local*. That's the one that always seems to be a bit... subtle. I need to refresh my understanding of how this particular function fits into the larger memory management picture. I remember it's part of a critical section related to memory locking, but I need to make sure I understand the precise role. Is it dealing with a specific type of resource drain? Is it dealing with a per-CPU or global lock? Is it dealing with removing all pages, or just specific pages? I need to think about that process: what is the point of *drain*? How does it relate to unlocking? What is the *local* aspect of the drain about? Is this a per-CPU operation? Is this a function I see when there are lock-ups or memory issues? I need to go over the documentation on this. Time to dig in and clarify its purpose and the specific context in which it's used.
|
| 23/2 |
2026/05/03 02:40 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/mlock.c IncludeLines:true Name:mlock_drain_local]
Results: map[SourceCode: 212: void mlock_drain_local(void)
213: {
214: struct folio_batch *fbatch;
215:
216: local_lock(&mlock_fbatch.lock);
217: fbatch = this_cpu_ptr(&mlock_fbatch.fbatch);
218: if (folio_batch_count(fbatch))
219: mlock_folio_batch(fbatch);
220: local_unlock(&mlock_fbatch.lock);
221: }
SourceFile:mm/mlock.c]
|
| 24/2 |
2026/05/03 02:40 |
llm |
expert |
1m
Model:
Tokens:
input: 13988
output: 22
thoughts: 1254
Thoughts: **Analyzing the Per-CPU Folio Batch Draining Mechanism**
Okay, so I'm digging into this `mlock_drain_local` function and how it interacts with the per-CPU folio batch mechanism. The core of the issue seems to be a race condition involving the `fbatch->nr` counter, which is managed by both `mlock_folio` (writer) and `__lru_add_drain_all` (reader).
On one CPU (let's say CPU 1), `mlock_folio` is called. This calls `mlock_folio_batch`, which in turn leads to `folios_put`, then `folios_put_refs`, and finally `folio_batch_reinit`. Critically, `folio_batch_reinit` sets `fbatch->nr` to zero. Simultaneously, on another CPU (CPU 0), `__lru_add_drain_all` is running. This function calls `cpu_needs_drain(1)` -> `need_mlock_drain(1)` which eventually reads `fbatch->nr` via `folio_batch_count`.
The `__lru_add_drain_all` function is essentially checking whether a drain operation needs to be scheduled. The worry is that it might read a stale value of `fbatch->nr` and make an incorrect decision about whether to schedule a drain.
If the reader sees `nr > 0` and it was *just* set to 0, it will schedule a no-op due to how `mlock_drain_local` functions. If it sees `nr == 0`, but it was *just* set to `> 0`, it might miss a drain this round.
Looking at the code, `__lru_add_drain_all` includes `smp_mb()` calls, suggesting an attempt to enforce memory ordering. The comments highlight that it wants to ensure the writer's updates to the folio batch counters are visible to other CPUs *before* the reader loads the drain generation.
Now, let's look at the writer, `mlock_folio`. It acquires `mlock_fbatch.lock`, increments `fbatch->nr` using `folio_batch_add`, *then* potentially calls `mlock_folio_batch` which sets it to zero later via the function call stack. The crucial part here is the sequence of operations within the lock. The writer modifies `nr`. The critical part, as far as I can see, is if the counter changes at an inopportune moment.
The question then is, how harmful is it if `__lru_add_drain_all` misses a drain? It seems to be used to make sure the pages are drained before some operation which requires them to be in a certain state (i.e. not in the per-cpu batch). It's called from `invalidate_bdev` during `ext4_put_super`, so missing a drain there would be...bad. If `__lru_add_drain_all` misses a CPU with pages in its `mlock_fbatch`, those pages won't be drained, potentially leading to incorrect behavior. The `smp_mb()` should help, but it's not a silver bullet, and KCSAN might still complain.
Regarding data races: I'm not worried about torn reads because `nr` is a single byte. A torn read is impossible. I also don't think compiler optimization is a problem here, as the read is within a `for_each_online_cpu(cpu)` loop.
The comments in `__lru_add_drain_all` refer to `smp_load_acquire()` (B) and a "full memory barrier" to ensure visibility. I want to be certain that the latest `folio_batch` counters are visible.
If `mlock_folio` just added a folio, `nr` is 1, so the drain would be scheduled. However, if `__lru_add_drain_all` sees `nr == 0`, it won't drain and could miss pages.
The `smp_mb()` at 877 does seem to ensure the `lru_drain_gen` increment is visible before reading `nr`. However, there's no corresponding barrier *after* updating `nr` in the writer. This is my main concern. `local_unlock` may have some implicit ordering guarantees. It might act as a compiler barrier or involve `preempt_enable`. Given it's in a per-cpu context, it looks like it just disabled/enables preemption. Let me just double check how `local_lock` is defined... ah, found it in `linux/local_lock.h`.
|
| 25/2 |
2026/05/03 02:42 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:[8250_pci.h acct.h acpi.h acpi_amd_wbrf.h acpi_dma.h acpi_iort.h acpi_mdio.h acpi_pmtmr.h acpi_rimt.h acpi_viot.h adb.h adfs_fs.h adi-axi-common.h adreno-smmu-priv.h adxl.h aer.h agp_backend.h agpgart.h ahci-remap.h ahci_platform.h aio.h alarmtimer.h alcor_pci.h align.h alloc_tag.h altera_jtaguart.h altera_uart.h amd-iommu.h amd-pmf-io.h annotate.h anon_inodes.h aperture.h apm-emulation.h apm_bios.h apple-gmux.h arch_topology.h args.h arm-cci.h arm-smccc.h arm_ffa.h arm_mpam.h arm_sdei.h armada-37xx-rwtm-mailbox.h array_size.h ascii85.h asn1.h asn1_ber_bytecode.h asn1_decoder.h asn1_encoder.h assoc_array.h assoc_array_priv.h async.h async_tx.h ata.h ata_platform.h atalk.h atm.h atm_tcp.h atmdev.h atmel-isc-media.h atmel-ssc.h atmel_pdc.h atomic.h attribute_container.h audit.h audit_arch.h auto_dev-ioctl.h auto_fs.h auxiliary_bus.h auxvec.h average.h backing-dev-defs.h backing-dev.h backing-file.h backlight.h badblocks.h balloon.h base64.h bcd.h bch.h bcm47xx_nvram.h bcm47xx_sprom.h bcm47xx_wdt.h bcm963xx_nvram.h bcm963xx_tag.h binfmts.h bio-integrity.h bio.h bit_spinlock.h bitfield.h bitmap-str.h bitmap.h bitops.h bitrev.h bits.h blk-cgroup.h blk-crypto-profile.h blk-crypto.h blk-integrity.h blk-mq-dma.h blk-mq.h blk-pm.h blk_types.h blkdev.h blkpg.h blktrace_api.h blockgroup_lock.h bma150.h bootconfig.h bootmem_info.h bottom_half.h bpf-cgroup-defs.h bpf-cgroup.h bpf-netns.h bpf.h bpf_crypto.h bpf_lirc.h bpf_local_storage.h bpf_lsm.h bpf_mem_alloc.h bpf_mprog.h bpf_trace.h bpf_types.h bpf_verifier.h bpfptr.h brcmphy.h bsearch.h bsg-lib.h bsg.h btf.h btf_ids.h btree-128.h btree-type.h btree.h btrfs.h buffer_head.h bug.h build-salt.h build_bug.h buildid.h bvec.h c2port.h cache.h cache_coherency.h cacheflush.h cacheinfo.h call_once.h capability.h cb710.h cc_platform.h cciss_ioctl.h ccp.h cdev.h cdrom.h cfag12864b.h cfi.h cfi_types.h cgroup-defs.h cgroup.h cgroup_api.h cgroup_dmem.h cgroup_namespace.h cgroup_rdma.h cgroup_refcnt.h cgroup_subsys.h circ_buf.h cleanup.h clk-provider.h clk.h clkdev.h clockchips.h clocksource.h clocksource_ids.h closure.h cma.h cmpxchg-emu.h cn_proc.h cnt32_to_63.h coda.h codetag.h compaction.h compat.h compiler-clang.h compiler-context-analysis.h compiler-gcc.h compiler-version.h compiler.h compiler_attributes.h compiler_types.h completion.h component.h configfs.h connector.h console.h console_struct.h consolemap.h const.h container.h container_of.h context_tracking.h context_tracking_irq.h context_tracking_state.h cookie.h cordic.h coreboot.h coredump.h coresight-pmu.h coresight-stm.h coresight.h count_zeros.h counter.h cper.h cpu.h cpu_cooling.h cpu_pm.h cpu_rmap.h cpu_smt.h cpufeature.h cpufreq.h cpuhotplug.h cpuhplock.h cpuidle.h cpuidle_haltpoll.h cpumask.h cpumask_api.h cpumask_types.h cpuset.h crash_core.h crash_dump.h crash_reserve.h crc-ccitt.h crc-itu-t.h crc-t10dif.h crc16.h crc32.h crc32c.h crc32poly.h crc4.h crc64.h crc7.h crc8.h cred.h crypto.h cs5535.h ctype.h cuda.h damon.h dasd_mod.h davinci_emac.h dax.h dca.h dcache.h dccp.h debug_locks.h debugfs.h debugobjects.h delay.h delayacct.h delayed_call.h dev_printk.h devcoredump.h devfreq-event.h devfreq-governor.h devfreq.h devfreq_cooling.h device-mapper.h device.h device_cgroup.h devm-helpers.h devpts_fs.h dfl.h dibs.h digsig.h dim.h dio.h dirent.h dlm.h dlm_plock.h dm-bufio.h dm-dirty-log.h dm-io.h dm-kcopyd.h dm-region-hash.h dm-verity-loadpin.h dm9000.h dma-buf-mapping.h dma-buf.h dma-direct.h dma-direction.h dma-fence-array.h dma-fence-chain.h dma-fence-unwrap.h dma-fence.h dma-heap.h dma-map-ops.h dma-mapping.h dma-resv.h dmaengine.h dmapool.h dmar.h dmi.h dnotify.h dns_resolver.h dpll.h dqblk_qtree.h dqblk_v1.h dqblk_v2.h drbd.h drbd_config.h drbd_genl.h drbd_genl_api.h drbd_limits.h ds2782_battery.h dtlk.h dtpm.h dw_apb_timer.h dynamic_debug.h dynamic_queue_limits.h earlycpio.h ecryptfs.h edac.h edd.h eeprom_93cx6.h efi-bgrt.h efi.h efi_embedded_fw.h efs_vh.h ehl_pse_io_aux.h eisa.h elf-fdpic.h elf-randomize.h elf.h elfcore-compat.h elfcore.h elfnote-lto.h elfnote.h enclosure.h energy_model.h entry-common.h entry-virt.h err.h errname.h errno.h error-injection.h errqueue.h errseq.h etherdevice.h ethtool.h ethtool_netlink.h eventfd.h eventpoll.h evm.h execmem.h export-internal.h export.h exportfs.h ext2_fs.h extable.h extcon-provider.h extcon.h f2fs_fs.h f75375s.h falloc.h fanotify.h fault-inject-usercopy.h fault-inject.h fb.h fbcon.h fcdevice.h fcntl.h fd.h fddidevice.h fdtable.h fec.h fiemap.h file.h file_ref.h fileattr.h filelock.h filter.h find.h fips.h firewire.h firmware-map.h firmware.h fixp-arith.h flat.h flex_proportions.h folio_batch.h folio_queue.h font.h fortify-string.h fprobe.h fpu.h freezer.h fs.h fs_api.h fs_context.h fs_dirent.h fs_parser.h fs_pin.h fs_stack.h fs_struct.h fscache-cache.h fscache.h fscrypt.h fserror.h fsi-occ.h fsi-sbefifo.h fsi.h fsl-diu-fb.h fsl_devices.h fsl_hypervisor.h fsl_ifc.h fsldma.h fsnotify.h fsnotify_backend.h fsverity.h ftrace.h ftrace_irq.h ftrace_regs.h futex.h fw_table.h fwctl.h fwnode.h fwnode_mdio.h gameport.h gcd.h genalloc.h generic-radix-tree.h genl_magic_func.h genl_magic_struct.h gfp.h gfp_api.h gfp_types.h glob.h gnss.h goldfish.h gpio-pxa.h gpio.h gpio_keys.h gpu_buddy.h greybus.h group_cpus.h hardirq.h hash.h hashtable.h hashtable_api.h hdlc.h hdmi.h hex.h hfs_common.h hid-debug.h hid-over-i2c.h hid-over-spi.h hid-roccat.h hid-sensor-hub.h hid-sensor-ids.h hid.h hid_bpf.h hidden.h hiddev.h hidraw.h highmem-internal.h highmem.h highuid.h hil.h hil_mlc.h hisi_acc_qm.h hmm-dma.h hmm.h host1x.h host1x_context_bus.h hp_sdc.h hpet.h hrtimer.h hrtimer_api.h hrtimer_defs.h hrtimer_rearm.h hrtimer_types.h hte.h huge_mm.h hugetlb.h hugetlb_cgroup.h hugetlb_inline.h hung_task.h hw_bitfield.h hw_breakpoint.h hw_random.h hwmon-sysfs.h hwmon-vid.h hwmon.h hwspinlock.h hyperv.h hypervisor.h i2c-algo-bit.h i2c-algo-pca.h i2c-algo-pcf.h i2c-atr.h i2c-dev.h i2c-mux.h i2c-of-prober.h i2c-smbus.h i2c.h i8042.h i8253.h i8254.h icmp.h icmpv6.h idle_inject.h idr.h ieee80211-eht.h ieee80211-he.h ieee80211-ht.h ieee80211-mesh.h ieee80211-nan.h ieee80211-p2p.h ieee80211-s1g.h ieee80211-uhr.h ieee80211-vht.h ieee80211.h ieee802154.h if_arp.h if_bridge.h if_eql.h if_ether.h if_fddi.h if_hsr.h if_link.h if_macvlan.h if_phonet.h if_pppol2tp.h if_pppox.h if_rmnet.h if_tap.h if_team.h if_tun.h if_tunnel.h if_vlan.h igmp.h ihex.h ima.h imx-media.h in.h in6.h indirect_call_wrapper.h inet.h inet_diag.h inetdevice.h init.h init_ohci1394_dma.h init_syscalls.h init_task.h initrd.h inotify.h input.h instruction_pointer.h instrumentation.h instrumented.h int_log.h integrity.h intel-ish-client-if.h intel_dg_nvm_aux.h intel_pmt_features.h intel_rapl.h intel_tcc.h intel_th.h intel_tpmi.h intel_vsec.h interconnect-clk.h interconnect-provider.h interconnect.h interrupt.h interval_tree.h interval_tree_generic.h io-64-nonatomic-hi-lo.h io-64-nonatomic-lo-hi.h io-mapping.h io-pgtable.h io.h io_uring.h io_uring_types.h ioam6.h ioam6_genl.h ioam6_iptunnel.h iocontext.h iomap.h iommu-debug-pagealloc.h iommu-dma.h iommu-helper.h iommu.h iommufd.h iopoll.h ioport.h ioprio.h ioremap.h iosys-map.h iov_iter.h iova.h iova_bitmap.h ip.h ipack.h ipc.h ipc_namespace.h ipmi.h ipmi_smi.h ipv6.h ipv6_route.h irq-entry-common.h irq.h irq_poll.h irq_sim.h irq_work.h irq_work_types.h irqbypass.h irqchip.h irqdesc.h irqdomain.h irqdomain_defs.h irqflags.h irqflags_types.h irqhandler.h irqnr.h irqreturn.h isa-dma.h isa.h isapnp.h iscsi_boot_sysfs.h iscsi_ibft.h ism.h iversion.h jbd2.h jhash.h jiffies.h journal-head.h joystick.h jump_label.h jump_label_ratelimit.h jz4740-adc.h jz4780-nemc.h kallsyms.h kasan-checks.h kasan-enabled.h kasan-tags.h kasan.h kbd_diacr.h kbd_kern.h kbuild.h kconfig.h kcore.h kcov.h kcsan-checks.h kcsan.h kdb.h kdebug.h kdev_t.h kern_levels.h kernel-page-flags.h kernel.h kernel_read_file.h kernel_stat.h kernfs.h kexec.h kexec_handover.h key-type.h key.h keyboard.h keyctl.h kfence.h kfifo.h kgdb.h kho_radix_tree.h khugepaged.h klist.h kmemleak.h kmod.h kmsan-checks.h kmsan.h kmsan_string.h kmsan_types.h kmsg_dump.h kobj_map.h kobject.h kobject_api.h kobject_ns.h kprobes.h kref.h kref_api.h ks0108.h ks8842.h ks8851_mll.h ksm.h kstack_erase.h kstrtox.h ksysfs.h kthread.h ktime.h ktime_api.h kvm_dirty_ring.h kvm_host.h kvm_irqfd.h kvm_para.h kvm_types.h l2tp.h lantiq.h lapb.h latencytop.h lcd.h lcm.h leafops.h led-class-flash.h led-class-multicolor.h led-lm3530.h leds-bd2802.h leds-expresswire.h leds-lp3944.h leds-lp3952.h leds-pca9532.h leds-regulator.h leds-ti-lmu-common.h leds.h libata.h libfdt.h libfdt_env.h libgcc.h libnvdimm.h libps2.h license.h limits.h linear_range.h linkage.h linkmode.h linux_logo.h lis3lv02d.h list.h list_bl.h list_lru.h list_nulls.h list_private.h list_sort.h litex.h livepatch.h livepatch_external.h livepatch_helpers.h livepatch_sched.h liveupdate.h llc.h llist.h llist_api.h local_lock.h local_lock_internal.h lockdep.h lockdep_api.h lockdep_types.h lockref.h log2.h logic_iomem.h logic_pio.h lp.h lru_cache.h lsm_audit.h lsm_count.h lsm_hook_defs.h lsm_hooks.h lwq.h lz4.h lzo.h mailbox_client.h mailbox_controller.h maple.h maple_tree.h marvell_phy.h math.h math64.h mbcache.h mbus.h mc146818rtc.h mc33xs2410.h mc6821.h mcb.h mdev.h mdio-bitbang.h mdio-mux.h mdio.h mei_aux.h mei_cl_bus.h mem_encrypt.h memblock.h memcontrol.h memfd.h memory-failure.h memory-tiers.h memory.h memory_hotplug.h mempolicy.h mempool.h memregion.h memremap.h memstick.h mhi.h mhi_ep.h micrel_phy.h microchipphy.h migrate.h migrate_mode.h mii.h mii_timestamper.h min_heap.h minmax.h misc_cgroup.h miscdevice.h mm.h mm_api.h mm_inline.h mm_types.h mm_types_task.h mman.h mmap_lock.h mmdebug.h mmiotrace.h mmu_context.h mmu_notifier.h mmzone.h mnt_idmapping.h mnt_namespace.h mod_devicetable.h module.h module_signature.h module_symbol.h moduleloader.h moduleparam.h most.h mount.h moxtet.h mpage.h mpi.h mpls.h mpls_iptunnel.h mroute.h mroute6.h mroute_base.h msdos_fs.h msdos_partition.h msg.h msi.h msi_api.h mtio.h mutex.h mutex_api.h mutex_types.h mv643xx_eth.h mv643xx_i2c.h mvebu-pmsu.h mxm-wmi.h namei.h nd.h ndctl.h net.h net_tstamp.h netdev_features.h netdevice.h netdevice_xmit.h netfilter.h netfilter_bridge.h netfilter_defs.h netfilter_ipv4.h netfilter_ipv6.h netfilter_netdev.h netfs.h netlink.h netpoll.h nfs.h nfs3.h nfs4.h nfs_common.h nfs_fs.h nfs_fs_i.h nfs_fs_sb.h nfs_iostat.h nfs_page.h nfs_ssc.h nfs_xdr.h nfsacl.h nfslocalio.h nitro_enclaves.h nl802154.h nls.h nmi.h node.h nodemask.h nodemask_types.h nospec.h notifier.h ns_common.h nsc_gpio.h nsfs.h nsproxy.h nstree.h ntb.h ntb_transport.h nubus.h numa.h numa_memblks.h nvme-auth.h nvme-fc-driver.h nvme-fc.h nvme-keyring.h nvme-rdma.h nvme-tcp.h nvme.h nvmem-consumer.h nvmem-provider.h nvram.h oa_tc6.h objagg.h objpool.h objtool.h objtool_types.h of.h of_address.h of_clk.h of_device.h of_dma.h of_fdt.h of_graph.h of_iommu.h of_irq.h of_mdio.h of_net.h of_pci.h of_pdt.h of_platform.h of_reserved_mem.h oid_registry.h olpc-ec.h omap-dma.h omap-gpmc.h omap-iommu.h omap-mailbox.h omapfb.h once.h once_lite.h oom.h openvswitch.h osq_lock.h overflow.h packing.h padata.h page-flags-layout.h page-flags.h page-isolation.h page_counter.h page_ext.h page_frag_cache.h page_idle.h page_owner.h page_ref.h page_reporting.h page_table_check.h pageblock-flags.h pagemap.h pagewalk.h panic.h panic_notifier.h papr_scm.h parman.h parport.h parport_pc.h parser.h part_stat.h pata_arasan_cf_data.h patchkey.h path.h pch_dma.h pci-acpi.h pci-ats.h pci-bwctrl.h pci-doe.h pci-ecam.h pci-ep-cfs.h pci-ep-msi.h pci-epc.h pci-epf.h pci-ide.h pci-p2pdma.h pci-pwrctrl.h pci-tph.h pci-tsm.h pci.h pci_hotplug.h pci_ids.h pcie-dwc.h pcs-lynx.h pcs-rzn1-miic.h pe.h peci-cpu.h peci.h percpu-defs.h percpu-refcount.h percpu-rwsem.h percpu.h percpu_counter.h perf_event.h perf_event_api.h perf_regs.h personality.h pfn.h pgalloc.h pgalloc_tag.h pgtable.h pgtable_api.h phonet.h phy.h phy_fixed.h phy_led_triggers.h phy_link_topology.h phy_port.h phylib_stubs.h phylink.h pid.h pid_namespace.h pid_types.h pidfs.h pim.h pipe_fs_i.h pkeys.h pl320-ipc.h platform_device.h platform_profile.h pldmfw.h plist.h plist_types.h pm-trace.h pm.h pm_clock.h pm_domain.h pm_opp.h pm_qos.h pm_runtime.h pm_wakeirq.h pm_wakeup.h pmbus.h pmu.h pnp.h poison.h poll.h polynomial.h posix-clock.h posix-timers.h posix-timers_types.h posix_acl.h posix_acl_xattr.h power_supply.h powercap.h ppp-comp.h ppp_channel.h ppp_defs.h pps_gen_kernel.h pps_kernel.h pr.h prandom.h preempt.h prefetch.h prime_numbers.h printk.h prmt.h proc_fs.h proc_ns.h processor.h profile.h projid.h property.h pruss_driver.h psci.h pseudo_fs.h psi.h psi_types.h psp-platform-access.h psp-sev.h psp-tee.h psp.h pstore.h pstore_blk.h pstore_ram.h pstore_zone.h ptdump.h pti.h ptp_classify.h ptp_clock_kernel.h ptp_kvm.h ptp_mock.h ptp_pch.h ptr_ring.h ptrace.h ptrace_api.h purgatory.h pvclock_gtod.h pwm.h pwm_backlight.h pxa168_eth.h pxa2xx_ssp.h qnx6_fs.h quota.h quotaops.h radix-tree.h raid_class.h ramfs.h random.h randomize_kstack.h range.h ras.h ratelimit.h ratelimit_types.h rational.h rbtree.h rbtree_augmented.h rbtree_latch.h rbtree_types.h rcu_node_tree.h rcu_notifier.h rcu_segcblist.h rcu_sync.h rculist.h rculist_bl.h rculist_nulls.h rcupdate.h rcupdate_trace.h rcupdate_wait.h rcuref.h rcutiny.h rcutree.h rcuwait.h rcuwait_api.h reboot-mode.h reboot.h reciprocal_div.h ref_tracker.h refcount.h refcount_api.h refcount_types.h regmap.h regset.h relay.h remoteproc.h resctrl.h resctrl_types.h reset-controller.h reset.h resource.h resource_ext.h restart_block.h resume_user_mode.h rethook.h rfkill.h rhashtable-types.h rhashtable.h ring_buffer.h ring_buffer_types.h rio.h rio_drv.h rio_ids.h rio_regs.h rmap.h rmi.h rndis.h rodata_test.h rolling_buffer.h root_dev.h rpmb.h rpmsg.h rseq.h rseq_entry.h rseq_types.h rslib.h rtc.h rtmutex.h rtnetlink.h rtsx_common.h rtsx_pci.h rtsx_usb.h rv.h rw_hint.h rwbase_rt.h rwlock.h rwlock_api_smp.h rwlock_rt.h rwlock_types.h rwsem.h sbitmap.h scatterlist.h sched.h sched_clock.h scmi_imx_protocol.h scmi_protocol.h scpi_protocol.h screen_info.h scs.h sctp.h scx200.h scx200_gpio.h seccomp.h seccomp_types.h secretmem.h secure_boot.h securebits.h security.h sed-opal-key.h sed-opal.h seg6.h seg6_genl.h seg6_hmac.h seg6_iptunnel.h seg6_local.h selection.h sem.h sem_types.h semaphore.h seq_buf.h seq_file.h seq_file_net.h seqlock.h seqlock_api.h seqlock_types.h serdev.h serial.h serial_8250.h serial_bcm63xx.h serial_core.h serial_s3c.h serial_sci.h serio.h set_memory.h sfp.h sh_clk.h sh_dma.h sh_eth.h sh_intc.h sh_timer.h shdma-base.h shm.h shmem_fs.h shrinker.h signal.h signal_types.h signalfd.h simple_ring_buffer.h siox.h siphash.h sizes.h skb_array.h skbuff.h skbuff_ref.h skmsg.h slab.h slimbus.h sm501-regs.h sm501.h smc91x.h smp.h smp_types.h smpboot.h smsc911x.h smscphy.h sock_diag.h socket.h sockptr.h softirq.h sonet.h sonypi.h sort.h sound.h soundcard.h spinlock.h spinlock_api.h spinlock_api_smp.h spinlock_api_up.h spinlock_rt.h spinlock_types.h spinlock_types_raw.h spinlock_types_up.h spinlock_up.h splice.h spmi.h sprintf.h sram.h srcu.h srcutiny.h srcutree.h ssbi.h stackdepot.h stackprotector.h stacktrace.h start_kernel.h stat.h statfs.h static_call.h static_call_types.h static_key.h stdarg.h stddef.h stm.h stmmac.h stmp3xxx_rtc_wdt.h stmp_device.h stop_machine.h string.h string_choices.h string_helpers.h stringhash.h stringify.h sungem_phy.h sunserialcore.h sunxi-rsb.h surface_acpi_notify.h suspend.h svga.h sw842.h swab.h swait.h swait_api.h swap.h swap_cgroup.h swapfile.h swapops.h swiotlb.h switchtec.h sxgbe_platform.h sync_core.h sync_file.h synclink.h sys.h sys_info.h sys_soc.h syscall_user_dispatch.h syscall_user_dispatch_types.h syscalls.h syscalls_api.h syscore_ops.h sysctl.h sysfb.h sysfs.h syslog.h sysrq.h t10-pi.h task_io_accounting.h task_io_accounting_ops.h task_work.h taskstats_kern.h tboot.h tc.h tcp.h tee_core.h tee_drv.h tegra-icc.h tegra-mipi-cal.h termios_internal.h text-patching.h textsearch.h textsearch_fsm.h thermal.h thread_info.h threads.h thunderbolt.h ti-emif-sram.h ti_wilink_st.h tick.h tifm.h timb_dma.h timb_gpio.h time.h time32.h time64.h time_namespace.h timecounter.h timekeeper_internal.h timekeeping.h timer.h timer_types.h timerfd.h timeriomem-rng.h timerqueue.h timerqueue_types.h timex.h tnum.h topology.h torture.h toshiba.h tpm.h tpm_command.h tpm_eventlog.h tpm_svsm.h trace.h trace_clock.h trace_events.h trace_printk.h trace_recursion.h trace_remote.h trace_remote_event.h trace_seq.h tracefs.h tracepoint-defs.h tracepoint.h transport_class.h ts-nbus.h tsacct_kern.h tsm-mr.h tsm.h tty.h tty_buffer.h tty_driver.h tty_flip.h tty_ldisc.h tty_port.h turris-omnia-mcu-interface.h turris-signing-key.h typecheck.h types.h u64_stats_sync.h u64_stats_sync_api.h uacce.h uaccess.h ubsan.h ucopysize.h ucs2_string.h udp.h uidgid.h uidgid_types.h uio.h uio_driver.h umh.h unaligned.h unicode.h union_find.h units.h unroll.h unwind_deferred.h unwind_deferred_types.h unwind_user.h unwind_user_types.h uprobes.h usb.h usb_usual.h usbdevice_fs.h user-return-notifier.h user.h user_events.h user_namespace.h userfaultfd_k.h util_macros.h uts.h uts_namespace.h utsname.h uuid.h vbox_utils.h vdpa.h vdso_datastore.h verification.h vermagic.h vexpress.h vfio.h vfio_pci_core.h vfs.h vfsdebug.h vga_switcheroo.h vgaarb.h vhost_iotlb.h via-core.h via.h via_i2c.h videodev2.h virtio.h virtio_anchor.h virtio_byteorder.h virtio_config.h virtio_dma_buf.h virtio_features.h virtio_net.h virtio_pci_admin.h virtio_pci_legacy.h virtio_pci_modern.h virtio_ring.h virtio_vsock.h vm_event_item.h vmalloc.h vmcore_info.h vmpressure.h vmstat.h vmw_vmci_api.h vmw_vmci_defs.h vringh.h vt.h vt_buffer.h vt_kern.h vtime.h w1.h wait.h wait_api.h wait_bit.h watch_queue.h watchdog.h win_minmax.h wireless.h wkup_m3_ipc.h wm97xx.h wmi.h wordpart.h workqueue.h workqueue_api.h workqueue_types.h writeback.h ww_mutex.h wwan.h xarray.h xattr.h xxhash.h xz.h zconf.h zlib.h zorro.h zsmalloc.h zstd.h zstd_errors.h zstd_lib.h zswap.h zutil.h] Subdirs:[amba atomic avf bcma bnge bnxt bus byteorder can cdx ceph clk comedi crush decompress device dma dsa extcon firmware fpga framer fs fsl generic_pt gpio greybus habanalabs hsi i3c iio input io_uring irqchip kho lockd lsm mailbox mdio memory mfd misc mlx4 mlx5 mmc mtd mux net netfilter netfilter_arp netfilter_bridge netfilter_ipv4 netfilter_ipv6 ns pcs pds perf phy pinctrl platform_data power pse-pd pwrseq qat qed raid raspberrypi regulator remoteproc reset rpmsg rtc sched soc soundwire spi ssb sunrpc surface_aggregator ulpi unaligned usb]]
|
| 26/2 |
2026/05/03 02:42 |
llm |
expert |
0m
Model:
Tokens:
input: 29422
output: 50
thoughts: 29
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Diving into the Local Locking Mechanism**
Right, let's start by digging into the heart of the local locking implementation. My first move will be to examine the foundational header files. I need to refresh my understanding of the core data structures and mechanisms involved. I'm going to focus on `include/linux/local_lock.h` and `include/linux/local_lock_internal.h`. I'm expecting to find the definitions for the lock primitives themselves β like `local_lock`, `local_unlock`, `local_irq_save`, and the like β in `local_lock.h`. I also anticipate `local_lock_internal.h` to contain the nitty-gritty implementation details and any architecture-specific optimizations, possibly including the use of atomic operations or specific memory barrier instructions if there are any that I need to recall. I'm especially keen to see how the kernel handles contention and any potential for deadlock within this local context. It is important to know if it uses some form of spinlock. I'll need to know whether any architectural considerations or assumptions are baked in, too, to see if they might affect performance.
|
| 27/2 |
2026/05/03 02:42 |
tool |
read-file |
0m
Args: map[File:include/linux/local_lock.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_LOCAL_LOCK_H
#define _LINUX_LOCAL_LOCK_H
#include <linux/local_lock_internal.h>
/**
* local_lock_init - Runtime initialize a lock instance
* @lock: The lock variable
*/
#define local_lock_init(lock) __local_lock_init(lock)
/**
* local_lock - Acquire a per CPU local lock
* @lock: The lock variable
*/
#define local_lock(lock) __local_lock(__this_cpu_local_lock(lock))
/**
* local_lock_irq - Acquire a per CPU local lock and disable interrupts
* @lock: The lock variable
*/
#define local_lock_irq(lock) __local_lock_irq(__this_cpu_local_lock(lock))
/**
* local_lock_irqsave - Acquire a per CPU local lock, save and disable
* interrupts
* @lock: The lock variable
* @flags: Storage for interrupt flags
*/
#define local_lock_irqsave(lock, flags) \
__local_lock_irqsave(__this_cpu_local_lock(lock), flags)
/**
* local_unlock - Release a per CPU local lock
* @lock: The lock variable
*/
#define local_unlock(lock) __local_unlock(__this_cpu_local_lock(lock))
/**
* local_unlock_irq - Release a per CPU local lock and enable interrupts
* @lock: The lock variable
*/
#define local_unlock_irq(lock) __local_unlock_irq(__this_cpu_local_lock(lock))
/**
* local_unlock_irqrestore - Release a per CPU local lock and restore
* interrupt flags
* @lock: The lock variable
* @flags: Interrupt flags to restore
*/
#define local_unlock_irqrestore(lock, flags) \
__local_unlock_irqrestore(__this_cpu_local_lock(lock), flags)
/**
* local_trylock_init - Runtime initialize a lock instance
* @lock: The lock variable
*/
#define local_trylock_init(lock) __local_trylock_init(lock)
/**
* local_trylock - Try to acquire a per CPU local lock
* @lock: The lock variable
*
* The function can be used in any context such as NMI or HARDIRQ. Due to
* locking constrains it will _always_ fail to acquire the lock in NMI or
* HARDIRQ context on PREEMPT_RT.
*/
#define local_trylock(lock) __local_trylock(__this_cpu_local_lock(lock))
#define local_lock_is_locked(lock) __local_lock_is_locked(lock)
/**
* local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable
* interrupts if acquired
* @lock: The lock variable
* @flags: Storage for interrupt flags
*
* The function can be used in any context such as NMI or HARDIRQ. Due to
* locking constrains it will _always_ fail to acquire the lock in NMI or
* HARDIRQ context on PREEMPT_RT.
*/
#define local_trylock_irqsave(lock, flags) \
__local_trylock_irqsave(__this_cpu_local_lock(lock), flags)
DEFINE_LOCK_GUARD_1(local_lock, local_lock_t __percpu,
local_lock(_T->lock),
local_unlock(_T->lock))
DEFINE_LOCK_GUARD_1(local_lock_irq, local_lock_t __percpu,
local_lock_irq(_T->lock),
local_unlock_irq(_T->lock))
DEFINE_LOCK_GUARD_1(local_lock_irqsave, local_lock_t __percpu,
local_lock_irqsave(_T->lock, _T->flags),
local_unlock_irqrestore(_T->lock, _T->flags),
unsigned long flags)
#define local_lock_nested_bh(_lock) \
__local_lock_nested_bh(__this_cpu_local_lock(_lock))
#define local_unlock_nested_bh(_lock) \
__local_unlock_nested_bh(__this_cpu_local_lock(_lock))
DEFINE_LOCK_GUARD_1(local_lock_nested_bh, local_lock_t __percpu,
local_lock_nested_bh(_T->lock),
local_unlock_nested_bh(_T->lock))
DEFINE_LOCK_GUARD_1(local_lock_init, local_lock_t, local_lock_init(_T->lock), /* */)
DECLARE_LOCK_GUARD_1_ATTRS(local_lock, __acquires(_T), __releases(*(local_lock_t __percpu **)_T))
#define class_local_lock_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(local_lock, _T)
DECLARE_LOCK_GUARD_1_ATTRS(local_lock_irq, __acquires(_T), __releases(*(local_lock_t __percpu **)_T))
#define class_local_lock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(local_lock_irq, _T)
DECLARE_LOCK_GUARD_1_ATTRS(local_lock_irqsave, __acquires(_T), __releases(*(local_lock_t __percpu **)_T))
#define class_local_lock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(local_lock_irqsave, _T)
DECLARE_LOCK_GUARD_1_ATTRS(local_lock_nested_bh, __acquires(_T), __releases(*(local_lock_t __percpu **)_T))
#define class_local_lock_nested_bh_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(local_lock_nested_bh, _T)
DECLARE_LOCK_GUARD_1_ATTRS(local_lock_init, __acquires(_T), __releases(*(local_lock_t **)_T))
#define class_local_lock_init_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(local_lock_init, _T)
DEFINE_LOCK_GUARD_1(local_trylock_init, local_trylock_t, local_trylock_init(_T->lock), /* */)
DECLARE_LOCK_GUARD_1_ATTRS(local_trylock_init, __acquires(_T), __releases(*(local_trylock_t **)_T))
#define class_local_trylock_init_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(local_trylock_init, _T)
#endif
]
|
| 28/2 |
2026/05/03 02:42 |
tool |
read-file |
0m
Args: map[File:include/linux/local_lock_internal.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_LOCAL_LOCK_H
# error "Do not include directly, include linux/local_lock.h"
#endif
#include <linux/percpu-defs.h>
#include <linux/irqflags.h>
#include <linux/lockdep.h>
#include <linux/debug_locks.h>
#include <asm/current.h>
#ifndef CONFIG_PREEMPT_RT
context_lock_struct(local_lock) {
#ifdef CONFIG_DEBUG_LOCK_ALLOC
struct lockdep_map dep_map;
struct task_struct *owner;
#endif
};
typedef struct local_lock local_lock_t;
/* local_trylock() and local_trylock_irqsave() only work with local_trylock_t */
context_lock_struct(local_trylock) {
#ifdef CONFIG_DEBUG_LOCK_ALLOC
struct lockdep_map dep_map;
struct task_struct *owner;
#endif
u8 acquired;
};
typedef struct local_trylock local_trylock_t;
#ifdef CONFIG_DEBUG_LOCK_ALLOC
# define LOCAL_LOCK_DEBUG_INIT(lockname) \
.dep_map = { \
.name = #lockname, \
.wait_type_inner = LD_WAIT_CONFIG, \
.lock_type = LD_LOCK_PERCPU, \
}, \
.owner = NULL,
# define LOCAL_TRYLOCK_DEBUG_INIT(lockname) \
LOCAL_LOCK_DEBUG_INIT(lockname)
static inline void local_lock_acquire(local_lock_t *l)
{
lock_map_acquire(&l->dep_map);
DEBUG_LOCKS_WARN_ON(l->owner);
l->owner = current;
}
static inline void local_trylock_acquire(local_lock_t *l)
{
lock_map_acquire_try(&l->dep_map);
DEBUG_LOCKS_WARN_ON(l->owner);
l->owner = current;
}
static inline void local_lock_release(local_lock_t *l)
{
DEBUG_LOCKS_WARN_ON(l->owner != current);
l->owner = NULL;
lock_map_release(&l->dep_map);
}
static inline void local_lock_debug_init(local_lock_t *l)
{
l->owner = NULL;
}
#else /* CONFIG_DEBUG_LOCK_ALLOC */
# define LOCAL_LOCK_DEBUG_INIT(lockname)
# define LOCAL_TRYLOCK_DEBUG_INIT(lockname)
static inline void local_lock_acquire(local_lock_t *l) { }
static inline void local_trylock_acquire(local_lock_t *l) { }
static inline void local_lock_release(local_lock_t *l) { }
static inline void local_lock_debug_init(local_lock_t *l) { }
#endif /* !CONFIG_DEBUG_LOCK_ALLOC */
#define INIT_LOCAL_LOCK(lockname) { LOCAL_LOCK_DEBUG_INIT(lockname) }
#define INIT_LOCAL_TRYLOCK(lockname) { LOCAL_TRYLOCK_DEBUG_INIT(lockname) }
#define __local_lock_init(lock) \
do { \
static struct lock_class_key __key; \
\
debug_check_no_locks_freed((void *)lock, sizeof(*lock));\
lockdep_init_map_type(&(lock)->dep_map, #lock, &__key, \
0, LD_WAIT_CONFIG, LD_WAIT_INV, \
LD_LOCK_PERCPU); \
local_lock_debug_init(lock); \
} while (0)
#define __local_trylock_init(lock) \
do { \
__local_lock_init((local_lock_t *)lock); \
} while (0)
#define __spinlock_nested_bh_init(lock) \
do { \
static struct lock_class_key __key; \
\
debug_check_no_locks_freed((void *)lock, sizeof(*lock));\
lockdep_init_map_type(&(lock)->dep_map, #lock, &__key, \
0, LD_WAIT_CONFIG, LD_WAIT_INV, \
LD_LOCK_NORMAL); \
local_lock_debug_init(lock); \
} while (0)
#define __local_lock_acquire(lock) \
do { \
local_trylock_t *__tl; \
local_lock_t *__l; \
\
__l = (local_lock_t *)(lock); \
__tl = (local_trylock_t *)__l; \
_Generic((lock), \
local_trylock_t *: ({ \
lockdep_assert(__tl->acquired == 0); \
WRITE_ONCE(__tl->acquired, 1); \
}), \
local_lock_t *: (void)0); \
local_lock_acquire(__l); \
} while (0)
#define __local_lock(lock) \
do { \
preempt_disable(); \
__local_lock_acquire(lock); \
__acquire(lock); \
} while (0)
#define __local_lock_irq(lock) \
do { \
local_irq_disable(); \
__local_lock_acquire(lock); \
__acquire(lock); \
} while (0)
#define __local_lock_irqsave(lock, flags) \
do { \
local_irq_save(flags); \
__local_lock_acquire(lock); \
__acquire(lock); \
} while (0)
#define __local_trylock(lock) \
__try_acquire_ctx_lock(lock, ({ \
local_trylock_t *__tl; \
\
preempt_disable(); \
__tl = (lock); \
if (READ_ONCE(__tl->acquired)) { \
preempt_enable(); \
__tl = NULL; \
} else { \
WRITE_ONCE(__tl->acquired, 1); \
local_trylock_acquire( \
(local_lock_t *)__tl); \
} \
!!__tl; \
}))
#define __local_trylock_irqsave(lock, flags) \
__try_acquire_ctx_lock(lock, ({ \
local_trylock_t *__tl; \
\
local_irq_save(flags); \
__tl = (lock); \
if (READ_ONCE(__tl->acquired)) { \
local_irq_restore(flags); \
__tl = NULL; \
} else { \
WRITE_ONCE(__tl->acquired, 1); \
local_trylock_acquire( \
(local_lock_t *)__tl); \
} \
!!__tl; \
}))
/* preemption or migration must be disabled before calling __local_lock_is_locked */
#define __local_lock_is_locked(lock) READ_ONCE(this_cpu_ptr(lock)->acquired)
#define __local_lock_release(lock) \
do { \
local_trylock_t *__tl; \
local_lock_t *__l; \
\
__l = (local_lock_t *)(lock); \
__tl = (local_trylock_t *)__l; \
local_lock_release(__l); \
_Generic((lock), \
local_trylock_t *: ({ \
lockdep_assert(__tl->acquired == 1); \
WRITE_ONCE(__tl->acquired, 0); \
}), \
local_lock_t *: (void)0); \
} while (0)
#define __local_unlock(lock) \
do { \
__release(lock); \
__local_lock_release(lock); \
preempt_enable(); \
} while (0)
#define __local_unlock_irq(lock) \
do { \
__release(lock); \
__local_lock_release(lock); \
local_irq_enable(); \
} while (0)
#define __local_unlock_irqrestore(lock, flags) \
do { \
__release(lock); \
__local_lock_release(lock); \
local_irq_restore(flags); \
} while (0)
#define __local_lock_nested_bh(lock) \
do { \
lockdep_assert_in_softirq(); \
local_lock_acquire((lock)); \
__acquire(lock); \
} while (0)
#define __local_unlock_nested_bh(lock) \
do { \
__release(lock); \
local_lock_release((lock)); \
} while (0)
#else /* !CONFIG_PREEMPT_RT */
#include <linux/sched.h>
#include <linux/spinlock.h>
/*
* On PREEMPT_RT local_lock maps to a per CPU spinlock, which protects the
* critical section while staying preemptible.
*/
typedef spinlock_t local_lock_t;
typedef spinlock_t local_trylock_t;
#define INIT_LOCAL_LOCK(lockname) __LOCAL_SPIN_LOCK_UNLOCKED((lockname))
#define INIT_LOCAL_TRYLOCK(lockname) __LOCAL_SPIN_LOCK_UNLOCKED((lockname))
#define __local_lock_init(__l) \
do { \
local_spin_lock_init((__l)); \
} while (0)
#define __local_trylock_init(__l) __local_lock_init(__l)
#define __local_lock(__lock) \
do { \
migrate_disable(); \
spin_lock((__lock)); \
} while (0)
#define __local_lock_irq(lock) __local_lock(lock)
#define __local_lock_irqsave(lock, flags) \
do { \
typecheck(unsigned long, flags); \
flags = 0; \
__local_lock(lock); \
} while (0)
#define __local_unlock(__lock) \
do { \
spin_unlock((__lock)); \
migrate_enable(); \
} while (0)
#define __local_unlock_irq(lock) __local_unlock(lock)
#define __local_unlock_irqrestore(lock, flags) __local_unlock(lock)
#define __local_lock_nested_bh(lock) \
do { \
lockdep_assert_in_softirq_func(); \
spin_lock((lock)); \
} while (0)
#define __local_unlock_nested_bh(lock) \
do { \
spin_unlock((lock)); \
} while (0)
#define __local_trylock(lock) \
__try_acquire_ctx_lock(lock, context_unsafe(({ \
int __locked; \
\
if (in_nmi() | in_hardirq()) { \
__locked = 0; \
} else { \
migrate_disable(); \
__locked = spin_trylock((lock)); \
if (!__locked) \
migrate_enable(); \
} \
__locked; \
})))
#define __local_trylock_irqsave(lock, flags) \
__try_acquire_ctx_lock(lock, ({ \
typecheck(unsigned long, flags); \
flags = 0; \
__local_trylock(lock); \
}))
/* migration must be disabled before calling __local_lock_is_locked */
#define __local_lock_is_locked(__lock) \
(rt_mutex_owner(&this_cpu_ptr(__lock)->lock) == current)
#endif /* CONFIG_PREEMPT_RT */
#if defined(WARN_CONTEXT_ANALYSIS) && !defined(__CHECKER__)
/*
* Because the compiler only knows about the base per-CPU variable, use this
* helper function to make the compiler think we lock/unlock the @base variable,
* and hide the fact we actually pass the per-CPU instance to lock/unlock
* functions.
*/
static __always_inline local_lock_t *__this_cpu_local_lock(local_lock_t __percpu *base)
__returns_ctx_lock(base) __attribute__((overloadable))
{
return this_cpu_ptr(base);
}
#ifndef CONFIG_PREEMPT_RT
static __always_inline local_trylock_t *__this_cpu_local_lock(local_trylock_t __percpu *base)
__returns_ctx_lock(base) __attribute__((overloadable))
{
return this_cpu_ptr(base);
}
#endif /* CONFIG_PREEMPT_RT */
#else /* WARN_CONTEXT_ANALYSIS */
#define __this_cpu_local_lock(base) this_cpu_ptr(base)
#endif /* WARN_CONTEXT_ANALYSIS */
]
|
| 29/2 |
2026/05/03 02:42 |
llm |
expert |
0m
Model:
Tokens:
input: 34849
output: 43
thoughts: 156
Thoughts: **Analyzing the Memory Barrier Concerns in Non-RT Locking and Folio Batch Operations**
Okay, so I'm looking at the non-RT side of things, specifically how locks work. In a non-RT context, `local_lock` basically just boils down to a `preempt_disable()` call, along with some lockdep checks. The important thing here is that there's no explicit memory barrier involved. No mechanism to synchronize memory operations with other CPUs. That feels... risky, especially considering what I'm diving into.
Now I'm tracing the flow, and I see that `__lru_add_drain_all` is using `smp_mb()` and `smp_load_acquire`. These are definitely memory barrier operations β a stronger signal. But, what's concerning me is that the writer, the `mlock_folio` function, doesn't seem to have a corresponding barrier. That feels like a potential race condition and I need to explore this.
Let me see... `mlock_folio` calls `folio_batch_add`. Bingo.
And looking into `folio_batch_add` now, this is in `include/linux/folio_batch.h` - ahh, that makes more sense in the context of batches of folios and potential synchronization issues there. This is where the real potential for trouble lies. The code's probably doing something interesting, but if `mlock_folio` isn't properly synchronizing, we've got a problem. I've got to investigate the `folio_batch_add` function's interactions.
|
| 30/2 |
2026/05/03 02:42 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/linux/folio_batch.h IncludeLines:true Name:folio_batch_add]
Results: map[SourceCode: 64: /**
65: * folio_batch_add() - Add a folio to a batch.
66: * @fbatch: The folio batch.
67: * @folio: The folio to add.
68: *
69: * The folio is added to the end of the batch.
70: * The batch must have previously been initialised using folio_batch_init().
71: *
72: * Return: The number of slots still available.
73: */
74: static inline unsigned folio_batch_add(struct folio_batch *fbatch,
75: struct folio *folio)
76: {
77: fbatch->folios[fbatch->nr++] = folio;
78: return folio_batch_space(fbatch);
79: }
SourceFile:include/linux/folio_batch.h]
|
| 31/2 |
2026/05/03 02:42 |
llm |
expert |
0m
Model:
Tokens:
input: 35638
output: 42
thoughts: 771
Thoughts: **Analyzing the Folio Batching Race Condition**
Okay, so I'm looking at this folio batching code and I'm trying to figure out if there's a potential race condition. The basic idea is pretty straightforward: `fbatch->nr++` is just an increment to track the number of folios in the batch. The real race is around this `fbatch->nr` counter.
The key context here is `__lru_add_drain_all`. It's used to ensure that all batched folios are drained, meaning they're moved to the LRU lists or wherever they need to go after the draining process. My initial thought is, if a folio is added *after* `__lru_add_drain_all` starts, missing it in that particular drain cycle might not be catastrophic. The call *shouldn't* expect it to be there, as long as it handles it on its next call. But, if it was added *before* the function starts, it *must* be drained.
Now, that `smp_mb()` in `__lru_add_drain_all` at line 832 - it's a memory barrier. I had to double-check the comment, and it seems the barrier's job is to guarantee that the folio batch updates on *this* CPU are visible to other CPUs before it grabs the current drain generation. This covers the case where the current CPU added something to its batch and then is the one calling `lru_add_drain_all`. That makes sense.
Then we have the barrier at line 877, where we update `lru_drain_gen`. The barrier is crucial here: it makes sure that `lru_drain_gen` is updated before other CPUs can read their `nr` value and potentially make decisions based on that version of the generation. However, I see a problem: what about other CPUs? They don't have a barrier after updating `nr`. So even with the `smp_mb()` barrier on CPU 0, it might not see the updates from other CPUs, like CPU 1, if CPU 1 doesn't also have a barrier. We have a classic read-after-write dependency here.
Now, let's trace this back. I know that `mlock_folio` is called from `__folio_add_anon_rmap`. The call stack looks like: `__folio_add_anon_rmap`, then `folio_add_anon_rmap_ptes`, then `remove_migration_pte`. Within `remove_migration_pte` it calls either `folio_add_anon_rmap_ptes` or `folio_add_file_rmap_ptes`, based on whether the folio is anonymous.
So, `__folio_add_anon_rmap` calls `mlock_vma_folio`, which calls `mlock_folio`. Okay, I will now inspect `__folio_add_anon_rmap` directly. This seems like a critical path to examine.
|
| 32/2 |
2026/05/03 02:42 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/rmap.c IncludeLines:true Name:__folio_add_anon_rmap]
Results: map[SourceCode:1516: static __always_inline void __folio_add_anon_rmap(struct folio *folio,
1517: struct page *page, int nr_pages, struct vm_area_struct *vma,
1518: unsigned long address, rmap_t flags, enum pgtable_level level)
1519: {
1520: int i;
1521:
1522: VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
1523:
1524: __folio_add_rmap(folio, page, nr_pages, vma, level);
1525:
1526: if (likely(!folio_test_ksm(folio)))
1527: __page_check_anon_rmap(folio, page, vma, address);
1528:
1529: if (flags & RMAP_EXCLUSIVE) {
1530: switch (level) {
1531: case PGTABLE_LEVEL_PTE:
1532: for (i = 0; i < nr_pages; i++)
1533: SetPageAnonExclusive(page + i);
1534: break;
1535: case PGTABLE_LEVEL_PMD:
1536: SetPageAnonExclusive(page);
1537: break;
1538: case PGTABLE_LEVEL_PUD:
1539: /*
1540: * Keep the compiler happy, we don't support anonymous
1541: * PUD mappings.
1542: */
1543: WARN_ON_ONCE(1);
1544: break;
1545: default:
1546: BUILD_BUG();
1547: }
1548: }
1549:
1550: VM_WARN_ON_FOLIO(!folio_test_large(folio) && PageAnonExclusive(page) &&
1551: atomic_read(&folio->_mapcount) > 0, folio);
1552: for (i = 0; i < nr_pages; i++) {
1553: struct page *cur_page = page + i;
1554:
1555: VM_WARN_ON_FOLIO(folio_test_large(folio) &&
1556: folio_entire_mapcount(folio) > 1 &&
1557: PageAnonExclusive(cur_page), folio);
1558: if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT))
1559: continue;
1560:
1561: /*
1562: * While PTE-mapping a THP we have a PMD and a PTE
1563: * mapping.
1564: */
1565: VM_WARN_ON_FOLIO(atomic_read(&cur_page->_mapcount) > 0 &&
1566: PageAnonExclusive(cur_page), folio);
1567: }
1568:
1569: /*
1570: * Only mlock it if the folio is fully mapped to the VMA.
1571: *
1572: * Partially mapped folios can be split on reclaim and part outside
1573: * of mlocked VMA can be evicted or freed.
1574: */
1575: if (folio_nr_pages(folio) == nr_pages)
1576: mlock_vma_folio(folio, vma);
1577: }
SourceFile:mm/rmap.c]
|
| 33/2 |
2026/05/03 02:42 |
llm |
expert |
0m
Model:
Tokens:
input: 38065
output: 41
thoughts: 150
Thoughts: **Analyzing `mlock_vma_folio`**
Okay, so I'm looking at this inline function `mlock_vma_folio` and immediately my brain's going through the mechanics. It seems to be using the `VMA`'s `mlock_count` to avoid redundant `mlock` calls, which is a common optimization to prevent unnecessary system calls. The code snippet says: "A value > 0 means we've already mlocked this folio in this VMA." That makes perfect sense. We only need to lock the folio if the `VMA`'s `vm_flags` has the `VM_LOCKED` flag set. If it is locked already, then call mlock_folio, which is the call to mlock.
But wait, hold on. Let me just quickly confirm my assumptions and ensure I'm getting the full picture by checking the actual definition of `mlock_vma_folio` in `mm/internal.h`. I've got a feeling there might be a nuance or two in the way this function is implemented that I should make sure I fully grasp. Context is everything when dealing with memory management.
|
| 34/2 |
2026/05/03 02:42 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/internal.h IncludeLines:true Name:mlock_vma_folio]
Results: map[SourceCode:1187: static inline void mlock_vma_folio(struct folio *folio,
1188: struct vm_area_struct *vma)
1189: {
1190: /*
1191: * The VM_SPECIAL check here serves two purposes.
1192: * 1) VM_IO check prevents migration from double-counting during mlock.
1193: * 2) Although mmap_region() and mlock_fixup() take care that VM_LOCKED
1194: * is never left set on a VM_SPECIAL vma, there is an interval while
1195: * file->f_op->mmap() is using vm_insert_page(s), when VM_LOCKED may
1196: * still be set while VM_SPECIAL bits are added: so ignore it then.
1197: */
1198: if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED))
1199: mlock_folio(folio);
1200: }
SourceFile:mm/internal.h]
|
| 35/2 |
2026/05/03 02:42 |
llm |
expert |
0m
Model:
Tokens:
input: 38813
output: 40
thoughts: 1967
Thoughts: **Analyzing the `mlock_fbatch` Data Race**
Okay, so I'm digging into this potential data race around `mlock_fbatch->nr`, the folio batch count. The core of the issue seems to be the synchronization (or lack thereof) between writers updating `nr` and readers checking it. Specifically, `folios_put_refs` in one context, and `need_mlock_drain` and `lru_add_drain_all` in other contexts.
The critical path looks like this: `mlock_folio` adds a folio to a per-CPU batch, increasing `nr`. Then, either the batch fills and `mlock_folio_batch` drains it, or `lru_add_drain_all` is called to force a drain. The problem arises if a reader of `nr` gets a stale value. If a drain is missed, pages could potentially stick around longer than they should. In extreme cases, this could mean that pages are not invalidated when they need to be, for example during a device unmount. However, the use of `mlock` is typically for pages in use, so that helps to narrow the potential concerns.
Let's drill down: `folios_put_refs` can set `nr` to either `0` or `j` (the number of folios to be freed). The writer can set `nr = 0` if the batch is reinitialized or `nr = j` when the batch has folios that need to be freed. The reader, `need_mlock_drain`, reads `nr` to determine whether a drain is needed. If the reader sees a stale `nr`, it might miss a drain.
`__lru_add_drain_all` is supposed to guarantee that everything is drained, to ensure pages are moved to the LRU lists or are properly freed. It uses `lru_drain_gen`, but this lacks a memory barrier on the writer side. So what if the batch count changes after the reader checks the count?
My focus is on understanding what happens during unmount or when `invalidate_bdev` is called. Specifically, `ext4_put_super` calls `invalidate_bdev` which calls `lru_add_drain_all`. We want to be absolutely sure all buffers/pages are gone. If a page gets missed because it's stuck in an mlock batch, it could be a real problem.
In the case where the batch *is* draining, `mlock_folio_batch` first processes the folios, then calls `folios_put` to release references. `folios_put` calls `folios_put_refs` which will eventually set `nr=0`. So, if `need_mlock_drain` sees the old value while the drain is happening, it isn't an issue. If `need_mlock_drain` sees `nr = 0` then the drain is already finished and it is ok.
However, the reported KCSAN race is between `folios_put_refs` writing and `need_mlock_drain` reading `nr`. The key is when `nr` is not zero in the first case.
In `folios_put_refs`, the variable `j` represents folios that need freeing. So `nr = j` happens if a folio has a refcount of zero.
Now I remember. `j` is the number of folios that were not *freed* yet because their refcount wasn't zero. That means the `nr=j` write is only happening if there are outstanding references. Those pages will eventually be freed later, but they might hang around in the page cache longer than expected.
Okay, let me review `folios_put_refs`:
* `j` counts folios to be freed.
* `j` is the number of folios that passed the `folio_ref_sub_and_test(folio, nr_refs)` check.
* Then it will be freed by `free_unref_folios`.
So if `need_mlock_drain` misses the batch when it's being populated by folios from `mlock_folio`, then they might linger longer than necessary. In the worst case, this could mean pages that should have been invalidated are still present, particularly during unmount. In general, a drain is not necessary if there are outstanding references.
In summary, there's a potential for a race, but the impact is probably not catastrophic given that `mlock` is used on pages in use. However, it *could* lead to pages lingering longer than they should, which warrants careful consideration in contexts like device unmounts where we need to ensure all resources are released. The absence of a strong memory barrier on the writer side is the main concern here
|
| 36/2 |
2026/05/03 02:42 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/swap.c IncludeLines:true Name:free_unref_folios]
Results: map[SourceCode:2989: /*
2990: * Free a batch of folios
2991: */
2992: void free_unref_folios(struct folio_batch *folios)
2993: {
2994: struct per_cpu_pages *pcp = NULL;
2995: struct zone *locked_zone = NULL;
2996: int i, j;
2997:
2998: /* Prepare folios for freeing */
2999: for (i = 0, j = 0; i < folios->nr; i++) {
3000: struct folio *folio = folios->folios[i];
3001: unsigned long pfn = folio_pfn(folio);
3002: unsigned int order = folio_order(folio);
3003:
3004: if (!__free_pages_prepare(&folio->page, order, FPI_NONE))
3005: continue;
3006: /*
3007: * Free orders not handled on the PCP directly to the
3008: * allocator.
3009: */
3010: if (!pcp_allowed_order(order)) {
3011: free_one_page(folio_zone(folio), &folio->page,
3012: pfn, order, FPI_NONE);
3013: continue;
3014: }
3015: folio->private = (void *)(unsigned long)order;
3016: if (j != i)
3017: folios->folios[j] = folio;
3018: j++;
3019: }
3020: folios->nr = j;
3021:
3022: for (i = 0; i < folios->nr; i++) {
3023: struct folio *folio = folios->folios[i];
3024: struct zone *zone = folio_zone(folio);
3025: unsigned long pfn = folio_pfn(folio);
3026: unsigned int order = (unsigned long)folio->private;
3027: int migratetype;
3028:
3029: folio->private = NULL;
3030: migratetype = get_pfnblock_migratetype(&folio->page, pfn);
3031:
3032: /* Different zone requires a different pcp lock */
3033: if (zone != locked_zone ||
3034: is_migrate_isolate(migratetype)) {
3035: if (pcp) {
3036: pcp_spin_unlock(pcp);
3037: locked_zone = NULL;
3038: pcp = NULL;
3039: }
3040:
3041: /*
3042: * Free isolated pages directly to the
3043: * allocator, see comment in free_frozen_pages.
3044: */
3045: if (is_migrate_isolate(migratetype)) {
3046: free_one_page(zone, &folio->page, pfn,
3047: order, FPI_NONE);
3048: continue;
3049: }
3050:
3051: /*
3052: * trylock is necessary as folios may be getting freed
3053: * from IRQ or SoftIRQ context after an IO completion.
3054: */
3055: pcp = pcp_spin_trylock(zone->per_cpu_pageset);
3056: if (unlikely(!pcp)) {
3057: free_one_page(zone, &folio->page, pfn,
3058: order, FPI_NONE);
3059: continue;
3060: }
3061: locked_zone = zone;
3062: }
3063:
3064: /*
3065: * Non-isolated types over MIGRATE_PCPTYPES get added
3066: * to the MIGRATE_MOVABLE pcp list.
3067: */
3068: if (unlikely(migratetype >= MIGRATE_PCPTYPES))
3069: migratetype = MIGRATE_MOVABLE;
3070:
3071: trace_mm_page_free_batched(&folio->page);
3072: if (!free_frozen_page_commit(zone, pcp, &folio->page,
3073: migratetype, order, FPI_NONE)) {
3074: pcp = NULL;
3075: locked_zone = NULL;
3076: }
3077: }
3078:
3079: if (pcp)
3080: pcp_spin_unlock(pcp);
3081: folio_batch_reinit(folios);
3082: }
SourceFile:mm/page_alloc.c]
|
| 37/2 |
2026/05/03 02:42 |
llm |
expert |
0m
Model:
Tokens:
input: 43209
output: 40
thoughts: 1777
Thoughts: **Analyzing Potential Data Races in mlock Folio Batch Handling**
Okay, so I'm looking into how `mlock_folio_batch` works, specifically focusing on potential data races in the interaction between adding folios to the batch and draining the batch. The initial confusion about `free_unref_folios` being in `mm/swap.c` was quickly cleared up; it's actually in `mm/page_alloc.c`. This is where my journey started.
My concern centers around the `fbatch->nr` counter, which tracks the number of folios in the batch. The `folios_put` sequence calls `free_unref_folios`, which in turn, manipulates this counter. The `need_mlock_drain` function reads this counter to determine if a drain is required. The challenge is ensuring that readers of `nr` see a consistent view, and that a drain isn't missed because the counter has been updated.
The primary worry is the scenario where `mlock_folio` adds a folio (incrementing `nr` from 0 to 1) and a reader observes an intermediate zero value. This would mean that the drain might be missed, which sounds like a potential bug.
I'm thinking about the implications. If `lru_add_drain_all` misses a page that was just added, and returns, it means that a caller of `lru_add_drain_all` might get an incomplete job done. For example, in the context of `ext4_put_super`, which calls `invalidate_bdev` and `lru_add_drain_all`, this could lead to a block device not being fully invalidated, which potentially causes issues during unmount. However, the use case for the `mlock_fbatch` is specifically `mlock`, and pages being mlocked are in use, so that might be an invalid concern.
Now, considering the stack trace, I see that `mlock_folio` is called during migration (`remove_migration_pte`), which means that the folio has to be mlocked after migration. It's a bit of a race anyway if `lru_add_drain_all` is called while migration is happening. The presence of a KCSAN data race alert is the thing that kicked off this investigation.
The `lru_add_drain_all` code explicitly acknowledges that it's doing lockless reads of `nr`, but doesn't use `READ_ONCE`, which seems concerning. `cpu_needs_drain` just reads the value of the counter. The kernel frequently uses "lockless check then take lock" patterns, but often includes `data_race()` or `READ_ONCE()` annotations. The absence of these is suspicious.
The critical question is: is there any actual harm? My brain is thinking through various scenarios: If the compiler reloads `fbatch->nr` and sees different values, it could be a problem. In `cpu_needs_drain`, it's just a boolean check, and seems safe because the caller of this code will eventually take a lock. If `cpu_needs_drain` sees `true`, it schedules work, the `mlock_drain_local` that's called takes the lock, and re-checks `nr`. This part is safe. The danger is, of course, the opposite case, where a `false` is seen when a `true` should have been. The code misses the drain.
Missing the drain *could* be a problem for callers of `lru_add_drain_all` where the call is *not* best-effort. Specifically, it can be a problem if it's required for correctness, as is the case for `alloc_contig_range` or `offline_pages`.
The intention with the `lru_drain_gen` logic, is that this ordering should be handled correctly, but it only works if there are matching barriers on the writer side. It is critical that I check that `mlock_folio` *does* have these barriers.
And now I'm checking the source. It turns out that `mlock_folio` uses `local_lock` and `local_unlock`. On non-RT kernels, `local_unlock` is just `preempt_enable()`. No barrier. Thus, this is the crux. So CPU 1 can update `nr` and that change might not be visible to CPU 0 for a while. This is a classic data race scenario. The existing `smp_mb()` on CPU 0 won't help if CPU 1 doesn't have a barrier.
It clicks that `lru_add_drain_all` has been a known source of KCSAN reports. In the past, there was a patch series to add `READ_ONCE
|
| 38/2 |
2026/05/03 02:42 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/swap.c IncludeLines:true Name:folio_add_lru]
Results: map[SourceCode: 497: /**
498: * folio_add_lru - Add a folio to an LRU list.
499: * @folio: The folio to be added to the LRU.
500: *
501: * Queue the folio for addition to the LRU. The decision on whether
502: * to add the page to the [in]active [file|anon] list is deferred until the
503: * folio_batch is drained. This gives a chance for the caller of folio_add_lru()
504: * have the folio added to the active list using folio_mark_accessed().
505: */
506: void folio_add_lru(struct folio *folio)
507: {
508: VM_BUG_ON_FOLIO(folio_test_active(folio) &&
509: folio_test_unevictable(folio), folio);
510: VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
511:
512: /* see the comment in lru_gen_folio_seq() */
513: if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
514: lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))
515: folio_set_active(folio);
516:
517: folio_batch_add_and_move(folio, lru_add);
518: }
SourceFile:mm/swap.c]
|
| 39/2 |
2026/05/03 02:42 |
llm |
expert |
0m
Model:
Tokens:
input: 46489
output: 43
thoughts: 13
Thoughts: Okay, here's my interpretation of those thoughts, crafted with the understanding that I'm discussing this with a fellow expert:
**Navigating the `folio_batch_add_and_move` Function**
Hmm, so we're looking at `folio_batch_add_and_move`, are we? What's going on with that particular function in our code? That's what I'm focused on right now. What's the context? What exactly are we using it for? Are we adding new things to the folio? And *then* moving them? What are the implications of that dual operation? I need to think this through. Is the order of those actions critical? Are there potential concurrency issues if we're doing this in a multi-threaded environment? Perhaps the function uses locking internally, or maybe it's relying on some other mechanism to ensure data consistency. I'm wondering if there are any gotchas, edge cases, or optimizations we should be aware of. I need to dig deeper into the implementation details of this function. I'll need to go through its documentation or, better yet, dive into the source code to really understand its behavior. The details of how this function executes are what I am considering.
|
| 40/2 |
2026/05/03 02:43 |
tool |
codesearch-definition-source |
0m
Error:
Args: map[ContextFile:mm/swap.c IncludeLines:true Name:folio_batch_add_and_move]
Results: map[SourceCode: SourceFile:]
|
| 41/2 |
2026/05/03 02:43 |
llm |
expert |
0m
Model:
Tokens:
input: 46826
output: 25
thoughts: 147
Thoughts: **Analyzing a Macro Definition**
Okay, so looking at this code, my initial instinct is that it's likely a macro. The ` #define ` pretty much screams that. The naming convention, `folio_batch_add_and_move`, suggests it's designed to perform some kind of batch operation with folios. The `do { ... } while (0)` construct is a classic macro pattern, designed to handle multi-statement blocks safely.
Let me break it down further. I see `struct folio_batch *fbatch;` which defines a local pointer to something called `folio_batch`. Then, there's a local lock and access to per-CPU data using `this_cpu_ptr`. Iβm guessing it's some form of batching or queueing for folios, per CPU. The `folio_batch_add` call itself looks like the core operation. If that fails, based on the `if` statement, we run `__lru_add_drain_cpu`. The locking and unlocking around the critical section are also what I expect.
Just to be sure, and to understand its usage context better, let me quickly grep for instances of this macro in the codebase. That'll give me a clearer picture of how it's actually used and how it fits into the larger architecture.
|
| 42/2 |
2026/05/03 02:43 |
tool |
grepper |
0m
Args: map[Expression:#define folio_batch_add_and_move]
Results: map[Output:mm/swap.c=182=static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
--
mm/swap.c-203-
mm/swap.c:204:#define folio_batch_add_and_move(folio, op) \
mm/swap.c-205- __folio_batch_add_and_move( \
]
|
| 43/2 |
2026/05/03 02:43 |
llm |
expert |
0m
Model:
Tokens:
input: 47390
output: 20
thoughts: 145
Thoughts: **Analyzing Folio Batch Management**
Okay, so I'm looking at this `__folio_batch_add_and_move` function. Let me break down what's happening. My first thought is to consider the context of its execution. It seems to be related to managing batches of `folio` structures, likely for efficient memory management. I see it accepts a per-CPU `folio_batch` structure, the `folio` itself, and a flag, `lru_cache_disabled`.
The core logic revolves around adding the given `folio` to the batch. I can see a local lock being acquired on `cpu_fbatches.lock`. That suggests that this function is managing a shared resource, specifically a collection of folio batches, likely one per CPU, which aligns with my understanding of `this_cpu_ptr`. I should really check how `cpu_fbatches` is defined and how this lock protects access to the per-CPU batches.
The `folio_batch_add` function is called to attempt adding the `folio` to the current CPU's batch. Crucially, the outcome of `folio_batch_add` is checked. If it fails *or* the `lru_cache_disabled` flag is true, we then proceed to drain the LRU for the CPU. This seems to imply that either the batch is full, or that the LRU is not able to be updated. Itβs a mechanism to ensure that the memory isn't just left accumulating in the batch if it canβt be immediately managed efficiently.
Finally, the local lock is released. This ensures that concurrent access to the `cpu_fbatches` is handled carefully.
|
| 44/2 |
2026/05/03 02:43 |
tool |
read-file |
0m
Args:
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/mm/swap.c
*
* Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
*/
/*
* This file contains the default values for the operation of the
* Linux VM subsystem. Fine-tuning documentation can be found in
* Documentation/admin-guide/sysctl/vm.rst.
* Started 18.12.91
* Swap aging added 23.2.95, Stephen Tweedie.
* Buffermem limits added 12.3.98, Rik van Riel.
*/
#include <linux/mm.h>
#include <linux/sched.h>
#include <linux/kernel_stat.h>
#include <linux/swap.h>
#include <linux/mman.h>
#include <linux/pagemap.h>
#include <linux/folio_batch.h>
#include <linux/init.h>
#include <linux/export.h>
#include <linux/mm_inline.h>
#include <linux/percpu_counter.h>
#include <linux/memremap.h>
#include <linux/percpu.h>
#include <linux/cpu.h>
#include <linux/notifier.h>
#include <linux/backing-dev.h>
#include <linux/memcontrol.h>
#include <linux/gfp.h>
#include <linux/uio.h>
#include <linux/hugetlb.h>
#include <linux/page_idle.h>
#include <linux/local_lock.h>
#include <linux/buffer_head.h>
#include "internal.h"
#define CREATE_TRACE_POINTS
#include <trace/events/pagemap.h>
/* How many pages do we try to swap or page in/out together? As a power of 2 */
int page_cluster;
static const int page_cluster_max = 31;
struct cpu_fbatches {
/*
* The following folio batches are grouped together because they are protected
* by disabling preemption (and interrupts remain enabled).
*/
local_lock_t lock;
struct folio_batch lru_add;
struct folio_batch lru_deactivate_file;
struct folio_batch lru_deactivate;
struct folio_batch lru_lazyfree;
#ifdef CONFIG_SMP
struct folio_batch lru_activate;
#endif
/* Protecting the following batches which require disabling interrupts */
local_lock_t lock_irq;
struct folio_batch lru_move_tail;
};
static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = {
.lock = INIT_LOCAL_LOCK(lock),
.lock_irq = INIT_LOCAL_LOCK(lock_irq),
};
static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp,
unsigned long *flagsp)
{
if (folio_test_lru(folio)) {
folio_lruvec_relock_irqsave(folio, lruvecp, flagsp);
lruvec_del_folio(*lruvecp, folio);
__folio_clear_lru_flags(folio);
}
}
/*
* This path almost never happens for VM activity - pages are normally freed
* in batches. But it gets used by networking - and for compound pages.
*/
static void page_cache_release(struct folio *folio)
{
struct lruvec *lruvec = NULL;
unsigned long flags;
__page_cache_release(folio, &lruvec, &flags);
if (lruvec)
lruvec_unlock_irqrestore(lruvec, flags);
}
void __folio_put(struct folio *folio)
{
if (unlikely(folio_is_zone_device(folio))) {
free_zone_device_folio(folio);
return;
}
if (folio_test_hugetlb(folio)) {
free_huge_folio(folio);
return;
}
page_cache_release(folio);
folio_unqueue_deferred_split(folio);
mem_cgroup_uncharge(folio);
free_frozen_pages(&folio->page, folio_order(folio));
}
EXPORT_SYMBOL(__folio_put);
typedef void (*move_fn_t)(struct lruvec *lruvec, struct folio *folio);
static void lru_add(struct lruvec *lruvec, struct folio *folio)
{
int was_unevictable = folio_test_clear_unevictable(folio);
long nr_pages = folio_nr_pages(folio);
VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
/*
* Is an smp_mb__after_atomic() still required here, before
* folio_evictable() tests the mlocked flag, to rule out the possibility
* of stranding an evictable folio on an unevictable LRU? I think
* not, because __munlock_folio() only clears the mlocked flag
* while the LRU lock is held.
*
* (That is not true of __page_cache_release(), and not necessarily
* true of folios_put(): but those only clear the mlocked flag after
* folio_put_testzero() has excluded any other users of the folio.)
*/
if (folio_evictable(folio)) {
if (was_unevictable)
__count_vm_events(UNEVICTABLE_PGRESCUED, nr_pages);
} else {
folio_clear_active(folio);
folio_set_unevictable(folio);
/*
* folio->mlock_count = !!folio_test_mlocked(folio)?
* But that leaves __mlock_folio() in doubt whether another
* actor has already counted the mlock or not. Err on the
* safe side, underestimate, let page reclaim fix it, rather
* than leaving a page on the unevictable LRU indefinitely.
*/
folio->mlock_count = 0;
if (!was_unevictable)
__count_vm_events(UNEVICTABLE_PGCULLED, nr_pages);
}
lruvec_add_folio(lruvec, folio);
trace_mm_lru_insertion(folio);
}
static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
{
int i;
struct lruvec *lruvec = NULL;
unsigned long flags = 0;
for (i = 0; i < folio_batch_count(fbatch); i++) {
struct folio *folio = fbatch->folios[i];
/* block memcg migration while the folio moves between lru */
if (move_fn != lru_add && !folio_test_clear_lru(folio))
continue;
folio_lruvec_relock_irqsave(folio, &lruvec, &flags);
move_fn(lruvec, folio);
folio_set_lru(folio);
}
if (lruvec)
lruvec_unlock_irqrestore(lruvec, flags);
folios_put(fbatch);
}
static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
struct folio *folio, move_fn_t move_fn, bool disable_irq)
{
unsigned long flags;
folio_get(folio);
if (disable_irq)
local_lock_irqsave(&cpu_fbatches.lock_irq, flags);
else
local_lock(&cpu_fbatches.lock);
if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
!folio_may_be_lru_cached(folio) || lru_cache_disabled())
folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);
if (disable_irq)
local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags);
else
local_unlock(&cpu_fbatches.lock);
}
#define folio_batch_add_and_move(folio, op) \
__folio_batch_add_and_move( \
&cpu_fbatches.op, \
folio, \
op, \
offsetof(struct cpu_fbatches, op) >= \
offsetof(struct cpu_fbatches, lock_irq) \
)
static void lru_move_tail(struct lruvec *lruvec, struct folio *folio)
{
if (folio_test_unevictable(folio))
return;
lruvec_del_folio(lruvec, folio);
folio_clear_active(folio);
lruvec_add_folio_tail(lruvec, folio);
__count_vm_events(PGROTATED, folio_nr_pages(folio));
}
/*
* Writeback is about to end against a folio which has been marked for
* immediate reclaim. If it still appears to be reclaimable, move it
* to the tail of the inactive list.
*
* folio_rotate_reclaimable() must disable IRQs, to prevent nasty races.
*/
void folio_rotate_reclaimable(struct folio *folio)
{
if (folio_test_locked(folio) || folio_test_dirty(folio) ||
folio_test_unevictable(folio) || !folio_test_lru(folio))
return;
folio_batch_add_and_move(folio, lru_move_tail);
}
void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
unsigned int nr_io, unsigned int nr_rotated)
__releases(lruvec->lru_lock)
__releases(rcu)
{
unsigned long cost;
/*
* Reflect the relative cost of incurring IO and spending CPU
* time on rotations. This doesn't attempt to make a precise
* comparison, it just says: if reloads are about comparable
* between the LRU lists, or rotations are overwhelmingly
* different between them, adjust scan balance for CPU work.
*/
cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated;
if (!cost) {
spin_unlock_irq(&lruvec->lru_lock);
rcu_read_unlock();
return;
}
for (;;) {
unsigned long lrusize;
/* Record cost event */
if (file)
lruvec->file_cost += cost;
else
lruvec->anon_cost += cost;
/*
* Decay previous events
*
* Because workloads change over time (and to avoid
* overflow) we keep these statistics as a floating
* average, which ends up weighing recent refaults
* more than old ones.
*/
lrusize = lruvec_page_state(lruvec, NR_INACTIVE_ANON) +
lruvec_page_state(lruvec, NR_ACTIVE_ANON) +
lruvec_page_state(lruvec, NR_INACTIVE_FILE) +
lruvec_page_state(lruvec, NR_ACTIVE_FILE);
if (lruvec->file_cost + lruvec->anon_cost > lrusize / 4) {
lruvec->file_cost /= 2;
lruvec->anon_cost /= 2;
}
spin_unlock_irq(&lruvec->lru_lock);
lruvec = parent_lruvec(lruvec);
if (!lruvec) {
rcu_read_unlock();
break;
}
spin_lock_irq(&lruvec->lru_lock);
}
}
void lru_note_cost_refault(struct folio *folio)
{
struct lruvec *lruvec;
lruvec = folio_lruvec_lock_irq(folio);
lru_note_cost_unlock_irq(lruvec, folio_is_file_lru(folio),
folio_nr_pages(folio), 0);
}
static void lru_activate(struct lruvec *lruvec, struct folio *folio)
{
long nr_pages = folio_nr_pages(folio);
if (folio_test_active(folio) || folio_test_unevictable(folio))
return;
lruvec_del_folio(lruvec, folio);
folio_set_active(folio);
lruvec_add_folio(lruvec, folio);
trace_mm_lru_activate(folio);
__count_vm_events(PGACTIVATE, nr_pages);
count_memcg_events(lruvec_memcg(lruvec), PGACTIVATE, nr_pages);
}
#ifdef CONFIG_SMP
static void folio_activate_drain(int cpu)
{
struct folio_batch *fbatch = &per_cpu(cpu_fbatches.lru_activate, cpu);
if (folio_batch_count(fbatch))
folio_batch_move_lru(fbatch, lru_activate);
}
void folio_activate(struct folio *folio)
{
if (folio_test_active(folio) || folio_test_unevictable(folio) ||
!folio_test_lru(folio))
return;
folio_batch_add_and_move(folio, lru_activate);
}
#else
static inline void folio_activate_drain(int cpu)
{
}
void folio_activate(struct folio *folio)
{
struct lruvec *lruvec;
if (!folio_test_clear_lru(folio))
return;
lruvec = folio_lruvec_lock_irq(folio);
lru_activate(lruvec, folio);
lruvec_unlock_irq(lruvec);
folio_set_lru(folio);
}
#endif
static void __lru_cache_activate_folio(struct folio *folio)
{
struct folio_batch *fbatch;
int i;
local_lock(&cpu_fbatches.lock);
fbatch = this_cpu_ptr(&cpu_fbatches.lru_add);
/*
* Search backwards on the optimistic assumption that the folio being
* activated has just been added to this batch. Note that only
* the local batch is examined as a !LRU folio could be in the
* process of being released, reclaimed, migrated or on a remote
* batch that is currently being drained. Furthermore, marking
* a remote batch's folio active potentially hits a race where
* a folio is marked active just after it is added to the inactive
* list causing accounting errors and BUG_ON checks to trigger.
*/
for (i = folio_batch_count(fbatch) - 1; i >= 0; i--) {
struct folio *batch_folio = fbatch->folios[i];
if (batch_folio == folio) {
folio_set_active(folio);
break;
}
}
local_unlock(&cpu_fbatches.lock);
}
#ifdef CONFIG_LRU_GEN
static void lru_gen_inc_refs(struct folio *folio)
{
unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
if (folio_test_unevictable(folio))
return;
/* see the comment on LRU_REFS_FLAGS */
if (!folio_test_referenced(folio)) {
set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
return;
}
do {
if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
if (!folio_test_workingset(folio))
folio_set_workingset(folio);
return;
}
new_flags = old_flags + BIT(LRU_REFS_PGOFF);
} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
}
static bool lru_gen_clear_refs(struct folio *folio)
{
int gen = folio_lru_gen(folio);
int type = folio_is_file_lru(folio);
unsigned long seq;
if (gen < 0)
return true;
set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS | BIT(PG_workingset), 0);
rcu_read_lock();
seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
rcu_read_unlock();
/* whether can do without shuffling under the LRU lock */
return gen == lru_gen_from_seq(seq);
}
#else /* !CONFIG_LRU_GEN */
static void lru_gen_inc_refs(struct folio *folio)
{
}
static bool lru_gen_clear_refs(struct folio *folio)
{
return false;
}
#endif /* CONFIG_LRU_GEN */
/**
* folio_mark_accessed - Mark a folio as having seen activity.
* @folio: The folio to mark.
*
* This function will perform one of the following transitions:
*
* * inactive,unreferenced -> inactive,referenced
* * inactive,referenced -> active,unreferenced
* * active,unreferenced -> active,referenced
*
* When a newly allocated folio is not yet visible, so safe for non-atomic ops,
* __folio_set_referenced() may be substituted for folio_mark_accessed().
*/
void folio_mark_accessed(struct folio *folio)
{
if (folio_test_dropbehind(folio))
return;
if (lru_gen_enabled()) {
lru_gen_inc_refs(folio);
return;
}
if (!folio_test_referenced(folio)) {
folio_set_referenced(folio);
} else if (folio_test_unevictable(folio)) {
/*
* Unevictable pages are on the "LRU_UNEVICTABLE" list. But,
* this list is never rotated or maintained, so marking an
* unevictable page accessed has no effect.
*/
} else if (!folio_test_active(folio)) {
/*
* If the folio is on the LRU, queue it for activation via
* cpu_fbatches.lru_activate. Otherwise, assume the folio is in a
* folio_batch, mark it active and it'll be moved to the active
* LRU on the next drain.
*/
if (folio_test_lru(folio))
folio_activate(folio);
else
__lru_cache_activate_folio(folio);
folio_clear_referenced(folio);
workingset_activation(folio);
}
if (folio_test_idle(folio))
folio_clear_idle(folio);
}
EXPORT_SYMBOL(folio_mark_accessed);
/**
* folio_add_lru - Add a folio to an LRU list.
* @folio: The folio to be added to the LRU.
*
* Queue the folio for addition to the LRU. The decision on whether
* to add the page to the [in]active [file|anon] list is deferred until the
* folio_batch is drained. This gives a chance for the caller of folio_add_lru()
* have the folio added to the active list using folio_mark_accessed().
*/
void folio_add_lru(struct folio *folio)
{
VM_BUG_ON_FOLIO(folio_test_active(folio) &&
folio_test_unevictable(folio), folio);
VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
/* see the comment in lru_gen_folio_seq() */
if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))
folio_set_active(folio);
folio_batch_add_and_move(folio, lru_add);
}
EXPORT_SYMBOL(folio_add_lru);
/**
* folio_add_lru_vma() - Add a folio to the appropriate LRU list for this VMA.
* @folio: The folio to be added to the LRU.
* @vma: VMA in which the folio is mapped.
*
* If the VMA is mlocked, @folio is added to the unevictable list.
* Otherwise, it is treated the same way as folio_add_lru().
*/
void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma)
{
VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
if (unlikely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) == VM_LOCKED))
mlock_new_folio(folio);
else
folio_add_lru(folio);
}
/*
* If the folio cannot be invalidated, it is moved to the
* inactive list to speed up its reclaim. It is moved to the
* head of the list, rather than the tail, to give the flusher
* threads some time to write it out, as this is much more
* effective than the single-page writeout from reclaim.
*
* If the folio isn't mapped and dirty/writeback, the folio
* could be reclaimed asap using the reclaim flag.
*
* 1. active, mapped folio -> none
* 2. active, dirty/writeback folio -> inactive, head, reclaim
* 3. inactive, mapped folio -> none
* 4. inactive, dirty/writeback folio -> inactive, head, reclaim
* 5. inactive, clean -> inactive, tail
* 6. Others -> none
*
* In 4, it moves to the head of the inactive list so the folio is
* written out by flusher threads as this is much more efficient
* than the single-page writeout from reclaim.
*/
static void lru_deactivate_file(struct lruvec *lruvec, struct folio *folio)
{
bool active = folio_test_active(folio) || lru_gen_enabled();
long nr_pages = folio_nr_pages(folio);
if (folio_test_unevictable(folio))
return;
/* Some processes are using the folio */
if (folio_mapped(folio))
return;
lruvec_del_folio(lruvec, folio);
folio_clear_active(folio);
folio_clear_referenced(folio);
if (folio_test_writeback(folio) || folio_test_dirty(folio)) {
/*
* Setting the reclaim flag could race with
* folio_end_writeback() and confuse readahead. But the
* race window is _really_ small and it's not a critical
* problem.
*/
lruvec_add_folio(lruvec, folio);
folio_set_reclaim(folio);
} else {
/*
* The folio's writeback ended while it was in the batch.
* We move that folio to the tail of the inactive list.
*/
lruvec_add_folio_tail(lruvec, folio);
__count_vm_events(PGROTATED, nr_pages);
}
if (active) {
__count_vm_events(PGDEACTIVATE, nr_pages);
count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE,
nr_pages);
}
}
static void lru_deactivate(struct lruvec *lruvec, struct folio *folio)
{
long nr_pages = folio_nr_pages(folio);
if (folio_test_unevictable(folio) || !(folio_test_active(folio) || lru_gen_enabled()))
return;
lruvec_del_folio(lruvec, folio);
folio_clear_active(folio);
folio_clear_referenced(folio);
lruvec_add_folio(lruvec, folio);
__count_vm_events(PGDEACTIVATE, nr_pages);
count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_pages);
}
static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
{
long nr_pages = folio_nr_pages(folio);
if (!folio_test_anon(folio) || !folio_test_swapbacked(folio) ||
folio_test_swapcache(folio) || folio_test_unevictable(folio))
return;
lruvec_del_folio(lruvec, folio);
folio_clear_active(folio);
if (lru_gen_enabled())
lru_gen_clear_refs(folio);
else
folio_clear_referenced(folio);
/*
* Lazyfree folios are clean anonymous folios. They have
* the swapbacked flag cleared, to distinguish them from normal
* anonymous folios
*/
folio_clear_swapbacked(folio);
lruvec_add_folio(lruvec, folio);
__count_vm_events(PGLAZYFREE, nr_pages);
count_memcg_events(lruvec_memcg(lruvec), PGLAZYFREE, nr_pages);
}
/*
* Drain pages out of the cpu's folio_batch.
* Either "cpu" is the current CPU, and preemption has already been
* disabled; or "cpu" is being hot-unplugged, and is already dead.
*/
void lru_add_drain_cpu(int cpu)
{
struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
struct folio_batch *fbatch = &fbatches->lru_add;
if (folio_batch_count(fbatch))
folio_batch_move_lru(fbatch, lru_add);
fbatch = &fbatches->lru_move_tail;
/* Disabling interrupts below acts as a compiler barrier. */
if (data_race(folio_batch_count(fbatch))) {
unsigned long flags;
/* No harm done if a racing interrupt already did this */
local_lock_irqsave(&cpu_fbatches.lock_irq, flags);
folio_batch_move_lru(fbatch, lru_move_tail);
local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags);
}
fbatch = &fbatches->lru_deactivate_file;
if (folio_batch_count(fbatch))
folio_batch_move_lru(fbatch, lru_deactivate_file);
fbatch = &fbatches->lru_deactivate;
if (folio_batch_count(fbatch))
folio_batch_move_lru(fbatch, lru_deactivate);
fbatch = &fbatches->lru_lazyfree;
if (folio_batch_count(fbatch))
folio_batch_move_lru(fbatch, lru_lazyfree);
folio_activate_drain(cpu);
}
/**
* deactivate_file_folio() - Deactivate a file folio.
* @folio: Folio to deactivate.
*
* This function hints to the VM that @folio is a good reclaim candidate,
* for example if its invalidation fails due to the folio being dirty
* or under writeback.
*
* Context: Caller holds a reference on the folio.
*/
void deactivate_file_folio(struct folio *folio)
{
/* Deactivating an unevictable folio will not accelerate reclaim */
if (folio_test_unevictable(folio) || !folio_test_lru(folio))
return;
if (lru_gen_enabled() && lru_gen_clear_refs(folio))
return;
folio_batch_add_and_move(folio, lru_deactivate_file);
}
/*
* folio_deactivate - deactivate a folio
* @folio: folio to deactivate
*
* folio_deactivate() moves @folio to the inactive list if @folio was on the
* active list and was not unevictable. This is done to accelerate the
* reclaim of @folio.
*/
void folio_deactivate(struct folio *folio)
{
if (folio_test_unevictable(folio) || !folio_test_lru(folio))
return;
if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio))
return;
folio_batch_add_and_move(folio, lru_deactivate);
}
/**
* folio_mark_lazyfree - make an anon folio lazyfree
* @folio: folio to deactivate
*
* folio_mark_lazyfree() moves @folio to the inactive file list.
* This is done to accelerate the reclaim of @folio.
*/
void folio_mark_lazyfree(struct folio *folio)
{
if (!folio_test_anon(folio) || !folio_test_swapbacked(folio) ||
!folio_test_lru(folio) ||
folio_test_swapcache(folio) || folio_test_unevictable(folio))
return;
folio_batch_add_and_move(folio, lru_lazyfree);
}
void lru_add_drain(void)
{
local_lock(&cpu_fbatches.lock);
lru_add_drain_cpu(smp_processor_id());
local_unlock(&cpu_fbatches.lock);
mlock_drain_local();
}
/*
* It's called from per-cpu workqueue context in SMP case so
* lru_add_drain_cpu and invalidate_bh_lrus_cpu should run on
* the same cpu. It shouldn't be a problem in !SMP case since
* the core is only one and the locks will disable preemption.
*/
static void lru_add_and_bh_lrus_drain(void)
{
local_lock(&cpu_fbatches.lock);
lru_add_drain_cpu(smp_processor_id());
local_unlock(&cpu_fbatches.lock);
invalidate_bh_lrus_cpu();
mlock_drain_local();
}
void lru_add_drain_cpu_zone(struct zone *zone)
{
local_lock(&cpu_fbatches.lock);
lru_add_drain_cpu(smp_processor_id());
drain_local_pages(zone);
local_unlock(&cpu_fbatches.lock);
mlock_drain_local();
}
#ifdef CONFIG_SMP
static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work);
static void lru_add_drain_per_cpu(struct work_struct *dummy)
{
lru_add_and_bh_lrus_drain();
}
static bool cpu_needs_drain(unsigned int cpu)
{
struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
/* Check these in order of likelihood that they're not zero */
return folio_batch_count(&fbatches->lru_add) ||
folio_batch_count(&fbatches->lru_move_tail) ||
folio_batch_count(&fbatches->lru_deactivate_file) ||
folio_batch_count(&fbatches->lru_deactivate) ||
folio_batch_count(&fbatches->lru_lazyfree) ||
folio_batch_count(&fbatches->lru_activate) ||
need_mlock_drain(cpu) ||
has_bh_in_lru(cpu, NULL);
}
/*
* Doesn't need any cpu hotplug locking because we do rely on per-cpu
* kworkers being shut down before our page_alloc_cpu_dead callback is
* executed on the offlined cpu.
* Calling this function with cpu hotplug locks held can actually lead
* to obscure indirect dependencies via WQ context.
*/
static inline void __lru_add_drain_all(bool force_all_cpus)
{
/*
* lru_drain_gen - Global pages generation number
*
* (A) Definition: global lru_drain_gen = x implies that all generations
* 0 < n <= x are already *scheduled* for draining.
*
* This is an optimization for the highly-contended use case where a
* user space workload keeps constantly generating a flow of pages for
* each CPU.
*/
static unsigned int lru_drain_gen;
static struct cpumask has_work;
static DEFINE_MUTEX(lock);
unsigned cpu, this_gen;
/*
* Make sure nobody triggers this path before mm_percpu_wq is fully
* initialized.
*/
if (WARN_ON(!mm_percpu_wq))
return;
/*
* Guarantee folio_batch counter stores visible by this CPU
* are visible to other CPUs before loading the current drain
* generation.
*/
smp_mb();
/*
* (B) Locally cache global LRU draining generation number
*
* The read barrier ensures that the counter is loaded before the mutex
* is taken. It pairs with smp_mb() inside the mutex critical section
* at (D).
*/
this_gen = smp_load_acquire(&lru_drain_gen);
/* It helps everyone if we do our own local drain immediately. */
lru_add_drain();
mutex_lock(&lock);
/*
* (C) Exit the draining operation if a newer generation, from another
* lru_add_drain_all(), was already scheduled for draining. Check (A).
*/
if (unlikely(this_gen != lru_drain_gen && !force_all_cpus))
goto done;
/*
* (D) Increment global generation number
*
* Pairs with smp_load_acquire() at (B), outside of the critical
* section. Use a full memory barrier to guarantee that the
* new global drain generation number is stored before loading
* folio_batch counters.
*
* This pairing must be done here, before the for_each_online_cpu loop
* below which drains the page vectors.
*
* Let x, y, and z represent some system CPU numbers, where x < y < z.
* Assume CPU #z is in the middle of the for_each_online_cpu loop
* below and has already reached CPU #y's per-cpu data. CPU #x comes
* along, adds some pages to its per-cpu vectors, then calls
* lru_add_drain_all().
*
* If the paired barrier is done at any later step, e.g. after the
* loop, CPU #x will just exit at (C) and miss flushing out all of its
* added pages.
*/
WRITE_ONCE(lru_drain_gen, lru_drain_gen + 1);
smp_mb();
cpumask_clear(&has_work);
for_each_online_cpu(cpu) {
struct work_struct *work = &per_cpu(lru_add_drain_work, cpu);
if (cpu_needs_drain(cpu)) {
INIT_WORK(work, lru_add_drain_per_cpu);
queue_work_on(cpu, mm_percpu_wq, work);
__cpumask_set_cpu(cpu, &has_work);
}
}
for_each_cpu(cpu, &has_work)
flush_work(&per_cpu(lru_add_drain_work, cpu));
done:
mutex_unlock(&lock);
}
void lru_add_drain_all(void)
{
__lru_add_drain_all(false);
}
#else
void lru_add_drain_all(void)
{
lru_add_drain();
}
#endif /* CONFIG_SMP */
atomic_t lru_disable_count = ATOMIC_INIT(0);
/*
* lru_cache_disable() needs to be called before we start compiling
* a list of folios to be migrated using folio_isolate_lru().
* It drains folios on LRU cache and then disable on all cpus until
* lru_cache_enable is called.
*
* Must be paired with a call to lru_cache_enable().
*/
void lru_cache_disable(void)
{
atomic_inc(&lru_disable_count);
/*
* Readers of lru_disable_count are protected by either disabling
* preemption or rcu_read_lock:
*
* preempt_disable, local_irq_disable [bh_lru_lock()]
* rcu_read_lock [rt_spin_lock CONFIG_PREEMPT_RT]
* preempt_disable [local_lock !CONFIG_PREEMPT_RT]
*
* Since v5.1 kernel, synchronize_rcu() is guaranteed to wait on
* preempt_disable() regions of code. So any CPU which sees
* lru_disable_count = 0 will have exited the critical
* section when synchronize_rcu() returns.
*/
synchronize_rcu_expedited();
#ifdef CONFIG_SMP
__lru_add_drain_all(true);
#else
lru_add_and_bh_lrus_drain();
#endif
}
/**
* folios_put_refs - Reduce the reference count on a batch of folios.
* @folios: The folios.
* @refs: The number of refs to subtract from each folio.
*
* Like folio_put(), but for a batch of folios. This is more efficient
* than writing the loop yourself as it will optimise the locks which need
* to be taken if the folios are freed. The folios batch is returned
* empty and ready to be reused for another batch; there is no need
* to reinitialise it. If @refs is NULL, we subtract one from each
* folio refcount.
*
* Context: May be called in process or interrupt context, but not in NMI
* context. May be called while holding a spinlock.
*/
void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
{
int i, j;
struct lruvec *lruvec = NULL;
unsigned long flags = 0;
for (i = 0, j = 0; i < folios->nr; i++) {
struct folio *folio = folios->folios[i];
unsigned int nr_refs = refs ? refs[i] : 1;
if (is_huge_zero_folio(folio))
continue;
if (folio_is_zone_device(folio)) {
if (lruvec) {
lruvec_unlock_irqrestore(lruvec, flags);
lruvec = NULL;
}
if (folio_ref_sub_and_test(folio, nr_refs))
free_zone_device_folio(folio);
continue;
}
if (!folio_ref_sub_and_test(folio, nr_refs))
continue;
/* hugetlb has its own memcg */
if (folio_test_hugetlb(folio)) {
if (lruvec) {
lruvec_unlock_irqrestore(lruvec, flags);
lruvec = NULL;
}
free_huge_folio(folio);
continue;
}
folio_unqueue_deferred_split(folio);
__page_cache_release(folio, &lruvec, &flags);
if (j != i)
folios->folios[j] = folio;
j++;
}
if (lruvec)
lruvec_unlock_irqrestore(lruvec, flags);
if (!j) {
folio_batch_reinit(folios);
return;
}
folios->nr = j;
mem_cgroup_uncharge_folios(folios);
free_unref_folios(folios);
}
EXPORT_SYMBOL(folios_put_refs);
/**
* release_pages - batched put_page()
* @arg: array of pages to release
* @nr: number of pages
*
* Decrement the reference count on all the pages in @arg. If it
* fell to zero, remove the page from the LRU and free it.
*
* Note that the argument can be an array of pages, encoded pages,
* or folio pointers. We ignore any encoded bits, and turn any of
* them into just a folio that gets free'd.
*/
void release_pages(release_pages_arg arg, int nr)
{
struct folio_batch fbatch;
int refs[FOLIO_BATCH_SIZE];
struct encoded_page **encoded = arg.encoded_pages;
int i;
folio_batch_init(&fbatch);
for (i = 0; i < nr; i++) {
/* Turn any of the argument types into a folio */
struct folio *folio = page_folio(encoded_page_ptr(encoded[i]));
/* Is our next entry actually "nr_pages" -> "nr_refs" ? */
refs[fbatch.nr] = 1;
if (unlikely(encoded_page_flags(encoded[i]) &
ENCODED_PAGE_BIT_NR_PAGES_NEXT))
refs[fbatch.nr] = encoded_nr_pages(encoded[++i]);
if (folio_batch_add(&fbatch, folio) > 0)
continue;
folios_put_refs(&fbatch, refs);
}
if (fbatch.nr)
folios_put_refs(&fbatch, refs);
}
EXPORT_SYMBOL(release_pages);
/*
* The folios which we're about to release may be in the deferred lru-addition
* queues. That would prevent them from really being freed right now. That's
* OK from a correctness point of view but is inefficient - those folios may be
* cache-warm and we want to give them back to the page allocator ASAP.
*
* So __folio_batch_release() will drain those queues here.
* folio_batch_move_lru() calls folios_put() directly to avoid
* mutual recursion.
*/
void __folio_batch_release(struct folio_batch *fbatch)
{
if (!fbatch->percpu_pvec_drained) {
lru_add_drain();
fbatch->percpu_pvec_drained = true;
}
folios_put(fbatch);
}
EXPORT_SYMBOL(__folio_batch_release);
/**
* folio_batch_remove_exceptionals() - Prune non-folios from a batch.
* @fbatch: The batch to prune
*
* find_get_entries() fills a batch with both folios and shadow/swap/DAX
* entries. This function prunes all the non-folio entries from @fbatch
* without leaving holes, so that it can be passed on to folio-only batch
* operations.
*/
void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
{
unsigned int i, j;
for (i = 0, j = 0; i < folio_batch_count(fbatch); i++) {
struct folio *folio = fbatch->folios[i];
if (!xa_is_value(folio))
fbatch->folios[j++] = folio;
}
fbatch->nr = j;
}
#ifdef CONFIG_MEMCG
static void lruvec_reparent_lru(struct lruvec *child_lruvec,
struct lruvec *parent_lruvec,
enum lru_list lru, int nid)
{
int zid;
struct zone *zone;
if (lru != LRU_UNEVICTABLE)
list_splice_tail_init(&child_lruvec->lists[lru], &parent_lruvec->lists[lru]);
for_each_managed_zone_pgdat(zone, NODE_DATA(nid), zid, MAX_NR_ZONES - 1) {
unsigned long size = mem_cgroup_get_zone_lru_size(child_lruvec, lru, zid);
mem_cgroup_update_lru_size(parent_lruvec, lru, zid, size);
}
}
void lru_reparent_memcg(struct mem_cgroup *memcg, struct mem_cgroup *parent, int nid)
{
enum lru_list lru;
struct lruvec *child_lruvec, *parent_lruvec;
child_lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
parent_lruvec = mem_cgroup_lruvec(parent, NODE_DATA(nid));
parent_lruvec->anon_cost += child_lruvec->anon_cost;
parent_lruvec->file_cost += child_lruvec->file_cost;
for_each_lru(lru)
lruvec_reparent_lru(child_lruvec, parent_lruvec, lru, nid);
}
#endif
static const struct ctl_table swap_sysctl_table[] = {
{
.procname = "page-cluster",
.data = &page_cluster,
.maxlen = sizeof(int),
.mode = 0644,
.proc_handler = proc_dointvec_minmax,
.extra1 = SYSCTL_ZERO,
.extra2 = (void *)&page_cluster_max,
}
};
/*
* Perform any setup for the swap system
*/
void __init swap_setup(void)
{
unsigned long megs = PAGES_TO_MB(totalram_pages());
/* Use a smaller cluster for small-memory machines */
if (megs < 16)
page_cluster = 2;
else
page_cluster = 3;
/*
* Right now other parts of the system means that we
* _really_ don't want to cluster much more
*/
register_sysctl_init("vm", swap_sysctl_table);
}
]
|
| 45/2 |
2026/05/03 02:43 |
llm |
expert |
0m
Model:
Tokens:
input: 59379
output: 19
thoughts: 3062
Thoughts: **Analysis of Potential Data Race in `lru_add_drain_cpu` and `mlock_fbatch`**
Okay, here's the deal. I've been digging into this KCSAN report, specifically around the `lru_move_tail` in `lru_add_drain_cpu` and its interaction with `mlock_fbatch`. It's pointed out that the `data_race()` is used on `lru_move_tail` but not in `need_mlock_drain`. I need to figure out why KCSAN is flagging the absence of marking in `need_mlock_drain` and if it is, in fact, a problem.
First, I looked at `need_mlock_drain` which reads the `folio_batch_count` for `mlock_fbatch`. The concern is that `folio_batch_count` is reading `nr` which is written in `folio_batch_reinit`, and also updated by `folio_batch_add`. `need_mlock_drain` doesn't have any explicit synchronization. The fact that the call for `lru_move_tail` *does* use a `data_race()` call suggests that someone was at least aware of the possible race, even if it's considered harmless.
Then I checked that the `folio_batch_count` is also read for `lru_add` in `cpu_needs_drain`, which also has the same problem!
It seems the race is between the writer (`folio_batch_reinit` setting `nr = 0` or `folio_batch_add` incrementing `nr`) and the reader (`folio_batch_count` returning `nr`). The reader could either see a stale value (0 if it was reset, or a previous non-zero value), or see the current value.
Now, from the analysis, it's not immediately obvious that this will cause a crash. Seeing a stale value is probably not going to cause any severe problems because even if `folio_batch_count` sees a non-zero value and schedules a drain, the drain will *then* acquire a lock and re-check. Seeing a 0 when it should be non-zero will only cause a missed drain, which might be okay. It is potentially problematic if a global drain *needs* to happen *now* and misses a CPU. But KCSAN's report and the code suggest it's likely a BENIGN race.
The global drain (`lru_add_drain_all`) calls are *meant* to be a global drain, but it has a comment explaining that the caller is usually the one needing the drain. The caller is often the one that needs the visibility of their local operations. If it's the other way around, the normal synchronization is expected to happen via locks and barriers.
The current barrier in `lru_add_drain_all` is for the *caller*βs actions. It does *not* help with remote batches, like `mlock_fbatch`. So, if `lru_add_drain_all` is called, it might miss a page in a CPU's batch. It still should not be a serious problem unless the drain must happen right now.
The question then becomes: *should* we add the barrier? Maybe `mlock_folio` (and `need_mlock_drain`) needs a `READ_ONCE` or `WRITE_ONCE`?
Looking at it again, `lru_add_drain_all` *does* have a comment about needing to guarantee visibility for batch counts. But that is for the caller.
However, remote batches are a different story, since the caller is not directly adding to the remote batch.
On the race, the concern is that if CPU 1 is doing its mlocking, CPU 0 might call `lru_add_drain_all`. CPU 0 might see whatever is there, it might miss a page from CPU 1 that was recently added. If CPU 0 and CPU 1 coordinate, there would be synchronization. So the race is likely to cause a harmless, missed drain. But it's still correct.
So KCSAN is correct in reporting the race. We *likely* need to use `READ_ONCE` and `WRITE_ONCE`, or `data_race()` to mark the accesses.
But wait, if we change `folio_batch_reinit` and `folio_batch_add`, it might impact performance in many places. The access of `folio_batch` is on hot paths.
So, let's look at the specific race again. The writer is in `folios_put_refs`, and calls `folio_batch_reinit`. The reader is in `need_
|
| 46/2 |
2026/05/03 02:43 |
tool |
grepper |
0m
Args: map[Expression:mlock_fbatch]
Results: map[Output:mm/mlock.c-30-
mm/mlock.c:31:struct mlock_fbatch {
mm/mlock.c-32- local_lock_t lock;
--
mm/mlock.c-35-
mm/mlock.c:36:static DEFINE_PER_CPU(struct mlock_fbatch, mlock_fbatch) = {
mm/mlock.c-37- .lock = INIT_LOCAL_LOCK(lock),
--
mm/mlock.c=122=static struct lruvec *__munlock_folio(struct folio *folio, struct lruvec *lruvec)
--
mm/mlock.c-164-/*
mm/mlock.c:165: * Flags held in the low bits of a struct folio pointer on the mlock_fbatch.
mm/mlock.c-166- */
--
mm/mlock.c=212=void mlock_drain_local(void)
--
mm/mlock.c-215-
mm/mlock.c:216: local_lock(&mlock_fbatch.lock);
mm/mlock.c:217: fbatch = this_cpu_ptr(&mlock_fbatch.fbatch);
mm/mlock.c-218- if (folio_batch_count(fbatch))
mm/mlock.c-219- mlock_folio_batch(fbatch);
mm/mlock.c:220: local_unlock(&mlock_fbatch.lock);
mm/mlock.c-221-}
--
mm/mlock.c=223=void mlock_drain_remote(int cpu)
--
mm/mlock.c-227- WARN_ON_ONCE(cpu_online(cpu));
mm/mlock.c:228: fbatch = &per_cpu(mlock_fbatch.fbatch, cpu);
mm/mlock.c-229- if (folio_batch_count(fbatch))
--
mm/mlock.c=233=bool need_mlock_drain(int cpu)
mm/mlock.c-234-{
mm/mlock.c:235: return folio_batch_count(&per_cpu(mlock_fbatch.fbatch, cpu));
mm/mlock.c-236-}
--
mm/mlock.c=242=void mlock_folio(struct folio *folio)
--
mm/mlock.c-245-
mm/mlock.c:246: local_lock(&mlock_fbatch.lock);
mm/mlock.c:247: fbatch = this_cpu_ptr(&mlock_fbatch.fbatch);
mm/mlock.c-248-
--
mm/mlock.c-259- mlock_folio_batch(fbatch);
mm/mlock.c:260: local_unlock(&mlock_fbatch.lock);
mm/mlock.c-261-}
--
mm/mlock.c=267=void mlock_new_folio(struct folio *folio)
--
mm/mlock.c-271-
mm/mlock.c:272: local_lock(&mlock_fbatch.lock);
mm/mlock.c:273: fbatch = this_cpu_ptr(&mlock_fbatch.fbatch);
mm/mlock.c-274- folio_set_mlocked(folio);
--
mm/mlock.c-282- mlock_folio_batch(fbatch);
mm/mlock.c:283: local_unlock(&mlock_fbatch.lock);
mm/mlock.c-284-}
--
mm/mlock.c=290=void munlock_folio(struct folio *folio)
--
mm/mlock.c-293-
mm/mlock.c:294: local_lock(&mlock_fbatch.lock);
mm/mlock.c:295: fbatch = this_cpu_ptr(&mlock_fbatch.fbatch);
mm/mlock.c-296- /*
--
mm/mlock.c-303- mlock_folio_batch(fbatch);
mm/mlock.c:304: local_unlock(&mlock_fbatch.lock);
mm/mlock.c-305-}
]
|
| 47/2 |
2026/05/03 02:43 |
llm |
expert |
0m
Model:
Tokens:
input: 64607
output: 936
thoughts: 3932
Thoughts: **Analysis of Potential Data Race in Folio Batch Management**
Okay, so I'm looking at this code and trying to figure out if there's a real problem here. It seems like `mlock_drain_remote` does a lockless check of the folio batch count on a potentially offline CPU. Given that it's offline, I think we are safe there as there can't be concurrent writers.
The main concern is with `need_mlock_drain`. I need to figure out how `folio_batch_count` is used. I see that it's also used in `mm/swap.c` in `cpu_needs_drain` and there are a bunch of lockless reads of `nr` there. That makes me wonder if there's a wider issue.
I need to confirm if there is a risk of a value larger than `FOLIO_BATCH_SIZE` being read. However, `nr` is just an `unsigned char` and `FOLIO_BATCH_SIZE` is just 15, so that should be okay. The writer increments `nr` and the reader can see the values: 0, 1, ..., 15. The compiler would also need to reload `nr` from memory in `folio_batch_count`. So, this looks like a benign race that just needs to be properly marked.
Thinking about this from a memory ordering perspective, the "happens-before" relationship: `lru_add_drain_all` is meant to see all updates that happened before it was called. So if CPU 1 does `mlock_folio` and then CPU 0 calls `lru_add_drain_all()`, we need CPU 0 to see the update to `nr`. But there's no memory barrier on CPU 1 after the update to `nr`.
Is this a bug? It is only a problem if `lru_add_drain_all` has to be synchronous. But `lru_add_drain_all` is not generally synchronous with respect to other CPUs actions unless there's explicit synchronization. I need to be careful with `lru_cache_disable()` though, because that code expects it to be synchronous. `synchronize_rcu_expedited()` ensures that all CPUs have seen the `lru_disable_count` increment, but it doesn't ensure that the `nr` updates are visible.
If a CPU is in `mlock_folio`, preemption is disabled. `synchronize_rcu_expedited()` will wait for it to finish. But `preempt_enable` might not be a full memory barrier. However, `synchronize_rcu` involves barriers. So, if `mlock_folio` finished before `synchronize_rcu`, then its updates should be visible. If `mlock_folio` starts AFTER `lru_cache_disable` increments `lru_disable_count`, `mlock_folio` will see `lru_cache_disabled() == true` and call `mlock_folio_batch` immediately. But `lru_cache_disabled()` checks `lru_disable_count`. So, if the cache is disabled, it won't batch. The only problematic case is when it was added to the batch *just* before `lru_disable_count` was incremented, but `synchronize_rcu` should ensure visibility. The logic seems sound, just a missing annotation.
Then there's the `folios_put_refs` function. It looks like it does:
```c
folios->nr = j;
mem_cgroup_uncharge_folios(folios);
free_unref_folios(folios);
```
If `need_mlock_drain` sees `nr = j`, it schedules a drain. But the drain will wait for the `local_lock`. `folios_put_refs` is *not* holding the `local_lock`! But `mlock_folio_batch` calls `folios_put(fbatch)`. `folios_put` calls `folios_put_refs`. The drain function runs under the `local_lock` on the *same* CPU as `mlock_folio`. Okay, so `folios_put_refs` IS called under the lock when that particular batch is involved.
What about other callers of `folios_put_refs`? `release_pages` calls it, but it does *not* hold a `local_lock` on the same specific `folio_batch`. It uses a stack allocated batch `fbatch`. But `need_mlock_drain` reads from `per_cpu(mlock_fbatch.fbatch, cpu)`. It's not reading from a stack-allocated batch. That means the race is specifically on the
|
| 48/2 |
2026/05/03 02:44 |
llm |
expert |
0m
Model:
Tokens:
input: 56351
output: 19
thoughts: 192
Thoughts: **Benign Data Race Analysis and Reporting**
Okay, so I'm being prompted to remember to use the `set-results` tool, which is good β gotta keep the process moving. I've already dug into the KCSAN report and, after careful examination, I'm confident in my conclusion: this data race is completely BENIGN. It's a race between `folios_put_refs`, the writer, and `need_mlock_drain`, the reader, specifically on the `fbatch->nr` variable.
Now, the crucial part is the reader. It's a lockless check within `lru_add_drain_all`, basically seeing if a CPU needs to do some work. The writer, on the other hand, is correctly protected by a `local_lock`, so that's good. The readerβs value is used as a hint β a performance optimization, really. If it reads a stale value, the worst-case scenarios are pretty low impact: either the reader might schedule some redundant work (which won't find anything to do under the actual lock) or it could miss a drain entirely. Missing a drain is acceptable for this best-effort flush, since `nr` is an `unsigned char`, meaning no tearing.
So, benign, harmless, and no big deal. Time to report it. I'll get `set-results` called right away and then get the final response formulated.
|
| 49/2 |
2026/05/03 02:44 |
tool |
set-results |
0m
Args: map[Benign:true Confident:true]
Results: map[Benign:true Confident:true]
|
| 50/2 |
2026/05/03 02:44 |
llm |
expert |
0m
Model:
Tokens:
input: 56882
output: 846
thoughts: 22
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and presented as if it were my own internal monologue:
**Finalizing the Response: `set-results` and the Grand Finale**
Right, `set-results` has been invoked. That's the signal. The core processing is done; the intermediary stages are complete. Now, I need to focus on delivering the ultimate, distilled response as per the established protocol. This isn't just about regurgitating an answer, of course. It's about ensuring the information is in the correct format, completely integrated, and tailored to the context of the interaction. I need to make sure I'm adhering to all of the specified parameters for the final output. Think of it like finishing a complex mathematical proof. You've worked through the calculations, meticulously tested your variables, and now you must present the complete solution in a way that is both elegant and absolutely irrefutable. Time to go over my output one last time and deliver the goods.
|
| 51/1 |
2026/05/03 02:44 |
action |
format-explanation |
0m
Results: map[Explanation:The data race occurs between **`folios_put_refs()`** (the writer) and
**`need_mlock_drain()`** (the reader) on the `nr` field of a `struct
folio_batch`. This batch is part of the per-CPU `mlock_fbatch` structure used to
amortize the cost of `mlock` operations.
### Analysis
**1. Context:**
- **Writer (`mlock_folio` path):** Task 25650 is executing `mlock_folio()`,
which adds a folio to the per-CPU `mlock_fbatch`. If the batch becomes full or
if LRU caching is disabled, it calls `mlock_folio_batch()`, which eventually
calls `folios_put_refs()`. The writer holds the `mlock_fbatch.lock` (a
`local_lock`, which on non-RT kernels is a preemption-disable region).
- **Reader (`lru_add_drain_all` path):** Task 25645 is executing
`lru_add_drain_all()`, which iterates over all online CPUs and checks if their
per-CPU batches need draining. It calls `cpu_needs_drain()`, which in turn calls
`need_mlock_drain()`. This check is performed **locklessly** on remote CPUs to
avoid the overhead of acquiring locks for every CPU.
**2. Data Flow:**
- The reader `need_mlock_drain()` reads `fbatch->nr` to decide whether to
schedule a `lru_add_drain_per_cpu` work item on that CPU.
- If it sees `nr > 0`, it schedules the work. The work item, when it runs on the
target CPU, will acquire the `mlock_fbatch.lock` and re-check `nr` before
processing the batch.
- If it sees `nr == 0` (stale or new), it simply skips that CPU for the current
draining round.
**3. Safety:**
- **Torn Accesses:** The `nr` field is an `unsigned char` (1 byte). Torn
reads/writes are impossible for single-byte accesses on all supported
architectures.
- **Logic Soundness:**
- If the reader sees a non-zero value while the writer is already clearing
the batch (setting `nr = 0`), it will schedule redundant work. This work
will safely see `nr == 0` after acquiring the lock and do nothing.
- If the reader misses a non-zero value because it sees a stale `0`, it
might delay the drain. However, `lru_add_drain_all()` is generally a
best-effort flush. In cases where strict synchronization is required (e.g.,
`lru_cache_disable()`), other mechanisms like `synchronize_rcu()` ensure
that preceding batch updates are visible.
- The `local_lock` (preemption disable) on the writer side ensures that the
scheduled work item cannot start on the same CPU until the writer has
finished its critical section and released the lock, ensuring the work sees
a consistent state.
### Final Classification: **BENIGN**
**Reasoning:**
The race is a classic "lockless-check-before-lock" optimization. The read is
used as a heuristic hint to avoid unnecessary work scheduling. Correctness is
guaranteed because the actual processing of the batch is performed under the
`mlock_fbatch.lock`. The variable is a single byte, precluding torn accesses.
**Recommended Annotations:**
The lockless read in `need_mlock_drain()` should be wrapped in `data_race()` or
`READ_ONCE()`.
```c
// mm/mlock.c
bool need_mlock_drain(int cpu)
{
return data_race(folio_batch_count(&per_cpu(mlock_fbatch.fbatch, cpu)));
}
```]
|