| 0/0 |
2026/02/06 07:03 |
flow |
assessment-kcsan |
10m
Results: map[Benign:false Confident:true Explanation:The data race occurs between `__perf_event_read_value` (reader) and `perf_event_set_state` (writer, specifically via `perf_event_update_time` and `__perf_update_times`).
### Analysis
The fields involved in the race are `total_time_enabled` and `total_time_running` in `struct perf_event`. These are plain `u64` fields.
**Writer Side:**
The writer is `event_sched_out`, which is called during a task context switch (`__schedule`). The sequence of operations in `event_sched_out` is:
1. `event->oncpu = -1;`
2. `perf_event_set_state(event, PERF_EVENT_STATE_INACTIVE);`
- This calls `perf_event_update_time(event)`, which updates `total_time_enabled` and `total_time_running`.
- Then it sets `event->state = PERF_EVENT_STATE_INACTIVE;`.
**Reader Side:**
The reader is `__perf_event_read_value`, called from a `read()` syscall (in this case via `io_uring`). It performs the following:
1. Calls `perf_event_read(event, false);`.
2. Reads `event->total_time_enabled` and `event->total_time_running`.
The `perf_event_read` function is supposed to synchronize with the CPU where the event is active. However, it has a synchronization hole:
```c
4827: if (state == PERF_EVENT_STATE_ACTIVE) {
...
4838: event_cpu = READ_ONCE(event->oncpu);
4839: if ((unsigned)event_cpu >= nr_cpu_ids)
4840: return 0;
```
If `perf_event_read` sees `state == PERF_EVENT_STATE_ACTIVE` but `event->oncpu == -1`, it returns `0` without performing any synchronization (like sending an IPI or taking a lock). This exact condition occurs during `event_sched_out` because `oncpu` is set to `-1` *before* the times are updated and before the state is changed to `INACTIVE`.
### Harmfulness
This data race is **harmful** for several reasons:
1. **Tearing on 32-bit systems:** Since `total_time_enabled` and `total_time_running` are plain `u64` variables, 32-bit systems will perform the read and write in two 32-bit chunks. A concurrent read and write can result in the reader seeing a "torn" value (e.g., high 32 bits from the old value and low 32 bits from the new value), leading to completely incorrect timing data.
2. **Inconsistency on 64-bit systems:** Even on 64-bit systems where 8-byte aligned accesses are atomic, the reader can see an inconsistent state. For example, it might read an updated `count` (from a previous step in `perf_event_read`) but an old `total_time_enabled`. This inconsistency leads to incorrect scaling of performance counters, as the ratio of `count` to `enabled_time` will be wrong.
3. **Lack of Synchronization:** The race highlights a bug in the `perf_event_read` logic where it fails to properly synchronize with an event that is in the process of being scheduled out.
Given that these values are used for critical performance measurements and the race can lead to memory corruption (in the form of garbage data) on 32-bit systems, it is not a benign race.]
|
| 1/1 |
2026/02/06 07:03 |
action |
kernel-checkouter |
0m
Results: map[KernelSrc:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/src/e33557cda8a98db97b344f14955a278b02136c34]
|
| 2/1 |
2026/02/06 07:04 |
action |
kernel-builder |
3m
Results: map[KernelObj:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/build/2d2949499f4390f6d1c9f5b049dddf111913cdc1]
|
| 3/1 |
2026/02/06 07:08 |
action |
codesearch-prepare |
2m
Results: map[Index:codesearch-index]
|
| 4/1 |
2026/02/06 07:11 |
agent |
expert |
3m
Model:
Results: map[Benign:false Confident:true]
Instruction:
You are an experienced Linux kernel developer tasked with determining if the given kernel
data race is benign or not. The data race report is from KCSAN tool.
It contains 2 stack traces of the memory accesses that constitute a data race.
A "benign" data races are on a simple int/bool variable or similar field,
and the accesses are not supposed to be protected by any mutual exclusion primitives.
Common examples of such "benign" data races are accesses to various flags fields,
statistics counters, and similar. A "benign" data race does not lead to memory corruption/crash
with a conservative compiler that compiles memory accesses to primitive types
effectively as atomic.
A non-benign (or "harmful" data race) can lead to corruption/crash even with
a conservative compiler that compiles memory accesses to primitive types
effectively as atomic. A common example of a "harmful" data races is race on
a complex container (list/hashmap/etc), where accesses are supposed to be protected
by a mutual exclusion primitive.
Also consider races that happen at the same time with the given one.
If there is no synchronization in between, other memory accesses in the involved threads
race with each other if they access the same memory. For example, if both threads execute:
some_struct->have_elements = true;
list_add(new_node, &some_struct->list_head);
the race on some_struct->have_elements may appear benign, however it also implies there
is a race on some_struct->list_head which is not benign, since the list is not thread-safe.
Take into account that on 32-bit systems 64-bit memory accesses may be split into two accesses,
and thus even with a conservative compiler may not be fully atomic. However, such races may
still be benign depending on how writes are done, and how read data is used.
In the final reply explain why you think the given data race is benign or is harmful.
Use the provided tools to confirm any assumptions, variables/fields being accessed, etc.
In particular, don't make assumptions about the kernel source code,
use codesearch tools to read the actual source code.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt:
The data race report is:
==================================================================
BUG: KCSAN: data-race in __perf_event_read_value / perf_event_set_state
write to 0xffff8881311a8538 of 8 bytes by task 13031 on cpu 1:
__perf_update_times kernel/events/core.c:-1 [inline]
perf_event_update_time kernel/events/core.c:735 [inline]
perf_event_set_state+0x1c4/0x440 kernel/events/core.c:754
event_sched_out+0x2d4/0x4d0 kernel/events/core.c:2391
group_sched_out kernel/events/core.c:2415 [inline]
__pmu_ctx_sched_out+0x3e7/0x530 kernel/events/core.c:3458
ctx_sched_out+0x273/0x2d0 kernel/events/core.c:3539
task_ctx_sched_out+0x4d/0x70 kernel/events/core.c:2859
perf_event_context_sched_out kernel/events/core.c:3746 [inline]
__perf_event_task_sched_out+0x2df/0x3c0 kernel/events/core.c:3846
perf_event_task_sched_out include/linux/perf_event.h:1654 [inline]
prepare_task_switch kernel/sched/core.c:5049 [inline]
context_switch kernel/sched/core.c:5205 [inline]
__schedule+0xbaa/0xc90 kernel/sched/core.c:6867
__schedule_loop kernel/sched/core.c:6949 [inline]
schedule+0x5e/0xd0 kernel/sched/core.c:6964
futex_do_wait kernel/futex/waitwake.c:358 [inline]
__futex_wait+0x121/0x260 kernel/futex/waitwake.c:687
futex_wait+0x9c/0x1d0 kernel/futex/waitwake.c:715
do_futex+0x2bf/0x380 kernel/futex/syscalls.c:130
__do_sys_futex kernel/futex/syscalls.c:207 [inline]
__se_sys_futex+0x2f6/0x370 kernel/futex/syscalls.c:188
__x64_sys_futex+0x78/0x90 kernel/futex/syscalls.c:188
x64_sys_call+0x2bc2/0x3000 arch/x86/include/generated/asm/syscalls_64.h:203
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0xc0/0x2a0 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
read to 0xffff8881311a8538 of 8 bytes by task 13182 on cpu 0:
__perf_event_read_value+0xb6/0x1d0 kernel/events/core.c:5894
perf_read_one kernel/events/core.c:6049 [inline]
__perf_read kernel/events/core.c:6102 [inline]
perf_read+0x171/0x4f0 kernel/events/core.c:6119
loop_rw_iter+0x2c6/0x3f0 include/linux/uio.h:-1
io_iter_do_read io_uring/rw.c:836 [inline]
__io_read+0xbf6/0xc50 io_uring/rw.c:950
io_read+0x4a/0x190 io_uring/rw.c:1030
__io_issue_sqe+0xfd/0x2d0 io_uring/io_uring.c:1793
io_issue_sqe+0x56/0xa70 io_uring/io_uring.c:1816
io_wq_submit_work+0x3f7/0x5f0 io_uring/io_uring.c:1928
io_worker_handle_work+0x41e/0x950 io_uring/io-wq.c:650
io_wq_worker+0x22d/0x860 io_uring/io-wq.c:704
ret_from_fork+0x148/0x280 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
value changed: 0x000000004a891b66 -> 0x000000004a89847d
Reported by Kernel Concurrency Sanitizer on:
CPU: 0 UID: 0 PID: 13182 Comm: iou-wrk-13031 Not tainted syzkaller #0 PREEMPT(voluntary)
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/25/2025
==================================================================
Reply: The data race occurs between `__perf_event_read_value` (reader) and `perf_event_set_state` (writer, specifically via `perf_event_update_time` and `__perf_update_times`).
### Analysis
The fields involved in the race are `total_time_enabled` and `total_time_running` in `struct perf_event`. These are plain `u64` fields.
**Writer Side:**
The writer is `event_sched_out`, which is called during a task context switch (`__schedule`). The sequence of operations in `event_sched_out` is:
1. `event->oncpu = -1;`
2. `perf_event_set_state(event, PERF_EVENT_STATE_INACTIVE);`
- This calls `perf_event_update_time(event)`, which updates `total_time_enabled` and `total_time_running`.
- Then it sets `event->state = PERF_EVENT_STATE_INACTIVE;`.
**Reader Side:**
The reader is `__perf_event_read_value`, called from a `read()` syscall (in this case via `io_uring`). It performs the following:
1. Calls `perf_event_read(event, false);`.
2. Reads `event->total_time_enabled` and `event->total_time_running`.
The `perf_event_read` function is supposed to synchronize with the CPU where the event is active. However, it has a synchronization hole:
```c
4827: if (state == PERF_EVENT_STATE_ACTIVE) {
...
4838: event_cpu = READ_ONCE(event->oncpu);
4839: if ((unsigned)event_cpu >= nr_cpu_ids)
4840: return 0;
```
If `perf_event_read` sees `state == PERF_EVENT_STATE_ACTIVE` but `event->oncpu == -1`, it returns `0` without performing any synchronization (like sending an IPI or taking a lock). This exact condition occurs during `event_sched_out` because `oncpu` is set to `-1` *before* the times are updated and before the state is changed to `INACTIVE`.
### Harmfulness
This data race is **harmful** for several reasons:
1. **Tearing on 32-bit systems:** Since `total_time_enabled` and `total_time_running` are plain `u64` variables, 32-bit systems will perform the read and write in two 32-bit chunks. A concurrent read and write can result in the reader seeing a "torn" value (e.g., high 32 bits from the old value and low 32 bits from the new value), leading to completely incorrect timing data.
2. **Inconsistency on 64-bit systems:** Even on 64-bit systems where 8-byte aligned accesses are atomic, the reader can see an inconsistent state. For example, it might read an updated `count` (from a previous step in `perf_event_read`) but an old `total_time_enabled`. This inconsistency leads to incorrect scaling of performance counters, as the ratio of `count` to `enabled_time` will be wrong.
3. **Lack of Synchronization:** The race highlights a bug in the `perf_event_read` logic where it fails to properly synchronize with an event that is in the process of being scheduled out.
Given that these values are used for critical performance measurements and the race can lead to memory corruption (in the form of garbage data) on 32-bit systems, it is not a benign race.
|
| 5/2 |
2026/02/06 07:11 |
llm |
expert |
0m
Model:
Tokens:
input: 4471
output: 86
thoughts: 218
Thoughts: **Analyzing a Data Race in the Perf Event System**
Okay, so the report flags a nasty data race β a concurrent read and write conflict β between `__perf_event_read_value` and `perf_event_set_state`, specifically in the `__perf_update_times` function. The address in question is `0xffff8881311a8538`, and the access is 8 bytes wide for both the read and the write. Crucially, the value at this location went from `0x000000004a891b66` to `0x000000004a89847d`. This demands a deep dive.
The first thing to do, is to understand what data field is the cause of the race. The write happens within `__perf_update_times`, which in turn is called from `perf_event_set_state`. The read comes from `__perf_event_read_value`. That helps narrow the scope considerably.
My plan of attack is clear: I'll now head straight into the kernel source, focusing on `kernel/events/core.c`. I'm going to start by scrutinizing the implementations of `__perf_update_times` and `__perf_event_read_value`. I'm looking for the shared data and the locking mechanisms (or lack thereof) that govern access to that memory location. I expect the problem lies in the synchronization, or perhaps some subtle ordering issue. I need to understand exactly what's being updated in `__perf_update_times` and why it's also being read in `__perf_event_read_value` at the same time. The value change indicates this is likely a timing-sensitive bug.
|
| 6/2 |
2026/02/06 07:11 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:__perf_update_times]
Results: map[SourceCode: 716: static __always_inline void
717: __perf_update_times(struct perf_event *event, u64 now, u64 *enabled, u64 *running)
718: {
719: enum perf_event_state state = __perf_effective_state(event);
720: u64 delta = now - event->tstamp;
721:
722: *enabled = event->total_time_enabled;
723: if (state >= PERF_EVENT_STATE_INACTIVE)
724: *enabled += delta;
725:
726: *running = event->total_time_running;
727: if (state >= PERF_EVENT_STATE_ACTIVE)
728: *running += delta;
729: }
SourceFile:kernel/events/core.c]
|
| 7/2 |
2026/02/06 07:11 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:__perf_event_read_value]
Results: map[SourceCode:5879: static u64 __perf_event_read_value(struct perf_event *event, u64 *enabled, u64 *running)
5880: {
5881: struct perf_event *child;
5882: u64 total = 0;
5883:
5884: *enabled = 0;
5885: *running = 0;
5886:
5887: mutex_lock(&event->child_mutex);
5888:
5889: (void)perf_event_read(event, false);
5890: total += perf_event_count(event, false);
5891:
5892: *enabled += event->total_time_enabled +
5893: atomic64_read(&event->child_total_time_enabled);
5894: *running += event->total_time_running +
5895: atomic64_read(&event->child_total_time_running);
5896:
5897: list_for_each_entry(child, &event->child_list, child_list) {
5898: (void)perf_event_read(child, false);
5899: total += perf_event_count(child, false);
5900: *enabled += child->total_time_enabled;
5901: *running += child->total_time_running;
5902: }
5903: mutex_unlock(&event->child_mutex);
5904:
5905: return total;
5906: }
SourceFile:kernel/events/core.c]
|
| 8/2 |
2026/02/06 07:11 |
llm |
expert |
0m
Model:
Tokens:
input: 5872
output: 86
thoughts: 313
Thoughts: **Analyzing Potential Concurrency Issues in Perf Event Time Tracking**
Okay, let's break this down. In `__perf_event_read_value`, specifically line 5894, I see that we're adding to `*running` using `event->total_time_running` and `event->child_total_time_running`. The first read is the main interest here, `event->total_time_running`. I'm concerned about race conditions.
Now, looking at `__perf_update_times` at line 726, something jumps out. It *reads* `event->total_time_running` and updates the pointer `running`. But, it doesn't *write* to `event->total_time_running` itself! This implies that `event->total_time_running` is updated elsewhere. This is key, so I need to understand *where* and *how* it's being updated to ensure this read isn't stale.
Ah, `perf_event_set_state` calls `perf_event_update_time`, which in turn calls `__perf_update_times`. That's the chain I need to trace.
Wait, here's some helpful information from the report: The write to address `0xffff8881311a8538` (8 bytes) is done by task 13031 on CPU 1. This write occurs within `__perf_update_times` which is inlined, and is called by `perf_event_update_time` and subsequently from `perf_event_set_state`. So this confirms the write, and its calling context.
I need to dive into `perf_event_update_time` and `perf_event_set_state` immediately to understand the context of the write and to see if the timing is correct relative to the read in `__perf_event_read_value`. It's likely that a write to `event->total_time_running` is happening in `perf_event_set_state` or a function it calls. I need to figure out the concurrency pattern here and confirm that there's proper synchronization to prevent data corruption. This seems like a potential race condition waiting to happen.
|
| 9/2 |
2026/02/06 07:11 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:perf_event_update_time]
Results: map[SourceCode: 731: static void perf_event_update_time(struct perf_event *event)
732: {
733: u64 now = perf_event_time(event);
734:
735: __perf_update_times(event, now, &event->total_time_enabled,
736: &event->total_time_running);
737: event->tstamp = now;
738: }
SourceFile:kernel/events/core.c]
|
| 10/2 |
2026/02/06 07:11 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:perf_event_set_state]
Results: map[SourceCode: 748: static void
749: perf_event_set_state(struct perf_event *event, enum perf_event_state state)
750: {
751: if (event->state == state)
752: return;
753:
754: perf_event_update_time(event);
755: /*
756: * If a group leader gets enabled/disabled all its siblings
757: * are affected too.
758: */
759: if ((event->state < 0) ^ (state < 0))
760: perf_event_update_sibling_time(event);
761:
762: WRITE_ONCE(event->state, state);
763: }
SourceFile:kernel/events/core.c]
|
| 11/2 |
2026/02/06 07:11 |
llm |
expert |
0m
Model:
Tokens:
input: 7156
output: 34
thoughts: 569
Thoughts: **Analyzing Perf Event Time Updates**
Okay, so I'm looking at how performance event time is updated. Specifically, `perf_event_update_time` calls `__perf_update_times`, passing the addresses of `event->total_time_enabled` and `event->total_time_running`. I need to understand what's happening to these members.
Let's break down `__perf_update_times`. I see it's setting `*enabled` to `event->total_time_enabled` initially. Then, if the event's `state` is at least `PERF_EVENT_STATE_INACTIVE`, it adds `delta` to `*enabled`. The same logic is applied to `*running` and `event->total_time_running` with `PERF_EVENT_STATE_ACTIVE`.
Hang on, because `__perf_update_times` is `__always_inline`, meaning it gets inlined directly into the caller, `perf_event_update_time`. If the compiler inlines that, the assignments become:
`event->total_time_enabled = event->total_time_enabled;` (with an optional increment with `delta`)
`event->total_time_running = event->total_time_running;` (with an optional increment with `delta`)
So, the write operations are directly to the `event->total_time_enabled` or `event->total_time_running` members of the `perf_event` struct. The crash report shows a write of 8 bytes to an address and a read from the same address, which corresponds to the address of `total_time_enabled`.
I also see that `__perf_event_read_value` reads the values of `total_time_enabled` and `total_time_running` from `perf_event`.
Now I should probably look up the exact offsets of those fields within `struct perf_event` to verify the memory access.
|
| 12/2 |
2026/02/06 07:11 |
tool |
codesearch-struct-layout |
0m
Args: map[ContextFile:kernel/events/core.c Name:perf_event]
Results: map[Fields:[map[Name:event_entry OffsetBits:0 SizeBits:128] map[Name:sibling_list OffsetBits:128 SizeBits:128] map[Name:active_list OffsetBits:256 SizeBits:128] map[Name:group_node OffsetBits:384 SizeBits:192] map[Name:group_index OffsetBits:576 SizeBits:64] map[Name:migrate_entry OffsetBits:640 SizeBits:128] map[Name:hlist_entry OffsetBits:768 SizeBits:128] map[Name:active_entry OffsetBits:896 SizeBits:128] map[Name:nr_siblings OffsetBits:1024 SizeBits:32] map[Name:event_caps OffsetBits:1056 SizeBits:32] map[Name:group_caps OffsetBits:1088 SizeBits:32] map[Name:group_generation OffsetBits:1120 SizeBits:32] map[Name:group_leader OffsetBits:1152 SizeBits:64] map[Name:pmu OffsetBits:1216 SizeBits:64] map[Name:pmu_private OffsetBits:1280 SizeBits:64] map[Name:state OffsetBits:1344 SizeBits:32] map[Name:attach_state OffsetBits:1376 SizeBits:32] map[Name:count OffsetBits:1408 SizeBits:64] map[Name:child_count OffsetBits:1472 SizeBits:64] map[Name:total_time_enabled OffsetBits:1536 SizeBits:64] map[Name:total_time_running OffsetBits:1600 SizeBits:64] map[Name:tstamp OffsetBits:1664 SizeBits:64] map[Name:attr OffsetBits:1728 SizeBits:1152] map[Name:header_size OffsetBits:2880 SizeBits:16] map[Name:id_header_size OffsetBits:2896 SizeBits:16] map[Name:read_size OffsetBits:2912 SizeBits:16] map[Name:hw OffsetBits:2944 SizeBits:1664] map[Name:ctx OffsetBits:4608 SizeBits:64] map[Name:pmu_ctx OffsetBits:4672 SizeBits:64] map[Name:refcount OffsetBits:4736 SizeBits:64] map[Name:child_total_time_enabled OffsetBits:4800 SizeBits:64] map[Name:child_total_time_running OffsetBits:4864 SizeBits:64] map[Name:child_mutex OffsetBits:4928 SizeBits:256] map[Name:child_list OffsetBits:5184 SizeBits:128] map[Name:parent OffsetBits:5312 SizeBits:64] map[Name:oncpu OffsetBits:5376 SizeBits:32] map[Name:cpu OffsetBits:5408 SizeBits:32] map[Name:owner_entry OffsetBits:5440 SizeBits:128] map[Name:owner OffsetBits:5568 SizeBits:64] map[Name:mmap_mutex OffsetBits:5632 SizeBits:256] map[Name:mmap_count OffsetBits:5888 SizeBits:32] map[Name:rb OffsetBits:5952 SizeBits:64] map[Name:rb_entry OffsetBits:6016 SizeBits:128] map[Name:rcu_batches OffsetBits:6144 SizeBits:64] map[Name:rcu_pending OffsetBits:6208 SizeBits:32] map[Name:waitq OffsetBits:6272 SizeBits:192] map[Name:fasync OffsetBits:6464 SizeBits:64] map[Name:pending_wakeup OffsetBits:6528 SizeBits:32] map[Name:pending_kill OffsetBits:6560 SizeBits:32] map[Name:pending_disable OffsetBits:6592 SizeBits:32] map[Name:pending_addr OffsetBits:6656 SizeBits:64] map[Name:pending_irq OffsetBits:6720 SizeBits:256] map[Name:pending_disable_irq OffsetBits:6976 SizeBits:256] map[Name:pending_task OffsetBits:7232 SizeBits:128] map[Name:pending_work OffsetBits:7360 SizeBits:32] map[Name:event_limit OffsetBits:7392 SizeBits:32] map[Name:addr_filters OffsetBits:7424 SizeBits:192] map[Name:addr_filter_ranges OffsetBits:7616 SizeBits:64] map[Name:addr_filters_gen OffsetBits:7680 SizeBits:64] map[Name:aux_event OffsetBits:7744 SizeBits:64] map[Name:destroy OffsetBits:7808 SizeBits:64] map[Name:callback_head OffsetBits:7872 SizeBits:128] map[Name:ns OffsetBits:8000 SizeBits:64] map[Name:id OffsetBits:8064 SizeBits:64] map[Name:lost_samples OffsetBits:8128 SizeBits:64] map[Name:clock OffsetBits:8192 SizeBits:64] map[Name:overflow_handler OffsetBits:8256 SizeBits:64] map[Name:overflow_handler_context OffsetBits:8320 SizeBits:64] map[Name:prog OffsetBits:8384 SizeBits:64] map[Name:bpf_cookie OffsetBits:8448 SizeBits:64] map[Name:tp_event OffsetBits:8512 SizeBits:64] map[Name:filter OffsetBits:8576 SizeBits:64] map[Name:cgrp OffsetBits:8640 SizeBits:64] map[Name:security OffsetBits:8704 SizeBits:64] map[Name:sb_list OffsetBits:8768 SizeBits:128] map[Name:pmu_list OffsetBits:8896 SizeBits:128] map[Name:orig_type OffsetBits:9024 SizeBits:32]]]
|
| 13/2 |
2026/02/06 07:11 |
llm |
expert |
0m
Model:
Tokens:
input: 9900
output: 39
thoughts: 1267
Thoughts: **Analyzing a Potential Race Condition in Perf Event Time Tracking**
Okay, so I'm looking at this report showing a write to an 8-byte value by task 13031 on CPU 1, specifically related to `__perf_update_times` within the kernel's event handling. The addresses mentioned (offsets 192 and 200 bytes for `total_time_enabled` and `total_time_running` respectively, within the `perf_event` struct) suggest these are the targets. The report specifies an address, 0xffff8881311a8538, and if I had the base address, I could identify the event structure.
The core of the issue seems to be in `__perf_update_times`, specifically this section:
```c
722: *enabled = event->total_time_enabled;
723: if (state >= PERF_EVENT_STATE_INACTIVE)
724: *enabled += delta;
```
Line 722 is initially setting the *enabled value to the total_time_enabled, and then line 724 might add delta to it. However, the `*enabled = event->total_time_enabled;` line is a write of the current value of enabled back to itself. Then delta is added.
I also see that `__perf_event_read_value` is involved in reading these values:
```c
5892: *enabled += event->total_time_enabled +
5893: atomic64_read(&event->child_total_time_enabled);
5894: *running += event->total_time_running +
5895: atomic64_read(&event->child_total_time_running);
```
So, there's a potential race on either `total_time_enabled` or `total_time_running`. The writer is `perf_event_set_state` (called during context switches via `__perf_event_task_sched_out`), and the reader is `__perf_event_read_value` which is used in `perf_read` called through `io_uring`.
`perf_event_set_state` is triggered when the event's state changes. In this context it appears during an `event_sched_out`.
```c
static void
event_sched_out(struct perf_event *event,
struct perf_event_context *ctx)
{
...
perf_event_set_state(event, PERF_EVENT_STATE_INACTIVE);
...
}
```
This then calls `perf_event_update_time(event)`, which updates the problematic fields. The reader function `__perf_event_read_value` is reading these values, potentially during this update.
The important question now is, is this race actually harmful? The variables, being `u64`, suggest that 8-byte aligned accesses *should* be atomic on a 64-bit system. The values did change between 0x000000004a891b66 and 0x000000004a89847d, which *looks* like a normal update, perhaps a timestamp increment.
However, `__perf_update_times` contains the following potential issue:
```c
722: *enabled = event->total_time_enabled;
723: if (state >= PERF_EVENT_STATE_INACTIVE)
724: *enabled += delta;
```
This isn't *guaranteed* to be atomic if the compiler optimizes it into multiple steps, effectively becoming a read, modify, write operation. It might be interpreted as `*enabled = event->total_time_enabled + delta;`. Since `*enabled` is `&event->total_time_enabled`, the code could be translated into:
1. `event->total_time_enabled = event->total_time_enabled;`
2. `event->total_time_enabled += delta;`
The first write is seemingly redundant, unless the state is not inactive. If `state < PERF_EVENT_STATE_INACTIVE`, then the values remain the same.
The compiler *could* generate this:
1. Load `event->total_time_enabled` into a register.
2. Store the register value back to `event->total_time_enabled`.
3. Conditional add 'delta' to the register if `state >= ...`.
4.
|
| 14/2 |
2026/02/06 07:11 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:perf_read]
Results: map[SourceCode:6107: static ssize_t
6108: perf_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
6109: {
6110: struct perf_event *event = file->private_data;
6111: struct perf_event_context *ctx;
6112: int ret;
6113:
6114: ret = security_perf_event_read(event);
6115: if (ret)
6116: return ret;
6117:
6118: ctx = perf_event_ctx_lock(event);
6119: ret = __perf_read(event, buf, count);
6120: perf_event_ctx_unlock(event, ctx);
6121:
6122: return ret;
6123: }
SourceFile:kernel/events/core.c]
|
| 15/2 |
2026/02/06 07:11 |
llm |
expert |
0m
Model:
Tokens:
input: 12498
output: 41
thoughts: 389
Thoughts: **Analyzing a Potential Race Condition in Perf Event Handling**
Okay, so here's what's running through my head as I examine this. `perf_read` starts by calling `perf_event_ctx_lock(event)`, which in turn locks the context. Seems straightforward enough. The problem, I'm thinking, lies in `event_sched_out`, which is triggered during a context switch. Let's trace it back: `__schedule` -> `context_switch` -> `prepare_task_switch` -> `perf_event_task_sched_out` and so on, all the way down to `__perf_update_times`. This whole chain happens on the CPU where the task is being switched out. And if I'm understanding this correctly, the `perf_event_context` being dealt with is *likely* the context of that very task.
Now, the potential for racing is what's catching my attention. If `perf_read` gets called from *another* task, or even the same task via something like an io_uring worker, we could have a nasty concurrency issue. The report mentions the reader being `iou-wrk-13031` and the writer being task `13031`. Ah, so task 13031 is being scheduled out, and its io_uring worker is attempting to read its perf event. *That* makes it much clearer.
`perf_event_ctx_lock` specifically locks `ctx->mutex`. The question now is: does `event_sched_out` hold *that* `ctx->mutex` during context switch? Because that's the crucial point. It's called with interrupts disabled, and the context's spinlock *might* be held, but not necessarily the mutex, right?
Wait a minute, let me re-examine the context: `ctx_sched_out` and its associated functions are called with the context spinlock held. But, does that mean anything regarding the mutex? I need to double-check the `perf_event_context` structure and its locking mechanisms. I need to be absolutely sure what locks are held where. Let's see...
|
| 16/2 |
2026/02/06 07:11 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:perf_event_context]
Results: map[SourceCode:1002: /**
1003: * struct perf_event_context - event context structure
1004: *
1005: * Used as a container for task events and CPU events as well:
1006: */
1007: struct perf_event_context {
1008: /*
1009: * Protect the states of the events in the list,
1010: * nr_active, and the list:
1011: */
1012: raw_spinlock_t lock;
1013: /*
1014: * Protect the list of events. Locking either mutex or lock
1015: * is sufficient to ensure the list doesn't change; to change
1016: * the list you need to lock both the mutex and the spinlock.
1017: */
1018: struct mutex mutex;
1019:
1020: struct list_head pmu_ctx_list;
1021: struct perf_event_groups pinned_groups;
1022: struct perf_event_groups flexible_groups;
1023: struct list_head event_list;
1024:
1025: int nr_events;
1026: int nr_user;
1027: int is_active;
1028:
1029: int nr_stat;
1030: int nr_freq;
1031: int rotate_disable;
1032:
1033: refcount_t refcount; /* event <-> ctx */
1034: struct task_struct *task;
1035:
1036: /*
1037: * Context clock, runs when context enabled.
1038: */
1039: u64 time;
1040: u64 timestamp;
1041: u64 timeoffset;
1042:
1043: /*
1044: * These fields let us detect when two contexts have both
1045: * been cloned (inherited) from a common ancestor.
1046: */
1047: struct perf_event_context *parent_ctx;
1048: u64 parent_gen;
1049: u64 generation;
1050: int pin_count;
1051: #ifdef CONFIG_CGROUP_PERF
1052: int nr_cgroups; /* cgroup evts */
1053: #endif
1054: struct rcu_head rcu_head;
1055:
1056: /*
1057: * The count of events for which using the switch-out fast path
1058: * should be avoided.
1059: *
1060: * Sum (event->pending_work + events with
1061: * (attr->inherit && (attr->sample_type & PERF_SAMPLE_READ)))
1062: *
1063: * The SIGTRAP is targeted at ctx->task, as such it won't do changing
1064: * that until the signal is delivered.
1065: */
1066: local_t nr_no_switch_fast;
1067: };
SourceFile:include/linux/perf_event.h]
|
| 17/2 |
2026/02/06 07:11 |
llm |
expert |
0m
Model:
Tokens:
input: 14321
output: 43
thoughts: 80
Thoughts: Here's my summary, as I understand the situation:
**Potential Concurrency Issue in Performance Event Handling**
Okay, so I'm looking at how `perf_read` interacts with context switches. It seems like `perf_read` calls `perf_event_ctx_lock(event)`, and my initial thought was that this function was just locking `ctx->mutex`. But then I recalled `event_sched_out`, which happens during a context switch, holds `ctx->lock`, which is the raw spinlock. This is where things get interesting and raises a question, right?
Wait, hold on a second... is `perf_event_ctx_lock` doing more than I initially thought? Does it *also* disable interrupts and take the spinlock itself? I need to verify that. My understanding of the locking primitives here is critical, so let's dig into that function's implementation right now to be absolutely sure.
|
| 18/2 |
2026/02/06 07:12 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:perf_event_ctx_lock]
Results: map[SourceCode:1390: static inline struct perf_event_context *
1391: perf_event_ctx_lock(struct perf_event *event)
1392: {
1393: return perf_event_ctx_lock_nested(event, 0);
1394: }
SourceFile:kernel/events/core.c]
|
| 19/2 |
2026/02/06 07:12 |
llm |
expert |
0m
Model:
Tokens:
input: 14748
output: 45
thoughts: 12
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:
**Delving into `perf_event_ctx_lock_nested`**
Alright, so I'm thinking about `perf_event_ctx_lock_nested`. Itβs probably related to managing concurrency within the performance event context, right? Specifically, I'm guessing this function is designed to address a nested locking scenario. This screams "potential for deadlock" to me, and I'm immediately trying to understand the context in which this is used. Why a nested lock? What resources are being protected and why are these specific nested lock scenarios arising? Are we dealing with something like a CPU context switch within a profiling context, or perhaps an interrupt handler that can interact with the performance monitoring code? My mind immediately goes to the locking hierarchy and the potential for a programmer to inadvertently create a circular dependency. I'd need to trace the call graph to see what other locks are held when this is acquired. The βnestedβ part is key here; it implies a more complex locking scheme than a simple, single mutex. Are we trying to avoid priority inversion or are we optimizing for a very specific read-write concurrency pattern? Is it a read-write lock perhaps? I am hoping the documentation explains the full scope of what it protects. Understanding this is critical for debugging, performance analysis, and, crucially, avoiding introducing subtle, hard-to-find bugs down the line. It demands a closer look. I'll need to dig into the source code to confirm my assumptions and grasp the nuances of its behavior.
|
| 20/2 |
2026/02/06 07:12 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:perf_event_ctx_lock_nested]
Results: map[SourceCode:1299: /*
1300: * Because of perf_event::ctx migration in sys_perf_event_open::move_group and
1301: * perf_pmu_migrate_context() we need some magic.
1302: *
1303: * Those places that change perf_event::ctx will hold both
1304: * perf_event_ctx::mutex of the 'old' and 'new' ctx value.
1305: *
1306: * Lock ordering is by mutex address. There are two other sites where
1307: * perf_event_context::mutex nests and those are:
1308: *
1309: * - perf_event_exit_task_context() [ child , 0 ]
1310: * perf_event_exit_event()
1311: * put_event() [ parent, 1 ]
1312: *
1313: * - perf_event_init_context() [ parent, 0 ]
1314: * inherit_task_group()
1315: * inherit_group()
1316: * inherit_event()
1317: * perf_event_alloc()
1318: * perf_init_event()
1319: * perf_try_init_event() [ child , 1 ]
1320: *
1321: * While it appears there is an obvious deadlock here -- the parent and child
1322: * nesting levels are inverted between the two. This is in fact safe because
1323: * life-time rules separate them. That is an exiting task cannot fork, and a
1324: * spawning task cannot (yet) exit.
1325: *
1326: * But remember that these are parent<->child context relations, and
1327: * migration does not affect children, therefore these two orderings should not
1328: * interact.
1329: *
1330: * The change in perf_event::ctx does not affect children (as claimed above)
1331: * because the sys_perf_event_open() case will install a new event and break
1332: * the ctx parent<->child relation, and perf_pmu_migrate_context() is only
1333: * concerned with cpuctx and that doesn't have children.
1334: *
1335: * The places that change perf_event::ctx will issue:
1336: *
1337: * perf_remove_from_context();
1338: * synchronize_rcu();
1339: * perf_install_in_context();
1340: *
1341: * to affect the change. The remove_from_context() + synchronize_rcu() should
1342: * quiesce the event, after which we can install it in the new location. This
1343: * means that only external vectors (perf_fops, prctl) can perturb the event
1344: * while in transit. Therefore all such accessors should also acquire
1345: * perf_event_context::mutex to serialize against this.
1346: *
1347: * However; because event->ctx can change while we're waiting to acquire
1348: * ctx->mutex we must be careful and use the below perf_event_ctx_lock()
1349: * function.
1350: *
1351: * Lock order:
1352: * exec_update_lock
1353: * task_struct::perf_event_mutex
1354: * perf_event_context::mutex
1355: * perf_event::child_mutex;
1356: * perf_event_context::lock
1357: * mmap_lock
1358: * perf_event::mmap_mutex
1359: * perf_buffer::aux_mutex
1360: * perf_addr_filters_head::lock
1361: *
1362: * cpu_hotplug_lock
1363: * pmus_lock
1364: * cpuctx->mutex / perf_event_context::mutex
1365: */
1366: static struct perf_event_context *
1367: perf_event_ctx_lock_nested(struct perf_event *event, int nesting)
1368: {
1369: struct perf_event_context *ctx;
1370:
1371: again:
1372: rcu_read_lock();
1373: ctx = READ_ONCE(event->ctx);
1374: if (!refcount_inc_not_zero(&ctx->refcount)) {
1375: rcu_read_unlock();
1376: goto again;
1377: }
1378: rcu_read_unlock();
1379:
1380: mutex_lock_nested(&ctx->mutex, nesting);
1381: if (event->ctx != ctx) {
1382: mutex_unlock(&ctx->mutex);
1383: put_ctx(ctx);
1384: goto again;
1385: }
1386:
1387: return ctx;
1388: }
SourceFile:kernel/events/core.c]
|
| 21/2 |
2026/02/06 07:12 |
llm |
expert |
0m
Model:
Tokens:
input: 16607
output: 41
thoughts: 1172
Thoughts: **Analyzing Potential Data Races in Perf Event Time Tracking**
Okay, so here's the situation I'm looking at: I'm focusing on potential race conditions related to how `total_time_enabled` and `total_time_running` are accessed in the perf event system. My initial observation is that `perf_event_ctx_lock` only grabs the `ctx->mutex`, not the `ctx->lock` (the spinlock). This is a crucial detail.
Now, in `__perf_event_read_value`, I see that it reads `event->total_time_enabled` and `event->total_time_running`. The code reads these *without* holding `ctx->lock`. This immediately triggers my concern: are we potentially reading these values while they are being modified by another thread? Specifically, I see these lines of concern:
```c
5892: *enabled += event->total_time_enabled +
5893: atomic64_read(&event->child_total_time_enabled);
5894: *running += event->total_time_running +
5895: atomic64_read(&event->child_total_time_running);
```
Let's trace where these values are *written*. `perf_event_set_state` (and, subsequently, `event_sched_out`) is called with `ctx->lock` *held*. Let's confirm: `ctx_sched_out` calls `__pmu_ctx_sched_out`, which calls `group_sched_out`. Then `task_ctx_sched_out` holds `ctx->lock`. So updates within this path *are* protected.
So the core problem: `__perf_event_read_value` reads without the lock while updates are done under the lock. This raises the question of whether the values being read are atomic. These are `u64` values, which *should* be atomic on 64-bit architectures, provided they are naturally aligned. But is that guaranteed?
Looking deeper, `__perf_update_times` seems relevant:
```c
722: *enabled = event->total_time_enabled;
723: if (state >= PERF_EVENT_STATE_INACTIVE)
724: *enabled += delta;
```
If `*enabled` is indeed `&event->total_time_enabled`, this becomes:
1. `event->total_time_enabled = event->total_time_enabled;`
2. `event->total_time_enabled += delta;`
Wait a minute...why is the first line there? Isn't it redundant? Is the compiler optimizing this out? If the compiler does *not* optimize, this is two separate stores. Even if not optimized, this is atomic, assuming 8-byte stores are atomic, so it can only be either the old value, or the updated one.
The real problem lies in the cases where the values are non-atomic. But on x86-64, 8-byte stores *are* atomic.
But the plot thickens! KCSAN has flagged a data race: `value changed: 0x000000004a891b66 -> 0x000000004a89847d`. This indicates that the value *did* change between two reads by KCSAN or during a single read. This is the smoking gun!
If `__perf_event_read_value` is intended to provide a consistent snapshot of `total_time_enabled` and `total_time_running` β and, given the nature of the read, I believe it is β then we've got a problem. The current access pattern is potentially broken.
Now I remember that `__perf_event_read_value` also calls `perf_event_read(event, false)`. `perf_event_read` updates the event count.
```c
5889: (void)perf_event_read(event, false);
5890: total += perf_event_count(event, false);
```
Let's check `perf_event_read` to see what changes are made.
So, overall, this data race report needs to be investigated further, since there is a problem somewhere.
|
| 22/2 |
2026/02/06 07:12 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:perf_event_read]
Results: map[SourceCode:4817: static int perf_event_read(struct perf_event *event, bool group)
4818: {
4819: enum perf_event_state state = READ_ONCE(event->state);
4820: int event_cpu, ret = 0;
4821:
4822: /*
4823: * If event is enabled and currently active on a CPU, update the
4824: * value in the event structure:
4825: */
4826: again:
4827: if (state == PERF_EVENT_STATE_ACTIVE) {
4828: struct perf_read_data data;
4829:
4830: /*
4831: * Orders the ->state and ->oncpu loads such that if we see
4832: * ACTIVE we must also see the right ->oncpu.
4833: *
4834: * Matches the smp_wmb() from event_sched_in().
4835: */
4836: smp_rmb();
4837:
4838: event_cpu = READ_ONCE(event->oncpu);
4839: if ((unsigned)event_cpu >= nr_cpu_ids)
4840: return 0;
4841:
4842: data = (struct perf_read_data){
4843: .event = event,
4844: .group = group,
4845: .ret = 0,
4846: };
4847:
4848: preempt_disable();
4849: event_cpu = __perf_event_read_cpu(event, event_cpu);
4850:
4851: /*
4852: * Purposely ignore the smp_call_function_single() return
4853: * value.
4854: *
4855: * If event_cpu isn't a valid CPU it means the event got
4856: * scheduled out and that will have updated the event count.
4857: *
4858: * Therefore, either way, we'll have an up-to-date event count
4859: * after this.
4860: */
4861: (void)smp_call_function_single(event_cpu, __perf_event_read, &data, 1);
4862: preempt_enable();
4863: ret = data.ret;
4864:
4865: } else if (state == PERF_EVENT_STATE_INACTIVE) {
4866: struct perf_event_context *ctx = event->ctx;
4867: unsigned long flags;
4868:
4869: raw_spin_lock_irqsave(&ctx->lock, flags);
4870: state = event->state;
4871: if (state != PERF_EVENT_STATE_INACTIVE) {
4872: raw_spin_unlock_irqrestore(&ctx->lock, flags);
4873: goto again;
4874: }
4875:
4876: /*
4877: * May read while context is not active (e.g., thread is
4878: * blocked), in that case we cannot update context time
4879: */
4880: ctx_time_update_event(ctx, event);
4881:
4882: perf_event_update_time(event);
4883: if (group)
4884: perf_event_update_sibling_time(event);
4885: raw_spin_unlock_irqrestore(&ctx->lock, flags);
4886: }
4887:
4888: return ret;
4889: }
SourceFile:kernel/events/core.c]
|
| 23/2 |
2026/02/06 07:12 |
llm |
expert |
0m
Model:
Tokens:
input: 19848
output: 41
thoughts: 6979
Thoughts: **Analyzing the perf_event Data Race**
Okay, here's the situation as I understand it. I'm looking at a potential data race within the performance monitoring (perf) subsystem. The core issue revolves around how the `total_time_enabled` and `total_time_running` values of a `perf_event` are read and updated when an event is scheduled out or read.
My initial thoughts were around `perf_event_read` function. If the event's state is `PERF_EVENT_STATE_ACTIVE`, this code *doesn't* acquire `ctx->lock`, and instead uses `smp_call_function_single` to execute `__perf_event_read` on the CPU where the event is active. The `__perf_event_read` function *does* call `perf_event_update_time`, which is where things get tricky.
I initially theorized that the IPI call would act as synchronization; if `perf_event_read` sees `ACTIVE`, then sends an IPI to update time. That might not be the case because `__perf_event_read_value` reads those values after the `perf_event_read` has returned.
On the other hand, the writer (the part scheduling out the event, in this case `event_sched_out`) calls `perf_event_set_state`. Critically, this function calls `perf_event_update_time` *before* changing the event state to `INACTIVE`. So, while the IPI should have serialized the reader and writer, the reader could still race with `__perf_update_times`.
The actual read, in `__perf_event_read_value`, happens *after* the initial `perf_event_read` call. And the read is just a plain `event->total_time_enabled`. There's no locking to protect this read.
My confusion stems from why the IPI, which is supposed to run on the writer's CPU, isn't providing enough synchronization. Because if `perf_event_read` has returned, then the IPI on the writer's CPU should have finished. Therefore the values should be consistent when read by the reader.
I think there are multiple possible race conditions. If `perf_event_read` sees the event as `OFF` (or any state that isn't `ACTIVE`), it returns without any explicit synchronization. But by the time the `__perf_event_read_value` reads, things may have changed. The IPI might not be the source of protection for all cases.
I see that `perf_event_read` is using `READ_ONCE` for the event's state (`event->state`) and CPU (`event->oncpu`). Then, if the reader sees `ACTIVE` and the CPU is valid, it sends an IPI to that CPU.
I'm starting to think that the issue might be if `event->oncpu` becomes invalid *between* `READ_ONCE` calls. If it reads a valid CPU initially, an IPI is sent. If `event->oncpu` changes *before* the IPI runs (e.g., event is scheduled out *again*), the IPI still runs, but the time values become inconsistent.
Aha! This is what it is: If `perf_event_read` (on CPU 0) sees `ACTIVE` and a valid CPU (CPU 1). It sends IPI to CPU 1. However, after the reader (on CPU 0) saw `ACTIVE`, but *before* the IPI runs (on CPU 1) the writer (on CPU 1) is now in `event_sched_out` (context switch), it calls `perf_event_set_state` (setting state to INACTIVE).
* `perf_event_update_time(event)`
* `WRITE_ONCE(event->state, INACTIVE)`
*If* the `perf_event_read` calls `READ_ONCE` on state and then checks on the CPU, a race would occur. This might happen because `event->oncpu` and state could change between being read.
The reader is reading `total_time_enabled` *after* the potentially out-of-date information from the writer on CPU 1, which the IPI should have ensured consistency.
I think the KCSAN report is correct because there's a race between `event->total_time_enabled` read on CPU 0 in `__perf_event_read_value` and its write (via `__perf_update_times`) in `__perf_event_read` called by the IPI on CPU 1. Even with IPIs and interrupts disabled in various places, the
|
| 24/2 |
2026/02/06 07:12 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:event_sched_out]
Results: map[SourceCode:2358: static void
2359: event_sched_out(struct perf_event *event, struct perf_event_context *ctx)
2360: {
2361: struct perf_event_pmu_context *epc = event->pmu_ctx;
2362: struct perf_cpu_pmu_context *cpc = this_cpc(epc->pmu);
2363: enum perf_event_state state = PERF_EVENT_STATE_INACTIVE;
2364:
2365: // XXX cpc serialization, probably per-cpu IRQ disabled
2366:
2367: WARN_ON_ONCE(event->ctx != ctx);
2368: lockdep_assert_held(&ctx->lock);
2369:
2370: if (event->state != PERF_EVENT_STATE_ACTIVE)
2371: return;
2372:
2373: /*
2374: * Asymmetry; we only schedule events _IN_ through ctx_sched_in(), but
2375: * we can schedule events _OUT_ individually through things like
2376: * __perf_remove_from_context().
2377: */
2378: list_del_init(&event->active_list);
2379:
2380: perf_pmu_disable(event->pmu);
2381:
2382: event->pmu->del(event, 0);
2383: event->oncpu = -1;
2384:
2385: if (event->pending_disable) {
2386: event->pending_disable = 0;
2387: perf_cgroup_event_disable(event, ctx);
2388: state = PERF_EVENT_STATE_OFF;
2389: }
2390:
2391: perf_event_set_state(event, state);
2392:
2393: if (!is_software_event(event))
2394: cpc->active_oncpu--;
2395: if (is_event_in_freq_mode(event)) {
2396: ctx->nr_freq--;
2397: epc->nr_freq--;
2398: }
2399: if (event->attr.exclusive || !cpc->active_oncpu)
2400: cpc->exclusive = 0;
2401:
2402: perf_pmu_enable(event->pmu);
2403: }
SourceFile:kernel/events/core.c]
|
| 25/2 |
2026/02/06 07:12 |
llm |
expert |
0m
Model:
Tokens:
input: 28613
output: 19
thoughts: 3130
Thoughts: **Analysis of a Potential Data Race in perf_event_read**
Okay, so I've been digging into this KCSAN report, and it looks like we have a data race issue with how `perf_event_read` interacts with event scheduling. Here's what I've got:
It seems the race happens when an event is being scheduled out (`event_sched_out`). The writer (on one CPU) first sets `event->oncpu = -1;` and *then* calls `perf_event_set_state`. Critically, `perf_event_set_state` contains the `__perf_update_times` call, which writes to `total_time_enabled`.
The reader, on the other hand (perhaps a user-space process calling `read()` on the perf event file descriptor), might read `event->state` as `ACTIVE`, but then read `event->oncpu` as `-1`. This is the crux of it. The existing code checks for a valid `oncpu` *after* checking if the event is active. If the `oncpu` check fails (because it's -1), the function returns 0. Crucially, it skips the synchronization that would be needed to ensure consistency.
Here's the problem: The reader *then* proceeds to read `total_time_enabled`. But the *writer* is actively updating this value in `__perf_update_times`!
On a 64-bit system, the `u64` read/write of `total_time_enabled` is likely atomic, so we are unlikely to see torn values. However, on a 32-bit system, the `u64` access is *not* atomic. The reader might see the high 32 bits from the old value and the low 32 bits from the new value β leading to a completely corrupted value. This would lead to incorrect scaling calculations. That's a definite problem.
The code seems to return 0 when this happens, which means the reader gets an un-updated (or partially updated) `total_time_enabled`. The comment in the code suggests that the code *should* update the value in this case, but the actual code path doesn't do that.
We really need a robust synchronization mechanism in `perf_event_read`. If we see `oncpu == -1` while the event state is `ACTIVE`, we should probably acquire the `ctx->lock`, as the writer *should* be holding that lock during `perf_event_set_state` (and, importantly, `__perf_update_times`). We can then safely read or wait for the state to transition.
The existing code checks the state again under the lock after the initial read. However, this doesn't protect the critical window where `oncpu` changes *before* the state update.
So, in short, on 32-bit, this is a harmful data race as a `u64` read/write is not atomic. While on 64-bit, this might be benign (as we're unlikely to tear the value) it is a race as an inconsistent value can be read by the reader. This leads to inaccurate perf event readings.
|
| 26/2 |
2026/02/06 07:13 |
tool |
grepper |
0m
Args: map[Expression:total_time_enabled]
Results: map[Output:include/linux/perf_event.h=764=struct perf_event {
--
include/linux/perf_event.h-820- */
include/linux/perf_event.h:821: u64 total_time_enabled;
include/linux/perf_event.h-822- u64 total_time_running;
--
include/linux/perf_event.h-843- */
include/linux/perf_event.h:844: atomic64_t child_total_time_enabled;
include/linux/perf_event.h-845- atomic64_t child_total_time_running;
--
kernel/events/core.c=717=__perf_update_times(struct perf_event *event, u64 now, u64 *enabled, u64 *running)
--
kernel/events/core.c-721-
kernel/events/core.c:722: *enabled = event->total_time_enabled;
kernel/events/core.c-723- if (state >= PERF_EVENT_STATE_INACTIVE)
--
kernel/events/core.c=731=static void perf_event_update_time(struct perf_event *event)
--
kernel/events/core.c-734-
kernel/events/core.c:735: __perf_update_times(event, now, &event->total_time_enabled,
kernel/events/core.c-736- &event->total_time_running);
--
kernel/events/core.c=3580=static void __perf_event_sync_stat(struct perf_event *event,
--
kernel/events/core.c-3606-
kernel/events/core.c:3607: swap(event->total_time_enabled, next_event->total_time_enabled);
kernel/events/core.c-3608- swap(event->total_time_running, next_event->total_time_running);
--
kernel/events/core.c=5879=static u64 __perf_event_read_value(struct perf_event *event, u64 *enabled, u64 *running)
--
kernel/events/core.c-5891-
kernel/events/core.c:5892: *enabled += event->total_time_enabled +
kernel/events/core.c:5893: atomic64_read(&event->child_total_time_enabled);
kernel/events/core.c-5894- *running += event->total_time_running +
--
kernel/events/core.c-5899- total += perf_event_count(child, false);
kernel/events/core.c:5900: *enabled += child->total_time_enabled;
kernel/events/core.c-5901- *running += child->total_time_running;
--
kernel/events/core.c=5921=static int __perf_read_group_add(struct perf_event *leader,
--
kernel/events/core.c-5968- if (read_format & PERF_FORMAT_TOTAL_TIME_ENABLED) {
kernel/events/core.c:5969: values[n++] += leader->total_time_enabled +
kernel/events/core.c:5970: atomic64_read(&leader->child_total_time_enabled);
kernel/events/core.c-5971- }
--
kernel/events/core.c=6539=void perf_event_update_userpage(struct perf_event *event)
--
kernel/events/core.c-6550- /*
kernel/events/core.c:6551: * compute total_time_enabled, total_time_running
kernel/events/core.c-6552- * based on snapshot values taken when the event
--
kernel/events/core.c-6574- userpg->time_enabled = enabled +
kernel/events/core.c:6575: atomic64_read(&event->child_total_time_enabled);
kernel/events/core.c-6576-
--
kernel/events/core.c=7772=static void perf_output_read_one(struct perf_output_handle *handle,
--
kernel/events/core.c-7782- values[n++] = enabled +
kernel/events/core.c:7783: atomic64_read(&event->child_total_time_enabled);
kernel/events/core.c-7784- }
--
kernel/events/core.c=7865=static void perf_output_read(struct perf_output_handle *handle,
--
kernel/events/core.c-7871- /*
kernel/events/core.c:7872: * compute total_time_enabled, total_time_running
kernel/events/core.c-7873- * based on snapshot values taken when the event
--
kernel/events/core.c=14091=static void sync_child_event(struct perf_event *child_event,
--
kernel/events/core.c-14107- atomic64_add(child_val, &parent_event->child_count);
kernel/events/core.c:14108: atomic64_add(child_event->total_time_enabled,
kernel/events/core.c:14109: &parent_event->child_total_time_enabled);
kernel/events/core.c-14110- atomic64_add(child_event->total_time_running,
]
|
| 27/2 |
2026/02/06 07:13 |
llm |
expert |
0m
Model:
Tokens:
input: 33900
output: 43
thoughts: 1813
Thoughts: **Race Condition Analysis in Perf Event Reading**
Okay, let's break this down. It seems there's a potential race condition with how `total_time_enabled` is handled in the perf event code, specifically in the `perf_event_read` path. I'm seeing that `total_time_enabled` is treated as a plain `u64`, unlike `child_total_time_enabled`, which is an `atomic64_t`. That immediately triggers my alarm bells. If this value is being updated while an event is active, it needs some form of protection, either a lock or atomics, to ensure data integrity.
My initial thought was that updates to `total_time_enabled` are serialized during event scheduling or through `perf_event_read`. The IPI used by `perf_event_read` when an event is active seemed to provide serialization for that specific CPU, particularly because interrupts are disabled, which is good. But the `oncpu == -1` case in `perf_event_read` has thrown a wrench into the works.
I'm seeing a problem when the event *isn't* active. The assumption is that `ctx->lock` should protect it. But I'm going through the code for `__perf_event_read_value` and it looks like it reads `total_time_enabled` *without* holding `ctx->lock`. That's an immediate red flag. It seems `__perf_event_read_value` holds `child_mutex` but not `ctx->lock`. `perf_event_read` is called internally and, even if it does take `ctx->lock` internally, it *releases* it before returning. So, when the code dereferences `event->total_time_enabled`, no lock is held.
Now, if the event is "INACTIVE", it *should* be stable unless someone else is updating it. But who can update it? Another thread calling `perf_event_read`? They *should* take `ctx->lock`, and then release it, so that should prevent any races. But what about a call to `event_sched_out`? That's holding `ctx->lock`, and it's looking like `perf_event_read` can return when the state is `ACTIVE` but `oncpu == -1` without taking the lock, and potentially read `total_time_enabled` while `event_sched_out` is *still* updating it!
Yep, there's a race here, and it's a real one. On a 64-bit system, it could lead to inconsistent reads, which can lead to problems, but on a 32-bit system, there could be *torn* reads, which are *much* worse. And since the kernel supports 32-bit, that makes this a serious issue. Even on 64-bit, the inconsistencies in the `enabled_time` and `running_time` can lead to seriously weird results for the user, especially when the values are used for scaling the raw counter values.
Okay, let me back up a step. I should really check how `perf_event_read` is *meant* to be used. Since `__perf_event_read_value` is used by `perf_read`, which is a system call, users *expect* consistent results from `read()`. And if I have a group of events, I would want them to be consistent with each other. `__perf_event_read_value` doesn't seem to offer consistency across multiple children, which iterates them individually, but for a single event at least, it should be consistent with itself!
Hmm, there's a known issue of races around perf time updates. Let me see if there are any recent patches addressing this area. I cannot directly search for related patches, but I can re-examine the code.
Alright, I see something interesting with `perf_event_set_state` using `WRITE_ONCE` and `perf_event_read` using `READ_ONCE`. This is good. It shows that there is thought of data races related to the state of the events. However, `total_time_enabled` *isn't* using `READ_ONCE`/`WRITE_ONCE`.
If I look at `__perf_update_times` again, the code reads and updates `total_time_enabled`, and this becomes: `event->total_time_enabled = event->total_time_enabled + delta;`. This is a read-modify-write operation on a plain `u64`. If another thread (the reader) accesses it mid-operation, it gets a bad value. On x86_64, an `add` instruction can potentially be
|
| 28/2 |
2026/02/06 07:13 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:perf_event_read_value]
Results: map[SourceCode:5908: u64 perf_event_read_value(struct perf_event *event, u64 *enabled, u64 *running)
5909: {
5910: struct perf_event_context *ctx;
5911: u64 count;
5912:
5913: ctx = perf_event_ctx_lock(event);
5914: count = __perf_event_read_value(event, enabled, running);
5915: perf_event_ctx_unlock(event, ctx);
5916:
5917: return count;
5918: }
SourceFile:kernel/events/core.c]
|
| 29/2 |
2026/02/06 07:13 |
llm |
expert |
0m
Model:
Tokens:
input: 36978
output: 44
thoughts: 273
Thoughts: **Analyzing the Perf Event Data Synchronization**
Okay, so I'm looking at this `perf_event_ctx_lock` and it's clear it's *only* grabbing a mutex. That's interesting, and it's making me think about how the perf subsystem is designed to handle potential data races. It seems the assumption is that the `perf_event_read` function provides the necessary synchronization. But, as I've demonstrated, that's just not the case. This feels like a potential issue.
I need to figure out if this is a known, "benign" race condition. I have to find some evidence if that's the case. Typically, races involving `u64` values aren't considered benign unless we're talking about statistics where tearing isn't a huge concern, and even then, thatβs really only true on 64-bit architectures. Remember, `perf` is used on 32-bit systems as well, like ARM and x86, which makes the possibility of tearing more significant.
Hold on, I recall seeing a similar problem before. I need to re-examine the `__perf_update_times` function; it's used in a lot of different places, so there's probably a lot of opportunities for issues here.
Specifically, I'm going to look again at how `perf_event_update_userpage` works. Examining the code:
```c
6574: userpg->time_enabled = enabled +
6575: atomic64_read(&event->child_total_time_enabled);
```
Here, `enabled` is a local variable that was calculated earlier, and I need to double-check that calculation of `enabled` to see if it is synchronized properly.
|
| 30/2 |
2026/02/06 07:13 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/events/core.c IncludeLines:true Name:perf_event_update_userpage]
Results: map[SourceCode:6534: /*
6535: * Callers need to ensure there can be no nesting of this function, otherwise
6536: * the seqlock logic goes bad. We can not serialize this because the arch
6537: * code calls this from NMI context.
6538: */
6539: void perf_event_update_userpage(struct perf_event *event)
6540: {
6541: struct perf_event_mmap_page *userpg;
6542: struct perf_buffer *rb;
6543: u64 enabled, running, now;
6544:
6545: rcu_read_lock();
6546: rb = rcu_dereference(event->rb);
6547: if (!rb)
6548: goto unlock;
6549:
6550: /*
6551: * compute total_time_enabled, total_time_running
6552: * based on snapshot values taken when the event
6553: * was last scheduled in.
6554: *
6555: * we cannot simply called update_context_time()
6556: * because of locking issue as we can be called in
6557: * NMI context
6558: */
6559: calc_timer_values(event, &now, &enabled, &running);
6560:
6561: userpg = rb->user_page;
6562: /*
6563: * Disable preemption to guarantee consistent time stamps are stored to
6564: * the user page.
6565: */
6566: preempt_disable();
6567: ++userpg->lock;
6568: barrier();
6569: userpg->index = perf_event_index(event);
6570: userpg->offset = perf_event_count(event, false);
6571: if (userpg->index)
6572: userpg->offset -= local64_read(&event->hw.prev_count);
6573:
6574: userpg->time_enabled = enabled +
6575: atomic64_read(&event->child_total_time_enabled);
6576:
6577: userpg->time_running = running +
6578: atomic64_read(&event->child_total_time_running);
6579:
6580: arch_perf_update_userpage(event, userpg, now);
6581:
6582: barrier();
6583: ++userpg->lock;
6584: preempt_enable();
6585: unlock:
6586: rcu_read_unlock();
6587: }
SourceFile:kernel/events/core.c]
|
| 31/2 |
2026/02/06 07:13 |
llm |
expert |
0m
Model:
Tokens:
input: 38464
output: 23
thoughts: 2805
Thoughts: **Data Race in Perf Event Time Tracking**
Okay, here's what I've got. I've been tracing how the performance event timer values are calculated and read, and I think I've found a pretty serious data race. It looks like the core issue boils down to the way `total_time_enabled` and `total_time_running` are handled in conjunction with `perf_event_read`.
It starts with `calc_timer_values`, which appears safe in how it updates local variables, passing them to `__perf_update_times`. That's good. However, the problem arises in `perf_event_update_time`. This function directly modifies the `total_time_enabled` and `total_time_running` fields of the `perf_event` structure. And, in parallel, `__perf_event_read_value` directly reads these values.
The key here is that `total_time_enabled` is just a plain `u64`. There's no `WRITE_ONCE` for the write and no `READ_ONCE` for the read. That's a huge red flag. On 32-bit systems, this leads to tearing of values; on 64-bit systems, it's still a data race that can give inconsistent results, as one read will happen at one time, and the values may update in a manner not atomic.
Initially, I thought, is there some reason this is safe that I'm missing? Could the values be "close enough"? But tearing, or even race conditions, are never acceptable.
I checked `perf_event_set_state` to see if it was being called with interrupts disabled. It is, from `event_sched_out`, which *does* hold `ctx->lock`, so writers on the same CPU *are* protected. But the *readers* can be on other CPUs, and that's the problem.
`perf_event_read` is supposed to synchronize here. But there's a hole.
I found a possible commit about this race, but I can't search for commits at this time.
Let's look at `perf_event_read` again. If I were fixing it, it's be nice if it didn't return 0 if `oncpu == -1` and `state == ACTIVE`. It should probably wait for `state` to become something else, or take the lock.
If `oncpu == -1`, the event isn't running on a CPU. That means `total_time_running` should *not* be increasing. But, `total_time_enabled` might still be increasing if the context is active.
`event_sched_out` is called when an event is being removed from a CPU. The writer is updating the time to reflect how long it was active. Now, if the reader reads the values *during* this update, it might get the value *before* the update or *after* the update. Both are "correct" but if the kernel doesn't wait, tearing and inconsistency are the issue.
I was originally thinking the kernel developers would just say a `u64` on 64-bit is atomic. But, KCSAN is running on 64-bit, and it's complaining. And KCSAN typically ignores races marked `data_race()` or protected with `READ_ONCE`, etc., so the fact it's complaining means it's a real issue.
I noticed `perf_event_count` uses `local64_read` for its `count` field, which is `local64_t`. But `total_time_enabled` is just a `u64`. Why this difference? Probably because `count` is updated far more often. But if something is accessed from multiple CPUs, it has to be protected.
So, *this is a bug*.
Is it harmful? Yes, because of tearing on 32-bit and inconsistency on 64-bit. `perf_event_read` is supposed to fix this by updating everything, but it fails in the `oncpu == -1` case.
I checked `total_time_enabled` and if the code is updating it without `ctx->lock`. If `state == INACTIVE`, it takes `ctx->lock`. If `state == ACTIVE`, it sends an IPI. *Both* cases are synchronized. The only unsynchronized case is that hole I found.
Here's the race:
* **Reader:** `perf_event_read` sees `ACTIVE`, `oncpu == -1`, returns 0, and reads `total_time_enabled`.
* **Writer:** `event_sched_out` sets `oncpu = -1
|
| 32/2 |
2026/02/06 07:14 |
tool |
codesearch-find-references |
0m
Args: map[Name:perf_event_read]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__perf_event_read_value SourceFile:kernel/events/core.c SourceLine:5889] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__perf_event_read_value SourceFile:kernel/events/core.c SourceLine:5898] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__perf_read_group_add SourceFile:kernel/events/core.c SourceLine:5930] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:_perf_event_reset SourceFile:kernel/events/core.c SourceLine:6160]] TruncatedOutput:false]
|
| 33/2 |
2026/02/06 07:14 |
llm |
expert |
0m
Model:
Tokens:
input: 42547
output: 19
thoughts: 124
Thoughts: **Analyzing the Lock Acquisition in Perf Event Operations**
Okay, so I'm looking at how this `ctx->lock` is being acquired. It seems like the critical code paths β namely, `__perf_event_read_value`, `__perf_read_group_add`, and `_perf_event_reset` β are all originating from within syscall context, specifically `read` and `ioctl`. This suggests that acquiring the lock within those functions shouldn't pose any immediate thread safety concerns, which is good. Weβre in a protected environment at that point.
Hang on though⦠`_perf_event_reset` is a little more involved. It appears within `perf_event_for_each_child` which in turn is called by `perf_event_ioctl`. Ah, yes, still syscall context. So the lock acquisition should still be safe there.
This analysis, if correct, indicates that the problematic race condition in `perf_event_read` might indeed be resolvable with the lock. We could be looking at a straightforward fix.
The severity of the race condition means itβs critical that we address this issue. A harmful race condition is something we want to eliminate.
|
| 34/2 |
2026/02/06 07:14 |
tool |
set-results |
0m
Args: map[Benign:false Confident:true]
Results: map[Benign:false Confident:true]
|
| 35/2 |
2026/02/06 07:14 |
llm |
expert |
0m
Model:
Tokens:
input: 42960
output: 827
thoughts: 357
Thoughts: **My Analysis of the Perf Event Data Race**
Okay, so I've pinned down a pretty nasty data race, and here's the breakdown of why it's a problem. I'm focusing on the `total_time_enabled` and `total_time_running` fields within the `struct perf_event` structure. Those are just plain `u64` values, no atomic operations, no memory barriers β nothing to prevent concurrent access.
The write side is clear: it's `event_sched_out`, triggered during a context switch. It updates those fields through the `perf_event_set_state` chain, specifically in `perf_event_update_time`. This is where the time values are being modified.
The read happens within `__perf_event_read_value`, which is called through `perf_read` via the io_uring interface. It reads those time fields after invoking `perf_event_read`.
Here's where the critical flaw lies: `perf_event_read` has a synchronization window. If it sees the event state as `ACTIVE`, but the `oncpu` field as `-1` (and this `-1` condition does occur during `event_sched_out` as `oncpu` is set to `-1` *before* the time update), it simply returns without any additional synchronization. Effectively, we have a clear concurrent read and write happening on those `u64` fields.
This is where things get ugly. On 32-bit systems, we're almost certainly going to see torn reads β the reader will get a garbage mixture of old and new values for the high and low 32 bits. On 64-bit systems, even though we won't have torn reads, we can still have inconsistent results, the read values of `count`, `enabled_time`, and `running_time` will not align. In effect, we get the wrong scale and thus an incorrect performance counter reading.
This synchronization hole in `perf_event_read` is a real bug and needs to be addressed immediately.
|