| 0/0 |
2026/03/15 01:08 |
flow |
moderation |
17m
Results: map[Actionable:true Confident:true Explanation:The bug report describes a KASAN slab-use-after-free in `__bpf_trace_run`. Based on the provided stack traces and kernel source code analysis, the report is self-consistent and actionable.
### Analysis:
1. **Access Stack Trace**: The use-after-free occurs in `__bpf_trace_run` (inlined in `bpf_trace_run4`), which is called as a probe for the `mm_page_alloc` tracepoint. The access is at offset 24 (0x18) of the `link` object, which corresponds to the `prog` field of `struct bpf_link` (the base of `struct bpf_raw_tp_link`).
2. **Allocation Stack Trace**: The object was allocated in `bpf_raw_tp_link_attach`, which creates a `bpf_raw_tp_link` to attach a BPF program to a raw tracepoint.
3. **Free Stack Trace**: The object was freed via RCU (`rcu_do_batch`). The "Last potentially related work creation" shows that `call_rcu` was invoked from `bpf_link_put_direct` during the release of a BPF link (likely when its file descriptor was closed).
### Root Cause:
The underlying issue is a synchronization mismatch between the tracepoint subsystem and BPF link deallocation:
* **Tracepoint Protection**: In recent kernels (e.g., 6.11+), regular tracepoints (defined via `TRACE_EVENT`) use `tracepoint_srcu` (SRCU) to protect the execution of their probes.
* **BPF Link Deallocation**: When a `bpf_raw_tp_link` is released, it is first unregistered from the tracepoint. However, `tracepoint_probe_unregister` does not wait for a grace period. The BPF link infrastructure then defers the actual freeing of the `link` object using `call_rcu` (regular RCU) if the link is not marked as sleepable.
* **The Mismatch**: A regular RCU grace period (`call_rcu`) does not necessarily wait for an SRCU grace period (`tracepoint_srcu`) to complete, especially in `PREEMPT_RT` or `PREEMPT_FULL` configurations where SRCU-fast readers are not implicit RCU readers.
* **Result**: The `bpf_raw_tp_link` object is freed while a tracepoint caller (like `udevd` in the report) is still executing the BPF probe within an SRCU read-side critical section, leading to a use-after-free when the probe tries to access `link->link.prog`.
### Conclusion:
The report provides all the necessary information to identify this synchronization bug. It is a clear case of an object being freed after the wrong type of grace period. The report is highly actionable for kernel developers.]
|
| 1/1 |
2026/03/15 01:08 |
action |
kernel-checkouter |
0m
Results: map[KernelSrc:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/src/0ace5a22bf58e4c37f72d8e2ed6b5cc1da24d6d1]
|
| 2/1 |
2026/03/15 01:08 |
action |
kernel-builder |
5m
Results: map[KernelObj:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/build/0a82e5a4cad5f01fcdea81a7856329efb097675d]
|
| 3/1 |
2026/03/15 01:14 |
action |
codesearch-prepare |
7m
Results: map[Index:codesearch-index]
|
| 4/1 |
2026/03/15 01:22 |
agent |
expert |
3m
Model:
Results: map[Actionable:true Confident:true]
Instruction:
You are an experienced Linux kernel developer tasked with determining if the given kernel bug
report is actionable or not. Actionable means that it contains enough info to root cause
the underlying bug, and that the report is self-consistent and makes sense, rather than
e.g. a one-off nonsensical crash induced by a previous memory corruption.
The bug report is about a use-after-free bug generated by KASAN tool.
It should contain 3 stack traces: the bad memory access stack, the heap block allocation stack,
and the heap block free stack. If the report does not contain 3 stacks, it's not actionable.
All 3 stack traces should be related to the same object type,
and usually be in the same kernel subsystem (at least leaf stack frames).
An example of an actionable and consistent report would be: first access stack relates
to an access to a field of struct Foo, allocation/free stacks relate to allocation/free
of the struct Foo.
In inconsistent/nonsensical reports an access may be to a struct Foo, but allocation
stack allocates a different structure in a different subsystem.
Look for other suspicious signals/inconsistencies that can make this report hard to
debug/understand.
In the final reply explain why you think the report is self-consistent and actionable,
or why it's inconsistent and/or not actionable.
Use the provided tools to confirm any assumptions, variables/fields being accessed, etc.
In particular, don't make assumptions about the kernel source code,
use codesearch tools to read the actual source code.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt:
The bug report is:
==================================================================
BUG: KASAN: slab-use-after-free in __bpf_trace_run kernel/trace/bpf_trace.c:2075 [inline]
BUG: KASAN: slab-use-after-free in bpf_trace_run4+0xe6/0x850 kernel/trace/bpf_trace.c:2131
Read of size 8 at addr ffff8880361a0318 by task udevd/5299
CPU: 0 UID: 0 PID: 5299 Comm: udevd Not tainted syzkaller #0 PREEMPT(full)
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0xe8/0x150 lib/dump_stack.c:120
print_address_description mm/kasan/report.c:378 [inline]
print_report+0xba/0x230 mm/kasan/report.c:482
kasan_report+0x117/0x150 mm/kasan/report.c:595
__bpf_trace_run kernel/trace/bpf_trace.c:2075 [inline]
bpf_trace_run4+0xe6/0x850 kernel/trace/bpf_trace.c:2131
__traceiter_mm_page_alloc+0x3d/0x60 include/trace/events/kmem.h:180
__do_trace_mm_page_alloc include/trace/events/kmem.h:180 [inline]
trace_mm_page_alloc+0x149/0x180 include/trace/events/kmem.h:180
__alloc_frozen_pages_noprof+0x1de/0x380 mm/page_alloc.c:5272
alloc_pages_mpol+0x232/0x4a0 mm/mempolicy.c:2484
folio_alloc_mpol_noprof+0x39/0x70 mm/mempolicy.c:2503
swap_cache_alloc_folio+0xd5/0x240 mm/swap_state.c:571
swap_cluster_readahead+0x369/0x690 mm/swap_state.c:749
swapin_readahead+0x196/0xc50 mm/swap_state.c:924
do_swap_page+0x56f/0x5a20 mm/memory.c:4802
handle_pte_fault mm/memory.c:6320 [inline]
__handle_mm_fault mm/memory.c:6455 [inline]
handle_mm_fault+0x12d2/0x3310 mm/memory.c:6624
do_user_addr_fault+0xa73/0x1340 arch/x86/mm/fault.c:1334
handle_page_fault arch/x86/mm/fault.c:1474 [inline]
exc_page_fault+0x6a/0xc0 arch/x86/mm/fault.c:1527
asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:618
RIP: 0033:0x55f04dab55f0
Code: c0 0f 85 0e 19 00 00 4c 8b 73 18 c7 44 24 28 00 00 00 00 49 89 dc 4c 8d 3d e9 95 02 00 4c 89 34 24 66 0f 1f 84 00 00 00 00 00 <41> 0f b6 1e 80 fb 35 0f 87 e3 01 00 00 0f b6 c3 49 63 04 87 4c 01
RSP: 002b:00007fff48a349c0 EFLAGS: 00010206
RAX: 000055f051ea3770 RBX: 0000000000000034 RCX: 0000000000000063
RDX: 0000000000000381 RSI: 000055f051eb5b50 RDI: 000055f051ecfdae
RBP: 000055f052096b80 R08: 000055f04daf2100 R09: 000055f04daf2140
R10: 0000000000000000 R11: 0000000000000000 R12: 000055f051eb5270
R13: 000055f051eb02c0 R14: 000055f051ea4574 R15: 000055f04dadebcc
</TASK>
Allocated by task 5326:
kasan_save_stack mm/kasan/common.c:57 [inline]
kasan_save_track+0x3e/0x80 mm/kasan/common.c:78
poison_kmalloc_redzone mm/kasan/common.c:398 [inline]
__kasan_kmalloc+0x93/0xb0 mm/kasan/common.c:415
kasan_kmalloc include/linux/kasan.h:263 [inline]
__kmalloc_cache_noprof+0x31c/0x660 mm/slub.c:5383
kmalloc_noprof include/linux/slab.h:950 [inline]
kzalloc_noprof include/linux/slab.h:1188 [inline]
bpf_raw_tp_link_attach+0x278/0x700 kernel/bpf/syscall.c:4264
bpf_raw_tracepoint_open+0x1b2/0x220 kernel/bpf/syscall.c:4312
__sys_bpf+0x846/0x950 kernel/bpf/syscall.c:6270
__do_sys_bpf kernel/bpf/syscall.c:6341 [inline]
__se_sys_bpf kernel/bpf/syscall.c:6339 [inline]
__x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6339
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0x14d/0xf80 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x77/0x7f
Freed by task 15:
kasan_save_stack mm/kasan/common.c:57 [inline]
kasan_save_track+0x3e/0x80 mm/kasan/common.c:78
kasan_save_free_info+0x46/0x50 mm/kasan/generic.c:584
poison_slab_object mm/kasan/common.c:253 [inline]
__kasan_slab_free+0x5c/0x80 mm/kasan/common.c:285
kasan_slab_free include/linux/kasan.h:235 [inline]
slab_free_hook mm/slub.c:2692 [inline]
slab_free mm/slub.c:6168 [inline]
kfree+0x1c1/0x630 mm/slub.c:6486
rcu_do_batch kernel/rcu/tree.c:2617 [inline]
rcu_core+0x7cd/0x1070 kernel/rcu/tree.c:2869
handle_softirqs+0x22a/0x870 kernel/softirq.c:622
run_ksoftirqd+0x36/0x60 kernel/softirq.c:1063
smpboot_thread_fn+0x541/0xa50 kernel/smpboot.c:160
kthread+0x388/0x470 kernel/kthread.c:436
ret_from_fork+0x51e/0xb90 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
Last potentially related work creation:
kasan_save_stack+0x3e/0x60 mm/kasan/common.c:57
kasan_record_aux_stack+0xbd/0xd0 mm/kasan/generic.c:556
__call_rcu_common kernel/rcu/tree.c:3131 [inline]
call_rcu+0xee/0x890 kernel/rcu/tree.c:3251
bpf_link_put_direct kernel/bpf/syscall.c:3323 [inline]
bpf_link_release+0x6b/0x80 kernel/bpf/syscall.c:3330
__fput+0x44f/0xa70 fs/file_table.c:469
task_work_run+0x1d9/0x270 kernel/task_work.c:233
exit_task_work include/linux/task_work.h:40 [inline]
do_exit+0x70f/0x23c0 kernel/exit.c:976
do_group_exit+0x21b/0x2d0 kernel/exit.c:1118
get_signal+0x1284/0x1330 kernel/signal.c:3034
arch_do_signal_or_restart+0xbc/0x830 arch/x86/kernel/signal.c:337
__exit_to_user_mode_loop kernel/entry/common.c:64 [inline]
exit_to_user_mode_loop kernel/entry/common.c:98 [inline]
__exit_to_user_mode_prepare include/linux/irq-entry-common.h:226 [inline]
irqentry_exit_to_user_mode_prepare include/linux/irq-entry-common.h:270 [inline]
irqentry_exit_to_user_mode include/linux/irq-entry-common.h:339 [inline]
irqentry_exit+0x176/0x620 kernel/entry/common.c:219
asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:618
The buggy address belongs to the object at ffff8880361a0300
which belongs to the cache kmalloc-192 of size 192
The buggy address is located 24 bytes inside of
freed 192-byte region [ffff8880361a0300, ffff8880361a03c0)
The buggy address belongs to the physical page:
page: refcount:0 mapcount:0 mapping:0000000000000000 index:0xffff8880361a0400 pfn:0x361a0
flags: 0x4fff00000000200(workingset|node=1|zone=1|lastcpupid=0x7ff)
page_type: f5(slab)
raw: 04fff00000000200 ffff88801ac413c0 ffffea0000d7ce90 ffffea0000e14e90
raw: ffff8880361a0400 000000080010000f 00000000f5000000 0000000000000000
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 0, migratetype Unmovable, gfp_mask 0xd2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 922, tgid 922 (kworker/0:3), ts 22003490640, free_ts 22003061349
set_page_owner include/linux/page_owner.h:32 [inline]
post_alloc_hook+0x231/0x280 mm/page_alloc.c:1889
prep_new_page mm/page_alloc.c:1897 [inline]
get_page_from_freelist+0x24dc/0x2580 mm/page_alloc.c:3962
__alloc_frozen_pages_noprof+0x18d/0x380 mm/page_alloc.c:5250
alloc_slab_page mm/slub.c:3296 [inline]
allocate_slab+0x77/0x660 mm/slub.c:3485
new_slab mm/slub.c:3543 [inline]
refill_objects+0x331/0x3c0 mm/slub.c:7178
__pcs_replace_empty_main+0x2f9/0x5e0 mm/slub.c:-1
alloc_from_pcs mm/slub.c:4720 [inline]
slab_alloc_node mm/slub.c:4854 [inline]
__do_kmalloc_node mm/slub.c:5262 [inline]
__kmalloc_noprof+0x474/0x760 mm/slub.c:5275
kmalloc_noprof include/linux/slab.h:954 [inline]
virtio_gpu_array_alloc+0x26/0xc0 drivers/gpu/drm/virtio/virtgpu_gem.c:170
virtio_gpu_update_dumb_bo drivers/gpu/drm/virtio/virtgpu_plane.c:171 [inline]
virtio_gpu_primary_plane_update+0x38d/0x13a0 drivers/gpu/drm/virtio/virtgpu_plane.c:265
drm_atomic_helper_commit_planes+0x60f/0xec0 drivers/gpu/drm/drm_atomic_helper.c:3038
drm_atomic_helper_commit_tail+0x5f/0x500 drivers/gpu/drm/drm_atomic_helper.c:1989
commit_tail+0x29a/0x3a0 drivers/gpu/drm/drm_atomic_helper.c:2074
drm_atomic_helper_commit+0xa6e/0xb10 drivers/gpu/drm/drm_atomic_helper.c:2312
drm_atomic_commit+0x246/0x2b0 drivers/gpu/drm/drm_atomic.c:1775
drm_atomic_helper_dirtyfb+0xdec/0xf80 drivers/gpu/drm/drm_damage_helper.c:183
drm_fbdev_shmem_helper_fb_dirty+0x160/0x2d0 drivers/gpu/drm/drm_fbdev_shmem.c:117
page last free pid 70 tgid 70 stack trace:
reset_page_owner include/linux/page_owner.h:25 [inline]
__free_pages_prepare mm/page_alloc.c:1433 [inline]
__free_frozen_pages+0xc2b/0xdb0 mm/page_alloc.c:2978
___free_pages_bulk mm/kasan/shadow.c:333 [inline]
__kasan_populate_vmalloc_do mm/kasan/shadow.c:385 [inline]
__kasan_populate_vmalloc+0x137/0x1d0 mm/kasan/shadow.c:424
kasan_populate_vmalloc include/linux/kasan.h:580 [inline]
alloc_vmap_area+0xd73/0x14b0 mm/vmalloc.c:2129
__get_vm_area_node+0x1f8/0x300 mm/vmalloc.c:3232
__vmalloc_node_range_noprof+0x372/0x1730 mm/vmalloc.c:4024
__vmalloc_node_noprof+0xc2/0x100 mm/vmalloc.c:4124
alloc_thread_stack_node kernel/fork.c:355 [inline]
dup_task_struct+0x228/0x9a0 kernel/fork.c:924
copy_process+0x508/0x3cf0 kernel/fork.c:2050
kernel_clone+0x248/0x8e0 kernel/fork.c:2654
user_mode_thread+0x110/0x180 kernel/fork.c:2730
call_usermodehelper_exec_work+0x5c/0x230 kernel/umh.c:171
process_one_work kernel/workqueue.c:3275 [inline]
process_scheduled_works+0xb02/0x1830 kernel/workqueue.c:3358
worker_thread+0xa50/0xfc0 kernel/workqueue.c:3439
kthread+0x388/0x470 kernel/kthread.c:436
ret_from_fork+0x51e/0xb90 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
Memory state around the buggy address:
ffff8880361a0200: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff8880361a0280: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
>ffff8880361a0300: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff8880361a0380: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
ffff8880361a0400: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================
Reply: The bug report describes a KASAN slab-use-after-free in `__bpf_trace_run`. Based on the provided stack traces and kernel source code analysis, the report is self-consistent and actionable.
### Analysis:
1. **Access Stack Trace**: The use-after-free occurs in `__bpf_trace_run` (inlined in `bpf_trace_run4`), which is called as a probe for the `mm_page_alloc` tracepoint. The access is at offset 24 (0x18) of the `link` object, which corresponds to the `prog` field of `struct bpf_link` (the base of `struct bpf_raw_tp_link`).
2. **Allocation Stack Trace**: The object was allocated in `bpf_raw_tp_link_attach`, which creates a `bpf_raw_tp_link` to attach a BPF program to a raw tracepoint.
3. **Free Stack Trace**: The object was freed via RCU (`rcu_do_batch`). The "Last potentially related work creation" shows that `call_rcu` was invoked from `bpf_link_put_direct` during the release of a BPF link (likely when its file descriptor was closed).
### Root Cause:
The underlying issue is a synchronization mismatch between the tracepoint subsystem and BPF link deallocation:
* **Tracepoint Protection**: In recent kernels (e.g., 6.11+), regular tracepoints (defined via `TRACE_EVENT`) use `tracepoint_srcu` (SRCU) to protect the execution of their probes.
* **BPF Link Deallocation**: When a `bpf_raw_tp_link` is released, it is first unregistered from the tracepoint. However, `tracepoint_probe_unregister` does not wait for a grace period. The BPF link infrastructure then defers the actual freeing of the `link` object using `call_rcu` (regular RCU) if the link is not marked as sleepable.
* **The Mismatch**: A regular RCU grace period (`call_rcu`) does not necessarily wait for an SRCU grace period (`tracepoint_srcu`) to complete, especially in `PREEMPT_RT` or `PREEMPT_FULL` configurations where SRCU-fast readers are not implicit RCU readers.
* **Result**: The `bpf_raw_tp_link` object is freed while a tracepoint caller (like `udevd` in the report) is still executing the BPF probe within an SRCU read-side critical section, leading to a use-after-free when the probe tries to access `link->link.prog`.
### Conclusion:
The report provides all the necessary information to identify this synchronization bug. It is a clear case of an object being freed after the wrong type of grace period. The report is highly actionable for kernel developers.
|
| 5/2 |
2026/03/15 01:22 |
llm |
expert |
0m
Model:
Tokens:
input: 7888
output: 46
thoughts: 740
Thoughts: **Analyzing a KASAN Slab-Use-After-Free in `__bpf_trace_run`**
Okay, so I'm looking at a KASAN slab-use-after-free, specifically in the `__bpf_trace_run` function. I need to piece together what's going on here.
First, the access stack trace gives me the entry point: It's `__bpf_trace_run`, which then calls `bpf_trace_run4`, and that gets triggered through `trace_mm_page_alloc`. So a BPF program is running because of the `mm_page_alloc` tracepoint. That's my starting point.
Next, I need to know what object is being freed. The allocation stack shows that the culprit is related to `bpf_raw_tp_link_attach`. That's a key piece of the puzzle. I'm thinking I need to find the link between the bpf trace and the `bpf_raw_tp_link`.
Then I see the free stack shows the object was freed using RCU through `rcu_core`. That's important. RCU can be tricky, especially with concurrent access, so that tells me to focus on possible race conditions.
Finally, looking at the last potentially related work creation I'm getting confirmation that the object in question is, in fact, a `bpf_link`. It's a `bpf_raw_tp_link` specifically, based on the allocation call. Now I need to check the link between the `bpf_trace_run` and the `bpf_raw_tp_link` and look for an issue.
Now my next step is examining the code at `kernel/trace/bpf_trace.c:2075` to pinpoint the exact location of the potential use-after-free and, hopefully, figure out why it's happening in this tracepoint context.
|
| 6/2 |
2026/03/15 01:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/trace/bpf_trace.c IncludeLines:true Name:__bpf_trace_run]
Results: map[SourceCode:2072: static __always_inline
2073: void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)
2074: {
2075: struct bpf_prog *prog = link->link.prog;
2076: struct bpf_run_ctx *old_run_ctx;
2077: struct bpf_trace_run_ctx run_ctx;
2078:
2079: rcu_read_lock_dont_migrate();
2080: if (unlikely(!bpf_prog_get_recursion_context(prog))) {
2081: bpf_prog_inc_misses_counter(prog);
2082: goto out;
2083: }
2084:
2085: run_ctx.bpf_cookie = link->cookie;
2086: old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
2087:
2088: (void) bpf_prog_run(prog, args);
2089:
2090: bpf_reset_run_ctx(old_run_ctx);
2091: out:
2092: bpf_prog_put_recursion_context(prog);
2093: rcu_read_unlock_migrate();
2094: }
SourceFile:kernel/trace/bpf_trace.c]
|
| 7/2 |
2026/03/15 01:22 |
llm |
expert |
0m
Model:
Tokens:
input: 9498
output: 46
thoughts: 89
Thoughts: Okay, here's my take on those lines of code, as if it's my internal thought process.
**Analyzing a Potential Use-After-Free in BPF Tracepoints**
Right, so I've been digging through this BPF code, and it looks like I've found a potential Use-After-Free (UAF) issue. Specifically, I'm focusing on line 2075, which reads `struct bpf_prog *prog = link->link.prog;`. The key thing here is the `link` variable β it's a `struct bpf_raw_tp_link *`. The way I'm reading this, there's a problem when the code tries to access `link->link.prog`. It's a classic smell, a pointer dereference that could go horribly wrong if the `prog` pointer has been freed or is no longer valid. I need to figure out where that `link` object is coming from, and the lifecycle of that `prog` pointer, to confirm this.
Now, I need to check where `bpf_trace_run4` is. It's in `kernel/trace/bpf_trace.c:2131`. This is where the action's happening. I'm going to look at the surrounding code to determine how this `link` variable is being initialized, and when/where the referenced `prog` pointer might be freed, invalidated, or overwritten. This is where the real work begins to trace down the root cause. My expertise is that the pointer and memory-management should be checked.
|
| 8/2 |
2026/03/15 01:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/trace/bpf_trace.c IncludeLines:true Name:bpf_trace_run4]
Results: map[SourceCode:2131: BPF_TRACE_DEFN_x(4);
SourceFile:kernel/trace/bpf_trace.c]
|
| 9/2 |
2026/03/15 01:22 |
llm |
expert |
0m
Model:
Tokens:
input: 10024
output: 25
thoughts: 24
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Analyzing the Macro: BPF_TRACE_DEFN_x**
Right, so I'm looking at this code and immediately see a macro. My first thought is always to understand the abstraction. Specifically, I need to know what `BPF_TRACE_DEFN_x` is expanding into. Is it a function call? A struct declaration? A complex set of preprocessor directives? The "x" is probably a placeholder so there are several of these definitions. I need to get into the definition itself to assess how it's setting things up. What's it doing with those BPF helpers? Is it hooking into probes? And what about the data layout and arguments? Understanding this macro is key to understanding the larger BPF program logic.
|
| 10/2 |
2026/03/15 01:22 |
tool |
grepper |
0m
Args: map[Expression:#define BPF_TRACE_DEFN_x]
Results: map[Output:kernel/trace/bpf_trace.c=2073=void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)
--
kernel/trace/bpf_trace.c-2118-
kernel/trace/bpf_trace.c:2119:#define BPF_TRACE_DEFN_x(x) \
kernel/trace/bpf_trace.c-2120- void bpf_trace_run##x(struct bpf_raw_tp_link *link, \
]
|
| 11/2 |
2026/03/15 01:22 |
llm |
expert |
0m
Model:
Tokens:
input: 10397
output: 25
thoughts: 19
Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience:
**Diving into the `BPF_TRACE_DEFN_x` Macro**
Okay, so I need to get a really solid grasp on what's going on with these `BPF_TRACE_DEFN_x` macros. They're fundamental to how we define and structure BPF tracepoints, and a loose understanding is simply not going to cut it. I need the *full* definition here β no abstractions, no paraphrasing. I want to see the nitty-gritty details. What's it expanding to? Are there any hidden preprocessor tricks or conditional compilations I need to be aware of? Are there platform-specific considerations encoded within it? Any interaction with other core BPF infrastructure components? Time to get down to the actual code. Let's see it.
|
| 12/2 |
2026/03/15 01:22 |
tool |
read-file |
0m
Args: map[File:kernel/trace/bpf_trace.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2011-2015 PLUMgrid, http://plumgrid.com
* Copyright (c) 2016 Facebook
*/
#include <linux/kernel.h>
#include <linux/types.h>
#include <linux/slab.h>
#include <linux/bpf.h>
#include <linux/bpf_verifier.h>
#include <linux/bpf_perf_event.h>
#include <linux/btf.h>
#include <linux/filter.h>
#include <linux/uaccess.h>
#include <linux/ctype.h>
#include <linux/kprobes.h>
#include <linux/spinlock.h>
#include <linux/syscalls.h>
#include <linux/error-injection.h>
#include <linux/btf_ids.h>
#include <linux/bpf_lsm.h>
#include <linux/fprobe.h>
#include <linux/bsearch.h>
#include <linux/sort.h>
#include <linux/key.h>
#include <linux/namei.h>
#include <net/bpf_sk_storage.h>
#include <uapi/linux/bpf.h>
#include <uapi/linux/btf.h>
#include <asm/tlb.h>
#include "trace_probe.h"
#include "trace.h"
#define CREATE_TRACE_POINTS
#include "bpf_trace.h"
#define bpf_event_rcu_dereference(p) \
rcu_dereference_protected(p, lockdep_is_held(&bpf_event_mutex))
#define MAX_UPROBE_MULTI_CNT (1U << 20)
#define MAX_KPROBE_MULTI_CNT (1U << 20)
#ifdef CONFIG_MODULES
struct bpf_trace_module {
struct module *module;
struct list_head list;
};
static LIST_HEAD(bpf_trace_modules);
static DEFINE_MUTEX(bpf_module_mutex);
static struct bpf_raw_event_map *bpf_get_raw_tracepoint_module(const char *name)
{
struct bpf_raw_event_map *btp, *ret = NULL;
struct bpf_trace_module *btm;
unsigned int i;
mutex_lock(&bpf_module_mutex);
list_for_each_entry(btm, &bpf_trace_modules, list) {
for (i = 0; i < btm->module->num_bpf_raw_events; ++i) {
btp = &btm->module->bpf_raw_events[i];
if (!strcmp(btp->tp->name, name)) {
if (try_module_get(btm->module))
ret = btp;
goto out;
}
}
}
out:
mutex_unlock(&bpf_module_mutex);
return ret;
}
#else
static struct bpf_raw_event_map *bpf_get_raw_tracepoint_module(const char *name)
{
return NULL;
}
#endif /* CONFIG_MODULES */
u64 bpf_get_stackid(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
u64 bpf_get_stack(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
u64 flags, const struct btf **btf,
s32 *btf_id);
static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx);
static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx);
static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx);
static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx);
/**
* trace_call_bpf - invoke BPF program
* @call: tracepoint event
* @ctx: opaque context pointer
*
* kprobe handlers execute BPF programs via this helper.
* Can be used from static tracepoints in the future.
*
* Return: BPF programs always return an integer which is interpreted by
* kprobe handler as:
* 0 - return from kprobe (event is filtered out)
* 1 - store kprobe event into ring buffer
* Other values are reserved and currently alias to 1
*/
unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx)
{
unsigned int ret;
cant_sleep();
if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) {
/*
* since some bpf program is already running on this cpu,
* don't call into another bpf program (same or different)
* and don't send kprobe event into ring-buffer,
* so return zero here
*/
rcu_read_lock();
bpf_prog_inc_misses_counters(rcu_dereference(call->prog_array));
rcu_read_unlock();
ret = 0;
goto out;
}
/*
* Instead of moving rcu_read_lock/rcu_dereference/rcu_read_unlock
* to all call sites, we did a bpf_prog_array_valid() there to check
* whether call->prog_array is empty or not, which is
* a heuristic to speed up execution.
*
* If bpf_prog_array_valid() fetched prog_array was
* non-NULL, we go into trace_call_bpf() and do the actual
* proper rcu_dereference() under RCU lock.
* If it turns out that prog_array is NULL then, we bail out.
* For the opposite, if the bpf_prog_array_valid() fetched pointer
* was NULL, you'll skip the prog_array with the risk of missing
* out of events when it was updated in between this and the
* rcu_dereference() which is accepted risk.
*/
rcu_read_lock();
ret = bpf_prog_run_array(rcu_dereference(call->prog_array),
ctx, bpf_prog_run);
rcu_read_unlock();
out:
__this_cpu_dec(bpf_prog_active);
return ret;
}
#ifdef CONFIG_BPF_KPROBE_OVERRIDE
BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc)
{
regs_set_return_value(regs, rc);
override_function_with_return(regs);
return 0;
}
static const struct bpf_func_proto bpf_override_return_proto = {
.func = bpf_override_return,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_ANYTHING,
};
#endif
static __always_inline int
bpf_probe_read_user_common(void *dst, u32 size, const void __user *unsafe_ptr)
{
int ret;
ret = copy_from_user_nofault(dst, unsafe_ptr, size);
if (unlikely(ret < 0))
memset(dst, 0, size);
return ret;
}
BPF_CALL_3(bpf_probe_read_user, void *, dst, u32, size,
const void __user *, unsafe_ptr)
{
return bpf_probe_read_user_common(dst, size, unsafe_ptr);
}
const struct bpf_func_proto bpf_probe_read_user_proto = {
.func = bpf_probe_read_user,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
static __always_inline int
bpf_probe_read_user_str_common(void *dst, u32 size,
const void __user *unsafe_ptr)
{
int ret;
/*
* NB: We rely on strncpy_from_user() not copying junk past the NUL
* terminator into `dst`.
*
* strncpy_from_user() does long-sized strides in the fast path. If the
* strncpy does not mask out the bytes after the NUL in `unsafe_ptr`,
* then there could be junk after the NUL in `dst`. If user takes `dst`
* and keys a hash map with it, then semantically identical strings can
* occupy multiple entries in the map.
*/
ret = strncpy_from_user_nofault(dst, unsafe_ptr, size);
if (unlikely(ret < 0))
memset(dst, 0, size);
return ret;
}
BPF_CALL_3(bpf_probe_read_user_str, void *, dst, u32, size,
const void __user *, unsafe_ptr)
{
return bpf_probe_read_user_str_common(dst, size, unsafe_ptr);
}
const struct bpf_func_proto bpf_probe_read_user_str_proto = {
.func = bpf_probe_read_user_str,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_probe_read_kernel, void *, dst, u32, size,
const void *, unsafe_ptr)
{
return bpf_probe_read_kernel_common(dst, size, unsafe_ptr);
}
const struct bpf_func_proto bpf_probe_read_kernel_proto = {
.func = bpf_probe_read_kernel,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
static __always_inline int
bpf_probe_read_kernel_str_common(void *dst, u32 size, const void *unsafe_ptr)
{
int ret;
/*
* The strncpy_from_kernel_nofault() call will likely not fill the
* entire buffer, but that's okay in this circumstance as we're probing
* arbitrary memory anyway similar to bpf_probe_read_*() and might
* as well probe the stack. Thus, memory is explicitly cleared
* only in error case, so that improper users ignoring return
* code altogether don't copy garbage; otherwise length of string
* is returned that can be used for bpf_perf_event_output() et al.
*/
ret = strncpy_from_kernel_nofault(dst, unsafe_ptr, size);
if (unlikely(ret < 0))
memset(dst, 0, size);
return ret;
}
BPF_CALL_3(bpf_probe_read_kernel_str, void *, dst, u32, size,
const void *, unsafe_ptr)
{
return bpf_probe_read_kernel_str_common(dst, size, unsafe_ptr);
}
const struct bpf_func_proto bpf_probe_read_kernel_str_proto = {
.func = bpf_probe_read_kernel_str,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
BPF_CALL_3(bpf_probe_read_compat, void *, dst, u32, size,
const void *, unsafe_ptr)
{
if ((unsigned long)unsafe_ptr < TASK_SIZE) {
return bpf_probe_read_user_common(dst, size,
(__force void __user *)unsafe_ptr);
}
return bpf_probe_read_kernel_common(dst, size, unsafe_ptr);
}
static const struct bpf_func_proto bpf_probe_read_compat_proto = {
.func = bpf_probe_read_compat,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_probe_read_compat_str, void *, dst, u32, size,
const void *, unsafe_ptr)
{
if ((unsigned long)unsafe_ptr < TASK_SIZE) {
return bpf_probe_read_user_str_common(dst, size,
(__force void __user *)unsafe_ptr);
}
return bpf_probe_read_kernel_str_common(dst, size, unsafe_ptr);
}
static const struct bpf_func_proto bpf_probe_read_compat_str_proto = {
.func = bpf_probe_read_compat_str,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
#endif /* CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE */
BPF_CALL_3(bpf_probe_write_user, void __user *, unsafe_ptr, const void *, src,
u32, size)
{
/*
* Ensure we're in user context which is safe for the helper to
* run. This helper has no business in a kthread.
*
* access_ok() should prevent writing to non-user memory, but in
* some situations (nommu, temporary switch, etc) access_ok() does
* not provide enough validation, hence the check on KERNEL_DS.
*
* nmi_uaccess_okay() ensures the probe is not run in an interim
* state, when the task or mm are switched. This is specifically
* required to prevent the use of temporary mm.
*/
if (unlikely(in_interrupt() ||
current->flags & (PF_KTHREAD | PF_EXITING)))
return -EPERM;
if (unlikely(!nmi_uaccess_okay()))
return -EPERM;
return copy_to_user_nofault(unsafe_ptr, src, size);
}
static const struct bpf_func_proto bpf_probe_write_user_proto = {
.func = bpf_probe_write_user,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
};
#define MAX_TRACE_PRINTK_VARARGS 3
#define BPF_TRACE_PRINTK_SIZE 1024
BPF_CALL_5(bpf_trace_printk, char *, fmt, u32, fmt_size, u64, arg1,
u64, arg2, u64, arg3)
{
u64 args[MAX_TRACE_PRINTK_VARARGS] = { arg1, arg2, arg3 };
struct bpf_bprintf_data data = {
.get_bin_args = true,
.get_buf = true,
};
int ret;
ret = bpf_bprintf_prepare(fmt, fmt_size, args,
MAX_TRACE_PRINTK_VARARGS, &data);
if (ret < 0)
return ret;
ret = bstr_printf(data.buf, MAX_BPRINTF_BUF, fmt, data.bin_args);
trace_bpf_trace_printk(data.buf);
bpf_bprintf_cleanup(&data);
return ret;
}
static const struct bpf_func_proto bpf_trace_printk_proto = {
.func = bpf_trace_printk,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
};
static void __set_printk_clr_event(struct work_struct *work)
{
/*
* This program might be calling bpf_trace_printk,
* so enable the associated bpf_trace/bpf_trace_printk event.
* Repeat this each time as it is possible a user has
* disabled bpf_trace_printk events. By loading a program
* calling bpf_trace_printk() however the user has expressed
* the intent to see such events.
*/
if (trace_set_clr_event("bpf_trace", "bpf_trace_printk", 1))
pr_warn_ratelimited("could not enable bpf_trace_printk events");
}
static DECLARE_WORK(set_printk_work, __set_printk_clr_event);
const struct bpf_func_proto *bpf_get_trace_printk_proto(void)
{
schedule_work(&set_printk_work);
return &bpf_trace_printk_proto;
}
BPF_CALL_4(bpf_trace_vprintk, char *, fmt, u32, fmt_size, const void *, args,
u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
.get_buf = true,
};
int ret, num_args;
if (data_len & 7 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
ret = bpf_bprintf_prepare(fmt, fmt_size, args, num_args, &data);
if (ret < 0)
return ret;
ret = bstr_printf(data.buf, MAX_BPRINTF_BUF, fmt, data.bin_args);
trace_bpf_trace_printk(data.buf);
bpf_bprintf_cleanup(&data);
return ret;
}
static const struct bpf_func_proto bpf_trace_vprintk_proto = {
.func = bpf_trace_vprintk,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
};
const struct bpf_func_proto *bpf_get_trace_vprintk_proto(void)
{
schedule_work(&set_printk_work);
return &bpf_trace_vprintk_proto;
}
BPF_CALL_5(bpf_seq_printf, struct seq_file *, m, char *, fmt, u32, fmt_size,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len & 7 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
err = bpf_bprintf_prepare(fmt, fmt_size, args, num_args, &data);
if (err < 0)
return err;
seq_bprintf(m, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return seq_has_overflowed(m) ? -EOVERFLOW : 0;
}
BTF_ID_LIST_SINGLE(btf_seq_file_ids, struct, seq_file)
static const struct bpf_func_proto bpf_seq_printf_proto = {
.func = bpf_seq_printf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_seq_file_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len)
{
return seq_write(m, data, len) ? -EOVERFLOW : 0;
}
static const struct bpf_func_proto bpf_seq_write_proto = {
.func = bpf_seq_write,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_seq_file_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_4(bpf_seq_printf_btf, struct seq_file *, m, struct btf_ptr *, ptr,
u32, btf_ptr_size, u64, flags)
{
const struct btf *btf;
s32 btf_id;
int ret;
ret = bpf_btf_printf_prepare(ptr, btf_ptr_size, flags, &btf, &btf_id);
if (ret)
return ret;
return btf_type_seq_show_flags(btf, btf_id, ptr->ptr, m, flags);
}
static const struct bpf_func_proto bpf_seq_printf_btf_proto = {
.func = bpf_seq_printf_btf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_seq_file_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
static __always_inline int
get_map_perf_counter(struct bpf_map *map, u64 flags,
u64 *value, u64 *enabled, u64 *running)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
unsigned int cpu = smp_processor_id();
u64 index = flags & BPF_F_INDEX_MASK;
struct bpf_event_entry *ee;
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
if (index == BPF_F_CURRENT_CPU)
index = cpu;
if (unlikely(index >= array->map.max_entries))
return -E2BIG;
ee = READ_ONCE(array->ptrs[index]);
if (!ee)
return -ENOENT;
return perf_event_read_local(ee->event, value, enabled, running);
}
BPF_CALL_2(bpf_perf_event_read, struct bpf_map *, map, u64, flags)
{
u64 value = 0;
int err;
err = get_map_perf_counter(map, flags, &value, NULL, NULL);
/*
* this api is ugly since we miss [-22..-2] range of valid
* counter values, but that's uapi
*/
if (err)
return err;
return value;
}
const struct bpf_func_proto bpf_perf_event_read_proto = {
.func = bpf_perf_event_read,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
};
BPF_CALL_4(bpf_perf_event_read_value, struct bpf_map *, map, u64, flags,
struct bpf_perf_event_value *, buf, u32, size)
{
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_perf_event_value)))
goto clear;
err = get_map_perf_counter(map, flags, &buf->counter, &buf->enabled,
&buf->running);
if (unlikely(err))
goto clear;
return 0;
clear:
memset(buf, 0, size);
return err;
}
static const struct bpf_func_proto bpf_perf_event_read_value_proto = {
.func = bpf_perf_event_read_value,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
};
const struct bpf_func_proto *bpf_get_perf_event_read_value_proto(void)
{
return &bpf_perf_event_read_value_proto;
}
static __always_inline u64
__bpf_perf_event_output(struct pt_regs *regs, struct bpf_map *map,
u64 flags, struct perf_raw_record *raw,
struct perf_sample_data *sd)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
unsigned int cpu = smp_processor_id();
u64 index = flags & BPF_F_INDEX_MASK;
struct bpf_event_entry *ee;
struct perf_event *event;
if (index == BPF_F_CURRENT_CPU)
index = cpu;
if (unlikely(index >= array->map.max_entries))
return -E2BIG;
ee = READ_ONCE(array->ptrs[index]);
if (!ee)
return -ENOENT;
event = ee->event;
if (unlikely(event->attr.type != PERF_TYPE_SOFTWARE ||
event->attr.config != PERF_COUNT_SW_BPF_OUTPUT))
return -EINVAL;
if (unlikely(event->oncpu != cpu))
return -EOPNOTSUPP;
perf_sample_save_raw_data(sd, event, raw);
return perf_event_output(event, sd, regs);
}
/*
* Support executing tracepoints in normal, irq, and nmi context that each call
* bpf_perf_event_output
*/
struct bpf_trace_sample_data {
struct perf_sample_data sds[3];
};
static DEFINE_PER_CPU(struct bpf_trace_sample_data, bpf_trace_sds);
static DEFINE_PER_CPU(int, bpf_trace_nest_level);
BPF_CALL_5(bpf_perf_event_output, struct pt_regs *, regs, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
struct bpf_trace_sample_data *sds;
struct perf_raw_record raw = {
.frag = {
.size = size,
.data = data,
},
};
struct perf_sample_data *sd;
int nest_level, err;
preempt_disable();
sds = this_cpu_ptr(&bpf_trace_sds);
nest_level = this_cpu_inc_return(bpf_trace_nest_level);
if (WARN_ON_ONCE(nest_level > ARRAY_SIZE(sds->sds))) {
err = -EBUSY;
goto out;
}
sd = &sds->sds[nest_level - 1];
if (unlikely(flags & ~(BPF_F_INDEX_MASK))) {
err = -EINVAL;
goto out;
}
perf_sample_data_init(sd, 0, 0);
err = __bpf_perf_event_output(regs, map, flags, &raw, sd);
out:
this_cpu_dec(bpf_trace_nest_level);
preempt_enable();
return err;
}
static const struct bpf_func_proto bpf_perf_event_output_proto = {
.func = bpf_perf_event_output,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
static DEFINE_PER_CPU(int, bpf_event_output_nest_level);
struct bpf_nested_pt_regs {
struct pt_regs regs[3];
};
static DEFINE_PER_CPU(struct bpf_nested_pt_regs, bpf_pt_regs);
static DEFINE_PER_CPU(struct bpf_trace_sample_data, bpf_misc_sds);
u64 bpf_event_output(struct bpf_map *map, u64 flags, void *meta, u64 meta_size,
void *ctx, u64 ctx_size, bpf_ctx_copy_t ctx_copy)
{
struct perf_raw_frag frag = {
.copy = ctx_copy,
.size = ctx_size,
.data = ctx,
};
struct perf_raw_record raw = {
.frag = {
{
.next = ctx_size ? &frag : NULL,
},
.size = meta_size,
.data = meta,
},
};
struct perf_sample_data *sd;
struct pt_regs *regs;
int nest_level;
u64 ret;
preempt_disable();
nest_level = this_cpu_inc_return(bpf_event_output_nest_level);
if (WARN_ON_ONCE(nest_level > ARRAY_SIZE(bpf_misc_sds.sds))) {
ret = -EBUSY;
goto out;
}
sd = this_cpu_ptr(&bpf_misc_sds.sds[nest_level - 1]);
regs = this_cpu_ptr(&bpf_pt_regs.regs[nest_level - 1]);
perf_fetch_caller_regs(regs);
perf_sample_data_init(sd, 0, 0);
ret = __bpf_perf_event_output(regs, map, flags, &raw, sd);
out:
this_cpu_dec(bpf_event_output_nest_level);
preempt_enable();
return ret;
}
BPF_CALL_0(bpf_get_current_task)
{
return (long) current;
}
const struct bpf_func_proto bpf_get_current_task_proto = {
.func = bpf_get_current_task,
.gpl_only = true,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_task_btf)
{
return (unsigned long) current;
}
const struct bpf_func_proto bpf_get_current_task_btf_proto = {
.func = bpf_get_current_task_btf,
.gpl_only = true,
.ret_type = RET_PTR_TO_BTF_ID_TRUSTED,
.ret_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
};
BPF_CALL_1(bpf_task_pt_regs, struct task_struct *, task)
{
return (unsigned long) task_pt_regs(task);
}
BTF_ID_LIST_SINGLE(bpf_task_pt_regs_ids, struct, pt_regs)
const struct bpf_func_proto bpf_task_pt_regs_proto = {
.func = bpf_task_pt_regs,
.gpl_only = true,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.ret_type = RET_PTR_TO_BTF_ID,
.ret_btf_id = &bpf_task_pt_regs_ids[0],
};
struct send_signal_irq_work {
struct irq_work irq_work;
struct task_struct *task;
u32 sig;
enum pid_type type;
bool has_siginfo;
struct kernel_siginfo info;
};
static DEFINE_PER_CPU(struct send_signal_irq_work, send_signal_work);
static void do_bpf_send_signal(struct irq_work *entry)
{
struct send_signal_irq_work *work;
struct kernel_siginfo *siginfo;
work = container_of(entry, struct send_signal_irq_work, irq_work);
siginfo = work->has_siginfo ? &work->info : SEND_SIG_PRIV;
group_send_sig_info(work->sig, siginfo, work->task, work->type);
put_task_struct(work->task);
}
static int bpf_send_signal_common(u32 sig, enum pid_type type, struct task_struct *task, u64 value)
{
struct send_signal_irq_work *work = NULL;
struct kernel_siginfo info;
struct kernel_siginfo *siginfo;
if (!task) {
task = current;
siginfo = SEND_SIG_PRIV;
} else {
clear_siginfo(&info);
info.si_signo = sig;
info.si_errno = 0;
info.si_code = SI_KERNEL;
info.si_pid = 0;
info.si_uid = 0;
info.si_value.sival_ptr = (void __user __force *)(unsigned long)value;
siginfo = &info;
}
/* Similar to bpf_probe_write_user, task needs to be
* in a sound condition and kernel memory access be
* permitted in order to send signal to the current
* task.
*/
if (unlikely(task->flags & (PF_KTHREAD | PF_EXITING)))
return -EPERM;
if (unlikely(!nmi_uaccess_okay()))
return -EPERM;
/* Task should not be pid=1 to avoid kernel panic. */
if (unlikely(is_global_init(task)))
return -EPERM;
if (preempt_count() != 0 || irqs_disabled()) {
/* Do an early check on signal validity. Otherwise,
* the error is lost in deferred irq_work.
*/
if (unlikely(!valid_signal(sig)))
return -EINVAL;
work = this_cpu_ptr(&send_signal_work);
if (irq_work_is_busy(&work->irq_work))
return -EBUSY;
/* Add the current task, which is the target of sending signal,
* to the irq_work. The current task may change when queued
* irq works get executed.
*/
work->task = get_task_struct(task);
work->has_siginfo = siginfo == &info;
if (work->has_siginfo)
copy_siginfo(&work->info, &info);
work->sig = sig;
work->type = type;
irq_work_queue(&work->irq_work);
return 0;
}
return group_send_sig_info(sig, siginfo, task, type);
}
BPF_CALL_1(bpf_send_signal, u32, sig)
{
return bpf_send_signal_common(sig, PIDTYPE_TGID, NULL, 0);
}
const struct bpf_func_proto bpf_send_signal_proto = {
.func = bpf_send_signal,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_send_signal_thread, u32, sig)
{
return bpf_send_signal_common(sig, PIDTYPE_PID, NULL, 0);
}
const struct bpf_func_proto bpf_send_signal_thread_proto = {
.func = bpf_send_signal_thread,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_d_path, const struct path *, path, char *, buf, u32, sz)
{
struct path copy;
long len;
char *p;
if (!sz)
return 0;
/*
* The path pointer is verified as trusted and safe to use,
* but let's double check it's valid anyway to workaround
* potentially broken verifier.
*/
len = copy_from_kernel_nofault(©, path, sizeof(*path));
if (len < 0)
return len;
p = d_path(©, buf, sz);
if (IS_ERR(p)) {
len = PTR_ERR(p);
} else {
len = buf + sz - p;
memmove(buf, p, len);
}
return len;
}
BTF_SET_START(btf_allowlist_d_path)
#ifdef CONFIG_SECURITY
BTF_ID(func, security_file_permission)
BTF_ID(func, security_inode_getattr)
BTF_ID(func, security_file_open)
#endif
#ifdef CONFIG_SECURITY_PATH
BTF_ID(func, security_path_truncate)
#endif
BTF_ID(func, vfs_truncate)
BTF_ID(func, vfs_fallocate)
BTF_ID(func, dentry_open)
BTF_ID(func, vfs_getattr)
BTF_ID(func, filp_close)
BTF_SET_END(btf_allowlist_d_path)
static bool bpf_d_path_allowed(const struct bpf_prog *prog)
{
if (prog->type == BPF_PROG_TYPE_TRACING &&
prog->expected_attach_type == BPF_TRACE_ITER)
return true;
if (prog->type == BPF_PROG_TYPE_LSM)
return bpf_lsm_is_sleepable_hook(prog->aux->attach_btf_id);
return btf_id_set_contains(&btf_allowlist_d_path,
prog->aux->attach_btf_id);
}
BTF_ID_LIST_SINGLE(bpf_d_path_btf_ids, struct, path)
static const struct bpf_func_proto bpf_d_path_proto = {
.func = bpf_d_path,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &bpf_d_path_btf_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_WRITE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.allowed = bpf_d_path_allowed,
};
#define BTF_F_ALL (BTF_F_COMPACT | BTF_F_NONAME | \
BTF_F_PTR_RAW | BTF_F_ZERO)
static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
u64 flags, const struct btf **btf,
s32 *btf_id)
{
const struct btf_type *t;
if (unlikely(flags & ~(BTF_F_ALL)))
return -EINVAL;
if (btf_ptr_size != sizeof(struct btf_ptr))
return -EINVAL;
*btf = bpf_get_btf_vmlinux();
if (IS_ERR_OR_NULL(*btf))
return IS_ERR(*btf) ? PTR_ERR(*btf) : -EINVAL;
if (ptr->type_id > 0)
*btf_id = ptr->type_id;
else
return -EINVAL;
if (*btf_id > 0)
t = btf_type_by_id(*btf, *btf_id);
if (*btf_id <= 0 || !t)
return -ENOENT;
return 0;
}
BPF_CALL_5(bpf_snprintf_btf, char *, str, u32, str_size, struct btf_ptr *, ptr,
u32, btf_ptr_size, u64, flags)
{
const struct btf *btf;
s32 btf_id;
int ret;
ret = bpf_btf_printf_prepare(ptr, btf_ptr_size, flags, &btf, &btf_id);
if (ret)
return ret;
return btf_type_snprintf_show(btf, btf_id, ptr->ptr, str, str_size,
flags);
}
const struct bpf_func_proto bpf_snprintf_btf_proto = {
.func = bpf_snprintf_btf,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE,
.arg5_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_get_func_ip_tracing, void *, ctx)
{
/* This helper call is inlined by verifier. */
return ((u64 *)ctx)[-2];
}
static const struct bpf_func_proto bpf_get_func_ip_proto_tracing = {
.func = bpf_get_func_ip_tracing,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
static inline unsigned long get_entry_ip(unsigned long fentry_ip)
{
#ifdef CONFIG_X86_KERNEL_IBT
if (is_endbr((void *)(fentry_ip - ENDBR_INSN_SIZE)))
fentry_ip -= ENDBR_INSN_SIZE;
#endif
return fentry_ip;
}
BPF_CALL_1(bpf_get_func_ip_kprobe, struct pt_regs *, regs)
{
struct bpf_trace_run_ctx *run_ctx __maybe_unused;
struct kprobe *kp;
#ifdef CONFIG_UPROBES
run_ctx = container_of(current->bpf_ctx, struct bpf_trace_run_ctx, run_ctx);
if (run_ctx->is_uprobe)
return ((struct uprobe_dispatch_data *)current->utask->vaddr)->bp_addr;
#endif
kp = kprobe_running();
if (!kp || !(kp->flags & KPROBE_FLAG_ON_FUNC_ENTRY))
return 0;
return get_entry_ip((uintptr_t)kp->addr);
}
static const struct bpf_func_proto bpf_get_func_ip_proto_kprobe = {
.func = bpf_get_func_ip_kprobe,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_func_ip_kprobe_multi, struct pt_regs *, regs)
{
return bpf_kprobe_multi_entry_ip(current->bpf_ctx);
}
static const struct bpf_func_proto bpf_get_func_ip_proto_kprobe_multi = {
.func = bpf_get_func_ip_kprobe_multi,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_attach_cookie_kprobe_multi, struct pt_regs *, regs)
{
return bpf_kprobe_multi_cookie(current->bpf_ctx);
}
static const struct bpf_func_proto bpf_get_attach_cookie_proto_kmulti = {
.func = bpf_get_attach_cookie_kprobe_multi,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_func_ip_uprobe_multi, struct pt_regs *, regs)
{
return bpf_uprobe_multi_entry_ip(current->bpf_ctx);
}
static const struct bpf_func_proto bpf_get_func_ip_proto_uprobe_multi = {
.func = bpf_get_func_ip_uprobe_multi,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_attach_cookie_uprobe_multi, struct pt_regs *, regs)
{
return bpf_uprobe_multi_cookie(current->bpf_ctx);
}
static const struct bpf_func_proto bpf_get_attach_cookie_proto_umulti = {
.func = bpf_get_attach_cookie_uprobe_multi,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_attach_cookie_trace, void *, ctx)
{
struct bpf_trace_run_ctx *run_ctx;
run_ctx = container_of(current->bpf_ctx, struct bpf_trace_run_ctx, run_ctx);
return run_ctx->bpf_cookie;
}
static const struct bpf_func_proto bpf_get_attach_cookie_proto_trace = {
.func = bpf_get_attach_cookie_trace,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_attach_cookie_pe, struct bpf_perf_event_data_kern *, ctx)
{
return ctx->event->bpf_cookie;
}
static const struct bpf_func_proto bpf_get_attach_cookie_proto_pe = {
.func = bpf_get_attach_cookie_pe,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_attach_cookie_tracing, void *, ctx)
{
struct bpf_trace_run_ctx *run_ctx;
run_ctx = container_of(current->bpf_ctx, struct bpf_trace_run_ctx, run_ctx);
return run_ctx->bpf_cookie;
}
static const struct bpf_func_proto bpf_get_attach_cookie_proto_tracing = {
.func = bpf_get_attach_cookie_tracing,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_3(bpf_get_branch_snapshot, void *, buf, u32, size, u64, flags)
{
static const u32 br_entry_size = sizeof(struct perf_branch_entry);
u32 entry_cnt = size / br_entry_size;
entry_cnt = static_call(perf_snapshot_branch_stack)(buf, entry_cnt);
if (unlikely(flags))
return -EINVAL;
if (!entry_cnt)
return -ENOENT;
return entry_cnt * br_entry_size;
}
const struct bpf_func_proto bpf_get_branch_snapshot_proto = {
.func = bpf_get_branch_snapshot,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(get_func_arg, void *, ctx, u32, n, u64 *, value)
{
/* This helper call is inlined by verifier. */
u64 nr_args = ((u64 *)ctx)[-1] & 0xFF;
if ((u64) n >= nr_args)
return -EINVAL;
*value = ((u64 *)ctx)[n];
return 0;
}
static const struct bpf_func_proto bpf_get_func_arg_proto = {
.func = get_func_arg,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg3_size = sizeof(u64),
};
BPF_CALL_2(get_func_ret, void *, ctx, u64 *, value)
{
/* This helper call is inlined by verifier. */
u64 nr_args = ((u64 *)ctx)[-1] & 0xFF;
*value = ((u64 *)ctx)[nr_args];
return 0;
}
static const struct bpf_func_proto bpf_get_func_ret_proto = {
.func = get_func_ret,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg2_size = sizeof(u64),
};
BPF_CALL_1(get_func_arg_cnt, void *, ctx)
{
/* This helper call is inlined by verifier. */
return ((u64 *)ctx)[-1] & 0xFF;
}
static const struct bpf_func_proto bpf_get_func_arg_cnt_proto = {
.func = get_func_arg_cnt,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
static const struct bpf_func_proto *
bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
const struct bpf_func_proto *func_proto;
switch (func_id) {
case BPF_FUNC_get_smp_processor_id:
return &bpf_get_smp_processor_id_proto;
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
case BPF_FUNC_probe_read:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_compat_proto;
case BPF_FUNC_probe_read_str:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_compat_str_proto;
#endif
case BPF_FUNC_get_func_ip:
return &bpf_get_func_ip_proto_tracing;
default:
break;
}
func_proto = bpf_base_func_proto(func_id, prog);
if (func_proto)
return func_proto;
if (!bpf_token_capable(prog->aux->token, CAP_SYS_ADMIN))
return NULL;
switch (func_id) {
case BPF_FUNC_probe_write_user:
return security_locked_down(LOCKDOWN_BPF_WRITE_USER) < 0 ?
NULL : &bpf_probe_write_user_proto;
default:
return NULL;
}
}
static bool is_kprobe_multi(const struct bpf_prog *prog)
{
return prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI ||
prog->expected_attach_type == BPF_TRACE_KPROBE_SESSION;
}
static inline bool is_kprobe_session(const struct bpf_prog *prog)
{
return prog->type == BPF_PROG_TYPE_KPROBE &&
prog->expected_attach_type == BPF_TRACE_KPROBE_SESSION;
}
static inline bool is_uprobe_multi(const struct bpf_prog *prog)
{
return prog->expected_attach_type == BPF_TRACE_UPROBE_MULTI ||
prog->expected_attach_type == BPF_TRACE_UPROBE_SESSION;
}
static inline bool is_uprobe_session(const struct bpf_prog *prog)
{
return prog->type == BPF_PROG_TYPE_KPROBE &&
prog->expected_attach_type == BPF_TRACE_UPROBE_SESSION;
}
static inline bool is_trace_fsession(const struct bpf_prog *prog)
{
return prog->type == BPF_PROG_TYPE_TRACING &&
prog->expected_attach_type == BPF_TRACE_FSESSION;
}
static const struct bpf_func_proto *
kprobe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_perf_event_output:
return &bpf_perf_event_output_proto;
case BPF_FUNC_get_stackid:
return &bpf_get_stackid_proto;
case BPF_FUNC_get_stack:
return prog->sleepable ? &bpf_get_stack_sleepable_proto : &bpf_get_stack_proto;
#ifdef CONFIG_BPF_KPROBE_OVERRIDE
case BPF_FUNC_override_return:
return &bpf_override_return_proto;
#endif
case BPF_FUNC_get_func_ip:
if (is_kprobe_multi(prog))
return &bpf_get_func_ip_proto_kprobe_multi;
if (is_uprobe_multi(prog))
return &bpf_get_func_ip_proto_uprobe_multi;
return &bpf_get_func_ip_proto_kprobe;
case BPF_FUNC_get_attach_cookie:
if (is_kprobe_multi(prog))
return &bpf_get_attach_cookie_proto_kmulti;
if (is_uprobe_multi(prog))
return &bpf_get_attach_cookie_proto_umulti;
return &bpf_get_attach_cookie_proto_trace;
default:
return bpf_tracing_func_proto(func_id, prog);
}
}
/* bpf+kprobe programs can access fields of 'struct pt_regs' */
static bool kprobe_prog_is_valid_access(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
if (off < 0 || off >= sizeof(struct pt_regs))
return false;
if (off % size != 0)
return false;
/*
* Assertion for 32 bit to make sure last 8 byte access
* (BPF_DW) to the last 4 byte member is disallowed.
*/
if (off + size > sizeof(struct pt_regs))
return false;
if (type == BPF_WRITE)
prog->aux->kprobe_write_ctx = true;
return true;
}
const struct bpf_verifier_ops kprobe_verifier_ops = {
.get_func_proto = kprobe_prog_func_proto,
.is_valid_access = kprobe_prog_is_valid_access,
};
const struct bpf_prog_ops kprobe_prog_ops = {
};
BPF_CALL_5(bpf_perf_event_output_tp, void *, tp_buff, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
struct pt_regs *regs = *(struct pt_regs **)tp_buff;
/*
* r1 points to perf tracepoint buffer where first 8 bytes are hidden
* from bpf program and contain a pointer to 'struct pt_regs'. Fetch it
* from there and call the same bpf_perf_event_output() helper inline.
*/
return ____bpf_perf_event_output(regs, map, flags, data, size);
}
static const struct bpf_func_proto bpf_perf_event_output_proto_tp = {
.func = bpf_perf_event_output_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_get_stackid_tp, void *, tp_buff, struct bpf_map *, map,
u64, flags)
{
struct pt_regs *regs = *(struct pt_regs **)tp_buff;
/*
* Same comment as in bpf_perf_event_output_tp(), only that this time
* the other helper's function body cannot be inlined due to being
* external, thus we need to call raw helper function.
*/
return bpf_get_stackid((unsigned long) regs, (unsigned long) map,
flags, 0, 0);
}
static const struct bpf_func_proto bpf_get_stackid_proto_tp = {
.func = bpf_get_stackid_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_4(bpf_get_stack_tp, void *, tp_buff, void *, buf, u32, size,
u64, flags)
{
struct pt_regs *regs = *(struct pt_regs **)tp_buff;
return bpf_get_stack((unsigned long) regs, (unsigned long) buf,
(unsigned long) size, flags, 0);
}
static const struct bpf_func_proto bpf_get_stack_proto_tp = {
.func = bpf_get_stack_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
static const struct bpf_func_proto *
tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_perf_event_output:
return &bpf_perf_event_output_proto_tp;
case BPF_FUNC_get_stackid:
return &bpf_get_stackid_proto_tp;
case BPF_FUNC_get_stack:
return &bpf_get_stack_proto_tp;
case BPF_FUNC_get_attach_cookie:
return &bpf_get_attach_cookie_proto_trace;
default:
return bpf_tracing_func_proto(func_id, prog);
}
}
static bool tp_prog_is_valid_access(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
if (off < sizeof(void *) || off >= PERF_MAX_TRACE_SIZE)
return false;
if (type != BPF_READ)
return false;
if (off % size != 0)
return false;
BUILD_BUG_ON(PERF_MAX_TRACE_SIZE % sizeof(__u64));
return true;
}
const struct bpf_verifier_ops tracepoint_verifier_ops = {
.get_func_proto = tp_prog_func_proto,
.is_valid_access = tp_prog_is_valid_access,
};
const struct bpf_prog_ops tracepoint_prog_ops = {
};
BPF_CALL_3(bpf_perf_prog_read_value, struct bpf_perf_event_data_kern *, ctx,
struct bpf_perf_event_value *, buf, u32, size)
{
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_perf_event_value)))
goto clear;
err = perf_event_read_local(ctx->event, &buf->counter, &buf->enabled,
&buf->running);
if (unlikely(err))
goto clear;
return 0;
clear:
memset(buf, 0, size);
return err;
}
static const struct bpf_func_proto bpf_perf_prog_read_value_proto = {
.func = bpf_perf_prog_read_value,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE,
};
BPF_CALL_4(bpf_read_branch_records, struct bpf_perf_event_data_kern *, ctx,
void *, buf, u32, size, u64, flags)
{
static const u32 br_entry_size = sizeof(struct perf_branch_entry);
struct perf_branch_stack *br_stack = ctx->data->br_stack;
u32 to_copy;
if (unlikely(flags & ~BPF_F_GET_BRANCH_RECORDS_SIZE))
return -EINVAL;
if (unlikely(!(ctx->data->sample_flags & PERF_SAMPLE_BRANCH_STACK)))
return -ENOENT;
if (unlikely(!br_stack))
return -ENOENT;
if (flags & BPF_F_GET_BRANCH_RECORDS_SIZE)
return br_stack->nr * br_entry_size;
if (!buf || (size % br_entry_size != 0))
return -EINVAL;
to_copy = min_t(u32, br_stack->nr * br_entry_size, size);
memcpy(buf, br_stack->entries, to_copy);
return to_copy;
}
static const struct bpf_func_proto bpf_read_branch_records_proto = {
.func = bpf_read_branch_records,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
static const struct bpf_func_proto *
pe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_perf_event_output:
return &bpf_perf_event_output_proto_tp;
case BPF_FUNC_get_stackid:
return &bpf_get_stackid_proto_pe;
case BPF_FUNC_get_stack:
return &bpf_get_stack_proto_pe;
case BPF_FUNC_perf_prog_read_value:
return &bpf_perf_prog_read_value_proto;
case BPF_FUNC_read_branch_records:
return &bpf_read_branch_records_proto;
case BPF_FUNC_get_attach_cookie:
return &bpf_get_attach_cookie_proto_pe;
default:
return bpf_tracing_func_proto(func_id, prog);
}
}
/*
* bpf_raw_tp_regs are separate from bpf_pt_regs used from skb/xdp
* to avoid potential recursive reuse issue when/if tracepoints are added
* inside bpf_*_event_output, bpf_get_stackid and/or bpf_get_stack.
*
* Since raw tracepoints run despite bpf_prog_active, support concurrent usage
* in normal, irq, and nmi context.
*/
struct bpf_raw_tp_regs {
struct pt_regs regs[3];
};
static DEFINE_PER_CPU(struct bpf_raw_tp_regs, bpf_raw_tp_regs);
static DEFINE_PER_CPU(int, bpf_raw_tp_nest_level);
static struct pt_regs *get_bpf_raw_tp_regs(void)
{
struct bpf_raw_tp_regs *tp_regs = this_cpu_ptr(&bpf_raw_tp_regs);
int nest_level = this_cpu_inc_return(bpf_raw_tp_nest_level);
if (nest_level > ARRAY_SIZE(tp_regs->regs)) {
this_cpu_dec(bpf_raw_tp_nest_level);
return ERR_PTR(-EBUSY);
}
return &tp_regs->regs[nest_level - 1];
}
static void put_bpf_raw_tp_regs(void)
{
this_cpu_dec(bpf_raw_tp_nest_level);
}
BPF_CALL_5(bpf_perf_event_output_raw_tp, struct bpf_raw_tracepoint_args *, args,
struct bpf_map *, map, u64, flags, void *, data, u64, size)
{
struct pt_regs *regs = get_bpf_raw_tp_regs();
int ret;
if (IS_ERR(regs))
return PTR_ERR(regs);
perf_fetch_caller_regs(regs);
ret = ____bpf_perf_event_output(regs, map, flags, data, size);
put_bpf_raw_tp_regs();
return ret;
}
static const struct bpf_func_proto bpf_perf_event_output_proto_raw_tp = {
.func = bpf_perf_event_output_raw_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
extern const struct bpf_func_proto bpf_skb_output_proto;
extern const struct bpf_func_proto bpf_xdp_output_proto;
extern const struct bpf_func_proto bpf_xdp_get_buff_len_trace_proto;
BPF_CALL_3(bpf_get_stackid_raw_tp, struct bpf_raw_tracepoint_args *, args,
struct bpf_map *, map, u64, flags)
{
struct pt_regs *regs = get_bpf_raw_tp_regs();
int ret;
if (IS_ERR(regs))
return PTR_ERR(regs);
perf_fetch_caller_regs(regs);
/* similar to bpf_perf_event_output_tp, but pt_regs fetched differently */
ret = bpf_get_stackid((unsigned long) regs, (unsigned long) map,
flags, 0, 0);
put_bpf_raw_tp_regs();
return ret;
}
static const struct bpf_func_proto bpf_get_stackid_proto_raw_tp = {
.func = bpf_get_stackid_raw_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args,
void *, buf, u32, size, u64, flags)
{
struct pt_regs *regs = get_bpf_raw_tp_regs();
int ret;
if (IS_ERR(regs))
return PTR_ERR(regs);
perf_fetch_caller_regs(regs);
ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf,
(unsigned long) size, flags, 0);
put_bpf_raw_tp_regs();
return ret;
}
static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
.func = bpf_get_stack_raw_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
static const struct bpf_func_proto *
raw_tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_perf_event_output:
return &bpf_perf_event_output_proto_raw_tp;
case BPF_FUNC_get_stackid:
return &bpf_get_stackid_proto_raw_tp;
case BPF_FUNC_get_stack:
return &bpf_get_stack_proto_raw_tp;
case BPF_FUNC_get_attach_cookie:
return &bpf_get_attach_cookie_proto_tracing;
default:
return bpf_tracing_func_proto(func_id, prog);
}
}
const struct bpf_func_proto *
tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
const struct bpf_func_proto *fn;
switch (func_id) {
#ifdef CONFIG_NET
case BPF_FUNC_skb_output:
return &bpf_skb_output_proto;
case BPF_FUNC_xdp_output:
return &bpf_xdp_output_proto;
case BPF_FUNC_skc_to_tcp6_sock:
return &bpf_skc_to_tcp6_sock_proto;
case BPF_FUNC_skc_to_tcp_sock:
return &bpf_skc_to_tcp_sock_proto;
case BPF_FUNC_skc_to_tcp_timewait_sock:
return &bpf_skc_to_tcp_timewait_sock_proto;
case BPF_FUNC_skc_to_tcp_request_sock:
return &bpf_skc_to_tcp_request_sock_proto;
case BPF_FUNC_skc_to_udp6_sock:
return &bpf_skc_to_udp6_sock_proto;
case BPF_FUNC_skc_to_unix_sock:
return &bpf_skc_to_unix_sock_proto;
case BPF_FUNC_skc_to_mptcp_sock:
return &bpf_skc_to_mptcp_sock_proto;
case BPF_FUNC_sk_storage_get:
return &bpf_sk_storage_get_tracing_proto;
case BPF_FUNC_sk_storage_delete:
return &bpf_sk_storage_delete_tracing_proto;
case BPF_FUNC_sock_from_file:
return &bpf_sock_from_file_proto;
case BPF_FUNC_get_socket_cookie:
return &bpf_get_socket_ptr_cookie_proto;
case BPF_FUNC_xdp_get_buff_len:
return &bpf_xdp_get_buff_len_trace_proto;
#endif
case BPF_FUNC_seq_printf:
return prog->expected_attach_type == BPF_TRACE_ITER ?
&bpf_seq_printf_proto :
NULL;
case BPF_FUNC_seq_write:
return prog->expected_attach_type == BPF_TRACE_ITER ?
&bpf_seq_write_proto :
NULL;
case BPF_FUNC_seq_printf_btf:
return prog->expected_attach_type == BPF_TRACE_ITER ?
&bpf_seq_printf_btf_proto :
NULL;
case BPF_FUNC_d_path:
return &bpf_d_path_proto;
case BPF_FUNC_get_func_arg:
if (bpf_prog_has_trampoline(prog) ||
prog->expected_attach_type == BPF_TRACE_RAW_TP)
return &bpf_get_func_arg_proto;
return NULL;
case BPF_FUNC_get_func_ret:
return bpf_prog_has_trampoline(prog) ? &bpf_get_func_ret_proto : NULL;
case BPF_FUNC_get_func_arg_cnt:
if (bpf_prog_has_trampoline(prog) ||
prog->expected_attach_type == BPF_TRACE_RAW_TP)
return &bpf_get_func_arg_cnt_proto;
return NULL;
case BPF_FUNC_get_attach_cookie:
if (prog->type == BPF_PROG_TYPE_TRACING &&
prog->expected_attach_type == BPF_TRACE_RAW_TP)
return &bpf_get_attach_cookie_proto_tracing;
return bpf_prog_has_trampoline(prog) ? &bpf_get_attach_cookie_proto_tracing : NULL;
default:
fn = raw_tp_prog_func_proto(func_id, prog);
if (!fn && prog->expected_attach_type == BPF_TRACE_ITER)
fn = bpf_iter_get_func_proto(func_id, prog);
return fn;
}
}
static bool raw_tp_prog_is_valid_access(int off, int size,
enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
return bpf_tracing_ctx_access(off, size, type);
}
static bool tracing_prog_is_valid_access(int off, int size,
enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
}
int __weak bpf_prog_test_run_tracing(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
const struct bpf_verifier_ops raw_tracepoint_verifier_ops = {
.get_func_proto = raw_tp_prog_func_proto,
.is_valid_access = raw_tp_prog_is_valid_access,
};
const struct bpf_prog_ops raw_tracepoint_prog_ops = {
#ifdef CONFIG_NET
.test_run = bpf_prog_test_run_raw_tp,
#endif
};
const struct bpf_verifier_ops tracing_verifier_ops = {
.get_func_proto = tracing_prog_func_proto,
.is_valid_access = tracing_prog_is_valid_access,
};
const struct bpf_prog_ops tracing_prog_ops = {
.test_run = bpf_prog_test_run_tracing,
};
static bool raw_tp_writable_prog_is_valid_access(int off, int size,
enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
if (off == 0) {
if (size != sizeof(u64) || type != BPF_READ)
return false;
info->reg_type = PTR_TO_TP_BUFFER;
}
return raw_tp_prog_is_valid_access(off, size, type, prog, info);
}
const struct bpf_verifier_ops raw_tracepoint_writable_verifier_ops = {
.get_func_proto = raw_tp_prog_func_proto,
.is_valid_access = raw_tp_writable_prog_is_valid_access,
};
const struct bpf_prog_ops raw_tracepoint_writable_prog_ops = {
};
static bool pe_prog_is_valid_access(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
const int size_u64 = sizeof(u64);
if (off < 0 || off >= sizeof(struct bpf_perf_event_data))
return false;
if (type != BPF_READ)
return false;
if (off % size != 0) {
if (sizeof(unsigned long) != 4)
return false;
if (size != 8)
return false;
if (off % size != 4)
return false;
}
switch (off) {
case bpf_ctx_range(struct bpf_perf_event_data, sample_period):
bpf_ctx_record_field_size(info, size_u64);
if (!bpf_ctx_narrow_access_ok(off, size, size_u64))
return false;
break;
case bpf_ctx_range(struct bpf_perf_event_data, addr):
bpf_ctx_record_field_size(info, size_u64);
if (!bpf_ctx_narrow_access_ok(off, size, size_u64))
return false;
break;
default:
if (size != sizeof(long))
return false;
}
return true;
}
static u32 pe_prog_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog, u32 *target_size)
{
struct bpf_insn *insn = insn_buf;
switch (si->off) {
case offsetof(struct bpf_perf_event_data, sample_period):
*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_perf_event_data_kern,
data), si->dst_reg, si->src_reg,
offsetof(struct bpf_perf_event_data_kern, data));
*insn++ = BPF_LDX_MEM(BPF_DW, si->dst_reg, si->dst_reg,
bpf_target_off(struct perf_sample_data, period, 8,
target_size));
break;
case offsetof(struct bpf_perf_event_data, addr):
*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_perf_event_data_kern,
data), si->dst_reg, si->src_reg,
offsetof(struct bpf_perf_event_data_kern, data));
*insn++ = BPF_LDX_MEM(BPF_DW, si->dst_reg, si->dst_reg,
bpf_target_off(struct perf_sample_data, addr, 8,
target_size));
break;
default:
*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_perf_event_data_kern,
regs), si->dst_reg, si->src_reg,
offsetof(struct bpf_perf_event_data_kern, regs));
*insn++ = BPF_LDX_MEM(BPF_SIZEOF(long), si->dst_reg, si->dst_reg,
si->off);
break;
}
return insn - insn_buf;
}
const struct bpf_verifier_ops perf_event_verifier_ops = {
.get_func_proto = pe_prog_func_proto,
.is_valid_access = pe_prog_is_valid_access,
.convert_ctx_access = pe_prog_convert_ctx_access,
};
const struct bpf_prog_ops perf_event_prog_ops = {
};
static DEFINE_MUTEX(bpf_event_mutex);
#define BPF_TRACE_MAX_PROGS 64
int perf_event_attach_bpf_prog(struct perf_event *event,
struct bpf_prog *prog,
u64 bpf_cookie)
{
struct bpf_prog_array *old_array;
struct bpf_prog_array *new_array;
int ret = -EEXIST;
/*
* Kprobe override only works if they are on the function entry,
* and only if they are on the opt-in list.
*/
if (prog->kprobe_override &&
(!trace_kprobe_on_func_entry(event->tp_event) ||
!trace_kprobe_error_injectable(event->tp_event)))
return -EINVAL;
mutex_lock(&bpf_event_mutex);
if (event->prog)
goto unlock;
old_array = bpf_event_rcu_dereference(event->tp_event->prog_array);
if (old_array &&
bpf_prog_array_length(old_array) >= BPF_TRACE_MAX_PROGS) {
ret = -E2BIG;
goto unlock;
}
ret = bpf_prog_array_copy(old_array, NULL, prog, bpf_cookie, &new_array);
if (ret < 0)
goto unlock;
/* set the new array to event->tp_event and set event->prog */
event->prog = prog;
event->bpf_cookie = bpf_cookie;
rcu_assign_pointer(event->tp_event->prog_array, new_array);
bpf_prog_array_free_sleepable(old_array);
unlock:
mutex_unlock(&bpf_event_mutex);
return ret;
}
void perf_event_detach_bpf_prog(struct perf_event *event)
{
struct bpf_prog_array *old_array;
struct bpf_prog_array *new_array;
struct bpf_prog *prog = NULL;
int ret;
mutex_lock(&bpf_event_mutex);
if (!event->prog)
goto unlock;
old_array = bpf_event_rcu_dereference(event->tp_event->prog_array);
if (!old_array)
goto put;
ret = bpf_prog_array_copy(old_array, event->prog, NULL, 0, &new_array);
if (ret < 0) {
bpf_prog_array_delete_safe(old_array, event->prog);
} else {
rcu_assign_pointer(event->tp_event->prog_array, new_array);
bpf_prog_array_free_sleepable(old_array);
}
put:
prog = event->prog;
event->prog = NULL;
unlock:
mutex_unlock(&bpf_event_mutex);
if (prog) {
/*
* It could be that the bpf_prog is not sleepable (and will be freed
* via normal RCU), but is called from a point that supports sleepable
* programs and uses tasks-trace-RCU.
*/
synchronize_rcu_tasks_trace();
bpf_prog_put(prog);
}
}
int perf_event_query_prog_array(struct perf_event *event, void __user *info)
{
struct perf_event_query_bpf __user *uquery = info;
struct perf_event_query_bpf query = {};
struct bpf_prog_array *progs;
u32 *ids, prog_cnt, ids_len;
int ret;
if (!perfmon_capable())
return -EPERM;
if (event->attr.type != PERF_TYPE_TRACEPOINT)
return -EINVAL;
if (copy_from_user(&query, uquery, sizeof(query)))
return -EFAULT;
ids_len = query.ids_len;
if (ids_len > BPF_TRACE_MAX_PROGS)
return -E2BIG;
ids = kcalloc(ids_len, sizeof(u32), GFP_USER | __GFP_NOWARN);
if (!ids)
return -ENOMEM;
/*
* The above kcalloc returns ZERO_SIZE_PTR when ids_len = 0, which
* is required when user only wants to check for uquery->prog_cnt.
* There is no need to check for it since the case is handled
* gracefully in bpf_prog_array_copy_info.
*/
mutex_lock(&bpf_event_mutex);
progs = bpf_event_rcu_dereference(event->tp_event->prog_array);
ret = bpf_prog_array_copy_info(progs, ids, ids_len, &prog_cnt);
mutex_unlock(&bpf_event_mutex);
if (copy_to_user(&uquery->prog_cnt, &prog_cnt, sizeof(prog_cnt)) ||
copy_to_user(uquery->ids, ids, ids_len * sizeof(u32)))
ret = -EFAULT;
kfree(ids);
return ret;
}
extern struct bpf_raw_event_map __start__bpf_raw_tp[];
extern struct bpf_raw_event_map __stop__bpf_raw_tp[];
struct bpf_raw_event_map *bpf_get_raw_tracepoint(const char *name)
{
struct bpf_raw_event_map *btp = __start__bpf_raw_tp;
for (; btp < __stop__bpf_raw_tp; btp++) {
if (!strcmp(btp->tp->name, name))
return btp;
}
return bpf_get_raw_tracepoint_module(name);
}
void bpf_put_raw_tracepoint(struct bpf_raw_event_map *btp)
{
struct module *mod;
guard(rcu)();
mod = __module_address((unsigned long)btp);
module_put(mod);
}
static __always_inline
void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)
{
struct bpf_prog *prog = link->link.prog;
struct bpf_run_ctx *old_run_ctx;
struct bpf_trace_run_ctx run_ctx;
rcu_read_lock_dont_migrate();
if (unlikely(!bpf_prog_get_recursion_context(prog))) {
bpf_prog_inc_misses_counter(prog);
goto out;
}
run_ctx.bpf_cookie = link->cookie;
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
(void) bpf_prog_run(prog, args);
bpf_reset_run_ctx(old_run_ctx);
out:
bpf_prog_put_recursion_context(prog);
rcu_read_unlock_migrate();
}
#define UNPACK(...) __VA_ARGS__
#define REPEAT_1(FN, DL, X, ...) FN(X)
#define REPEAT_2(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_1(FN, DL, __VA_ARGS__)
#define REPEAT_3(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_2(FN, DL, __VA_ARGS__)
#define REPEAT_4(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_3(FN, DL, __VA_ARGS__)
#define REPEAT_5(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_4(FN, DL, __VA_ARGS__)
#define REPEAT_6(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_5(FN, DL, __VA_ARGS__)
#define REPEAT_7(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_6(FN, DL, __VA_ARGS__)
#define REPEAT_8(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_7(FN, DL, __VA_ARGS__)
#define REPEAT_9(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_8(FN, DL, __VA_ARGS__)
#define REPEAT_10(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_9(FN, DL, __VA_ARGS__)
#define REPEAT_11(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_10(FN, DL, __VA_ARGS__)
#define REPEAT_12(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_11(FN, DL, __VA_ARGS__)
#define REPEAT(X, FN, DL, ...) REPEAT_##X(FN, DL, __VA_ARGS__)
#define SARG(X) u64 arg##X
#define COPY(X) args[X] = arg##X
#define __DL_COM (,)
#define __DL_SEM (;)
#define __SEQ_0_11 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
#define BPF_TRACE_DEFN_x(x) \
void bpf_trace_run##x(struct bpf_raw_tp_link *link, \
REPEAT(x, SARG, __DL_COM, __SEQ_0_11)) \
{ \
u64 args[x]; \
REPEAT(x, COPY, __DL_SEM, __SEQ_0_11); \
__bpf_trace_run(link, args); \
} \
EXPORT_SYMBOL_GPL(bpf_trace_run##x)
BPF_TRACE_DEFN_x(1);
BPF_TRACE_DEFN_x(2);
BPF_TRACE_DEFN_x(3);
BPF_TRACE_DEFN_x(4);
BPF_TRACE_DEFN_x(5);
BPF_TRACE_DEFN_x(6);
BPF_TRACE_DEFN_x(7);
BPF_TRACE_DEFN_x(8);
BPF_TRACE_DEFN_x(9);
BPF_TRACE_DEFN_x(10);
BPF_TRACE_DEFN_x(11);
BPF_TRACE_DEFN_x(12);
int bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_raw_tp_link *link)
{
struct tracepoint *tp = btp->tp;
struct bpf_prog *prog = link->link.prog;
/*
* check that program doesn't access arguments beyond what's
* available in this tracepoint
*/
if (prog->aux->max_ctx_offset > btp->num_args * sizeof(u64))
return -EINVAL;
if (prog->aux->max_tp_access > btp->writable_size)
return -EINVAL;
return tracepoint_probe_register_may_exist(tp, (void *)btp->bpf_func, link);
}
int bpf_probe_unregister(struct bpf_raw_event_map *btp, struct bpf_raw_tp_link *link)
{
return tracepoint_probe_unregister(btp->tp, (void *)btp->bpf_func, link);
}
int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id,
u32 *fd_type, const char **buf,
u64 *probe_offset, u64 *probe_addr,
unsigned long *missed)
{
bool is_tracepoint, is_syscall_tp;
struct bpf_prog *prog;
int flags, err = 0;
prog = event->prog;
if (!prog)
return -ENOENT;
/* not supporting BPF_PROG_TYPE_PERF_EVENT yet */
if (prog->type == BPF_PROG_TYPE_PERF_EVENT)
return -EOPNOTSUPP;
*prog_id = prog->aux->id;
flags = event->tp_event->flags;
is_tracepoint = flags & TRACE_EVENT_FL_TRACEPOINT;
is_syscall_tp = is_syscall_trace_event(event->tp_event);
if (is_tracepoint || is_syscall_tp) {
*buf = is_tracepoint ? event->tp_event->tp->name
: event->tp_event->name;
/* We allow NULL pointer for tracepoint */
if (fd_type)
*fd_type = BPF_FD_TYPE_TRACEPOINT;
if (probe_offset)
*probe_offset = 0x0;
if (probe_addr)
*probe_addr = 0x0;
} else {
/* kprobe/uprobe */
err = -EOPNOTSUPP;
#ifdef CONFIG_KPROBE_EVENTS
if (flags & TRACE_EVENT_FL_KPROBE)
err = bpf_get_kprobe_info(event, fd_type, buf,
probe_offset, probe_addr, missed,
event->attr.type == PERF_TYPE_TRACEPOINT);
#endif
#ifdef CONFIG_UPROBE_EVENTS
if (flags & TRACE_EVENT_FL_UPROBE)
err = bpf_get_uprobe_info(event, fd_type, buf,
probe_offset, probe_addr,
event->attr.type == PERF_TYPE_TRACEPOINT);
#endif
}
return err;
}
static int __init send_signal_irq_work_init(void)
{
int cpu;
struct send_signal_irq_work *work;
for_each_possible_cpu(cpu) {
work = per_cpu_ptr(&send_signal_work, cpu);
init_irq_work(&work->irq_work, do_bpf_send_signal);
}
return 0;
}
subsys_initcall(send_signal_irq_work_init);
#ifdef CONFIG_MODULES
static int bpf_event_notify(struct notifier_block *nb, unsigned long op,
void *module)
{
struct bpf_trace_module *btm, *tmp;
struct module *mod = module;
int ret = 0;
if (mod->num_bpf_raw_events == 0 ||
(op != MODULE_STATE_COMING && op != MODULE_STATE_GOING))
goto out;
mutex_lock(&bpf_module_mutex);
switch (op) {
case MODULE_STATE_COMING:
btm = kzalloc_obj(*btm);
if (btm) {
btm->module = module;
list_add(&btm->list, &bpf_trace_modules);
} else {
ret = -ENOMEM;
}
break;
case MODULE_STATE_GOING:
list_for_each_entry_safe(btm, tmp, &bpf_trace_modules, list) {
if (btm->module == module) {
list_del(&btm->list);
kfree(btm);
break;
}
}
break;
}
mutex_unlock(&bpf_module_mutex);
out:
return notifier_from_errno(ret);
}
static struct notifier_block bpf_module_nb = {
.notifier_call = bpf_event_notify,
};
static int __init bpf_event_init(void)
{
register_module_notifier(&bpf_module_nb);
return 0;
}
fs_initcall(bpf_event_init);
#endif /* CONFIG_MODULES */
struct bpf_session_run_ctx {
struct bpf_run_ctx run_ctx;
bool is_return;
void *data;
};
#ifdef CONFIG_FPROBE
struct bpf_kprobe_multi_link {
struct bpf_link link;
struct fprobe fp;
unsigned long *addrs;
u64 *cookies;
u32 cnt;
u32 mods_cnt;
struct module **mods;
};
struct bpf_kprobe_multi_run_ctx {
struct bpf_session_run_ctx session_ctx;
struct bpf_kprobe_multi_link *link;
unsigned long entry_ip;
};
struct user_syms {
const char **syms;
char *buf;
};
#ifndef CONFIG_HAVE_FTRACE_REGS_HAVING_PT_REGS
static DEFINE_PER_CPU(struct pt_regs, bpf_kprobe_multi_pt_regs);
#define bpf_kprobe_multi_pt_regs_ptr() this_cpu_ptr(&bpf_kprobe_multi_pt_regs)
#else
#define bpf_kprobe_multi_pt_regs_ptr() (NULL)
#endif
static unsigned long ftrace_get_entry_ip(unsigned long fentry_ip)
{
unsigned long ip = ftrace_get_symaddr(fentry_ip);
return ip ? : fentry_ip;
}
static int copy_user_syms(struct user_syms *us, unsigned long __user *usyms, u32 cnt)
{
unsigned long __user usymbol;
const char **syms = NULL;
char *buf = NULL, *p;
int err = -ENOMEM;
unsigned int i;
syms = kvmalloc_array(cnt, sizeof(*syms), GFP_KERNEL);
if (!syms)
goto error;
buf = kvmalloc_array(cnt, KSYM_NAME_LEN, GFP_KERNEL);
if (!buf)
goto error;
for (p = buf, i = 0; i < cnt; i++) {
if (__get_user(usymbol, usyms + i)) {
err = -EFAULT;
goto error;
}
err = strncpy_from_user(p, (const char __user *) usymbol, KSYM_NAME_LEN);
if (err == KSYM_NAME_LEN)
err = -E2BIG;
if (err < 0)
goto error;
syms[i] = p;
p += err + 1;
}
us->syms = syms;
us->buf = buf;
return 0;
error:
if (err) {
kvfree(syms);
kvfree(buf);
}
return err;
}
static void kprobe_multi_put_modules(struct module **mods, u32 cnt)
{
u32 i;
for (i = 0; i < cnt; i++)
module_put(mods[i]);
}
static void free_user_syms(struct user_syms *us)
{
kvfree(us->syms);
kvfree(us->buf);
}
static void bpf_kprobe_multi_link_release(struct bpf_link *link)
{
struct bpf_kprobe_multi_link *kmulti_link;
kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link);
unregister_fprobe(&kmulti_link->fp);
kprobe_multi_put_modules(kmulti_link->mods, kmulti_link->mods_cnt);
}
static void bpf_kprobe_multi_link_dealloc(struct bpf_link *link)
{
struct bpf_kprobe_multi_link *kmulti_link;
kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link);
kvfree(kmulti_link->addrs);
kvfree(kmulti_link->cookies);
kfree(kmulti_link->mods);
kfree(kmulti_link);
}
static int bpf_kprobe_multi_link_fill_link_info(const struct bpf_link *link,
struct bpf_link_info *info)
{
u64 __user *ucookies = u64_to_user_ptr(info->kprobe_multi.cookies);
u64 __user *uaddrs = u64_to_user_ptr(info->kprobe_multi.addrs);
struct bpf_kprobe_multi_link *kmulti_link;
u32 ucount = info->kprobe_multi.count;
int err = 0, i;
if (!uaddrs ^ !ucount)
return -EINVAL;
if (ucookies && !ucount)
return -EINVAL;
kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link);
info->kprobe_multi.count = kmulti_link->cnt;
info->kprobe_multi.flags = kmulti_link->link.flags;
info->kprobe_multi.missed = kmulti_link->fp.nmissed;
if (!uaddrs)
return 0;
if (ucount < kmulti_link->cnt)
err = -ENOSPC;
else
ucount = kmulti_link->cnt;
if (ucookies) {
if (kmulti_link->cookies) {
if (copy_to_user(ucookies, kmulti_link->cookies, ucount * sizeof(u64)))
return -EFAULT;
} else {
for (i = 0; i < ucount; i++) {
if (put_user(0, ucookies + i))
return -EFAULT;
}
}
}
if (kallsyms_show_value(current_cred())) {
if (copy_to_user(uaddrs, kmulti_link->addrs, ucount * sizeof(u64)))
return -EFAULT;
} else {
for (i = 0; i < ucount; i++) {
if (put_user(0, uaddrs + i))
return -EFAULT;
}
}
return err;
}
#ifdef CONFIG_PROC_FS
static void bpf_kprobe_multi_show_fdinfo(const struct bpf_link *link,
struct seq_file *seq)
{
struct bpf_kprobe_multi_link *kmulti_link;
bool has_cookies;
kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link);
has_cookies = !!kmulti_link->cookies;
seq_printf(seq,
"kprobe_cnt:\t%u\n"
"missed:\t%lu\n",
kmulti_link->cnt,
kmulti_link->fp.nmissed);
seq_printf(seq, "%s\t %s\n", "cookie", "func");
for (int i = 0; i < kmulti_link->cnt; i++) {
seq_printf(seq,
"%llu\t %pS\n",
has_cookies ? kmulti_link->cookies[i] : 0,
(void *)kmulti_link->addrs[i]);
}
}
#endif
static const struct bpf_link_ops bpf_kprobe_multi_link_lops = {
.release = bpf_kprobe_multi_link_release,
.dealloc_deferred = bpf_kprobe_multi_link_dealloc,
.fill_link_info = bpf_kprobe_multi_link_fill_link_info,
#ifdef CONFIG_PROC_FS
.show_fdinfo = bpf_kprobe_multi_show_fdinfo,
#endif
};
static void bpf_kprobe_multi_cookie_swap(void *a, void *b, int size, const void *priv)
{
const struct bpf_kprobe_multi_link *link = priv;
unsigned long *addr_a = a, *addr_b = b;
u64 *cookie_a, *cookie_b;
cookie_a = link->cookies + (addr_a - link->addrs);
cookie_b = link->cookies + (addr_b - link->addrs);
/* swap addr_a/addr_b and cookie_a/cookie_b values */
swap(*addr_a, *addr_b);
swap(*cookie_a, *cookie_b);
}
static int bpf_kprobe_multi_addrs_cmp(const void *a, const void *b)
{
const unsigned long *addr_a = a, *addr_b = b;
if (*addr_a == *addr_b)
return 0;
return *addr_a < *addr_b ? -1 : 1;
}
static int bpf_kprobe_multi_cookie_cmp(const void *a, const void *b, const void *priv)
{
return bpf_kprobe_multi_addrs_cmp(a, b);
}
static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx)
{
struct bpf_kprobe_multi_run_ctx *run_ctx;
struct bpf_kprobe_multi_link *link;
u64 *cookie, entry_ip;
unsigned long *addr;
if (WARN_ON_ONCE(!ctx))
return 0;
run_ctx = container_of(current->bpf_ctx, struct bpf_kprobe_multi_run_ctx,
session_ctx.run_ctx);
link = run_ctx->link;
if (!link->cookies)
return 0;
entry_ip = run_ctx->entry_ip;
addr = bsearch(&entry_ip, link->addrs, link->cnt, sizeof(entry_ip),
bpf_kprobe_multi_addrs_cmp);
if (!addr)
return 0;
cookie = link->cookies + (addr - link->addrs);
return *cookie;
}
static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
{
struct bpf_kprobe_multi_run_ctx *run_ctx;
run_ctx = container_of(current->bpf_ctx, struct bpf_kprobe_multi_run_ctx,
session_ctx.run_ctx);
return run_ctx->entry_ip;
}
static __always_inline int
kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link,
unsigned long entry_ip, struct ftrace_regs *fregs,
bool is_return, void *data)
{
struct bpf_kprobe_multi_run_ctx run_ctx = {
.session_ctx = {
.is_return = is_return,
.data = data,
},
.link = link,
.entry_ip = entry_ip,
};
struct bpf_run_ctx *old_run_ctx;
struct pt_regs *regs;
int err;
/*
* graph tracer framework ensures we won't migrate, so there is no need
* to use migrate_disable for bpf_prog_run again. The check here just for
* __this_cpu_inc_return.
*/
cant_sleep();
if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) {
bpf_prog_inc_misses_counter(link->link.prog);
err = 1;
goto out;
}
rcu_read_lock();
regs = ftrace_partial_regs(fregs, bpf_kprobe_multi_pt_regs_ptr());
old_run_ctx = bpf_set_run_ctx(&run_ctx.session_ctx.run_ctx);
err = bpf_prog_run(link->link.prog, regs);
bpf_reset_run_ctx(old_run_ctx);
ftrace_partial_regs_update(fregs, bpf_kprobe_multi_pt_regs_ptr());
rcu_read_unlock();
out:
__this_cpu_dec(bpf_prog_active);
return err;
}
static int
kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip,
unsigned long ret_ip, struct ftrace_regs *fregs,
void *data)
{
struct bpf_kprobe_multi_link *link;
int err;
link = container_of(fp, struct bpf_kprobe_multi_link, fp);
err = kprobe_multi_link_prog_run(link, ftrace_get_entry_ip(fentry_ip),
fregs, false, data);
return is_kprobe_session(link->link.prog) ? err : 0;
}
static void
kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip,
unsigned long ret_ip, struct ftrace_regs *fregs,
void *data)
{
struct bpf_kprobe_multi_link *link;
link = container_of(fp, struct bpf_kprobe_multi_link, fp);
kprobe_multi_link_prog_run(link, ftrace_get_entry_ip(fentry_ip),
fregs, true, data);
}
static int symbols_cmp_r(const void *a, const void *b, const void *priv)
{
const char **str_a = (const char **) a;
const char **str_b = (const char **) b;
return strcmp(*str_a, *str_b);
}
struct multi_symbols_sort {
const char **funcs;
u64 *cookies;
};
static void symbols_swap_r(void *a, void *b, int size, const void *priv)
{
const struct multi_symbols_sort *data = priv;
const char **name_a = a, **name_b = b;
swap(*name_a, *name_b);
/* If defined, swap also related cookies. */
if (data->cookies) {
u64 *cookie_a, *cookie_b;
cookie_a = data->cookies + (name_a - data->funcs);
cookie_b = data->cookies + (name_b - data->funcs);
swap(*cookie_a, *cookie_b);
}
}
struct modules_array {
struct module **mods;
int mods_cnt;
int mods_cap;
};
static int add_module(struct modules_array *arr, struct module *mod)
{
struct module **mods;
if (arr->mods_cnt == arr->mods_cap) {
arr->mods_cap = max(16, arr->mods_cap * 3 / 2);
mods = krealloc_array(arr->mods, arr->mods_cap, sizeof(*mods), GFP_KERNEL);
if (!mods)
return -ENOMEM;
arr->mods = mods;
}
arr->mods[arr->mods_cnt] = mod;
arr->mods_cnt++;
return 0;
}
static bool has_module(struct modules_array *arr, struct module *mod)
{
int i;
for (i = arr->mods_cnt - 1; i >= 0; i--) {
if (arr->mods[i] == mod)
return true;
}
return false;
}
static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u32 addrs_cnt)
{
struct modules_array arr = {};
u32 i, err = 0;
for (i = 0; i < addrs_cnt; i++) {
bool skip_add = false;
struct module *mod;
scoped_guard(rcu) {
mod = __module_address(addrs[i]);
/* Either no module or it's already stored */
if (!mod || has_module(&arr, mod)) {
skip_add = true;
break; /* scoped_guard */
}
if (!try_module_get(mod))
err = -EINVAL;
}
if (skip_add)
continue;
if (err)
break;
err = add_module(&arr, mod);
if (err) {
module_put(mod);
break;
}
}
/* We return either err < 0 in case of error, ... */
if (err) {
kprobe_multi_put_modules(arr.mods, arr.mods_cnt);
kfree(arr.mods);
return err;
}
/* or number of modules found if everything is ok. */
*mods = arr.mods;
return arr.mods_cnt;
}
static int addrs_check_error_injection_list(unsigned long *addrs, u32 cnt)
{
u32 i;
for (i = 0; i < cnt; i++) {
if (!within_error_injection_list(addrs[i]))
return -EINVAL;
}
return 0;
}
int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
struct bpf_kprobe_multi_link *link = NULL;
struct bpf_link_primer link_primer;
void __user *ucookies;
unsigned long *addrs;
u32 flags, cnt, size;
void __user *uaddrs;
u64 *cookies = NULL;
void __user *usyms;
int err;
/* no support for 32bit archs yet */
if (sizeof(u64) != sizeof(void *))
return -EOPNOTSUPP;
if (attr->link_create.flags)
return -EINVAL;
if (!is_kprobe_multi(prog))
return -EINVAL;
/* Writing to context is not allowed for kprobes. */
if (prog->aux->kprobe_write_ctx)
return -EINVAL;
flags = attr->link_create.kprobe_multi.flags;
if (flags & ~BPF_F_KPROBE_MULTI_RETURN)
return -EINVAL;
uaddrs = u64_to_user_ptr(attr->link_create.kprobe_multi.addrs);
usyms = u64_to_user_ptr(attr->link_create.kprobe_multi.syms);
if (!!uaddrs == !!usyms)
return -EINVAL;
cnt = attr->link_create.kprobe_multi.cnt;
if (!cnt)
return -EINVAL;
if (cnt > MAX_KPROBE_MULTI_CNT)
return -E2BIG;
size = cnt * sizeof(*addrs);
addrs = kvmalloc_array(cnt, sizeof(*addrs), GFP_KERNEL);
if (!addrs)
return -ENOMEM;
ucookies = u64_to_user_ptr(attr->link_create.kprobe_multi.cookies);
if (ucookies) {
cookies = kvmalloc_array(cnt, sizeof(*addrs), GFP_KERNEL);
if (!cookies) {
err = -ENOMEM;
goto error;
}
if (copy_from_user(cookies, ucookies, size)) {
err = -EFAULT;
goto error;
}
}
if (uaddrs) {
if (copy_from_user(addrs, uaddrs, size)) {
err = -EFAULT;
goto error;
}
} else {
struct multi_symbols_sort data = {
.cookies = cookies,
};
struct user_syms us;
err = copy_user_syms(&us, usyms, cnt);
if (err)
goto error;
if (cookies)
data.funcs = us.syms;
sort_r(us.syms, cnt, sizeof(*us.syms), symbols_cmp_r,
symbols_swap_r, &data);
err = ftrace_lookup_symbols(us.syms, cnt, addrs);
free_user_syms(&us);
if (err)
goto error;
}
if (prog->kprobe_override && addrs_check_error_injection_list(addrs, cnt)) {
err = -EINVAL;
goto error;
}
link = kzalloc_obj(*link);
if (!link) {
err = -ENOMEM;
goto error;
}
bpf_link_init(&link->link, BPF_LINK_TYPE_KPROBE_MULTI,
&bpf_kprobe_multi_link_lops, prog, attr->link_create.attach_type);
err = bpf_link_prime(&link->link, &link_primer);
if (err)
goto error;
if (!(flags & BPF_F_KPROBE_MULTI_RETURN))
link->fp.entry_handler = kprobe_multi_link_handler;
if ((flags & BPF_F_KPROBE_MULTI_RETURN) || is_kprobe_session(prog))
link->fp.exit_handler = kprobe_multi_link_exit_handler;
if (is_kprobe_session(prog))
link->fp.entry_data_size = sizeof(u64);
link->addrs = addrs;
link->cookies = cookies;
link->cnt = cnt;
link->link.flags = flags;
if (cookies) {
/*
* Sorting addresses will trigger sorting cookies as well
* (check bpf_kprobe_multi_cookie_swap). This way we can
* find cookie based on the address in bpf_get_attach_cookie
* helper.
*/
sort_r(addrs, cnt, sizeof(*addrs),
bpf_kprobe_multi_cookie_cmp,
bpf_kprobe_multi_cookie_swap,
link);
}
err = get_modules_for_addrs(&link->mods, addrs, cnt);
if (err < 0) {
bpf_link_cleanup(&link_primer);
return err;
}
link->mods_cnt = err;
err = register_fprobe_ips(&link->fp, addrs, cnt);
if (err) {
kprobe_multi_put_modules(link->mods, link->mods_cnt);
bpf_link_cleanup(&link_primer);
return err;
}
return bpf_link_settle(&link_primer);
error:
kfree(link);
kvfree(addrs);
kvfree(cookies);
return err;
}
#else /* !CONFIG_FPROBE */
int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
return -EOPNOTSUPP;
}
static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx)
{
return 0;
}
static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
{
return 0;
}
#endif
#ifdef CONFIG_UPROBES
struct bpf_uprobe_multi_link;
struct bpf_uprobe {
struct bpf_uprobe_multi_link *link;
loff_t offset;
unsigned long ref_ctr_offset;
u64 cookie;
struct uprobe *uprobe;
struct uprobe_consumer consumer;
bool session;
};
struct bpf_uprobe_multi_link {
struct path path;
struct bpf_link link;
u32 cnt;
struct bpf_uprobe *uprobes;
struct task_struct *task;
};
struct bpf_uprobe_multi_run_ctx {
struct bpf_session_run_ctx session_ctx;
unsigned long entry_ip;
struct bpf_uprobe *uprobe;
};
static void bpf_uprobe_unregister(struct bpf_uprobe *uprobes, u32 cnt)
{
u32 i;
for (i = 0; i < cnt; i++)
uprobe_unregister_nosync(uprobes[i].uprobe, &uprobes[i].consumer);
if (cnt)
uprobe_unregister_sync();
}
static void bpf_uprobe_multi_link_release(struct bpf_link *link)
{
struct bpf_uprobe_multi_link *umulti_link;
umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
bpf_uprobe_unregister(umulti_link->uprobes, umulti_link->cnt);
if (umulti_link->task)
put_task_struct(umulti_link->task);
path_put(&umulti_link->path);
}
static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
{
struct bpf_uprobe_multi_link *umulti_link;
umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
kvfree(umulti_link->uprobes);
kfree(umulti_link);
}
static int bpf_uprobe_multi_link_fill_link_info(const struct bpf_link *link,
struct bpf_link_info *info)
{
u64 __user *uref_ctr_offsets = u64_to_user_ptr(info->uprobe_multi.ref_ctr_offsets);
u64 __user *ucookies = u64_to_user_ptr(info->uprobe_multi.cookies);
u64 __user *uoffsets = u64_to_user_ptr(info->uprobe_multi.offsets);
u64 __user *upath = u64_to_user_ptr(info->uprobe_multi.path);
u32 upath_size = info->uprobe_multi.path_size;
struct bpf_uprobe_multi_link *umulti_link;
u32 ucount = info->uprobe_multi.count;
int err = 0, i;
char *p, *buf;
long left = 0;
if (!upath ^ !upath_size)
return -EINVAL;
if ((uoffsets || uref_ctr_offsets || ucookies) && !ucount)
return -EINVAL;
umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
info->uprobe_multi.count = umulti_link->cnt;
info->uprobe_multi.flags = umulti_link->link.flags;
info->uprobe_multi.pid = umulti_link->task ?
task_pid_nr_ns(umulti_link->task, task_active_pid_ns(current)) : 0;
upath_size = upath_size ? min_t(u32, upath_size, PATH_MAX) : PATH_MAX;
buf = kmalloc(upath_size, GFP_KERNEL);
if (!buf)
return -ENOMEM;
p = d_path(&umulti_link->path, buf, upath_size);
if (IS_ERR(p)) {
kfree(buf);
return PTR_ERR(p);
}
upath_size = buf + upath_size - p;
if (upath)
left = copy_to_user(upath, p, upath_size);
kfree(buf);
if (left)
return -EFAULT;
info->uprobe_multi.path_size = upath_size;
if (!uoffsets && !ucookies && !uref_ctr_offsets)
return 0;
if (ucount < umulti_link->cnt)
err = -ENOSPC;
else
ucount = umulti_link->cnt;
for (i = 0; i < ucount; i++) {
if (uoffsets &&
put_user(umulti_link->uprobes[i].offset, uoffsets + i))
return -EFAULT;
if (uref_ctr_offsets &&
put_user(umulti_link->uprobes[i].ref_ctr_offset, uref_ctr_offsets + i))
return -EFAULT;
if (ucookies &&
put_user(umulti_link->uprobes[i].cookie, ucookies + i))
return -EFAULT;
}
return err;
}
#ifdef CONFIG_PROC_FS
static void bpf_uprobe_multi_show_fdinfo(const struct bpf_link *link,
struct seq_file *seq)
{
struct bpf_uprobe_multi_link *umulti_link;
char *p, *buf;
pid_t pid;
umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
buf = kmalloc(PATH_MAX, GFP_KERNEL);
if (!buf)
return;
p = d_path(&umulti_link->path, buf, PATH_MAX);
if (IS_ERR(p)) {
kfree(buf);
return;
}
pid = umulti_link->task ?
task_pid_nr_ns(umulti_link->task, task_active_pid_ns(current)) : 0;
seq_printf(seq,
"uprobe_cnt:\t%u\n"
"pid:\t%u\n"
"path:\t%s\n",
umulti_link->cnt, pid, p);
seq_printf(seq, "%s\t %s\t %s\n", "cookie", "offset", "ref_ctr_offset");
for (int i = 0; i < umulti_link->cnt; i++) {
seq_printf(seq,
"%llu\t %#llx\t %#lx\n",
umulti_link->uprobes[i].cookie,
umulti_link->uprobes[i].offset,
umulti_link->uprobes[i].ref_ctr_offset);
}
kfree(buf);
}
#endif
static const struct bpf_link_ops bpf_uprobe_multi_link_lops = {
.release = bpf_uprobe_multi_link_release,
.dealloc_deferred = bpf_uprobe_multi_link_dealloc,
.fill_link_info = bpf_uprobe_multi_link_fill_link_info,
#ifdef CONFIG_PROC_FS
.show_fdinfo = bpf_uprobe_multi_show_fdinfo,
#endif
};
static int uprobe_prog_run(struct bpf_uprobe *uprobe,
unsigned long entry_ip,
struct pt_regs *regs,
bool is_return, void *data)
{
struct bpf_uprobe_multi_link *link = uprobe->link;
struct bpf_uprobe_multi_run_ctx run_ctx = {
.session_ctx = {
.is_return = is_return,
.data = data,
},
.entry_ip = entry_ip,
.uprobe = uprobe,
};
struct bpf_prog *prog = link->link.prog;
bool sleepable = prog->sleepable;
struct bpf_run_ctx *old_run_ctx;
int err;
if (link->task && !same_thread_group(current, link->task))
return 0;
if (sleepable)
rcu_read_lock_trace();
else
rcu_read_lock();
migrate_disable();
old_run_ctx = bpf_set_run_ctx(&run_ctx.session_ctx.run_ctx);
err = bpf_prog_run(link->link.prog, regs);
bpf_reset_run_ctx(old_run_ctx);
migrate_enable();
if (sleepable)
rcu_read_unlock_trace();
else
rcu_read_unlock();
return err;
}
static bool
uprobe_multi_link_filter(struct uprobe_consumer *con, struct mm_struct *mm)
{
struct bpf_uprobe *uprobe;
uprobe = container_of(con, struct bpf_uprobe, consumer);
return uprobe->link->task->mm == mm;
}
static int
uprobe_multi_link_handler(struct uprobe_consumer *con, struct pt_regs *regs,
__u64 *data)
{
struct bpf_uprobe *uprobe;
int ret;
uprobe = container_of(con, struct bpf_uprobe, consumer);
ret = uprobe_prog_run(uprobe, instruction_pointer(regs), regs, false, data);
if (uprobe->session)
return ret ? UPROBE_HANDLER_IGNORE : 0;
return 0;
}
static int
uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, struct pt_regs *regs,
__u64 *data)
{
struct bpf_uprobe *uprobe;
uprobe = container_of(con, struct bpf_uprobe, consumer);
uprobe_prog_run(uprobe, func, regs, true, data);
return 0;
}
static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
{
struct bpf_uprobe_multi_run_ctx *run_ctx;
run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx,
session_ctx.run_ctx);
return run_ctx->entry_ip;
}
static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
{
struct bpf_uprobe_multi_run_ctx *run_ctx;
run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx,
session_ctx.run_ctx);
return run_ctx->uprobe->cookie;
}
int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
struct bpf_uprobe_multi_link *link = NULL;
unsigned long __user *uref_ctr_offsets;
struct bpf_link_primer link_primer;
struct bpf_uprobe *uprobes = NULL;
struct task_struct *task = NULL;
unsigned long __user *uoffsets;
u64 __user *ucookies;
void __user *upath;
u32 flags, cnt, i;
struct path path;
char *name;
pid_t pid;
int err;
/* no support for 32bit archs yet */
if (sizeof(u64) != sizeof(void *))
return -EOPNOTSUPP;
if (attr->link_create.flags)
return -EINVAL;
if (!is_uprobe_multi(prog))
return -EINVAL;
flags = attr->link_create.uprobe_multi.flags;
if (flags & ~BPF_F_UPROBE_MULTI_RETURN)
return -EINVAL;
/*
* path, offsets and cnt are mandatory,
* ref_ctr_offsets and cookies are optional
*/
upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path);
uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets);
cnt = attr->link_create.uprobe_multi.cnt;
pid = attr->link_create.uprobe_multi.pid;
if (!upath || !uoffsets || !cnt || pid < 0)
return -EINVAL;
if (cnt > MAX_UPROBE_MULTI_CNT)
return -E2BIG;
uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets);
ucookies = u64_to_user_ptr(attr->link_create.uprobe_multi.cookies);
name = strndup_user(upath, PATH_MAX);
if (IS_ERR(name)) {
err = PTR_ERR(name);
return err;
}
err = kern_path(name, LOOKUP_FOLLOW, &path);
kfree(name);
if (err)
return err;
if (!d_is_reg(path.dentry)) {
err = -EBADF;
goto error_path_put;
}
if (pid) {
rcu_read_lock();
task = get_pid_task(find_vpid(pid), PIDTYPE_TGID);
rcu_read_unlock();
if (!task) {
err = -ESRCH;
goto error_path_put;
}
}
err = -ENOMEM;
link = kzalloc_obj(*link);
uprobes = kvzalloc_objs(*uprobes, cnt);
if (!uprobes || !link)
goto error_free;
for (i = 0; i < cnt; i++) {
if (__get_user(uprobes[i].offset, uoffsets + i)) {
err = -EFAULT;
goto error_free;
}
if (uprobes[i].offset < 0) {
err = -EINVAL;
goto error_free;
}
if (uref_ctr_offsets && __get_user(uprobes[i].ref_ctr_offset, uref_ctr_offsets + i)) {
err = -EFAULT;
goto error_free;
}
if (ucookies && __get_user(uprobes[i].cookie, ucookies + i)) {
err = -EFAULT;
goto error_free;
}
uprobes[i].link = link;
if (!(flags & BPF_F_UPROBE_MULTI_RETURN))
uprobes[i].consumer.handler = uprobe_multi_link_handler;
if (flags & BPF_F_UPROBE_MULTI_RETURN || is_uprobe_session(prog))
uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
if (is_uprobe_session(prog))
uprobes[i].session = true;
if (pid)
uprobes[i].consumer.filter = uprobe_multi_link_filter;
}
link->cnt = cnt;
link->uprobes = uprobes;
link->path = path;
link->task = task;
link->link.flags = flags;
bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
&bpf_uprobe_multi_link_lops, prog, attr->link_create.attach_type);
for (i = 0; i < cnt; i++) {
uprobes[i].uprobe = uprobe_register(d_real_inode(link->path.dentry),
uprobes[i].offset,
uprobes[i].ref_ctr_offset,
&uprobes[i].consumer);
if (IS_ERR(uprobes[i].uprobe)) {
err = PTR_ERR(uprobes[i].uprobe);
link->cnt = i;
goto error_unregister;
}
}
err = bpf_link_prime(&link->link, &link_primer);
if (err)
goto error_unregister;
return bpf_link_settle(&link_primer);
error_unregister:
bpf_uprobe_unregister(uprobes, link->cnt);
error_free:
kvfree(uprobes);
kfree(link);
if (task)
put_task_struct(task);
error_path_put:
path_put(&path);
return err;
}
#else /* !CONFIG_UPROBES */
int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
return -EOPNOTSUPP;
}
static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
{
return 0;
}
static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
{
return 0;
}
#endif /* CONFIG_UPROBES */
__bpf_kfunc_start_defs();
__bpf_kfunc bool bpf_session_is_return(void *ctx)
{
struct bpf_session_run_ctx *session_ctx;
session_ctx = container_of(current->bpf_ctx, struct bpf_session_run_ctx, run_ctx);
return session_ctx->is_return;
}
__bpf_kfunc __u64 *bpf_session_cookie(void *ctx)
{
struct bpf_session_run_ctx *session_ctx;
session_ctx = container_of(current->bpf_ctx, struct bpf_session_run_ctx, run_ctx);
return session_ctx->data;
}
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(session_kfunc_set_ids)
BTF_ID_FLAGS(func, bpf_session_is_return)
BTF_ID_FLAGS(func, bpf_session_cookie)
BTF_KFUNCS_END(session_kfunc_set_ids)
static int bpf_session_filter(const struct bpf_prog *prog, u32 kfunc_id)
{
if (!btf_id_set8_contains(&session_kfunc_set_ids, kfunc_id))
return 0;
if (!is_kprobe_session(prog) && !is_uprobe_session(prog) && !is_trace_fsession(prog))
return -EACCES;
return 0;
}
static const struct btf_kfunc_id_set bpf_session_kfunc_set = {
.owner = THIS_MODULE,
.set = &session_kfunc_set_ids,
.filter = bpf_session_filter,
};
static int __init bpf_trace_kfuncs_init(void)
{
int err = 0;
err = err ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_KPROBE, &bpf_session_kfunc_set);
err = err ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_session_kfunc_set);
return err;
}
late_initcall(bpf_trace_kfuncs_init);
typedef int (*copy_fn_t)(void *dst, const void *src, u32 size, struct task_struct *tsk);
/*
* The __always_inline is to make sure the compiler doesn't
* generate indirect calls into callbacks, which is expensive,
* on some kernel configurations. This allows compiler to put
* direct calls into all the specific callback implementations
* (copy_user_data_sleepable, copy_user_data_nofault, and so on)
*/
static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u64 doff, u64 size,
const void *unsafe_src,
copy_fn_t str_copy_fn,
struct task_struct *tsk)
{
struct bpf_dynptr_kern *dst;
u64 chunk_sz, off;
void *dst_slice;
int cnt, err;
char buf[256];
dst_slice = bpf_dynptr_slice_rdwr(dptr, doff, NULL, size);
if (likely(dst_slice))
return str_copy_fn(dst_slice, unsafe_src, size, tsk);
dst = (struct bpf_dynptr_kern *)dptr;
if (bpf_dynptr_check_off_len(dst, doff, size))
return -E2BIG;
for (off = 0; off < size; off += chunk_sz - 1) {
chunk_sz = min_t(u64, sizeof(buf), size - off);
/* Expect str_copy_fn to return count of copied bytes, including
* zero terminator. Next iteration increment off by chunk_sz - 1 to
* overwrite NUL.
*/
cnt = str_copy_fn(buf, unsafe_src + off, chunk_sz, tsk);
if (cnt < 0)
return cnt;
err = __bpf_dynptr_write(dst, doff + off, buf, cnt, 0);
if (err)
return err;
if (cnt < chunk_sz || chunk_sz == 1) /* we are done */
return off + cnt;
}
return off;
}
static __always_inline int __bpf_dynptr_copy(const struct bpf_dynptr *dptr, u64 doff,
u64 size, const void *unsafe_src,
copy_fn_t copy_fn, struct task_struct *tsk)
{
struct bpf_dynptr_kern *dst;
void *dst_slice;
char buf[256];
u64 off, chunk_sz;
int err;
dst_slice = bpf_dynptr_slice_rdwr(dptr, doff, NULL, size);
if (likely(dst_slice))
return copy_fn(dst_slice, unsafe_src, size, tsk);
dst = (struct bpf_dynptr_kern *)dptr;
if (bpf_dynptr_check_off_len(dst, doff, size))
return -E2BIG;
for (off = 0; off < size; off += chunk_sz) {
chunk_sz = min_t(u64, sizeof(buf), size - off);
err = copy_fn(buf, unsafe_src + off, chunk_sz, tsk);
if (err)
return err;
err = __bpf_dynptr_write(dst, doff + off, buf, chunk_sz, 0);
if (err)
return err;
}
return 0;
}
static __always_inline int copy_user_data_nofault(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
return copy_from_user_nofault(dst, (const void __user *)unsafe_src, size);
}
static __always_inline int copy_user_data_sleepable(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
int ret;
if (!tsk) { /* Read from the current task */
ret = copy_from_user(dst, (const void __user *)unsafe_src, size);
if (ret)
return -EFAULT;
return 0;
}
ret = access_process_vm(tsk, (unsigned long)unsafe_src, dst, size, 0);
if (ret != size)
return -EFAULT;
return 0;
}
static __always_inline int copy_kernel_data_nofault(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
return copy_from_kernel_nofault(dst, unsafe_src, size);
}
static __always_inline int copy_user_str_nofault(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
return strncpy_from_user_nofault(dst, (const void __user *)unsafe_src, size);
}
static __always_inline int copy_user_str_sleepable(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
int ret;
if (unlikely(size == 0))
return 0;
if (tsk) {
ret = copy_remote_vm_str(tsk, (unsigned long)unsafe_src, dst, size, 0);
} else {
ret = strncpy_from_user(dst, (const void __user *)unsafe_src, size - 1);
/* strncpy_from_user does not guarantee NUL termination */
if (ret >= 0)
((char *)dst)[ret] = '\0';
}
if (ret < 0)
return ret;
return ret + 1;
}
static __always_inline int copy_kernel_str_nofault(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
return strncpy_from_kernel_nofault(dst, unsafe_src, size);
}
__bpf_kfunc_start_defs();
__bpf_kfunc int bpf_send_signal_task(struct task_struct *task, int sig, enum pid_type type,
u64 value)
{
if (type != PIDTYPE_PID && type != PIDTYPE_TGID)
return -EINVAL;
return bpf_send_signal_common(sig, type, task, value);
}
__bpf_kfunc int bpf_probe_read_user_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_data_nofault, NULL);
}
__bpf_kfunc int bpf_probe_read_kernel_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void *unsafe_ptr__ign)
{
return __bpf_dynptr_copy(dptr, off, size, unsafe_ptr__ign,
copy_kernel_data_nofault, NULL);
}
__bpf_kfunc int bpf_probe_read_user_str_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_str_nofault, NULL);
}
__bpf_kfunc int bpf_probe_read_kernel_str_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void *unsafe_ptr__ign)
{
return __bpf_dynptr_copy_str(dptr, off, size, unsafe_ptr__ign,
copy_kernel_str_nofault, NULL);
}
__bpf_kfunc int bpf_copy_from_user_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_data_sleepable, NULL);
}
__bpf_kfunc int bpf_copy_from_user_str_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_str_sleepable, NULL);
}
__bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign,
struct task_struct *tsk)
{
return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_data_sleepable, tsk);
}
__bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign,
struct task_struct *tsk)
{
return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_str_sleepable, tsk);
}
__bpf_kfunc_end_defs();
]
|
| 13/2 |
2026/03/15 01:22 |
llm |
expert |
0m
Model:
Tokens:
input: 49614
output: 47
thoughts: 826
Thoughts: **Analyzing a Potential Use-After-Free Vulnerability in BPF Tracepoints**
Okay, so I'm looking at this `__bpf_trace_run` function and immediately a red flag pops up at line 2075: `struct bpf_prog *prog = link->link.prog;`. If `link` (a `struct bpf_raw_tp_link *`) has already been freed, that's a classic Use-After-Free (UAF). The code's trying to access memory that's been deallocated, which is never good.
I need to trace back how `link` gets allocated and freed to confirm this. The allocation seems to be in `bpf_raw_tp_link_attach`. Then I see the corresponding free happening via RCU, triggered by `bpf_link_put_direct`, which is ultimately called by `bpf_link_release` in `__fput`.
But, wait... this function is called from a tracepoint. Tracepoints are *supposed* to use RCU to protect against concurrent access during the execution of a probe. The key here is the link between the `bpf_raw_tp_link` and the tracepoint itself. `bpf_raw_tp_link` is attached to a tracepoint in `bpf_probe_register`:
```c
2158: int bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_raw_tp_link *link)
2159: {
...
2172: return tracepoint_probe_register_may_exist(tp, (void *)btp->bpf_func, link);
2173: }
```
The function `btp->bpf_func`, is one of the `bpf_trace_run1`, `bpf_trace_run2`, etc., which *calls* my suspect `__bpf_trace_run` function. The `link` itself is passed as a first argument to the probe function when the tracepoint is triggered. The RCU should protect me.
When a tracepoint is hit, the probe function is called. RCU is supposed to make sure all of this is thread-safe and prevent race conditions. So if this `bpf_raw_tp_link` has been freed, it should *absolutely* have been unregistered from the tracepoint first. The unregistration would happen in `bpf_raw_tp_link_release` (or something similar).
I need to dig deeper into the `bpf_raw_tp_link` and how it's released in `kernel/bpf/syscall.c` and figure out the exact sequence. Something is clearly broken if this UAF exists in a function protected by RCU. I need to understand the timing of the registration, unregistration, and execution of the probe. There's a gap in my mental model here, and I need to close it.
|
| 14/2 |
2026/03/15 01:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/syscall.c IncludeLines:true Name:bpf_raw_tp_link_attach]
Results: map[SourceCode:4223: static int bpf_raw_tp_link_attach(struct bpf_prog *prog,
4224: const char __user *user_tp_name, u64 cookie,
4225: enum bpf_attach_type attach_type)
4226: {
4227: struct bpf_link_primer link_primer;
4228: struct bpf_raw_tp_link *link;
4229: struct bpf_raw_event_map *btp;
4230: const char *tp_name;
4231: char buf[128];
4232: int err;
4233:
4234: switch (prog->type) {
4235: case BPF_PROG_TYPE_TRACING:
4236: case BPF_PROG_TYPE_EXT:
4237: case BPF_PROG_TYPE_LSM:
4238: if (user_tp_name)
4239: /* The attach point for this category of programs
4240: * should be specified via btf_id during program load.
4241: */
4242: return -EINVAL;
4243: if (prog->type == BPF_PROG_TYPE_TRACING &&
4244: prog->expected_attach_type == BPF_TRACE_RAW_TP) {
4245: tp_name = prog->aux->attach_func_name;
4246: break;
4247: }
4248: return bpf_tracing_prog_attach(prog, 0, 0, 0, attach_type);
4249: case BPF_PROG_TYPE_RAW_TRACEPOINT:
4250: case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:
4251: if (strncpy_from_user(buf, user_tp_name, sizeof(buf) - 1) < 0)
4252: return -EFAULT;
4253: buf[sizeof(buf) - 1] = 0;
4254: tp_name = buf;
4255: break;
4256: default:
4257: return -EINVAL;
4258: }
4259:
4260: btp = bpf_get_raw_tracepoint(tp_name);
4261: if (!btp)
4262: return -ENOENT;
4263:
4264: link = kzalloc_obj(*link, GFP_USER);
4265: if (!link) {
4266: err = -ENOMEM;
4267: goto out_put_btp;
4268: }
4269: bpf_link_init_sleepable(&link->link, BPF_LINK_TYPE_RAW_TRACEPOINT,
4270: &bpf_raw_tp_link_lops, prog, attach_type,
4271: tracepoint_is_faultable(btp->tp));
4272: link->btp = btp;
4273: link->cookie = cookie;
4274:
4275: err = bpf_link_prime(&link->link, &link_primer);
4276: if (err) {
4277: kfree(link);
4278: goto out_put_btp;
4279: }
4280:
4281: err = bpf_probe_register(link->btp, link);
4282: if (err) {
4283: bpf_link_cleanup(&link_primer);
4284: goto out_put_btp;
4285: }
4286:
4287: return bpf_link_settle(&link_primer);
4288:
4289: out_put_btp:
4290: bpf_put_raw_tracepoint(btp);
4291: return err;
4292: }
SourceFile:kernel/bpf/syscall.c]
|
| 15/2 |
2026/03/15 01:22 |
llm |
expert |
0m
Model:
Tokens:
input: 52263
output: 48
thoughts: 14
Thoughts: Okay, here's my summary, taking on the persona you requested:
**Deconstructing `bpf_raw_tp_link_lops`**
Alright, let's break this down. When I see `bpf_raw_tp_link_lops`, I immediately start thinking about tracing and eBPF. The "bpf" prefix screams Berkeley Packet Filter, which in its extended form is eBPF. The "raw" suggests we're dealing with something very close to the metal, probably interacting directly with raw tracepoints. The "tp" strongly indicates tracepoints specifically, likely kernel tracepoints. Finally, "link_lops"... that's the interesting part. "Link" hints at how we attach the eBPF programs, likely to connect to the tracepoints in order to intercept events, and "lops" is probably the name of the operation. So, in effect, this sounds like the underlying function that links eBPF programs to raw kernel tracepoints. I'm almost certainly talking about attaching and detaching eBPF programs to tracepoint events that don't involve the usual kprobe/uprobe systems, bypassing the standard attachment logic. This implies an architecture level control mechanism, that allows finer grained interaction and control, rather than using the usual kprobe/uprobe methods. This suggests advanced use cases where we need extremely high performance or very precise control over event capture. I need to look into its implementation details to see the specific operations for connecting and disconnecting.
|
| 16/2 |
2026/03/15 01:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/syscall.c IncludeLines:true Name:bpf_raw_tp_link_lops]
Results: map[SourceCode:3850: static const struct bpf_link_ops bpf_raw_tp_link_lops = {
3851: .release = bpf_raw_tp_link_release,
3852: .dealloc_deferred = bpf_raw_tp_link_dealloc,
3853: .show_fdinfo = bpf_raw_tp_link_show_fdinfo,
3854: .fill_link_info = bpf_raw_tp_link_fill_link_info,
3855: };
SourceFile:kernel/bpf/syscall.c]
|
| 17/2 |
2026/03/15 01:22 |
llm |
expert |
0m
Model:
Tokens:
input: 52809
output: 47
thoughts: 30
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Examining BPF Tracepoint Link Release and Deallocation**
Alright, let's dive in. My mind is immediately drawn to the internals of `bpf_raw_tp_link_release` and `bpf_raw_tp_link_dealloc`. I need to ensure the correct interaction between these two functions. I'm thinking about the reference counting, resource clean-up, and potential race conditions. Is the release function just decrementing a reference, or is it doing more intricate bookkeeping with regards to the BPF program associated with the raw tracepoint? Dealloc... is that just free'ing the memory? Or is there any complex state that needs to be taken into account before the memory is released? I am mentally sketching out the code flow, looking for potential areas of concern, particularly around concurrent access to shared data structures like the link's metadata or any global lists. I need to refresh my understanding of how the tracepoint links are managed within the broader BPF framework to ensure that any potential logic errors, like double frees or memory leaks, are avoided.
|
| 18/2 |
2026/03/15 01:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/syscall.c IncludeLines:true Name:bpf_raw_tp_link_release]
Results: map[SourceCode:3779: static void bpf_raw_tp_link_release(struct bpf_link *link)
3780: {
3781: struct bpf_raw_tp_link *raw_tp =
3782: container_of(link, struct bpf_raw_tp_link, link);
3783:
3784: bpf_probe_unregister(raw_tp->btp, raw_tp);
3785: bpf_put_raw_tracepoint(raw_tp->btp);
3786: }
SourceFile:kernel/bpf/syscall.c]
|
| 19/2 |
2026/03/15 01:22 |
llm |
expert |
0m
Model:
Tokens:
input: 53305
output: 48
thoughts: 14
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:
**My Internal Processing on `bpf_raw_tp_link_dealloc`**
Right, let's break this down. The mention of `bpf_raw_tp_link_dealloc` immediately signals a deep dive into the BPF (Berkeley Packet Filter) world, specifically the realm of tracepoints. I need to think about what this function *does*. The name itself screams "deallocation," "link," and "raw tracepoint." So, I'm already visualizing the underlying data structures: the tracepoint link itself, likely some associated metadata, and of course, the BPF program it's connected to.
My first thought is around memory management. This deallocation function must be handling the critical aspects of freeing up resources tied to a BPF tracepoint link. I'm picturing the potential complexities β ensuring proper reference counting if there are multiple BPF programs linked to the same tracepoint, managing any associated internal state, and, most importantly, preventing any use-after-free scenarios. This is critical in a kernel environment.
Then I start asking myself about the potential failure modes. What are the potential error conditions this function needs to handle? Null pointer checks are a given, but what about race conditions during deallocation? Concurrent access from different contexts is a huge concern in the kernel. I need to be thinking about locks, atomic operations, and how this function ensures data integrity during the deallocation process. Are there any specific error codes I need to be aware of?
Beyond memory management, I'm considering what side effects this might have on the rest of the system. Does this deallocation necessitate any changes to the tracepoint's state? Does it require notifying other subsystems? Is there a need to invalidate any caches or perform any cleanup actions? I have to consider the ripple effects through the system.
Essentially, I'm mentally walking through the logic, step-by-step, making sure all bases are covered. I'm looking for potential vulnerabilities, bottlenecks, and areas for optimization. This function is a cornerstone, so its implementation, correctness, and efficiency are paramount. I need to fully understand its function.
|
| 20/2 |
2026/03/15 01:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/syscall.c IncludeLines:true Name:bpf_raw_tp_link_dealloc]
Results: map[SourceCode:3788: static void bpf_raw_tp_link_dealloc(struct bpf_link *link)
3789: {
3790: struct bpf_raw_tp_link *raw_tp =
3791: container_of(link, struct bpf_raw_tp_link, link);
3792:
3793: kfree(raw_tp);
3794: }
SourceFile:kernel/bpf/syscall.c]
|
| 21/2 |
2026/03/15 01:22 |
llm |
expert |
0m
Model:
Tokens:
input: 53981
output: 45
thoughts: 176
Thoughts: **Deconstructing BPF Link Initialization and Deallocation**
Okay, so I'm looking at this `bpf_raw_tp_link_attach` function and immediately my eye is drawn to the `bpf_link_init_sleepable` call. The inclusion of "sleepable" in the function name immediately flags something to me. If it's *sleepable*, there's a good chance it's employing a mechanism like `synchronize_rcu_tasks_trace()` or a similar RCU synchronization primitive. That suggests a need to gracefully handle deallocation, especially in scenarios involving ongoing tracing or event processing.
So, where is this link getting deallocated? The crucial part will be how `bpf_link_put_direct` handles everything. It's the deallocation side that I'll be particularly interested in. Let me jump into that and see how they are handling potential race conditions and graceful shutdown of this link. I bet there's some interesting locking or RCU-related logic lurking there.
|
| 22/2 |
2026/03/15 01:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/syscall.c IncludeLines:true Name:bpf_link_put_direct]
Results: map[SourceCode:3319: static void bpf_link_put_direct(struct bpf_link *link)
3320: {
3321: if (!atomic64_dec_and_test(&link->refcnt))
3322: return;
3323: bpf_link_free(link);
3324: }
SourceFile:kernel/bpf/syscall.c]
|
| 23/2 |
2026/03/15 01:22 |
llm |
expert |
0m
Model:
Tokens:
input: 54541
output: 43
thoughts: 9
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:
**My BPF Link Management Musings**
Alright, so I'm thinking about `bpf_link_free`. Let's break this down. We're talking about *freeing* a BPF link. We all know how crucial proper resource management is in BPF, and this function is at the heart of that. When a BPF link is no longer needed β maybe we're tearing down a specific program attachment, or a tracepoint is being unregistered β we *must* release it. Otherwise, weβre looking at memory leaks, resource exhaustion, and potentially a very unstable system, right?
I'm probably considering the various scenarios where `bpf_link_free` gets called. Is it always a direct call from my program, or are we dealing with cleanup routines triggered by the kernel, perhaps within a `struct bpf_link` lifecycle? It's important to understand the ownership and responsibility here. Did I create the link, so I am responsible for freeing it? And the atomic nature of the operations: what's the guaranteed consistency?
Furthermore, I'm likely picturing the internal workings. What's actually happening at the kernel level? Does `bpf_link_free` dereference and deallocate the associated data structures, removing the link from any internal linked lists or hash tables? Are there any reference counts involved, and how are those handled to prevent double-frees or premature deallocations? Iβd be thinking of race conditions as well: what protections are in place to ensure that the link isn't in use by another BPF program or subsystem while I'm attempting to free it? We *cannot* have corruption.
And, of course, error handling. What happens if `bpf_link_free` fails? Does it return an error code, and what are the implications of that? What kind of recovery mechanisms, if any, are needed in my code to handle those failures gracefully, to ensure continued stability and prevent resource leaks, even in the face of unexpected issues? Error propagation is key.
Basically, I'm thinking about the *implications* of this function call. Itβs not just a technicality; itβs a critical piece of the BPF puzzle for long-lived systems. Failure to use it correctly guarantees pain later.
|
| 24/2 |
2026/03/15 01:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/syscall.c IncludeLines:true Name:bpf_link_free]
Results: map[SourceCode:3272: /* bpf_link_free is guaranteed to be called from process context */
3273: static void bpf_link_free(struct bpf_link *link)
3274: {
3275: const struct bpf_link_ops *ops = link->ops;
3276:
3277: bpf_link_free_id(link->id);
3278: /* detach BPF program, clean up used resources */
3279: if (link->prog)
3280: ops->release(link);
3281: if (ops->dealloc_deferred) {
3282: /* Schedule BPF link deallocation, which will only then
3283: * trigger putting BPF program refcount.
3284: * If underlying BPF program is sleepable or BPF link's target
3285: * attach hookpoint is sleepable or otherwise requires RCU GPs
3286: * to ensure link and its underlying BPF program is not
3287: * reachable anymore, we need to first wait for RCU tasks
3288: * trace sync, and then go through "classic" RCU grace period
3289: */
3290: if (link->sleepable || (link->prog && link->prog->sleepable))
3291: call_rcu_tasks_trace(&link->rcu, bpf_link_defer_dealloc_mult_rcu_gp);
3292: else
3293: call_rcu(&link->rcu, bpf_link_defer_dealloc_rcu_gp);
3294: } else if (ops->dealloc) {
3295: bpf_link_dealloc(link);
3296: }
3297: }
SourceFile:kernel/bpf/syscall.c]
|
| 25/2 |
2026/03/15 01:23 |
llm |
expert |
0m
Model:
Tokens:
input: 55599
output: 23
thoughts: 815
Thoughts: **Analyzing a Potential Use-After-Free in BPF Tracepoint Handling**
Okay, let's break this down. I'm seeing a possible Use-After-Free (UAF) issue within the BPF tracepoint infrastructure, specifically when dealing with the `mm_page_alloc` tracepoint. The code path involved in freeing a `bpf_raw_tp_link` looks like this:
1. `bpf_link_free` is called.
2. Within that, `ops->release(link)` is invoked. For `bpf_raw_tp_link`, this resolves to `bpf_raw_tp_link_release`.
3. `bpf_raw_tp_link_release` then calls `bpf_probe_unregister`.
4. `bpf_probe_unregister` delegates to `tracepoint_probe_unregister`.
5. Finally, either `call_rcu` or `call_rcu_tasks_trace` is used to schedule the deallocation, which is done through `bpf_raw_tp_link_dealloc`. This part looks correct.
Now, if `link->sleepable` is false, it uses `call_rcu`. `link->sleepable` depends on `tracepoint_is_faultable(btp->tp)`. Based on the name and general nature of `mm_page_alloc`, it's almost certainly *not* a faultable tracepoint. So we're likely using `call_rcu`. The crucial point here is that `call_rcu` is supposed to give a grace period before the actual deallocation, preventing the UAF.
`tracepoint_probe_unregister` itself doesn't wait; it merely removes the probe from the list. The caller, in this case `bpf_link_free` which calls the link's `release` function, is responsible for waiting. `call_rcu` is how `bpf_link_free` waits.
The RCU grace period should prevent the UAF:
1. `bpf_raw_tp_link_release` leads to `tracepoint_probe_unregister`.
2. `call_rcu` is then initiated, waiting for the RCU grace period.
3. Eventually, a `bpf_link_defer_dealloc_rcu_gp` function is called, which then invokes `ops->dealloc_deferred`, which, for this type of link, results in `bpf_raw_tp_link_dealloc` and finally `kfree(link)`.
The UAF is reported to be in `__bpf_trace_run`. This function is where the BPF program actually executes when a tracepoint fires. Specifically, it's called from the tracepoint itself: `__traceiter_mm_page_alloc` via the `bpf_trace_run4` function. Regular tracepoints *should* protect probe iteration with `rcu_read_lock()`. Therefore, `__bpf_trace_run` must be inside an RCU read-side critical section. The `kfree(link)` in the deferred deallocation should only happen *after* all RCU read-side critical sections that were active when the RCU call was initiated have finished.
So, the question is: **Why are we getting a UAF?** The code path *should* be safe.
The key is that the tracepoint uses RCU. Let's look at the stack trace again:
```
__bpf_trace_run kernel/trace/bpf_trace.c:2075 [inline]
bpf_trace_run4+0xe6/0x850 kernel/trace/bpf_trace.c:2131
__traceiter_mm_page_alloc+0x3d/0x60 include/trace/events/kmem.h:180
__do_trace_mm_page_alloc include/trace/events/kmem.h:180 [inline]
trace_mm_page_alloc+0x149/0x180 include/trace/events/kmem.h:180
```
That confirms the direct connection. Now, let's examine how tracepoints are *called* within `include/linux/tracepoint.h`. I need to figure out how the `__traceiter_mm_page_alloc` function calls its probes to confirm it is actually using regular RCU. This is the crucial missing piece. If it bypasses the normal RCU mechanisms for some reason, then we have a race condition.
|
| 26/2 |
2026/03/15 01:23 |
tool |
read-file |
0m
Args: map[File:include/linux/tracepoint.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0-only */
#ifndef _LINUX_TRACEPOINT_H
#define _LINUX_TRACEPOINT_H
/*
* Kernel Tracepoint API.
*
* See Documentation/trace/tracepoints.rst.
*
* Copyright (C) 2008-2014 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
*
* Heavily inspired from the Linux Kernel Markers.
*/
#include <linux/smp.h>
#include <linux/srcu.h>
#include <linux/errno.h>
#include <linux/types.h>
#include <linux/rcupdate.h>
#include <linux/rcupdate_trace.h>
#include <linux/tracepoint-defs.h>
#include <linux/static_call.h>
struct module;
struct tracepoint;
struct notifier_block;
struct trace_eval_map {
const char *system;
const char *eval_string;
unsigned long eval_value;
};
#define TRACEPOINT_DEFAULT_PRIO 10
extern int
tracepoint_probe_register(struct tracepoint *tp, void *probe, void *data);
extern int
tracepoint_probe_register_prio(struct tracepoint *tp, void *probe, void *data,
int prio);
extern int
tracepoint_probe_register_prio_may_exist(struct tracepoint *tp, void *probe, void *data,
int prio);
extern int
tracepoint_probe_unregister(struct tracepoint *tp, void *probe, void *data);
static inline int
tracepoint_probe_register_may_exist(struct tracepoint *tp, void *probe,
void *data)
{
return tracepoint_probe_register_prio_may_exist(tp, probe, data,
TRACEPOINT_DEFAULT_PRIO);
}
extern void
for_each_kernel_tracepoint(void (*fct)(struct tracepoint *tp, void *priv),
void *priv);
#ifdef CONFIG_MODULES
struct tp_module {
struct list_head list;
struct module *mod;
};
bool trace_module_has_bad_taint(struct module *mod);
extern int register_tracepoint_module_notifier(struct notifier_block *nb);
extern int unregister_tracepoint_module_notifier(struct notifier_block *nb);
void for_each_module_tracepoint(void (*fct)(struct tracepoint *,
struct module *, void *),
void *priv);
void for_each_tracepoint_in_module(struct module *,
void (*fct)(struct tracepoint *,
struct module *, void *),
void *priv);
#else
static inline bool trace_module_has_bad_taint(struct module *mod)
{
return false;
}
static inline
int register_tracepoint_module_notifier(struct notifier_block *nb)
{
return 0;
}
static inline
int unregister_tracepoint_module_notifier(struct notifier_block *nb)
{
return 0;
}
static inline
void for_each_module_tracepoint(void (*fct)(struct tracepoint *,
struct module *, void *),
void *priv)
{
}
static inline
void for_each_tracepoint_in_module(struct module *mod,
void (*fct)(struct tracepoint *,
struct module *, void *),
void *priv)
{
}
#endif /* CONFIG_MODULES */
/*
* tracepoint_synchronize_unregister must be called between the last tracepoint
* probe unregistration and the end of module exit to make sure there is no
* caller executing a probe when it is freed.
*
* An alternative is to use the following for batch reclaim associated
* with a given tracepoint:
*
* - tracepoint_is_faultable() == false: call_srcu()
* - tracepoint_is_faultable() == true: call_rcu_tasks_trace()
*/
#ifdef CONFIG_TRACEPOINTS
extern struct srcu_struct tracepoint_srcu;
static inline void tracepoint_synchronize_unregister(void)
{
synchronize_rcu_tasks_trace();
synchronize_srcu(&tracepoint_srcu);
}
static inline bool tracepoint_is_faultable(struct tracepoint *tp)
{
return tp->ext && tp->ext->faultable;
}
#else
static inline void tracepoint_synchronize_unregister(void)
{ }
static inline bool tracepoint_is_faultable(struct tracepoint *tp)
{
return false;
}
#endif
#ifdef CONFIG_HAVE_SYSCALL_TRACEPOINTS
extern int syscall_regfunc(void);
extern void syscall_unregfunc(void);
#endif /* CONFIG_HAVE_SYSCALL_TRACEPOINTS */
#ifndef PARAMS
#define PARAMS(args...) args
#endif
#define TRACE_DEFINE_ENUM(x)
#define TRACE_DEFINE_SIZEOF(x)
#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS
static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)
{
return offset_to_ptr(p);
}
#define __TRACEPOINT_ENTRY(name) \
asm(" .section \"__tracepoints_ptrs\", \"a\" \n" \
" .balign 4 \n" \
" .long __tracepoint_" #name " - . \n" \
" .previous \n")
#else
static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)
{
return *p;
}
#define __TRACEPOINT_ENTRY(name) \
static tracepoint_ptr_t __tracepoint_ptr_##name __used \
__section("__tracepoints_ptrs") = &__tracepoint_##name
#endif
#endif /* _LINUX_TRACEPOINT_H */
/*
* Note: we keep the TRACE_EVENT and DECLARE_TRACE outside the include
* file ifdef protection.
* This is due to the way trace events work. If a file includes two
* trace event headers under one "CREATE_TRACE_POINTS" the first include
* will override the TRACE_EVENT and break the second include.
*/
#ifndef DECLARE_TRACE
#define TP_PROTO(args...) args
#define TP_ARGS(args...) args
#define TP_CONDITION(args...) args
/*
* Individual subsystem my have a separate configuration to
* enable their tracepoints. By default, this file will create
* the tracepoints if CONFIG_TRACEPOINTS is defined. If a subsystem
* wants to be able to disable its tracepoints from being created
* it can define NOTRACE before including the tracepoint headers.
*/
#if defined(CONFIG_TRACEPOINTS) && !defined(NOTRACE)
#define TRACEPOINTS_ENABLED
#endif
#ifdef TRACEPOINTS_ENABLED
#ifdef CONFIG_HAVE_STATIC_CALL
#define __DO_TRACE_CALL(name, args) \
do { \
struct tracepoint_func *it_func_ptr; \
void *__data; \
it_func_ptr = \
rcu_dereference_raw((&__tracepoint_##name)->funcs); \
if (it_func_ptr) { \
__data = (it_func_ptr)->data; \
static_call(tp_func_##name)(__data, args); \
} \
} while (0)
#else
#define __DO_TRACE_CALL(name, args) __traceiter_##name(NULL, args)
#endif /* CONFIG_HAVE_STATIC_CALL */
/*
* Declare an exported function that Rust code can call to trigger this
* tracepoint. This function does not include the static branch; that is done
* in Rust to avoid a function call when the tracepoint is disabled.
*/
#define DEFINE_RUST_DO_TRACE(name, proto, args)
#define __DEFINE_RUST_DO_TRACE(name, proto, args) \
notrace void rust_do_trace_##name(proto) \
{ \
__do_trace_##name(args); \
}
/*
* When a tracepoint is used, it's name is added to the __tracepoint_check
* section. This section is only used at build time to make sure all
* defined tracepoints are used. It is discarded after the build.
*/
# define TRACEPOINT_CHECK(name) \
static const char __used __section("__tracepoint_check") \
__trace_check_##name[] = #name;
/*
* Make sure the alignment of the structure in the __tracepoints section will
* not add unwanted padding between the beginning of the section and the
* structure. Force alignment to the same alignment as the section start.
*
* When lockdep is enabled, we make sure to always test if RCU is
* "watching" regardless if the tracepoint is enabled or not. Tracepoints
* require RCU to be active, and it should always warn at the tracepoint
* site if it is not watching, as it will need to be active when the
* tracepoint is enabled.
*/
#define __DECLARE_TRACE_COMMON(name, proto, args, data_proto) \
extern int __traceiter_##name(data_proto); \
DECLARE_STATIC_CALL(tp_func_##name, __traceiter_##name); \
extern struct tracepoint __tracepoint_##name; \
extern void rust_do_trace_##name(proto); \
static inline int \
register_trace_##name(void (*probe)(data_proto), void *data) \
{ \
return tracepoint_probe_register(&__tracepoint_##name, \
(void *)probe, data); \
} \
static inline int \
register_trace_prio_##name(void (*probe)(data_proto), void *data,\
int prio) \
{ \
return tracepoint_probe_register_prio(&__tracepoint_##name, \
(void *)probe, data, prio); \
} \
static inline int \
unregister_trace_##name(void (*probe)(data_proto), void *data) \
{ \
return tracepoint_probe_unregister(&__tracepoint_##name,\
(void *)probe, data); \
} \
static inline void \
check_trace_callback_type_##name(void (*cb)(data_proto)) \
{ \
} \
static inline bool \
trace_##name##_enabled(void) \
{ \
return static_branch_unlikely(&__tracepoint_##name.key);\
}
#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \
__DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_proto)) \
static inline void __do_trace_##name(proto) \
{ \
TRACEPOINT_CHECK(name) \
if (cond) { \
guard(srcu_fast_notrace)(&tracepoint_srcu); \
__DO_TRACE_CALL(name, TP_ARGS(args)); \
} \
} \
static inline void trace_##name(proto) \
{ \
if (static_branch_unlikely(&__tracepoint_##name.key)) \
__do_trace_##name(args); \
if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \
WARN_ONCE(!rcu_is_watching(), \
"RCU not watching for tracepoint"); \
} \
}
#define __DECLARE_TRACE_SYSCALL(name, proto, args, data_proto) \
__DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_proto)) \
static inline void __do_trace_##name(proto) \
{ \
TRACEPOINT_CHECK(name) \
guard(rcu_tasks_trace)(); \
__DO_TRACE_CALL(name, TP_ARGS(args)); \
} \
static inline void trace_##name(proto) \
{ \
might_fault(); \
if (static_branch_unlikely(&__tracepoint_##name.key)) \
__do_trace_##name(args); \
if (IS_ENABLED(CONFIG_LOCKDEP)) { \
WARN_ONCE(!rcu_is_watching(), \
"RCU not watching for tracepoint"); \
} \
}
/*
* We have no guarantee that gcc and the linker won't up-align the tracepoint
* structures, so we create an array of pointers that will be used for iteration
* on the tracepoints.
*
* it_func[0] is never NULL because there is at least one element in the array
* when the array itself is non NULL.
*/
#define __DEFINE_TRACE_EXT(_name, _ext, proto, args) \
static const char __tpstrtab_##_name[] \
__section("__tracepoints_strings") = #_name; \
extern struct static_call_key STATIC_CALL_KEY(tp_func_##_name); \
int __traceiter_##_name(void *__data, proto); \
void __probestub_##_name(void *__data, proto); \
struct tracepoint __tracepoint_##_name __used \
__section("__tracepoints") = { \
.name = __tpstrtab_##_name, \
.key = STATIC_KEY_FALSE_INIT, \
.static_call_key = &STATIC_CALL_KEY(tp_func_##_name), \
.static_call_tramp = STATIC_CALL_TRAMP_ADDR(tp_func_##_name), \
.iterator = &__traceiter_##_name, \
.probestub = &__probestub_##_name, \
.funcs = NULL, \
.ext = _ext, \
}; \
__TRACEPOINT_ENTRY(_name); \
int __traceiter_##_name(void *__data, proto) \
{ \
struct tracepoint_func *it_func_ptr; \
void *it_func; \
\
it_func_ptr = \
rcu_dereference_raw((&__tracepoint_##_name)->funcs); \
if (it_func_ptr) { \
do { \
it_func = READ_ONCE((it_func_ptr)->func); \
__data = (it_func_ptr)->data; \
((void(*)(void *, proto))(it_func))(__data, args); \
} while ((++it_func_ptr)->func); \
} \
return 0; \
} \
void __probestub_##_name(void *__data, proto) \
{ \
} \
DEFINE_STATIC_CALL(tp_func_##_name, __traceiter_##_name); \
DEFINE_RUST_DO_TRACE(_name, TP_PROTO(proto), TP_ARGS(args))
#define DEFINE_TRACE_FN(_name, _reg, _unreg, _proto, _args) \
static struct tracepoint_ext __tracepoint_ext_##_name = { \
.regfunc = _reg, \
.unregfunc = _unreg, \
.faultable = false, \
}; \
__DEFINE_TRACE_EXT(_name, &__tracepoint_ext_##_name, PARAMS(_proto), PARAMS(_args));
#define DEFINE_TRACE_SYSCALL(_name, _reg, _unreg, _proto, _args) \
static struct tracepoint_ext __tracepoint_ext_##_name = { \
.regfunc = _reg, \
.unregfunc = _unreg, \
.faultable = true, \
}; \
__DEFINE_TRACE_EXT(_name, &__tracepoint_ext_##_name, PARAMS(_proto), PARAMS(_args));
#define DEFINE_TRACE(_name, _proto, _args) \
__DEFINE_TRACE_EXT(_name, NULL, PARAMS(_proto), PARAMS(_args));
#define EXPORT_TRACEPOINT_SYMBOL_GPL(name) \
TRACEPOINT_CHECK(name) \
EXPORT_SYMBOL_GPL(__tracepoint_##name); \
EXPORT_SYMBOL_GPL(__traceiter_##name); \
EXPORT_STATIC_CALL_GPL(tp_func_##name)
#define EXPORT_TRACEPOINT_SYMBOL(name) \
TRACEPOINT_CHECK(name) \
EXPORT_SYMBOL(__tracepoint_##name); \
EXPORT_SYMBOL(__traceiter_##name); \
EXPORT_STATIC_CALL(tp_func_##name)
#else /* !TRACEPOINTS_ENABLED */
#define __DECLARE_TRACE_COMMON(name, proto, args, data_proto) \
static inline void trace_##name(proto) \
{ } \
static inline int \
register_trace_##name(void (*probe)(data_proto), \
void *data) \
{ \
return -ENOSYS; \
} \
static inline int \
unregister_trace_##name(void (*probe)(data_proto), \
void *data) \
{ \
return -ENOSYS; \
} \
static inline void check_trace_callback_type_##name(void (*cb)(data_proto)) \
{ \
} \
static inline bool \
trace_##name##_enabled(void) \
{ \
return false; \
}
#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \
__DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_proto))
#define __DECLARE_TRACE_SYSCALL(name, proto, args, data_proto) \
__DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_proto))
#define DEFINE_TRACE_FN(name, reg, unreg, proto, args)
#define DEFINE_TRACE_SYSCALL(name, reg, unreg, proto, args)
#define DEFINE_TRACE(name, proto, args)
#define EXPORT_TRACEPOINT_SYMBOL_GPL(name)
#define EXPORT_TRACEPOINT_SYMBOL(name)
#endif /* TRACEPOINTS_ENABLED */
#ifdef CONFIG_TRACING
/**
* tracepoint_string - register constant persistent string to trace system
* @str - a constant persistent string that will be referenced in tracepoints
*
* If constant strings are being used in tracepoints, it is faster and
* more efficient to just save the pointer to the string and reference
* that with a printf "%s" instead of saving the string in the ring buffer
* and wasting space and time.
*
* The problem with the above approach is that userspace tools that read
* the binary output of the trace buffers do not have access to the string.
* Instead they just show the address of the string which is not very
* useful to users.
*
* With tracepoint_string(), the string will be registered to the tracing
* system and exported to userspace via the debugfs/tracing/printk_formats
* file that maps the string address to the string text. This way userspace
* tools that read the binary buffers have a way to map the pointers to
* the ASCII strings they represent.
*
* The @str used must be a constant string and persistent as it would not
* make sense to show a string that no longer exists. But it is still fine
* to be used with modules, because when modules are unloaded, if they
* had tracepoints, the ring buffers are cleared too. As long as the string
* does not change during the life of the module, it is fine to use
* tracepoint_string() within a module.
*/
#define tracepoint_string(str) \
({ \
static const char *___tp_str __tracepoint_string = str; \
___tp_str; \
})
#define __tracepoint_string __used __section("__tracepoint_str")
#else
/*
* tracepoint_string() is used to save the string address for userspace
* tracing tools. When tracing isn't configured, there's no need to save
* anything.
*/
# define tracepoint_string(str) str
# define __tracepoint_string
#endif
#define DECLARE_TRACE(name, proto, args) \
__DECLARE_TRACE(name##_tp, PARAMS(proto), PARAMS(args), \
cpu_online(raw_smp_processor_id()), \
PARAMS(void *__data, proto))
#define DECLARE_TRACE_CONDITION(name, proto, args, cond) \
__DECLARE_TRACE(name##_tp, PARAMS(proto), PARAMS(args), \
cpu_online(raw_smp_processor_id()) && (PARAMS(cond)), \
PARAMS(void *__data, proto))
#define DECLARE_TRACE_SYSCALL(name, proto, args) \
__DECLARE_TRACE_SYSCALL(name##_tp, PARAMS(proto), PARAMS(args), \
PARAMS(void *__data, proto))
#define DECLARE_TRACE_EVENT(name, proto, args) \
__DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \
cpu_online(raw_smp_processor_id()), \
PARAMS(void *__data, proto))
#define DECLARE_TRACE_EVENT_CONDITION(name, proto, args, cond) \
__DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \
cpu_online(raw_smp_processor_id()) && (PARAMS(cond)), \
PARAMS(void *__data, proto))
#define DECLARE_TRACE_EVENT_SYSCALL(name, proto, args) \
__DECLARE_TRACE_SYSCALL(name, PARAMS(proto), PARAMS(args), \
PARAMS(void *__data, proto))
#define TRACE_EVENT_FLAGS(event, flag)
#define TRACE_EVENT_PERF_PERM(event, expr...)
#endif /* DECLARE_TRACE */
#ifndef TRACE_EVENT
/*
* For use with the TRACE_EVENT macro:
*
* We define a tracepoint, its arguments, its printk format
* and its 'fast binary record' layout.
*
* Firstly, name your tracepoint via TRACE_EVENT(name : the
* 'subsystem_event' notation is fine.
*
* Think about this whole construct as the
* 'trace_sched_switch() function' from now on.
*
*
* TRACE_EVENT(sched_switch,
*
* *
* * A function has a regular function arguments
* * prototype, declare it via TP_PROTO():
* *
*
* TP_PROTO(struct rq *rq, struct task_struct *prev,
* struct task_struct *next),
*
* *
* * Define the call signature of the 'function'.
* * (Design sidenote: we use this instead of a
* * TP_PROTO1/TP_PROTO2/TP_PROTO3 ugliness.)
* *
*
* TP_ARGS(rq, prev, next),
*
* *
* * Fast binary tracing: define the trace record via
* * TP_STRUCT__entry(). You can think about it like a
* * regular C structure local variable definition.
* *
* * This is how the trace record is structured and will
* * be saved into the ring buffer. These are the fields
* * that will be exposed to user-space in
* * /sys/kernel/tracing/events/<*>/format.
* *
* * The declared 'local variable' is called '__entry'
* *
* * __field(pid_t, prev_pid) is equivalent to a standard declaration:
* *
* * pid_t prev_pid;
* *
* * __array(char, prev_comm, TASK_COMM_LEN) is equivalent to:
* *
* * char prev_comm[TASK_COMM_LEN];
* *
*
* TP_STRUCT__entry(
* __array( char, prev_comm, TASK_COMM_LEN )
* __field( pid_t, prev_pid )
* __field( int, prev_prio )
* __array( char, next_comm, TASK_COMM_LEN )
* __field( pid_t, next_pid )
* __field( int, next_prio )
* ),
*
* *
* * Assign the entry into the trace record, by embedding
* * a full C statement block into TP_fast_assign(). You
* * can refer to the trace record as '__entry' -
* * otherwise you can put arbitrary C code in here.
* *
* * Note: this C code will execute every time a trace event
* * happens, on an active tracepoint.
* *
*
* TP_fast_assign(
* memcpy(__entry->next_comm, next->comm, TASK_COMM_LEN);
* __entry->prev_pid = prev->pid;
* __entry->prev_prio = prev->prio;
* memcpy(__entry->prev_comm, prev->comm, TASK_COMM_LEN);
* __entry->next_pid = next->pid;
* __entry->next_prio = next->prio;
* ),
*
* *
* * Formatted output of a trace record via TP_printk().
* * This is how the tracepoint will appear under ftrace
* * plugins that make use of this tracepoint.
* *
* * (raw-binary tracing wont actually perform this step.)
* *
*
* TP_printk("task %s:%d [%d] ==> %s:%d [%d]",
* __entry->prev_comm, __entry->prev_pid, __entry->prev_prio,
* __entry->next_comm, __entry->next_pid, __entry->next_prio),
*
* );
*
* This macro construct is thus used for the regular printk format
* tracing setup, it is used to construct a function pointer based
* tracepoint callback (this is used by programmatic plugins and
* can also by used by generic instrumentation like SystemTap), and
* it is also used to expose a structured trace record in
* /sys/kernel/tracing/events/.
*
* A set of (un)registration functions can be passed to the variant
* TRACE_EVENT_FN to perform any (un)registration work.
*/
#define DECLARE_EVENT_CLASS(name, proto, args, tstruct, assign, print)
#define DEFINE_EVENT(template, name, proto, args) \
DECLARE_TRACE_EVENT(name, PARAMS(proto), PARAMS(args))
#define DEFINE_EVENT_FN(template, name, proto, args, reg, unreg)\
DECLARE_TRACE_EVENT(name, PARAMS(proto), PARAMS(args))
#define DEFINE_EVENT_PRINT(template, name, proto, args, print) \
DECLARE_TRACE_EVENT(name, PARAMS(proto), PARAMS(args))
#define DEFINE_EVENT_CONDITION(template, name, proto, \
args, cond) \
DECLARE_TRACE_EVENT_CONDITION(name, PARAMS(proto), \
PARAMS(args), PARAMS(cond))
#define TRACE_EVENT(name, proto, args, struct, assign, print) \
DECLARE_TRACE_EVENT(name, PARAMS(proto), PARAMS(args))
#define TRACE_EVENT_FN(name, proto, args, struct, \
assign, print, reg, unreg) \
DECLARE_TRACE_EVENT(name, PARAMS(proto), PARAMS(args))
#define TRACE_EVENT_FN_COND(name, proto, args, cond, struct, \
assign, print, reg, unreg) \
DECLARE_TRACE_EVENT_CONDITION(name, PARAMS(proto), \
PARAMS(args), PARAMS(cond))
#define TRACE_EVENT_CONDITION(name, proto, args, cond, \
struct, assign, print) \
DECLARE_TRACE_EVENT_CONDITION(name, PARAMS(proto), \
PARAMS(args), PARAMS(cond))
#define TRACE_EVENT_SYSCALL(name, proto, args, struct, assign, \
print, reg, unreg) \
DECLARE_TRACE_EVENT_SYSCALL(name, PARAMS(proto), PARAMS(args))
#define TRACE_EVENT_FLAGS(event, flag)
#define TRACE_EVENT_PERF_PERM(event, expr...)
#define DECLARE_EVENT_NOP(name, proto, args) \
static inline void trace_##name(proto) \
{ } \
static inline bool trace_##name##_enabled(void) \
{ \
return false; \
}
#define TRACE_EVENT_NOP(name, proto, args, struct, assign, print) \
DECLARE_EVENT_NOP(name, PARAMS(proto), PARAMS(args))
#define DECLARE_EVENT_CLASS_NOP(name, proto, args, tstruct, assign, print)
#define DEFINE_EVENT_NOP(template, name, proto, args) \
DECLARE_EVENT_NOP(name, PARAMS(proto), PARAMS(args))
#endif /* ifdef TRACE_EVENT (see note above) */
]
|
| 27/2 |
2026/03/15 01:23 |
llm |
expert |
0m
Model:
Tokens:
input: 64867
output: 23
thoughts: 1127
Thoughts: **BPF Link Handling and Tracepoint Protection: A Potential Race Condition**
Okay, so I've been digging into how BPF links interact with tracepoints, and I think I've spotted a potential issue. Looking at the `__DECLARE_TRACE` macro, specifically lines 213-222, it's clear that tracepoints declared this way are protected by SRCU, specifically using `guard(srcu_fast_notrace)(&tracepoint_srcu)`. This is my first key observation: tracepoint execution under this macro is SRCU-protected.
Then I shifted my attention to `bpf_link_free`. Lines 3290-3293 show that the link is freed using either `call_rcu_tasks_trace` or `call_rcu`, depending on `link->sleepable`. Crucially, *neither* of these functions explicitly waits for the `tracepoint_srcu` grace period. This is the crux of the problem. If a BPF link is freed while a tracepoint using `__DECLARE_TRACE` is still active (i.e., within the SRCU grace period), we could have a race condition. The link could be deallocated before the SRCU grace period completes.
To confirm this, I looked at how `bpf_link_attach` initializes the BPF link, particularly lines 4269-4271. It passes `tracepoint_is_faultable(btp->tp)` as the `sleepable` argument to `bpf_link_init_sleepable`. If `tracepoint_is_faultable` is false, it means `link->sleepable` is false, and `bpf_link_free` will use `call_rcu`. This is where the discrepancy emerges. The tracepoint execution is protected by SRCU, but the BPF link deallocation might not be! A regular RCU grace period (initiated by `call_rcu`) does *not* wait for an SRCU grace period, which is needed by the tracepoint. This is a potential bug.
I quickly double-checked the `include/linux/tracepoint.h` header to make sure my understanding of `srcu_fast_notrace` was correct (line 219). Yes, it's used in `__DECLARE_TRACE`. But then, I looked at `DECLARE_TRACE_SYSCALL` (lines 233-240). Here, the protection mechanism is `guard(rcu_tasks_trace)()`. Fortunately, `bpf_link_free` *does* handle this scenario when `link->sleepable` is true (using `call_rcu_tasks_trace`). But with regular tracepoints (using `tracepoint_srcu`), the `call_rcu` or `call_rcu_tasks_trace` is not enough, as confirmed by my quick review of `tracepoint_is_faultable` (lines 113-116). If the `faultable` member of the `tracepoint` structure's `ext` field is false, SRCU is used. If it's true, `rcu_tasks_trace` is used, so the `bpf_link` free handles this case.
I think I was right in the first place - if a non-faultable tracepoint uses SRCU, but the BPF link is freed via a regular RCU grace period (because the `tracepoint` passed to the link has its `faultable` bit set to false), we've got a potential race. I need to dig deeper into the link deallocation logic to confirm this conclusively.
|
| 28/2 |
2026/03/15 01:23 |
tool |
grepper |
0m
Error:
Args: map[Expression:#define srcu_fast_notrace]
Results:
|
| 29/2 |
2026/03/15 01:23 |
llm |
expert |
0m
Model:
Tokens:
input: 66819
output: 23
thoughts: 221
Thoughts: **Analyzing Tracepoint Synchronization**
Okay, so I'm thinking, based on what I'm seeing, this is potentially related to a `DEFINE_GUARD`, isn't it? Let's double check the usage of `tracepoint_srcu` in `include/linux/tracepoint.h`. Ah, I have the contents right here. Let me parse through that quickly.
Right, so lines 107 to 113 are pertinent here. If I'm understanding this correctly, the inclusion is guarded by `CONFIG_TRACEPOINTS`, of course. I see that `tracepoint_srcu` is declared as an `extern struct srcu_struct`. We also have `tracepoint_synchronize_unregister()`, which calls both `synchronize_rcu_tasks_trace()` and `synchronize_srcu(&tracepoint_srcu)`. So, it waits for *both* synchronize calls.
Now, the important part... I'm pretty sure BPF doesn't go through the `tracepoint_synchronize_unregister()` call flow. That's a key distinction. My understanding is that BPF relies on either `call_rcu` or `call_rcu_tasks_trace` for synchronization, correct? So, what I need to do now is figure out where `tracepoint_srcu` is actually *defined* and *used* within the larger context. This is the next thing I will have to find, so I can see what is using this data structure.
|
| 30/2 |
2026/03/15 01:23 |
tool |
codesearch-find-references |
0m
Args: map[Name:tracepoint_srcu]
Results: map[References:<nil> TruncatedOutput:false]
|
| 31/2 |
2026/03/15 01:23 |
llm |
expert |
0m
Model:
Tokens:
input: 67406
output: 19
thoughts: 11
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:
**Initial Reaction & Immediate Solutioning**
Well, that's not what I anticipated. Huh. Let me take a moment to consider what's happening. I immediately need to get a handle on this, so my first instinct is to try `grep`. Itβs a workhorse, a fast way to tease out the pertinent details, especially when I'm dealing with something I donβt fully understand yet. It's often the quickest way to confirm my initial hypothesis, or discard it and move onto a broader scope of searching.
|
| 32/2 |
2026/03/15 01:23 |
tool |
grepper |
0m
Args: map[Expression:tracepoint_srcu]
Results: map[Output:include/linux/tracepoint.h=95=void for_each_tracepoint_in_module(struct module *mod,
--
include/linux/tracepoint.h-114-#ifdef CONFIG_TRACEPOINTS
include/linux/tracepoint.h:115:extern struct srcu_struct tracepoint_srcu;
include/linux/tracepoint.h-116-static inline void tracepoint_synchronize_unregister(void)
--
include/linux/tracepoint.h-118- synchronize_rcu_tasks_trace();
include/linux/tracepoint.h:119: synchronize_srcu(&tracepoint_srcu);
include/linux/tracepoint.h-120-}
--
include/linux/tracepoint.h=158=static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)
--
include/linux/tracepoint.h-284- if (cond) { \
include/linux/tracepoint.h:285: guard(srcu_fast_notrace)(&tracepoint_srcu); \
include/linux/tracepoint.h-286- __DO_TRACE_CALL(name, TP_ARGS(args)); \
--
kernel/trace/trace_events_filter.c=1171=int filter_match_preds(struct event_filter *filter, void *rec)
--
kernel/trace/trace_events_filter.c-1179-
kernel/trace/trace_events_filter.c:1180: /* Protected by either SRCU(tracepoint_srcu) or preempt_disable */
kernel/trace/trace_events_filter.c-1181- prog = rcu_dereference_raw(filter->prog);
--
kernel/tracepoint.c=35=struct tp_transition_snapshot {
--
kernel/tracepoint.c-40-
kernel/tracepoint.c:41:DEFINE_SRCU_FAST(tracepoint_srcu);
kernel/tracepoint.c:42:EXPORT_SYMBOL_GPL(tracepoint_srcu);
kernel/tracepoint.c-43-
--
kernel/tracepoint.c=47=static void tp_rcu_get_state(enum tp_transition_sync sync)
--
kernel/tracepoint.c-52- snapshot->rcu = get_state_synchronize_rcu();
kernel/tracepoint.c:53: snapshot->srcu_gp = start_poll_synchronize_srcu(&tracepoint_srcu);
kernel/tracepoint.c-54- snapshot->ongoing = true;
--
kernel/tracepoint.c=57=static void tp_rcu_cond_sync(enum tp_transition_sync sync)
--
kernel/tracepoint.c-63- cond_synchronize_rcu(snapshot->rcu);
kernel/tracepoint.c:64: if (!poll_state_synchronize_srcu(&tracepoint_srcu, snapshot->srcu_gp))
kernel/tracepoint.c:65: synchronize_srcu(&tracepoint_srcu);
kernel/tracepoint.c-66- snapshot->ongoing = false;
--
kernel/tracepoint.c=115=static inline void release_probes(struct tracepoint *tp, struct tracepoint_func *old)
--
kernel/tracepoint.c-124- } else {
kernel/tracepoint.c:125: call_srcu(&tracepoint_srcu, &tp_probes->rcu,
kernel/tracepoint.c-126- rcu_free_old_probes);
]
|
| 33/2 |
2026/03/15 01:23 |
llm |
expert |
0m
Model:
Tokens:
input: 68469
output: 21
thoughts: 159
Thoughts: **Potential Race Condition in Tracepoint Unregistration**
Okay, here's what's running through my mind. I'm looking at `kernel/tracepoint.c`, specifically how `release_probes` handles the cleanup of tracepoint functions. I see that it uses `call_srcu` with `tracepoint_srcu` to synchronize the freeing of the `tracepoint_func` array. That makes sense, standard approach for something like this.
But here's the rub: the `data` being passed to the probe function β in our case, the `bpf_raw_tp_link` β is *not* controlled by the tracepoint system itself. BPF is in charge of that. And if BPF unlinks and frees that `bpf_raw_tp_link` using something like `call_rcu`, which is common for BPF object cleanup, a race is possible.
Essentially, we have a scenario where the tracepoint subsystem might still be *executing* a probe function that's now operating on already-freed memory! The call to `call_srcu` is protecting the *function array* itself, but not the external data being pointed to by that array. That's a classic concurrency issue, and we need to investigate.
I need to dig into `kernel/tracepoint.c` now, really understand the call flow of the probes and, crucially, how the unregistration process plays out under the hood. Specifically, I'll focus on how the `bpf_raw_tp_link` is unlinked and cleaned up in relation to the `tracepoint_srcu` protection.
|
| 34/2 |
2026/03/15 01:23 |
tool |
read-file |
0m
Args: map[File:kernel/tracepoint.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-or-later
/*
* Copyright (C) 2008-2014 Mathieu Desnoyers
*/
#include <linux/module.h>
#include <linux/mutex.h>
#include <linux/types.h>
#include <linux/jhash.h>
#include <linux/list.h>
#include <linux/rcupdate.h>
#include <linux/tracepoint.h>
#include <linux/err.h>
#include <linux/slab.h>
#include <linux/sched/signal.h>
#include <linux/sched/task.h>
#include <linux/static_key.h>
enum tp_func_state {
TP_FUNC_0,
TP_FUNC_1,
TP_FUNC_2,
TP_FUNC_N,
};
extern tracepoint_ptr_t __start___tracepoints_ptrs[];
extern tracepoint_ptr_t __stop___tracepoints_ptrs[];
enum tp_transition_sync {
TP_TRANSITION_SYNC_1_0_1,
TP_TRANSITION_SYNC_N_2_1,
_NR_TP_TRANSITION_SYNC,
};
struct tp_transition_snapshot {
unsigned long rcu;
unsigned long srcu_gp;
bool ongoing;
};
DEFINE_SRCU_FAST(tracepoint_srcu);
EXPORT_SYMBOL_GPL(tracepoint_srcu);
/* Protected by tracepoints_mutex */
static struct tp_transition_snapshot tp_transition_snapshot[_NR_TP_TRANSITION_SYNC];
static void tp_rcu_get_state(enum tp_transition_sync sync)
{
struct tp_transition_snapshot *snapshot = &tp_transition_snapshot[sync];
/* Keep the latest get_state snapshot. */
snapshot->rcu = get_state_synchronize_rcu();
snapshot->srcu_gp = start_poll_synchronize_srcu(&tracepoint_srcu);
snapshot->ongoing = true;
}
static void tp_rcu_cond_sync(enum tp_transition_sync sync)
{
struct tp_transition_snapshot *snapshot = &tp_transition_snapshot[sync];
if (!snapshot->ongoing)
return;
cond_synchronize_rcu(snapshot->rcu);
if (!poll_state_synchronize_srcu(&tracepoint_srcu, snapshot->srcu_gp))
synchronize_srcu(&tracepoint_srcu);
snapshot->ongoing = false;
}
/* Set to 1 to enable tracepoint debug output */
static const int tracepoint_debug;
#ifdef CONFIG_MODULES
/*
* Tracepoint module list mutex protects the local module list.
*/
static DEFINE_MUTEX(tracepoint_module_list_mutex);
/* Local list of struct tp_module */
static LIST_HEAD(tracepoint_module_list);
#endif /* CONFIG_MODULES */
/*
* tracepoints_mutex protects the builtin and module tracepoints.
* tracepoints_mutex nests inside tracepoint_module_list_mutex.
*/
static DEFINE_MUTEX(tracepoints_mutex);
/*
* Note about RCU :
* It is used to delay the free of multiple probes array until a quiescent
* state is reached.
*/
struct tp_probes {
struct rcu_head rcu;
struct tracepoint_func probes[];
};
/* Called in removal of a func but failed to allocate a new tp_funcs */
static void tp_stub_func(void)
{
return;
}
static inline void *allocate_probes(int count)
{
struct tp_probes *p = kmalloc_flex(*p, probes, count);
return p == NULL ? NULL : p->probes;
}
static void rcu_free_old_probes(struct rcu_head *head)
{
kfree(container_of(head, struct tp_probes, rcu));
}
static inline void release_probes(struct tracepoint *tp, struct tracepoint_func *old)
{
if (old) {
struct tp_probes *tp_probes = container_of(old,
struct tp_probes, probes[0]);
if (tracepoint_is_faultable(tp)) {
call_rcu_tasks_trace(&tp_probes->rcu,
rcu_free_old_probes);
} else {
call_srcu(&tracepoint_srcu, &tp_probes->rcu,
rcu_free_old_probes);
}
}
}
static void debug_print_probes(struct tracepoint_func *funcs)
{
int i;
if (!tracepoint_debug || !funcs)
return;
for (i = 0; funcs[i].func; i++)
printk(KERN_DEBUG "Probe %d : %pSb\n", i, funcs[i].func);
}
static struct tracepoint_func *
func_add(struct tracepoint_func **funcs, struct tracepoint_func *tp_func,
int prio)
{
struct tracepoint_func *old, *new;
int iter_probes; /* Iterate over old probe array. */
int nr_probes = 0; /* Counter for probes */
int pos = -1; /* Insertion position into new array */
if (WARN_ON(!tp_func->func))
return ERR_PTR(-EINVAL);
debug_print_probes(*funcs);
old = *funcs;
if (old) {
/* (N -> N+1), (N != 0, 1) probes */
for (iter_probes = 0; old[iter_probes].func; iter_probes++) {
if (old[iter_probes].func == tp_stub_func)
continue; /* Skip stub functions. */
if (old[iter_probes].func == tp_func->func &&
old[iter_probes].data == tp_func->data)
return ERR_PTR(-EEXIST);
nr_probes++;
}
}
/* + 2 : one for new probe, one for NULL func */
new = allocate_probes(nr_probes + 2);
if (new == NULL)
return ERR_PTR(-ENOMEM);
if (old) {
nr_probes = 0;
for (iter_probes = 0; old[iter_probes].func; iter_probes++) {
if (old[iter_probes].func == tp_stub_func)
continue;
/* Insert before probes of lower priority */
if (pos < 0 && old[iter_probes].prio < prio)
pos = nr_probes++;
new[nr_probes++] = old[iter_probes];
}
if (pos < 0)
pos = nr_probes++;
/* nr_probes now points to the end of the new array */
} else {
pos = 0;
nr_probes = 1; /* must point at end of array */
}
new[pos] = *tp_func;
new[nr_probes].func = NULL;
*funcs = new;
debug_print_probes(*funcs);
return old;
}
static void *func_remove(struct tracepoint_func **funcs,
struct tracepoint_func *tp_func)
{
int nr_probes = 0, nr_del = 0, i;
struct tracepoint_func *old, *new;
old = *funcs;
if (!old)
return ERR_PTR(-ENOENT);
debug_print_probes(*funcs);
/* (N -> M), (N > 1, M >= 0) probes */
if (tp_func->func) {
for (nr_probes = 0; old[nr_probes].func; nr_probes++) {
if ((old[nr_probes].func == tp_func->func &&
old[nr_probes].data == tp_func->data) ||
old[nr_probes].func == tp_stub_func)
nr_del++;
}
}
/*
* If probe is NULL, then nr_probes = nr_del = 0, and then the
* entire entry will be removed.
*/
if (nr_probes - nr_del == 0) {
/* N -> 0, (N > 1) */
*funcs = NULL;
debug_print_probes(*funcs);
return old;
} else {
int j = 0;
/* N -> M, (N > 1, M > 0) */
/* + 1 for NULL */
new = allocate_probes(nr_probes - nr_del + 1);
if (new) {
for (i = 0; old[i].func; i++) {
if ((old[i].func != tp_func->func ||
old[i].data != tp_func->data) &&
old[i].func != tp_stub_func)
new[j++] = old[i];
}
new[nr_probes - nr_del].func = NULL;
*funcs = new;
} else {
/*
* Failed to allocate, replace the old function
* with calls to tp_stub_func.
*/
for (i = 0; old[i].func; i++) {
if (old[i].func == tp_func->func &&
old[i].data == tp_func->data)
WRITE_ONCE(old[i].func, tp_stub_func);
}
*funcs = old;
}
}
debug_print_probes(*funcs);
return old;
}
/*
* Count the number of functions (enum tp_func_state) in a tp_funcs array.
*/
static enum tp_func_state nr_func_state(const struct tracepoint_func *tp_funcs)
{
if (!tp_funcs)
return TP_FUNC_0;
if (!tp_funcs[1].func)
return TP_FUNC_1;
if (!tp_funcs[2].func)
return TP_FUNC_2;
return TP_FUNC_N; /* 3 or more */
}
static void tracepoint_update_call(struct tracepoint *tp, struct tracepoint_func *tp_funcs)
{
void *func = tp->iterator;
/* Synthetic events do not have static call sites */
if (!tp->static_call_key)
return;
if (nr_func_state(tp_funcs) == TP_FUNC_1)
func = tp_funcs[0].func;
__static_call_update(tp->static_call_key, tp->static_call_tramp, func);
}
/*
* Add the probe function to a tracepoint.
*/
static int tracepoint_add_func(struct tracepoint *tp,
struct tracepoint_func *func, int prio,
bool warn)
{
struct tracepoint_func *old, *tp_funcs;
int ret;
if (tp->ext && tp->ext->regfunc && !static_key_enabled(&tp->key)) {
ret = tp->ext->regfunc();
if (ret < 0)
return ret;
}
tp_funcs = rcu_dereference_protected(tp->funcs,
lockdep_is_held(&tracepoints_mutex));
old = func_add(&tp_funcs, func, prio);
if (IS_ERR(old)) {
WARN_ON_ONCE(warn && PTR_ERR(old) != -ENOMEM);
return PTR_ERR(old);
}
/*
* rcu_assign_pointer has as smp_store_release() which makes sure
* that the new probe callbacks array is consistent before setting
* a pointer to it. This array is referenced by __DO_TRACE from
* include/linux/tracepoint.h using rcu_dereference_sched().
*/
switch (nr_func_state(tp_funcs)) {
case TP_FUNC_1: /* 0->1 */
/*
* Make sure new static func never uses old data after a
* 1->0->1 transition sequence.
*/
tp_rcu_cond_sync(TP_TRANSITION_SYNC_1_0_1);
/* Set static call to first function */
tracepoint_update_call(tp, tp_funcs);
/* Both iterator and static call handle NULL tp->funcs */
rcu_assign_pointer(tp->funcs, tp_funcs);
static_branch_enable(&tp->key);
break;
case TP_FUNC_2: /* 1->2 */
/* Set iterator static call */
tracepoint_update_call(tp, tp_funcs);
/*
* Iterator callback installed before updating tp->funcs.
* Requires ordering between RCU assign/dereference and
* static call update/call.
*/
fallthrough;
case TP_FUNC_N: /* N->N+1 (N>1) */
rcu_assign_pointer(tp->funcs, tp_funcs);
/*
* Make sure static func never uses incorrect data after a
* N->...->2->1 (N>1) transition sequence.
*/
if (tp_funcs[0].data != old[0].data)
tp_rcu_get_state(TP_TRANSITION_SYNC_N_2_1);
break;
default:
WARN_ON_ONCE(1);
break;
}
release_probes(tp, old);
return 0;
}
/*
* Remove a probe function from a tracepoint.
* Note: only waiting an RCU period after setting elem->call to the empty
* function insures that the original callback is not used anymore. This insured
* by preempt_disable around the call site.
*/
static int tracepoint_remove_func(struct tracepoint *tp,
struct tracepoint_func *func)
{
struct tracepoint_func *old, *tp_funcs;
tp_funcs = rcu_dereference_protected(tp->funcs,
lockdep_is_held(&tracepoints_mutex));
old = func_remove(&tp_funcs, func);
if (WARN_ON_ONCE(IS_ERR(old)))
return PTR_ERR(old);
if (tp_funcs == old)
/* Failed allocating new tp_funcs, replaced func with stub */
return 0;
switch (nr_func_state(tp_funcs)) {
case TP_FUNC_0: /* 1->0 */
/* Removed last function */
if (tp->ext && tp->ext->unregfunc && static_key_enabled(&tp->key))
tp->ext->unregfunc();
static_branch_disable(&tp->key);
/* Set iterator static call */
tracepoint_update_call(tp, tp_funcs);
/* Both iterator and static call handle NULL tp->funcs */
rcu_assign_pointer(tp->funcs, NULL);
/*
* Make sure new static func never uses old data after a
* 1->0->1 transition sequence.
*/
tp_rcu_get_state(TP_TRANSITION_SYNC_1_0_1);
break;
case TP_FUNC_1: /* 2->1 */
rcu_assign_pointer(tp->funcs, tp_funcs);
/*
* Make sure static func never uses incorrect data after a
* N->...->2->1 (N>2) transition sequence. If the first
* element's data has changed, then force the synchronization
* to prevent current readers that have loaded the old data
* from calling the new function.
*/
if (tp_funcs[0].data != old[0].data)
tp_rcu_get_state(TP_TRANSITION_SYNC_N_2_1);
tp_rcu_cond_sync(TP_TRANSITION_SYNC_N_2_1);
/* Set static call to first function */
tracepoint_update_call(tp, tp_funcs);
break;
case TP_FUNC_2: /* N->N-1 (N>2) */
fallthrough;
case TP_FUNC_N:
rcu_assign_pointer(tp->funcs, tp_funcs);
/*
* Make sure static func never uses incorrect data after a
* N->...->2->1 (N>2) transition sequence.
*/
if (tp_funcs[0].data != old[0].data)
tp_rcu_get_state(TP_TRANSITION_SYNC_N_2_1);
break;
default:
WARN_ON_ONCE(1);
break;
}
release_probes(tp, old);
return 0;
}
/**
* tracepoint_probe_register_prio_may_exist - Connect a probe to a tracepoint with priority
* @tp: tracepoint
* @probe: probe handler
* @data: tracepoint data
* @prio: priority of this function over other registered functions
*
* Same as tracepoint_probe_register_prio() except that it will not warn
* if the tracepoint is already registered.
*/
int tracepoint_probe_register_prio_may_exist(struct tracepoint *tp, void *probe,
void *data, int prio)
{
struct tracepoint_func tp_func;
int ret;
mutex_lock(&tracepoints_mutex);
tp_func.func = probe;
tp_func.data = data;
tp_func.prio = prio;
ret = tracepoint_add_func(tp, &tp_func, prio, false);
mutex_unlock(&tracepoints_mutex);
return ret;
}
EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio_may_exist);
/**
* tracepoint_probe_register_prio - Connect a probe to a tracepoint with priority
* @tp: tracepoint
* @probe: probe handler
* @data: tracepoint data
* @prio: priority of this function over other registered functions
*
* Returns 0 if ok, error value on error.
* Note: if @tp is within a module, the caller is responsible for
* unregistering the probe before the module is gone. This can be
* performed either with a tracepoint module going notifier, or from
* within module exit functions.
*/
int tracepoint_probe_register_prio(struct tracepoint *tp, void *probe,
void *data, int prio)
{
struct tracepoint_func tp_func;
int ret;
mutex_lock(&tracepoints_mutex);
tp_func.func = probe;
tp_func.data = data;
tp_func.prio = prio;
ret = tracepoint_add_func(tp, &tp_func, prio, true);
mutex_unlock(&tracepoints_mutex);
return ret;
}
EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio);
/**
* tracepoint_probe_register - Connect a probe to a tracepoint
* @tp: tracepoint
* @probe: probe handler
* @data: tracepoint data
*
* Returns 0 if ok, error value on error.
* Note: if @tp is within a module, the caller is responsible for
* unregistering the probe before the module is gone. This can be
* performed either with a tracepoint module going notifier, or from
* within module exit functions.
*/
int tracepoint_probe_register(struct tracepoint *tp, void *probe, void *data)
{
return tracepoint_probe_register_prio(tp, probe, data, TRACEPOINT_DEFAULT_PRIO);
}
EXPORT_SYMBOL_GPL(tracepoint_probe_register);
/**
* tracepoint_probe_unregister - Disconnect a probe from a tracepoint
* @tp: tracepoint
* @probe: probe function pointer
* @data: tracepoint data
*
* Returns 0 if ok, error value on error.
*/
int tracepoint_probe_unregister(struct tracepoint *tp, void *probe, void *data)
{
struct tracepoint_func tp_func;
int ret;
mutex_lock(&tracepoints_mutex);
tp_func.func = probe;
tp_func.data = data;
ret = tracepoint_remove_func(tp, &tp_func);
mutex_unlock(&tracepoints_mutex);
return ret;
}
EXPORT_SYMBOL_GPL(tracepoint_probe_unregister);
static void for_each_tracepoint_range(
tracepoint_ptr_t *begin, tracepoint_ptr_t *end,
void (*fct)(struct tracepoint *tp, void *priv),
void *priv)
{
tracepoint_ptr_t *iter;
if (!begin)
return;
for (iter = begin; iter < end; iter++)
fct(tracepoint_ptr_deref(iter), priv);
}
#ifdef CONFIG_MODULES
bool trace_module_has_bad_taint(struct module *mod)
{
return mod->taints & ~((1 << TAINT_OOT_MODULE) | (1 << TAINT_CRAP) |
(1 << TAINT_UNSIGNED_MODULE) | (1 << TAINT_TEST) |
(1 << TAINT_LIVEPATCH));
}
static BLOCKING_NOTIFIER_HEAD(tracepoint_notify_list);
/**
* register_tracepoint_module_notifier - register tracepoint coming/going notifier
* @nb: notifier block
*
* Notifiers registered with this function are called on module
* coming/going with the tracepoint_module_list_mutex held.
* The notifier block callback should expect a "struct tp_module" data
* pointer.
*/
int register_tracepoint_module_notifier(struct notifier_block *nb)
{
struct tp_module *tp_mod;
int ret;
mutex_lock(&tracepoint_module_list_mutex);
ret = blocking_notifier_chain_register(&tracepoint_notify_list, nb);
if (ret)
goto end;
list_for_each_entry(tp_mod, &tracepoint_module_list, list)
(void) nb->notifier_call(nb, MODULE_STATE_COMING, tp_mod);
end:
mutex_unlock(&tracepoint_module_list_mutex);
return ret;
}
EXPORT_SYMBOL_GPL(register_tracepoint_module_notifier);
/**
* unregister_tracepoint_module_notifier - unregister tracepoint coming/going notifier
* @nb: notifier block
*
* The notifier block callback should expect a "struct tp_module" data
* pointer.
*/
int unregister_tracepoint_module_notifier(struct notifier_block *nb)
{
struct tp_module *tp_mod;
int ret;
mutex_lock(&tracepoint_module_list_mutex);
ret = blocking_notifier_chain_unregister(&tracepoint_notify_list, nb);
if (ret)
goto end;
list_for_each_entry(tp_mod, &tracepoint_module_list, list)
(void) nb->notifier_call(nb, MODULE_STATE_GOING, tp_mod);
end:
mutex_unlock(&tracepoint_module_list_mutex);
return ret;
}
EXPORT_SYMBOL_GPL(unregister_tracepoint_module_notifier);
/*
* Ensure the tracer unregistered the module's probes before the module
* teardown is performed. Prevents leaks of probe and data pointers.
*/
static void tp_module_going_check_quiescent(struct tracepoint *tp, void *priv)
{
WARN_ON_ONCE(tp->funcs);
}
static int tracepoint_module_coming(struct module *mod)
{
struct tp_module *tp_mod;
if (!mod->num_tracepoints)
return 0;
/*
* We skip modules that taint the kernel, especially those with different
* module headers (for forced load), to make sure we don't cause a crash.
* Staging, out-of-tree, unsigned GPL, and test modules are fine.
*/
if (trace_module_has_bad_taint(mod))
return 0;
tp_mod = kmalloc_obj(struct tp_module);
if (!tp_mod)
return -ENOMEM;
tp_mod->mod = mod;
mutex_lock(&tracepoint_module_list_mutex);
list_add_tail(&tp_mod->list, &tracepoint_module_list);
blocking_notifier_call_chain(&tracepoint_notify_list,
MODULE_STATE_COMING, tp_mod);
mutex_unlock(&tracepoint_module_list_mutex);
return 0;
}
static void tracepoint_module_going(struct module *mod)
{
struct tp_module *tp_mod;
if (!mod->num_tracepoints)
return;
mutex_lock(&tracepoint_module_list_mutex);
list_for_each_entry(tp_mod, &tracepoint_module_list, list) {
if (tp_mod->mod == mod) {
blocking_notifier_call_chain(&tracepoint_notify_list,
MODULE_STATE_GOING, tp_mod);
list_del(&tp_mod->list);
kfree(tp_mod);
/*
* Called the going notifier before checking for
* quiescence.
*/
for_each_tracepoint_range(mod->tracepoints_ptrs,
mod->tracepoints_ptrs + mod->num_tracepoints,
tp_module_going_check_quiescent, NULL);
break;
}
}
/*
* In the case of modules that were tainted at "coming", we'll simply
* walk through the list without finding it. We cannot use the "tainted"
* flag on "going", in case a module taints the kernel only after being
* loaded.
*/
mutex_unlock(&tracepoint_module_list_mutex);
}
static int tracepoint_module_notify(struct notifier_block *self,
unsigned long val, void *data)
{
struct module *mod = data;
int ret = 0;
switch (val) {
case MODULE_STATE_COMING:
ret = tracepoint_module_coming(mod);
break;
case MODULE_STATE_LIVE:
break;
case MODULE_STATE_GOING:
tracepoint_module_going(mod);
break;
case MODULE_STATE_UNFORMED:
break;
}
return notifier_from_errno(ret);
}
static struct notifier_block tracepoint_module_nb = {
.notifier_call = tracepoint_module_notify,
.priority = 0,
};
static __init int init_tracepoints(void)
{
int ret;
ret = register_module_notifier(&tracepoint_module_nb);
if (ret)
pr_warn("Failed to register tracepoint module enter notifier\n");
return ret;
}
__initcall(init_tracepoints);
/**
* for_each_tracepoint_in_module - iteration on all tracepoints in a module
* @mod: module
* @fct: callback
* @priv: private data
*/
void for_each_tracepoint_in_module(struct module *mod,
void (*fct)(struct tracepoint *tp,
struct module *mod, void *priv),
void *priv)
{
tracepoint_ptr_t *begin, *end, *iter;
lockdep_assert_held(&tracepoint_module_list_mutex);
if (!mod)
return;
begin = mod->tracepoints_ptrs;
end = mod->tracepoints_ptrs + mod->num_tracepoints;
for (iter = begin; iter < end; iter++)
fct(tracepoint_ptr_deref(iter), mod, priv);
}
/**
* for_each_module_tracepoint - iteration on all tracepoints in all modules
* @fct: callback
* @priv: private data
*/
void for_each_module_tracepoint(void (*fct)(struct tracepoint *tp,
struct module *mod, void *priv),
void *priv)
{
struct tp_module *tp_mod;
mutex_lock(&tracepoint_module_list_mutex);
list_for_each_entry(tp_mod, &tracepoint_module_list, list)
for_each_tracepoint_in_module(tp_mod->mod, fct, priv);
mutex_unlock(&tracepoint_module_list_mutex);
}
#endif /* CONFIG_MODULES */
/**
* for_each_kernel_tracepoint - iteration on all kernel tracepoints
* @fct: callback
* @priv: private data
*/
void for_each_kernel_tracepoint(void (*fct)(struct tracepoint *tp, void *priv),
void *priv)
{
for_each_tracepoint_range(__start___tracepoints_ptrs,
__stop___tracepoints_ptrs, fct, priv);
}
EXPORT_SYMBOL_GPL(for_each_kernel_tracepoint);
#ifdef CONFIG_HAVE_SYSCALL_TRACEPOINTS
/* NB: reg/unreg are called while guarded with the tracepoints_mutex */
static int sys_tracepoint_refcount;
int syscall_regfunc(void)
{
struct task_struct *p, *t;
if (!sys_tracepoint_refcount) {
read_lock(&tasklist_lock);
for_each_process_thread(p, t) {
set_task_syscall_work(t, SYSCALL_TRACEPOINT);
}
read_unlock(&tasklist_lock);
}
sys_tracepoint_refcount++;
return 0;
}
void syscall_unregfunc(void)
{
struct task_struct *p, *t;
sys_tracepoint_refcount--;
if (!sys_tracepoint_refcount) {
read_lock(&tasklist_lock);
for_each_process_thread(p, t) {
clear_task_syscall_work(t, SYSCALL_TRACEPOINT);
}
read_unlock(&tasklist_lock);
}
}
#endif
]
|
| 35/2 |
2026/03/15 01:23 |
llm |
expert |
0m
Model:
Tokens:
input: 76482
output: 25
thoughts: 858
Thoughts: **Root Cause Analysis: Tracepoint SRCU vs. BPF RCU Conflict**
Okay, I'm diving into this `release_probes` function in `kernel/tracepoint.c`. My initial observation is that this function only cleans up the `tp_probes` structure β the array holding the tracepoint functions and associated data pointers. Crucially, it **doesn't** synchronize or wait for a grace period before returning. This is where the potential problem begins to surface.
The control flow is this: `bpf_raw_tp_link_release` calls `bpf_probe_unregister`, which eventually calls `tracepoint_probe_unregister`. This sequence removes the probe from the array and schedules the old array for deletion. The key thing is that the `data` itself β and by extension our `link` β is still in use by any concurrent tracepoint calls.
Now, BPF schedules the `link`'s deletion using `call_rcu`. The risk arises if a tracepoint caller is *still* running and actively *using* the `link` data. And if `call_rcu` finishes before the tracepoint caller does, we get a classic Use-After-Free (UAF) situation. That looks dangerous to me.
I quickly remember that `mm_page_alloc` is a regular tracepoint. Now, let's think about how it's handled. Looking in `include/linux/tracepoint.h`, I see `guard(srcu_fast_notrace)(&tracepoint_srcu);`. Bingo! The tracepoint *does* use `tracepoint_srcu`.
The issue becomes much clearer when I see that BPF's `bpf_link_free` (for non-sleepable links) also calls `call_rcu`. The regular RCU grace period *doesn't* account for, or wait for, the SRCU grace period. This confirms my initial concern: `link` could very well be freed while a tracepoint, using SRCU, is still running and potentially accessing it.
It's seeming more and more likely that this is the root cause.
Now, the next thing I should determine is if `bpf_raw_tp_link` is *always* non-sleepable. I check `bpf_raw_tp_link_attach`. Here, I find that `bpf_link_init_sleepable` uses `tracepoint_is_faultable(btp->tp)`. For `mm_page_alloc`, this is false. That means `link->sleepable` is also false.
If the `link->sleepable` is indeed false, BPF does call `call_rcu`, which further amplifies the vulnerability. Remember, `call_rcu` is not synchronized with the SRCU use of the tracepoint!
One more thing I should check. Is it possible that the `mm_page_alloc` tracepoint is *not* using SRCU, even though I saw the `guard(srcu_fast_notrace)`? Let me double-check the definition. I'll search `include/trace/events/kmem.h`. This feels like it might confirm it.
|
| 36/2 |
2026/03/15 01:23 |
tool |
read-file |
0m
Args: map[File:include/trace/events/kmem.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0 */
#undef TRACE_SYSTEM
#define TRACE_SYSTEM kmem
#if !defined(_TRACE_KMEM_H) || defined(TRACE_HEADER_MULTI_READ)
#define _TRACE_KMEM_H
#include <linux/types.h>
#include <linux/tracepoint.h>
#include <trace/events/mmflags.h>
TRACE_EVENT(kmem_cache_alloc,
TP_PROTO(unsigned long call_site,
const void *ptr,
struct kmem_cache *s,
gfp_t gfp_flags,
int node),
TP_ARGS(call_site, ptr, s, gfp_flags, node),
TP_STRUCT__entry(
__field( unsigned long, call_site )
__field( const void *, ptr )
__string( name, s->name )
__field( size_t, bytes_req )
__field( size_t, bytes_alloc )
__field( unsigned long, gfp_flags )
__field( int, node )
__field( bool, accounted )
),
TP_fast_assign(
__entry->call_site = call_site;
__entry->ptr = ptr;
__assign_str(name);
__entry->bytes_req = s->object_size;
__entry->bytes_alloc = s->size;
__entry->gfp_flags = (__force unsigned long)gfp_flags;
__entry->node = node;
__entry->accounted = IS_ENABLED(CONFIG_MEMCG) ?
((gfp_flags & __GFP_ACCOUNT) ||
(s->flags & SLAB_ACCOUNT)) : false;
),
TP_printk("call_site=%pS ptr=%p name=%s bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d accounted=%s",
(void *)__entry->call_site,
__entry->ptr,
__get_str(name),
__entry->bytes_req,
__entry->bytes_alloc,
show_gfp_flags(__entry->gfp_flags),
__entry->node,
__entry->accounted ? "true" : "false")
);
TRACE_EVENT(kmalloc,
TP_PROTO(unsigned long call_site,
const void *ptr,
size_t bytes_req,
size_t bytes_alloc,
gfp_t gfp_flags,
int node),
TP_ARGS(call_site, ptr, bytes_req, bytes_alloc, gfp_flags, node),
TP_STRUCT__entry(
__field( unsigned long, call_site )
__field( const void *, ptr )
__field( size_t, bytes_req )
__field( size_t, bytes_alloc )
__field( unsigned long, gfp_flags )
__field( int, node )
),
TP_fast_assign(
__entry->call_site = call_site;
__entry->ptr = ptr;
__entry->bytes_req = bytes_req;
__entry->bytes_alloc = bytes_alloc;
__entry->gfp_flags = (__force unsigned long)gfp_flags;
__entry->node = node;
),
TP_printk("call_site=%pS ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d accounted=%s",
(void *)__entry->call_site,
__entry->ptr,
__entry->bytes_req,
__entry->bytes_alloc,
show_gfp_flags(__entry->gfp_flags),
__entry->node,
(IS_ENABLED(CONFIG_MEMCG) &&
(__entry->gfp_flags & (__force unsigned long)__GFP_ACCOUNT)) ? "true" : "false")
);
TRACE_EVENT(kfree,
TP_PROTO(unsigned long call_site, const void *ptr),
TP_ARGS(call_site, ptr),
TP_STRUCT__entry(
__field( unsigned long, call_site )
__field( const void *, ptr )
),
TP_fast_assign(
__entry->call_site = call_site;
__entry->ptr = ptr;
),
TP_printk("call_site=%pS ptr=%p",
(void *)__entry->call_site, __entry->ptr)
);
TRACE_EVENT(kmem_cache_free,
TP_PROTO(unsigned long call_site, const void *ptr, const struct kmem_cache *s),
TP_ARGS(call_site, ptr, s),
TP_STRUCT__entry(
__field( unsigned long, call_site )
__field( const void *, ptr )
__string( name, s->name )
),
TP_fast_assign(
__entry->call_site = call_site;
__entry->ptr = ptr;
__assign_str(name);
),
TP_printk("call_site=%pS ptr=%p name=%s",
(void *)__entry->call_site, __entry->ptr, __get_str(name))
);
TRACE_EVENT(mm_page_free,
TP_PROTO(struct page *page, unsigned int order),
TP_ARGS(page, order),
TP_STRUCT__entry(
__field( unsigned long, pfn )
__field( unsigned int, order )
),
TP_fast_assign(
__entry->pfn = page_to_pfn(page);
__entry->order = order;
),
TP_printk("page=%p pfn=0x%lx order=%d",
pfn_to_page(__entry->pfn),
__entry->pfn,
__entry->order)
);
TRACE_EVENT(mm_page_free_batched,
TP_PROTO(struct page *page),
TP_ARGS(page),
TP_STRUCT__entry(
__field( unsigned long, pfn )
),
TP_fast_assign(
__entry->pfn = page_to_pfn(page);
),
TP_printk("page=%p pfn=0x%lx order=0",
pfn_to_page(__entry->pfn),
__entry->pfn)
);
TRACE_EVENT(mm_page_alloc,
TP_PROTO(struct page *page, unsigned int order,
gfp_t gfp_flags, int migratetype),
TP_ARGS(page, order, gfp_flags, migratetype),
TP_STRUCT__entry(
__field( unsigned long, pfn )
__field( unsigned int, order )
__field( unsigned long, gfp_flags )
__field( int, migratetype )
),
TP_fast_assign(
__entry->pfn = page ? page_to_pfn(page) : -1UL;
__entry->order = order;
__entry->gfp_flags = (__force unsigned long)gfp_flags;
__entry->migratetype = migratetype;
),
TP_printk("page=%p pfn=0x%lx order=%d migratetype=%d gfp_flags=%s",
__entry->pfn != -1UL ? pfn_to_page(__entry->pfn) : NULL,
__entry->pfn != -1UL ? __entry->pfn : 0,
__entry->order,
__entry->migratetype,
show_gfp_flags(__entry->gfp_flags))
);
DECLARE_EVENT_CLASS(mm_page,
TP_PROTO(struct page *page, unsigned int order, int migratetype,
int percpu_refill),
TP_ARGS(page, order, migratetype, percpu_refill),
TP_STRUCT__entry(
__field( unsigned long, pfn )
__field( unsigned int, order )
__field( int, migratetype )
__field( int, percpu_refill )
),
TP_fast_assign(
__entry->pfn = page ? page_to_pfn(page) : -1UL;
__entry->order = order;
__entry->migratetype = migratetype;
__entry->percpu_refill = percpu_refill;
),
TP_printk("page=%p pfn=0x%lx order=%u migratetype=%d percpu_refill=%d",
__entry->pfn != -1UL ? pfn_to_page(__entry->pfn) : NULL,
__entry->pfn != -1UL ? __entry->pfn : 0,
__entry->order,
__entry->migratetype,
__entry->percpu_refill)
);
DEFINE_EVENT(mm_page, mm_page_alloc_zone_locked,
TP_PROTO(struct page *page, unsigned int order, int migratetype,
int percpu_refill),
TP_ARGS(page, order, migratetype, percpu_refill)
);
TRACE_EVENT(mm_page_pcpu_drain,
TP_PROTO(struct page *page, unsigned int order, int migratetype),
TP_ARGS(page, order, migratetype),
TP_STRUCT__entry(
__field( unsigned long, pfn )
__field( unsigned int, order )
__field( int, migratetype )
),
TP_fast_assign(
__entry->pfn = page ? page_to_pfn(page) : -1UL;
__entry->order = order;
__entry->migratetype = migratetype;
),
TP_printk("page=%p pfn=0x%lx order=%d migratetype=%d",
pfn_to_page(__entry->pfn), __entry->pfn,
__entry->order, __entry->migratetype)
);
TRACE_EVENT(mm_page_alloc_extfrag,
TP_PROTO(struct page *page,
int alloc_order, int fallback_order,
int alloc_migratetype, int fallback_migratetype),
TP_ARGS(page,
alloc_order, fallback_order,
alloc_migratetype, fallback_migratetype),
TP_STRUCT__entry(
__field( unsigned long, pfn )
__field( int, alloc_order )
__field( int, fallback_order )
__field( int, alloc_migratetype )
__field( int, fallback_migratetype )
__field( int, change_ownership )
),
TP_fast_assign(
__entry->pfn = page_to_pfn(page);
__entry->alloc_order = alloc_order;
__entry->fallback_order = fallback_order;
__entry->alloc_migratetype = alloc_migratetype;
__entry->fallback_migratetype = fallback_migratetype;
__entry->change_ownership = (alloc_migratetype ==
get_pageblock_migratetype(page));
),
TP_printk("page=%p pfn=0x%lx alloc_order=%d fallback_order=%d pageblock_order=%d alloc_migratetype=%d fallback_migratetype=%d fragmenting=%d change_ownership=%d",
pfn_to_page(__entry->pfn),
__entry->pfn,
__entry->alloc_order,
__entry->fallback_order,
pageblock_order,
__entry->alloc_migratetype,
__entry->fallback_migratetype,
__entry->fallback_order < pageblock_order,
__entry->change_ownership)
);
TRACE_EVENT(mm_setup_per_zone_wmarks,
TP_PROTO(struct zone *zone),
TP_ARGS(zone),
TP_STRUCT__entry(
__field(int, node_id)
__string(name, zone->name)
__field(unsigned long, watermark_min)
__field(unsigned long, watermark_low)
__field(unsigned long, watermark_high)
__field(unsigned long, watermark_promo)
),
TP_fast_assign(
__entry->node_id = zone->zone_pgdat->node_id;
__assign_str(name);
__entry->watermark_min = zone->_watermark[WMARK_MIN];
__entry->watermark_low = zone->_watermark[WMARK_LOW];
__entry->watermark_high = zone->_watermark[WMARK_HIGH];
__entry->watermark_promo = zone->_watermark[WMARK_PROMO];
),
TP_printk("node_id=%d zone name=%s watermark min=%lu low=%lu high=%lu promo=%lu",
__entry->node_id,
__get_str(name),
__entry->watermark_min,
__entry->watermark_low,
__entry->watermark_high,
__entry->watermark_promo)
);
TRACE_EVENT(mm_setup_per_zone_lowmem_reserve,
TP_PROTO(struct zone *zone, struct zone *upper_zone, long lowmem_reserve),
TP_ARGS(zone, upper_zone, lowmem_reserve),
TP_STRUCT__entry(
__field(int, node_id)
__string(name, zone->name)
__string(upper_name, upper_zone->name)
__field(long, lowmem_reserve)
),
TP_fast_assign(
__entry->node_id = zone->zone_pgdat->node_id;
__assign_str(name);
__assign_str(upper_name);
__entry->lowmem_reserve = lowmem_reserve;
),
TP_printk("node_id=%d zone name=%s upper_zone name=%s lowmem_reserve_pages=%ld",
__entry->node_id,
__get_str(name),
__get_str(upper_name),
__entry->lowmem_reserve)
);
TRACE_EVENT(mm_calculate_totalreserve_pages,
TP_PROTO(unsigned long totalreserve_pages),
TP_ARGS(totalreserve_pages),
TP_STRUCT__entry(
__field(unsigned long, totalreserve_pages)
),
TP_fast_assign(
__entry->totalreserve_pages = totalreserve_pages;
),
TP_printk("totalreserve_pages=%lu", __entry->totalreserve_pages)
);
/*
* Required for uniquely and securely identifying mm in rss_stat tracepoint.
*/
#ifndef __PTR_TO_HASHVAL
static unsigned int __maybe_unused mm_ptr_to_hash(const void *ptr)
{
int ret;
unsigned long hashval;
ret = ptr_to_hashval(ptr, &hashval);
if (ret)
return 0;
/* The hashed value is only 32-bit */
return (unsigned int)hashval;
}
#define __PTR_TO_HASHVAL
#endif
#define TRACE_MM_PAGES \
EM(MM_FILEPAGES) \
EM(MM_ANONPAGES) \
EM(MM_SWAPENTS) \
EMe(MM_SHMEMPAGES)
#undef EM
#undef EMe
#define EM(a) TRACE_DEFINE_ENUM(a);
#define EMe(a) TRACE_DEFINE_ENUM(a);
TRACE_MM_PAGES
#undef EM
#undef EMe
#define EM(a) { a, #a },
#define EMe(a) { a, #a }
TRACE_EVENT(rss_stat,
TP_PROTO(struct mm_struct *mm,
int member),
TP_ARGS(mm, member),
TP_STRUCT__entry(
__field(unsigned int, mm_id)
__field(unsigned int, curr)
__field(int, member)
__field(long, size)
),
TP_fast_assign(
__entry->mm_id = mm_ptr_to_hash(mm);
/*
* curr is true if the mm matches the current task's mm_struct.
* Since kthreads (PF_KTHREAD) have no mm_struct of their own
* but can borrow one via kthread_use_mm(), we must filter them
* out to avoid incorrectly attributing the RSS update to them.
*/
__entry->curr = current->mm == mm && !(current->flags & PF_KTHREAD);
__entry->member = member;
__entry->size = (percpu_counter_sum_positive(&mm->rss_stat[member])
<< PAGE_SHIFT);
),
TP_printk("mm_id=%u curr=%d type=%s size=%ldB",
__entry->mm_id,
__entry->curr,
__print_symbolic(__entry->member, TRACE_MM_PAGES),
__entry->size)
);
#endif /* _TRACE_KMEM_H */
/* This part must be outside protection */
#include <trace/define_trace.h>
]
|
| 37/2 |
2026/03/15 01:23 |
llm |
expert |
0m
Model:
Tokens:
input: 82443
output: 53
thoughts: 2441
Thoughts: **My Analysis of a Potential UAF Vulnerability in BPF with Tracepoints**
Okay, so I've been tracing the logic here, and it looks like we might have a race condition leading to a Use-After-Free (UAF) issue. It stems from how BPF links are handled in conjunction with tracepoints, specifically when those tracepoints utilize SRCU (Source Read-Copy-Update) for synchronization, which is the default for most regular tracepoints defined via `TRACE_EVENT`.
Here's the breakdown as I understand it:
1. Tracepoints defined with `TRACE_EVENT` are set up to use `tracepoint_srcu` for read-side critical sections. This is confirmed by tracing the macro definitions to `__DECLARE_TRACE`. The macro definition itself includes `guard(srcu_fast_notrace)(&tracepoint_srcu)`.
2. BPF programs can be attached to tracepoints via `BPF_PROG_TYPE_RAW_TRACEPOINT`. This is handled by `bpf_raw_tp_link`.
3. When a BPF program is attached, `tracepoint_probe_register_may_exist` is called, which essentially "hooks" the BPF program into the tracepoint's execution.
4. The issue arises during link deallocation. In `bpf_link_free`, the code determines whether to use `call_rcu` or `call_rcu_tasks_trace` based on `link->sleepable`. Notably, it doesn't consider whether the target tracepoint uses SRCU.
5. If `link->sleepable` is false (and the link isn't a syscall), it uses `call_rcu` or `call_rcu_tasks_trace`, which only waits for a regular RCU grace period. This will be an issue because the tracepoint function which is running is protected by SRCU, not RCU.
6. The BPF link, and the BPF program it references, are then freed after only the RCU grace period. But because the tracepoint uses SRCU, there's a window where a tracepoint can *still* be executing and trying to access this freed memory. This is the UAF.
Specifically, it looks like `bpf_raw_tp_link` doesn't fully respect the synchronization requirements of the tracepoints it's attached to.
The core of the problem seems to be that the BPF infrastructure is not aware of, or isn't properly handling, the SRCU grace period required by these standard tracepoints. The BPF code should be waiting for SRCU to complete before freeing the BPF link, but it's not.
My assumption is that there's a mismatch between the BPF code and the tracepoint infrastructure's synchronization mechanisms. It seems like the BPF code needs to be updated to account for SRCU when a BPF link is associated with a tracepoint that uses SRCU. The key question is whether this is an oversight, or if there's a reason BPF *can't* wait on SRCU, which I don't see yet.
I need to confirm if this mismatch is recent or if BPF hasn't caught up with the use of SRCU in the tracepoint infrastructure.
I'm checking the definition of `SRCU_FAST` now to see what its impact on the grace period is. I am also checking to see if `bpf_raw_tp_link` is protected by RCU anywhere, as it may provide some insight. But it looks like BPF uses its own RCU locks in a different function, and not the link object directly.
I'm also looking into `bpf_link_defer_dealloc_mult_rcu_gp` to understand how it impacts things.
|
| 38/2 |
2026/03/15 01:24 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/syscall.c IncludeLines:true Name:bpf_link_defer_dealloc_mult_rcu_gp]
Results: map[SourceCode:3264: static void bpf_link_defer_dealloc_mult_rcu_gp(struct rcu_head *rcu)
3265: {
3266: if (rcu_trace_implies_rcu_gp())
3267: bpf_link_defer_dealloc_rcu_gp(rcu);
3268: else
3269: call_rcu(rcu, bpf_link_defer_dealloc_rcu_gp);
3270: }
SourceFile:kernel/bpf/syscall.c]
|
| 39/2 |
2026/03/15 01:24 |
llm |
expert |
0m
Model:
Tokens:
input: 85920
output: 46
thoughts: 36
Thoughts: Okay, here's my interpretation of those thoughts, tailored for an expert audience, as if they were my own:
**RCU Grace Period Wait and BPF Link Initialization**
Right, so we're still potentially waiting on the RCU grace period. This is the expected behavior, especially if we're dealing with concurrent operations. No SRCU in sight, which makes sense given the context, or at least, not yet.
Hold on... Let's switch gears for a second. That reminder that the current situation relates to the behavior, not an unusual case, makes me consider what is happening with `bpf_link_init_sleepable`. I wonder if there might be a subtle interaction there. Let's trace it through.
|
| 40/2 |
2026/03/15 01:24 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/syscall.c IncludeLines:true Name:bpf_link_init_sleepable]
Results: map[SourceCode:3183: /* bpf_link_init_sleepable() allows to specify whether BPF link itself has
3184: * "sleepable" semantics, which normally would mean that BPF link's attach
3185: * hook can dereference link or link's underlying program for some time after
3186: * detachment due to RCU Tasks Trace-based lifetime protection scheme.
3187: * BPF program itself can be non-sleepable, yet, because it's transitively
3188: * reachable through BPF link, its freeing has to be delayed until after RCU
3189: * Tasks Trace GP.
3190: */
3191: void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type,
3192: const struct bpf_link_ops *ops, struct bpf_prog *prog,
3193: enum bpf_attach_type attach_type, bool sleepable)
3194: {
3195: WARN_ON(ops->dealloc && ops->dealloc_deferred);
3196: atomic64_set(&link->refcnt, 1);
3197: link->type = type;
3198: link->sleepable = sleepable;
3199: link->id = 0;
3200: link->ops = ops;
3201: link->prog = prog;
3202: link->attach_type = attach_type;
3203: }
SourceFile:kernel/bpf/syscall.c]
|
| 41/2 |
2026/03/15 01:24 |
llm |
expert |
0m
Model:
Tokens:
input: 86586
output: 23
thoughts: 789
Thoughts: **Analyzing BPF Link Detachment with Tracepoints**
Okay, so the comment mentions "sleepable" semantics and RCU Tasks Trace for BPF link detachment, meaning the BPF link's attach hook might dereference the link or its program after detachment. But it *only* mentions RCU. My first thought is: what about SRCU? If the tracepoint uses SRCU, shouldn't the `bpf_link` also wait for SRCU? I need to figure out how this interaction works.
First instinct is to check if `tracepoint_srcu` is new, maybe look at the history of `include/linux/tracepoint.h` or search the code for "tracepoint_srcu bpf." Wait, maybe a simpler approach, check if other `bpf_link` types use SRCU to get a sense of the broader context.
Let's dive back into `bpf_raw_tp_link_attach`. Ah, here's `tracepoint_is_faultable(btp->tp)`. Right, that returns true for syscall tracepoints and false otherwise. If it's false, the link isn't "sleepable" and `bpf_link_free` uses `call_rcu`. But the tracepoint uses SRCU! That's the conflict I'm seeing.
Okay, I see `tracepoint_synchronize_unregister` in `include/linux/tracepoint.h`:
```c
109: static inline void tracepoint_synchronize_unregister(void)
110: {
111: synchronize_rcu_tasks_trace();
112: synchronize_srcu(&tracepoint_srcu);
113: }
```
That waits for *both* RCU and SRCU. If BPF used this directly, it'd be safe. But the BPF code aims for asynchronicity (using `call_rcu`), so it doesn't wait for the SRCU.
Now, is there a `call_srcu(&tracepoint_srcu, ...)`? Yes, it's used in `release_probes`. Hmm. If BPF wants to use `call_srcu`, it *needs* to know about `tracepoint_srcu`.
Aha! Maybe I'm on to something, maybe this is a known issue, or even a recent change. Time to check if `tracepoint_srcu` is used in `__do_trace_##name` in *all* cases. The code from `read-file` looks promising, I see the calls to `guard(srcu_fast_notrace)(&tracepoint_srcu)`. Okay, yes, it's there.
I'm probably safe assuming `CONFIG_TRACEPOINTS` is defined, because if it wasn't, tracepoints wouldn't exist!
Now, the lingering question is if `tracepoint_srcu` is some kind of accelerated SRCU that is synchronized with RCU. No, SRCU is SRCU.
Let's revisit `kernel/tracepoint.c`. There it is:
```c
41: DEFINE_SRCU_FAST(tracepoint_srcu);
```
Hmm, `DEFINE_SRCU_FAST`. That doesn't sound like a standard SRCU macro. I need to understand what this macro does!
|
| 42/2 |
2026/03/15 01:24 |
tool |
grepper |
0m
Args: map[Expression:#define DEFINE_SRCU_FAST]
Results: map[Output:include/linux/srcutiny.h=32=void srcu_drive_gp(struct work_struct *wp);
--
include/linux/srcutiny.h-49- static struct srcu_struct name = __SRCU_STRUCT_INIT(name, name, name, name)
include/linux/srcutiny.h:50:#define DEFINE_SRCU_FAST(name) DEFINE_SRCU(name)
include/linux/srcutiny.h-51-#define DEFINE_STATIC_SRCU_FAST(name) \
include/linux/srcutiny.h-52- static struct srcu_struct name = __SRCU_STRUCT_INIT(name, name, name, name)
include/linux/srcutiny.h:53:#define DEFINE_SRCU_FAST_UPDOWN(name) DEFINE_SRCU(name)
include/linux/srcutiny.h-54-#define DEFINE_STATIC_SRCU_FAST_UPDOWN(name) \
--
include/linux/srcutree.h=104=struct srcu_struct {
--
include/linux/srcutree.h-228-#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, 0, static)
include/linux/srcutree.h:229:#define DEFINE_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST, /* not static */)
include/linux/srcutree.h-230-#define DEFINE_STATIC_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST, static)
include/linux/srcutree.h:231:#define DEFINE_SRCU_FAST_UPDOWN(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST_UPDOWN, \
include/linux/srcutree.h-232- /* not static */)
]
|
| 43/2 |
2026/03/15 01:24 |
llm |
expert |
0m
Model:
Tokens:
input: 88622
output: 24
thoughts: 55
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Analyzing the SRCU Read Flavor**
My initial instinct is pointing towards `SRCU_READ_FLAVOR_FAST`. That's where I'm leaning, but I need to be sure. My memory's triggered some related SRCU nuances, so itβs time to double-check. I'm going to jump into `include/linux/srcutree.h` to refresh my memory and verify the specific definition and implications of `SRCU_READ_FLAVOR_FAST`. I want to make sure I have all the relevant context, especially considering different SRCU flavors can have subtle but critical performance trade-offs in various scenarios. This will help confirm that this is indeed the appropriate flavor for the task at hand.
|
| 44/2 |
2026/03/15 01:24 |
tool |
read-file |
0m
Args: map[File:include/linux/srcutree.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0+ */
/*
* Sleepable Read-Copy Update mechanism for mutual exclusion,
* tree variant.
*
* Copyright (C) IBM Corporation, 2017
*
* Author: Paul McKenney <paulmck@linux.ibm.com>
*/
#ifndef _LINUX_SRCU_TREE_H
#define _LINUX_SRCU_TREE_H
#include <linux/rcu_node_tree.h>
#include <linux/completion.h>
struct srcu_node;
struct srcu_struct;
/* One element of the srcu_data srcu_ctrs array. */
struct srcu_ctr {
atomic_long_t srcu_locks; /* Locks per CPU. */
atomic_long_t srcu_unlocks; /* Unlocks per CPU. */
};
/*
* Per-CPU structure feeding into leaf srcu_node, similar in function
* to rcu_node.
*/
struct srcu_data {
/* Read-side state. */
struct srcu_ctr srcu_ctrs[2]; /* Locks and unlocks per CPU. */
int srcu_reader_flavor; /* Reader flavor for srcu_struct structure? */
/* Values: SRCU_READ_FLAVOR_.* */
/* Update-side state. */
spinlock_t __private lock ____cacheline_internodealigned_in_smp;
struct rcu_segcblist srcu_cblist; /* List of callbacks.*/
unsigned long srcu_gp_seq_needed; /* Furthest future GP needed. */
unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */
bool srcu_cblist_invoking; /* Invoking these CBs? */
struct timer_list delay_work; /* Delay for CB invoking */
struct work_struct work; /* Context for CB invoking. */
struct rcu_head srcu_barrier_head; /* For srcu_barrier() use. */
struct rcu_head srcu_ec_head; /* For srcu_expedite_current() use. */
int srcu_ec_state; /* State for srcu_expedite_current(). */
struct srcu_node *mynode; /* Leaf srcu_node. */
unsigned long grpmask; /* Mask for leaf srcu_node */
/* ->srcu_data_have_cbs[]. */
int cpu;
struct srcu_struct *ssp;
};
/*
* Node in SRCU combining tree, similar in function to rcu_data.
*/
struct srcu_node {
spinlock_t __private lock;
unsigned long srcu_have_cbs[4]; /* GP seq for children having CBs, but only */
/* if greater than ->srcu_gp_seq. */
unsigned long srcu_data_have_cbs[4]; /* Which srcu_data structs have CBs for given GP? */
unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */
struct srcu_node *srcu_parent; /* Next up in tree. */
int grplo; /* Least CPU for node. */
int grphi; /* Biggest CPU for node. */
};
/*
* Per-SRCU-domain structure, update-side data linked from srcu_struct.
*/
struct srcu_usage {
struct srcu_node *node; /* Combining tree. */
struct srcu_node *level[RCU_NUM_LVLS + 1];
/* First node at each level. */
int srcu_size_state; /* Small-to-big transition state. */
struct mutex srcu_cb_mutex; /* Serialize CB preparation. */
spinlock_t __private lock; /* Protect counters and size state. */
struct mutex srcu_gp_mutex; /* Serialize GP work. */
unsigned long srcu_gp_seq; /* Grace-period seq #. */
unsigned long srcu_gp_seq_needed; /* Latest gp_seq needed. */
unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */
unsigned long srcu_gp_start; /* Last GP start timestamp (jiffies) */
unsigned long srcu_last_gp_end; /* Last GP end timestamp (ns) */
unsigned long srcu_size_jiffies; /* Current contention-measurement interval. */
unsigned long srcu_n_lock_retries; /* Contention events in current interval. */
unsigned long srcu_n_exp_nodelay; /* # expedited no-delays in current GP phase. */
bool sda_is_static; /* May ->sda be passed to free_percpu()? */
unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */
struct mutex srcu_barrier_mutex; /* Serialize barrier ops. */
struct completion srcu_barrier_completion;
/* Awaken barrier rq at end. */
atomic_t srcu_barrier_cpu_cnt; /* # CPUs not yet posting a */
/* callback for the barrier */
/* operation. */
unsigned long reschedule_jiffies;
unsigned long reschedule_count;
struct delayed_work work;
struct srcu_struct *srcu_ssp;
};
/*
* Per-SRCU-domain structure, similar in function to rcu_state.
*/
struct srcu_struct {
struct srcu_ctr __percpu *srcu_ctrp;
struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */
u8 srcu_reader_flavor;
struct lockdep_map dep_map;
struct srcu_usage *srcu_sup; /* Update-side data. */
};
// Values for size state variable (->srcu_size_state). Once the state
// has been set to SRCU_SIZE_ALLOC, the grace-period code advances through
// this state machine one step per grace period until the SRCU_SIZE_BIG state
// is reached. Otherwise, the state machine remains in the SRCU_SIZE_SMALL
// state indefinitely.
#define SRCU_SIZE_SMALL 0 // No srcu_node combining tree, ->node == NULL
#define SRCU_SIZE_ALLOC 1 // An srcu_node tree is being allocated, initialized,
// and then referenced by ->node. It will not be used.
#define SRCU_SIZE_WAIT_BARRIER 2 // The srcu_node tree starts being used by everything
// except call_srcu(), especially by srcu_barrier().
// By the end of this state, all CPUs and threads
// are aware of this tree's existence.
#define SRCU_SIZE_WAIT_CALL 3 // The srcu_node tree starts being used by call_srcu().
// By the end of this state, all of the call_srcu()
// invocations that were running on a non-boot CPU
// and using the boot CPU's callback queue will have
// completed.
#define SRCU_SIZE_WAIT_CBS1 4 // Don't trust the ->srcu_have_cbs[] grace-period
#define SRCU_SIZE_WAIT_CBS2 5 // sequence elements or the ->srcu_data_have_cbs[]
#define SRCU_SIZE_WAIT_CBS3 6 // CPU-bitmask elements until all four elements of
#define SRCU_SIZE_WAIT_CBS4 7 // each array have been initialized.
#define SRCU_SIZE_BIG 8 // The srcu_node combining tree is fully initialized
// and all aspects of it are being put to use.
/* Values for state variable (bottom bits of ->srcu_gp_seq). */
#define SRCU_STATE_IDLE 0
#define SRCU_STATE_SCAN1 1
#define SRCU_STATE_SCAN2 2
/* Values for srcu_expedite_current() state (->srcu_ec_state). */
#define SRCU_EC_IDLE 0
#define SRCU_EC_PENDING 1
#define SRCU_EC_REPOST 2
/*
* Values for initializing gp sequence fields. Higher values allow wrap arounds to
* occur earlier.
* The second value with state is useful in the case of static initialization of
* srcu_usage where srcu_gp_seq_needed is expected to have some state value in its
* lower bits (or else it will appear to be already initialized within
* the call check_init_srcu_struct()).
*/
#define SRCU_GP_SEQ_INITIAL_VAL ((0UL - 100UL) << RCU_SEQ_CTR_SHIFT)
#define SRCU_GP_SEQ_INITIAL_VAL_WITH_STATE (SRCU_GP_SEQ_INITIAL_VAL - 1)
#define __SRCU_USAGE_INIT(name) \
{ \
.lock = __SPIN_LOCK_UNLOCKED(name.lock), \
.srcu_gp_seq = SRCU_GP_SEQ_INITIAL_VAL, \
.srcu_gp_seq_needed = SRCU_GP_SEQ_INITIAL_VAL_WITH_STATE, \
.srcu_gp_seq_needed_exp = SRCU_GP_SEQ_INITIAL_VAL, \
.work = __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \
}
#define __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \
.srcu_sup = &usage_name, \
.srcu_reader_flavor = fast, \
__SRCU_DEP_MAP_INIT(name)
#define __SRCU_STRUCT_INIT_MODULE(name, usage_name, fast) \
{ \
__SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \
}
#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name, fast) \
{ \
.sda = &pcpu_name, \
.srcu_ctrp = &pcpu_name.srcu_ctrs[0], \
__SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \
}
/*
* Define and initialize a srcu struct at build time.
* Do -not- call init_srcu_struct() nor cleanup_srcu_struct() on it.
*
* Note that although DEFINE_STATIC_SRCU() hides the name from other
* files, the per-CPU variable rules nevertheless require that the
* chosen name be globally unique. These rules also prohibit use of
* DEFINE_STATIC_SRCU() within a function. If these rules are too
* restrictive, declare the srcu_struct manually. For example, in
* each file:
*
* static struct srcu_struct my_srcu;
*
* Then, before the first use of each my_srcu, manually initialize it:
*
* init_srcu_struct(&my_srcu);
*
* See include/linux/percpu-defs.h for the rules on per-CPU variables.
*
* DEFINE_SRCU_FAST() and DEFINE_STATIC_SRCU_FAST create an srcu_struct
* and associated structures whose readers must be of the SRCU-fast variety.
* DEFINE_SRCU_FAST_UPDOWN() and DEFINE_STATIC_SRCU_FAST_UPDOWN() create
* an srcu_struct and associated structures whose readers must be of the
* SRCU-fast-updown variety. The key point (aside from error checking) with
* both varieties is that the grace periods must use synchronize_rcu()
* instead of smp_mb(), and given that the first (for example)
* srcu_read_lock_fast() might race with the first synchronize_srcu(),
* this different must be specified at initialization time.
*/
#ifdef MODULE
# define __DEFINE_SRCU(name, fast, is_static) \
static struct srcu_usage name##_srcu_usage = __SRCU_USAGE_INIT(name##_srcu_usage); \
is_static struct srcu_struct name = __SRCU_STRUCT_INIT_MODULE(name, name##_srcu_usage, \
fast); \
extern struct srcu_struct * const __srcu_struct_##name; \
struct srcu_struct * const __srcu_struct_##name \
__section("___srcu_struct_ptrs") = &name
#else
# define __DEFINE_SRCU(name, fast, is_static) \
static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \
static struct srcu_usage name##_srcu_usage = __SRCU_USAGE_INIT(name##_srcu_usage); \
is_static struct srcu_struct name = \
__SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data, fast)
#endif
#define DEFINE_SRCU(name) __DEFINE_SRCU(name, 0, /* not static */)
#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, 0, static)
#define DEFINE_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST, /* not static */)
#define DEFINE_STATIC_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST, static)
#define DEFINE_SRCU_FAST_UPDOWN(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST_UPDOWN, \
/* not static */)
#define DEFINE_STATIC_SRCU_FAST_UPDOWN(name) \
__DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST_UPDOWN, static)
int __srcu_read_lock(struct srcu_struct *ssp) __acquires_shared(ssp);
void synchronize_srcu_expedited(struct srcu_struct *ssp);
void srcu_barrier(struct srcu_struct *ssp);
void srcu_expedite_current(struct srcu_struct *ssp);
void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf);
// Converts a per-CPU pointer to an ->srcu_ctrs[] array element to that
// element's index.
static inline bool __srcu_ptr_to_ctr(struct srcu_struct *ssp, struct srcu_ctr __percpu *scpp)
{
return scpp - &ssp->sda->srcu_ctrs[0];
}
// Converts an integer to a per-CPU pointer to the corresponding
// ->srcu_ctrs[] array element.
static inline struct srcu_ctr __percpu *__srcu_ctr_to_ptr(struct srcu_struct *ssp, int idx)
{
return &ssp->sda->srcu_ctrs[idx];
}
/*
* Counts the new reader in the appropriate per-CPU element of the
* srcu_struct. Returns a pointer that must be passed to the matching
* srcu_read_unlock_fast().
*
* Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side
* critical sections either because they disables interrupts, because
* they are a single instruction, or because they are read-modify-write
* atomic operations, depending on the whims of the architecture.
* This matters because the SRCU-fast grace-period mechanism uses either
* synchronize_rcu() or synchronize_rcu_expedited(), that is, RCU,
* *not* SRCU, in order to eliminate the need for the read-side smp_mb()
* invocations that are used by srcu_read_lock() and srcu_read_unlock().
* The __srcu_read_unlock_fast() function also relies on this same RCU
* (again, *not* SRCU) trick to eliminate the need for smp_mb().
*
* The key point behind this RCU trick is that if any part of a given
* RCU reader precedes the beginning of a given RCU grace period, then
* the entirety of that RCU reader and everything preceding it happens
* before the end of that same RCU grace period. Similarly, if any part
* of a given RCU reader follows the end of a given RCU grace period,
* then the entirety of that RCU reader and everything following it
* happens after the beginning of that same RCU grace period. Therefore,
* the operations labeled Y in __srcu_read_lock_fast() and those labeled Z
* in __srcu_read_unlock_fast() are ordered against the corresponding SRCU
* read-side critical section from the viewpoint of the SRCU grace period.
* This is all the ordering that is required, hence no calls to smp_mb().
*
* This means that __srcu_read_lock_fast() is not all that fast
* on architectures that support NMIs but do not supply NMI-safe
* implementations of this_cpu_inc().
*/
static inline struct srcu_ctr __percpu notrace *__srcu_read_lock_fast(struct srcu_struct *ssp)
__acquires_shared(ssp)
{
struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp);
if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))
this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader.
else
atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); // Y, and implicit RCU reader.
barrier(); /* Avoid leaking the critical section. */
__acquire_shared(ssp);
return scp;
}
/*
* Removes the count for the old reader from the appropriate
* per-CPU element of the srcu_struct. Note that this may well be a
* different CPU than that which was incremented by the corresponding
* srcu_read_lock_fast(), but it must be within the same task.
*
* Please see the __srcu_read_lock_fast() function's header comment for
* information on implicit RCU readers and NMI safety.
*/
static inline void notrace
__srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp)
__releases_shared(ssp)
{
__release_shared(ssp);
barrier(); /* Avoid leaking the critical section. */
if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))
this_cpu_inc(scp->srcu_unlocks.counter); // Z, and implicit RCU reader.
else
atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RCU reader.
}
/*
* Counts the new reader in the appropriate per-CPU element of the
* srcu_struct. Returns a pointer that must be passed to the matching
* srcu_read_unlock_fast_updown(). This type of reader is compatible
* with srcu_down_read_fast() and srcu_up_read_fast().
*
* See the __srcu_read_lock_fast() comment for more details.
*/
static inline
struct srcu_ctr __percpu notrace *__srcu_read_lock_fast_updown(struct srcu_struct *ssp)
__acquires_shared(ssp)
{
struct srcu_ctr __percpu *scp = READ_ONCE(ssp->srcu_ctrp);
if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))
this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader.
else
atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); // Y, and implicit RCU reader.
barrier(); /* Avoid leaking the critical section. */
__acquire_shared(ssp);
return scp;
}
/*
* Removes the count for the old reader from the appropriate
* per-CPU element of the srcu_struct. Note that this may well be a
* different CPU than that which was incremented by the corresponding
* srcu_read_lock_fast(), but it must be within the same task.
*
* Please see the __srcu_read_lock_fast() function's header comment for
* information on implicit RCU readers and NMI safety.
*/
static inline void notrace
__srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp)
__releases_shared(ssp)
{
__release_shared(ssp);
barrier(); /* Avoid leaking the critical section. */
if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))
this_cpu_inc(scp->srcu_unlocks.counter); // Z, and implicit RCU reader.
else
atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RCU reader.
}
void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor);
// Record SRCU-reader usage type only for CONFIG_PROVE_RCU=y kernels.
static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor)
{
if (IS_ENABLED(CONFIG_PROVE_RCU))
__srcu_check_read_flavor(ssp, read_flavor);
}
#endif
]
|
| 45/2 |
2026/03/15 01:24 |
llm |
expert |
0m
Model:
Tokens:
input: 94191
output: 23
thoughts: 1355
Thoughts: **Analyzing SRCU-Fast and RCU Interaction in Tracepoints**
Okay, let's break this down. That comment on `__srcu_read_lock_fast` is very telling. It highlights that the SRCU-fast mechanism leverages RCU for grace periods via `synchronize_rcu()` or `synchronize_rcu_expedited()`. This is key. The comment explicitly states RCU is used, not SRCU, for grace periods in this context. If `tracepoint_srcu` is a `DEFINE_SRCU_FAST`, then its grace period is indeed tied to RCU.
If that's the case, then `call_rcu()` in BPF *should* be sufficient to wait for the readers. Previously I thought there was some mismatch between SRCU and RCU but this points at that being wrong, so let's re-evaluate.
Let's look more closely at the comment. It notes that "if any part of a given RCU reader precedes the beginning of a given RCU grace period, then the entirety of that RCU reader and everything preceding it happens before the end of that same RCU grace period". Also, in the initialization "this different must be specified at initialization time".
If `synchronize_srcu(&tracepoint_srcu)` is called, it will use RCU grace periods. Now, `call_rcu()` uses a regular RCU grace period. So, does a regular RCU grace period wait for the `tracepoint_srcu` readers, as defined in the SRCU-fast case? The comment clarifies: "Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side critical sections...This matters because the SRCU-fast grace-period mechanism uses either synchronize_rcu() ... that is, RCU, *not* SRCU". So it looks like `this_cpu_inc()` is an RCU read-side critical section, and thus `synchronize_rcu()` should wait for it.
If `tracepoint_srcu` is SRCU-fast, then any reader is *also* an RCU reader. Therefore, `synchronize_rcu()` (and `call_rcu()`) should indeed wait for those readers. This means my initial hypothesis about a possible SRCU/RCU mismatch is likely incorrect.
Now, let's re-examine the stack trace. The UAF is in `__bpf_trace_run`. The task is `udevd/5299`, and it's doing a page allocation. The allocation of the `link` was by task 5326, and the free was by task 15 (likely `rcu_core`). I see that the `link` was being released because its FD was closed, and `call_rcu` was being used.
Given my current understanding, the link *shouldn't* have been freed until the tracepoint finished. But, there's a crucial detail β the report indicates `PREEMPT(full)`.
With `PREEMPT(full)`, a single instruction (like `this_cpu_inc()`) might *not* inherently be an RCU read-side critical section. It depends on whether `rcu_read_lock()` is used.
Does `__srcu_read_lock_fast` call `rcu_read_lock()`? No, and the comment clarifies: "Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side critical sections either because they disables interrupts, because they are a single instruction, or because they are read-modify-write atomic operations, depending on the whims of the architecture."
In `PREEMPT(full)`, a single instruction on x86, even if with a prefix, might be preempted if it is not atomic, so does not become an RCU reader by itself. The comment is saying that RCU readers *are* tracked in `PREEMPT(full)`.
If `synchronize_rcu()` is used, it waits for all RCU readers. If `this_cpu_inc()` is *not* an RCU reader, then `synchronize_rcu()` *won't* wait for it, which could explain the UAF. The key must lie in how `__do_trace_##name` handles this.
The code shows that `guard(srcu_fast_notrace)` is used, and this would likely call `srcu_read_lock_fast()`. Now, I need to check `include/linux/srcu.h` to confirm the exact behavior of `srcu_read_lock_fast` in relation to `PREEMPT(full)` and how it handles potential preemption during `this_cpu_inc()
|
| 46/2 |
2026/03/15 01:24 |
tool |
read-file |
0m
Args: map[File:include/linux/srcu.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0+ */
/*
* Sleepable Read-Copy Update mechanism for mutual exclusion
*
* Copyright (C) IBM Corporation, 2006
* Copyright (C) Fujitsu, 2012
*
* Author: Paul McKenney <paulmck@linux.ibm.com>
* Lai Jiangshan <laijs@cn.fujitsu.com>
*
* For detailed explanation of Read-Copy Update mechanism see -
* Documentation/RCU/ *.txt
*
*/
#ifndef _LINUX_SRCU_H
#define _LINUX_SRCU_H
#include <linux/mutex.h>
#include <linux/rcupdate.h>
#include <linux/workqueue.h>
#include <linux/rcu_segcblist.h>
context_lock_struct(srcu_struct, __reentrant_ctx_lock);
#ifdef CONFIG_DEBUG_LOCK_ALLOC
int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct lock_class_key *key);
#ifndef CONFIG_TINY_SRCU
int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, struct lock_class_key *key);
int __init_srcu_struct_fast_updown(struct srcu_struct *ssp, const char *name,
struct lock_class_key *key);
#endif // #ifndef CONFIG_TINY_SRCU
#define init_srcu_struct(ssp) \
({ \
static struct lock_class_key __srcu_key; \
\
__init_srcu_struct((ssp), #ssp, &__srcu_key); \
})
#define init_srcu_struct_fast(ssp) \
({ \
static struct lock_class_key __srcu_key; \
\
__init_srcu_struct_fast((ssp), #ssp, &__srcu_key); \
})
#define init_srcu_struct_fast_updown(ssp) \
({ \
static struct lock_class_key __srcu_key; \
\
__init_srcu_struct_fast_updown((ssp), #ssp, &__srcu_key); \
})
#define __SRCU_DEP_MAP_INIT(srcu_name) .dep_map = { .name = #srcu_name },
#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
int init_srcu_struct(struct srcu_struct *ssp);
#ifndef CONFIG_TINY_SRCU
int init_srcu_struct_fast(struct srcu_struct *ssp);
int init_srcu_struct_fast_updown(struct srcu_struct *ssp);
#endif // #ifndef CONFIG_TINY_SRCU
#define __SRCU_DEP_MAP_INIT(srcu_name)
#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
/* Values for SRCU Tree srcu_data ->srcu_reader_flavor, but also used by rcutorture. */
#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock().
#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe().
// 0x4 // SRCU-lite is no longer with us.
#define SRCU_READ_FLAVOR_FAST 0x4 // srcu_read_lock_fast().
#define SRCU_READ_FLAVOR_FAST_UPDOWN 0x8 // srcu_read_lock_fast().
#define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_NMI | \
SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN)
// All of the above.
#define SRCU_READ_FLAVOR_SLOWGP (SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN)
// Flavors requiring synchronize_rcu()
// instead of smp_mb().
void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases_shared(ssp);
#ifdef CONFIG_TINY_SRCU
#include <linux/srcutiny.h>
#elif defined(CONFIG_TREE_SRCU)
#include <linux/srcutree.h>
#else
#error "Unknown SRCU implementation specified to kernel configuration"
#endif
void call_srcu(struct srcu_struct *ssp, struct rcu_head *head,
void (*func)(struct rcu_head *head));
void cleanup_srcu_struct(struct srcu_struct *ssp);
void synchronize_srcu(struct srcu_struct *ssp);
#define SRCU_GET_STATE_COMPLETED 0x1
/**
* get_completed_synchronize_srcu - Return a pre-completed polled state cookie
*
* Returns a value that poll_state_synchronize_srcu() will always treat
* as a cookie whose grace period has already completed.
*/
static inline unsigned long get_completed_synchronize_srcu(void)
{
return SRCU_GET_STATE_COMPLETED;
}
unsigned long get_state_synchronize_srcu(struct srcu_struct *ssp);
unsigned long start_poll_synchronize_srcu(struct srcu_struct *ssp);
bool poll_state_synchronize_srcu(struct srcu_struct *ssp, unsigned long cookie);
// Maximum number of unsigned long values corresponding to
// not-yet-completed SRCU grace periods.
#define NUM_ACTIVE_SRCU_POLL_OLDSTATE 2
/**
* same_state_synchronize_srcu - Are two old-state values identical?
* @oldstate1: First old-state value.
* @oldstate2: Second old-state value.
*
* The two old-state values must have been obtained from either
* get_state_synchronize_srcu(), start_poll_synchronize_srcu(), or
* get_completed_synchronize_srcu(). Returns @true if the two values are
* identical and @false otherwise. This allows structures whose lifetimes
* are tracked by old-state values to push these values to a list header,
* allowing those structures to be slightly smaller.
*/
static inline bool same_state_synchronize_srcu(unsigned long oldstate1, unsigned long oldstate2)
{
return oldstate1 == oldstate2;
}
#ifdef CONFIG_NEED_SRCU_NMI_SAFE
int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires_shared(ssp);
void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) __releases_shared(ssp);
#else
static inline int __srcu_read_lock_nmisafe(struct srcu_struct *ssp)
__acquires_shared(ssp)
{
return __srcu_read_lock(ssp);
}
static inline void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx)
__releases_shared(ssp)
{
__srcu_read_unlock(ssp, idx);
}
#endif /* CONFIG_NEED_SRCU_NMI_SAFE */
void srcu_init(void);
#ifdef CONFIG_DEBUG_LOCK_ALLOC
/**
* srcu_read_lock_held - might we be in SRCU read-side critical section?
* @ssp: The srcu_struct structure to check
*
* If CONFIG_DEBUG_LOCK_ALLOC is selected, returns nonzero iff in an SRCU
* read-side critical section. In absence of CONFIG_DEBUG_LOCK_ALLOC,
* this assumes we are in an SRCU read-side critical section unless it can
* prove otherwise.
*
* Checks debug_lockdep_rcu_enabled() to prevent false positives during boot
* and while lockdep is disabled.
*
* Note that SRCU is based on its own statemachine and it doesn't
* relies on normal RCU, it can be called from the CPU which
* is in the idle loop from an RCU point of view or offline.
*/
static inline int srcu_read_lock_held(const struct srcu_struct *ssp)
{
if (!debug_lockdep_rcu_enabled())
return 1;
return lock_is_held(&ssp->dep_map);
}
/*
* Annotations provide deadlock detection for SRCU.
*
* Similar to other lockdep annotations, except there is an additional
* srcu_lock_sync(), which is basically an empty *write*-side critical section,
* see lock_sync() for more information.
*/
/* Annotates a srcu_read_lock() */
static inline void srcu_lock_acquire(struct lockdep_map *map)
{
lock_map_acquire_read(map);
}
/* Annotates a srcu_read_lock() */
static inline void srcu_lock_release(struct lockdep_map *map)
{
lock_map_release(map);
}
/* Annotates a synchronize_srcu() */
static inline void srcu_lock_sync(struct lockdep_map *map)
{
lock_map_sync(map);
}
#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
static inline int srcu_read_lock_held(const struct srcu_struct *ssp)
{
return 1;
}
#define srcu_lock_acquire(m) do { } while (0)
#define srcu_lock_release(m) do { } while (0)
#define srcu_lock_sync(m) do { } while (0)
#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
/*
* No-op helper to denote that ssp must be held. Because SRCU-protected pointers
* should still be marked with __rcu_guarded, and we do not want to mark them
* with __guarded_by(ssp) as it would complicate annotations for writers, we
* choose the following strategy: srcu_dereference_check() calls this helper
* that checks that the passed ssp is held, and then fake-acquires 'RCU'.
*/
static inline void __srcu_read_lock_must_hold(const struct srcu_struct *ssp) __must_hold_shared(ssp) { }
/**
* srcu_dereference_check - fetch SRCU-protected pointer for later dereferencing
* @p: the pointer to fetch and protect for later dereferencing
* @ssp: pointer to the srcu_struct, which is used to check that we
* really are in an SRCU read-side critical section.
* @c: condition to check for update-side use
*
* If PROVE_RCU is enabled, invoking this outside of an RCU read-side
* critical section will result in an RCU-lockdep splat, unless @c evaluates
* to 1. The @c argument will normally be a logical expression containing
* lockdep_is_held() calls.
*/
#define srcu_dereference_check(p, ssp, c) \
({ \
__srcu_read_lock_must_hold(ssp); \
__acquire_shared_ctx_lock(RCU); \
__auto_type __v = __rcu_dereference_check((p), __UNIQUE_ID(rcu), \
(c) || srcu_read_lock_held(ssp), __rcu); \
__release_shared_ctx_lock(RCU); \
__v; \
})
/**
* srcu_dereference - fetch SRCU-protected pointer for later dereferencing
* @p: the pointer to fetch and protect for later dereferencing
* @ssp: pointer to the srcu_struct, which is used to check that we
* really are in an SRCU read-side critical section.
*
* Makes rcu_dereference_check() do the dirty work. If PROVE_RCU
* is enabled, invoking this outside of an RCU read-side critical
* section will result in an RCU-lockdep splat.
*/
#define srcu_dereference(p, ssp) srcu_dereference_check((p), (ssp), 0)
/**
* srcu_dereference_notrace - no tracing and no lockdep calls from here
* @p: the pointer to fetch and protect for later dereferencing
* @ssp: pointer to the srcu_struct, which is used to check that we
* really are in an SRCU read-side critical section.
*/
#define srcu_dereference_notrace(p, ssp) srcu_dereference_check((p), (ssp), 1)
/**
* srcu_read_lock - register a new reader for an SRCU-protected structure.
* @ssp: srcu_struct in which to register the new reader.
*
* Enter an SRCU read-side critical section. Note that SRCU read-side
* critical sections may be nested. However, it is illegal to
* call anything that waits on an SRCU grace period for the same
* srcu_struct, whether directly or indirectly. Please note that
* one way to indirectly wait on an SRCU grace period is to acquire
* a mutex that is held elsewhere while calling synchronize_srcu() or
* synchronize_srcu_expedited().
*
* The return value from srcu_read_lock() is guaranteed to be
* non-negative. This value must be passed unaltered to the matching
* srcu_read_unlock(). Note that srcu_read_lock() and the matching
* srcu_read_unlock() must occur in the same context, for example, it is
* illegal to invoke srcu_read_unlock() in an irq handler if the matching
* srcu_read_lock() was invoked in process context. Or, for that matter to
* invoke srcu_read_unlock() from one task and the matching srcu_read_lock()
* from another.
*/
static inline int srcu_read_lock(struct srcu_struct *ssp)
__acquires_shared(ssp)
{
int retval;
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
retval = __srcu_read_lock(ssp);
srcu_lock_acquire(&ssp->dep_map);
return retval;
}
/**
* srcu_read_lock_fast - register a new reader for an SRCU-protected structure.
* @ssp: srcu_struct in which to register the new reader.
*
* Enter an SRCU read-side critical section, but for a light-weight
* smp_mb()-free reader. See srcu_read_lock() for more information. This
* function is NMI-safe, in a manner similar to srcu_read_lock_nmisafe().
*
* For srcu_read_lock_fast() to be used on an srcu_struct structure,
* that structure must have been defined using either DEFINE_SRCU_FAST()
* or DEFINE_STATIC_SRCU_FAST() on the one hand or initialized with
* init_srcu_struct_fast() on the other. Such an srcu_struct structure
* cannot be passed to any non-fast variant of srcu_read_{,un}lock() or
* srcu_{down,up}_read(). In kernels built with CONFIG_PROVE_RCU=y,
* __srcu_check_read_flavor() will complain bitterly if you ignore this
* restriction.
*
* Grace-period auto-expediting is disabled for SRCU-fast srcu_struct
* structures because SRCU-fast expedited grace periods invoke
* synchronize_rcu_expedited(), IPIs and all. If you need expedited
* SRCU-fast grace periods, use synchronize_srcu_expedited().
*
* The srcu_read_lock_fast() function can be invoked only from those
* contexts where RCU is watching, that is, from contexts where it would
* be legal to invoke rcu_read_lock(). Otherwise, lockdep will complain.
*/
static inline struct srcu_ctr __percpu *srcu_read_lock_fast(struct srcu_struct *ssp) __acquires_shared(ssp)
__acquires_shared(ssp)
{
struct srcu_ctr __percpu *retval;
RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_fast().");
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST);
retval = __srcu_read_lock_fast(ssp);
rcu_try_lock_acquire(&ssp->dep_map);
return retval;
}
/**
* srcu_read_lock_fast_updown - register a new reader for an SRCU-fast-updown structure.
* @ssp: srcu_struct in which to register the new reader.
*
* Enter an SRCU read-side critical section, but for a light-weight
* smp_mb()-free reader. See srcu_read_lock() for more information.
* This function is compatible with srcu_down_read_fast(), but is not
* NMI-safe.
*
* For srcu_read_lock_fast_updown() to be used on an srcu_struct
* structure, that structure must have been defined using either
* DEFINE_SRCU_FAST_UPDOWN() or DEFINE_STATIC_SRCU_FAST_UPDOWN() on the one
* hand or initialized with init_srcu_struct_fast_updown() on the other.
* Such an srcu_struct structure cannot be passed to any non-fast-updown
* variant of srcu_read_{,un}lock() or srcu_{down,up}_read(). In kernels
* built with CONFIG_PROVE_RCU=y, __srcu_check_read_flavor() will complain
* bitterly if you ignore this * restriction.
*
* Grace-period auto-expediting is disabled for SRCU-fast-updown
* srcu_struct structures because SRCU-fast-updown expedited grace periods
* invoke synchronize_rcu_expedited(), IPIs and all. If you need expedited
* SRCU-fast-updown grace periods, use synchronize_srcu_expedited().
*
* The srcu_read_lock_fast_updown() function can be invoked only from
* those contexts where RCU is watching, that is, from contexts where
* it would be legal to invoke rcu_read_lock(). Otherwise, lockdep will
* complain.
*/
static inline struct srcu_ctr __percpu *srcu_read_lock_fast_updown(struct srcu_struct *ssp)
__acquires_shared(ssp)
{
struct srcu_ctr __percpu *retval;
RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock_fast_updown().");
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN);
retval = __srcu_read_lock_fast_updown(ssp);
rcu_try_lock_acquire(&ssp->dep_map);
return retval;
}
/*
* Used by tracing, cannot be traced and cannot call lockdep.
* See srcu_read_lock_fast() for more information.
*/
static inline struct srcu_ctr __percpu *srcu_read_lock_fast_notrace(struct srcu_struct *ssp)
__acquires_shared(ssp)
{
struct srcu_ctr __percpu *retval;
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST);
retval = __srcu_read_lock_fast(ssp);
return retval;
}
/**
* srcu_down_read_fast - register a new reader for an SRCU-protected structure.
* @ssp: srcu_struct in which to register the new reader.
*
* Enter a semaphore-like SRCU read-side critical section, but for
* a light-weight smp_mb()-free reader. See srcu_read_lock_fast() and
* srcu_down_read() for more information.
*
* The same srcu_struct may be used concurrently by srcu_down_read_fast()
* and srcu_read_lock_fast(). However, the same definition/initialization
* requirements called out for srcu_read_lock_safe() apply.
*/
static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_struct *ssp) __acquires_shared(ssp)
{
WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi());
RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read_fast().");
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN);
return __srcu_read_lock_fast_updown(ssp);
}
/**
* srcu_read_lock_nmisafe - register a new reader for an SRCU-protected structure.
* @ssp: srcu_struct in which to register the new reader.
*
* Enter an SRCU read-side critical section, but in an NMI-safe manner.
* See srcu_read_lock() for more information.
*
* If srcu_read_lock_nmisafe() is ever used on an srcu_struct structure,
* then none of the other flavors may be used, whether before, during,
* or after.
*/
static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp)
__acquires_shared(ssp)
{
int retval;
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NMI);
retval = __srcu_read_lock_nmisafe(ssp);
rcu_try_lock_acquire(&ssp->dep_map);
return retval;
}
/* Used by tracing, cannot be traced and cannot invoke lockdep. */
static inline notrace int
srcu_read_lock_notrace(struct srcu_struct *ssp)
__acquires_shared(ssp)
{
int retval;
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
retval = __srcu_read_lock(ssp);
return retval;
}
/**
* srcu_down_read - register a new reader for an SRCU-protected structure.
* @ssp: srcu_struct in which to register the new reader.
*
* Enter a semaphore-like SRCU read-side critical section. Note that
* SRCU read-side critical sections may be nested. However, it is
* illegal to call anything that waits on an SRCU grace period for the
* same srcu_struct, whether directly or indirectly. Please note that
* one way to indirectly wait on an SRCU grace period is to acquire
* a mutex that is held elsewhere while calling synchronize_srcu() or
* synchronize_srcu_expedited(). But if you want lockdep to help you
* keep this stuff straight, you should instead use srcu_read_lock().
*
* The semaphore-like nature of srcu_down_read() means that the matching
* srcu_up_read() can be invoked from some other context, for example,
* from some other task or from an irq handler. However, neither
* srcu_down_read() nor srcu_up_read() may be invoked from an NMI handler.
*
* Calls to srcu_down_read() may be nested, similar to the manner in
* which calls to down_read() may be nested. The same srcu_struct may be
* used concurrently by srcu_down_read() and srcu_read_lock().
*/
static inline int srcu_down_read(struct srcu_struct *ssp)
__acquires_shared(ssp)
{
WARN_ON_ONCE(in_nmi());
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
return __srcu_read_lock(ssp);
}
/**
* srcu_read_unlock - unregister a old reader from an SRCU-protected structure.
* @ssp: srcu_struct in which to unregister the old reader.
* @idx: return value from corresponding srcu_read_lock().
*
* Exit an SRCU read-side critical section.
*/
static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx)
__releases_shared(ssp)
{
WARN_ON_ONCE(idx & ~0x1);
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
srcu_lock_release(&ssp->dep_map);
__srcu_read_unlock(ssp, idx);
}
/**
* srcu_read_unlock_fast - unregister a old reader from an SRCU-protected structure.
* @ssp: srcu_struct in which to unregister the old reader.
* @scp: return value from corresponding srcu_read_lock_fast().
*
* Exit a light-weight SRCU read-side critical section.
*/
static inline void srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp)
__releases_shared(ssp)
{
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST);
srcu_lock_release(&ssp->dep_map);
__srcu_read_unlock_fast(ssp, scp);
RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlock_fast().");
}
/**
* srcu_read_unlock_fast_updown - unregister a old reader from an SRCU-fast-updown structure.
* @ssp: srcu_struct in which to unregister the old reader.
* @scp: return value from corresponding srcu_read_lock_fast_updown().
*
* Exit an SRCU-fast-updown read-side critical section.
*/
static inline void
srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp) __releases_shared(ssp)
{
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN);
srcu_lock_release(&ssp->dep_map);
__srcu_read_unlock_fast_updown(ssp, scp);
RCU_LOCKDEP_WARN(!rcu_is_watching(),
"RCU must be watching srcu_read_unlock_fast_updown().");
}
/*
* Used by tracing, cannot be traced and cannot call lockdep.
* See srcu_read_unlock_fast() for more information.
*/
static inline void srcu_read_unlock_fast_notrace(struct srcu_struct *ssp,
struct srcu_ctr __percpu *scp) __releases_shared(ssp)
{
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST);
__srcu_read_unlock_fast(ssp, scp);
}
/**
* srcu_up_read_fast - unregister a old reader from an SRCU-protected structure.
* @ssp: srcu_struct in which to unregister the old reader.
* @scp: return value from corresponding srcu_read_lock_fast().
*
* Exit an SRCU read-side critical section, but not necessarily from
* the same context as the maching srcu_down_read_fast().
*/
static inline void srcu_up_read_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp)
__releases_shared(ssp)
{
WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi());
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN);
__srcu_read_unlock_fast_updown(ssp, scp);
RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_up_read_fast_updown().");
}
/**
* srcu_read_unlock_nmisafe - unregister a old reader from an SRCU-protected structure.
* @ssp: srcu_struct in which to unregister the old reader.
* @idx: return value from corresponding srcu_read_lock_nmisafe().
*
* Exit an SRCU read-side critical section, but in an NMI-safe manner.
*/
static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx)
__releases_shared(ssp)
{
WARN_ON_ONCE(idx & ~0x1);
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NMI);
rcu_lock_release(&ssp->dep_map);
__srcu_read_unlock_nmisafe(ssp, idx);
}
/* Used by tracing, cannot be traced and cannot call lockdep. */
static inline notrace void
srcu_read_unlock_notrace(struct srcu_struct *ssp, int idx) __releases_shared(ssp)
{
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
__srcu_read_unlock(ssp, idx);
}
/**
* srcu_up_read - unregister a old reader from an SRCU-protected structure.
* @ssp: srcu_struct in which to unregister the old reader.
* @idx: return value from corresponding srcu_read_lock().
*
* Exit an SRCU read-side critical section, but not necessarily from
* the same context as the maching srcu_down_read().
*/
static inline void srcu_up_read(struct srcu_struct *ssp, int idx)
__releases_shared(ssp)
{
WARN_ON_ONCE(idx & ~0x1);
WARN_ON_ONCE(in_nmi());
srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_NORMAL);
__srcu_read_unlock(ssp, idx);
}
/**
* smp_mb__after_srcu_read_unlock - ensure full ordering after srcu_read_unlock
*
* Converts the preceding srcu_read_unlock into a two-way memory barrier.
*
* Call this after srcu_read_unlock, to guarantee that all memory operations
* that occur after smp_mb__after_srcu_read_unlock will appear to happen after
* the preceding srcu_read_unlock.
*/
static inline void smp_mb__after_srcu_read_unlock(void)
{
/* __srcu_read_unlock has smp_mb() internally so nothing to do here. */
}
/**
* smp_mb__after_srcu_read_lock - ensure full ordering after srcu_read_lock
*
* Converts the preceding srcu_read_lock into a two-way memory barrier.
*
* Call this after srcu_read_lock, to guarantee that all memory operations
* that occur after smp_mb__after_srcu_read_lock will appear to happen after
* the preceding srcu_read_lock.
*/
static inline void smp_mb__after_srcu_read_lock(void)
{
/* __srcu_read_lock has smp_mb() internally so nothing to do here. */
}
DEFINE_LOCK_GUARD_1(srcu, struct srcu_struct,
_T->idx = srcu_read_lock(_T->lock),
srcu_read_unlock(_T->lock, _T->idx),
int idx)
DECLARE_LOCK_GUARD_1_ATTRS(srcu, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
#define class_srcu_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu, _T)
DEFINE_LOCK_GUARD_1(srcu_fast, struct srcu_struct,
_T->scp = srcu_read_lock_fast(_T->lock),
srcu_read_unlock_fast(_T->lock, _T->scp),
struct srcu_ctr __percpu *scp)
DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
#define class_srcu_fast_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast, _T)
DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct,
_T->scp = srcu_read_lock_fast_notrace(_T->lock),
srcu_read_unlock_fast_notrace(_T->lock, _T->scp),
struct srcu_ctr __percpu *scp)
DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
#define class_srcu_fast_notrace_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T)
#endif
]
|
| 47/2 |
2026/03/15 01:24 |
llm |
expert |
0m
Model:
Tokens:
input: 104832
output: 40
thoughts: 3617
Thoughts: **Analyzing BPF and SRCU Interactions in Tracepoints**
Okay, so I'm diving into this issue where BPF programs attached to tracepoints might be causing use-after-free (UAF) errors, specifically in the context of the `mm_page_alloc` tracepoint. The core of the problem seems to stem from how the kernel handles releasing resources associated with BPF links when tracepoints are involved.
My initial thought is that the function `guard(srcu_fast_notrace)` uses `srcu_read_lock_fast_notrace` in `__do_trace_##name`, and `__srcu_read_lock_fast` relies on `this_cpu_inc` as an RCU reader. But with `PREEMPT(full)` enabled, `this_cpu_inc` isn't an RCU reader, as `current->rcu_read_lock_nesting` is used to track readers, and `this_cpu_inc` doesn't increment that counter. This means a preemption could occur between `this_cpu_inc` and probe execution, potentially leading RCU to think there are no readers, and prematurely finishing the grace period.
However, the `guard` is held during `__DO_TRACE_CALL`, which is called inside the conditional `if (cond)`. If the read lock is held during `__DO_TRACE_CALL`, `synchronize_srcu` would wait for any outstanding readers. The problem arises because BPF uses `call_rcu`, not `synchronize_srcu`, and `tracepoint_srcu` is an `SRCU_READ_FLAVOR_FAST` SRCU. The grace period for this flavor uses `synchronize_rcu`, which does wait for all RCU readers.
The trick here is that `synchronize_rcu` waits for all RCU readers. If `srcu_read_lock_fast` readers are *not* RCU readers, then `synchronize_rcu` won't wait for them. This is the crux. In `PREEMPT(full)`, RCU readers are defined by `rcu_read_lock()` and `rcu_read_unlock()`, and `srcu_read_lock_fast` doesn't use those. I also see that `RCU_LOCKDEP_WARN(!rcu_is_watching(), ...)` is used in `srcu_read_lock_fast` which only checks if RCU is aware of the CPU, not whether the current context is in an RCU read-side critical section.
So, if `tracepoint_srcu` readers are NOT RCU readers, then `call_rcu` in BPF is definitely not sufficient. I found a recent kernel commit titled "bpf: Use SRCU for tracepoint protection" that is relevant here.
Looking deeper, the key is the `bpf_link_free` function, and how it handles different link types. If the tracepoint uses SRCU, the BPF link should use `call_srcu`. However, `bpf_link` is generic, so it doesn't know it's attached to a tracepoint that uses `tracepoint_srcu`. But `bpf_raw_tp_link` does. While `bpf_link_free` is the generic function, `bpf_raw_tp_link` uses `dealloc_deferred`. The core issue then, is whether `call_rcu` in `bpf_link_free` waits long enough for the SRCU grace period when used with tracepoints.
If the BPF link is sleepable or has a sleepable program, `call_rcu_tasks_trace` is used. If not, `call_rcu` is used, and it seems this is where the issue lies for tracepoint links. Regular tracepoints now use SRCU, meaning BPF is now potentially broken.
A recent patch series "tracepoint: Optimize tracepoints using SRCU" was merged, indicating that tracepoints are now using SRCU. Checking the kernel code, `DEFINE_SRCU_FAST(tracepoint_srcu)` and `guard(srcu_fast_notrace)(&tracepoint_srcu)` confirms this. However, `bpf_link_free` in `kernel/bpf/syscall.c` doesn't seem to have been updated to account for this change. The `bpf_raw_tp_link` struct now needs to wait for `tracepoint_srcu`'s grace period, but the `bpf_link_free` function in BPF uses `call_rcu` which doesn't know about SRCU.
Further analysis of `bpf_link_free` reveals it calls `ops->release
|
| 48/2 |
2026/03/15 01:25 |
tool |
codesearch-struct-layout |
0m
Args: map[ContextFile:kernel/bpf/syscall.c Name:bpf_raw_tp_link]
Results: map[Fields:[map[Name:link OffsetBits:0 SizeBits:960] map[Name:btp OffsetBits:960 SizeBits:64] map[Name:cookie OffsetBits:1024 SizeBits:64]]]
|
| 49/2 |
2026/03/15 01:25 |
llm |
expert |
0m
Model:
Tokens:
input: 109586
output: 36
thoughts: 57
Thoughts: **Analyzing the BPF Link Structure**
Okay, so I'm looking at this `link` variable, and I see it's starting at offset 0. That's good to know, makes things a bit simpler. Now, the `struct bpf_link` size β let's see, 960 bits, which translates to 120 bytes. That's the overall footprint we're dealing with. Now, the key thing I need to get at is `link->link.prog`. To access that member, I need to know the layout of the `struct bpf_link`. It will be necessary to know the internal composition of this structure to understand how to access that embedded field correctly, that `link.prog` field that I'm interested in!
|
| 50/2 |
2026/03/15 01:25 |
tool |
codesearch-struct-layout |
0m
Args: map[ContextFile:include/linux/bpf.h Name:bpf_link]
Results: map[Fields:[map[Name:refcnt OffsetBits:0 SizeBits:64] map[Name:id OffsetBits:64 SizeBits:32] map[Name:type OffsetBits:96 SizeBits:32] map[Name:ops OffsetBits:128 SizeBits:64] map[Name:prog OffsetBits:192 SizeBits:64] map[Name:flags OffsetBits:256 SizeBits:32] map[Name:attach_type OffsetBits:288 SizeBits:32] map[Name: OffsetBits:320 SizeBits:576] map[Name:sleepable OffsetBits:896 SizeBits:8]]]
|
| 51/2 |
2026/03/15 01:25 |
llm |
expert |
0m
Model:
Tokens:
input: 110030
output: 26
thoughts: 2017
Thoughts: **Analysis of a Use-After-Free Vulnerability in BPF Tracepoint Links**
Okay, so I'm looking at this report, and the clues are starting to align. The initial report places the problem: `prog` is at `OffsetBits: 192`, or 24 bytes from the start of a freed memory region. It's a textbook use-after-free, consistent with what I'm seeing. The report says, and I agree, that the problem stems from `bpf_raw_tp_link` being freed after an RCU grace period. But it's used by tracepoints, which, crucially, are protected by an SRCU grace period.
I have to verify this SRCU isn't just an RCU alias. Checking in `kernel/tracepoint.c`, I can see: `DEFINE_SRCU_FAST(tracepoint_srcu);`. Going further into the `include/linux/srcutree.h` header, it confirms this is a true SRCU. So it *is* a real SRCU.
Now, I need to understand if `synchronize_rcu` waits for SRCU readers. Digging into the SRCU implementation is tricky, so I need to go at this another way. But even if `synchronize_rcu` did wait for them, `call_rcu` might not, because they can use different grace periods. In fact, `call_rcu` and `synchronize_rcu` use the same grace period mechanism, which is interesting.
Hereβs the thing, in `PREEMPT(full)`, `synchronize_rcu` waits for tasks in an RCU critical section. If `srcu_read_lock_fast` doesn't use `rcu_read_lock()`, it *isn't* an RCU critical section in `PREEMPT(full)`. Let's check `include/linux/srcutree.h` again. Ah, here it is: the comment mentions the SRCU-fast mechanism uses `synchronize_rcu()` or `synchronize_rcu_expedited()`, which is RCU, not SRCU. That strongly suggests `synchronize_srcu()` calls `synchronize_rcu()`, but doesn't *guarantee* that `synchronize_rcu()` will wait for it.
The vital part is, if `synchronize_srcu()` calls `synchronize_rcu()`, it's because it *wants* to ensure all readers started before `synchronize_rcu()` have finished. And for that to work, the readers must be RCU readers. If they're *not* RCU readers (because `rcu_read_lock()` wasn't called), `synchronize_rcu()` won't wait.
Got it! Looking deeper in `include/linux/srcutree.h` I can see this: `this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader.` The comment notes it's an "implicit RCU reader." On many configurations, this is true because the operation is non-preemptible or atomic. But in `PREEMPT(full)`, it *is* preemptible and unless it's within an `rcu_read_lock()` it isn't a guaranteed RCU reader.
I think I've found it: a bug in SRCU-fast on `PREEMPT(full)`, or, more likely, BPF is misusing it. I'm leaning toward BPF misuse.
If `tracepoint_srcu` is used, BPF MUST wait for it with `synchronize_srcu(&tracepoint_srcu)` or `call_srcu(&tracepoint_srcu, ...)` because relying on that "implicit RCU reader" is dangerous and apparently failing here.
Is this actionable? Yes, absolutely. We have a clear trace: access stack (tracepoint execution), allocation stack (BPF link creation), free stack (BPF link destruction via RCU), and a mismatch between the protection mechanism (SRCU) and the freeing mechanism (RCU). The report is coherent and points to a real BPF/tracepoint integration bug.
One more thing: `udevd` is triggering `mm_page_alloc`, and that's a *very* common tracepoint. This suggests a pattern of frequent raw tracepoint opening/closing, or maybe just bad luck with allocations.
Now, should `bpf_link_free` be using `call_srcu` for *all* tracepoint links? The `bpf_link` struct doesn't have a direct pointer to `tracepoint_srcu`, but it's a global, so it's accessible. If
|
| 52/2 |
2026/03/15 01:25 |
tool |
read-file |
0m
Args: map[File:kernel/trace/trace_event_perf.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0
/*
* trace event based perf event profiling/tracing
*
* Copyright (C) 2009 Red Hat Inc, Peter Zijlstra
* Copyright (C) 2009-2010 Frederic Weisbecker <fweisbec@gmail.com>
*/
#include <linux/module.h>
#include <linux/kprobes.h>
#include <linux/security.h>
#include "trace.h"
#include "trace_probe.h"
static char __percpu *perf_trace_buf[PERF_NR_CONTEXTS];
/*
* Force it to be aligned to unsigned long to avoid misaligned accesses
* surprises
*/
typedef typeof(unsigned long [PERF_MAX_TRACE_SIZE / sizeof(unsigned long)])
perf_trace_t;
/* Count the events in use (per event id, not per instance) */
static int total_ref_count;
static int perf_trace_event_perm(struct trace_event_call *tp_event,
struct perf_event *p_event)
{
int ret;
if (tp_event->perf_perm) {
ret = tp_event->perf_perm(tp_event, p_event);
if (ret)
return ret;
}
/*
* We checked and allowed to create parent,
* allow children without checking.
*/
if (p_event->parent)
return 0;
/*
* It's ok to check current process (owner) permissions in here,
* because code below is called only via perf_event_open syscall.
*/
/* The ftrace function trace is allowed only for root. */
if (ftrace_event_is_function(tp_event)) {
ret = perf_allow_tracepoint();
if (ret)
return ret;
if (!is_sampling_event(p_event))
return 0;
/*
* We don't allow user space callchains for function trace
* event, due to issues with page faults while tracing page
* fault handler and its overall trickiness nature.
*/
if (!p_event->attr.exclude_callchain_user)
return -EINVAL;
/*
* Same reason to disable user stack dump as for user space
* callchains above.
*/
if (p_event->attr.sample_type & PERF_SAMPLE_STACK_USER)
return -EINVAL;
}
/* No tracing, just counting, so no obvious leak */
if (!(p_event->attr.sample_type & PERF_SAMPLE_RAW))
return 0;
/* Some events are ok to be traced by non-root users... */
if (p_event->attach_state == PERF_ATTACH_TASK) {
if (tp_event->flags & TRACE_EVENT_FL_CAP_ANY)
return 0;
}
/*
* ...otherwise raw tracepoint data can be a severe data leak,
* only allow root to have these.
*/
ret = perf_allow_tracepoint();
if (ret)
return ret;
return 0;
}
static int perf_trace_event_reg(struct trace_event_call *tp_event,
struct perf_event *p_event)
{
struct hlist_head __percpu *list;
int ret = -ENOMEM;
int cpu;
p_event->tp_event = tp_event;
if (tp_event->perf_refcount++ > 0)
return 0;
list = alloc_percpu(struct hlist_head);
if (!list)
goto fail;
for_each_possible_cpu(cpu)
INIT_HLIST_HEAD(per_cpu_ptr(list, cpu));
tp_event->perf_events = list;
if (!total_ref_count) {
char __percpu *buf;
int i;
for (i = 0; i < PERF_NR_CONTEXTS; i++) {
buf = (char __percpu *)alloc_percpu(perf_trace_t);
if (!buf)
goto fail;
perf_trace_buf[i] = buf;
}
}
ret = tp_event->class->reg(tp_event, TRACE_REG_PERF_REGISTER, NULL);
if (ret)
goto fail;
total_ref_count++;
return 0;
fail:
if (!total_ref_count) {
int i;
for (i = 0; i < PERF_NR_CONTEXTS; i++) {
free_percpu(perf_trace_buf[i]);
perf_trace_buf[i] = NULL;
}
}
if (!--tp_event->perf_refcount) {
free_percpu(tp_event->perf_events);
tp_event->perf_events = NULL;
}
return ret;
}
static void perf_trace_event_unreg(struct perf_event *p_event)
{
struct trace_event_call *tp_event = p_event->tp_event;
int i;
if (--tp_event->perf_refcount > 0)
return;
tp_event->class->reg(tp_event, TRACE_REG_PERF_UNREGISTER, NULL);
/*
* Ensure our callback won't be called anymore. The buffers
* will be freed after that.
*/
tracepoint_synchronize_unregister();
free_percpu(tp_event->perf_events);
tp_event->perf_events = NULL;
if (!--total_ref_count) {
for (i = 0; i < PERF_NR_CONTEXTS; i++) {
free_percpu(perf_trace_buf[i]);
perf_trace_buf[i] = NULL;
}
}
}
static int perf_trace_event_open(struct perf_event *p_event)
{
struct trace_event_call *tp_event = p_event->tp_event;
return tp_event->class->reg(tp_event, TRACE_REG_PERF_OPEN, p_event);
}
static void perf_trace_event_close(struct perf_event *p_event)
{
struct trace_event_call *tp_event = p_event->tp_event;
tp_event->class->reg(tp_event, TRACE_REG_PERF_CLOSE, p_event);
}
static int perf_trace_event_init(struct trace_event_call *tp_event,
struct perf_event *p_event)
{
int ret;
ret = perf_trace_event_perm(tp_event, p_event);
if (ret)
return ret;
ret = perf_trace_event_reg(tp_event, p_event);
if (ret)
return ret;
ret = perf_trace_event_open(p_event);
if (ret) {
perf_trace_event_unreg(p_event);
return ret;
}
return 0;
}
int perf_trace_init(struct perf_event *p_event)
{
struct trace_event_call *tp_event;
u64 event_id = p_event->attr.config;
int ret = -EINVAL;
mutex_lock(&event_mutex);
list_for_each_entry(tp_event, &ftrace_events, list) {
if (tp_event->event.type == event_id &&
tp_event->class && tp_event->class->reg &&
trace_event_try_get_ref(tp_event)) {
ret = perf_trace_event_init(tp_event, p_event);
if (ret)
trace_event_put_ref(tp_event);
break;
}
}
mutex_unlock(&event_mutex);
return ret;
}
void perf_trace_destroy(struct perf_event *p_event)
{
mutex_lock(&event_mutex);
perf_trace_event_close(p_event);
perf_trace_event_unreg(p_event);
trace_event_put_ref(p_event->tp_event);
mutex_unlock(&event_mutex);
}
#ifdef CONFIG_KPROBE_EVENTS
int perf_kprobe_init(struct perf_event *p_event, bool is_retprobe)
{
int ret;
char *func = NULL;
struct trace_event_call *tp_event;
if (p_event->attr.kprobe_func) {
func = strndup_user(u64_to_user_ptr(p_event->attr.kprobe_func),
KSYM_NAME_LEN);
if (IS_ERR(func)) {
ret = PTR_ERR(func);
return (ret == -EINVAL) ? -E2BIG : ret;
}
if (func[0] == '\0') {
kfree(func);
func = NULL;
}
}
tp_event = create_local_trace_kprobe(
func, (void *)(unsigned long)(p_event->attr.kprobe_addr),
p_event->attr.probe_offset, is_retprobe);
if (IS_ERR(tp_event)) {
ret = PTR_ERR(tp_event);
goto out;
}
mutex_lock(&event_mutex);
ret = perf_trace_event_init(tp_event, p_event);
if (ret)
destroy_local_trace_kprobe(tp_event);
mutex_unlock(&event_mutex);
out:
kfree(func);
return ret;
}
void perf_kprobe_destroy(struct perf_event *p_event)
{
mutex_lock(&event_mutex);
perf_trace_event_close(p_event);
perf_trace_event_unreg(p_event);
trace_event_put_ref(p_event->tp_event);
mutex_unlock(&event_mutex);
destroy_local_trace_kprobe(p_event->tp_event);
}
#endif /* CONFIG_KPROBE_EVENTS */
#ifdef CONFIG_UPROBE_EVENTS
int perf_uprobe_init(struct perf_event *p_event,
unsigned long ref_ctr_offset, bool is_retprobe)
{
int ret;
char *path = NULL;
struct trace_event_call *tp_event;
if (!p_event->attr.uprobe_path)
return -EINVAL;
path = strndup_user(u64_to_user_ptr(p_event->attr.uprobe_path),
PATH_MAX);
if (IS_ERR(path)) {
ret = PTR_ERR(path);
return (ret == -EINVAL) ? -E2BIG : ret;
}
if (path[0] == '\0') {
ret = -EINVAL;
goto out;
}
tp_event = create_local_trace_uprobe(path, p_event->attr.probe_offset,
ref_ctr_offset, is_retprobe);
if (IS_ERR(tp_event)) {
ret = PTR_ERR(tp_event);
goto out;
}
/*
* local trace_uprobe need to hold event_mutex to call
* uprobe_buffer_enable() and uprobe_buffer_disable().
* event_mutex is not required for local trace_kprobes.
*/
mutex_lock(&event_mutex);
ret = perf_trace_event_init(tp_event, p_event);
if (ret)
destroy_local_trace_uprobe(tp_event);
mutex_unlock(&event_mutex);
out:
kfree(path);
return ret;
}
void perf_uprobe_destroy(struct perf_event *p_event)
{
mutex_lock(&event_mutex);
perf_trace_event_close(p_event);
perf_trace_event_unreg(p_event);
trace_event_put_ref(p_event->tp_event);
mutex_unlock(&event_mutex);
destroy_local_trace_uprobe(p_event->tp_event);
}
#endif /* CONFIG_UPROBE_EVENTS */
int perf_trace_add(struct perf_event *p_event, int flags)
{
struct trace_event_call *tp_event = p_event->tp_event;
struct hw_perf_event *hwc = &p_event->hw;
if (!(flags & PERF_EF_START))
p_event->hw.state = PERF_HES_STOPPED;
if (is_sampling_event(p_event)) {
hwc->last_period = hwc->sample_period;
perf_swevent_set_period(p_event);
}
/*
* If TRACE_REG_PERF_ADD returns false; no custom action was performed
* and we need to take the default action of enqueueing our event on
* the right per-cpu hlist.
*/
if (!tp_event->class->reg(tp_event, TRACE_REG_PERF_ADD, p_event)) {
struct hlist_head __percpu *pcpu_list;
struct hlist_head *list;
pcpu_list = tp_event->perf_events;
if (WARN_ON_ONCE(!pcpu_list))
return -EINVAL;
list = this_cpu_ptr(pcpu_list);
hlist_add_head_rcu(&p_event->hlist_entry, list);
}
return 0;
}
void perf_trace_del(struct perf_event *p_event, int flags)
{
struct trace_event_call *tp_event = p_event->tp_event;
/*
* If TRACE_REG_PERF_DEL returns false; no custom action was performed
* and we need to take the default action of dequeueing our event from
* the right per-cpu hlist.
*/
if (!tp_event->class->reg(tp_event, TRACE_REG_PERF_DEL, p_event))
hlist_del_rcu(&p_event->hlist_entry);
}
void *perf_trace_buf_alloc(int size, struct pt_regs **regs, int *rctxp)
{
char *raw_data;
int rctx;
BUILD_BUG_ON(PERF_MAX_TRACE_SIZE % sizeof(unsigned long));
if (WARN_ONCE(size > PERF_MAX_TRACE_SIZE,
"perf buffer not large enough, wanted %d, have %d",
size, PERF_MAX_TRACE_SIZE))
return NULL;
*rctxp = rctx = perf_swevent_get_recursion_context();
if (rctx < 0)
return NULL;
if (regs)
*regs = this_cpu_ptr(&__perf_regs[rctx]);
raw_data = this_cpu_ptr(perf_trace_buf[rctx]);
/* zero the dead bytes from align to not leak stack to user */
memset(&raw_data[size - sizeof(u64)], 0, sizeof(u64));
return raw_data;
}
EXPORT_SYMBOL_GPL(perf_trace_buf_alloc);
NOKPROBE_SYMBOL(perf_trace_buf_alloc);
void perf_trace_buf_update(void *record, u16 type)
{
struct trace_entry *entry = record;
tracing_generic_entry_update(entry, type, tracing_gen_ctx());
}
NOKPROBE_SYMBOL(perf_trace_buf_update);
#ifdef CONFIG_FUNCTION_TRACER
static void
perf_ftrace_function_call(unsigned long ip, unsigned long parent_ip,
struct ftrace_ops *ops, struct ftrace_regs *fregs)
{
struct ftrace_entry *entry;
struct perf_event *event;
struct hlist_head head;
struct pt_regs regs;
int rctx;
int bit;
if (!rcu_is_watching())
return;
bit = ftrace_test_recursion_trylock(ip, parent_ip);
if (bit < 0)
return;
if ((unsigned long)ops->private != smp_processor_id())
goto out;
event = container_of(ops, struct perf_event, ftrace_ops);
/*
* @event->hlist entry is NULL (per INIT_HLIST_NODE), and all
* the perf code does is hlist_for_each_entry_rcu(), so we can
* get away with simply setting the @head.first pointer in order
* to create a singular list.
*/
head.first = &event->hlist_entry;
#define ENTRY_SIZE (ALIGN(sizeof(struct ftrace_entry) + sizeof(u32), \
sizeof(u64)) - sizeof(u32))
BUILD_BUG_ON(ENTRY_SIZE > PERF_MAX_TRACE_SIZE);
memset(®s, 0, sizeof(regs));
perf_fetch_caller_regs(®s);
entry = perf_trace_buf_alloc(ENTRY_SIZE, NULL, &rctx);
if (!entry)
goto out;
entry->ip = ip;
entry->parent_ip = parent_ip;
perf_trace_buf_submit(entry, ENTRY_SIZE, rctx, TRACE_FN,
1, ®s, &head, NULL);
out:
ftrace_test_recursion_unlock(bit);
#undef ENTRY_SIZE
}
static int perf_ftrace_function_register(struct perf_event *event)
{
struct ftrace_ops *ops = &event->ftrace_ops;
ops->func = perf_ftrace_function_call;
ops->private = (void *)(unsigned long)nr_cpu_ids;
return register_ftrace_function(ops);
}
static int perf_ftrace_function_unregister(struct perf_event *event)
{
struct ftrace_ops *ops = &event->ftrace_ops;
int ret = unregister_ftrace_function(ops);
ftrace_free_filter(ops);
return ret;
}
int perf_ftrace_event_register(struct trace_event_call *call,
enum trace_reg type, void *data)
{
struct perf_event *event = data;
switch (type) {
case TRACE_REG_REGISTER:
case TRACE_REG_UNREGISTER:
break;
case TRACE_REG_PERF_REGISTER:
case TRACE_REG_PERF_UNREGISTER:
return 0;
case TRACE_REG_PERF_OPEN:
return perf_ftrace_function_register(data);
case TRACE_REG_PERF_CLOSE:
return perf_ftrace_function_unregister(data);
case TRACE_REG_PERF_ADD:
event->ftrace_ops.private = (void *)(unsigned long)smp_processor_id();
return 1;
case TRACE_REG_PERF_DEL:
event->ftrace_ops.private = (void *)(unsigned long)nr_cpu_ids;
return 1;
}
return -EINVAL;
}
#endif /* CONFIG_FUNCTION_TRACER */
]
|
| 53/2 |
2026/03/15 01:25 |
llm |
expert |
0m
Model:
Tokens:
input: 117928
output: 19
thoughts: 1327
Thoughts: **Actionable Bug Report: UAF in BPF Tracepoint Link**
Okay, so here's what I've got. It looks like there's a serious use-after-free vulnerability in the interaction between BPF tracepoint links and the tracepoint infrastructure. I've broken it down step-by-step:
First, in the regular tracepoint unregistration, `perf_trace_event_unreg` calls `tracepoint_synchronize_unregister()`, which waits for `tracepoint_srcu`. This is good; it ensures safety.
However, `bpf_raw_tp_link`, which is used by BPF, skips that synchronization. It goes straight to `bpf_probe_unregister`, which *only* calls `tracepoint_probe_unregister`. This is the problem because it doesn't wait for a grace period.
I considered whether `bpf_probe_unregister` *should* call `tracepoint_synchronize_unregister()`, but that would block, and BPF links try to avoid blocking in release. This is where I started to realize this is a genuine bug.
I know that `bpf_link_free` is already deferred. My thinking is that if `bpf_link_free` used `call_srcu(&tracepoint_srcu, ...)` instead of `call_rcu`, then this would be safe and non-blocking. This is the key insight.
I've got a comprehensive picture of the problem:
1. We have a UAF on a `struct bpf_raw_tp_link`.
2. The UAF happens in `__bpf_trace_run`, which is called from a tracepoint.
3. Tracepoints are supposed to be protected by `tracepoint_srcu` (SRCU).
4. `bpf_raw_tp_link` is unregistered from the tracepoint in `bpf_raw_tp_link_release`.
5. `bpf_raw_tp_link` gets freed in `bpf_link_free` after an RCU grace period via `call_rcu`.
6. An RCU grace period doesn't wait for an SRCU grace period.
7. Therefore, the `link` object can be freed while a tracepoint is *still* executing the BPF probe.
8. The KASAN report is consistent: the access offset (24 bytes) matches the `prog` field in `struct bpf_link`, which is at the start of `struct bpf_raw_tp_link`.
This report is actionable because it provides all 3 call stacks, which allow for identifying the root cause. I can confirm this is a real problem.
Okay, let's consider the fix. `bpf_link_free` is called from `bpf_link_put_direct`. Now, if `bpf_link_put_direct` is called from a context that *can't* sleep, then we can't use `synchronize_srcu`. The answer is `call_srcu` which is always available and non-blocking.
So, the fix is to make sure `bpf_raw_tp_link` (and potentially other BPF links tied to tracepoints) waits for the `tracepoint_srcu` grace period before freeing.
I verified that `bpf_raw_tp_link` seems to be the only one using `bpf_probe_register`, which keeps the fix targeted.
A point to note: `bpf_link_free` is generic; it doesn't know about `tracepoint_srcu`. But we could make it know, or we could handle deferral inside `bpf_raw_tp_link`. That's an implementation detail.
After a final review, I can confirm that the report is actionable, and these are the specifics:
- BUG: KASAN: slab-use-after-free in `__bpf_trace_run`
- Read of size 8 at address `ffff8880361a0318` by task `udevd/5299`
- Access stack: `__bpf_trace_run` <- `bpf_trace_run4` <- `trace_mm_page_alloc`
- Allocation stack: `bpf_raw_tp_link_attach`
- Free stack: `kfree` <- `rcu_do_batch` <- `rcu_core`
- Last potentially related work creation: `call_rcu` <- `bpf_link_put_direct` <- `bpf_link_release` <- `__fput`
|
| 54/2 |
2026/03/15 01:25 |
tool |
set-results |
0m
Args: map[Actionable:true Confident:true]
Results: map[Actionable:true Confident:true]
|
| 55/2 |
2026/03/15 01:25 |
llm |
expert |
0m
Model:
Tokens:
input: 120315
output: 637
thoughts: 45
Thoughts: Okay, here's my take on those thoughts, framed from my own expert perspective:
**Analysis Complete: The Core Issue is Clear**
Right, so I've finished digging into this, and I'm feeling confident in the conclusions. The report itself is solid; the recommendations are definitely something we can act on, and it all hangs together logically. The core issue, as I've been saying, boils down to how tracepoints are interacting with the BPF link's memory management. Specifically, it's the incompatibility between the Synchronized Read-Copy-Update (SRCU) protection utilized by the tracepoints and the Read-Copy-Update (RCU)-based deferred deallocation approach used by the BPF links that's causing the problem. We've got a fundamental conflict in how these two mechanisms are handling and protecting memory, and that's the root of the instability. We need to figure out how to reconcile these.
|