| 0/0 |
2026/05/18 17:16 |
flow |
patching |
3h48m
Results: map[AckedBy:<nil> Fixes:map[Hash:48cac3f4a96ddf08df8e53809ed066de0dc93915 Title:bpf: Implement formatted output helpers with bstr_printf] KernelBranch:master KernelCommit:5200f5f493f79f14bbdc349e402a40dfb32f23c8 KernelRepo:git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git PatchDescription:bpf: Fix out-of-bounds write in bpf_bprintf_prepare()
The `bpf_bprintf_prepare()` function formats arguments into a temporary
512-byte buffer. There are two distinct bugs in this function that allow
`tmp_buf` to advance past `tmp_buf_end`, bypassing bounds checks and
causing out-of-bounds memory writes.
First, when handling the `%pB` and `%pI4`/`%pI6` format specifiers,
`snprintf()` is used to format the string. `snprintf()` returns the
number of characters that would have been written if the buffer was
large enough. If the remaining buffer space is smaller than the
formatted string, the return value will be greater than or equal to the
remaining space. The code previously advanced `tmp_buf` by `err + 1`,
which could push `tmp_buf` past `tmp_buf_end`.
Second, when handling standard format specifiers, the code aligns
`tmp_buf` and checks if there is enough space using `tmp_buf_end -
tmp_buf < sizeof_cur_arg`. Since `sizeof_cur_arg` is an unsigned
`size_t`, the signed `ptrdiff_t` result of the pointer subtraction is
implicitly cast to unsigned. If `tmp_buf` has already been advanced past
`tmp_buf_end`, the negative difference becomes a huge positive number,
bypassing the check and allowing further out-of-bounds writes.
Fix these issues by checking the return value of `snprintf()` for `%pB`
and `%pI4`/`%pI6`. If it indicates truncation or an error, return
`-ENOSPC` or `-EINVAL` respectively, ensuring the buffer is not overrun
and `bstr_printf()` does not read malformed data. Additionally, rewrite
the bounds checks to use pointer addition (`tmp_buf + sizeof_cur_arg >
tmp_buf_end`), which eliminates the integer signedness bypass, and
harden equality checks for buffer fullness from `tmp_buf_end == tmp_buf`
to `tmp_buf >= tmp_buf_end`.
This prevents memory corruption that can lead to soft lockups and other
crashes. PatchDiff:diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2bb60200c..0e3d3bddd 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -921,10 +921,22 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
+ err = -ENOSPC;
+ goto out;
+ }
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += (err + 1);
}
@@ -944,12 +956,13 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
- if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
+ sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
+
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
@@ -964,6 +977,14 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += err + 1;
num_spec++;
@@ -982,7 +1003,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1004,7 +1025,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1038,7 +1059,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
- if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
Recipients:[map[Email:andrii@kernel.org Name:Andrii Nakryiko To:false] map[Email:ast@kernel.org Name:Alexei Starovoitov To:false] map[Email:bpf@vger.kernel.org Name:open list:BPF [GENERAL] (Safe Dynamic Programs and Tools) To:false] map[Email:daniel@iogearbox.net Name:Daniel Borkmann To:false] map[Email:eddyz87@gmail.com Name:Eduard Zingerman To:false] map[Email:jolsa@kernel.org Name:Jiri Olsa To:false] map[Email:linux-kernel@vger.kernel.org Name: To:true] map[Email:martin.lau@linux.dev Name:Martin KaFai Lau To:false] map[Email:memxor@gmail.com Name:Kumar Kartikeya Dwivedi To:false] map[Email:song@kernel.org Name:Song Liu To:false] map[Email:yonghong.song@linux.dev Name:Yonghong Song To:false]] ReportedBy:<nil> ReviewedBy:<nil> TestedBy:<nil>]
|
| 1/1 |
2026/05/18 17:16 |
action |
base-commit-picker |
0m
Results: map[KernelBranch:master KernelCommit:5200f5f493f79f14bbdc349e402a40dfb32f23c8 KernelRepo:git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git]
|
| 2/1 |
2026/05/18 17:16 |
action |
syz-repro-to-c-repro |
0m
Results: map[SimplifiedCRepro:// autogenerated by syzkaller (https://github.com/google/syzkaller)
#define _GNU_SOURCE
#include <endian.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mount.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>
#ifndef __NR_bpf
#define __NR_bpf 321
#endif
#define BITMASK(bf_off,bf_len) (((1ull << (bf_len)) - 1) << (bf_off))
#define STORE_BY_BITMASK(type,htobe,addr,val,bf_off,bf_len) *(type*)(addr) = htobe((htobe(*(type*)(addr)) & ~BITMASK((bf_off), (bf_len))) | (((type)(val) << (bf_off)) & BITMASK((bf_off), (bf_len))))
uint64_t r[2] = {0xffffffffffffffff, 0xffffffffffffffff};
int main(void)
{
syscall(__NR_mmap, /*addr=*/0x1ffffffff000ul, /*len=*/0x1000ul, /*prot=*/0ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
syscall(__NR_mmap, /*addr=*/0x200000000000ul, /*len=*/0x1000000ul, /*prot=PROT_WRITE|PROT_READ|PROT_EXEC*/7ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
syscall(__NR_mmap, /*addr=*/0x200001000000ul, /*len=*/0x1000ul, /*prot=*/0ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
const char* reason;
(void)reason;
intptr_t res = 0;
if (write(1, "executing program\n", sizeof("executing program\n") - 1)) {}
// socket arguments: [
// domain: socket_domain = 0xa (8 bytes)
// type: socket_type = 0x5 (8 bytes)
// proto: int32 = 0x0 (4 bytes)
// ]
// returns sock
res = syscall(__NR_socket, /*domain=AF_INET6*/0xaul, /*type=SOCK_SEQPACKET*/5ul, /*proto=*/0);
if (res != -1)
r[0] = res;
// bpf$PROG_LOAD arguments: [
// cmd: const = 0x5 (8 bytes)
// arg: ptr[in, bpf_prog_t[flags[bpf_prog_type, int32], bpf_prog_attach_types, bpf_btf_id[opt], fd_bpf_prog[opt]]] {
// bpf_prog_t[flags[bpf_prog_type, int32], bpf_prog_attach_types, bpf_btf_id[opt], fd_bpf_prog[opt]] {
// type: bpf_prog_type = 0x5 (4 bytes)
// ninsn: bytesize8 = 0x2c (4 bytes)
// insns: ptr[in, bpf_instructions] {
// union bpf_instructions {
// ringbuf: bpf_program_ringbuf {
// initr0: bpf_insn_init_r0 {
// code: const = 0x18 (1 bytes)
// dst: const = 0x0 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: int32 = 0x2 (4 bytes)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: int32 = 0x20 (4 bytes)
// }
// reserve: bpf_insn_ringbuf_reserve {
// insn1: bpf_insn_map_fd_t[const[BPF_REG_1, int8:4], ringbuf_map_fd] {
// code: const = 0x18 (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x1 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: ringbuf_map_fd (resource)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
// insn2: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_2, int8:4], const[0, int8:4], const[0, int16], const[20, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x14 (4 bytes)
// }
// insn3: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_3, int8:4], const[0, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x3 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn4: bpf_insn_call_helper_t[const[BPF_FUNC_ringbuf_reserve, int32]] {
// code: const = 0x85 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// func: const = 0x83 (4 bytes)
// }
// insn5: bpf_insn_alu_t[BPF_ALU64, BPF_X0, BPF_MOV0, const[BPF_REG_9, int8:4], const[BPF_REG_0, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x1 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x9 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// }
// null_check: bpf_insn_null_check[BPF_REG_9] {
// cond_jump: bpf_insn_not_null_jmp[BPF_REG_9] {
// code_class: const = 0x5 (0 bytes)
// code_s: const = 0x0 (0 bytes)
// code_op: const = 0x5 (1 bytes)
// dst: const = 0x9 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x1 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// exit: bpf_insn_exit {
// code: const = 0x95 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// }
// body: array[bpf_insn] {
// union bpf_insn {
// snprintf: bpf_insn_snprintf {
// insn1: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_8, int8:4], const[0, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x8 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn2: bpf_insn_ldst_t[BPF_STX, BPF_DW0, BPF_MEM0, const[BPF_REG_10, int8:4], const[BPF_REG_8, int8:4], const[-8, int16], const[0, int32]] {
// code_class: int8 = 0x3 (0 bytes)
// code_size: int8 = 0x3 (0 bytes)
// code_mode: int8 = 0x3 (1 bytes)
// dst: const = 0xa (0 bytes)
// src: const = 0x8 (1 bytes)
// off: const = 0xfff8 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn3: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_8, int8:4], const[0, int8:4], const[0, int16], int32] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x8 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: int32 = 0x9 (4 bytes)
// }
// insn4: bpf_insn_ldst_t[BPF_STX, BPF_DW0, BPF_MEM0, const[BPF_REG_10, int8:4], const[BPF_REG_8, int8:4], const[-16, int16], const[0, int32]] {
// code_class: int8 = 0x3 (0 bytes)
// code_size: int8 = 0x3 (0 bytes)
// code_mode: int8 = 0x3 (1 bytes)
// dst: const = 0xa (0 bytes)
// src: const = 0x8 (1 bytes)
// off: const = 0xfff0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn5: bpf_insn_alu_t[BPF_ALU64, BPF_X0, BPF_MOV0, const[BPF_REG_1, int8:4], const[BPF_REG_10, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x1 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0xa (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn6: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_ADD0, const[BPF_REG_1, int8:4], const[0, int8:4], const[0, int16], const[-8, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0x0 (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0xfffffff8 (4 bytes)
// }
// insn7: bpf_insn_alu_t[BPF_ALU64, BPF_X0, BPF_MOV0, const[BPF_REG_4, int8:4], const[BPF_REG_10, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x1 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x4 (0 bytes)
// src: const = 0xa (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn8: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_ADD0, const[BPF_REG_4, int8:4], const[0, int8:4], const[0, int16], const[-16, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0x0 (1 bytes)
// dst: const = 0x4 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0xfffffff0 (4 bytes)
// }
// insn9: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_2, int8:4], const[0, int8:4], const[0, int16], const[8, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x8 (4 bytes)
// }
// insn10: bpf_insn_map_value_t[const[BPF_REG_3, int8:4], bpf_frozen_const_str, const[0, int32]] {
// code: const = 0x18 (1 bytes)
// dst: const = 0x3 (0 bytes)
// src: const = 0x2 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: bpf_frozen_const_str (resource)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
// insn11: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_5, int8:4], const[0, int8:4], const[0, int16], const[8, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x5 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x8 (4 bytes)
// }
// insn12: bpf_insn_call_helper_t[const[BPF_FUNC_snprintf, int32]] {
// code: const = 0x85 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// func: const = 0xa5 (4 bytes)
// }
// }
// }
// union bpf_insn {
// alu: bpf_insn_alu_t[bpf_alu_insn, bpf_alu_source, bpf_alu_op, flags[bpf_reg, int8:4], flags[bpf_reg, int8:4], flags[bpf_insn_offsets, int16], flags[bpf_insn_immediates, int32]] {
// code_class: bpf_alu_insn = 0x7 (0 bytes)
// code_s: bpf_alu_source = 0x1 (0 bytes)
// code_op: bpf_alu_op = 0x9 (1 bytes)
// dst: bpf_reg = 0x5 (0 bytes)
// src: bpf_reg = 0x2 (1 bytes)
// off: bpf_insn_offsets = 0xfffffffffffffffc (2 bytes)
// imm: bpf_insn_immediates = 0xfffffffffffffffc (4 bytes)
// }
// }
// union bpf_insn {
// cb_func: bpf_insn_cb_func {
// code: const = 0x18 (1 bytes)
// dst: bpf_reg = 0x4 (0 bytes)
// src: const = 0x4 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: int32 = 0x7 (4 bytes)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
// }
// union bpf_insn {
// ringbuf_query: bpf_insn_ringbuf_query {
// insn1: bpf_insn_map_fd_t[const[BPF_REG_1, int8:4], tail_call_map] {
// code: const = 0x18 (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x1 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: tail_call_map (resource)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
// insn2: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_2, int8:4], const[0, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn3: bpf_insn_call_helper_t[const[BPF_FUNC_ringbuf_query, int32]] {
// code: const = 0x85 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// func: const = 0x86 (4 bytes)
// }
// }
// }
// union bpf_insn {
// ringbuf_output: bpf_insn_ringbuf_output {
// insn1: bpf_insn_map_fd_t[const[BPF_REG_1, int8:4], tail_call_map] {
// code: const = 0x18 (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x1 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: tail_call_map (resource)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
// insn2: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_8, int8:4], const[0, int8:4], const[0, int16], int32] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x8 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: int32 = 0x10000 (4 bytes)
// }
// insn3: bpf_insn_ldst_t[BPF_STX, BPF_DW0, BPF_MEM0, const[BPF_REG_10, int8:4], const[BPF_REG_8, int8:4], const[-8, int16], const[0, int32]] {
// code_class: int8 = 0x3 (0 bytes)
// code_size: int8 = 0x3 (0 bytes)
// code_mode: int8 = 0x3 (1 bytes)
// dst: const = 0xa (0 bytes)
// src: const = 0x8 (1 bytes)
// off: const = 0xfff8 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn4: bpf_insn_alu_t[BPF_ALU64, BPF_X0, BPF_MOV0, const[BPF_REG_2, int8:4], const[BPF_REG_10, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x1 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0xa (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn5: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_ADD0, const[BPF_REG_2, int8:4], const[0, int8:4], const[0, int16], const[-8, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0x0 (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0xfffffff8 (4 bytes)
// }
// insn6: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_3, int8:4], const[0, int8:4], const[0, int16], const[8, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x3 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x8 (4 bytes)
// }
// insn7: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_4, int8:4], const[0, int8:4], const[0, int16], flags[bpf_ringbuf_wakeup_flags, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x4 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: bpf_ringbuf_wakeup_flags = 0x0 (4 bytes)
// }
// insn8: bpf_insn_call_helper_t[const[BPF_FUNC_ringbuf_output, int32]] {
// code: const = 0x85 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// func: const = 0x82 (4 bytes)
// }
// }
// }
// }
// free: bpf_insn_ringbuf_free {
// insn1: bpf_insn_alu_t[BPF_ALU64, BPF_X0, BPF_MOV0, const[BPF_REG_1, int8:4], const[BPF_REG_9, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x1 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x9 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn2: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_2, int8:4], const[0, int8:4], const[0, int16], flags[bpf_ringbuf_wakeup_flags, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: bpf_ringbuf_wakeup_flags = 0x1 (4 bytes)
// }
// insn3: bpf_insn_call_helper_t[flags[bpf_helpers_ringbuf_free, int32]] {
// code: const = 0x85 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// func: bpf_helpers_ringbuf_free = 0x85 (4 bytes)
// }
// insn4: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_0, int8:4], const[0, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x0 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// }
// exit: bpf_insn_exit {
// code: const = 0x95 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// }
// }
// }
// license: nil
// loglev: int32 = 0x8 (4 bytes)
// logsize: len = 0x0 (4 bytes)
// log: nil
// kern_version: bpf_kern_version = 0x41100 (4 bytes)
// flags: bpf_prog_load_flags = 0x2 (4 bytes)
// prog_name: buffer: {00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00} (length 0x10)
// prog_ifindex: ifindex (resource)
// expected_attach_type: union bpf_prog_attach_types {
// fallback: bpf_attach_types = 0x0 (4 bytes)
// }
// btf_fd: fd_btf (resource)
// func_info_rec_size: const = 0x8 (4 bytes)
// func_info: nil
// func_info_cnt: len = 0x0 (4 bytes)
// line_info_rec_size: const = 0x10 (4 bytes)
// line_info: nil
// line_info_cnt: len = 0x0 (4 bytes)
// attach_btf_id: bpf_btf_id (resource)
// attach_prog_fd: fd_bpf_prog (resource)
// core_relo_cnt: len = 0x0 (4 bytes)
// fd_array: nil
// core_relos: nil
// core_relo_rec_size: const = 0x10 (4 bytes)
// log_true_size: int32 = 0x1 (4 bytes)
// prog_token_fd: union _bpf_prog_t[flags[bpf_prog_type, int32], bpf_prog_attach_types, bpf_btf_id[opt], fd_bpf_prog[opt]]_prog_token_fd_wrapper {
// void: buffer: {} (length 0x0)
// }
// pad: union _bpf_prog_t[flags[bpf_prog_type, int32], bpf_prog_attach_types, bpf_btf_id[opt], fd_bpf_prog[opt]]_pad_wrapper {
// value: const = 0x0 (4 bytes)
// }
// }
// }
// size: len = 0x90 (8 bytes)
// ]
// returns fd_bpf_prog
*(uint32_t*)0x200000000c40 = 5;
*(uint32_t*)0x200000000c44 = 0x2c;
*(uint64_t*)0x200000000c48 = 0x200000000080;
*(uint8_t*)0x200000000080 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000081, 0, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000081, 0, 4, 4);
*(uint16_t*)0x200000000082 = 0;
*(uint32_t*)0x200000000084 = 2;
*(uint8_t*)0x200000000088 = 0;
*(uint8_t*)0x200000000089 = 0;
*(uint16_t*)0x20000000008a = 0;
*(uint32_t*)0x20000000008c = 0x20;
*(uint8_t*)0x200000000090 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000091, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000091, 1, 4, 4);
*(uint16_t*)0x200000000092 = 0;
*(uint32_t*)0x200000000094 = -1;
*(uint8_t*)0x200000000098 = 0;
*(uint8_t*)0x200000000099 = 0;
*(uint16_t*)0x20000000009a = 0;
*(uint32_t*)0x20000000009c = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000a0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a1, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a1, 0, 4, 4);
*(uint16_t*)0x2000000000a2 = 0;
*(uint32_t*)0x2000000000a4 = 0x14;
STORE_BY_BITMASK(uint8_t, , 0x2000000000a8, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a8, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a8, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a9, 3, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a9, 0, 4, 4);
*(uint16_t*)0x2000000000aa = 0;
*(uint32_t*)0x2000000000ac = 0;
*(uint8_t*)0x2000000000b0 = 0x85;
*(uint8_t*)0x2000000000b1 = 0;
*(uint16_t*)0x2000000000b2 = 0;
*(uint32_t*)0x2000000000b4 = 0x83;
STORE_BY_BITMASK(uint8_t, , 0x2000000000b8, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000b8, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000b8, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000b9, 9, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000b9, 0, 4, 4);
*(uint16_t*)0x2000000000ba = 0;
*(uint32_t*)0x2000000000bc = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000c0, 5, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000c0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000c0, 5, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000c1, 9, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000c1, 0, 4, 4);
*(uint16_t*)0x2000000000c2 = 1;
*(uint32_t*)0x2000000000c4 = 0;
*(uint8_t*)0x2000000000c8 = 0x95;
*(uint8_t*)0x2000000000c9 = 0;
*(uint16_t*)0x2000000000ca = 0;
*(uint32_t*)0x2000000000cc = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000d0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d1, 8, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d1, 0, 4, 4);
*(uint16_t*)0x2000000000d2 = 0;
*(uint32_t*)0x2000000000d4 = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000d8, 3, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d8, 3, 3, 2);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d8, 3, 5, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d9, 0xa, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d9, 8, 4, 4);
*(uint16_t*)0x2000000000da = 0xfff8;
*(uint32_t*)0x2000000000dc = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000e0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e1, 8, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e1, 0, 4, 4);
*(uint16_t*)0x2000000000e2 = 0;
*(uint32_t*)0x2000000000e4 = 9;
STORE_BY_BITMASK(uint8_t, , 0x2000000000e8, 3, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e8, 3, 3, 2);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e8, 3, 5, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e9, 0xa, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e9, 8, 4, 4);
*(uint16_t*)0x2000000000ea = 0xfff0;
*(uint32_t*)0x2000000000ec = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000f0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f0, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f1, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f1, 0xa, 4, 4);
*(uint16_t*)0x2000000000f2 = 0;
*(uint32_t*)0x2000000000f4 = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000f8, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f8, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f8, 0, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f9, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f9, 0, 4, 4);
*(uint16_t*)0x2000000000fa = 0;
*(uint32_t*)0x2000000000fc = 0xfffffff8;
STORE_BY_BITMASK(uint8_t, , 0x200000000100, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000100, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000100, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000101, 4, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000101, 0xa, 4, 4);
*(uint16_t*)0x200000000102 = 0;
*(uint32_t*)0x200000000104 = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000108, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000108, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000108, 0, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000109, 4, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000109, 0, 4, 4);
*(uint16_t*)0x20000000010a = 0;
*(uint32_t*)0x20000000010c = 0xfffffff0;
STORE_BY_BITMASK(uint8_t, , 0x200000000110, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000110, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000110, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000111, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000111, 0, 4, 4);
*(uint16_t*)0x200000000112 = 0;
*(uint32_t*)0x200000000114 = 8;
*(uint8_t*)0x200000000118 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000119, 3, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000119, 2, 4, 4);
*(uint16_t*)0x20000000011a = 0;
*(uint32_t*)0x20000000011c = -1;
*(uint8_t*)0x200000000120 = 0;
*(uint8_t*)0x200000000121 = 0;
*(uint16_t*)0x200000000122 = 0;
*(uint32_t*)0x200000000124 = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000128, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000128, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000128, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000129, 5, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000129, 0, 4, 4);
*(uint16_t*)0x20000000012a = 0;
*(uint32_t*)0x20000000012c = 8;
*(uint8_t*)0x200000000130 = 0x85;
*(uint8_t*)0x200000000131 = 0;
*(uint16_t*)0x200000000132 = 0;
*(uint32_t*)0x200000000134 = 0xa5;
STORE_BY_BITMASK(uint8_t, , 0x200000000138, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000138, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000138, 9, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000139, 5, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000139, 2, 4, 4);
*(uint16_t*)0x20000000013a = 0xfffc;
*(uint32_t*)0x20000000013c = 0xfffffffc;
*(uint8_t*)0x200000000140 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000141, 4, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000141, 4, 4, 4);
*(uint16_t*)0x200000000142 = 0;
*(uint32_t*)0x200000000144 = 7;
*(uint8_t*)0x200000000148 = 0;
*(uint8_t*)0x200000000149 = 0;
*(uint16_t*)0x20000000014a = 0;
*(uint32_t*)0x20000000014c = 0;
*(uint8_t*)0x200000000150 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000151, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000151, 1, 4, 4);
*(uint16_t*)0x200000000152 = 0;
*(uint32_t*)0x200000000154 = -1;
*(uint8_t*)0x200000000158 = 0;
*(uint8_t*)0x200000000159 = 0;
*(uint16_t*)0x20000000015a = 0;
*(uint32_t*)0x20000000015c = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000160, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000160, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000160, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000161, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000161, 0, 4, 4);
*(uint16_t*)0x200000000162 = 0;
*(uint32_t*)0x200000000164 = 0;
*(uint8_t*)0x200000000168 = 0x85;
*(uint8_t*)0x200000000169 = 0;
*(uint16_t*)0x20000000016a = 0;
*(uint32_t*)0x20000000016c = 0x86;
*(uint8_t*)0x200000000170 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000171, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000171, 1, 4, 4);
*(uint16_t*)0x200000000172 = 0;
*(uint32_t*)0x200000000174 = -1;
*(uint8_t*)0x200000000178 = 0;
*(uint8_t*)0x200000000179 = 0;
*(uint16_t*)0x20000000017a = 0;
*(uint32_t*)0x20000000017c = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000180, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000180, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000180, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000181, 8, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000181, 0, 4, 4);
*(uint16_t*)0x200000000182 = 0;
*(uint32_t*)0x200000000184 = 0x10000;
STORE_BY_BITMASK(uint8_t, , 0x200000000188, 3, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000188, 3, 3, 2);
STORE_BY_BITMASK(uint8_t, , 0x200000000188, 3, 5, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000189, 0xa, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000189, 8, 4, 4);
*(uint16_t*)0x20000000018a = 0xfff8;
*(uint32_t*)0x20000000018c = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000190, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000190, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000190, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000191, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000191, 0xa, 4, 4);
*(uint16_t*)0x200000000192 = 0;
*(uint32_t*)0x200000000194 = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000198, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000198, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000198, 0, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000199, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000199, 0, 4, 4);
*(uint16_t*)0x20000000019a = 0;
*(uint32_t*)0x20000000019c = 0xfffffff8;
STORE_BY_BITMASK(uint8_t, , 0x2000000001a0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a1, 3, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a1, 0, 4, 4);
*(uint16_t*)0x2000000001a2 = 0;
*(uint32_t*)0x2000000001a4 = 8;
STORE_BY_BITMASK(uint8_t, , 0x2000000001a8, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a8, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a8, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a9, 4, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a9, 0, 4, 4);
*(uint16_t*)0x2000000001aa = 0;
*(uint32_t*)0x2000000001ac = 0;
*(uint8_t*)0x2000000001b0 = 0x85;
*(uint8_t*)0x2000000001b1 = 0;
*(uint16_t*)0x2000000001b2 = 0;
*(uint32_t*)0x2000000001b4 = 0x82;
STORE_BY_BITMASK(uint8_t, , 0x2000000001b8, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000001b8, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000001b8, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001b9, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001b9, 9, 4, 4);
*(uint16_t*)0x2000000001ba = 0;
*(uint32_t*)0x2000000001bc = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000001c0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000001c0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000001c0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001c1, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001c1, 0, 4, 4);
*(uint16_t*)0x2000000001c2 = 0;
*(uint32_t*)0x2000000001c4 = 1;
*(uint8_t*)0x2000000001c8 = 0x85;
*(uint8_t*)0x2000000001c9 = 0;
*(uint16_t*)0x2000000001ca = 0;
*(uint32_t*)0x2000000001cc = 0x85;
STORE_BY_BITMASK(uint8_t, , 0x2000000001d0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000001d0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000001d0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001d1, 0, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001d1, 0, 4, 4);
*(uint16_t*)0x2000000001d2 = 0;
*(uint32_t*)0x2000000001d4 = 0;
*(uint8_t*)0x2000000001d8 = 0x95;
*(uint8_t*)0x2000000001d9 = 0;
*(uint16_t*)0x2000000001da = 0;
*(uint32_t*)0x2000000001dc = 0;
*(uint64_t*)0x200000000c50 = 0;
*(uint32_t*)0x200000000c58 = 8;
*(uint32_t*)0x200000000c5c = 0;
*(uint64_t*)0x200000000c60 = 0;
*(uint32_t*)0x200000000c68 = 0x41100;
*(uint32_t*)0x200000000c6c = 2;
memset((void*)0x200000000c70, 0, 16);
*(uint32_t*)0x200000000c80 = 0;
*(uint32_t*)0x200000000c84 = 0;
*(uint32_t*)0x200000000c88 = -1;
*(uint32_t*)0x200000000c8c = 8;
*(uint64_t*)0x200000000c90 = 0;
*(uint32_t*)0x200000000c98 = 0;
*(uint32_t*)0x200000000c9c = 0x10;
*(uint64_t*)0x200000000ca0 = 0;
*(uint32_t*)0x200000000ca8 = 0;
*(uint32_t*)0x200000000cac = 0;
*(uint32_t*)0x200000000cb0 = -1;
*(uint32_t*)0x200000000cb4 = 0;
*(uint64_t*)0x200000000cb8 = 0;
*(uint64_t*)0x200000000cc0 = 0;
*(uint32_t*)0x200000000cc8 = 0x10;
*(uint32_t*)0x200000000ccc = 1;
*(uint32_t*)0x200000000cd0 = 0;
syscall(__NR_bpf, /*cmd=*/5ul, /*arg=*/0x200000000c40ul, /*size=*/0x90ul);
// ioctl$ifreq_SIOCGIFINDEX_team arguments: [
// fd: sock (resource)
// cmd: const = 0x8933 (4 bytes)
// arg: ptr[out, ifreq_dev_t["team0", ifindex_team]] {
// ifreq_dev_t["team0", ifindex_team] {
// ifr_ifrn: buffer: {74 65 61 6d 30 00 00 00 00 00 00 00 00 00 00 00} (length 0x10)
// elem: ifindex_team (resource)
// pad = 0x0 (20 bytes)
// }
// }
// ]
memcpy((void*)0x200000000600, "team0\000\000\000\000\000\000\000\000\000\000\000", 16);
syscall(__NR_ioctl, /*fd=*/r[0], /*cmd=*/0x8933, /*arg=*/0x200000000600ul);
// socket$netlink arguments: [
// domain: const = 0x10 (8 bytes)
// type: const = 0x3 (8 bytes)
// proto: netlink_proto = 0x0 (4 bytes)
// ]
// returns sock_netlink
res = syscall(__NR_socket, /*domain=*/0x10ul, /*type=*/3ul, /*proto=*/0);
if (res != -1)
r[1] = res;
// sendmsg$nl_route_sched arguments: [
// fd: sock_nl_route (resource)
// msg: ptr[in, msghdr_netlink[netlink_msg_route_sched]] {
// msghdr_netlink[netlink_msg_route_sched] {
// addr: nil
// addrlen: len = 0x0 (4 bytes)
// pad = 0x0 (4 bytes)
// vec: ptr[in, iovec[in, netlink_msg_route_sched]] {
// iovec[in, netlink_msg_route_sched] {
// addr: ptr[in, netlink_msg_route_sched] {
// union netlink_msg_route_sched {
// newqdisc: netlink_msg_t[const[RTM_NEWQDISC, int16], tcmsg[AF_UNSPEC], rtm_tca_policy] {
// len: len = 0x0 (4 bytes)
// type: const = 0x24 (2 bytes)
// flags: netlink_msg_flags = 0x0 (2 bytes)
// seq: int32 = 0x0 (4 bytes)
// pid: int32 = 0x0 (4 bytes)
// payload: tcmsg[AF_UNSPEC] {
// family: const = 0x0 (1 bytes)
// tcm__pad1: const = 0x0 (1 bytes)
// tcm__pad2: const = 0x0 (2 bytes)
// ifindex: ifindex (resource)
// tcm_handle: tcm_handle {
// minor: tcm_handle_offsets = 0x0 (2 bytes)
// major: tcm_handle_offsets = 0x0 (2 bytes)
// }
// tcm_parent: tcm_handle {
// minor: tcm_handle_offsets = 0x0 (2 bytes)
// major: tcm_handle_offsets = 0x0 (2 bytes)
// }
// tcm_info: tcm_handle {
// minor: tcm_handle_offsets = 0x0 (2 bytes)
// major: tcm_handle_offsets = 0x0 (2 bytes)
// }
// }
// attrs: array[rtm_tca_policy] {
// }
// }
// }
// }
// len: len = 0xa4 (8 bytes)
// }
// }
// vlen: const = 0x1 (8 bytes)
// ctrl: const = 0x0 (8 bytes)
// ctrllen: const = 0x0 (8 bytes)
// f: send_flags = 0x0 (4 bytes)
// pad = 0x0 (4 bytes)
// }
// }
// f: send_flags = 0x0 (8 bytes)
// ]
*(uint64_t*)0x2000000007c0 = 0;
*(uint32_t*)0x2000000007c8 = 0;
*(uint64_t*)0x2000000007d0 = 0x200000000780;
*(uint64_t*)0x200000000780 = 0x200000000000;
*(uint32_t*)0x200000000000 = 0;
*(uint16_t*)0x200000000004 = 0x24;
*(uint16_t*)0x200000000006 = 0;
*(uint32_t*)0x200000000008 = 0;
*(uint32_t*)0x20000000000c = 0;
*(uint8_t*)0x200000000010 = 0;
*(uint8_t*)0x200000000011 = 0;
*(uint16_t*)0x200000000012 = 0;
*(uint32_t*)0x200000000014 = 0;
*(uint16_t*)0x200000000018 = 0;
*(uint16_t*)0x20000000001a = 0;
*(uint16_t*)0x20000000001c = 0;
*(uint16_t*)0x20000000001e = 0;
*(uint16_t*)0x200000000020 = 0;
*(uint16_t*)0x200000000022 = 0;
*(uint64_t*)0x200000000788 = 0xa4;
*(uint64_t*)0x2000000007d8 = 1;
*(uint64_t*)0x2000000007e0 = 0;
*(uint64_t*)0x2000000007e8 = 0;
*(uint32_t*)0x2000000007f0 = 0;
syscall(__NR_sendmsg, /*fd=*/r[1], /*msg=*/0x2000000007c0ul, /*f=*/0ul);
return 0;
}
]
|
| 3/1 |
2026/05/18 17:16 |
action |
kernel-checkouter |
1m
Results: map[KernelSrc:/app/workdir/cache/src/d5a02ed91a4f019017141f71e695c75b27c3612e]
|
| 4/1 |
2026/05/18 17:18 |
action |
kernel-builder |
29m
Results: map[KernelObj:/app/workdir/cache/build/cd54c62d8f8877071fb7eea687773c2acac9d846]
|
| 5/1 |
2026/05/18 17:47 |
action |
crash-reproducer |
28m
Results: map[OtherCrashReports:<nil> ReproducedBugTitle:BUG: soft lockup in smp_call_function ReproducedCrashReport:watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/u9:4:1179]
CPU#0 Utilization every 4000ms during lockup:
#1: 100% system, 1% softirq, 1% hardirq, 0% idle
#2: 100% system, 1% softirq, 0% hardirq, 0% idle
#3: 100% system, 1% softirq, 1% hardirq, 0% idle
#4: 100% system, 1% softirq, 1% hardirq, 0% idle
#5: 100% system, 1% softirq, 1% hardirq, 0% idle
Modules linked in:
irq event stamp: 386488
hardirqs last enabled at (386487): [<ffffffff8100148a>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
hardirqs last disabled at (386488): [<ffffffff8b9eb11e>] sysvec_apic_timer_interrupt+0xe/0xb0 arch/x86/kernel/apic/apic.c:1061
softirqs last enabled at (386486): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
softirqs last disabled at (386409): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
CPU: 0 UID: 0 PID: 1179 Comm: kworker/u9:4 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:342 [inline]
RIP: 0010:smp_call_function_many_cond+0xfcb/0x13b0 kernel/smp.c:892
Code: 79 45 8b 2e 44 89 ee 83 e6 01 31 ff e8 be c6 0b 00 41 83 e5 01 49 bd 00 00 00 00 00 fc ff df 75 07 e8 69 c2 0b 00 eb 37 f3 90 <43> 0f b6 04 2c 84 c0 75 10 41 f7 06 01 00 00 00 74 1e e8 4e c2 0b
RSP: 0018:ffffc90005fcf720 EFLAGS: 00000293
RAX: ffffffff81834d62 RBX: ffff88806943ad08 RCX: ffff88801d361c00
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90005fcf860 R08: ffffffff908e5af7 R09: 1ffffffff211cb5e
R10: dffffc0000000000 R11: fffffbfff211cb5f R12: 1ffff1100d2a7edd
R13: dffffc0000000000 R14: ffff88806953f6e8 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff8880d5abe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555578641490 CR3: 000000000eaf4000 CR4: 00000000000006f0
Call Trace:
<TASK>
on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1057
on_each_cpu include/linux/smp.h:72 [inline]
smp_text_poke_sync_each_cpu arch/x86/kernel/alternative.c:2773 [inline]
smp_text_poke_batch_finish+0x5e6/0x1090 arch/x86/kernel/alternative.c:2983
arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
static_key_enable_cpuslocked+0x128/0x240 kernel/jump_label.c:210
static_key_enable+0x1a/0x20 kernel/jump_label.c:223
toggle_allocation_gate+0xab/0x290 mm/kfence/core.c:906
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xad1/0x1740 kernel/workqueue.c:3397
worker_thread+0xa59/0xfc0 kernel/workqueue.c:3478
kthread+0x388/0x470 kernel/kthread.c:436
ret_from_fork+0x437/0x9d0 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:__remove_hrtimer+0x8e/0x3e0 kernel/time/hrtimer.c:1145
Code: ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d
RSP: 0018:ffffc900001b8d90 EFLAGS: 00000006
RAX: ffffffff817d161e RBX: ffff888069527f40 RCX: 0000000000010000
RDX: ffff8880160c8000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: dffffc0000000000 R08: ffffffff94f48153 R09: 1ffffffff29e902a
R10: dffffc0000000000 R11: fffffbfff29e902b R12: 0000000000000000
R13: 0000000000000001 R14: ffff888024042300 R15: ffff888069528000
FS: 0000000000000000(0000) GS:ffff8880d5bbe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f83db077470 CR3: 000000001f132000 CR4: 00000000000006f0
Call Trace:
<IRQ>
__run_hrtimer kernel/time/hrtimer.c:1910 [inline]
__hrtimer_run_queues+0x299/0xa10 kernel/time/hrtimer.c:1994
hrtimer_interrupt+0x44b/0x950 kernel/time/hrtimer.c:2113
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1050 [inline]
__sysvec_apic_timer_interrupt+0x102/0x420 arch/x86/kernel/apic/apic.c:1067
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1061 [inline]
sysvec_apic_timer_interrupt+0x8b/0xb0 arch/x86/kernel/apic/apic.c:1061
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:native_irq_disable arch/x86/include/asm/irqflags.h:37 [inline]
RIP: 0010:arch_local_irq_disable arch/x86/include/asm/irqflags.h:114 [inline]
RIP: 0010:default_idle+0xf/0x20 arch/x86/kernel/process.c:768
Code: 4c 01 c7 4c 29 c2 e9 6e ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 13 a4 23 00 fb f4 <fa> e9 8b 8d 02 00 cc 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
RSP: 0018:ffffc9000014fe28 EFLAGS: 00000202
RAX: 00000000006a8f01 RBX: ffffffff81654d3a RCX: 0000000080000001
RDX: 0000000000000001 RSI: ffffffff8e3ef513 RDI: ffffffff8c21df20
RBP: ffffc9000014ff20 R08: ffff888069532993 R09: 1ffff1100d2a6532
R10: dffffc0000000000 R11: ffffed100d2a6533 R12: 0000000000000001
R13: 1ffff11002c19000 R14: 0000000000000001 R15: 1ffff11002c19000
default_idle_call+0x70/0xb0 kernel/sched/idle.c:122
cpuidle_idle_call kernel/sched/idle.c:199 [inline]
do_idle+0x36a/0x5e0 kernel/sched/idle.c:352
cpu_startup_entry+0x43/0x60 kernel/sched/idle.c:451
start_secondary+0x101/0x110 arch/x86/kernel/smpboot.c:312
common_startup_64+0x12c/0x137
</TASK>
ReproducedFaultInjection:]
|
| 6/1 |
2026/05/18 18:15 |
action |
codesearch-prepare |
18m
Results: map[Index:codesearch-index]
|
| 7/1 |
2026/05/18 18:34 |
agent |
debugger |
34m
Model:
Instruction:
You are an experienced Linux kernel developer tasked with debugging a kernel crash root cause.
You need to provide a detailed explanation of the root cause for another developer to be
able to write a fix for the bug based on your explanation. Include all relevant details
into the response: function/struct/field/etc names, code snippets, line numbers,
macro/enum values, etc.
Prefer calling several tools at the same time to save round-trips.
Prompt:
The crash is:
watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/u9:4:1179]
CPU#0 Utilization every 4000ms during lockup:
#1: 100% system, 1% softirq, 1% hardirq, 0% idle
#2: 100% system, 1% softirq, 0% hardirq, 0% idle
#3: 100% system, 1% softirq, 1% hardirq, 0% idle
#4: 100% system, 1% softirq, 1% hardirq, 0% idle
#5: 100% system, 1% softirq, 1% hardirq, 0% idle
Modules linked in:
irq event stamp: 386488
hardirqs last enabled at (386487): [<ffffffff8100148a>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
hardirqs last disabled at (386488): [<ffffffff8b9eb11e>] sysvec_apic_timer_interrupt+0xe/0xb0 arch/x86/kernel/apic/apic.c:1061
softirqs last enabled at (386486): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
softirqs last disabled at (386409): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
CPU: 0 UID: 0 PID: 1179 Comm: kworker/u9:4 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:342 [inline]
RIP: 0010:smp_call_function_many_cond+0xfcb/0x13b0 kernel/smp.c:892
Code: 79 45 8b 2e 44 89 ee 83 e6 01 31 ff e8 be c6 0b 00 41 83 e5 01 49 bd 00 00 00 00 00 fc ff df 75 07 e8 69 c2 0b 00 eb 37 f3 90 <43> 0f b6 04 2c 84 c0 75 10 41 f7 06 01 00 00 00 74 1e e8 4e c2 0b
RSP: 0018:ffffc90005fcf720 EFLAGS: 00000293
RAX: ffffffff81834d62 RBX: ffff88806943ad08 RCX: ffff88801d361c00
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90005fcf860 R08: ffffffff908e5af7 R09: 1ffffffff211cb5e
R10: dffffc0000000000 R11: fffffbfff211cb5f R12: 1ffff1100d2a7edd
R13: dffffc0000000000 R14: ffff88806953f6e8 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff8880d5abe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555578641490 CR3: 000000000eaf4000 CR4: 00000000000006f0
Call Trace:
<TASK>
on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1057
on_each_cpu include/linux/smp.h:72 [inline]
smp_text_poke_sync_each_cpu arch/x86/kernel/alternative.c:2773 [inline]
smp_text_poke_batch_finish+0x5e6/0x1090 arch/x86/kernel/alternative.c:2983
arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
static_key_enable_cpuslocked+0x128/0x240 kernel/jump_label.c:210
static_key_enable+0x1a/0x20 kernel/jump_label.c:223
toggle_allocation_gate+0xab/0x290 mm/kfence/core.c:906
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xad1/0x1740 kernel/workqueue.c:3397
worker_thread+0xa59/0xfc0 kernel/workqueue.c:3478
kthread+0x388/0x470 kernel/kthread.c:436
ret_from_fork+0x437/0x9d0 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:__remove_hrtimer+0x8e/0x3e0 kernel/time/hrtimer.c:1145
Code: ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d
RSP: 0018:ffffc900001b8d90 EFLAGS: 00000006
RAX: ffffffff817d161e RBX: ffff888069527f40 RCX: 0000000000010000
RDX: ffff8880160c8000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: dffffc0000000000 R08: ffffffff94f48153 R09: 1ffffffff29e902a
R10: dffffc0000000000 R11: fffffbfff29e902b R12: 0000000000000000
R13: 0000000000000001 R14: ffff888024042300 R15: ffff888069528000
FS: 0000000000000000(0000) GS:ffff8880d5bbe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f83db077470 CR3: 000000001f132000 CR4: 00000000000006f0
Call Trace:
<IRQ>
__run_hrtimer kernel/time/hrtimer.c:1910 [inline]
__hrtimer_run_queues+0x299/0xa10 kernel/time/hrtimer.c:1994
hrtimer_interrupt+0x44b/0x950 kernel/time/hrtimer.c:2113
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1050 [inline]
__sysvec_apic_timer_interrupt+0x102/0x420 arch/x86/kernel/apic/apic.c:1067
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1061 [inline]
sysvec_apic_timer_interrupt+0x8b/0xb0 arch/x86/kernel/apic/apic.c:1061
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:native_irq_disable arch/x86/include/asm/irqflags.h:37 [inline]
RIP: 0010:arch_local_irq_disable arch/x86/include/asm/irqflags.h:114 [inline]
RIP: 0010:default_idle+0xf/0x20 arch/x86/kernel/process.c:768
Code: 4c 01 c7 4c 29 c2 e9 6e ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 13 a4 23 00 fb f4 <fa> e9 8b 8d 02 00 cc 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
RSP: 0018:ffffc9000014fe28 EFLAGS: 00000202
RAX: 00000000006a8f01 RBX: ffffffff81654d3a RCX: 0000000080000001
RDX: 0000000000000001 RSI: ffffffff8e3ef513 RDI: ffffffff8c21df20
RBP: ffffc9000014ff20 R08: ffff888069532993 R09: 1ffff1100d2a6532
R10: dffffc0000000000 R11: ffffed100d2a6533 R12: 0000000000000001
R13: 1ffff11002c19000 R14: 0000000000000001 R15: 1ffff11002c19000
default_idle_call+0x70/0xb0 kernel/sched/idle.c:122
cpuidle_idle_call kernel/sched/idle.c:199 [inline]
do_idle+0x36a/0x5e0 kernel/sched/idle.c:352
cpu_startup_entry+0x43/0x60 kernel/sched/idle.c:451
start_secondary+0x101/0x110 arch/x86/kernel/smpboot.c:312
common_startup_64+0x12c/0x137
</TASK>
The following C code is a draft of the vulnerable syscall sequence. Keep in mind that
it may lack the precise threading, sandboxing, and some arguments of a working reproducer:
// autogenerated by syzkaller (https://github.com/google/syzkaller)
#define _GNU_SOURCE
#include <endian.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mount.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>
#ifndef __NR_bpf
#define __NR_bpf 321
#endif
#define BITMASK(bf_off,bf_len) (((1ull << (bf_len)) - 1) << (bf_off))
#define STORE_BY_BITMASK(type,htobe,addr,val,bf_off,bf_len) *(type*)(addr) = htobe((htobe(*(type*)(addr)) & ~BITMASK((bf_off), (bf_len))) | (((type)(val) << (bf_off)) & BITMASK((bf_off), (bf_len))))
uint64_t r[2] = {0xffffffffffffffff, 0xffffffffffffffff};
int main(void)
{
syscall(__NR_mmap, /*addr=*/0x1ffffffff000ul, /*len=*/0x1000ul, /*prot=*/0ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
syscall(__NR_mmap, /*addr=*/0x200000000000ul, /*len=*/0x1000000ul, /*prot=PROT_WRITE|PROT_READ|PROT_EXEC*/7ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
syscall(__NR_mmap, /*addr=*/0x200001000000ul, /*len=*/0x1000ul, /*prot=*/0ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
const char* reason;
(void)reason;
intptr_t res = 0;
if (write(1, "executing program\n", sizeof("executing program\n") - 1)) {}
// socket arguments: [
// domain: socket_domain = 0xa (8 bytes)
// type: socket_type = 0x5 (8 bytes)
// proto: int32 = 0x0 (4 bytes)
// ]
// returns sock
res = syscall(__NR_socket, /*domain=AF_INET6*/0xaul, /*type=SOCK_SEQPACKET*/5ul, /*proto=*/0);
if (res != -1)
r[0] = res;
// bpf$PROG_LOAD arguments: [
// cmd: const = 0x5 (8 bytes)
// arg: ptr[in, bpf_prog_t[flags[bpf_prog_type, int32], bpf_prog_attach_types, bpf_btf_id[opt], fd_bpf_prog[opt]]] {
// bpf_prog_t[flags[bpf_prog_type, int32], bpf_prog_attach_types, bpf_btf_id[opt], fd_bpf_prog[opt]] {
// type: bpf_prog_type = 0x5 (4 bytes)
// ninsn: bytesize8 = 0x2c (4 bytes)
// insns: ptr[in, bpf_instructions] {
// union bpf_instructions {
// ringbuf: bpf_program_ringbuf {
// initr0: bpf_insn_init_r0 {
// code: const = 0x18 (1 bytes)
// dst: const = 0x0 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: int32 = 0x2 (4 bytes)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: int32 = 0x20 (4 bytes)
// }
// reserve: bpf_insn_ringbuf_reserve {
// insn1: bpf_insn_map_fd_t[const[BPF_REG_1, int8:4], ringbuf_map_fd] {
// code: const = 0x18 (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x1 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: ringbuf_map_fd (resource)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
// insn2: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_2, int8:4], const[0, int8:4], const[0, int16], const[20, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x14 (4 bytes)
// }
// insn3: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_3, int8:4], const[0, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x3 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn4: bpf_insn_call_helper_t[const[BPF_FUNC_ringbuf_reserve, int32]] {
// code: const = 0x85 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// func: const = 0x83 (4 bytes)
// }
// insn5: bpf_insn_alu_t[BPF_ALU64, BPF_X0, BPF_MOV0, const[BPF_REG_9, int8:4], const[BPF_REG_0, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x1 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x9 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// }
// null_check: bpf_insn_null_check[BPF_REG_9] {
// cond_jump: bpf_insn_not_null_jmp[BPF_REG_9] {
// code_class: const = 0x5 (0 bytes)
// code_s: const = 0x0 (0 bytes)
// code_op: const = 0x5 (1 bytes)
// dst: const = 0x9 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x1 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// exit: bpf_insn_exit {
// code: const = 0x95 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// }
// body: array[bpf_insn] {
// union bpf_insn {
// snprintf: bpf_insn_snprintf {
// insn1: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_8, int8:4], const[0, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x8 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn2: bpf_insn_ldst_t[BPF_STX, BPF_DW0, BPF_MEM0, const[BPF_REG_10, int8:4], const[BPF_REG_8, int8:4], const[-8, int16], const[0, int32]] {
// code_class: int8 = 0x3 (0 bytes)
// code_size: int8 = 0x3 (0 bytes)
// code_mode: int8 = 0x3 (1 bytes)
// dst: const = 0xa (0 bytes)
// src: const = 0x8 (1 bytes)
// off: const = 0xfff8 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn3: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_8, int8:4], const[0, int8:4], const[0, int16], int32] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x8 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: int32 = 0x9 (4 bytes)
// }
// insn4: bpf_insn_ldst_t[BPF_STX, BPF_DW0, BPF_MEM0, const[BPF_REG_10, int8:4], const[BPF_REG_8, int8:4], const[-16, int16], const[0, int32]] {
// code_class: int8 = 0x3 (0 bytes)
// code_size: int8 = 0x3 (0 bytes)
// code_mode: int8 = 0x3 (1 bytes)
// dst: const = 0xa (0 bytes)
// src: const = 0x8 (1 bytes)
// off: const = 0xfff0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn5: bpf_insn_alu_t[BPF_ALU64, BPF_X0, BPF_MOV0, const[BPF_REG_1, int8:4], const[BPF_REG_10, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x1 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0xa (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn6: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_ADD0, const[BPF_REG_1, int8:4], const[0, int8:4], const[0, int16], const[-8, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0x0 (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0xfffffff8 (4 bytes)
// }
// insn7: bpf_insn_alu_t[BPF_ALU64, BPF_X0, BPF_MOV0, const[BPF_REG_4, int8:4], const[BPF_REG_10, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x1 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x4 (0 bytes)
// src: const = 0xa (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn8: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_ADD0, const[BPF_REG_4, int8:4], const[0, int8:4], const[0, int16], const[-16, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0x0 (1 bytes)
// dst: const = 0x4 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0xfffffff0 (4 bytes)
// }
// insn9: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_2, int8:4], const[0, int8:4], const[0, int16], const[8, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x8 (4 bytes)
// }
// insn10: bpf_insn_map_value_t[const[BPF_REG_3, int8:4], bpf_frozen_const_str, const[0, int32]] {
// code: const = 0x18 (1 bytes)
// dst: const = 0x3 (0 bytes)
// src: const = 0x2 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: bpf_frozen_const_str (resource)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
// insn11: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_5, int8:4], const[0, int8:4], const[0, int16], const[8, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x5 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x8 (4 bytes)
// }
// insn12: bpf_insn_call_helper_t[const[BPF_FUNC_snprintf, int32]] {
// code: const = 0x85 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// func: const = 0xa5 (4 bytes)
// }
// }
// }
// union bpf_insn {
// alu: bpf_insn_alu_t[bpf_alu_insn, bpf_alu_source, bpf_alu_op, flags[bpf_reg, int8:4], flags[bpf_reg, int8:4], flags[bpf_insn_offsets, int16], flags[bpf_insn_immediates, int32]] {
// code_class: bpf_alu_insn = 0x7 (0 bytes)
// code_s: bpf_alu_source = 0x1 (0 bytes)
// code_op: bpf_alu_op = 0x9 (1 bytes)
// dst: bpf_reg = 0x5 (0 bytes)
// src: bpf_reg = 0x2 (1 bytes)
// off: bpf_insn_offsets = 0xfffffffffffffffc (2 bytes)
// imm: bpf_insn_immediates = 0xfffffffffffffffc (4 bytes)
// }
// }
// union bpf_insn {
// cb_func: bpf_insn_cb_func {
// code: const = 0x18 (1 bytes)
// dst: bpf_reg = 0x4 (0 bytes)
// src: const = 0x4 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: int32 = 0x7 (4 bytes)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
// }
// union bpf_insn {
// ringbuf_query: bpf_insn_ringbuf_query {
// insn1: bpf_insn_map_fd_t[const[BPF_REG_1, int8:4], tail_call_map] {
// code: const = 0x18 (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x1 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: tail_call_map (resource)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
// insn2: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_2, int8:4], const[0, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn3: bpf_insn_call_helper_t[const[BPF_FUNC_ringbuf_query, int32]] {
// code: const = 0x85 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// func: const = 0x86 (4 bytes)
// }
// }
// }
// union bpf_insn {
// ringbuf_output: bpf_insn_ringbuf_output {
// insn1: bpf_insn_map_fd_t[const[BPF_REG_1, int8:4], tail_call_map] {
// code: const = 0x18 (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x1 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: tail_call_map (resource)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
// insn2: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_8, int8:4], const[0, int8:4], const[0, int16], int32] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x8 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: int32 = 0x10000 (4 bytes)
// }
// insn3: bpf_insn_ldst_t[BPF_STX, BPF_DW0, BPF_MEM0, const[BPF_REG_10, int8:4], const[BPF_REG_8, int8:4], const[-8, int16], const[0, int32]] {
// code_class: int8 = 0x3 (0 bytes)
// code_size: int8 = 0x3 (0 bytes)
// code_mode: int8 = 0x3 (1 bytes)
// dst: const = 0xa (0 bytes)
// src: const = 0x8 (1 bytes)
// off: const = 0xfff8 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn4: bpf_insn_alu_t[BPF_ALU64, BPF_X0, BPF_MOV0, const[BPF_REG_2, int8:4], const[BPF_REG_10, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x1 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0xa (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn5: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_ADD0, const[BPF_REG_2, int8:4], const[0, int8:4], const[0, int16], const[-8, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0x0 (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0xfffffff8 (4 bytes)
// }
// insn6: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_3, int8:4], const[0, int8:4], const[0, int16], const[8, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x3 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x8 (4 bytes)
// }
// insn7: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_4, int8:4], const[0, int8:4], const[0, int16], flags[bpf_ringbuf_wakeup_flags, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x4 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: bpf_ringbuf_wakeup_flags = 0x0 (4 bytes)
// }
// insn8: bpf_insn_call_helper_t[const[BPF_FUNC_ringbuf_output, int32]] {
// code: const = 0x85 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// func: const = 0x82 (4 bytes)
// }
// }
// }
// }
// free: bpf_insn_ringbuf_free {
// insn1: bpf_insn_alu_t[BPF_ALU64, BPF_X0, BPF_MOV0, const[BPF_REG_1, int8:4], const[BPF_REG_9, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x1 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x9 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// insn2: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_2, int8:4], const[0, int8:4], const[0, int16], flags[bpf_ringbuf_wakeup_flags, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x2 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: bpf_ringbuf_wakeup_flags = 0x1 (4 bytes)
// }
// insn3: bpf_insn_call_helper_t[flags[bpf_helpers_ringbuf_free, int32]] {
// code: const = 0x85 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// func: bpf_helpers_ringbuf_free = 0x85 (4 bytes)
// }
// insn4: bpf_insn_alu_t[BPF_ALU64, BPF_K0, BPF_MOV0, const[BPF_REG_0, int8:4], const[0, int8:4], const[0, int16], const[0, int32]] {
// code_class: int8 = 0x7 (0 bytes)
// code_s: int8 = 0x0 (0 bytes)
// code_op: int8 = 0xb (1 bytes)
// dst: const = 0x0 (0 bytes)
// src: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// }
// exit: bpf_insn_exit {
// code: const = 0x95 (1 bytes)
// regs: const = 0x0 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: const = 0x0 (4 bytes)
// }
// }
// }
// }
// license: nil
// loglev: int32 = 0x8 (4 bytes)
// logsize: len = 0x0 (4 bytes)
// log: nil
// kern_version: bpf_kern_version = 0x41100 (4 bytes)
// flags: bpf_prog_load_flags = 0x2 (4 bytes)
// prog_name: buffer: {00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00} (length 0x10)
// prog_ifindex: ifindex (resource)
// expected_attach_type: union bpf_prog_attach_types {
// fallback: bpf_attach_types = 0x0 (4 bytes)
// }
// btf_fd: fd_btf (resource)
// func_info_rec_size: const = 0x8 (4 bytes)
// func_info: nil
// func_info_cnt: len = 0x0 (4 bytes)
// line_info_rec_size: const = 0x10 (4 bytes)
// line_info: nil
// line_info_cnt: len = 0x0 (4 bytes)
// attach_btf_id: bpf_btf_id (resource)
// attach_prog_fd: fd_bpf_prog (resource)
// core_relo_cnt: len = 0x0 (4 bytes)
// fd_array: nil
// core_relos: nil
// core_relo_rec_size: const = 0x10 (4 bytes)
// log_true_size: int32 = 0x1 (4 bytes)
// prog_token_fd: union _bpf_prog_t[flags[bpf_prog_type, int32], bpf_prog_attach_types, bpf_btf_id[opt], fd_bpf_prog[opt]]_prog_token_fd_wrapper {
// void: buffer: {} (length 0x0)
// }
// pad: union _bpf_prog_t[flags[bpf_prog_type, int32], bpf_prog_attach_types, bpf_btf_id[opt], fd_bpf_prog[opt]]_pad_wrapper {
// value: const = 0x0 (4 bytes)
// }
// }
// }
// size: len = 0x90 (8 bytes)
// ]
// returns fd_bpf_prog
*(uint32_t*)0x200000000c40 = 5;
*(uint32_t*)0x200000000c44 = 0x2c;
*(uint64_t*)0x200000000c48 = 0x200000000080;
*(uint8_t*)0x200000000080 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000081, 0, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000081, 0, 4, 4);
*(uint16_t*)0x200000000082 = 0;
*(uint32_t*)0x200000000084 = 2;
*(uint8_t*)0x200000000088 = 0;
*(uint8_t*)0x200000000089 = 0;
*(uint16_t*)0x20000000008a = 0;
*(uint32_t*)0x20000000008c = 0x20;
*(uint8_t*)0x200000000090 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000091, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000091, 1, 4, 4);
*(uint16_t*)0x200000000092 = 0;
*(uint32_t*)0x200000000094 = -1;
*(uint8_t*)0x200000000098 = 0;
*(uint8_t*)0x200000000099 = 0;
*(uint16_t*)0x20000000009a = 0;
*(uint32_t*)0x20000000009c = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000a0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a1, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a1, 0, 4, 4);
*(uint16_t*)0x2000000000a2 = 0;
*(uint32_t*)0x2000000000a4 = 0x14;
STORE_BY_BITMASK(uint8_t, , 0x2000000000a8, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a8, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a8, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a9, 3, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000a9, 0, 4, 4);
*(uint16_t*)0x2000000000aa = 0;
*(uint32_t*)0x2000000000ac = 0;
*(uint8_t*)0x2000000000b0 = 0x85;
*(uint8_t*)0x2000000000b1 = 0;
*(uint16_t*)0x2000000000b2 = 0;
*(uint32_t*)0x2000000000b4 = 0x83;
STORE_BY_BITMASK(uint8_t, , 0x2000000000b8, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000b8, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000b8, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000b9, 9, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000b9, 0, 4, 4);
*(uint16_t*)0x2000000000ba = 0;
*(uint32_t*)0x2000000000bc = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000c0, 5, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000c0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000c0, 5, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000c1, 9, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000c1, 0, 4, 4);
*(uint16_t*)0x2000000000c2 = 1;
*(uint32_t*)0x2000000000c4 = 0;
*(uint8_t*)0x2000000000c8 = 0x95;
*(uint8_t*)0x2000000000c9 = 0;
*(uint16_t*)0x2000000000ca = 0;
*(uint32_t*)0x2000000000cc = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000d0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d1, 8, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d1, 0, 4, 4);
*(uint16_t*)0x2000000000d2 = 0;
*(uint32_t*)0x2000000000d4 = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000d8, 3, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d8, 3, 3, 2);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d8, 3, 5, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d9, 0xa, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000d9, 8, 4, 4);
*(uint16_t*)0x2000000000da = 0xfff8;
*(uint32_t*)0x2000000000dc = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000e0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e1, 8, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e1, 0, 4, 4);
*(uint16_t*)0x2000000000e2 = 0;
*(uint32_t*)0x2000000000e4 = 9;
STORE_BY_BITMASK(uint8_t, , 0x2000000000e8, 3, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e8, 3, 3, 2);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e8, 3, 5, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e9, 0xa, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000e9, 8, 4, 4);
*(uint16_t*)0x2000000000ea = 0xfff0;
*(uint32_t*)0x2000000000ec = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000f0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f0, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f1, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f1, 0xa, 4, 4);
*(uint16_t*)0x2000000000f2 = 0;
*(uint32_t*)0x2000000000f4 = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000000f8, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f8, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f8, 0, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f9, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000000f9, 0, 4, 4);
*(uint16_t*)0x2000000000fa = 0;
*(uint32_t*)0x2000000000fc = 0xfffffff8;
STORE_BY_BITMASK(uint8_t, , 0x200000000100, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000100, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000100, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000101, 4, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000101, 0xa, 4, 4);
*(uint16_t*)0x200000000102 = 0;
*(uint32_t*)0x200000000104 = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000108, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000108, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000108, 0, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000109, 4, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000109, 0, 4, 4);
*(uint16_t*)0x20000000010a = 0;
*(uint32_t*)0x20000000010c = 0xfffffff0;
STORE_BY_BITMASK(uint8_t, , 0x200000000110, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000110, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000110, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000111, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000111, 0, 4, 4);
*(uint16_t*)0x200000000112 = 0;
*(uint32_t*)0x200000000114 = 8;
*(uint8_t*)0x200000000118 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000119, 3, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000119, 2, 4, 4);
*(uint16_t*)0x20000000011a = 0;
*(uint32_t*)0x20000000011c = -1;
*(uint8_t*)0x200000000120 = 0;
*(uint8_t*)0x200000000121 = 0;
*(uint16_t*)0x200000000122 = 0;
*(uint32_t*)0x200000000124 = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000128, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000128, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000128, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000129, 5, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000129, 0, 4, 4);
*(uint16_t*)0x20000000012a = 0;
*(uint32_t*)0x20000000012c = 8;
*(uint8_t*)0x200000000130 = 0x85;
*(uint8_t*)0x200000000131 = 0;
*(uint16_t*)0x200000000132 = 0;
*(uint32_t*)0x200000000134 = 0xa5;
STORE_BY_BITMASK(uint8_t, , 0x200000000138, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000138, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000138, 9, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000139, 5, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000139, 2, 4, 4);
*(uint16_t*)0x20000000013a = 0xfffc;
*(uint32_t*)0x20000000013c = 0xfffffffc;
*(uint8_t*)0x200000000140 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000141, 4, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000141, 4, 4, 4);
*(uint16_t*)0x200000000142 = 0;
*(uint32_t*)0x200000000144 = 7;
*(uint8_t*)0x200000000148 = 0;
*(uint8_t*)0x200000000149 = 0;
*(uint16_t*)0x20000000014a = 0;
*(uint32_t*)0x20000000014c = 0;
*(uint8_t*)0x200000000150 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000151, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000151, 1, 4, 4);
*(uint16_t*)0x200000000152 = 0;
*(uint32_t*)0x200000000154 = -1;
*(uint8_t*)0x200000000158 = 0;
*(uint8_t*)0x200000000159 = 0;
*(uint16_t*)0x20000000015a = 0;
*(uint32_t*)0x20000000015c = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000160, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000160, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000160, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000161, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000161, 0, 4, 4);
*(uint16_t*)0x200000000162 = 0;
*(uint32_t*)0x200000000164 = 0;
*(uint8_t*)0x200000000168 = 0x85;
*(uint8_t*)0x200000000169 = 0;
*(uint16_t*)0x20000000016a = 0;
*(uint32_t*)0x20000000016c = 0x86;
*(uint8_t*)0x200000000170 = 0x18;
STORE_BY_BITMASK(uint8_t, , 0x200000000171, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000171, 1, 4, 4);
*(uint16_t*)0x200000000172 = 0;
*(uint32_t*)0x200000000174 = -1;
*(uint8_t*)0x200000000178 = 0;
*(uint8_t*)0x200000000179 = 0;
*(uint16_t*)0x20000000017a = 0;
*(uint32_t*)0x20000000017c = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000180, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000180, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000180, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000181, 8, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000181, 0, 4, 4);
*(uint16_t*)0x200000000182 = 0;
*(uint32_t*)0x200000000184 = 0x10000;
STORE_BY_BITMASK(uint8_t, , 0x200000000188, 3, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000188, 3, 3, 2);
STORE_BY_BITMASK(uint8_t, , 0x200000000188, 3, 5, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000189, 0xa, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000189, 8, 4, 4);
*(uint16_t*)0x20000000018a = 0xfff8;
*(uint32_t*)0x20000000018c = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000190, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000190, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000190, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000191, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000191, 0xa, 4, 4);
*(uint16_t*)0x200000000192 = 0;
*(uint32_t*)0x200000000194 = 0;
STORE_BY_BITMASK(uint8_t, , 0x200000000198, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x200000000198, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x200000000198, 0, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000199, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x200000000199, 0, 4, 4);
*(uint16_t*)0x20000000019a = 0;
*(uint32_t*)0x20000000019c = 0xfffffff8;
STORE_BY_BITMASK(uint8_t, , 0x2000000001a0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a1, 3, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a1, 0, 4, 4);
*(uint16_t*)0x2000000001a2 = 0;
*(uint32_t*)0x2000000001a4 = 8;
STORE_BY_BITMASK(uint8_t, , 0x2000000001a8, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a8, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a8, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a9, 4, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001a9, 0, 4, 4);
*(uint16_t*)0x2000000001aa = 0;
*(uint32_t*)0x2000000001ac = 0;
*(uint8_t*)0x2000000001b0 = 0x85;
*(uint8_t*)0x2000000001b1 = 0;
*(uint16_t*)0x2000000001b2 = 0;
*(uint32_t*)0x2000000001b4 = 0x82;
STORE_BY_BITMASK(uint8_t, , 0x2000000001b8, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000001b8, 1, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000001b8, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001b9, 1, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001b9, 9, 4, 4);
*(uint16_t*)0x2000000001ba = 0;
*(uint32_t*)0x2000000001bc = 0;
STORE_BY_BITMASK(uint8_t, , 0x2000000001c0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000001c0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000001c0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001c1, 2, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001c1, 0, 4, 4);
*(uint16_t*)0x2000000001c2 = 0;
*(uint32_t*)0x2000000001c4 = 1;
*(uint8_t*)0x2000000001c8 = 0x85;
*(uint8_t*)0x2000000001c9 = 0;
*(uint16_t*)0x2000000001ca = 0;
*(uint32_t*)0x2000000001cc = 0x85;
STORE_BY_BITMASK(uint8_t, , 0x2000000001d0, 7, 0, 3);
STORE_BY_BITMASK(uint8_t, , 0x2000000001d0, 0, 3, 1);
STORE_BY_BITMASK(uint8_t, , 0x2000000001d0, 0xb, 4, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001d1, 0, 0, 4);
STORE_BY_BITMASK(uint8_t, , 0x2000000001d1, 0, 4, 4);
*(uint16_t*)0x2000000001d2 = 0;
*(uint32_t*)0x2000000001d4 = 0;
*(uint8_t*)0x2000000001d8 = 0x95;
*(uint8_t*)0x2000000001d9 = 0;
*(uint16_t*)0x2000000001da = 0;
*(uint32_t*)0x2000000001dc = 0;
*(uint64_t*)0x200000000c50 = 0;
*(uint32_t*)0x200000000c58 = 8;
*(uint32_t*)0x200000000c5c = 0;
*(uint64_t*)0x200000000c60 = 0;
*(uint32_t*)0x200000000c68 = 0x41100;
*(uint32_t*)0x200000000c6c = 2;
memset((void*)0x200000000c70, 0, 16);
*(uint32_t*)0x200000000c80 = 0;
*(uint32_t*)0x200000000c84 = 0;
*(uint32_t*)0x200000000c88 = -1;
*(uint32_t*)0x200000000c8c = 8;
*(uint64_t*)0x200000000c90 = 0;
*(uint32_t*)0x200000000c98 = 0;
*(uint32_t*)0x200000000c9c = 0x10;
*(uint64_t*)0x200000000ca0 = 0;
*(uint32_t*)0x200000000ca8 = 0;
*(uint32_t*)0x200000000cac = 0;
*(uint32_t*)0x200000000cb0 = -1;
*(uint32_t*)0x200000000cb4 = 0;
*(uint64_t*)0x200000000cb8 = 0;
*(uint64_t*)0x200000000cc0 = 0;
*(uint32_t*)0x200000000cc8 = 0x10;
*(uint32_t*)0x200000000ccc = 1;
*(uint32_t*)0x200000000cd0 = 0;
syscall(__NR_bpf, /*cmd=*/5ul, /*arg=*/0x200000000c40ul, /*size=*/0x90ul);
// ioctl$ifreq_SIOCGIFINDEX_team arguments: [
// fd: sock (resource)
// cmd: const = 0x8933 (4 bytes)
// arg: ptr[out, ifreq_dev_t["team0", ifindex_team]] {
// ifreq_dev_t["team0", ifindex_team] {
// ifr_ifrn: buffer: {74 65 61 6d 30 00 00 00 00 00 00 00 00 00 00 00} (length 0x10)
// elem: ifindex_team (resource)
// pad = 0x0 (20 bytes)
// }
// }
// ]
memcpy((void*)0x200000000600, "team0\000\000\000\000\000\000\000\000\000\000\000", 16);
syscall(__NR_ioctl, /*fd=*/r[0], /*cmd=*/0x8933, /*arg=*/0x200000000600ul);
// socket$netlink arguments: [
// domain: const = 0x10 (8 bytes)
// type: const = 0x3 (8 bytes)
// proto: netlink_proto = 0x0 (4 bytes)
// ]
// returns sock_netlink
res = syscall(__NR_socket, /*domain=*/0x10ul, /*type=*/3ul, /*proto=*/0);
if (res != -1)
r[1] = res;
// sendmsg$nl_route_sched arguments: [
// fd: sock_nl_route (resource)
// msg: ptr[in, msghdr_netlink[netlink_msg_route_sched]] {
// msghdr_netlink[netlink_msg_route_sched] {
// addr: nil
// addrlen: len = 0x0 (4 bytes)
// pad = 0x0 (4 bytes)
// vec: ptr[in, iovec[in, netlink_msg_route_sched]] {
// iovec[in, netlink_msg_route_sched] {
// addr: ptr[in, netlink_msg_route_sched] {
// union netlink_msg_route_sched {
// newqdisc: netlink_msg_t[const[RTM_NEWQDISC, int16], tcmsg[AF_UNSPEC], rtm_tca_policy] {
// len: len = 0x0 (4 bytes)
// type: const = 0x24 (2 bytes)
// flags: netlink_msg_flags = 0x0 (2 bytes)
// seq: int32 = 0x0 (4 bytes)
// pid: int32 = 0x0 (4 bytes)
// payload: tcmsg[AF_UNSPEC] {
// family: const = 0x0 (1 bytes)
// tcm__pad1: const = 0x0 (1 bytes)
// tcm__pad2: const = 0x0 (2 bytes)
// ifindex: ifindex (resource)
// tcm_handle: tcm_handle {
// minor: tcm_handle_offsets = 0x0 (2 bytes)
// major: tcm_handle_offsets = 0x0 (2 bytes)
// }
// tcm_parent: tcm_handle {
// minor: tcm_handle_offsets = 0x0 (2 bytes)
// major: tcm_handle_offsets = 0x0 (2 bytes)
// }
// tcm_info: tcm_handle {
// minor: tcm_handle_offsets = 0x0 (2 bytes)
// major: tcm_handle_offsets = 0x0 (2 bytes)
// }
// }
// attrs: array[rtm_tca_policy] {
// }
// }
// }
// }
// len: len = 0xa4 (8 bytes)
// }
// }
// vlen: const = 0x1 (8 bytes)
// ctrl: const = 0x0 (8 bytes)
// ctrllen: const = 0x0 (8 bytes)
// f: send_flags = 0x0 (4 bytes)
// pad = 0x0 (4 bytes)
// }
// }
// f: send_flags = 0x0 (8 bytes)
// ]
*(uint64_t*)0x2000000007c0 = 0;
*(uint32_t*)0x2000000007c8 = 0;
*(uint64_t*)0x2000000007d0 = 0x200000000780;
*(uint64_t*)0x200000000780 = 0x200000000000;
*(uint32_t*)0x200000000000 = 0;
*(uint16_t*)0x200000000004 = 0x24;
*(uint16_t*)0x200000000006 = 0;
*(uint32_t*)0x200000000008 = 0;
*(uint32_t*)0x20000000000c = 0;
*(uint8_t*)0x200000000010 = 0;
*(uint8_t*)0x200000000011 = 0;
*(uint16_t*)0x200000000012 = 0;
*(uint32_t*)0x200000000014 = 0;
*(uint16_t*)0x200000000018 = 0;
*(uint16_t*)0x20000000001a = 0;
*(uint16_t*)0x20000000001c = 0;
*(uint16_t*)0x20000000001e = 0;
*(uint16_t*)0x200000000020 = 0;
*(uint16_t*)0x200000000022 = 0;
*(uint64_t*)0x200000000788 = 0xa4;
*(uint64_t*)0x2000000007d8 = 1;
*(uint64_t*)0x2000000007e0 = 0;
*(uint64_t*)0x2000000007e8 = 0;
*(uint32_t*)0x2000000007f0 = 0;
syscall(__NR_sendmsg, /*fd=*/r[1], /*msg=*/0x2000000007c0ul, /*f=*/0ul);
return 0;
}
Reply: The root cause of this crash is a massive out-of-bounds memory write (buffer overflow) in the BPF helper `bpf_snprintf` (and other `bprintf`-like helpers) due to integer signedness and unchecked return values in `bpf_bprintf_prepare()`.
This memory corruption overwrites adjacent kernel memory, which in this case included `hrtimer` structures and their `timerqueue` rbtree nodes. When CPU 1 later attempted to remove a corrupted timer via `__remove_hrtimer` -> `timerqueue_linked_del` -> `rb_erase_linked`, it got stuck in an infinite loop traversing a cyclic rbtree. CPU 0 then detected a soft lockup because it was waiting for CPU 1 to acknowledge an IPI (`csd_lock_wait`), but CPU 1 had interrupts disabled while processing the timer queue.
### Detailed Analysis
In `kernel/bpf/helpers.c`, the `bpf_bprintf_prepare()` function formats arguments into a temporary 512-byte buffer (`buffers->bin_args`). There are two distinct bugs in this function that allow `tmp_buf` to advance past `tmp_buf_end`, bypassing bounds checks.
#### Bug 1: Unchecked `snprintf` return value for `%pB`
When handling the `%pB` format specifier, the code does:
```c
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
```
`snprintf` returns the number of characters that *would* have been written if the buffer was large enough, excluding the null terminator. If the remaining buffer space (`tmp_buf_end - tmp_buf`) is smaller than the formatted string, `err` will be greater than or equal to the remaining space. Consequently, `tmp_buf += (err + 1)` will advance `tmp_buf` **past** `tmp_buf_end`.
#### Bug 2: Signedness bug in bounds checking
When handling standard format specifiers (like `%d`, `%x`), the code aligns `tmp_buf` and checks if there is enough space:
```c
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
```
Here, `tmp_buf_end - tmp_buf` is of type `ptrdiff_t` (signed), and `sizeof_cur_arg` is of type `size_t` (unsigned). In C, this comparison forces the signed `ptrdiff_t` to be cast to an unsigned `size_t`.
If `tmp_buf` has already been advanced past `tmp_buf_end` (e.g., via the `%pB` bug above, or by using `%c` to make `tmp_buf` unaligned such that `PTR_ALIGN` pushes it past `tmp_buf_end`), `tmp_buf_end - tmp_buf` becomes negative. When cast to `size_t`, this negative value becomes a huge positive number (e.g., `0xFFFFFFFFFFFFFFF0`).
The condition `HUGE_NUMBER < sizeof_cur_arg` evaluates to `false`, bypassing the `-ENOSPC` check. The code then writes to `tmp_buf`, corrupting memory out of bounds.
Furthermore, if a `%s` specifier is used after `tmp_buf` is out of bounds:
```c
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
```
The equality check `tmp_buf_end == tmp_buf` fails to catch that `tmp_buf > tmp_buf_end`. Then `bpf_trace_copy_string` is called with a negative `bufsz` (`tmp_buf_end - tmp_buf`), which is cast to a huge `size_t`. This results in copying an arbitrarily large string into kernel memory, causing massive corruption.
### Fix
To fix this vulnerability:
1. For `%pB`, `tmp_buf` should only be advanced by the actual number of bytes written, and it must not exceed `tmp_buf_end`.
2. The bounds checks must correctly handle the case where `tmp_buf >= tmp_buf_end`. A safe way is to cast `tmp_buf_end - tmp_buf` to a signed type and check if it's less than the required size, or simply check `if (tmp_buf + sizeof_cur_arg > tmp_buf_end)`.
|
| 8/2 |
2026/05/18 18:34 |
llm |
debugger |
0m
Model:
Tokens:
input: 26842
output: 42
thoughts: 899
Thoughts: **Analyzing a Soft Lockup on CPU#0**
Okay, so I've got a soft lockup crash on CPU#0, and the stack trace is pointing towards `csd_lock_wait` within `smp_call_function_many_cond`. This is interesting; it's suggesting we're hitting a contention point related to inter-processor communication, potentially a lock. The trace goes through `smp_text_poke_batch_finish`, `static_key_enable`, and eventually to `toggle_allocation_gate` within the KFENCE code. That `static_key_enable` call is a potential area to investigate - could a key be getting enabled/disabled in a way that's causing this contention? And the call to `toggle_allocation_gate`… that's directly related to KFENCE, so definitely worth a closer look, especially if we're seeing issues around memory allocation or deallocation. Finally it traces into the kernel workqueues. I should look into what that workqueue is doing.
But wait, CPU 1 is also stuck. Now, the second stack trace is showing CPU 1 stuck in `__remove_hrtimer`! That's concerning, because it implies there's a problem with high-resolution timers. `__remove_hrtimer` suggests a timer is being removed or canceled, but it's getting hung up. It seems to be happening within the interrupt context of the APIC timer. That `native_irq_disable` call is also telling. It makes me wonder if there's an interrupt disabling issue combined with hrtimer management, creating a deadlock or a race condition. It's odd that CPU1 is in the interrupt handler and CPU0 is doing the workqueue work, so that looks like it's a potential area of contention. Let's look up how `__remove_hrtimer` behaves in the kernel source.
|
| 9/2 |
2026/05/18 18:34 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/time/hrtimer.c IncludeLines:true Name:__remove_hrtimer]
Results: map[SourceCode:1131: /*
1132: * __remove_hrtimer - internal function to remove a timer
1133: *
1134: * High resolution timer mode reprograms the clock event device when the
1135: * timer is the one which expires next. The caller can disable this by setting
1136: * reprogram to zero. This is useful, when the context does a reprogramming
1137: * anyway (e.g. timer interrupt)
1138: */
1139: static void __remove_hrtimer(struct hrtimer *timer, struct hrtimer_clock_base *base,
1140: bool newstate, bool reprogram)
1141: {
1142: struct hrtimer_cpu_base *cpu_base = base->cpu_base;
1143: bool was_first;
1144:
1145: lockdep_assert_held(&cpu_base->lock);
1146:
1147: if (!timer->is_queued)
1148: return;
1149:
1150: /* Pairs with the lockless read in hrtimer_is_queued() */
1151: WRITE_ONCE(timer->is_queued, newstate);
1152:
1153: was_first = !timerqueue_linked_prev(&timer->node);
1154:
1155: if (!timerqueue_linked_del(&base->active, &timer->node))
1156: cpu_base->active_bases &= ~(1 << base->index);
1157:
1158: /* Nothing to update if this was not the first timer in the base */
1159: if (!was_first)
1160: return;
1161:
1162: base_update_next_timer(base);
1163:
1164: /*
1165: * If reprogram is false don't update cpu_base->next_timer and do not
1166: * touch the clock event device.
1167: *
1168: * This happens when removing the first timer on a remote CPU, which
1169: * will be handled by the remote CPU's interrupt. It also happens when
1170: * a local timer is removed to be immediately restarted. That's handled
1171: * at the call site.
1172: */
1173: if (!reprogram || timer != cpu_base->next_timer || timer->is_lazy)
1174: return;
1175:
1176: if (cpu_base->deferred_rearm)
1177: cpu_base->deferred_needs_update = true;
1178: else
1179: hrtimer_force_reprogram(cpu_base, /* skip_equal */ true);
1180: }
SourceFile:kernel/time/hrtimer.c]
|
| 10/2 |
2026/05/18 18:34 |
llm |
debugger |
0m
Model:
Tokens:
input: 28981
output: 42
thoughts: 267
Thoughts: **Analyzing a Kernel Crash in `__remove_hrtimer`**
Okay, so I'm looking at this kernel crash, and the trace points me directly to `__remove_hrtimer` at line 1145. I see a few key checks there that could be at play: a `lockdep_assert_held` check on `cpu_base->lock`, a check for `timer->is_queued`, and then a `WRITE_ONCE` to update `timer->is_queued`, followed by a check for `was_first`. I'm immediately trying to connect the dots between these conditions and the crash.
Now, that assembly dump is what I really need to dig into. Let's see... looking at the code, I see the crucial instructions highlighted: `45 85 ed 0f 84 33 02 00 00`. I need to figure out what that sequence is doing - the registers and memory accesses are the key here. It's likely involved with conditional jumps based on comparisons, but I need to decipher the exact logic it is implementing, and relate that to the C code snippet.
I should also jump into `__run_hrtimer` to understand how the timer state could be related to the code.
|
| 11/2 |
2026/05/18 18:34 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/time/hrtimer.c IncludeLines:true Name:__run_hrtimer]
Results: map[SourceCode:1871: /*
1872: * The write_seqcount_barrier()s in __run_hrtimer() split the thing into 3
1873: * distinct sections:
1874: *
1875: * - queued: the timer is queued
1876: * - callback: the timer is being ran
1877: * - post: the timer is inactive or (re)queued
1878: *
1879: * On the read side we ensure we observe timer->is_queued and cpu_base->running
1880: * from the same section, if anything changed while we looked at it, we retry.
1881: * This includes timer->base changing because sequence numbers alone are
1882: * insufficient for that.
1883: *
1884: * The sequence numbers are required because otherwise we could still observe
1885: * a false negative if the read side got smeared over multiple consecutive
1886: * __run_hrtimer() invocations.
1887: */
1888: static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base, struct hrtimer_clock_base *base,
1889: struct hrtimer *timer, ktime_t now, unsigned long flags)
1890: __must_hold(&cpu_base->lock)
1891: {
1892: enum hrtimer_restart (*fn)(struct hrtimer *);
1893: bool expires_in_hardirq;
1894: int restart;
1895:
1896: lockdep_assert_held(&cpu_base->lock);
1897:
1898: debug_hrtimer_deactivate(timer);
1899: base->running = timer;
1900:
1901: /*
1902: * Separate the ->running assignment from the ->is_queued assignment.
1903: *
1904: * As with a regular write barrier, this ensures the read side in
1905: * hrtimer_active() cannot observe base->running == NULL &&
1906: * timer->is_queued == INACTIVE.
1907: */
1908: raw_write_seqcount_barrier(&base->seq);
1909:
1910: __remove_hrtimer(timer, base, HRTIMER_STATE_INACTIVE, false);
1911: fn = ACCESS_PRIVATE(timer, function);
1912:
1913: /*
1914: * Clear the 'is relative' flag for the TIME_LOW_RES case. If the
1915: * timer is restarted with a period then it becomes an absolute
1916: * timer. If its not restarted it does not matter.
1917: */
1918: if (IS_ENABLED(CONFIG_TIME_LOW_RES))
1919: timer->is_rel = false;
1920:
1921: /*
1922: * The timer is marked as running in the CPU base, so it is
1923: * protected against migration to a different CPU even if the lock
1924: * is dropped.
1925: */
1926: raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
1927: trace_hrtimer_expire_entry(timer, now);
1928: expires_in_hardirq = lockdep_hrtimer_enter(timer);
1929:
1930: restart = fn(timer);
1931:
1932: lockdep_hrtimer_exit(expires_in_hardirq);
1933: trace_hrtimer_expire_exit(timer);
1934: raw_spin_lock_irq(&cpu_base->lock);
1935:
1936: /*
1937: * Note: We clear the running state after enqueue_hrtimer and
1938: * we do not reprogram the event hardware. Happens either in
1939: * hrtimer_start_range_ns() or in hrtimer_interrupt()
1940: *
1941: * Note: Because we dropped the cpu_base->lock above,
1942: * hrtimer_start_range_ns() can have popped in and enqueued the timer
1943: * for us already.
1944: */
1945: if (restart == HRTIMER_RESTART && !timer->is_queued)
1946: enqueue_hrtimer(timer, base, HRTIMER_MODE_ABS, false);
1947:
1948: /*
1949: * Separate the ->running assignment from the ->is_queued assignment.
1950: *
1951: * As with a regular write barrier, this ensures the read side in
1952: * hrtimer_active() cannot observe base->running.timer == NULL &&
1953: * timer->is_queued == INACTIVE.
1954: */
1955: raw_write_seqcount_barrier(&base->seq);
1956:
1957: WARN_ON_ONCE(base->running != timer);
1958: base->running = NULL;
1959: }
SourceFile:kernel/time/hrtimer.c]
|
| 12/2 |
2026/05/18 18:34 |
llm |
debugger |
0m
Model:
Tokens:
input: 31001
output: 19
thoughts: 641
Thoughts: **Analyzing a Potential Soft Lockup**
Okay, so I'm seeing a potential soft lockup issue, and it looks like CPU 0 is the primary suspect. It's stuck in `csd_lock_wait`, which is called within a chain of functions: `smp_call_function_many_cond`, `on_each_cpu_cond_mask`, `smp_text_poke_sync_each_cpu`, and finally `smp_text_poke_batch_finish`. Right, `smp_text_poke_sync_each_cpu` is the crucial function, as it's designed to wait for IPI (Inter-Processor Interrupt) completion on all CPUs.
Now, here's where it gets interesting, and potentially tricky. CPU 1 is reported as being in `__remove_hrtimer`. Is CPU 1 hung? At first glance it looked that way, but now it appears CPU 1 is getting an interrupt, specifically a timer interrupt, while it's in the `default_idle` state. That's a normal occurrence.
Examining the call stack for CPU 1: the system is going from `__run_hrtimer` to `__hrtimer_run_queues` to `hrtimer_interrupt`, through the APIC interrupt handler and then finally back to `default_idle`. This makes perfect sense. CPU 1 looks to be just interrupted while in idle, and is likely still working as intended. I need to keep an eye on it.
Aha! Now I see, CPU 1's RIP is inside `__remove_hrtimer` and its exact instruction pointer is `__remove_hrtimer+0x8e/0x3e0`. The question is: is it *actually* stuck in a loop inside `__remove_hrtimer`? I need to analyze the assembly code of that function to understand what's happening.
|
| 13/2 |
2026/05/18 18:34 |
tool |
grepper |
0m
Args: map[Expression:__remove_hrtimer]
Results: map[Output:Documentation/trace/ftrace.rst=2160=Doing the same with chrt -r 5 and function-trace set.
--
Documentation/trace/ftrace.rst-2225- <idle>-0 3dN.1 16us : add_preempt_count <-_raw_spin_lock_irqsave
Documentation/trace/ftrace.rst:2226: <idle>-0 3dN.2 17us : __remove_hrtimer <-remove_hrtimer.part.16
Documentation/trace/ftrace.rst:2227: <idle>-0 3dN.2 17us : hrtimer_force_reprogram <-__remove_hrtimer
Documentation/trace/ftrace.rst-2228- <idle>-0 3dN.2 17us : tick_program_event <-hrtimer_force_reprogram
--
Documentation/trace/ftrace.rst=3098=Produces::
--
Documentation/trace/ftrace.rst-3112- <idle>-0 [003] dN.1 4228.547804: hrtimer_try_to_cancel <-hrtimer_cancel
Documentation/trace/ftrace.rst:3113: <idle>-0 [003] dN.2 4228.547805: hrtimer_force_reprogram <-__remove_hrtimer
Documentation/trace/ftrace.rst-3114- <idle>-0 [003] dN.1 4228.547805: hrtimer_forward <-tick_nohz_idle_exit
--
include/linux/hrtimer_defs.h=38=enum hrtimer_base_type {
--
include/linux/hrtimer_defs.h-77- *
include/linux/hrtimer_defs.h:78: * Note: next_timer is just an optimization for __remove_hrtimer().
include/linux/hrtimer_defs.h-79- * Do not dereference the pointer because it is not reliable on
--
kernel/time/hrtimer.c=1124=static inline void base_update_next_timer(struct hrtimer_clock_base *base)
--
kernel/time/hrtimer.c-1131-/*
kernel/time/hrtimer.c:1132: * __remove_hrtimer - internal function to remove a timer
kernel/time/hrtimer.c-1133- *
--
kernel/time/hrtimer.c-1138- */
kernel/time/hrtimer.c:1139:static void __remove_hrtimer(struct hrtimer *timer, struct hrtimer_clock_base *base,
kernel/time/hrtimer.c-1140- bool newstate, bool reprogram)
--
kernel/time/hrtimer.c=1182=static inline bool remove_hrtimer(struct hrtimer *timer, struct hrtimer_clock_base *base,
--
kernel/time/hrtimer.c-1201-
kernel/time/hrtimer.c:1202: __remove_hrtimer(timer, base, newstate, reprogram);
kernel/time/hrtimer.c-1203- return true;
--
kernel/time/hrtimer.c=1888=static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base, struct hrtimer_clock_base *base,
--
kernel/time/hrtimer.c-1909-
kernel/time/hrtimer.c:1910: __remove_hrtimer(timer, base, HRTIMER_STATE_INACTIVE, false);
kernel/time/hrtimer.c-1911- fn = ACCESS_PRIVATE(timer, function);
--
kernel/time/hrtimer.c=2438=static void migrate_hrtimer_list(struct hrtimer_clock_base *old_base,
--
kernel/time/hrtimer.c-2453- */
kernel/time/hrtimer.c:2454: __remove_hrtimer(timer, old_base, HRTIMER_STATE_ENQUEUED, false);
kernel/time/hrtimer.c-2455- timer->base = new_base;
]
|
| 14/2 |
2026/05/18 18:34 |
llm |
debugger |
0m
Model:
Tokens:
input: 33134
output: 183
thoughts: 218
Thoughts: **Analyzing the Crash in `__remove_hrtimer`**
Okay, so the crash happened in `__remove_hrtimer+0x8e`. Right, let's take a look at the code at that address. Hmm, here's the assembly: `Code: ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d`. Now, it's time to disassemble this and dig into it. We've got work to do here. Let's see if we can identify what's going wrong.
|
| 15/2 |
2026/05/18 18:34 |
tool |
codeexpert |
7m
Args: map[Question:Disassemble this x86_64 code snippet:
ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d
]
Results: map[Answer:The provided x86_64 code snippet is disassembled as follows:
```nasm
ee out dx,al
e8 f6 fd 11 00 call 0x11fdfb
45 85 ed test r13d,r13d
48 89 5c 24 08 mov QWORD PTR [rsp+0x8],rbx
74 2a je 0x40
48 8d 7b 18 lea rdi,[rbx+0x18]
be ff ff ff ff mov esi,0xffffffff
e8 ce a2 21 0a call 0xa21a2ce
41 89 c5 mov r13d,eax
31 ff xor edi,edi
89 c6 mov esi,eax
e8 d2 fd 11 00 call 0x11fdf9
45 85 ed test r13d,r13d
0f 84 33 02 00 00 je 0x27c
e8 84 f9 11 00 call 0x11f9b3
eb 05 jmp 0x3a
e8 7d f9 11 00 call 0x11f9b1
4d rex.WRB
```
### Analysis of the Code
The snippet appears to be from a kernel function, likely related to synchronization or waiting mechanisms (such as `io_uring` or `mutex` slow paths), given the patterns of testing return values and calling into other kernel sections.
1. **`out dx, al` (`ee`)**: This is an I/O instruction. In modern kernel code, it is relatively rare except in specific drivers or legacy support.
2. **`test r13d, r13d`**: This checks a 32-bit value in the `r13` register (likely a return code or status).
3. **`lea rdi, [rbx+0x18]`**: This calculates the address of a field at offset `0x18` (24 bytes) from the base address in `rbx`. Depending on the structure in `rbx`, this could be `&task->__state` (if `rbx` is `task_struct`) or a member of a synchronization primitive.
4. **`mov esi, 0xffffffff`**: Sets the second argument for the subsequent call to `-1`.
5. **`call 0xa21a2ce`**: A call with a large relative offset, often indicating a call between the kernel core and a module, or between two distant modules.
6. **`xor edi, edi; mov esi, eax; call ...`**: This prepares arguments `(0, eax)` for a function call, where `eax` is the return value of the previous call. This pattern is common in tracing (e.g., `trace_contention_end(lock, ret)` where `lock` is NULL).
7. **The `<45>` marker**: In the original input, the byte `45` was enclosed in `< >`. This is a standard convention in Linux kernel "oops" or "panic" logs to indicate the instruction pointer (RIP) at the time of the fault. The instruction starting at this byte is `45 85 ed` (`test r13d, r13d`).]
|
| 16/3 |
2026/05/18 18:34 |
agent |
codeexpert |
7m
Model:
Instruction:
You are a capable Linux kernel developer tasked with researching complex questions
about kernel source code. You will be given a concrete question, and need to provide
a concrete answer.
Use tools extensively while researching the question. Don't make assumptions,
or rely on your previous knowledge about the kernel source code, use available tools
to access the actual source code.
Use all available sources of information:
- kernel source code
- documentation in the Documentation dir in the source tree
- git commits descriptions, git blame
Prefer calling several tools at the same time to save round-trips.
Prompt: Disassemble this x86_64 code snippet:
ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d
Reply: The provided x86_64 code snippet is disassembled as follows:
```nasm
ee out dx,al
e8 f6 fd 11 00 call 0x11fdfb
45 85 ed test r13d,r13d
48 89 5c 24 08 mov QWORD PTR [rsp+0x8],rbx
74 2a je 0x40
48 8d 7b 18 lea rdi,[rbx+0x18]
be ff ff ff ff mov esi,0xffffffff
e8 ce a2 21 0a call 0xa21a2ce
41 89 c5 mov r13d,eax
31 ff xor edi,edi
89 c6 mov esi,eax
e8 d2 fd 11 00 call 0x11fdf9
45 85 ed test r13d,r13d
0f 84 33 02 00 00 je 0x27c
e8 84 f9 11 00 call 0x11f9b3
eb 05 jmp 0x3a
e8 7d f9 11 00 call 0x11f9b1
4d rex.WRB
```
### Analysis of the Code
The snippet appears to be from a kernel function, likely related to synchronization or waiting mechanisms (such as `io_uring` or `mutex` slow paths), given the patterns of testing return values and calling into other kernel sections.
1. **`out dx, al` (`ee`)**: This is an I/O instruction. In modern kernel code, it is relatively rare except in specific drivers or legacy support.
2. **`test r13d, r13d`**: This checks a 32-bit value in the `r13` register (likely a return code or status).
3. **`lea rdi, [rbx+0x18]`**: This calculates the address of a field at offset `0x18` (24 bytes) from the base address in `rbx`. Depending on the structure in `rbx`, this could be `&task->__state` (if `rbx` is `task_struct`) or a member of a synchronization primitive.
4. **`mov esi, 0xffffffff`**: Sets the second argument for the subsequent call to `-1`.
5. **`call 0xa21a2ce`**: A call with a large relative offset, often indicating a call between the kernel core and a module, or between two distant modules.
6. **`xor edi, edi; mov esi, eax; call ...`**: This prepares arguments `(0, eax)` for a function call, where `eax` is the return value of the previous call. This pattern is common in tracing (e.g., `trace_contention_end(lock, ret)` where `lock` is NULL).
7. **The `<45>` marker**: In the original input, the byte `45` was enclosed in `< >`. This is a standard convention in Linux kernel "oops" or "panic" logs to indicate the instruction pointer (RIP) at the time of the fault. The instruction starting at this byte is `45 85 ed` (`test r13d, r13d`).
|
| 17/4 |
2026/05/18 18:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 2421
output: 19
thoughts: 6225
Thoughts: **Disassembly and Analysis of an x86_64 Code Snippet**
Okay, here's what I'm thinking about this code snippet. The user wants me to break down this assembly, and it looks like a kernel oops. The `ee` at the beginning is probably garbage, and I'll ignore it. It is likely the tail end of the previous instruction. The code snippet begins with a `call` instruction, a `test r13d` instruction, which tests r13d. The following `mov`, `jz`, and `lea` instructions followed by a `mov esi, -1` sequence looks like a potential error handling or initialization sequence. The code calls a series of functions. There is a very large call offset to `0x0a21a2ce` which indicates that it might be calling a function far away. This could be within a kernel module or calling from a module to kernel.
The `45 85 ed` instruction, `test r13d, r13d`, appears multiple times, which looks suspicious. It's likely checking return values. There is a `jz` instruction that follows it, making me think it's a conditional check. The fact that the first instance of it is marked with `<45>` hints that this is the instruction pointer at the time of an error. `test r13d, r13d` is not likely to fault by itself, unless there is something very peculiar. The code appears to be saving the `rbx` register and then loading a field from an object pointed to by `rbx` into `rdi`. The instruction that moves `esi` to -1 is quite common. The relative calls suggest that the control flow is within a close range. The last `e8` call, `e8 7d f9 11 00`, is suspicious, since it is a relative call that might be a jump.
I'm interpreting the code like this:
1. A `call` to some function
2. Test the return value in `r13d`
3. Conditional jump if zero
4. Load address of object field into `rdi`
5. Call another function.
6. `C` to another function using return value of the previous `B` function call.
Based on the pattern, it seems like the code is trying to check a status, maybe from a previous call, and branch based on it. Also, the code appears to be operating on a structure where a field at offset 0x18 is used. The code appears to be saving `rbx` to the stack and calling functions. It also calls another function that is quite large. It is difficult to ascertain the exact function without more data, but the patterns are relatively clear.
My conclusion is that this looks like an error handling routine within some kernel code, possibly a module, or some function call between modules.
|
| 18/4 |
2026/05/18 18:35 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:f_lock OffsetBits:0 SizeBits:512] map[Name:f_mode OffsetBits:512 SizeBits:32] map[Name:f_op OffsetBits:576 SizeBits:64] map[Name:f_mapping OffsetBits:640 SizeBits:64] map[Name:private_data OffsetBits:704 SizeBits:64] map[Name:f_inode OffsetBits:768 SizeBits:64] map[Name:f_flags OffsetBits:832 SizeBits:32] map[Name:f_iocb_flags OffsetBits:864 SizeBits:32] map[Name:f_cred OffsetBits:896 SizeBits:64] map[Name:f_owner OffsetBits:960 SizeBits:64] map[Name: OffsetBits:1024 SizeBits:128] map[Name: OffsetBits:1152 SizeBits:1088] map[Name:f_pos OffsetBits:2240 SizeBits:64] map[Name:f_security OffsetBits:2304 SizeBits:64] map[Name:f_wb_err OffsetBits:2368 SizeBits:32] map[Name:f_sb_err OffsetBits:2400 SizeBits:32] map[Name:f_ep OffsetBits:2432 SizeBits:64] map[Name: OffsetBits:2496 SizeBits:256] map[Name:f_ref OffsetBits:2752 SizeBits:64]]]
|
| 19/4 |
2026/05/18 18:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 9716
output: 23
thoughts: 1987
Thoughts: **Dissecting a Code Snippet: A Journey into Kernel Internals**
Okay, so I've stumbled upon this assembly snippet and I'm trying to figure out its purpose within the kernel. My initial thought was about memory offsets within `struct file`, but that didn't pan out. I had a brief moment of confusion with `f_lock` and its size, but I quickly realized the x86_64 architecture uses a 4-byte `spinlock_t`. Still, I'm second-guessing if I'm even looking in the right place, so I've temporarily moved on from considering offsets.
My current approach is to understand the logic flow, so I'm trying to disassemble the instructions more accurately. I'm focusing on the call instructions (`e8`) and conditional jumps to get the overall structure. There seems to be a call to `func1`, followed by a conditional check (likely the return value of `func1`), then either `func2` or `func3` will be called. There's also a `jz` instruction that has a significant offset. This indicates a longer jump. This indicates a likely if/else construct, and the `0x233` jump offset has me suspicious that this is a case of handling an error.
I've identified the main pattern as `Call1()`, followed by an `if` statement based on the return value of `Call1`, which then either calls `Call2` and unconditionally jumps, or only calls `Call3`.
I'm zeroing in on `Call2`, because it takes `rbx + 0x18` and `-1` as arguments. Based on the pattern, I'm trying to understand the likely arguments. I'm assuming that `-1` is a special value (like `MAX_SCHEDULE_TIMEOUT` or some kind of error code). I'm also looking at what kind of struct `rbx + 0x18` might be, but it's not immediately obvious and could be a few different types of structs.
Then, there is `Call3` with a zero and the return of `Call2`, so I'm making guesses on what the return of `Call2` means. This could be something like a function that checks status, and then a function that interprets the status, such as a function that converts a status to an error.
I was thinking about searching for the byte sequence, but that may be difficult to interpret, even if I find something. I've realized that `r13d` is a callee-saved register, so it's a local variable. So I've shifted my attention to what these calls are doing with this local variable. I'm seeing if I can pinpoint which structs have offset 24 (0x18) to get a more concrete lead. I'm going to check `struct task_struct`, `struct kiocb`, and `struct io_kiocb` (for io_uring).
Let's start there.
|
| 20/4 |
2026/05/18 18:35 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name: OffsetBits:0 SizeBits:512] map[Name:opcode OffsetBits:512 SizeBits:8] map[Name:iopoll_completed OffsetBits:520 SizeBits:8] map[Name:buf_index OffsetBits:528 SizeBits:16] map[Name:nr_tw OffsetBits:544 SizeBits:32] map[Name:flags OffsetBits:576 SizeBits:64] map[Name:cqe OffsetBits:640 SizeBits:128] map[Name:ctx OffsetBits:768 SizeBits:64] map[Name:tctx OffsetBits:832 SizeBits:64] map[Name: OffsetBits:896 SizeBits:64] map[Name: OffsetBits:960 SizeBits:64] map[Name:file_node OffsetBits:1024 SizeBits:64] map[Name:refs OffsetBits:1088 SizeBits:32] map[Name:cancel_seq_set OffsetBits:1120 SizeBits:8] map[Name: OffsetBits:1152 SizeBits:128] map[Name: OffsetBits:1280 SizeBits:128] map[Name:apoll OffsetBits:1408 SizeBits:64] map[Name:async_data OffsetBits:1472 SizeBits:64] map[Name:poll_refs OffsetBits:1536 SizeBits:32] map[Name:link OffsetBits:1600 SizeBits:64] map[Name:creds OffsetBits:1664 SizeBits:64] map[Name:work OffsetBits:1728 SizeBits:128] map[Name:big_cqe OffsetBits:1856 SizeBits:128]]]
|
| 21/4 |
2026/05/18 18:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 12853
output: 21
thoughts: 3216
Thoughts: **Analyzing a Kernel Oops Code Snippet**
Okay, here's the breakdown of my thought process as I try to decipher this kernel oops:
Initially, I was trying to locate specific offsets within data structures like `io_kiocb`, `struct file`, and `struct socket` based on the given addresses, but that didn't pan out. I tried to use a byte sequence search, but couldn't.
Then I focused on the provided "Code:" line, interpreting the assembly instructions. I immediately recognized `45 85 ed` as `test r13d, r13d` and tried to determine where the faulting instruction was. Realized that the `<45>` might be marking the end of the previous instruction.
I recalled seeing a similar `Code:` line in previous bug reports and started to piece together some common instruction patterns. I noticed `mov [rsp+8], rbx`, a pattern frequently seen in function entry code or with specific compiler optimizations, especially if `rbx` is saved but the function doesn't use the usual `push` instruction.
The `e8 ...` call instructions with similar offsets were intriguing, suggesting calls to functions in a close area of the kernel. This helped to narrow down the context. I then tried to look for specific instructions like `lea rdi, [rbx+0x18]` to determine the data being accessed. Through this, I figured out that `rbx+0x18` likely pointed to `wait_queue_entry->entry` and considered the implications of `be ff ff ff ff` setting `esi` to `-1`.
I considered the `task_struct` layout at one point, but then tried to search for the specific byte sequence in the kernel. This also failed. I tried searching for function references, such as `__wait_event_common` to find the related code.
The `-1` value for `esi` made me think of `schedule_timeout`, but then I realized that the argument should be in `rsi`. Then I realized that `0xffffffff` corresponds to a very large number of jiffies and it made me consider what function took an `int` with a value of `-1`. I tried `wait_for_completion_interruptible_timeout`, but the argument was `long`. I considered if the code might have been related to `io_uring` based on the context of the `test r13d, r13d` instruction.
I was focusing on `mov [rsp+8], rbx` being a function prologue. The `jz +0x233` instruction suggested a large conditional block. I tried to search for the whole byte sequence to locate the code but failed.
I considered `trace_...` functions based on the code at the end of the snippet, and whether it may be from an inline wrapper.
I was getting nowhere with these ideas, so I considered where `45 85 ed 0f 84 33 02 00 00` may be used. I tried to look for where `jz +0x233` and other functions are used.
I then revisited the entire `Code:` line, breaking it down instruction by instruction and trying to determine the logic, especially with the calls. I noted that `mov r13d, eax` could mean that the `test r13d, r13d` uses the return value of some prior call. Then, I considered that maybe the `rbx+0x18` parameter in the earlier call could be `24` in a `task_struct`.
|
| 22/4 |
2026/05/18 18:35 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:thread_info OffsetBits:0 SizeBits:192] map[Name:__state OffsetBits:192 SizeBits:32] map[Name:saved_state OffsetBits:224 SizeBits:32] map[Name:stack OffsetBits:256 SizeBits:64] map[Name:usage OffsetBits:320 SizeBits:32] map[Name:flags OffsetBits:352 SizeBits:32] map[Name:ptrace OffsetBits:384 SizeBits:32] map[Name:on_cpu OffsetBits:416 SizeBits:32] map[Name:wake_entry OffsetBits:448 SizeBits:128] map[Name:wakee_flips OffsetBits:576 SizeBits:32] map[Name:wakee_flip_decay_ts OffsetBits:640 SizeBits:64] map[Name:last_wakee OffsetBits:704 SizeBits:64] map[Name:recent_used_cpu OffsetBits:768 SizeBits:32] map[Name:wake_cpu OffsetBits:800 SizeBits:32] map[Name:on_rq OffsetBits:832 SizeBits:32] map[Name:prio OffsetBits:864 SizeBits:32] map[Name:static_prio OffsetBits:896 SizeBits:32] map[Name:normal_prio OffsetBits:928 SizeBits:32] map[Name:rt_priority OffsetBits:960 SizeBits:32] map[Name:se OffsetBits:1024 SizeBits:2560] map[Name:rt OffsetBits:3584 SizeBits:384] map[Name:dl OffsetBits:3968 SizeBits:2176] map[Name:dl_server OffsetBits:6144 SizeBits:64] map[Name:sched_class OffsetBits:6208 SizeBits:64] map[Name:sched_task_group OffsetBits:6272 SizeBits:64] map[Name:sched_throttle_work OffsetBits:6336 SizeBits:128] map[Name:throttle_node OffsetBits:6464 SizeBits:128] map[Name:throttled OffsetBits:6592 SizeBits:8] map[Name:stats OffsetBits:6656 SizeBits:2048] map[Name:policy OffsetBits:8704 SizeBits:32] map[Name:max_allowed_capacity OffsetBits:8768 SizeBits:64] map[Name:nr_cpus_allowed OffsetBits:8832 SizeBits:32] map[Name:cpus_ptr OffsetBits:8896 SizeBits:64] map[Name:user_cpus_ptr OffsetBits:8960 SizeBits:64] map[Name:cpus_mask OffsetBits:9024 SizeBits:64] map[Name:migration_pending OffsetBits:9088 SizeBits:64] map[Name:migration_disabled OffsetBits:9152 SizeBits:16] map[Name:migration_flags OffsetBits:9168 SizeBits:16] map[Name:rcu_read_lock_nesting OffsetBits:9184 SizeBits:32] map[Name:rcu_read_unlock_special OffsetBits:9216 SizeBits:32] map[Name:rcu_node_entry OffsetBits:9280 SizeBits:128] map[Name:rcu_blocked_node OffsetBits:9408 SizeBits:64] map[Name:rcu_tasks_nvcsw OffsetBits:9472 SizeBits:64] map[Name:rcu_tasks_holdout OffsetBits:9536 SizeBits:8] map[Name:rcu_tasks_idx OffsetBits:9544 SizeBits:8] map[Name:rcu_tasks_idle_cpu OffsetBits:9568 SizeBits:32] map[Name:rcu_tasks_holdout_list OffsetBits:9600 SizeBits:128] map[Name:rcu_tasks_exit_cpu OffsetBits:9728 SizeBits:32] map[Name:rcu_tasks_exit_list OffsetBits:9792 SizeBits:128] map[Name:trc_reader_nesting OffsetBits:9920 SizeBits:32] map[Name:trc_reader_scp OffsetBits:9984 SizeBits:64] map[Name:sched_info OffsetBits:10048 SizeBits:512] map[Name:tasks OffsetBits:10560 SizeBits:128] map[Name:pushable_tasks OffsetBits:10688 SizeBits:320] map[Name:pushable_dl_tasks OffsetBits:11008 SizeBits:192] map[Name:mm OffsetBits:11200 SizeBits:64] map[Name:active_mm OffsetBits:11264 SizeBits:64] map[Name:exit_state OffsetBits:11328 SizeBits:32] map[Name:exit_code OffsetBits:11360 SizeBits:32] map[Name:exit_signal OffsetBits:11392 SizeBits:32] map[Name:pdeath_signal OffsetBits:11424 SizeBits:32] map[Name:jobctl OffsetBits:11456 SizeBits:64] map[Name:personality OffsetBits:11520 SizeBits:32] map[Name:sched_reset_on_fork OffsetBits:11552 SizeBits:1] map[Name:sched_contributes_to_load OffsetBits:11553 SizeBits:1] map[Name:sched_migrated OffsetBits:11554 SizeBits:1] map[Name:sched_task_hot OffsetBits:11555 SizeBits:1] map[Name: OffsetBits:11584 SizeBits:0] map[Name:sched_remote_wakeup OffsetBits:11584 SizeBits:1] map[Name:sched_rt_mutex OffsetBits:11585 SizeBits:1] map[Name:user_dumpable OffsetBits:11586 SizeBits:1] map[Name:in_execve OffsetBits:11587 SizeBits:1] map[Name:in_iowait OffsetBits:11588 SizeBits:1] map[Name:restore_sigmask OffsetBits:11589 SizeBits:1] map[Name:no_cgroup_migration OffsetBits:11590 SizeBits:1] map[Name:frozen OffsetBits:11591 SizeBits:1] map[Name:use_memdelay OffsetBits:11592 SizeBits:1] map[Name:in_memstall OffsetBits:11593 SizeBits:1] map[Name:in_page_owner OffsetBits:11594 SizeBits:1] map[Name:in_eventfd OffsetBits:11595 SizeBits:1] map[Name:reported_split_lock OffsetBits:11596 SizeBits:1] map[Name:in_nf_duplicate OffsetBits:11597 SizeBits:1] map[Name:atomic_flags OffsetBits:11648 SizeBits:64] map[Name:restart_block OffsetBits:11712 SizeBits:448] map[Name:pid OffsetBits:12160 SizeBits:32] map[Name:tgid OffsetBits:12192 SizeBits:32] map[Name:stack_canary OffsetBits:12224 SizeBits:64] map[Name:real_parent OffsetBits:12288 SizeBits:64] map[Name:parent OffsetBits:12352 SizeBits:64] map[Name:children OffsetBits:12416 SizeBits:128] map[Name:sibling OffsetBits:12544 SizeBits:128] map[Name:group_leader OffsetBits:12672 SizeBits:64] map[Name:ptraced OffsetBits:12736 SizeBits:128] map[Name:ptrace_entry OffsetBits:12864 SizeBits:128] map[Name:thread_pid OffsetBits:12992 SizeBits:64] map[Name:pid_links OffsetBits:13056 SizeBits:512] map[Name:thread_node OffsetBits:13568 SizeBits:128] map[Name:vfork_done OffsetBits:13696 SizeBits:64] map[Name:set_child_tid OffsetBits:13760 SizeBits:64] map[Name:clear_child_tid OffsetBits:13824 SizeBits:64] map[Name:worker_private OffsetBits:13888 SizeBits:64] map[Name:utime OffsetBits:13952 SizeBits:64] map[Name:stime OffsetBits:14016 SizeBits:64] map[Name:gtime OffsetBits:14080 SizeBits:64] map[Name:prev_cputime OffsetBits:14144 SizeBits:640] map[Name:vtime OffsetBits:14784 SizeBits:704] map[Name:nvcsw OffsetBits:15488 SizeBits:64] map[Name:nivcsw OffsetBits:15552 SizeBits:64] map[Name:start_time OffsetBits:15616 SizeBits:64] map[Name:start_boottime OffsetBits:15680 SizeBits:64] map[Name:min_flt OffsetBits:15744 SizeBits:64] map[Name:maj_flt OffsetBits:15808 SizeBits:64] map[Name:posix_cputimers OffsetBits:15872 SizeBits:640] map[Name:posix_cputimers_work OffsetBits:16512 SizeBits:1280] map[Name:ptracer_cred OffsetBits:17792 SizeBits:64] map[Name:real_cred OffsetBits:17856 SizeBits:64] map[Name:cred OffsetBits:17920 SizeBits:64] map[Name:cached_requested_key OffsetBits:17984 SizeBits:64] map[Name:comm OffsetBits:18048 SizeBits:128] map[Name:nameidata OffsetBits:18176 SizeBits:64] map[Name:sysvsem OffsetBits:18240 SizeBits:64] map[Name:sysvshm OffsetBits:18304 SizeBits:128] map[Name:last_switch_count OffsetBits:18432 SizeBits:64] map[Name:last_switch_time OffsetBits:18496 SizeBits:64] map[Name:fs OffsetBits:18560 SizeBits:64] map[Name:files OffsetBits:18624 SizeBits:64] map[Name:io_uring OffsetBits:18688 SizeBits:64] map[Name:io_uring_restrict OffsetBits:18752 SizeBits:64] map[Name:nsproxy OffsetBits:18816 SizeBits:64] map[Name:signal OffsetBits:18880 SizeBits:64] map[Name:sighand OffsetBits:18944 SizeBits:64] map[Name:blocked OffsetBits:19008 SizeBits:64] map[Name:real_blocked OffsetBits:19072 SizeBits:64] map[Name:saved_sigmask OffsetBits:19136 SizeBits:64] map[Name:pending OffsetBits:19200 SizeBits:192] map[Name:sas_ss_sp OffsetBits:19392 SizeBits:64] map[Name:sas_ss_size OffsetBits:19456 SizeBits:64] map[Name:sas_ss_flags OffsetBits:19520 SizeBits:32] map[Name:task_works OffsetBits:19584 SizeBits:64] map[Name:audit_context OffsetBits:19648 SizeBits:64] map[Name:loginuid OffsetBits:19712 SizeBits:32] map[Name:sessionid OffsetBits:19744 SizeBits:32] map[Name:seccomp OffsetBits:19776 SizeBits:128] map[Name:syscall_dispatch OffsetBits:19904 SizeBits:256] map[Name:parent_exec_id OffsetBits:20160 SizeBits:64] map[Name:self_exec_id OffsetBits:20224 SizeBits:64] map[Name:alloc_lock OffsetBits:20288 SizeBits:512] map[Name:pi_lock OffsetBits:20800 SizeBits:512] map[Name:wake_q OffsetBits:21312 SizeBits:64] map[Name:pi_waiters OffsetBits:21376 SizeBits:128] map[Name:pi_top_task OffsetBits:21504 SizeBits:64] map[Name:pi_blocked_on OffsetBits:21568 SizeBits:64] map[Name:blocked_on OffsetBits:21632 SizeBits:64] map[Name:blocked_lock OffsetBits:21696 SizeBits:512] map[Name:blocker OffsetBits:22208 SizeBits:64] map[Name:non_block_count OffsetBits:22272 SizeBits:32] map[Name:irqtrace OffsetBits:22336 SizeBits:448] map[Name:hardirq_threaded OffsetBits:22784 SizeBits:32] map[Name:hardirq_chain_key OffsetBits:22848 SizeBits:64] map[Name:softirqs_enabled OffsetBits:22912 SizeBits:32] map[Name:softirq_context OffsetBits:22944 SizeBits:32] map[Name:irq_config OffsetBits:22976 SizeBits:32] map[Name:curr_chain_key OffsetBits:23040 SizeBits:64] map[Name:lockdep_depth OffsetBits:23104 SizeBits:32] map[Name:lockdep_recursion OffsetBits:23136 SizeBits:32] map[Name:held_locks OffsetBits:23168 SizeBits:15360] map[Name:in_ubsan OffsetBits:38528 SizeBits:32] map[Name:journal_info OffsetBits:38592 SizeBits:64] map[Name:bio_list OffsetBits:38656 SizeBits:64] map[Name:plug OffsetBits:38720 SizeBits:64] map[Name:reclaim_state OffsetBits:38784 SizeBits:64] map[Name:io_context OffsetBits:38848 SizeBits:64] map[Name:capture_control OffsetBits:38912 SizeBits:64] map[Name:ptrace_message OffsetBits:38976 SizeBits:64] map[Name:last_siginfo OffsetBits:39040 SizeBits:64] map[Name:ioac OffsetBits:39104 SizeBits:448] map[Name:psi_flags OffsetBits:39552 SizeBits:32] map[Name:acct_rss_mem1 OffsetBits:39616 SizeBits:64] map[Name:acct_vm_mem1 OffsetBits:39680 SizeBits:64] map[Name:acct_timexpd OffsetBits:39744 SizeBits:64] map[Name:mems_allowed OffsetBits:39808 SizeBits:64] map[Name:mems_allowed_seq OffsetBits:39872 SizeBits:448] map[Name:cpuset_mem_spread_rotor OffsetBits:40320 SizeBits:32] map[Name:cgroups OffsetBits:40384 SizeBits:64] map[Name:cg_list OffsetBits:40448 SizeBits:128] map[Name:robust_list OffsetBits:40576 SizeBits:64] map[Name:pi_state_list OffsetBits:40640 SizeBits:128] map[Name:pi_state_cache OffsetBits:40768 SizeBits:64] map[Name:futex_exit_mutex OffsetBits:40832 SizeBits:1088] map[Name:futex_state OffsetBits:41920 SizeBits:32] map[Name:perf_recursion OffsetBits:41952 SizeBits:32] map[Name:perf_event_ctxp OffsetBits:41984 SizeBits:64] map[Name:perf_event_mutex OffsetBits:42048 SizeBits:1088] map[Name:perf_event_list OffsetBits:43136 SizeBits:128] map[Name:perf_ctx_data OffsetBits:43264 SizeBits:64] map[Name:preempt_disable_ip OffsetBits:43328 SizeBits:64] map[Name:mempolicy OffsetBits:43392 SizeBits:64] map[Name:il_prev OffsetBits:43456 SizeBits:16] map[Name:il_weight OffsetBits:43472 SizeBits:8] map[Name:pref_node_fork OffsetBits:43488 SizeBits:16] map[Name:numa_scan_seq OffsetBits:43520 SizeBits:32] map[Name:numa_scan_period OffsetBits:43552 SizeBits:32] map[Name:numa_scan_period_max OffsetBits:43584 SizeBits:32] map[Name:numa_preferred_nid OffsetBits:43616 SizeBits:32] map[Name:numa_migrate_retry OffsetBits:43648 SizeBits:64] map[Name:node_stamp OffsetBits:43712 SizeBits:64] map[Name:last_task_numa_placement OffsetBits:43776 SizeBits:64] map[Name:last_sum_exec_runtime OffsetBits:43840 SizeBits:64] map[Name:numa_work OffsetBits:43904 SizeBits:128] map[Name:numa_group OffsetBits:44032 SizeBits:64] map[Name:numa_faults OffsetBits:44096 SizeBits:64] map[Name:total_numa_faults OffsetBits:44160 SizeBits:64] map[Name:numa_faults_locality OffsetBits:44224 SizeBits:192] map[Name:numa_pages_migrated OffsetBits:44416 SizeBits:64] map[Name:rseq OffsetBits:44480 SizeBits:320] map[Name:mm_cid OffsetBits:44800 SizeBits:192] map[Name:tlb_ubc OffsetBits:44992 SizeBits:192] map[Name:splice_pipe OffsetBits:45184 SizeBits:64] map[Name:task_frag OffsetBits:45248 SizeBits:128] map[Name:make_it_fail OffsetBits:45376 SizeBits:32] map[Name:fail_nth OffsetBits:45408 SizeBits:32] map[Name:nr_dirtied OffsetBits:45440 SizeBits:32] map[Name:nr_dirtied_pause OffsetBits:45472 SizeBits:32] map[Name:dirty_paused_when OffsetBits:45504 SizeBits:64] map[Name:timer_slack_ns OffsetBits:45568 SizeBits:64] map[Name:default_timer_slack_ns OffsetBits:45632 SizeBits:64] map[Name:kasan_depth OffsetBits:45696 SizeBits:32] map[Name:trace_recursion OffsetBits:45760 SizeBits:64] map[Name:kcov_mode OffsetBits:45824 SizeBits:32] map[Name:kcov_size OffsetBits:45856 SizeBits:32] map[Name:kcov_area OffsetBits:45888 SizeBits:64] map[Name:kcov OffsetBits:45952 SizeBits:64] map[Name:kcov_handle OffsetBits:46016 SizeBits:64] map[Name:kcov_sequence OffsetBits:46080 SizeBits:32] map[Name:kcov_softirq OffsetBits:46112 SizeBits:32] map[Name:memcg_nr_pages_over_high OffsetBits:46144 SizeBits:32] map[Name:active_memcg OffsetBits:46208 SizeBits:64] map[Name:objcg OffsetBits:46272 SizeBits:64] map[Name:throttle_disk OffsetBits:46336 SizeBits:64] map[Name:sequential_io OffsetBits:46400 SizeBits:32] map[Name:sequential_io_avg OffsetBits:46432 SizeBits:32] map[Name:kmap_ctrl OffsetBits:46464 SizeBits:0] map[Name:task_state_change OffsetBits:46464 SizeBits:64] map[Name:rcu OffsetBits:46528 SizeBits:128] map[Name:rcu_users OffsetBits:46656 SizeBits:32] map[Name:pagefault_disabled OffsetBits:46688 SizeBits:32] map[Name:oom_reaper_list OffsetBits:46720 SizeBits:64] map[Name:oom_reaper_timer OffsetBits:46784 SizeBits:640] map[Name:stack_vm_area OffsetBits:47424 SizeBits:64] map[Name:stack_refcount OffsetBits:47488 SizeBits:32] map[Name:security OffsetBits:47552 SizeBits:64] map[Name:bpf_storage OffsetBits:47616 SizeBits:64] map[Name:bpf_ctx OffsetBits:47680 SizeBits:64] map[Name:bpf_net_context OffsetBits:47744 SizeBits:64] map[Name:mce_vaddr OffsetBits:47808 SizeBits:64] map[Name:mce_kflags OffsetBits:47872 SizeBits:64] map[Name:mce_addr OffsetBits:47936 SizeBits:64] map[Name:mce_ripv OffsetBits:48000 SizeBits:1] map[Name:mce_whole_page OffsetBits:48001 SizeBits:1] map[Name:__mce_reserved OffsetBits:48002 SizeBits:62] map[Name:mce_kill_me OffsetBits:48064 SizeBits:128] map[Name:mce_count OffsetBits:48192 SizeBits:32] map[Name:l1d_flush_kill OffsetBits:48256 SizeBits:128] map[Name:unwind_info OffsetBits:48384 SizeBits:320] map[Name:thread OffsetBits:48704 SizeBits:1216]]]
|
| 23/4 |
2026/05/18 18:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 23200
output: 44
thoughts: 1062
Thoughts: **Dissecting a Code Snippet: A Journey Through Memory and Context**
Okay, so I'm looking at this bit of code, and I need to figure out what it's doing. The first thing that jumps out is `0x18`. That offset immediately sets off alarm bells, so I have to try to figure out what is at that offset. It seems to be `lea rdi, [rbx+0x18]`, indicating an address calculation. `rbx` is the base, and I'm figuring that 0x18 is an offset into something.
My initial thought was `task_struct`, since that's a common structure, and I know the offset for the `__state` field is 192 bits (24 bytes). But wait, `__state` is just an `unsigned int`! Why would you be taking the *address* of a simple integer? That means it's likely being passed as a pointer to a function. Perhaps `signal_pending_state`? But, no, `signal_pending_state` expects a different signature.
Let's look at the instruction sequence: `Call2`, then a move to `r13d`, `xor edi, edi`, `mov esi, eax`, then `Call3`. So, `Call2` is likely returning a value that is then being passed to `Call3` with zero as the first argument. Is this a scheduling call? Probably not `trace_sched_switch` from what I see.
Okay, let's back up. Maybe `rbx` *isn't* a `task_struct`. What about a `wait_queue_head`? Its structure starts with a spinlock and a `list_head`, which I know from memory are a total of 20 bytes. That's not right.
Let's try a different tack. I can look for the sequence itself. I'm looking for the `lea rdi, [rbx+0x18]`, `mov esi, -1`, then a `call` instruction, followed by the `mov r13d, eax`, `xor edi, edi`, `mov esi, eax`, and the final `call`.
Okay, the `xor edi, edi` and `mov esi, eax` part suggests a function call where the arguments are `(0, eax)`. Where `eax` is the return value of the previous call. Let's try some function possibilities... nope.
Okay, back to square one. `48 8d 7b 18` is `lea rdi, [rbx+0x18]`. What if `rbx` is a `struct hrtimer_sleeper`? It has a `struct hrtimer timer` at the beginning, which is large. What's at offset 0x18 in that struct?
Aha! `struct kiocb`! `ki_ioprio` is at offset 24 (0x18) and it is an `unsigned short`. So `lea rdi, [rbx+0x18]` is the address of `kiocb->ki_ioprio`.
I'll check the structure of `struct io_kiocb` again. Hmmm.
Hang on... I see a pattern. The presence of `e8 ... 45 85 ed 0f 84 ...` looks like something like `mutex_lock` slow path. But the `lea rdi, [rbx+0x18]` is still very specific.
I can't assemble this mentally, so, I will attempt to apply some tools in my mental inventory... I'll look for functions and definitions in my codebase. Maybe an `io_uring` function like `io_cqring_wait`?
Now, let's look at `io_cqring_wait` in `io_uring/io_uring.c`. I have a feeling I'm close.
|
| 24/4 |
2026/05/18 18:36 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:io_uring/io_uring.c IncludeLines:true Name:io_cqring_wait]
Results: map[SourceCode: 185: /*
186: * Wait until events become available, if we don't already have some. The
187: * application must reap them itself, as they reside on the shared cq ring.
188: */
189: int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags,
190: struct ext_arg *ext_arg)
191: {
192: struct io_wait_queue iowq;
193: struct io_rings *rings;
194: ktime_t start_time;
195: int ret, nr_wait;
196:
197: min_events = min_t(int, min_events, ctx->cq_entries);
198:
199: if (!io_allowed_run_tw(ctx))
200: return -EEXIST;
201: if (io_local_work_pending(ctx))
202: io_run_local_work(ctx, min_events,
203: max(IO_LOCAL_TW_DEFAULT_MAX, min_events));
204: io_run_task_work();
205:
206: if (unlikely(test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)))
207: io_cqring_do_overflow_flush(ctx);
208:
209: rcu_read_lock();
210: rings = io_get_rings(ctx);
211: if (__io_cqring_events_user(ctx) >= min_events) {
212: rcu_read_unlock();
213: return 0;
214: }
215:
216: init_waitqueue_func_entry(&iowq.wq, io_wake_function);
217: iowq.wq.private = current;
218: INIT_LIST_HEAD(&iowq.wq.entry);
219: iowq.ctx = ctx;
220: iowq.cq_tail = READ_ONCE(rings->cq.head) + min_events;
221: iowq.cq_min_tail = READ_ONCE(rings->cq.tail);
222: nr_wait = (int) iowq.cq_tail - READ_ONCE(rings->cq.tail);
223: rcu_read_unlock();
224: rings = NULL;
225: iowq.nr_timeouts = atomic_read(&ctx->cq_timeouts);
226: iowq.hit_timeout = 0;
227: iowq.min_timeout = ext_arg->min_time;
228: iowq.timeout = KTIME_MAX;
229: start_time = io_get_time(ctx);
230:
231: if (ext_arg->ts_set) {
232: iowq.timeout = timespec64_to_ktime(ext_arg->ts);
233: if (flags & IORING_ENTER_ABS_TIMER)
234: iowq.timeout = timens_ktime_to_host(ctx->clockid,
235: iowq.timeout);
236: else
237: iowq.timeout = ktime_add(iowq.timeout, start_time);
238: }
239:
240: if (ext_arg->sig) {
241: #ifdef CONFIG_COMPAT
242: if (in_compat_syscall())
243: ret = set_compat_user_sigmask((const compat_sigset_t __user *)ext_arg->sig,
244: ext_arg->argsz);
245: else
246: #endif
247: ret = set_user_sigmask(ext_arg->sig, ext_arg->argsz);
248:
249: if (ret)
250: return ret;
251: }
252:
253: io_napi_busy_loop(ctx, &iowq);
254:
255: trace_io_uring_cqring_wait(ctx, min_events);
256: do {
257: unsigned long check_cq;
258:
259: if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) {
260: atomic_set(&ctx->cq_wait_nr, nr_wait);
261: set_current_state(TASK_INTERRUPTIBLE);
262: } else {
263: prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq,
264: TASK_INTERRUPTIBLE);
265: }
266:
267: ret = io_cqring_wait_schedule(ctx, &iowq, ext_arg, start_time);
268: __set_current_state(TASK_RUNNING);
269: atomic_set(&ctx->cq_wait_nr, IO_CQ_WAKE_INIT);
270:
271: /*
272: * Run task_work after scheduling and before io_should_wake().
273: * If we got woken because of task_work being processed, run it
274: * now rather than let the caller do another wait loop.
275: */
276: if (io_local_work_pending(ctx))
277: io_run_local_work(ctx, nr_wait, nr_wait);
278: io_run_task_work();
279:
280: /*
281: * Non-local task_work will be run on exit to userspace, but
282: * if we're using DEFER_TASKRUN, then we could have waited
283: * with a timeout for a number of requests. If the timeout
284: * hits, we could have some requests ready to process. Ensure
285: * this break is _after_ we have run task_work, to avoid
286: * deferring running potentially pending requests until the
287: * next time we wait for events.
288: */
289: if (ret < 0)
290: break;
291:
292: check_cq = READ_ONCE(ctx->check_cq);
293: if (unlikely(check_cq)) {
294: /* let the caller flush overflows, retry */
295: if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
296: io_cqring_do_overflow_flush(ctx);
297: if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) {
298: ret = -EBADR;
299: break;
300: }
301: }
302:
303: if (io_should_wake(&iowq)) {
304: ret = 0;
305: break;
306: }
307: cond_resched();
308:
309: /* if min timeout has been hit, don't reset wait count */
310: if (!iowq.hit_timeout)
311: scoped_guard(rcu)
312: nr_wait = (int) iowq.cq_tail -
313: READ_ONCE(io_get_rings(ctx)->cq.tail);
314: else
315: nr_wait = 1;
316: } while (1);
317:
318: if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN))
319: finish_wait(&ctx->cq_wait, &iowq.wq);
320: restore_saved_sigmask_unless(ret == -EINTR);
321:
322: guard(rcu)();
323: return READ_ONCE(io_get_rings(ctx)->cq.head) == READ_ONCE(io_get_rings(ctx)->cq.tail) ? ret : 0;
324: }
SourceFile:io_uring/wait.c]
|
| 25/4 |
2026/05/18 18:36 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 27522
output: 44
thoughts: 282
Thoughts: **Analyzing the `io_uring/wait.c` file**
Okay, so I'm digging into `io_uring/wait.c` because something isn't quite clicking. Let's start by revisiting the `io_cqring_wait_schedule` function.
Right, the key lines of code here are: `e8 ce a2 21 0a` (Call2), then `41 89 c5` (mov r13d, eax), `31 ff` (xor edi, edi), `89 c6` (mov esi, eax), and finally `e8 d2 fd 11 00` (Call3).
Hmm... let's break that down. `xor edi, edi` and `mov esi, eax` paired together usually translates to `Call3(0, res)`. I need to figure out the purpose of `Call3`. It seems like I'm looking for a function that takes two integer arguments and is called after a schedule operation completes. `signal_pending` doesn't quite fit.
Wait a minute... I remember seeing `restore_saved_sigmask_unless(ret == -EINTR)` around line 320. That seems relevant. And it's an inline or a macro! `restore_saved_sigmask_unless` calls `restore_saved_sigmask()` if the condition is false. Now, let's go look at the definition of `restore_saved_sigmask_unless`. Something seems to be connecting here...
|
| 26/4 |
2026/05/18 18:36 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:io_uring/wait.c IncludeLines:true Name:restore_saved_sigmask_unless]
Results: map[SourceCode: 549: static inline void restore_saved_sigmask_unless(bool interrupted)
550: {
551: if (interrupted)
552: WARN_ON(!signal_pending(current));
553: else
554: restore_saved_sigmask();
555: }
SourceFile:include/linux/sched/signal.h]
|
| 27/4 |
2026/05/18 18:36 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 28301
output: 25
thoughts: 879
Thoughts: **Deconstructing the IO Completion Queue Wait Logic**
Okay, let's break this down piece by piece. First, I noticed the potential confusion with the signal mask restoration. It's `__restore_saved_sigmask()`, not `set_restore_sigmask()` as I initially thought. Good to clarify that.
Now, focusing on the code sequence, I see `e8 d2 fd 11 00` which is a `Call3`, followed by `test r13d, r13d` and a conditional jump `jz +0x233`. The significance hinges on the value of `r13d`. If `r13d` is the return value, then the `jz` instruction is executed only if it's zero.
The key is that `io_cqring_wait` returns `ret`. Looking at line 323, it confirms that the function returns `ret` only if there are no completion queue entries, otherwise it returns 0.
Let's trace the execution flow:
`Call1`, `test r13d`, `mov [rsp+8], rbx`, `jz ...`, `lea rdi, [rbx+0x18]`, `mov esi, -1`, and then `Call2`, followed by `mov r13d, eax`, `xor edi, edi`, `mov esi, eax`, and finally, the `Call3` and another `test r13d` instruction.
My hunch is that `Call2` is `io_cqring_wait_schedule`, and based on the arguments to `Call3` – `0` and the return value of `Call2` – it is `Call3(0, res)`. This indicates that the `Call2` which is `io_cqring_wait_schedule` function is being followed.
Now, comparing this to `io_cqring_wait` implementation in the kernel, it is confirmed that `__set_current_state(TASK_RUNNING)` and `atomic_set(&ctx->cq_wait_nr, IO_CQ_WAKE_INIT)` calls are made after `io_cqring_wait_schedule`. Also, the function `io_run_local_work` is called, with the arguments `ctx, nr_wait, nr_wait`.
The key is in the `xor edi, edi` and `mov esi, eax` instructions. That's a `func(0, eax)` call where `eax` is the return value. This suggests a potential `restore_saved_sigmask_unless(ret == -EINTR)`. However, there's no comparison of `eax` with `-4` (the value for `EINTR`). The code does not compare `eax` with `EINTR`, so this theory does not hold.
I'll search for the sequence `45 85 ed 48 89 5c 24 08 74 2a` looking for `__mutex_lock_slowpath`, or similar. Let's see if that helps to clarify the call sequence and argument interpretation. I'll use `codesearch-find-references` to do this.
|
| 28/4 |
2026/05/18 18:36 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:__mutex_lock_slowpath]
Results: map[References:<nil> TruncatedOutput:false]
|
| 29/4 |
2026/05/18 18:36 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 29923
output: 22
thoughts: 28
Thoughts: Okay, here's my interpretation of those thoughts, presented as a summary:
**Digging into a Mutex Lock**
Right, so I'm suspecting the issue centers around a mutex lock. The function name that's coming to mind is `__mutex_lock`. Let me just double-check that by looking at the core implementation. I should probably head straight to `kernel/locking/mutex.c` to see how it's actually implemented. That's the best place to get the details I need.
|
| 30/4 |
2026/05/18 18:36 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:[irqflag-debug.c lock_events.c lock_events.h lock_events_list.h lockdep.c lockdep_internals.h lockdep_proc.c lockdep_states.h locktorture.c mcs_spinlock.h mutex-debug.c mutex.c mutex.h osq_lock.c percpu-rwsem.c qrwlock.c qspinlock.c qspinlock.h qspinlock_paravirt.h qspinlock_stat.h rtmutex.c rtmutex_api.c rtmutex_common.h rwbase_rt.c rwsem.c semaphore.c spinlock.c spinlock_debug.c spinlock_rt.c test-ww_mutex.c ww_mutex.h ww_rt_mutex.c] Subdirs:<nil>]
|
| 31/4 |
2026/05/18 18:36 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 30362
output: 26
thoughts: 14
Thoughts: Here's my take on the kernel's mutex implementation, from a seasoned perspective:
**Diving Deep into the Heart of Mutual Exclusion**
Alright, let's crack open `kernel/locking/mutex.c` and see what makes these mutexes tick. I want to understand the *why* as much as the *how*, so I'm thinking about the low-level mechanics and the implications for real-world concurrency issues. This isn't just about reading code; it's about seeing the *intent* behind the design choices.
First, I need to focus on the data structures. The `struct mutex` itself is probably pretty lean, focusing on an owner, a wait queue, and maybe some state flags. I'm guessing they're leveraging a clever mix of atomic operations and lock-free techniques where possible, perhaps using compare-and-swap or similar primitives to minimize contention. I need to look closely at how the wait queue is managed. Is it a simple linked list, or are they using a more sophisticated data structure like a priority queue to handle fairness or priority inheritance? How is this queue managed to avoid any race conditions?
Then, the acquisition and release paths are crucial. I imagine `mutex_lock()` will have a fast path for uncontended cases, just a simple atomic exchange or test-and-set. For contention, it's gotta involve placing the current task on the wait queue, and perhaps sleeping or yielding the CPU until the mutex becomes available. And the "wakeup" side of things, in `mutex_unlock()`, must be carefully crafted to avoid any lost wakeup issues. It needs to wake up *one* waiter, and it has to be done atomically. Is it using a spinlock around the queue manipulation to keep things safe? Or is there some more subtle way to do it?
I need to see how they handle recursive locking, and whether that is supported. Also, the handling of priority inheritance is a key area. How do they avoid priority inversion problems where a high-priority task can get blocked indefinitely by a lower-priority task holding the mutex? Is there a mechanism to temporarily boost the priority of the lower-priority task? I'm betting that's probably implemented here.
Finally, I'll be looking for any clever optimizations or special handling for different CPU architectures. Are there any platform-specific tricks, like using atomic instructions or lock elision techniques available on certain processors? I expect there will be lots of assembly-level code here that has been meticulously tuned for performance. I am also curious about the debugging and instrumentation. Are there any debug flags or tracing facilities to help identify mutex contention and performance bottlenecks?
This has the potential to be a fascinating exploration. Let's see what the kernel folks have come up with.
|
| 32/4 |
2026/05/18 18:36 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/locking/mutex.c]
Results: map[Entities:[map[Kind:function Name:__mutex_add_waiter] map[Kind:function Name:__mutex_clear_flag] map[Kind:function Name:__mutex_handoff] map[Kind:function Name:__mutex_init_generic] map[Kind:function Name:__mutex_lock] map[Kind:function Name:__mutex_lock_common] map[Kind:function Name:__mutex_remove_waiter] map[Kind:function Name:__mutex_set_flag] map[Kind:function Name:__mutex_trylock] map[Kind:function Name:__mutex_trylock_common] map[Kind:function Name:__mutex_trylock_or_handoff] map[Kind:function Name:__mutex_trylock_or_owner] map[Kind:function Name:__mutex_unlock_slowpath] map[Kind:function Name:__owner_flags] map[Kind:function Name:__owner_task] map[Kind:function Name:__ww_mutex_lock] map[Kind:function Name:_mutex_lock_killable] map[Kind:function Name:_mutex_lock_nest_lock] map[Kind:function Name:_mutex_trylock_nest_lock] map[Kind:function Name:atomic_dec_and_mutex_lock] map[Kind:function Name:mutex_can_spin_on_owner] map[Kind:function Name:mutex_get_owner] map[Kind:function Name:mutex_init_lockdep] map[Kind:function Name:mutex_is_locked] map[Kind:function Name:mutex_lock_interruptible_nested] map[Kind:function Name:mutex_lock_io_nested] map[Kind:function Name:mutex_lock_nested] map[Kind:function Name:mutex_optimistic_spin] map[Kind:function Name:mutex_spin_on_owner] map[Kind:function Name:mutex_unlock] map[Kind:function Name:ww_mutex_deadlock_injection] map[Kind:function Name:ww_mutex_lock] map[Kind:function Name:ww_mutex_lock_interruptible] map[Kind:function Name:ww_mutex_spin_on_owner] map[Kind:function Name:ww_mutex_trylock] map[Kind:function Name:ww_mutex_unlock] map[Kind:global_variable Name:__UNIQUE_ID_addressable___SCK__tp_func_contention_begin_589] map[Kind:global_variable Name:__UNIQUE_ID_addressable___SCK__tp_func_contention_end_593] map[Kind:global_variable Name:__UNIQUE_ID_addressable___SCT__tp_func_contention_begin_590] map[Kind:global_variable Name:__UNIQUE_ID_addressable___SCT__tp_func_contention_end_594] map[Kind:global_variable Name:__UNIQUE_ID_addressable___traceiter_contention_begin_588] map[Kind:global_variable Name:__UNIQUE_ID_addressable___traceiter_contention_end_592] map[Kind:global_variable Name:__UNIQUE_ID_addressable___tracepoint_contention_begin_587] map[Kind:global_variable Name:__UNIQUE_ID_addressable___tracepoint_contention_end_591] map[Kind:global_variable Name:__UNIQUE_ID_addressable__mutex_lock_killable_578] map[Kind:global_variable Name:__UNIQUE_ID_addressable__mutex_lock_nest_lock_577] map[Kind:global_variable Name:__UNIQUE_ID_addressable__mutex_trylock_nest_lock_586] map[Kind:global_variable Name:__UNIQUE_ID_addressable_atomic_dec_and_mutex_lock_595] map[Kind:global_variable Name:__UNIQUE_ID_addressable_mutex_init_lockdep_555] map[Kind:global_variable Name:__UNIQUE_ID_addressable_mutex_is_locked_553] map[Kind:global_variable Name:__UNIQUE_ID_addressable_mutex_lock_interruptible_nested_579] map[Kind:global_variable Name:__UNIQUE_ID_addressable_mutex_lock_io_nested_580] map[Kind:global_variable Name:__UNIQUE_ID_addressable_mutex_lock_nested_576] map[Kind:global_variable Name:__UNIQUE_ID_addressable_mutex_unlock_568] map[Kind:global_variable Name:__UNIQUE_ID_addressable_ww_mutex_lock_581] map[Kind:global_variable Name:__UNIQUE_ID_addressable_ww_mutex_lock_interruptible_582] map[Kind:global_variable Name:__UNIQUE_ID_addressable_ww_mutex_trylock_575] map[Kind:global_variable Name:__UNIQUE_ID_addressable_ww_mutex_unlock_569] map[Kind:global_variable Name:__trace_check_contention_begin] map[Kind:global_variable Name:__trace_check_contention_end]]]
|
| 33/4 |
2026/05/18 18:36 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 32378
output: 42
thoughts: 13
Thoughts: Okay, let's dive into this. Here's my take on those thoughts, as if they're my own:
**Analyzing `__mutex_lock_common`**
Alright, I'm focusing in on `__mutex_lock_common`. This is a critical function, the bedrock of our mutex implementation. I need to understand its mechanics inside and out. It's the point where contention really begins to manifest, and where efficiency gains or bottlenecks will become most apparent. I'm thinking about the various architectural implications: the atomic operations used, the spin-waiting vs. sleep strategy decisions, and how this common code path is optimized for the typical, uncontended case. I know from experience that micro-optimizations here can have a macro impact on overall system performance. I need to be thinking about lock ordering, deadlock avoidance, and how this function plays with the kernel's scheduler. The goal is a deep grasp of how to identify, and fix, any potential locking issues this function could exhibit.
|
| 34/4 |
2026/05/18 18:36 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/locking/mutex.c IncludeLines:true Name:__mutex_lock_common]
Results: map[SourceCode: 605: /*
606: * Lock a mutex (possibly interruptible), slowpath:
607: */
608: static __always_inline int __sched
609: __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclass,
610: struct lockdep_map *nest_lock, unsigned long ip,
611: struct ww_acquire_ctx *ww_ctx, const bool use_ww_ctx)
612: __cond_acquires(0, lock)
613: {
614: DEFINE_WAKE_Q(wake_q);
615: struct mutex_waiter waiter;
616: struct ww_mutex *ww;
617: unsigned long flags;
618: int ret;
619:
620: if (!use_ww_ctx)
621: ww_ctx = NULL;
622:
623: might_sleep();
624:
625: MUTEX_WARN_ON(lock->magic != lock);
626:
627: ww = container_of(lock, struct ww_mutex, base);
628: if (ww_ctx) {
629: if (unlikely(ww_ctx == READ_ONCE(ww->ctx)))
630: return -EALREADY;
631:
632: /*
633: * Reset the wounded flag after a kill. No other process can
634: * race and wound us here since they can't have a valid owner
635: * pointer if we don't have any locks held.
636: */
637: if (ww_ctx->acquired == 0)
638: ww_ctx->wounded = 0;
639:
640: #ifdef CONFIG_DEBUG_LOCK_ALLOC
641: nest_lock = &ww_ctx->dep_map;
642: #endif
643: }
644:
645: preempt_disable();
646: mutex_acquire_nest(&lock->dep_map, subclass, 0, nest_lock, ip);
647:
648: trace_contention_begin(lock, LCB_F_MUTEX | LCB_F_SPIN);
649: if (__mutex_trylock(lock) ||
650: mutex_optimistic_spin(lock, ww_ctx, NULL)) {
651: /* got the lock, yay! */
652: lock_acquired(&lock->dep_map, ip);
653: if (ww_ctx)
654: ww_mutex_set_context_fastpath(ww, ww_ctx);
655: trace_contention_end(lock, 0);
656: preempt_enable();
657: return 0;
658: }
659:
660: raw_spin_lock_irqsave(&lock->wait_lock, flags);
661: /*
662: * After waiting to acquire the wait_lock, try again.
663: */
664: if (__mutex_trylock(lock)) {
665: if (ww_ctx)
666: __ww_mutex_check_waiters(lock, ww_ctx, &wake_q);
667:
668: goto skip_wait;
669: }
670:
671: debug_mutex_lock_common(lock, &waiter);
672: waiter.task = current;
673: if (use_ww_ctx)
674: waiter.ww_ctx = ww_ctx;
675:
676: lock_contended(&lock->dep_map, ip);
677:
678: if (!use_ww_ctx) {
679: /* add waiting tasks to the end of the waitqueue (FIFO): */
680: __mutex_add_waiter(lock, &waiter, NULL);
681: } else {
682: /*
683: * Add in stamp order, waking up waiters that must kill
684: * themselves.
685: */
686: ret = __ww_mutex_add_waiter(&waiter, lock, ww_ctx, &wake_q);
687: if (ret)
688: goto err_early_kill;
689: }
690:
691: raw_spin_lock(¤t->blocked_lock);
692: __set_task_blocked_on(current, lock);
693: set_current_state(state);
694: trace_contention_begin(lock, LCB_F_MUTEX);
695: for (;;) {
696: bool first;
697:
698: /*
699: * Once we hold wait_lock, we're serialized against
700: * mutex_unlock() handing the lock off to us, do a trylock
701: * before testing the error conditions to make sure we pick up
702: * the handoff.
703: */
704: if (__mutex_trylock(lock))
705: break;
706:
707: raw_spin_unlock(¤t->blocked_lock);
708: /*
709: * Check for signals and kill conditions while holding
710: * wait_lock. This ensures the lock cancellation is ordered
711: * against mutex_unlock() and wake-ups do not go missing.
712: */
713: if (signal_pending_state(state, current)) {
714: ret = -EINTR;
715: goto err;
716: }
717:
718: if (ww_ctx) {
719: ret = __ww_mutex_check_kill(lock, &waiter, ww_ctx);
720: if (ret)
721: goto err;
722: }
723:
724: raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q);
725:
726: schedule_preempt_disabled();
727:
728: first = lock->first_waiter == &waiter;
729:
730: raw_spin_lock_irqsave(&lock->wait_lock, flags);
731: raw_spin_lock(¤t->blocked_lock);
732: /*
733: * As we likely have been woken up by task
734: * that has cleared our blocked_on state, re-set
735: * it to the lock we are trying to acquire.
736: */
737: __set_task_blocked_on(current, lock);
738: set_current_state(state);
739: /*
740: * Here we order against unlock; we must either see it change
741: * state back to RUNNING and fall through the next schedule(),
742: * or we must see its unlock and acquire.
743: */
744: if (__mutex_trylock_or_handoff(lock, first))
745: break;
746:
747: if (first) {
748: bool opt_acquired;
749:
750: /*
751: * mutex_optimistic_spin() can call schedule(), so
752: * we need to release these locks before calling it,
753: * and clear blocked on so we don't become unselectable
754: * to run.
755: */
756: __clear_task_blocked_on(current, lock);
757: raw_spin_unlock(¤t->blocked_lock);
758: raw_spin_unlock_irqrestore(&lock->wait_lock, flags);
759:
760: trace_contention_begin(lock, LCB_F_MUTEX | LCB_F_SPIN);
761: opt_acquired = mutex_optimistic_spin(lock, ww_ctx, &waiter);
762:
763: raw_spin_lock_irqsave(&lock->wait_lock, flags);
764: raw_spin_lock(¤t->blocked_lock);
765: __set_task_blocked_on(current, lock);
766:
767: if (opt_acquired)
768: break;
769: trace_contention_begin(lock, LCB_F_MUTEX);
770: }
771: }
772: __clear_task_blocked_on(current, lock);
773: __set_current_state(TASK_RUNNING);
774: raw_spin_unlock(¤t->blocked_lock);
775:
776: if (ww_ctx) {
777: /*
778: * Wound-Wait; we stole the lock (!first_waiter), check the
779: * waiters as anyone might want to wound us.
780: */
781: if (!ww_ctx->is_wait_die && lock->first_waiter != &waiter)
782: __ww_mutex_check_waiters(lock, ww_ctx, &wake_q);
783: }
784:
785: __mutex_remove_waiter(lock, &waiter);
786:
787: debug_mutex_free_waiter(&waiter);
788:
789: skip_wait:
790: /* got the lock - cleanup and rejoice! */
791: lock_acquired(&lock->dep_map, ip);
792: trace_contention_end(lock, 0);
793:
794: if (ww_ctx)
795: ww_mutex_lock_acquired(ww, ww_ctx);
796:
797: raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q);
798: preempt_enable();
799: return 0;
800:
801: err:
802: clear_task_blocked_on(current, lock);
803: __set_current_state(TASK_RUNNING);
804: __mutex_remove_waiter(lock, &waiter);
805: err_early_kill:
806: WARN_ON(get_task_blocked_on(current));
807: trace_contention_end(lock, ret);
808: raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q);
809: debug_mutex_free_waiter(&waiter);
810: mutex_release(&lock->dep_map, ip);
811: preempt_enable();
812: return ret;
813: }
SourceFile:kernel/locking/mutex.c]
|
| 35/4 |
2026/05/18 18:36 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 35783
output: 19
thoughts: 84
Thoughts: **Analyzing This Code Snippet**
Okay, this code block is fairly extensive. My initial instinct is to zero in on a specific pattern that often signals a particular operation. I'm focusing on the `lea rdi, [rbx+0x18]` sequence, paired with the subsequent `mov esi, -1`. That pairing frequently indicates some form of initialization or pointer setup, specifically around a wait lock.
Now, that `rbx+0x18` address... that’s piquing my interest. Could `rbx` be related to a mutex structure, and `+0x18` pointing to its `wait_lock` field? Seems plausible. I'm going to quickly cross-reference the `struct mutex` definition. I want to confirm the offset of that `wait_lock` member within the struct. Knowing the layout precisely will give me a better understanding of the overall purpose of this code. It's time to verify my hypothesis.
|
| 36/4 |
2026/05/18 18:36 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:owner OffsetBits:0 SizeBits:64] map[Name:wait_lock OffsetBits:64 SizeBits:512] map[Name:osq OffsetBits:576 SizeBits:32] map[Name:first_waiter OffsetBits:640 SizeBits:64] map[Name:magic OffsetBits:704 SizeBits:64] map[Name:dep_map OffsetBits:768 SizeBits:320]]]
|
| 37/4 |
2026/05/18 18:36 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 36225
output: 41
thoughts: 232
Thoughts: **Decoding That Memory Access**
Okay, so I'm looking at these memory offsets for this `struct mutex`, and something doesn't quite line up. I've got `wait_lock` at 8 bytes, `osq` at 72, `first_waiter` at 80... and now I see this offset of `0x18` (24 bytes) that's just hanging out, right? Seems out of place for a defined field within the mutex itself.
Wait a second... that instruction, `48 8d 7b 18`, is a `lea rdi, [rbx+0x18]`. The Register `rbx` immediately suggests `struct task_struct`, which I know like the back of my hand. And didn't I see earlier that `__state` is at 24 bytes (0x18) into the `task_struct`? Bingo! That `lea` is loading the address of `current->__state` into `rdi`.
So, what function could possibly be taking `¤t->__state` and `-1` as arguments? Gotta be something important. I recall seeing that combination used with `signal_pending_state(state, current)` from line 713 (which is, obviously, often inlined for performance). I should probably go back and refresh my memory on the inline definition of that function, as that will offer more insight. This is making more sense now.
|
| 38/4 |
2026/05/18 18:36 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/locking/mutex.c IncludeLines:true Name:signal_pending_state]
Results: map[SourceCode: 409: static inline int signal_pending_state(unsigned int state, struct task_struct *p)
410: {
411: if (!(state & (TASK_INTERRUPTIBLE | TASK_WAKEKILL)))
412: return 0;
413: if (!signal_pending(p))
414: return 0;
415:
416: return (state & TASK_INTERRUPTIBLE) || __fatal_signal_pending(p);
417: }
SourceFile:include/linux/sched/signal.h]
|
| 39/4 |
2026/05/18 18:36 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 36988
output: 44
thoughts: 797
Thoughts: **Deconstructing the Code**
Okay, so I'm looking at this snippet, and initially, the `lea rdi, [rbx+0x18]` and `mov esi, -1` threw me for a loop. It didn't immediately click. Let's think... that `be ff ff ff ff` is indeed `mov esi, -1`. Was that `__refrigerator(false)`? No. Let's look at the assembly...
Okay, I see these function calls: `e8 ce a2 21 0a` -> `Call2`, `41 89 c5` -> `mov r13d, eax`, `31 ff` -> `xor edi, edi`, `89 c6` -> `mov esi, eax`, `e8 d2 fd 11 00` -> `Call3`.
Aha! `xor edi, edi` and `mov esi, eax` means `Call3(0, eax)`. That `eax` is the return of `Call2`. Okay. Is `Call2` `schedule_timeout`? No, it takes a `long`.
Wait, the `lea rdi, [rbx+0x18]` is a key clue! It's `48 8d 7b 18`. If `rbx` is a `struct wait_queue_entry`, then `0x18` is an offset of 24 bytes, and `struct wait_queue_entry` has `entry` there, which is a `list_head`. So `lea rdi, [rbx+0x18]` is `&waiter->entry`.
So what function might take a `&waiter->entry` and `-1`? `list_del_init`? No, no.
Hmm... I can't use the hex string in the repo.
I can't use `codesearch-find-references` on `__SCT__tp_func_contention_begin`.
Let's look at the `Code:` again.
`ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d`
Okay, `45 85 ed` is `test r13d, r13d`. If `r13d` contains the return value... it's a conditional: `if (ret) { ... }`. So it's: `ret = Call2(&waiter->entry, -1);`, `Call3(0, ret);` and `if (ret == 0) goto ...`.
Right! `0f 84 33 02 00 00` is `jz`. If `ret == 0`, it jumps!
Aha! I think I've got it! This looks like something related to `io_uring` and waiting on the completion queue. I'll bet it's similar to `io_cqring_wait`. Let's confirm it, by looking at `io_cqring_wait_schedule` in `io_uring/wait.c`.
|
| 40/4 |
2026/05/18 18:36 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:io_uring/wait.c IncludeLines:true Name:io_cqring_wait_schedule]
Results: map[SourceCode: 165: /* If this returns > 0, the caller should retry */
166: static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,
167: struct io_wait_queue *iowq,
168: struct ext_arg *ext_arg,
169: ktime_t start_time)
170: {
171: if (unlikely(READ_ONCE(ctx->check_cq)))
172: return 1;
173: if (unlikely(io_local_work_pending(ctx)))
174: return 1;
175: if (unlikely(task_work_pending(current)))
176: return 1;
177: if (unlikely(task_sigpending(current)))
178: return -EINTR;
179: if (unlikely(io_should_wake(iowq)))
180: return 0;
181:
182: return __io_cqring_wait_schedule(ctx, iowq, ext_arg, start_time);
183: }
SourceFile:io_uring/wait.c]
|
| 41/4 |
2026/05/18 18:36 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 38991
output: 45
thoughts: 16
Thoughts: Okay, let's dive into this `__io_cqring_wait_schedule` function. I need to understand this kernel function thoroughly, as it's critical for I/O completion queue management, and that's a cornerstone of high-performance storage. First, I need to remind myself of the broader context. Where does this fit within the larger I/O stack? Is it a core scheduler function, or more of a low-level completion handler?
I'm thinking about the implications for different hardware architectures. Does this function have specific optimizations for different NVMe controllers or SAS HBAs? I'll need to check the surrounding code for architecture-specific branches or inline assembly. I'm also wondering about the potential for contention. How does this function handle concurrent accesses to the completion queue? Are there spinlocks, atomic operations, or other synchronization mechanisms at play?
More specifically, I need to know the waiting mechanism. Is it a busy-wait loop, or is it more sophisticated using something like a wait queue? What are the implications in terms of CPU power consumption and overall system responsiveness? If it's a sleep-based mechanism, what's the typical timeout interval, and how is it calculated? I want to understand the trade-offs between responsiveness and efficiency.
Finally, I want to see if any of the performance counters are affected. If so, which ones? I'm anticipating that understanding the impact on counters will be key to optimizing performance. I will be looking for any hints about potential bottlenecks, and any opportunities for improvement.
|
| 42/4 |
2026/05/18 18:36 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:io_uring/wait.c IncludeLines:true Name:__io_cqring_wait_schedule]
Results: map[SourceCode: 143: static int __io_cqring_wait_schedule(struct io_ring_ctx *ctx,
144: struct io_wait_queue *iowq,
145: struct ext_arg *ext_arg,
146: ktime_t start_time)
147: {
148: int ret = 0;
149:
150: /*
151: * Mark us as being in io_wait if we have pending requests, so cpufreq
152: * can take into account that the task is waiting for IO - turns out
153: * to be important for low QD IO.
154: */
155: if (ext_arg->iowait && current_pending_io())
156: current->in_iowait = 1;
157: if (iowq->timeout != KTIME_MAX || iowq->min_timeout)
158: ret = io_cqring_schedule_timeout(iowq, ctx->clockid, start_time);
159: else
160: schedule();
161: current->in_iowait = 0;
162: return ret;
163: }
SourceFile:io_uring/wait.c]
|
| 43/4 |
2026/05/18 18:36 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 39745
output: 37
thoughts: 1504
Thoughts: **Dissecting This Code Snippet**
Okay, this code isn't immediately obvious. It doesn't seem to fit any known pattern, so I need to carefully disassemble it. Let's start from the top again and break it down instruction by instruction.
Right, the `e8` calls are key. The proximity of `e8 d2 fd 11 00` and `e8 f6 fd 11 00` suggests they're likely calls to similar functions, potentially related to tracing. The differences are small, hinting at possible `trace_...` calls. But wait, these are direct calls, not conditional ones. That's unusual. It's also not `__fentry__`, the location is wrong.
Now, let me look at the `lea rdi, [rbx+0x18]` and `mov esi, -1` instructions. If `rbx` is a `wait_queue_entry`, then `0x18` makes sense but I have other thoughts about this. If `rbx` represents a mutex, 24 bytes could refer to a member within the wait lock. And if the value is negative one, then that translates to a timeout value.
Ah, I remember that `schedule_timeout(-1)` is actually `schedule_timeout(MAX_SCHEDULE_TIMEOUT)`! That resolves the `-1` issue. However, I still need to understand what type of timeout this refers to.
I'll search the kernel for the byte sequence `48 8d 7b 18 be ff ff ff ff` around `mutex_lock`. No luck, that's a dead end.
I'm going to revisit the disassembly. The `test r13d, r13d` and conditional jumps are important. `r13d` might be `ret`, so we have an `if (ret) {...}` structure. It looks like we're doing something like `ret = Call2(rbx+0x18, -1);` and then `Call3(0, ret);` with a conditional based on the return value.
Now, `Call3(0, ret)`... This looks very similar to tracing, particularly `trace_sched_stat_wait(current, ret)`. But wait, that takes a `struct task_struct *p`, but `0` is probably not the current task!
How about `trace_contention_end(lock, ret)`? That takes `(void *lock, int ret)`. I've got `xor edi, edi` and then `mov esi, eax`. The `0` could be assigned to a lock. I need to be careful with the register assignments, and compare this to `kernel/locking/mutex.c` line 807 where `trace_contention_end(lock, ret)` might appear.
If `rbx` is the lock, then `lea rdi, [rbx+0x18]` is `&lock->something`. But what is at offset `0x18` within `struct mutex`? I need to check the current version of the structure. I'm checking different offsets, but I'm looking at an old version of the `struct mutex`. I need to consult the most recent source code. Let's see...
|
| 44/4 |
2026/05/18 18:37 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/linux/mutex.h IncludeLines:true Name:mutex]
Results: map[SourceCode: 13: /*
14: * Simple, straightforward mutexes with strict semantics:
15: *
16: * - only one task can hold the mutex at a time
17: * - only the owner can unlock the mutex
18: * - multiple unlocks are not permitted
19: * - recursive locking is not permitted
20: * - a mutex object must be initialized via the API
21: * - a mutex object must not be initialized via memset or copying
22: * - task may not exit with mutex held
23: * - memory areas where held locks reside must not be freed
24: * - held mutexes must not be reinitialized
25: * - mutexes may not be used in hardware or software interrupt
26: * contexts such as tasklets and timers
27: *
28: * These semantics are fully enforced when DEBUG_MUTEXES is
29: * enabled. Furthermore, besides enforcing the above rules, the mutex
30: * debugging code also implements a number of additional features
31: * that make lock debugging easier and faster:
32: *
33: * - uses symbolic names of mutexes, whenever they are printed in debug output
34: * - point-of-acquire tracking, symbolic lookup of function names
35: * - list of all locks held in the system, printout of them
36: * - owner tracking
37: * - detects self-recursing locks and prints out all relevant info
38: * - detects multi-task circular deadlocks and prints out all affected
39: * locks and tasks (and only those tasks)
40: */
41: context_lock_struct(mutex) {
42: atomic_long_t owner;
43: raw_spinlock_t wait_lock;
44: #ifdef CONFIG_MUTEX_SPIN_ON_OWNER
45: struct optimistic_spin_queue osq; /* Spinner MCS lock */
46: #endif
47: struct mutex_waiter *first_waiter __guarded_by(&wait_lock);
48: #ifdef CONFIG_DEBUG_MUTEXES
49: void *magic;
50: #endif
51: #ifdef CONFIG_DEBUG_LOCK_ALLOC
52: struct lockdep_map dep_map;
53: #endif
54: };
SourceFile:include/linux/mutex_types.h]
|
| 45/4 |
2026/05/18 18:37 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 42662
output: 22
thoughts: 299
Thoughts: **Dissecting the Mutex Structure**
Okay, let's break down this mutex structure. My initial assumptions were based on standard x86_64 conventions, but let's be precise. I see `owner` as an `atomic_long_t`, so that's 8 bytes—standard for a 64-bit architecture, naturally. The `wait_lock` is a `raw_spinlock_t`, which, on x86_64, usually clocks in at 4 bytes. `osq`, a `struct optimistic_spin_queue`, is an `atomic_t`, so another 4 bytes. And `first_waiter` is a pointer, another 8 bytes in this 64-bit context.
So, adding it up, we have: `owner (8) + wait_lock (4) + osq (4) + first_waiter (8) = 24 bytes`. Now, the instruction `lea rdi, [rbx+0x18]` – that's 0x18, or 24 decimal, offset from `rbx`, which likely points to the mutex. That implies the member at this offset must be the "magic" number I am seeing.
However, the real puzzle is: Why would anyone take the *address* of something like "magic"? Perhaps my initial assumption about "magic" being the correct field is incorrect. What's the context here?
Aha! `CONFIG_MUTEX_SPIN_ON_OWNER` can change things; if that's *not* defined, then the `osq` field is actually *not* present, so we're starting with a different structure size! Further, if `CONFIG_DEBUG_MUTEXES` isn't defined, then the "magic" value I'm looking at isn't there, either! That complicates the memory layout.
Wait a moment, I think I'm onto something. If `CONFIG_DEBUG_LOCK_ALLOC` *is* defined, then the `dep_map` is there! And I know that thing can be *quite* large. I need to get the full picture of `struct lockdep_map` to understand the true size implications of this. I have to see how that affects the structure.
|
| 46/4 |
2026/05/18 18:37 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:key OffsetBits:0 SizeBits:64] map[Name:class_cache OffsetBits:64 SizeBits:128] map[Name:name OffsetBits:192 SizeBits:64] map[Name:wait_type_outer OffsetBits:256 SizeBits:8] map[Name:wait_type_inner OffsetBits:264 SizeBits:8] map[Name:lock_type OffsetBits:272 SizeBits:8]]]
|
| 47/4 |
2026/05/18 18:37 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 43611
output: 24
thoughts: 1039
Thoughts: **Dissecting the Assembly: A Mutex Locking Mystery**
Okay, let's break this down. First, I'm noting the structure of `lockdep_map`. Key, class cache, name – that's 40 bytes total. If `dep_map` sits at offset 24, as it seems, then the instruction `lea rdi, [rbx+0x18]` is definitely pointing to `&lock->dep_map`. We have owner, wait_lock, osq, first_waiter and *then* the `dep_map` at offset 24, which aligns with the conditions: `CONFIG_MUTEX_SPIN_ON_OWNER` and *no* `CONFIG_DEBUG_MUTEXES`.
So, `rdi` is confirmed as `&lock->dep_map`. The next clue is the `-1`, or `0xffffffff`. I need to figure out what function takes `&lock->dep_map` and a value that could be interpreted as `-1`.
My mind jumps to `mutex_acquire_nest`, but the `subclass` argument seems unlikely to be `-1` - that wouldn't make sense. Then, I remember the more general `lock_acquire` with its plethora of arguments: a `struct lockdep_map`, an `unsigned int subclass`, the trylock/read/check flags, and a nested lock.
Looking back at the assembly... `Call2` seems interesting. We only see `rdi` and `esi` being set: `rdi` as our `&lock->dep_map` and `esi` as `-1`. This strongly suggests that our function must take exactly those two arguments: `(lockdep_map*, int)`. I started looking into tracepoint calls.
`trace_contention_begin(lock, flags)` seems a possibility with a `(void *lock, unsigned int flags)` structure. However, the `flags` being -1 would not hold true.
So, I decided to focus on searching for the key byte sequence `48 8d 7b 18 be ff ff ff ff e8` that contains the `lea rdi, [rbx+0x18]` and `mov esi, -1` sequences. Searching for references to `__mutex_lock` directly seems difficult.
Maybe, the call comes from `__mutex_lock_common` in `kernel/locking/mutex.c`. But `trace_contention_begin` uses `LCB_F_MUTEX | LCB_F_SPIN` (1 | 2 = 3), not -1. So, that's not it.
Okay, let's rethink that `-1`... Could it *actually* be something other than -1 in a 64-bit context? Yes, `0xffffffff` is a large positive value in 64-bit.
I'll try to find the target of the `Call2` instruction. If the offset is `0x0a21a2ce`, that's huge! It's likely a call to something not in this current code section - a tracepoint or a function in a different part of the kernel.
I'll change tack and search for the byte sequence that precedes that call, `45 85 ed 48 89 5c 24 08 74 2a`, using `codesearch-find-references` on `mutex_optimistic_spin`. Let's see if that gets us anywhere.
|
| 48/4 |
2026/05/18 18:37 |
tool |
codesearch-find-references |
0m
Args: map[Name:mutex_optimistic_spin]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__mutex_lock_common SourceFile:kernel/locking/mutex.c SourceLine:650] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__mutex_lock_common SourceFile:kernel/locking/mutex.c SourceLine:761]] TruncatedOutput:false]
|
| 49/4 |
2026/05/18 18:37 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 45568
output: 21
thoughts: 622
Thoughts: **Analyzing the Mutex Lock and Optimistic Spin**
Okay, let's dive into this `__mutex_lock_common` function, specifically around line 761. I see `trace_contention_begin` there, which I know is a macro. It expands to check `trace_contention_begin_enabled` and then the actual tracepoint. This looks like a static branch, so I should keep that in mind.
I'm examining the assembly code now. The snippet has `ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d`. `45 85 ed` catches my eye; it's `test r13d, r13d`. If `r13d` is `opt_acquired`, then that `if (opt_acquired) break;` on line 767 makes perfect sense.
So, the `Call2` instruction I see, could that be `mutex_optimistic_spin`? The function signature is `(struct mutex *lock, struct ww_acquire_ctx *ww_ctx, struct mutex_waiter *waiter)`. The arguments are passed via registers: `rdi = lock`, `rsi = ww_ctx`, and `rdx = waiter`. However, looking at the snippet, I see `rdi = rbx + 0x18`, `rsi = -1`, and then `Call2`. That doesn't match the expected argument passing for `mutex_optimistic_spin`. `rsi = -1` makes no sense.
Okay, let's re-evaluate. Could `Call2` be `trace_contention_begin` instead? That function takes `(struct mutex *lock, LCB_F_MUTEX | LCB_F_SPIN)` or `(rdi = lock, rsi = 3)`. Still, the immediate operand `be ff ff ff ff` doesn't fit with this function either.
Alright, I'll switch gears and try a different approach. I'll search for the instruction sequence `48 8d 7b 18 be ff ff ff ff`. Perhaps the `rbx` register holds a `struct rw_semaphore` instance? It would make sense given the arguments. Time to check the layout of `struct rw_semaphore`... This should give me some clues as to what's going on!
|
| 50/4 |
2026/05/18 18:37 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:count OffsetBits:0 SizeBits:64] map[Name:owner OffsetBits:64 SizeBits:64] map[Name:osq OffsetBits:128 SizeBits:32] map[Name:wait_lock OffsetBits:192 SizeBits:512] map[Name:first_waiter OffsetBits:704 SizeBits:64] map[Name:magic OffsetBits:768 SizeBits:64] map[Name:dep_map OffsetBits:832 SizeBits:320]]]
|
| 51/4 |
2026/05/18 18:37 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 47016
output: 27
thoughts: 119
Thoughts: **Analyzing the Read-Write Semaphore Lock**
Okay, let's break this down. I'm seeing `count`, `owner`, and `osq` showing up frequently, eight times and four times respectively. Right, so `wait_lock` is at 192 bits – that's 24 bytes, or `0x18` in hex. Makes sense. Therefore, the instruction `lea rdi, [rbx+0x18]` is effectively getting the address of `rwsem->wait_lock`.
Now, the question is, what function is being called, passing the address of that `wait_lock` and a value of `-1`? The binary representation `be ff ff ff ff` immediately tells me this is the value `-1`. Let's confirm with the source.
Okay, time to consult `kernel/locking/rwsem.c`. That's where the read-write semaphore implementation should be. Let's see what that address and value are for.
|
| 52/4 |
2026/05/18 18:37 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/locking/rwsem.c]
Results: map[Entities:[map[Kind:function Name:__down_read] map[Kind:function Name:__down_read_common] map[Kind:function Name:__down_read_interruptible] map[Kind:function Name:__down_read_killable] map[Kind:function Name:__down_read_trylock] map[Kind:function Name:__down_write] map[Kind:function Name:__down_write_common] map[Kind:function Name:__down_write_killable] map[Kind:function Name:__down_write_trylock] map[Kind:function Name:__downgrade_write] map[Kind:function Name:__init_rwsem] map[Kind:function Name:__rwsem_del_waiter] map[Kind:function Name:__rwsem_set_reader_owned] map[Kind:function Name:__up_read] map[Kind:function Name:__up_write] map[Kind:function Name:_down_write_nest_lock] map[Kind:function Name:clear_nonspinnable] map[Kind:function Name:down_read] map[Kind:function Name:down_read_interruptible] map[Kind:function Name:down_read_killable] map[Kind:function Name:down_read_killable_nested] map[Kind:function Name:down_read_nested] map[Kind:function Name:down_read_non_owner] map[Kind:function Name:down_read_trylock] map[Kind:function Name:down_write] map[Kind:function Name:down_write_killable] map[Kind:function Name:down_write_killable_nested] map[Kind:function Name:down_write_nested] map[Kind:function Name:down_write_trylock] map[Kind:function Name:downgrade_write] map[Kind:function Name:is_rwsem_reader_owned] map[Kind:function Name:next_waiter] map[Kind:function Name:rwsem_can_spin_on_owner] map[Kind:function Name:rwsem_clear_owner] map[Kind:function Name:rwsem_clear_reader_owned] map[Kind:function Name:rwsem_cond_wake_waiter] map[Kind:function Name:rwsem_del_waiter] map[Kind:function Name:rwsem_del_wake_waiter] map[Kind:function Name:rwsem_down_read_slowpath] map[Kind:function Name:rwsem_down_write_slowpath] map[Kind:function Name:rwsem_downgrade_wake] map[Kind:function Name:rwsem_mark_wake] map[Kind:function Name:rwsem_optimistic_spin] map[Kind:function Name:rwsem_owner] map[Kind:function Name:rwsem_owner_flags] map[Kind:function Name:rwsem_owner_state] map[Kind:function Name:rwsem_read_trylock] map[Kind:function Name:rwsem_rspin_threshold] map[Kind:function Name:rwsem_set_nonspinnable] map[Kind:function Name:rwsem_set_owner] map[Kind:function Name:rwsem_set_reader_owned] map[Kind:function Name:rwsem_spin_on_owner] map[Kind:function Name:rwsem_test_oflags] map[Kind:function Name:rwsem_try_write_lock] map[Kind:function Name:rwsem_try_write_lock_unqueued] map[Kind:function Name:rwsem_wake] map[Kind:function Name:rwsem_write_trylock] map[Kind:function Name:up_read] map[Kind:function Name:up_read_non_owner] map[Kind:function Name:up_write] map[Kind:struct Name:rwsem_waiter] map[Kind:global_variable Name:__UNIQUE_ID_addressable___init_rwsem_334] map[Kind:global_variable Name:__UNIQUE_ID_addressable__down_write_nest_lock_364] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_read_352] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_read_interruptible_353] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_read_killable_354] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_read_killable_nested_363] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_read_nested_362] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_read_non_owner_365] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_read_trylock_355] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_write_356] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_write_killable_357] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_write_killable_nested_367] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_write_nested_366] map[Kind:global_variable Name:__UNIQUE_ID_addressable_down_write_trylock_358] map[Kind:global_variable Name:__UNIQUE_ID_addressable_downgrade_write_361] map[Kind:global_variable Name:__UNIQUE_ID_addressable_up_read_359] map[Kind:global_variable Name:__UNIQUE_ID_addressable_up_read_non_owner_369] map[Kind:global_variable Name:__UNIQUE_ID_addressable_up_write_360] map[Kind:enum Name:owner_state] map[Kind:enum Name:rwsem_waiter_type] map[Kind:enum Name:rwsem_wake_type]]]
|
| 53/4 |
2026/05/18 18:37 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 49055
output: 46
thoughts: 16
Thoughts: Okay, let's dive into this `rwsem_down_write_slowpath` function. I've seen this before, and it's always interesting to see how the kernel handles the write semaphore when things aren't straightforward. My immediate thought is, "Ah, the slow path... gotta be some contention or a special case involved here." I need to understand what triggers this specific path and what it's trying to achieve. Is this about blocking, spinning, or some form of fairness guarantee?
I immediately start thinking about potential scenarios: Are there existing readers holding the semaphore? Are there writers waiting ahead of me? Is there a priority boost somewhere? I need to analyze how it differentiates from the fast path, what the actual *symptoms* are that land it here. My experience tells me that slow paths frequently handle edge cases, error conditions, or situations demanding more complex resource management. Perhaps there's an attempt at deadlock prevention or starvation avoidance within. I'll need to look at the surrounding code to determine the exact conditions, and how its data structures are impacted.
|
| 54/4 |
2026/05/18 18:37 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/locking/rwsem.c IncludeLines:true Name:rwsem_down_write_slowpath]
Results: map[SourceCode:1135: /*
1136: * Wait until we successfully acquire the write lock
1137: */
1138: static struct rw_semaphore __sched *
1139: rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)
1140: {
1141: struct rwsem_waiter waiter, *first;
1142: DEFINE_WAKE_Q(wake_q);
1143:
1144: /* do optimistic spinning and steal lock if possible */
1145: if (rwsem_can_spin_on_owner(sem) && rwsem_optimistic_spin(sem)) {
1146: /* rwsem_optimistic_spin() implies ACQUIRE on success */
1147: return sem;
1148: }
1149:
1150: /*
1151: * Optimistic spinning failed, proceed to the slowpath
1152: * and block until we can acquire the sem.
1153: */
1154: waiter.task = current;
1155: waiter.type = RWSEM_WAITING_FOR_WRITE;
1156: waiter.timeout = jiffies + RWSEM_WAIT_TIMEOUT;
1157: waiter.handoff_set = false;
1158:
1159: raw_spin_lock_irq(&sem->wait_lock);
1160:
1161: first = sem->first_waiter;
1162: if (first) {
1163: list_add_tail(&waiter.list, &first->list);
1164: rwsem_cond_wake_waiter(sem, atomic_long_read(&sem->count),
1165: &wake_q);
1166: if (!wake_q_empty(&wake_q)) {
1167: /*
1168: * We want to minimize wait_lock hold time especially
1169: * when a large number of readers are to be woken up.
1170: */
1171: raw_spin_unlock_irq(&sem->wait_lock);
1172: wake_up_q(&wake_q);
1173: raw_spin_lock_irq(&sem->wait_lock);
1174: }
1175: } else {
1176: INIT_LIST_HEAD(&waiter.list);
1177: sem->first_waiter = &waiter;
1178: atomic_long_or(RWSEM_FLAG_WAITERS, &sem->count);
1179: }
1180:
1181: /* wait until we successfully acquire the lock */
1182: set_current_state(state);
1183: trace_contention_begin(sem, LCB_F_WRITE);
1184:
1185: if (state == TASK_UNINTERRUPTIBLE)
1186: hung_task_set_blocker(sem, BLOCKER_TYPE_RWSEM_WRITER);
1187:
1188: for (;;) {
1189: if (rwsem_try_write_lock(sem, &waiter)) {
1190: /* rwsem_try_write_lock() implies ACQUIRE on success */
1191: break;
1192: }
1193:
1194: raw_spin_unlock_irq(&sem->wait_lock);
1195:
1196: if (signal_pending_state(state, current))
1197: goto out_nolock;
1198:
1199: /*
1200: * After setting the handoff bit and failing to acquire
1201: * the lock, attempt to spin on owner to accelerate lock
1202: * transfer. If the previous owner is a on-cpu writer and it
1203: * has just released the lock, OWNER_NULL will be returned.
1204: * In this case, we attempt to acquire the lock again
1205: * without sleeping.
1206: */
1207: if (waiter.handoff_set) {
1208: enum owner_state owner_state;
1209:
1210: owner_state = rwsem_spin_on_owner(sem);
1211: if (owner_state == OWNER_NULL)
1212: goto trylock_again;
1213: }
1214:
1215: schedule_preempt_disabled();
1216: lockevent_inc(rwsem_sleep_writer);
1217: set_current_state(state);
1218: trylock_again:
1219: raw_spin_lock_irq(&sem->wait_lock);
1220: }
1221:
1222: if (state == TASK_UNINTERRUPTIBLE)
1223: hung_task_clear_blocker();
1224:
1225: __set_current_state(TASK_RUNNING);
1226: raw_spin_unlock_irq(&sem->wait_lock);
1227: lockevent_inc(rwsem_wlock);
1228: trace_contention_end(sem, 0);
1229: return sem;
1230:
1231: out_nolock:
1232: __set_current_state(TASK_RUNNING);
1233: raw_spin_lock_irq(&sem->wait_lock);
1234: rwsem_del_wake_waiter(sem, &waiter, &wake_q);
1235: lockevent_inc(rwsem_wlock_fail);
1236: trace_contention_end(sem, -EINTR);
1237: return ERR_PTR(-EINTR);
1238: }
SourceFile:kernel/locking/rwsem.c]
|
| 55/4 |
2026/05/18 18:37 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 50962
output: 19
thoughts: 858
Thoughts: **Dissecting a Kernel Code Snippet**
Okay, so I'm looking at this assembly and it's not immediately clicking. The initial instructions, specifically `lea rdi, [rbx+0x18]` and `mov esi, -1`, aren't quite lining up with my expectations. Time to dig deeper. I'll start by searching the kernel source for that byte sequence `48 8d 7b 18 be ff ff ff ff e8`. I'll use `codesearch-find-references` on `schedule_timeout`, just to get a broader context.
Hang on... I should analyze `schedule_timeout` assembly directly, maybe there's something to glean there.
Aha! That `be ff ff ff ff` is a dead giveaway: `mov esi, -1`. My initial hunch about this being related to timeout is likely correct. Could this be `schedule_timeout(MAX_SCHEDULE_TIMEOUT)`? Let's see... `MAX_SCHEDULE_TIMEOUT` is, of course, `0x7fffffffffffffff`, but `be ff ff ff ff` is a 32-bit `0xffffffff`. Ah, right, that's `-1`. If the function expects an `int`, `-1` is correct. If it's a `long`, then it would be `0x00000000ffffffff`.
Eureka! I think I've got something. I've found a sequence! I'm seeing `e8 ce a2 21 0a` which is `Call2`, followed by `41 89 c5` which is `mov r13d, eax`, then `31 ff` for `xor edi, edi`, and finally, `89 c6` which is `mov esi, eax`. And finally `e8 d2 fd 11 00` which is `Call3`.
This looks like `Call3(0, ret)` where `ret` is the return value of `Call2`. My gut says that this must be `trace_contention_end(sem, ret)`. If `edi` is zero, then `sem` is NULL. That's interesting, this suggests contention tracing is possibly being called with a NULL lock!
I need to see the conditional. Now, I should look for `45 85 ed 0f 84 33 02 00 00`, which is `test r13d, r13d`, and then `jz +0x233`.
Oh wait, I've already done that search! Right, I was using `codesearch-find-references` with `__mutex_lock_common` and I already tried the `test r13d, r13d` sequence.
Okay, let's go back and look at the `Code:` line again, and walk through it. I see `ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d`.
Hmm, this `e8 84 f9 11 00` and `e8 7d f9 11 00` are close, just 7 bytes different! And `eb 05` is `jmp +5`. This is likely a simple `if (condition) func1(); else func2();` pattern. The fact that the two calls are so close indicates that `func1` and `func2` are close in memory, and that's a common pattern with tracing.
I'll try searching for `48 8d 7b 18 be ff ff ff ff`, that should help define what `rbx` is. Perhaps `rbx` is a `struct socket`? That seems like a logical next step - I should check the `struct socket` layout and see if the offset at `0x18` makes sense.
|
| 56/4 |
2026/05/18 18:37 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:state OffsetBits:0 SizeBits:32] map[Name:type OffsetBits:32 SizeBits:16] map[Name:flags OffsetBits:64 SizeBits:64] map[Name:file OffsetBits:128 SizeBits:64] map[Name:sk OffsetBits:192 SizeBits:64] map[Name:ops OffsetBits:256 SizeBits:64] map[Name:wq OffsetBits:512 SizeBits:1024]]]
|
| 57/4 |
2026/05/18 18:37 |
llm |
codeexpert |
4m
Model:
Tokens:
input: 52970
output: 810
thoughts: 62910
|
| 58/2 |
2026/05/18 18:42 |
llm |
debugger |
0m
Model:
Tokens:
input: 34610
output: 43
thoughts: 233
Thoughts: **Dissecting a Potential Code Issue**
Okay, here's what's going through my mind as I analyze this code snippet. I initially saw `out dx, al` being represented as `ee`. My first thought was, "That's odd, `ee` *is* the opcode for `out dx, al`." But then I wondered if there's a potential prefix or a multi-byte instruction sequence, so I looked into it. Specifically, is `ee` part of a longer instruction like `... ee e8 ...`? Perhaps the `ee` is a remnant.
Then, I shifted my focus to `__remove_hrtimer`. I recall it has `lockdep_assert_held(&cpu_base->lock);`, so I want to make sure I'm following the proper lock discipline. I need to make sure I'm grasping the intent of this function. I'll read it one more time.
```c
if (!timer->is_queued)
return;
/* Pairs with the lockless read in hrtimer_is_queued() */
WRITE_ONCE(timer->is_queued, newstate);
was_first = !timerqueue_linked_prev(&timer->node);
if (!timerqueue_linked_del(&base->active, &timer->node))
cpu_base->active_bases &= ~(1 << base->index);
```
Okay, I see the conditional checks and the `WRITE_ONCE` usage. It seems like the core operation here is related to `timerqueue_linked_del`.
Given the surrounding context, I suspect the issue could be within `timerqueue_linked_del`. I'll delve deeper into the implementation of `timerqueue_linked_del` and examine its definition in `include/linux/timerqueue.h`. This feels like the right place to focus my investigation.
|
| 59/2 |
2026/05/18 18:42 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/linux/timerqueue.h IncludeLines:true Name:timerqueue_linked_del]
Results: map[SourceCode: 63: static __always_inline
64: bool timerqueue_linked_del(struct timerqueue_linked_head *head, struct timerqueue_linked_node *node)
65: {
66: return rb_erase_linked(&node->node, &head->rb_root);
67: }
SourceFile:include/linux/timerqueue.h]
|
| 60/2 |
2026/05/18 18:42 |
llm |
debugger |
0m
Model:
Tokens:
input: 35408
output: 39
thoughts: 57
Thoughts: Here's my summary, as though I'm thinking through this process:
**Diving into Timer Queue Deletion**
Okay, hold on a second. I thought `timerqueue_linked_del` was the one that ultimately called `rb_erase_linked`, but now I'm getting a flash that it's actually `timerqueue_del` that handles the red-black tree removal via `rb_erase_cached`. I need to confirm this before I go any further. Let me just double-check that by heading straight over to `lib/timerqueue.c` and digging into the source code for the `timerqueue_del` function. Gotta make sure I'm on the right track before assuming anything. Gotta stay sharp!
|
| 61/2 |
2026/05/18 18:42 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/timerqueue.c IncludeLines:true Name:timerqueue_del]
Results: map[SourceCode: 44: /**
45: * timerqueue_del - Removes a timer from the timerqueue.
46: *
47: * @head: head of timerqueue
48: * @node: timer node to be removed
49: *
50: * Removes the timer node from the timerqueue. Returns true if the queue is
51: * not empty after the remove.
52: */
53: bool timerqueue_del(struct timerqueue_head *head, struct timerqueue_node *node)
54: {
55: WARN_ON_ONCE(RB_EMPTY_NODE(&node->node));
56:
57: rb_erase_cached(&node->node, &head->rb_root);
58: RB_CLEAR_NODE(&node->node);
59:
60: return !RB_EMPTY_ROOT(&head->rb_root.rb_root);
61: }
SourceFile:lib/timerqueue.c]
|
| 62/2 |
2026/05/18 18:42 |
llm |
debugger |
0m
Model:
Tokens:
input: 35943
output: 41
thoughts: 292
Thoughts: **Delving into Timerqueue Functions**
Okay, first thought: Hang on, where *is* `timerqueue_linked_del`? I was sure I saw it in `lib/timerqueue.c`, but it's not there. Oh, right, it's more likely in `kernel/time/hrtimer.c`, isn't it? Let me just double-check… hmm, `timerqueue_del` or `timerqueue_linked_del`? Wait, is `timerqueue_linked_del` even a *standard* function? That feels off.
Let me just quickly verify in `kernel/time/hrtimer.c`… Aha! Line 1155 says: `if (!timerqueue_del(&base->active, &timer->node))`. Okay, that's interesting.
My previous codesearch for `__remove_hrtimer` seemed to show… let me check that... Ah, no, it showed:
```c
1153: was_first = !timerqueue_linked_prev(&timer->node);
1154:
1155: if (!timerqueue_linked_del(&base->active, &timer->node))
1156: cpu_base->active_bases &= ~(1 << base->index);
```
Wait, *wait* a second. I'm seeing `timerqueue_linked_del` again. Is this a custom implementation or something I'm missing? That function truly doesn't seem to exist in the standard kernel headers, so maybe it's a recent change? Let's go look at `timerqueue_del` in `include/linux/timerqueue.h` to see how it's handled.
|
| 63/2 |
2026/05/18 18:42 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/linux/timerqueue.h IncludeLines:true Name:timerqueue_del]
Results: map[SourceCode: 44: /**
45: * timerqueue_del - Removes a timer from the timerqueue.
46: *
47: * @head: head of timerqueue
48: * @node: timer node to be removed
49: *
50: * Removes the timer node from the timerqueue. Returns true if the queue is
51: * not empty after the remove.
52: */
53: bool timerqueue_del(struct timerqueue_head *head, struct timerqueue_node *node)
54: {
55: WARN_ON_ONCE(RB_EMPTY_NODE(&node->node));
56:
57: rb_erase_cached(&node->node, &head->rb_root);
58: RB_CLEAR_NODE(&node->node);
59:
60: return !RB_EMPTY_ROOT(&head->rb_root.rb_root);
61: }
SourceFile:lib/timerqueue.c]
|
| 64/2 |
2026/05/18 18:42 |
llm |
debugger |
0m
Model:
Tokens:
input: 36937
output: 23
thoughts: 33
Thoughts: Here's my thought process, summarized as you requested:
**Navigating the Timer Queue Implementation**
Okay, so I'm looking at `timerqueue_del`, and it's telling me that this function, which is crucial for removing timers from the queue, resides within `lib/timerqueue.c`. That's the first key piece of information. Now, to understand the function's declaration, data structures, and the overall API for timer queue manipulation, the next logical step is to dive into the header file. I need to examine `include/linux/timerqueue.h`. This is where I'll find the structures involved, the function prototypes, and any associated macros or inline functions that I need to be aware of to fully grasp how this timer queue implementation works. That header will give me the building blocks to understand how these timers are managed.
|
| 65/2 |
2026/05/18 18:42 |
tool |
read-file |
0m
Args: map[File:include/linux/timerqueue.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_TIMERQUEUE_H
#define _LINUX_TIMERQUEUE_H
#include <linux/rbtree.h>
#include <linux/timerqueue_types.h>
bool timerqueue_add(struct timerqueue_head *head, struct timerqueue_node *node);
bool timerqueue_del(struct timerqueue_head *head, struct timerqueue_node *node);
struct timerqueue_node *timerqueue_iterate_next(struct timerqueue_node *node);
bool timerqueue_linked_add(struct timerqueue_linked_head *head, struct timerqueue_linked_node *node);
/**
* timerqueue_getnext - Returns the timer with the earliest expiration time
*
* @head: head of timerqueue
*
* Returns a pointer to the timer node that has the earliest expiration time.
*/
static inline struct timerqueue_node *timerqueue_getnext(struct timerqueue_head *head)
{
struct rb_node *leftmost = rb_first_cached(&head->rb_root);
return rb_entry_safe(leftmost, struct timerqueue_node, node);
}
static inline void timerqueue_init(struct timerqueue_node *node)
{
RB_CLEAR_NODE(&node->node);
}
static inline bool timerqueue_node_queued(struct timerqueue_node *node)
{
return !RB_EMPTY_NODE(&node->node);
}
static inline void timerqueue_init_head(struct timerqueue_head *head)
{
head->rb_root = RB_ROOT_CACHED;
}
/* Timer queues with linked nodes */
static __always_inline
struct timerqueue_linked_node *timerqueue_linked_first(struct timerqueue_linked_head *head)
{
return rb_entry_safe(head->rb_root.rb_leftmost, struct timerqueue_linked_node, node);
}
static __always_inline
struct timerqueue_linked_node *timerqueue_linked_next(struct timerqueue_linked_node *node)
{
return rb_entry_safe(node->node.next, struct timerqueue_linked_node, node);
}
static __always_inline
struct timerqueue_linked_node *timerqueue_linked_prev(struct timerqueue_linked_node *node)
{
return rb_entry_safe(node->node.prev, struct timerqueue_linked_node, node);
}
static __always_inline
bool timerqueue_linked_del(struct timerqueue_linked_head *head, struct timerqueue_linked_node *node)
{
return rb_erase_linked(&node->node, &head->rb_root);
}
static __always_inline void timerqueue_linked_init(struct timerqueue_linked_node *node)
{
RB_CLEAR_LINKED_NODE(&node->node);
}
static __always_inline bool timerqueue_linked_node_queued(struct timerqueue_linked_node *node)
{
return !RB_EMPTY_LINKED_NODE(&node->node);
}
static __always_inline void timerqueue_linked_init_head(struct timerqueue_linked_head *head)
{
head->rb_root = RB_ROOT_LINKED;
}
#endif /* _LINUX_TIMERQUEUE_H */
]
|
| 66/2 |
2026/05/18 18:42 |
llm |
debugger |
0m
Model:
Tokens:
input: 37973
output: 42
thoughts: 50
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Analyzing the Timer Queue Deletion Mechanism**
Alright, so `timerqueue_linked_del` is the culprit. That function is clearly calling `rb_erase_linked`, interesting. I need to get a better handle on the actual implementation of the red-black tree deletion being used here. My first instinct is to dig into the details of `rb_erase_linked`. Given that it's a core data structure operation, the definition will either be in `include/linux/rbtree.h` for inline optimization or `lib/rbtree.c` for a more general implementation. I'll check both locations to see where the core logic resides and examine it closely. This is a critical point to understand for performance or potential race conditions.
|
| 67/2 |
2026/05/18 18:42 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/linux/rbtree.h IncludeLines:true Name:rb_erase_linked]
Results: map[SourceCode: 449: bool rb_erase_linked(struct rb_node_linked *node, struct rb_root_linked *root)
450: {
451: if (node->prev)
452: node->prev->next = node->next;
453: else
454: root->rb_leftmost = node->next;
455:
456: if (node->next)
457: node->next->prev = node->prev;
458:
459: rb_erase(&node->node, &root->rb_root);
460: RB_CLEAR_LINKED_NODE(node);
461:
462: return !!root->rb_leftmost;
463: }
SourceFile:lib/rbtree.c]
|
| 68/2 |
2026/05/18 18:42 |
llm |
debugger |
0m
Model:
Tokens:
input: 38479
output: 38
thoughts: 38
Thoughts: Here's my summary, as the thought process unfolds:
**Examining the Red-Black Tree Erase Implementation**
Okay, so I'm digging into how the red-black tree's erase function is implemented, specifically within the Ruby standard library. The trace shows a call to `rb_erase_linked`, so I need to find its definition. Right, it's telling me it's located in `lib/rbtree.c`. That makes sense; it's a core data structure implementation. Now, the next logical step is to analyze the `rb_erase` function itself. I need to understand its control flow, the cases it handles (node color, parent-child relationships, rebalancing operations), and any special considerations for linked nodes as hinted by `rb_erase_linked`. Let's get to the source code and start tracing the logic!
|
| 69/2 |
2026/05/18 18:42 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/rbtree.c IncludeLines:true Name:rb_erase]
Results: map[SourceCode: 440: void rb_erase(struct rb_node *node, struct rb_root *root)
441: {
442: struct rb_node *rebalance;
443: rebalance = __rb_erase_augmented(node, root, &dummy_callbacks);
444: if (rebalance)
445: ____rb_erase_color(rebalance, root, dummy_rotate);
446: }
SourceFile:lib/rbtree.c]
|
| 70/2 |
2026/05/18 18:42 |
llm |
debugger |
0m
Model:
Tokens:
input: 38885
output: 46
thoughts: 1725
Thoughts: **Analyzing a Kernel Bug: BPF, Locks, and Assembly**
Okay, so I'm looking at this crash in `__remove_hrtimer`, and the first thing that jumps out is that it's holding `cpu_base->lock`. That's important. I need to figure out why CPU 1 seems stuck in `rb_erase`. Let's dig back into the assembly code.
The assembly is kind of tricky to parse, but I notice this `call 0xa21a2ce` instruction. That's a huge clue! It's very likely this is a call to a BPF program. I need to understand what this BPF program is doing. Syzkaller reproducer always gives me an important lead. Let's look at the reproducer.
Aha! The reproducer is using a `bpf` syscall with a specific command and address. It is very likely a BPF program is loaded here, probably type 5. I need to be sure though. Checking the reproducer shows that it does `ioctl` on a team interface named `team0`. Then it creates a socket and sends a `netlink` message. It looks like it is setting up a traffic control queue discipline (`QDISC`). This is probably what the BPF program does.
So, the reproducer uses a BPF program and attaches it to something traffic control related. I need to confirm the BPF program type from `include/uapi/linux/bpf.h`. Then I need to see what the program does. The assembly and this reproducer are key! This looks like a race or lock issue related to BPF programs attached to traffic control. It is likely that the program messes with the timer which causes the lock to be held. Let's see how they interact.
|
| 71/2 |
2026/05/18 18:43 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/uapi/linux/bpf.h IncludeLines:true Name:bpf_prog_type]
Results: map[SourceCode:1052: /* Note that tracing related programs such as
1053: * BPF_PROG_TYPE_{KPROBE,TRACEPOINT,PERF_EVENT,RAW_TRACEPOINT}
1054: * are not subject to a stable API since kernel internal data
1055: * structures can change from release to release and may
1056: * therefore break existing tracing BPF programs. Tracing BPF
1057: * programs correspond to /a/ specific kernel which is to be
1058: * analyzed, and not /a/ specific kernel /and/ all future ones.
1059: */
1060: enum bpf_prog_type {
1061: BPF_PROG_TYPE_UNSPEC,
1062: BPF_PROG_TYPE_SOCKET_FILTER,
1063: BPF_PROG_TYPE_KPROBE,
1064: BPF_PROG_TYPE_SCHED_CLS,
1065: BPF_PROG_TYPE_SCHED_ACT,
1066: BPF_PROG_TYPE_TRACEPOINT,
1067: BPF_PROG_TYPE_XDP,
1068: BPF_PROG_TYPE_PERF_EVENT,
1069: BPF_PROG_TYPE_CGROUP_SKB,
1070: BPF_PROG_TYPE_CGROUP_SOCK,
1071: BPF_PROG_TYPE_LWT_IN,
1072: BPF_PROG_TYPE_LWT_OUT,
1073: BPF_PROG_TYPE_LWT_XMIT,
1074: BPF_PROG_TYPE_SOCK_OPS,
1075: BPF_PROG_TYPE_SK_SKB,
1076: BPF_PROG_TYPE_CGROUP_DEVICE,
1077: BPF_PROG_TYPE_SK_MSG,
1078: BPF_PROG_TYPE_RAW_TRACEPOINT,
1079: BPF_PROG_TYPE_CGROUP_SOCK_ADDR,
1080: BPF_PROG_TYPE_LWT_SEG6LOCAL,
1081: BPF_PROG_TYPE_LIRC_MODE2,
1082: BPF_PROG_TYPE_SK_REUSEPORT,
1083: BPF_PROG_TYPE_FLOW_DISSECTOR,
1084: BPF_PROG_TYPE_CGROUP_SYSCTL,
1085: BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE,
1086: BPF_PROG_TYPE_CGROUP_SOCKOPT,
1087: BPF_PROG_TYPE_TRACING,
1088: BPF_PROG_TYPE_STRUCT_OPS,
1089: BPF_PROG_TYPE_EXT,
1090: BPF_PROG_TYPE_LSM,
1091: BPF_PROG_TYPE_SK_LOOKUP,
1092: BPF_PROG_TYPE_SYSCALL, /* a program that can execute syscalls */
1093: BPF_PROG_TYPE_NETFILTER,
1094: __MAX_BPF_PROG_TYPE
1095: };
SourceFile:include/uapi/linux/bpf.h]
|
| 72/2 |
2026/05/18 18:43 |
llm |
debugger |
0m
Model:
Tokens:
input: 41940
output: 42
thoughts: 1072
Thoughts: **Analyzing a BPF-Related Soft Lockup**
Okay, so type 5 is `BPF_PROG_TYPE_TRACEPOINT`. The reproducer loads a tracepoint BPF program and then sends an `RTM_NEWQDISC` netlink message (0x24). I'm not seeing any explicit `bpf$PROG_ATTACH` or `perf_event_open` syscalls, but it's likely just loading the program for later use. Looking at the reproducer's `bpf$PROG_LOAD` arguments confirms this, with the type field set to 0x5. I don't see any attach though.
The soft lockup is happening in a pretty standard spot: `toggle_allocation_gate` -> `static_key_enable` -> `arch_jump_label_transform_apply` -> `smp_text_poke_batch_finish` -> `smp_text_poke_sync_each_cpu` -> `on_each_cpu_cond_mask` -> `smp_call_function_many_cond` -> `csd_lock_wait`. CPU 0 is waiting for an IPI from CPU 1.
CPU 1 is stuck in `__remove_hrtimer`. That's not good. Is the rbtree corrupted? That would explain the infinite loop in `rb_erase` or `timerqueue_del`. But, how could the rbtree get corrupted?
Wait a minute, let's look at the BPF program itself. It's using `BPF_FUNC_ringbuf_reserve`, `BPF_FUNC_snprintf`, `BPF_FUNC_ringbuf_query`, `BPF_FUNC_ringbuf_output` and then `bpf_helpers_ringbuf_free`.
Aha! `BPF_FUNC_snprintf`! That's interesting. Could there be a bug in `bpf_snprintf`? Time to dig into `kernel/bpf/helpers.c` and see what's going on there.
|
| 73/2 |
2026/05/18 18:43 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_snprintf]
Results: map[SourceCode:1064: BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
SourceFile:kernel/bpf/helpers.c]
|
| 74/2 |
2026/05/18 18:43 |
llm |
debugger |
0m
Model:
Tokens:
input: 43567
output: 43
thoughts: 15
Thoughts: Okay, let's dive into this. I need to understand this `bpf_snprintf` function in detail. I'm assuming it's related to BPF (Berkeley Packet Filter), but I need to be sure. I'm going to pull up the definitive documentation or source code for `bpf_snprintf` - wherever I can find the *complete* definition. I need to understand its parameters, return type, and, most importantly, how it interacts with the BPF context and environment. What's special about it that differentiates it from a standard `snprintf`? Are there any BPF-specific limitations or considerations? I need to know the specifics: data types, potential for memory access restrictions, any unique error handling, and the implications for performance within the BPF VM. Getting a solid grasp of this will give me the building blocks I require to work with it properly. It's time to dig in.
|
| 75/2 |
2026/05/18 18:43 |
tool |
codesearch-definition-source |
0m
Error:
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:BPF_CALL_5]
Results: map[SourceCode: SourceFile:]
|
| 76/2 |
2026/05/18 18:43 |
llm |
debugger |
0m
Model:
Tokens:
input: 43841
output: 26
thoughts: 23
Thoughts: Here's my thought process, as requested:
**Investigating BPF Helper Function Implementation**
Okay, so I need to understand how `bpf_snprintf` is implemented. I'm going to start by going directly to the source. The most logical place to begin is the kernel's BPF helper function implementation file, which is almost certainly in `kernel/bpf/helpers.c`. That's where I would expect to find the definition and logic for this function. Let's get to it. Time to dive into the code.
|
| 77/2 |
2026/05/18 18:43 |
tool |
grepper |
0m
Args: map[Expression:BPF_CALL_5\(bpf_snprintf]
Results: map[Output:kernel/bpf/helpers.c=818=int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
--
kernel/bpf/helpers.c-1063-
kernel/bpf/helpers.c:1064:BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
kernel/bpf/helpers.c-1065- const void *, args, u32, data_len)
--
kernel/trace/bpf_trace.c=976=static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
--
kernel/trace/bpf_trace.c-1005-
kernel/trace/bpf_trace.c:1006:BPF_CALL_5(bpf_snprintf_btf, char *, str, u32, str_size, struct btf_ptr *, ptr,
kernel/trace/bpf_trace.c-1007- u32, btf_ptr_size, u64, flags)
]
|
| 78/2 |
2026/05/18 18:43 |
llm |
debugger |
0m
Model:
Tokens:
input: 44310
output: 23
thoughts: 22
Thoughts: Here's a summary of my thought process, as requested:
**Diving into `helpers.c`**
Okay, time to get my hands dirty. I need to understand something specific about the BPF helper functions, so I'm heading directly into the source code. Let's start with `kernel/bpf/helpers.c`. My hunch is whatever I'm looking for is in that file, and I'm guessing it's around line 1064. That should be a good starting point for a targeted investigation. I'll need to remember to pay close attention to the surrounding context as well – no point in just looking at one line, I need to see how it fits into the broader picture. Let's see what we find.
|
| 79/2 |
2026/05/18 18:43 |
tool |
read-file |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*/
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/bpf-cgroup.h>
#include <linux/cgroup.h>
#include <linux/rcupdate.h>
#include <linux/random.h>
#include <linux/smp.h>
#include <linux/topology.h>
#include <linux/ktime.h>
#include <linux/sched.h>
#include <linux/uidgid.h>
#include <linux/filter.h>
#include <linux/ctype.h>
#include <linux/jiffies.h>
#include <linux/pid_namespace.h>
#include <linux/poison.h>
#include <linux/proc_ns.h>
#include <linux/sched/task.h>
#include <linux/security.h>
#include <linux/btf_ids.h>
#include <linux/bpf_mem_alloc.h>
#include <linux/kasan.h>
#include <linux/bpf_verifier.h>
#include <linux/uaccess.h>
#include <linux/verification.h>
#include <linux/task_work.h>
#include <linux/irq_work.h>
#include <linux/buildid.h>
#include "../../lib/kstrtox.h"
/* If kernel subsystem is allowing eBPF programs to call this function,
* inside its own verifier_ops->get_func_proto() callback it should return
* bpf_map_lookup_elem_proto, so that verifier can properly check the arguments
*
* Different map implementations will rely on rcu in map methods
* lookup/update/delete, therefore eBPF programs must run under rcu lock
* if program is allowed to access maps, so check rcu_read_lock_held() or
* rcu_read_lock_trace_held() in all three functions.
*/
BPF_CALL_2(bpf_map_lookup_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_elem(map, key);
}
const struct bpf_func_proto bpf_map_lookup_elem_proto = {
.func = bpf_map_lookup_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_4(bpf_map_update_elem, struct bpf_map *, map, void *, key,
void *, value, u64, flags)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_update_elem(map, key, value, flags);
}
const struct bpf_func_proto bpf_map_update_elem_proto = {
.func = bpf_map_update_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_PTR_TO_MAP_VALUE,
.arg4_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_delete_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_delete_elem(map, key);
}
const struct bpf_func_proto bpf_map_delete_elem_proto = {
.func = bpf_map_delete_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_3(bpf_map_push_elem, struct bpf_map *, map, void *, value, u64, flags)
{
return map->ops->map_push_elem(map, value, flags);
}
const struct bpf_func_proto bpf_map_push_elem_proto = {
.func = bpf_map_push_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_pop_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_pop_elem(map, value);
}
const struct bpf_func_proto bpf_map_pop_elem_proto = {
.func = bpf_map_pop_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_2(bpf_map_peek_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_peek_elem(map, value);
}
const struct bpf_func_proto bpf_map_peek_elem_proto = {
.func = bpf_map_peek_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_3(bpf_map_lookup_percpu_elem, struct bpf_map *, map, void *, key, u32, cpu)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_percpu_elem(map, key, cpu);
}
const struct bpf_func_proto bpf_map_lookup_percpu_elem_proto = {
.func = bpf_map_lookup_percpu_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_ANYTHING,
};
const struct bpf_func_proto bpf_get_prandom_u32_proto = {
.func = bpf_user_rnd_u32,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_smp_processor_id)
{
return smp_processor_id();
}
const struct bpf_func_proto bpf_get_smp_processor_id_proto = {
.func = bpf_get_smp_processor_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.allow_fastcall = true,
};
BPF_CALL_0(bpf_get_numa_node_id)
{
return numa_node_id();
}
const struct bpf_func_proto bpf_get_numa_node_id_proto = {
.func = bpf_get_numa_node_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_ns)
{
/* NMI safe access to clock monotonic */
return ktime_get_mono_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_ns_proto = {
.func = bpf_ktime_get_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_boot_ns)
{
/* NMI safe access to clock boottime */
return ktime_get_boot_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_boot_ns_proto = {
.func = bpf_ktime_get_boot_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_coarse_ns)
{
return ktime_get_coarse_ns();
}
const struct bpf_func_proto bpf_ktime_get_coarse_ns_proto = {
.func = bpf_ktime_get_coarse_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_tai_ns)
{
/* NMI safe access to clock tai */
return ktime_get_tai_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_tai_ns_proto = {
.func = bpf_ktime_get_tai_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_pid_tgid)
{
struct task_struct *task = current;
if (unlikely(!task))
return -EINVAL;
return (u64) task->tgid << 32 | task->pid;
}
const struct bpf_func_proto bpf_get_current_pid_tgid_proto = {
.func = bpf_get_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_uid_gid)
{
struct task_struct *task = current;
kuid_t uid;
kgid_t gid;
if (unlikely(!task))
return -EINVAL;
current_uid_gid(&uid, &gid);
return (u64) from_kgid(&init_user_ns, gid) << 32 |
from_kuid(&init_user_ns, uid);
}
const struct bpf_func_proto bpf_get_current_uid_gid_proto = {
.func = bpf_get_current_uid_gid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_2(bpf_get_current_comm, char *, buf, u32, size)
{
struct task_struct *task = current;
if (unlikely(!task))
goto err_clear;
/* Verifier guarantees that size > 0 */
strscpy_pad(buf, task->comm, size);
return 0;
err_clear:
memset(buf, 0, size);
return -EINVAL;
}
const struct bpf_func_proto bpf_get_current_comm_proto = {
.func = bpf_get_current_comm,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE,
};
#if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK)
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
union {
__u32 val;
arch_spinlock_t lock;
} u = { .lock = __ARCH_SPIN_LOCK_UNLOCKED };
compiletime_assert(u.val == 0, "__ARCH_SPIN_LOCK_UNLOCKED not 0");
BUILD_BUG_ON(sizeof(*l) != sizeof(__u32));
BUILD_BUG_ON(sizeof(*lock) != sizeof(__u32));
preempt_disable();
arch_spin_lock(l);
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
arch_spin_unlock(l);
preempt_enable();
}
#else
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
BUILD_BUG_ON(sizeof(*l) != sizeof(*lock));
do {
atomic_cond_read_relaxed(l, !VAL);
} while (atomic_xchg(l, 1));
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
atomic_set_release(l, 0);
}
#endif
static DEFINE_PER_CPU(unsigned long, irqsave_flags);
static inline void __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
{
unsigned long flags;
local_irq_save(flags);
__bpf_spin_lock(lock);
__this_cpu_write(irqsave_flags, flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
{
__bpf_spin_lock_irqsave(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_lock_proto = {
.func = bpf_spin_lock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
{
unsigned long flags;
flags = __this_cpu_read(irqsave_flags);
__bpf_spin_unlock(lock);
local_irq_restore(flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
{
__bpf_spin_unlock_irqrestore(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_unlock_proto = {
.func = bpf_spin_unlock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
bool lock_src)
{
struct bpf_spin_lock *lock;
if (lock_src)
lock = src + map->record->spin_lock_off;
else
lock = dst + map->record->spin_lock_off;
preempt_disable();
__bpf_spin_lock_irqsave(lock);
copy_map_value(map, dst, src);
__bpf_spin_unlock_irqrestore(lock);
preempt_enable();
}
BPF_CALL_0(bpf_jiffies64)
{
return get_jiffies_64();
}
const struct bpf_func_proto bpf_jiffies64_proto = {
.func = bpf_jiffies64,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
#ifdef CONFIG_CGROUPS
BPF_CALL_0(bpf_get_current_cgroup_id)
{
struct cgroup *cgrp;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
cgrp_id = cgroup_id(cgrp);
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_cgroup_id_proto = {
.func = bpf_get_current_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_1(bpf_get_current_ancestor_cgroup_id, int, ancestor_level)
{
struct cgroup *cgrp;
struct cgroup *ancestor;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
ancestor = cgroup_ancestor(cgrp, ancestor_level);
cgrp_id = ancestor ? cgroup_id(ancestor) : 0;
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto = {
.func = bpf_get_current_ancestor_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
#endif /* CONFIG_CGROUPS */
#define BPF_STRTOX_BASE_MASK 0x1F
static int __bpf_strtoull(const char *buf, size_t buf_len, u64 flags,
unsigned long long *res, bool *is_negative)
{
unsigned int base = flags & BPF_STRTOX_BASE_MASK;
const char *cur_buf = buf;
size_t cur_len = buf_len;
unsigned int consumed;
size_t val_len;
char str[64];
if (!buf || !buf_len || !res || !is_negative)
return -EINVAL;
if (base != 0 && base != 8 && base != 10 && base != 16)
return -EINVAL;
if (flags & ~BPF_STRTOX_BASE_MASK)
return -EINVAL;
while (cur_buf < buf + buf_len && isspace(*cur_buf))
++cur_buf;
*is_negative = (cur_buf < buf + buf_len && *cur_buf == '-');
if (*is_negative)
++cur_buf;
consumed = cur_buf - buf;
cur_len -= consumed;
if (!cur_len)
return -EINVAL;
cur_len = min(cur_len, sizeof(str) - 1);
memcpy(str, cur_buf, cur_len);
str[cur_len] = '\0';
cur_buf = str;
cur_buf = _parse_integer_fixup_radix(cur_buf, &base);
val_len = _parse_integer(cur_buf, base, res);
if (val_len & KSTRTOX_OVERFLOW)
return -ERANGE;
if (val_len == 0)
return -EINVAL;
cur_buf += val_len;
consumed += cur_buf - str;
return consumed;
}
static int __bpf_strtoll(const char *buf, size_t buf_len, u64 flags,
long long *res)
{
unsigned long long _res;
bool is_negative;
int err;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative) {
if ((long long)-_res > 0)
return -ERANGE;
*res = -_res;
} else {
if ((long long)_res < 0)
return -ERANGE;
*res = _res;
}
return err;
}
BPF_CALL_4(bpf_strtol, const char *, buf, size_t, buf_len, u64, flags,
s64 *, res)
{
long long _res;
int err;
*res = 0;
err = __bpf_strtoll(buf, buf_len, flags, &_res);
if (err < 0)
return err;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtol_proto = {
.func = bpf_strtol,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(s64),
};
BPF_CALL_4(bpf_strtoul, const char *, buf, size_t, buf_len, u64, flags,
u64 *, res)
{
unsigned long long _res;
bool is_negative;
int err;
*res = 0;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative)
return -EINVAL;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtoul_proto = {
.func = bpf_strtoul,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(u64),
};
BPF_CALL_3(bpf_strncmp, const char *, s1, u32, s1_sz, const char *, s2)
{
return strncmp(s1, s2, s1_sz);
}
static const struct bpf_func_proto bpf_strncmp_proto = {
.func = bpf_strncmp,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_CONST_STR,
};
BPF_CALL_4(bpf_get_ns_current_pid_tgid, u64, dev, u64, ino,
struct bpf_pidns_info *, nsdata, u32, size)
{
struct task_struct *task = current;
struct pid_namespace *pidns;
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_pidns_info)))
goto clear;
if (unlikely((u64)(dev_t)dev != dev))
goto clear;
if (unlikely(!task))
goto clear;
pidns = task_active_pid_ns(task);
if (unlikely(!pidns)) {
err = -ENOENT;
goto clear;
}
if (!ns_match(&pidns->ns, (dev_t)dev, ino))
goto clear;
nsdata->pid = task_pid_nr_ns(task, pidns);
nsdata->tgid = task_tgid_nr_ns(task, pidns);
return 0;
clear:
memset((void *)nsdata, 0, (size_t) size);
return err;
}
const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = {
.func = bpf_get_ns_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
};
static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
.func = bpf_get_raw_cpu_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
}
const struct bpf_func_proto bpf_event_output_data_proto = {
.func = bpf_event_output_data,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
const void __user *, user_ptr)
{
int ret = copy_from_user(dst, user_ptr, size);
if (unlikely(ret)) {
memset(dst, 0, size);
ret = -EFAULT;
}
return ret;
}
const struct bpf_func_proto bpf_copy_from_user_proto = {
.func = bpf_copy_from_user,
.gpl_only = false,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_5(bpf_copy_from_user_task, void *, dst, u32, size,
const void __user *, user_ptr, struct task_struct *, tsk, u64, flags)
{
int ret;
/* flags is not used yet */
if (unlikely(flags))
return -EINVAL;
if (unlikely(!size))
return 0;
ret = access_process_vm(tsk, (unsigned long)user_ptr, dst, size, 0);
if (ret == size)
return 0;
memset(dst, 0, size);
/* Return -EFAULT for partial read */
return ret < 0 ? ret : -EFAULT;
}
const struct bpf_func_proto bpf_copy_from_user_task_proto = {
.func = bpf_copy_from_user_task,
.gpl_only = true,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_BTF_ID,
.arg4_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.arg5_type = ARG_ANYTHING
};
BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
{
if (cpu >= nr_cpu_ids)
return (unsigned long)NULL;
return (unsigned long)per_cpu_ptr((const void __percpu *)(const uintptr_t)ptr, cpu);
}
const struct bpf_func_proto bpf_per_cpu_ptr_proto = {
.func = bpf_per_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | PTR_MAYBE_NULL | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
.arg2_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_this_cpu_ptr, const void *, percpu_ptr)
{
return (unsigned long)this_cpu_ptr((const void __percpu *)(const uintptr_t)percpu_ptr);
}
const struct bpf_func_proto bpf_this_cpu_ptr_proto = {
.func = bpf_this_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
};
static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,
size_t bufsz)
{
void __user *user_ptr = (__force void __user *)unsafe_ptr;
buf[0] = 0;
switch (fmt_ptype) {
case 's':
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
if ((unsigned long)unsafe_ptr < TASK_SIZE)
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
fallthrough;
#endif
case 'k':
return strncpy_from_kernel_nofault(buf, unsafe_ptr, bufsz);
case 'u':
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
}
return -EINVAL;
}
/* Support executing three nested bprintf helper calls on a given CPU */
#define MAX_BPRINTF_NEST_LEVEL 3
static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs);
static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
{
int nest_level;
preempt_disable();
nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
return -EBUSY;
}
*bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
return 0;
}
void bpf_put_buffers(void)
{
if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
return;
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
}
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
{
if (!data->bin_args && !data->buf)
return;
bpf_put_buffers();
}
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)
{
if (map->map_type == BPF_MAP_TYPE_ARRAY) {
struct bpf_array *array = container_of(map, struct bpf_array, map);
*arr_idx = ((char *)value - array->value) / array->elem_size;
return arr_idx;
}
return (void *)value - round_up(map->key_size, 8);
}
enum bpf_async_type {
BPF_ASYNC_TYPE_TIMER = 0,
BPF_ASYNC_TYPE_WQ,
};
enum bpf_async_op {
BPF_ASYNC_START,
BPF_ASYNC_CANCEL
};
struct bpf_async_cmd {
struct llist_node node;
u64 nsec;
u32 mode;
enum bpf_async_op op;
};
struct bpf_async_cb {
struct bpf_map *map;
struct bpf_prog *prog;
void __rcu *callback_fn;
void *value;
struct rcu_head rcu;
u64 flags;
struct irq_work worker;
refcount_t refcnt;
enum bpf_async_type type;
struct llist_head async_cmds;
};
/* BPF map elements can contain 'struct bpf_timer'.
* Such map owns all of its BPF timers.
* 'struct bpf_timer' is allocated as part of map element allocation
* and it's zero initialized.
* That space is used to keep 'struct bpf_async_kern'.
* bpf_timer_init() allocates 'struct bpf_hrtimer', inits hrtimer, and
* remembers 'struct bpf_map *' pointer it's part of.
* bpf_timer_set_callback() increments prog refcnt and assign bpf callback_fn.
* bpf_timer_start() arms the timer.
* If user space reference to a map goes to zero at this point
* ops->map_release_uref callback is responsible for cancelling the timers,
* freeing their memory, and decrementing prog's refcnts.
* bpf_timer_cancel() cancels the timer and decrements prog's refcnt.
* Inner maps can contain bpf timers as well. ops->map_release_uref is
* freeing the timers when inner map is replaced or deleted by user space.
*/
struct bpf_hrtimer {
struct bpf_async_cb cb;
struct hrtimer timer;
atomic_t cancelling;
};
struct bpf_work {
struct bpf_async_cb cb;
struct work_struct work;
};
/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */
struct bpf_async_kern {
union {
struct bpf_async_cb *cb;
struct bpf_hrtimer *timer;
struct bpf_work *work;
};
} __attribute__((aligned(8)));
static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);
static void bpf_async_refcount_put(struct bpf_async_cb *cb);
static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer)
{
struct bpf_hrtimer *t = container_of(hrtimer, struct bpf_hrtimer, timer);
struct bpf_map *map = t->cb.map;
void *value = t->cb.value;
bpf_callback_t callback_fn;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_timer);
callback_fn = rcu_dereference_check(t->cb.callback_fn, rcu_read_lock_bh_held());
if (!callback_fn)
goto out;
/* bpf_timer_cb() runs in hrtimer_run_softirq. It doesn't migrate and
* cannot be preempted by another bpf_timer_cb() on the same cpu.
* Remember the timer this callback is servicing to prevent
* deadlock if callback_fn() calls bpf_timer_cancel() or
* bpf_map_delete_elem() on the same timer.
*/
this_cpu_write(hrtimer_running, t);
key = map_key_from_value(map, value, &idx);
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
/* The verifier checked that return value is zero. */
this_cpu_write(hrtimer_running, NULL);
out:
return HRTIMER_NORESTART;
}
static void bpf_wq_work(struct work_struct *work)
{
struct bpf_work *w = container_of(work, struct bpf_work, work);
struct bpf_async_cb *cb = &w->cb;
struct bpf_map *map = cb->map;
bpf_callback_t callback_fn;
void *value = cb->value;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_wq);
callback_fn = READ_ONCE(cb->callback_fn);
if (!callback_fn)
return;
key = map_key_from_value(map, value, &idx);
rcu_read_lock_trace();
migrate_disable();
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
migrate_enable();
rcu_read_unlock_trace();
}
static void bpf_async_cb_rcu_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
/*
* Drop the last reference to prog only after RCU GP, as set_callback()
* may race with cancel_and_free()
*/
if (cb->prog)
bpf_prog_put(cb->prog);
kfree_nolock(cb);
}
/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */
static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
bool retry = false;
/*
* bpf_async_cancel_and_free() tried to cancel timer/wq, but it
* could have raced with timer/wq_start. Now refcnt is zero and
* srcu/rcu GP completed. Cancel timer/wq again.
*/
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER:
if (hrtimer_try_to_cancel(&t->timer) < 0)
retry = true;
break;
case BPF_ASYNC_TYPE_WQ:
if (!cancel_work(&w->work) && work_busy(&w->work))
retry = true;
break;
}
if (retry) {
/*
* hrtimer or wq callback may still be running. It must be
* in rcu_tasks_trace or rcu CS, so wait for GP again.
* It won't retry forever, since refcnt zero prevents all
* operations on timer/wq.
*/
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
return;
}
/* RCU Tasks Trace grace period implies RCU grace period. */
bpf_async_cb_rcu_free(rcu);
}
static void worker_for_call_rcu(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
static void bpf_async_refcount_put(struct bpf_async_cb *cb)
{
if (!refcount_dec_and_test(&cb->refcnt))
return;
if (irqs_disabled()) {
cb->worker = IRQ_WORK_INIT(worker_for_call_rcu);
irq_work_queue(&cb->worker);
} else {
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async);
static void bpf_async_irq_worker(struct irq_work *work);
static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags,
enum bpf_async_type type)
{
struct bpf_async_cb *cb, *old_cb;
struct bpf_hrtimer *t;
struct bpf_work *w;
clockid_t clockid;
size_t size;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
size = sizeof(struct bpf_hrtimer);
break;
case BPF_ASYNC_TYPE_WQ:
size = sizeof(struct bpf_work);
break;
default:
return -EINVAL;
}
old_cb = READ_ONCE(async->cb);
if (old_cb)
return -EBUSY;
cb = bpf_map_kmalloc_nolock(map, size, 0, map->numa_node);
if (!cb)
return -ENOMEM;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
clockid = flags & (MAX_CLOCKS - 1);
t = (struct bpf_hrtimer *)cb;
atomic_set(&t->cancelling, 0);
hrtimer_setup(&t->timer, bpf_timer_cb, clockid, HRTIMER_MODE_REL_SOFT);
cb->value = (void *)async - map->record->timer_off;
break;
case BPF_ASYNC_TYPE_WQ:
w = (struct bpf_work *)cb;
INIT_WORK(&w->work, bpf_wq_work);
cb->value = (void *)async - map->record->wq_off;
break;
}
cb->map = map;
cb->prog = NULL;
cb->flags = flags;
cb->worker = IRQ_WORK_INIT(bpf_async_irq_worker);
init_llist_head(&cb->async_cmds);
refcount_set(&cb->refcnt, 1); /* map's reference */
cb->type = type;
rcu_assign_pointer(cb->callback_fn, NULL);
old_cb = cmpxchg(&async->cb, NULL, cb);
if (old_cb) {
/* Lost the race to initialize this bpf_async_kern, drop the allocated object */
kfree_nolock(cb);
return -EBUSY;
}
/* Guarantee the order between async->cb and map->usercnt. So
* when there are concurrent uref release and bpf timer init, either
* bpf_timer_cancel_and_free() called by uref release reads a no-NULL
* timer or atomic64_read() below returns a zero usercnt.
*/
smp_mb();
if (!atomic64_read(&map->usercnt)) {
/* maps with timers must be either held by user space
* or pinned in bpffs.
*/
bpf_async_cancel_and_free(async);
return -EPERM;
}
return 0;
}
BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
u64, flags)
{
clock_t clockid = flags & (MAX_CLOCKS - 1);
BUILD_BUG_ON(MAX_CLOCKS != 16);
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer));
if (flags >= MAX_CLOCKS ||
/* similar to timerfd except _ALARM variants are not supported */
(clockid != CLOCK_MONOTONIC &&
clockid != CLOCK_REALTIME &&
clockid != CLOCK_BOOTTIME))
return -EINVAL;
return __bpf_async_init(timer, map, flags, BPF_ASYNC_TYPE_TIMER);
}
static const struct bpf_func_proto bpf_timer_init_proto = {
.func = bpf_timer_init,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
static int bpf_async_update_prog_callback(struct bpf_async_cb *cb,
struct bpf_prog *prog,
void *callback_fn)
{
struct bpf_prog *prev;
/* Acquire a guard reference on prog to prevent it from being freed during the loop */
if (prog) {
prog = bpf_prog_inc_not_zero(prog);
if (IS_ERR(prog))
return PTR_ERR(prog);
}
do {
if (prog)
prog = bpf_prog_inc_not_zero(prog);
prev = xchg(&cb->prog, prog);
rcu_assign_pointer(cb->callback_fn, callback_fn);
/*
* Release previous prog, make sure that if other CPU is contending,
* to set bpf_prog, references are not leaked as each iteration acquires and
* releases one reference.
*/
if (prev)
bpf_prog_put(prev);
} while (READ_ONCE(cb->prog) != prog ||
(void __force *)READ_ONCE(cb->callback_fn) != callback_fn);
if (prog)
bpf_prog_put(prog);
return 0;
}
static DEFINE_PER_CPU(struct bpf_async_cb *, async_cb_running);
static int bpf_async_schedule_op(struct bpf_async_cb *cb, enum bpf_async_op op,
u64 nsec, u32 timer_mode)
{
/*
* Do not schedule another operation on this cpu if it's in irq_work
* callback that is processing async_cmds queue. Otherwise the following
* loop is possible:
* bpf_timer_start() -> bpf_async_schedule_op() -> irq_work_queue().
* irqrestore -> bpf_async_irq_worker() -> tracepoint -> bpf_timer_start().
*/
if (this_cpu_read(async_cb_running) == cb) {
bpf_async_refcount_put(cb);
return -EDEADLK;
}
struct bpf_async_cmd *cmd = kmalloc_nolock(sizeof(*cmd), 0, NUMA_NO_NODE);
if (!cmd) {
bpf_async_refcount_put(cb);
return -ENOMEM;
}
init_llist_node(&cmd->node);
cmd->nsec = nsec;
cmd->mode = timer_mode;
cmd->op = op;
if (llist_add(&cmd->node, &cb->async_cmds))
irq_work_queue(&cb->worker);
return 0;
}
static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback_fn,
struct bpf_prog *prog)
{
struct bpf_async_cb *cb;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
return bpf_async_update_prog_callback(cb, prog, callback_fn);
}
BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
struct bpf_prog_aux *, aux)
{
return __bpf_async_set_callback(timer, callback_fn, aux->prog);
}
static const struct bpf_func_proto bpf_timer_set_callback_proto = {
.func = bpf_timer_set_callback,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_PTR_TO_FUNC,
};
static bool defer_timer_wq_op(void)
{
return in_hardirq() || irqs_disabled();
}
BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags)
{
struct bpf_hrtimer *t;
u32 mode;
if (flags & ~(BPF_F_TIMER_ABS | BPF_F_TIMER_CPU_PIN))
return -EINVAL;
t = READ_ONCE(async->timer);
if (!t || !READ_ONCE(t->cb.prog))
return -EINVAL;
if (flags & BPF_F_TIMER_ABS)
mode = HRTIMER_MODE_ABS_SOFT;
else
mode = HRTIMER_MODE_REL_SOFT;
if (flags & BPF_F_TIMER_CPU_PIN)
mode |= HRTIMER_MODE_PINNED;
/*
* bpf_async_cancel_and_free() could have dropped refcnt to zero. In
* such case BPF progs are not allowed to arm the timer to prevent UAF.
*/
if (!refcount_inc_not_zero(&t->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
hrtimer_start(&t->timer, ns_to_ktime(nsecs), mode);
bpf_async_refcount_put(&t->cb);
return 0;
} else {
return bpf_async_schedule_op(&t->cb, BPF_ASYNC_START, nsecs, mode);
}
}
static const struct bpf_func_proto bpf_timer_start_proto = {
.func = bpf_timer_start,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async)
{
struct bpf_hrtimer *t, *cur_t;
bool inc = false;
int ret = 0;
if (defer_timer_wq_op())
return -EOPNOTSUPP;
t = READ_ONCE(async->timer);
if (!t)
return -EINVAL;
cur_t = this_cpu_read(hrtimer_running);
if (cur_t == t) {
/* If bpf callback_fn is trying to bpf_timer_cancel()
* its own timer the hrtimer_cancel() will deadlock
* since it waits for callback_fn to finish.
*/
return -EDEADLK;
}
/* Only account in-flight cancellations when invoked from a timer
* callback, since we want to avoid waiting only if other _callbacks_
* are waiting on us, to avoid introducing lockups. Non-callback paths
* are ok, since nobody would synchronously wait for their completion.
*/
if (!cur_t)
goto drop;
atomic_inc(&t->cancelling);
/* Need full barrier after relaxed atomic_inc */
smp_mb__after_atomic();
inc = true;
if (atomic_read(&cur_t->cancelling)) {
/* We're cancelling timer t, while some other timer callback is
* attempting to cancel us. In such a case, it might be possible
* that timer t belongs to the other callback, or some other
* callback waiting upon it (creating transitive dependencies
* upon us), and we will enter a deadlock if we continue
* cancelling and waiting for it synchronously, since it might
* do the same. Bail!
*/
atomic_dec(&t->cancelling);
return -EDEADLK;
}
drop:
bpf_async_update_prog_callback(&t->cb, NULL, NULL);
/* Cancel the timer and wait for associated callback to finish
* if it was running.
*/
ret = hrtimer_cancel(&t->timer);
if (inc)
atomic_dec(&t->cancelling);
return ret;
}
static const struct bpf_func_proto bpf_timer_cancel_proto = {
.func = bpf_timer_cancel,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
};
static void bpf_async_process_op(struct bpf_async_cb *cb, u32 op,
u64 timer_nsec, u32 timer_mode)
{
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER: {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
switch (op) {
case BPF_ASYNC_START:
hrtimer_start(&t->timer, ns_to_ktime(timer_nsec), timer_mode);
break;
case BPF_ASYNC_CANCEL:
hrtimer_try_to_cancel(&t->timer);
break;
}
break;
}
case BPF_ASYNC_TYPE_WQ: {
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
switch (op) {
case BPF_ASYNC_START:
schedule_work(&w->work);
break;
case BPF_ASYNC_CANCEL:
cancel_work(&w->work);
break;
}
break;
}
}
bpf_async_refcount_put(cb);
}
static void bpf_async_irq_worker(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
struct llist_node *pos, *n, *list;
list = llist_del_all(&cb->async_cmds);
if (!list)
return;
list = llist_reverse_order(list);
this_cpu_write(async_cb_running, cb);
llist_for_each_safe(pos, n, list) {
struct bpf_async_cmd *cmd;
cmd = container_of(pos, struct bpf_async_cmd, node);
bpf_async_process_op(cb, cmd->op, cmd->nsec, cmd->mode);
kfree_nolock(cmd);
}
this_cpu_write(async_cb_running, NULL);
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async)
{
struct bpf_async_cb *cb;
if (!READ_ONCE(async->cb))
return;
cb = xchg(&async->cb, NULL);
if (!cb)
return;
bpf_async_update_prog_callback(cb, NULL, NULL);
/*
* No refcount_inc_not_zero(&cb->refcnt) here. Dropping the last
* refcnt. Either synchronously or asynchronously in irq_work.
*/
if (!defer_timer_wq_op()) {
bpf_async_process_op(cb, BPF_ASYNC_CANCEL, 0, 0);
} else {
(void)bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
/*
* bpf_async_schedule_op() either enqueues allocated cmd into llist
* or fails with ENOMEM and drop the last refcnt.
* This is unlikely, but safe, since bpf_async_cb_rcu_tasks_trace_free()
* callback will do additional timer/wq_cancel due to races anyway.
*/
}
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_timer_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_wq_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
{
unsigned long *kptr = dst;
/* This helper may be inlined by verifier. */
return xchg(kptr, (unsigned long)ptr);
}
/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg()
* helper is determined dynamically by the verifier. Use BPF_PTR_POISON to
* denote type that verifier will determine.
*/
static const struct bpf_func_proto bpf_kptr_xchg_proto = {
.func = bpf_kptr_xchg,
.gpl_only = false,
.ret_type = RET_PTR_TO_BTF_ID_OR_NULL,
.ret_btf_id = BPF_PTR_POISON,
.arg1_type = ARG_KPTR_XCHG_DEST,
.arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL | OBJ_RELEASE,
.arg2_btf_id = BPF_PTR_POISON,
};
struct bpf_dynptr_file_impl {
struct freader freader;
/* 64 bit offset and size overriding 32 bit ones in bpf_dynptr_kern */
u64 offset;
u64 size;
};
/* Since the upper 8 bits of dynptr->size is reserved, the
* maximum supported size is 2^24 - 1.
*/
#define DYNPTR_MAX_SIZE ((1UL << 24) - 1)
#define DYNPTR_TYPE_SHIFT 28
#define DYNPTR_SIZE_MASK 0xFFFFFF
#define DYNPTR_RDONLY_BIT BIT(31)
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr)
{
return ptr->size & DYNPTR_RDONLY_BIT;
}
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
{
ptr->size |= DYNPTR_RDONLY_BIT;
}
static void bpf_dynptr_set_type(struct bpf_dynptr_kern *ptr, enum bpf_dynptr_type type)
{
ptr->size |= type << DYNPTR_TYPE_SHIFT;
}
static enum bpf_dynptr_type bpf_dynptr_get_type(const struct bpf_dynptr_kern *ptr)
{
return (ptr->size & ~(DYNPTR_RDONLY_BIT)) >> DYNPTR_TYPE_SHIFT;
}
u64 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
return df->size;
}
return ptr->size & DYNPTR_SIZE_MASK;
}
static void bpf_dynptr_advance_offset(struct bpf_dynptr_kern *ptr, u64 off)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->offset += off;
return;
}
ptr->offset += off;
}
static void bpf_dynptr_set_size(struct bpf_dynptr_kern *ptr, u64 new_size)
{
u32 metadata = ptr->size & ~DYNPTR_SIZE_MASK;
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->size = new_size;
return;
}
ptr->size = (u32)new_size | metadata;
}
int bpf_dynptr_check_size(u64 size)
{
return size > DYNPTR_MAX_SIZE ? -E2BIG : 0;
}
static int bpf_file_fetch_bytes(struct bpf_dynptr_file_impl *df, u64 offset, void *buf, u64 len)
{
const void *ptr;
if (!buf)
return -EINVAL;
df->freader.buf = buf;
df->freader.buf_sz = len;
ptr = freader_fetch(&df->freader, offset + df->offset, len);
if (!ptr)
return df->freader.err;
if (ptr != buf) /* Force copying into the buffer */
memcpy(buf, ptr, len);
return 0;
}
void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size)
{
ptr->data = data;
ptr->offset = offset;
ptr->size = size;
bpf_dynptr_set_type(ptr, type);
}
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)
{
memset(ptr, 0, sizeof(*ptr));
}
BPF_CALL_4(bpf_dynptr_from_mem, void *, data, u64, size, u64, flags, struct bpf_dynptr_kern *, ptr)
{
int err;
BTF_TYPE_EMIT(struct bpf_dynptr);
err = bpf_dynptr_check_size(size);
if (err)
goto error;
/* flags is currently unsupported */
if (flags) {
err = -EINVAL;
goto error;
}
bpf_dynptr_init(ptr, data, BPF_DYNPTR_TYPE_LOCAL, 0, size);
return 0;
error:
bpf_dynptr_set_null(ptr);
return err;
}
static const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
.func = bpf_dynptr_from_mem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL | MEM_UNINIT | MEM_WRITE,
};
static int __bpf_dynptr_read(void *dst, u64 len, const struct bpf_dynptr_kern *src,
u64 offset, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!src->data || flags)
return -EINVAL;
err = bpf_dynptr_check_off_len(src, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(src);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst, src->data + src->offset + offset, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_XDP:
return __bpf_xdp_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_SKB_META:
memmove(dst, bpf_skb_meta_pointer(src->data, src->offset + offset), len);
return 0;
case BPF_DYNPTR_TYPE_FILE:
return bpf_file_fetch_bytes(src->data, offset, dst, len);
default:
WARN_ONCE(true, "bpf_dynptr_read: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_read, void *, dst, u64, len, const struct bpf_dynptr_kern *, src,
u64, offset, u64, flags)
{
return __bpf_dynptr_read(dst, len, src, offset, flags);
}
static const struct bpf_func_proto bpf_dynptr_read_proto = {
.func = bpf_dynptr_read,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
int __bpf_dynptr_write(const struct bpf_dynptr_kern *dst, u64 offset, void *src,
u64 len, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!dst->data || __bpf_dynptr_is_rdonly(dst))
return -EINVAL;
err = bpf_dynptr_check_off_len(dst, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(dst);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
if (flags)
return -EINVAL;
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst->data + dst->offset + offset, src, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_store_bytes(dst->data, dst->offset + offset, src, len,
flags);
case BPF_DYNPTR_TYPE_XDP:
if (flags)
return -EINVAL;
return __bpf_xdp_store_bytes(dst->data, dst->offset + offset, src, len);
case BPF_DYNPTR_TYPE_SKB_META:
return __bpf_skb_meta_store_bytes(dst->data, dst->offset + offset, src,
len, flags);
default:
WARN_ONCE(true, "bpf_dynptr_write: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_write, const struct bpf_dynptr_kern *, dst, u64, offset, void *, src,
u64, len, u64, flags)
{
return __bpf_dynptr_write(dst, offset, src, len, flags);
}
static const struct bpf_func_proto bpf_dynptr_write_proto = {
.func = bpf_dynptr_write,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_dynptr_data, const struct bpf_dynptr_kern *, ptr, u64, offset, u64, len)
{
enum bpf_dynptr_type type;
int err;
if (!ptr->data)
return 0;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return 0;
if (__bpf_dynptr_is_rdonly(ptr))
return 0;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return (unsigned long)(ptr->data + ptr->offset + offset);
case BPF_DYNPTR_TYPE_SKB:
case BPF_DYNPTR_TYPE_XDP:
case BPF_DYNPTR_TYPE_SKB_META:
/* skb and xdp dynptrs should use bpf_dynptr_slice / bpf_dynptr_slice_rdwr */
return 0;
default:
WARN_ONCE(true, "bpf_dynptr_data: unknown dynptr type %d\n", type);
return 0;
}
}
static const struct bpf_func_proto bpf_dynptr_data_proto = {
.func = bpf_dynptr_data,
.gpl_only = false,
.ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
};
const struct bpf_func_proto bpf_get_current_task_proto __weak;
const struct bpf_func_proto bpf_get_current_task_btf_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_str_proto __weak;
const struct bpf_func_proto bpf_task_pt_regs_proto __weak;
const struct bpf_func_proto bpf_perf_event_read_proto __weak;
const struct bpf_func_proto bpf_send_signal_proto __weak;
const struct bpf_func_proto bpf_send_signal_thread_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_sleepable_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_proto __weak;
const struct bpf_func_proto bpf_get_branch_snapshot_proto __weak;
const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_map_lookup_elem:
return &bpf_map_lookup_elem_proto;
case BPF_FUNC_map_update_elem:
return &bpf_map_update_elem_proto;
case BPF_FUNC_map_delete_elem:
return &bpf_map_delete_elem_proto;
case BPF_FUNC_map_push_elem:
return &bpf_map_push_elem_proto;
case BPF_FUNC_map_pop_elem:
return &bpf_map_pop_elem_proto;
case BPF_FUNC_map_peek_elem:
return &bpf_map_peek_elem_proto;
case BPF_FUNC_map_lookup_percpu_elem:
return &bpf_map_lookup_percpu_elem_proto;
case BPF_FUNC_get_prandom_u32:
return &bpf_get_prandom_u32_proto;
case BPF_FUNC_get_smp_processor_id:
return &bpf_get_raw_smp_processor_id_proto;
case BPF_FUNC_get_numa_node_id:
return &bpf_get_numa_node_id_proto;
case BPF_FUNC_tail_call:
return &bpf_tail_call_proto;
case BPF_FUNC_ktime_get_ns:
return &bpf_ktime_get_ns_proto;
case BPF_FUNC_ktime_get_boot_ns:
return &bpf_ktime_get_boot_ns_proto;
case BPF_FUNC_ktime_get_tai_ns:
return &bpf_ktime_get_tai_ns_proto;
case BPF_FUNC_ringbuf_output:
return &bpf_ringbuf_output_proto;
case BPF_FUNC_ringbuf_reserve:
return &bpf_ringbuf_reserve_proto;
case BPF_FUNC_ringbuf_submit:
return &bpf_ringbuf_submit_proto;
case BPF_FUNC_ringbuf_discard:
return &bpf_ringbuf_discard_proto;
case BPF_FUNC_ringbuf_query:
return &bpf_ringbuf_query_proto;
case BPF_FUNC_strncmp:
return &bpf_strncmp_proto;
case BPF_FUNC_strtol:
return &bpf_strtol_proto;
case BPF_FUNC_strtoul:
return &bpf_strtoul_proto;
case BPF_FUNC_get_current_pid_tgid:
return &bpf_get_current_pid_tgid_proto;
case BPF_FUNC_get_ns_current_pid_tgid:
return &bpf_get_ns_current_pid_tgid_proto;
case BPF_FUNC_get_current_uid_gid:
return &bpf_get_current_uid_gid_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_BPF))
return NULL;
switch (func_id) {
case BPF_FUNC_spin_lock:
return &bpf_spin_lock_proto;
case BPF_FUNC_spin_unlock:
return &bpf_spin_unlock_proto;
case BPF_FUNC_jiffies64:
return &bpf_jiffies64_proto;
case BPF_FUNC_per_cpu_ptr:
return &bpf_per_cpu_ptr_proto;
case BPF_FUNC_this_cpu_ptr:
return &bpf_this_cpu_ptr_proto;
case BPF_FUNC_timer_init:
return &bpf_timer_init_proto;
case BPF_FUNC_timer_set_callback:
return &bpf_timer_set_callback_proto;
case BPF_FUNC_timer_start:
return &bpf_timer_start_proto;
case BPF_FUNC_timer_cancel:
return &bpf_timer_cancel_proto;
case BPF_FUNC_kptr_xchg:
return &bpf_kptr_xchg_proto;
case BPF_FUNC_for_each_map_elem:
return &bpf_for_each_map_elem_proto;
case BPF_FUNC_loop:
return &bpf_loop_proto;
case BPF_FUNC_user_ringbuf_drain:
return &bpf_user_ringbuf_drain_proto;
case BPF_FUNC_ringbuf_reserve_dynptr:
return &bpf_ringbuf_reserve_dynptr_proto;
case BPF_FUNC_ringbuf_submit_dynptr:
return &bpf_ringbuf_submit_dynptr_proto;
case BPF_FUNC_ringbuf_discard_dynptr:
return &bpf_ringbuf_discard_dynptr_proto;
case BPF_FUNC_dynptr_from_mem:
return &bpf_dynptr_from_mem_proto;
case BPF_FUNC_dynptr_read:
return &bpf_dynptr_read_proto;
case BPF_FUNC_dynptr_write:
return &bpf_dynptr_write_proto;
case BPF_FUNC_dynptr_data:
return &bpf_dynptr_data_proto;
#ifdef CONFIG_CGROUPS
case BPF_FUNC_cgrp_storage_get:
return &bpf_cgrp_storage_get_proto;
case BPF_FUNC_cgrp_storage_delete:
return &bpf_cgrp_storage_delete_proto;
case BPF_FUNC_get_current_cgroup_id:
return &bpf_get_current_cgroup_id_proto;
case BPF_FUNC_get_current_ancestor_cgroup_id:
return &bpf_get_current_ancestor_cgroup_id_proto;
case BPF_FUNC_current_task_under_cgroup:
return &bpf_current_task_under_cgroup_proto;
#endif
#ifdef CONFIG_CGROUP_NET_CLASSID
case BPF_FUNC_get_cgroup_classid:
return &bpf_get_cgroup_classid_curr_proto;
#endif
case BPF_FUNC_task_storage_get:
return &bpf_task_storage_get_proto;
case BPF_FUNC_task_storage_delete:
return &bpf_task_storage_delete_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_PERFMON))
return NULL;
switch (func_id) {
case BPF_FUNC_trace_printk:
return bpf_get_trace_printk_proto();
case BPF_FUNC_get_current_task:
return &bpf_get_current_task_proto;
case BPF_FUNC_get_current_task_btf:
return &bpf_get_current_task_btf_proto;
case BPF_FUNC_get_current_comm:
return &bpf_get_current_comm_proto;
case BPF_FUNC_probe_read_user:
return &bpf_probe_read_user_proto;
case BPF_FUNC_probe_read_kernel:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_proto;
case BPF_FUNC_probe_read_user_str:
return &bpf_probe_read_user_str_proto;
case BPF_FUNC_probe_read_kernel_str:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_str_proto;
case BPF_FUNC_copy_from_user:
return &bpf_copy_from_user_proto;
case BPF_FUNC_copy_from_user_task:
return &bpf_copy_from_user_task_proto;
case BPF_FUNC_snprintf_btf:
return &bpf_snprintf_btf_proto;
case BPF_FUNC_snprintf:
return &bpf_snprintf_proto;
case BPF_FUNC_task_pt_regs:
return &bpf_task_pt_regs_proto;
case BPF_FUNC_trace_vprintk:
return bpf_get_trace_vprintk_proto();
case BPF_FUNC_perf_event_read_value:
return bpf_get_perf_event_read_value_proto();
case BPF_FUNC_perf_event_read:
return &bpf_perf_event_read_proto;
case BPF_FUNC_send_signal:
return &bpf_send_signal_proto;
case BPF_FUNC_send_signal_thread:
return &bpf_send_signal_thread_proto;
case BPF_FUNC_get_task_stack:
return prog->sleepable ? &bpf_get_task_stack_sleepable_proto
: &bpf_get_task_stack_proto;
case BPF_FUNC_get_branch_snapshot:
return &bpf_get_branch_snapshot_proto;
case BPF_FUNC_find_vma:
return &bpf_find_vma_proto;
default:
return NULL;
}
}
EXPORT_SYMBOL_GPL(bpf_base_func_proto);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock)
{
struct list_head *head = list_head, *orig_head = list_head;
BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
/* Do the actual list draining outside the lock to not hold the lock for
* too long, and also prevent deadlocks if tracing programs end up
* executing on entry/exit of functions called inside the critical
* section, and end up doing map ops that call bpf_list_head_free for
* the same map value again.
*/
__bpf_spin_lock_irqsave(spin_lock);
if (!head->next || list_empty(head))
goto unlock;
head = head->next;
unlock:
INIT_LIST_HEAD(orig_head);
__bpf_spin_unlock_irqrestore(spin_lock);
while (head != orig_head) {
void *obj = head;
obj -= field->graph_root.node_offset;
head = head->next;
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
/* Like rbtree_postorder_for_each_entry_safe, but 'pos' and 'n' are
* 'rb_node *', so field name of rb_node within containing struct is not
* needed.
*
* Since bpf_rb_tree's node type has a corresponding struct btf_field with
* graph_root.node_offset, it's not necessary to know field name
* or type of node struct
*/
#define bpf_rbtree_postorder_for_each_entry_safe(pos, n, root) \
for (pos = rb_first_postorder(root); \
pos && ({ n = rb_next_postorder(pos); 1; }); \
pos = n)
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock)
{
struct rb_root_cached orig_root, *root = rb_root;
struct rb_node *pos, *n;
void *obj;
BUILD_BUG_ON(sizeof(struct rb_root_cached) > sizeof(struct bpf_rb_root));
BUILD_BUG_ON(__alignof__(struct rb_root_cached) > __alignof__(struct bpf_rb_root));
__bpf_spin_lock_irqsave(spin_lock);
orig_root = *root;
*root = RB_ROOT_CACHED;
__bpf_spin_unlock_irqrestore(spin_lock);
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
obj = pos;
obj -= field->graph_root.node_offset;
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
__bpf_kfunc_start_defs();
/**
* bpf_obj_new() - allocate an object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate an object of the type identified by @local_type_id__k and
* initialize its special fields. BPF programs can use
* bpf_core_type_id_local() to provide @local_type_id__k. The verifier
* rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
void *p;
p = bpf_mem_alloc(&bpf_global_ma, size);
if (!p)
return NULL;
if (meta)
bpf_obj_init(meta->record, p);
return p;
}
__bpf_kfunc void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_obj_new(local_type_id__k, meta__ign);
}
/**
* bpf_percpu_obj_new() - allocate a percpu object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate a percpu object of the type identified by @local_type_id__k. BPF
* programs can use bpf_core_type_id_local() to provide @local_type_id__k.
* The verifier rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated percpu object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_percpu_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
/* The verifier has ensured that meta must be NULL */
return bpf_mem_alloc(&bpf_global_percpu_ma, size);
}
__bpf_kfunc void *bpf_percpu_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_percpu_obj_new(local_type_id__k, meta__ign);
}
/* Must be called under migrate_disable(), as required by bpf_mem_free */
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu)
{
struct bpf_mem_alloc *ma;
if (rec && rec->refcount_off >= 0 &&
!refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) {
/* Object is refcounted and refcount_dec didn't result in 0
* refcount. Return without freeing the object
*/
return;
}
if (rec)
bpf_obj_free_fields(rec, p);
if (percpu)
ma = &bpf_global_percpu_ma;
else
ma = &bpf_global_ma;
bpf_mem_free_rcu(ma, p);
}
/**
* bpf_obj_drop() - drop a previously allocated object
* @p__alloc: object to free
* @meta: verifier-supplied struct metadata
*
* Destroy special fields in @p__alloc as needed and free the object. The
* verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
void *p = p__alloc;
__bpf_obj_drop_impl(p, meta ? meta->record : NULL, false);
}
__bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
{
return bpf_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_percpu_obj_drop() - drop a previously allocated percpu object
* @p__alloc: percpu object to free
* @meta: verifier-supplied struct metadata
*
* Free @p__alloc. The verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_percpu_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
/* The verifier has ensured that meta must be NULL */
bpf_mem_free_rcu(&bpf_global_percpu_ma, p__alloc);
}
__bpf_kfunc void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign)
{
bpf_percpu_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_refcount_acquire() - turn a local kptr into an owning reference
* @p__refcounted_kptr: non-owning local kptr
* @meta: verifier-supplied struct metadata
*
* Increment the refcount for @p__refcounted_kptr. The verifier rewrites
* @meta; BPF programs do not set it.
*
* Return: Owning reference to @p__refcounted_kptr, or %NULL on failure.
*/
__bpf_kfunc void *bpf_refcount_acquire(void *p__refcounted_kptr, struct btf_struct_meta *meta)
{
struct bpf_refcount *ref;
/* Could just cast directly to refcount_t *, but need some code using
* bpf_refcount type so that it is emitted in vmlinux BTF
*/
ref = (struct bpf_refcount *)(p__refcounted_kptr + meta->record->refcount_off);
if (!refcount_inc_not_zero((refcount_t *)ref))
return NULL;
/* Verifier strips KF_RET_NULL if input is owned ref, see is_kfunc_ret_null
* in verifier.c
*/
return (void *)p__refcounted_kptr;
}
__bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta__ign)
{
return bpf_refcount_acquire(p__refcounted_kptr, meta__ign);
}
static int __bpf_list_add(struct bpf_list_node_kern *node,
struct bpf_list_head *head,
bool tail, struct btf_record *rec, u64 off)
{
struct list_head *n = &node->list_head, *h = (void *)head;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
/* node->owner != NULL implies !list_empty(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
tail ? list_add_tail(n, h) : list_add(n, h);
WRITE_ONCE(node->owner, head);
return 0;
}
/**
* bpf_list_push_front() - add a node to the front of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the front of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_front(head, node, meta__ign, off);
}
/**
* bpf_list_push_back() - add a node to the back of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the back of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_back(head, node, meta__ign, off);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
{
struct list_head *n, *h = (void *)head;
struct bpf_list_node_kern *node;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
if (list_empty(h))
return NULL;
n = tail ? h->prev : h->next;
node = container_of(n, struct bpf_list_node_kern, list_head);
if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))
return NULL;
list_del_init(n);
WRITE_ONCE(node->owner, NULL);
return (struct bpf_list_node *)n;
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
{
return __bpf_list_del(head, false);
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
{
return __bpf_list_del(head, true);
}
__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->next;
}
__bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->prev;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
struct rb_root_cached *r = (struct rb_root_cached *)root;
struct rb_node *n = &node_internal->rb_node;
/* node_internal->owner != root implies either RB_EMPTY_NODE(n) or
* n is owned by some other tree. No need to check RB_EMPTY_NODE(n)
*/
if (READ_ONCE(node_internal->owner) != root)
return NULL;
rb_erase_cached(n, r);
RB_CLEAR_NODE(n);
WRITE_ONCE(node_internal->owner, NULL);
return (struct bpf_rb_node *)n;
}
/* Need to copy rbtree_add_cached's logic here because our 'less' is a BPF
* program
*/
static int __bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node_kern *node,
void *less, struct btf_record *rec, u64 off)
{
struct rb_node **link = &((struct rb_root_cached *)root)->rb_root.rb_node;
struct rb_node *parent = NULL, *n = &node->rb_node;
bpf_callback_t cb = (bpf_callback_t)less;
bool leftmost = true;
/* node->owner != NULL implies !RB_EMPTY_NODE(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
while (*link) {
parent = *link;
if (cb((uintptr_t)node, (uintptr_t)parent, 0, 0, 0)) {
link = &parent->rb_left;
} else {
link = &parent->rb_right;
leftmost = false;
}
}
rb_link_node(n, parent, link);
rb_insert_color_cached(n, (struct rb_root_cached *)root, leftmost);
WRITE_ONCE(node->owner, root);
return 0;
}
/**
* bpf_rbtree_add() - add a node to a BPF rbtree
* @root: tree root
* @node: node to insert
* @less: comparator used to order nodes
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node into @root using @less. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked in a tree.
*/
__bpf_kfunc int bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_rb_node_kern *n = (void *)node;
return __bpf_rbtree_add(root, n, (void *)less, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
void *meta__ign, u64 off)
{
return bpf_rbtree_add(root, node, less, meta__ign, off);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)rb_first_cached(r);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_root(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)r->rb_root.rb_node;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_left(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_left;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_right(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_right;
}
/**
* bpf_task_acquire - Acquire a reference to a task. A task acquired by this
* kfunc which is not stored in a map as a kptr, must be released by calling
* bpf_task_release().
* @p: The task on which a reference is being acquired.
*/
__bpf_kfunc struct task_struct *bpf_task_acquire(struct task_struct *p)
{
if (refcount_inc_not_zero(&p->rcu_users))
return p;
return NULL;
}
/**
* bpf_task_release - Release the reference acquired on a task.
* @p: The task on which a reference is being released.
*/
__bpf_kfunc void bpf_task_release(struct task_struct *p)
{
put_task_struct_rcu_user(p);
}
__bpf_kfunc void bpf_task_release_dtor(void *p)
{
put_task_struct_rcu_user(p);
}
CFI_NOSEAL(bpf_task_release_dtor);
#ifdef CONFIG_CGROUPS
/**
* bpf_cgroup_acquire - Acquire a reference to a cgroup. A cgroup acquired by
* this kfunc which is not stored in a map as a kptr, must be released by
* calling bpf_cgroup_release().
* @cgrp: The cgroup on which a reference is being acquired.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)
{
return cgroup_tryget(cgrp) ? cgrp : NULL;
}
/**
* bpf_cgroup_release - Release the reference acquired on a cgroup.
* If this kfunc is invoked in an RCU read region, the cgroup is guaranteed to
* not be freed until the current grace period has ended, even if its refcount
* drops to 0.
* @cgrp: The cgroup on which a reference is being released.
*/
__bpf_kfunc void bpf_cgroup_release(struct cgroup *cgrp)
{
cgroup_put(cgrp);
}
__bpf_kfunc void bpf_cgroup_release_dtor(void *cgrp)
{
cgroup_put(cgrp);
}
CFI_NOSEAL(bpf_cgroup_release_dtor);
/**
* bpf_cgroup_ancestor - Perform a lookup on an entry in a cgroup's ancestor
* array. A cgroup returned by this kfunc which is not subsequently stored in a
* map, must be released by calling bpf_cgroup_release().
* @cgrp: The cgroup for which we're performing a lookup.
* @level: The level of ancestor to look up.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)
{
struct cgroup *ancestor;
if (level > cgrp->level || level < 0)
return NULL;
/* cgrp's refcnt could be 0 here, but ancestors can still be accessed */
ancestor = cgrp->ancestors[level];
if (!cgroup_tryget(ancestor))
return NULL;
return ancestor;
}
/**
* bpf_cgroup_from_id - Find a cgroup from its ID. A cgroup returned by this
* kfunc which is not subsequently stored in a map, must be released by calling
* bpf_cgroup_release().
* @cgid: cgroup id.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_from_id(u64 cgid)
{
struct cgroup *cgrp;
cgrp = __cgroup_get_from_id(cgid);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
/**
* bpf_task_under_cgroup - wrap task_under_cgroup_hierarchy() as a kfunc, test
* task's membership of cgroup ancestry.
* @task: the task to be tested
* @ancestor: possible ancestor of @task's cgroup
*
* Tests whether @task's default cgroup hierarchy is a descendant of @ancestor.
* It follows all the same rules as cgroup_is_descendant, and only applies
* to the default hierarchy.
*/
__bpf_kfunc long bpf_task_under_cgroup(struct task_struct *task,
struct cgroup *ancestor)
{
long ret;
rcu_read_lock();
ret = task_under_cgroup_hierarchy(task, ancestor);
rcu_read_unlock();
return ret;
}
BPF_CALL_2(bpf_current_task_under_cgroup, struct bpf_map *, map, u32, idx)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
struct cgroup *cgrp;
if (unlikely(idx >= array->map.max_entries))
return -E2BIG;
cgrp = READ_ONCE(array->ptrs[idx]);
if (unlikely(!cgrp))
return -EAGAIN;
return task_under_cgroup_hierarchy(current, cgrp);
}
const struct bpf_func_proto bpf_current_task_under_cgroup_proto = {
.func = bpf_current_task_under_cgroup,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
};
/**
* bpf_task_get_cgroup1 - Acquires the associated cgroup of a task within a
* specific cgroup1 hierarchy. The cgroup1 hierarchy is identified by its
* hierarchy ID.
* @task: The target task
* @hierarchy_id: The ID of a cgroup1 hierarchy
*
* On success, the cgroup is returen. On failure, NULL is returned.
*/
__bpf_kfunc struct cgroup *
bpf_task_get_cgroup1(struct task_struct *task, int hierarchy_id)
{
struct cgroup *cgrp = task_get_cgroup1(task, hierarchy_id);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
#endif /* CONFIG_CGROUPS */
/**
* bpf_task_from_pid - Find a struct task_struct from its pid by looking it up
* in the root pid namespace idr. If a task is returned, it must either be
* stored in a map, or released with bpf_task_release().
* @pid: The pid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_pid_ns(pid, &init_pid_ns);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_task_from_vpid - Find a struct task_struct from its vpid by looking it up
* in the pid namespace of the current task. If a task is returned, it must
* either be stored in a map, or released with bpf_task_release().
* @vpid: The vpid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_vpid(vpid);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_dynptr_slice() - Obtain a read-only pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* If the intention is to write to the data slice, please use
* bpf_dynptr_slice_rdwr.
*
* The user must check that the returned pointer is not null before using it.
*
* Please note that in the case of skb and xdp dynptrs, bpf_dynptr_slice
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a read-only
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
enum bpf_dynptr_type type;
u64 len = buffer__szk;
int err;
if (!ptr->data)
return NULL;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return NULL;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return ptr->data + ptr->offset + offset;
case BPF_DYNPTR_TYPE_SKB:
if (buffer__nullable)
return skb_header_pointer(ptr->data, ptr->offset + offset, len, buffer__nullable);
else
return skb_pointer_if_linear(ptr->data, ptr->offset + offset, len);
case BPF_DYNPTR_TYPE_XDP:
{
void *xdp_ptr = bpf_xdp_pointer(ptr->data, ptr->offset + offset, len);
if (!IS_ERR_OR_NULL(xdp_ptr))
return xdp_ptr;
if (!buffer__nullable)
return NULL;
bpf_xdp_copy_buf(ptr->data, ptr->offset + offset, buffer__nullable, len, false);
return buffer__nullable;
}
case BPF_DYNPTR_TYPE_SKB_META:
return bpf_skb_meta_pointer(ptr->data, ptr->offset + offset);
case BPF_DYNPTR_TYPE_FILE:
err = bpf_file_fetch_bytes(ptr->data, offset, buffer__nullable, buffer__szk);
return err ? NULL : buffer__nullable;
default:
WARN_ONCE(true, "unknown dynptr type %d\n", type);
return NULL;
}
}
/**
* bpf_dynptr_slice_rdwr() - Obtain a writable pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* The returned pointer is writable and may point to either directly the dynptr
* data at the requested offset or to the buffer if unable to obtain a direct
* data pointer to (example: the requested slice is to the paged area of an skb
* packet). In the case where the returned pointer is to the buffer, the user
* is responsible for persisting writes through calling bpf_dynptr_write(). This
* usually looks something like this pattern:
*
* struct eth_hdr *eth = bpf_dynptr_slice_rdwr(&dynptr, 0, buffer, sizeof(buffer));
* if (!eth)
* return TC_ACT_SHOT;
*
* // mutate eth header //
*
* if (eth == buffer)
* bpf_dynptr_write(&ptr, 0, buffer, sizeof(buffer), 0);
*
* Please note that, as in the example above, the user must check that the
* returned pointer is not null before using it.
*
* Please also note that in the case of skb and xdp dynptrs, bpf_dynptr_slice_rdwr
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice_rdwr will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data || __bpf_dynptr_is_rdonly(ptr))
return NULL;
/* bpf_dynptr_slice_rdwr is the same logic as bpf_dynptr_slice.
*
* For skb-type dynptrs, it is safe to write into the returned pointer
* if the bpf program allows skb data writes. There are two possibilities
* that may occur when calling bpf_dynptr_slice_rdwr:
*
* 1) The requested slice is in the head of the skb. In this case, the
* returned pointer is directly to skb data, and if the skb is cloned, the
* verifier will have uncloned it (see bpf_unclone_prologue()) already.
* The pointer can be directly written into.
*
* 2) Some portion of the requested slice is in the paged buffer area.
* In this case, the requested data will be copied out into the buffer
* and the returned pointer will be a pointer to the buffer. The skb
* will not be pulled. To persist the write, the user will need to call
* bpf_dynptr_write(), which will pull the skb and commit the write.
*
* Similarly for xdp programs, if the requested slice is not across xdp
* fragments, then a direct pointer will be returned, otherwise the data
* will be copied out into the buffer and the user will need to call
* bpf_dynptr_write() to commit changes.
*/
return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
}
__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 size;
if (!ptr->data || start > end)
return -EINVAL;
size = __bpf_dynptr_size(ptr);
if (start > size || end > size)
return -ERANGE;
bpf_dynptr_advance_offset(ptr, start);
bpf_dynptr_set_size(ptr, end - start);
return 0;
}
__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
return !ptr->data;
}
__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return false;
return __bpf_dynptr_is_rdonly(ptr);
}
__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return -EINVAL;
return __bpf_dynptr_size(ptr);
}
__bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
struct bpf_dynptr *clone__uninit)
{
struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data) {
bpf_dynptr_set_null(clone);
return -EINVAL;
}
*clone = *ptr;
return 0;
}
/**
* bpf_dynptr_copy() - Copy data from one dynptr to another.
* @dst_ptr: Destination dynptr - where data should be copied to
* @dst_off: Offset into the destination dynptr
* @src_ptr: Source dynptr - where data should be copied from
* @src_off: Offset into the source dynptr
* @size: Length of the data to copy from source to destination
*
* Copies data from source dynptr to destination dynptr.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
{
struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
void *src_slice, *dst_slice;
char buf[256];
u64 off;
src_slice = bpf_dynptr_slice(src_ptr, src_off, NULL, size);
dst_slice = bpf_dynptr_slice_rdwr(dst_ptr, dst_off, NULL, size);
if (src_slice && dst_slice) {
memmove(dst_slice, src_slice, size);
return 0;
}
if (src_slice)
return __bpf_dynptr_write(dst, dst_off, src_slice, size, 0);
if (dst_slice)
return __bpf_dynptr_read(dst_slice, size, src, src_off, 0);
if (bpf_dynptr_check_off_len(dst, dst_off, size) ||
bpf_dynptr_check_off_len(src, src_off, size))
return -E2BIG;
off = 0;
while (off < size) {
u64 chunk_sz = min_t(u64, sizeof(buf), size - off);
int err;
err = __bpf_dynptr_read(buf, chunk_sz, src, src_off + off, 0);
if (err)
return err;
err = __bpf_dynptr_write(dst, dst_off + off, buf, chunk_sz, 0);
if (err)
return err;
off += chunk_sz;
}
return 0;
}
/**
* bpf_dynptr_memset() - Fill dynptr memory with a constant byte.
* @p: Destination dynptr - where data will be filled
* @offset: Offset into the dynptr to start filling from
* @size: Number of bytes to fill
* @val: Constant byte to fill the memory with
*
* Fills the @size bytes of the memory area pointed to by @p
* at @offset with the constant byte @val.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 chunk_sz, write_off;
char buf[256];
void* slice;
int err;
slice = bpf_dynptr_slice_rdwr(p, offset, NULL, size);
if (likely(slice)) {
memset(slice, val, size);
return 0;
}
if (__bpf_dynptr_is_rdonly(ptr))
return -EINVAL;
err = bpf_dynptr_check_off_len(ptr, offset, size);
if (err)
return err;
/* Non-linear data under the dynptr, write from a local buffer */
chunk_sz = min_t(u64, sizeof(buf), size);
memset(buf, val, chunk_sz);
for (write_off = 0; write_off < size; write_off += chunk_sz) {
chunk_sz = min_t(u64, sizeof(buf), size - write_off);
err = __bpf_dynptr_write(ptr, offset + write_off, buf, chunk_sz, 0);
if (err)
return err;
}
return 0;
}
__bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
{
return obj;
}
__bpf_kfunc void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k)
{
return (void *)obj__ign;
}
__bpf_kfunc void bpf_rcu_read_lock(void)
{
rcu_read_lock();
}
__bpf_kfunc void bpf_rcu_read_unlock(void)
{
rcu_read_unlock();
}
struct bpf_throw_ctx {
struct bpf_prog_aux *aux;
u64 sp;
u64 bp;
int cnt;
};
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
{
struct bpf_throw_ctx *ctx = cookie;
struct bpf_prog *prog;
/*
* The RCU read lock is held to safely traverse the latch tree, but we
* don't need its protection when accessing the prog, since it has an
* active stack frame on the current stack trace, and won't disappear.
*/
rcu_read_lock();
prog = bpf_prog_ksym_find(ip);
rcu_read_unlock();
if (!prog)
return !ctx->cnt;
ctx->cnt++;
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
ctx->sp = sp;
ctx->bp = bp;
return false;
}
__bpf_kfunc void bpf_throw(u64 cookie)
{
struct bpf_throw_ctx ctx = {};
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
WARN_ON_ONCE(!ctx.aux->exception_boundary);
WARN_ON_ONCE(!ctx.bp);
WARN_ON_ONCE(!ctx.cnt);
/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
* deeper stack depths than ctx.sp as we do not return from bpf_throw,
* which skips compiler generated instrumentation to do the same.
*/
kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_map *map = p__map;
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq));
if (flags)
return -EINVAL;
return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ);
}
__bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_work *w;
if (flags)
return -EINVAL;
w = READ_ONCE(async->work);
if (!w || !READ_ONCE(w->cb.prog))
return -EINVAL;
if (!refcount_inc_not_zero(&w->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
schedule_work(&w->work);
bpf_async_refcount_put(&w->cb);
return 0;
} else {
return bpf_async_schedule_op(&w->cb, BPF_ASYNC_START, 0, 0);
}
}
__bpf_kfunc int bpf_wq_set_callback(struct bpf_wq *wq,
int (callback_fn)(void *map, int *key, void *value),
unsigned int flags,
struct bpf_prog_aux *aux)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
if (flags)
return -EINVAL;
return __bpf_async_set_callback(async, callback_fn, aux->prog);
}
__bpf_kfunc void bpf_preempt_disable(void)
{
preempt_disable();
}
__bpf_kfunc void bpf_preempt_enable(void)
{
preempt_enable();
}
struct bpf_iter_bits {
__u64 __opaque[2];
} __aligned(8);
#define BITS_ITER_NR_WORDS_MAX 511
struct bpf_iter_bits_kern {
union {
__u64 *bits;
__u64 bits_copy;
};
int nr_bits;
int bit;
} __aligned(8);
/* On 64-bit hosts, unsigned long and u64 have the same size, so passing
* a u64 pointer and an unsigned long pointer to find_next_bit() will
* return the same result, as both point to the same 8-byte area.
*
* For 32-bit little-endian hosts, using a u64 pointer or unsigned long
* pointer also makes no difference. This is because the first iterated
* unsigned long is composed of bits 0-31 of the u64 and the second unsigned
* long is composed of bits 32-63 of the u64.
*
* However, for 32-bit big-endian hosts, this is not the case. The first
* iterated unsigned long will be bits 32-63 of the u64, so swap these two
* ulong values within the u64.
*/
static void swap_ulong_in_u64(u64 *bits, unsigned int nr)
{
#if (BITS_PER_LONG == 32) && defined(__BIG_ENDIAN)
unsigned int i;
for (i = 0; i < nr; i++)
bits[i] = (bits[i] >> 32) | ((u64)(u32)bits[i] << 32);
#endif
}
/**
* bpf_iter_bits_new() - Initialize a new bits iterator for a given memory area
* @it: The new bpf_iter_bits to be created
* @unsafe_ptr__ign: A pointer pointing to a memory area to be iterated over
* @nr_words: The size of the specified memory area, measured in 8-byte units.
* The maximum value of @nr_words is @BITS_ITER_NR_WORDS_MAX. This limit may be
* further reduced by the BPF memory allocator implementation.
*
* This function initializes a new bpf_iter_bits structure for iterating over
* a memory area which is specified by the @unsafe_ptr__ign and @nr_words. It
* copies the data of the memory area to the newly created bpf_iter_bits @it for
* subsequent iteration operations.
*
* On success, 0 is returned. On failure, ERR is returned.
*/
__bpf_kfunc int
bpf_iter_bits_new(struct bpf_iter_bits *it, const u64 *unsafe_ptr__ign, u32 nr_words)
{
struct bpf_iter_bits_kern *kit = (void *)it;
u32 nr_bytes = nr_words * sizeof(u64);
u32 nr_bits = BYTES_TO_BITS(nr_bytes);
int err;
BUILD_BUG_ON(sizeof(struct bpf_iter_bits_kern) != sizeof(struct bpf_iter_bits));
BUILD_BUG_ON(__alignof__(struct bpf_iter_bits_kern) !=
__alignof__(struct bpf_iter_bits));
kit->nr_bits = 0;
kit->bits_copy = 0;
kit->bit = -1;
if (!unsafe_ptr__ign || !nr_words)
return -EINVAL;
if (nr_words > BITS_ITER_NR_WORDS_MAX)
return -E2BIG;
/* Optimization for u64 mask */
if (nr_bits == 64) {
err = bpf_probe_read_kernel_common(&kit->bits_copy, nr_bytes, unsafe_ptr__ign);
if (err)
return -EFAULT;
swap_ulong_in_u64(&kit->bits_copy, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
if (bpf_mem_alloc_check_size(false, nr_bytes))
return -E2BIG;
/* Fallback to memalloc */
kit->bits = bpf_mem_alloc(&bpf_global_ma, nr_bytes);
if (!kit->bits)
return -ENOMEM;
err = bpf_probe_read_kernel_common(kit->bits, nr_bytes, unsafe_ptr__ign);
if (err) {
bpf_mem_free(&bpf_global_ma, kit->bits);
return err;
}
swap_ulong_in_u64(kit->bits, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
/**
* bpf_iter_bits_next() - Get the next bit in a bpf_iter_bits
* @it: The bpf_iter_bits to be checked
*
* This function returns a pointer to a number representing the value of the
* next bit in the bits.
*
* If there are no further bits available, it returns NULL.
*/
__bpf_kfunc int *bpf_iter_bits_next(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
int bit = kit->bit, nr_bits = kit->nr_bits;
const void *bits;
if (!nr_bits || bit >= nr_bits)
return NULL;
bits = nr_bits == 64 ? &kit->bits_copy : kit->bits;
bit = find_next_bit(bits, nr_bits, bit + 1);
if (bit >= nr_bits) {
kit->bit = bit;
return NULL;
}
kit->bit = bit;
return &kit->bit;
}
/**
* bpf_iter_bits_destroy() - Destroy a bpf_iter_bits
* @it: The bpf_iter_bits to be destroyed
*
* Destroy the resource associated with the bpf_iter_bits.
*/
__bpf_kfunc void bpf_iter_bits_destroy(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
if (kit->nr_bits <= 64)
return;
bpf_mem_free(&bpf_global_ma, kit->bits);
}
/**
* bpf_copy_from_user_str() - Copy a string from an unsafe user address
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address, in user space.
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL-terminated string from userspace to BPF space. If user string is
* too long this will still ensure zero termination in the dst buffer unless
* buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst to 0 on success and
* memset all of @dst on failure.
*/
__bpf_kfunc int bpf_copy_from_user_str(void *dst, u32 dst__sz, const void __user *unsafe_ptr__ign, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(!dst__sz))
return 0;
ret = strncpy_from_user(dst, unsafe_ptr__ign, dst__sz - 1);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst + ret, 0, dst__sz - ret);
else
((char *)dst)[ret] = '\0';
return ret + 1;
}
/**
* bpf_copy_from_user_task_str() - Copy a string from an task's address space
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address in the task's address space.
* @tsk: The task whose address space will be used
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL terminated string from a task's address space to @dst__sz
* buffer. If user string is too long this will still ensure zero termination
* in the @dst__sz buffer unless buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst__sz to 0 on success
* and memset all of @dst__sz on failure.
*
* Return: The number of copied bytes on success including the NUL terminator.
* A negative error code on failure.
*/
__bpf_kfunc int bpf_copy_from_user_task_str(void *dst, u32 dst__sz,
const void __user *unsafe_ptr__ign,
struct task_struct *tsk, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(dst__sz == 0))
return 0;
ret = copy_remote_vm_str(tsk, (unsigned long)unsafe_ptr__ign, dst, dst__sz, 0);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset(dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset(dst + ret, 0, dst__sz - ret);
return ret + 1;
}
/* Keep unsinged long in prototype so that kfunc is usable when emitted to
* vmlinux.h in BPF programs directly, but note that while in BPF prog, the
* unsigned long always points to 8-byte region on stack, the kernel may only
* read and write the 4-bytes on 32-bit.
*/
__bpf_kfunc void bpf_local_irq_save(unsigned long *flags__irq_flag)
{
local_irq_save(*flags__irq_flag);
}
__bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)
{
local_irq_restore(*flags__irq_flag);
}
__bpf_kfunc void __bpf_trap(void)
{
}
/*
* Kfuncs for string operations.
*
* Since strings are not necessarily %NUL-terminated, we cannot directly call
* in-kernel implementations. Instead, we open-code the implementations using
* __get_kernel_nofault instead of plain dereference to make them safe.
*/
static int __bpf_strncasecmp(const char *s1, const char *s2, bool ignore_case, size_t len)
{
char c1, c2;
int i;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < len && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c1, s1, char, err_out);
__get_kernel_nofault(&c2, s2, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 != c2)
return c1 < c2 ? -1 : 1;
if (c1 == '\0')
return 0;
s1++;
s2++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : 0;
err_out:
return -EFAULT;
}
/**
* bpf_strcmp - Compare two strings
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, false, XATTR_SIZE_MAX);
}
/**
* bpf_strcasecmp - Compare two strings, ignoring the case of the characters
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasecmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, XATTR_SIZE_MAX);
}
/*
* bpf_strncasecmp - Compare two length-limited strings, ignoring case
* @s1__ign: One string
* @s2__ign: Another string
* @len: The maximum number of characters to compare
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasecmp(const char *s1__ign, const char *s2__ign, size_t len)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, len);
}
/**
* bpf_strnchr - Find a character in a length limited string
* @s__ign: The string to be searched
* @count: The number of characters to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in the first @count characters of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnchr(const char *s__ign, size_t count, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
return i;
if (sc == '\0')
return -ENOENT;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : -ENOENT;
err_out:
return -EFAULT;
}
/**
* bpf_strchr - Find the first occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - The index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchr(const char *s__ign, char c)
{
return bpf_strnchr(s__ign, XATTR_SIZE_MAX, c);
}
/**
* bpf_strchrnul - Find and return a character in a string, or end of string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign or index of
* the null byte at the end of @s__ign when @c is not found
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchrnul(const char *s__ign, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == '\0' || sc == c)
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strrchr - Find the last occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the last occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strrchr(const char *s__ign, int c)
{
char sc;
int i, last = -ENOENT;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
last = i;
if (sc == '\0')
return last;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strnlen - Calculate the length of a length-limited string
* @s__ign: The string
* @count: The maximum number of characters to count
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnlen(const char *s__ign, size_t count)
{
char c;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c, s__ign, char, err_out);
if (c == '\0')
return i;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : i;
err_out:
return -EFAULT;
}
/**
* bpf_strlen - Calculate the length of a string
* @s__ign: The string
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strlen(const char *s__ign)
{
return bpf_strnlen(s__ign, XATTR_SIZE_MAX);
}
/**
* bpf_strspn - Calculate the length of the initial substring of @s__ign which
* only contains letters in @accept__ign
* @s__ign: The string to be searched
* @accept__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which only
* contains letters from @accept__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strspn(const char *s__ign, const char *accept__ign)
{
char cs, ca;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(accept__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&ca, accept__ign + j, char, err_out);
if (cs == ca || ca == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (ca == '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strcspn - Calculate the length of the initial substring of @s__ign which
* does not contain letters in @reject__ign
* @s__ign: The string to be searched
* @reject__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which does not
* contain letters from @reject__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcspn(const char *s__ign, const char *reject__ign)
{
char cs, cr;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(reject__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&cr, reject__ign + j, char, err_out);
if (cs == cr || cr == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (cr != '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
static int __bpf_strnstr(const char *s1, const char *s2, size_t len,
bool ignore_case)
{
char c1, c2;
int i, j;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
for (j = 0; i + j <= len && j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&c2, s2 + j, char, err_out);
if (c2 == '\0')
return i;
/*
* We allow reading an extra byte from s2 (note the
* `i + j <= len` above) to cover the case when s2 is
* a suffix of the first len chars of s1.
*/
if (i + j == len)
break;
__get_kernel_nofault(&c1, s1 + j, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 == '\0')
return -ENOENT;
if (c1 != c2)
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (i + j == len)
return -ENOENT;
s1++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strstr - Find the first substring in a string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strstr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, false);
}
/**
* bpf_strcasestr - Find the first substring in a string, ignoring the case of
* the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasestr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, true);
}
/**
* bpf_strnstr - Find the first substring in a length-limited string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strnstr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, false);
}
/**
* bpf_strncasestr - Find the first substring in a length-limited string,
* ignoring the case of the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasestr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, true);
}
#ifdef CONFIG_KEYS
/**
* bpf_lookup_user_key - lookup a key by its serial
* @serial: key handle serial number
* @flags: lookup-specific flags
*
* Search a key with a given *serial* and the provided *flags*.
* If found, increment the reference count of the key by one, and
* return it in the bpf_key structure.
*
* The bpf_key structure must be passed to bpf_key_put() when done
* with it, so that the key reference count is decremented and the
* bpf_key structure is freed.
*
* Permission checks are deferred to the time the key is used by
* one of the available key-specific kfuncs.
*
* Set *flags* with KEY_LOOKUP_CREATE, to attempt creating a requested
* special keyring (e.g. session keyring), if it doesn't yet exist.
* Set *flags* with KEY_LOOKUP_PARTIAL, to lookup a key without waiting
* for the key construction, and to retrieve uninstantiated keys (keys
* without data attached to them).
*
* Return: a bpf_key pointer with a valid key pointer if the key is found, a
* NULL pointer otherwise.
*/
__bpf_kfunc struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags)
{
key_ref_t key_ref;
struct bpf_key *bkey;
if (flags & ~KEY_LOOKUP_ALL)
return NULL;
/*
* Permission check is deferred until the key is used, as the
* intent of the caller is unknown here.
*/
key_ref = lookup_user_key(serial, flags, KEY_DEFER_PERM_CHECK);
if (IS_ERR(key_ref))
return NULL;
bkey = kmalloc_obj(*bkey);
if (!bkey) {
key_put(key_ref_to_ptr(key_ref));
return NULL;
}
bkey->key = key_ref_to_ptr(key_ref);
bkey->has_ref = true;
return bkey;
}
/**
* bpf_lookup_system_key - lookup a key by a system-defined ID
* @id: key ID
*
* Obtain a bpf_key structure with a key pointer set to the passed key ID.
* The key pointer is marked as invalid, to prevent bpf_key_put() from
* attempting to decrement the key reference count on that pointer. The key
* pointer set in such way is currently understood only by
* verify_pkcs7_signature().
*
* Set *id* to one of the values defined in include/linux/verification.h:
* 0 for the primary keyring (immutable keyring of system keys);
* VERIFY_USE_SECONDARY_KEYRING for both the primary and secondary keyring
* (where keys can be added only if they are vouched for by existing keys
* in those keyrings); VERIFY_USE_PLATFORM_KEYRING for the platform
* keyring (primarily used by the integrity subsystem to verify a kexec'ed
* kerned image and, possibly, the initramfs signature).
*
* Return: a bpf_key pointer with an invalid key pointer set from the
* pre-determined ID on success, a NULL pointer otherwise
*/
__bpf_kfunc struct bpf_key *bpf_lookup_system_key(u64 id)
{
struct bpf_key *bkey;
if (system_keyring_id_check(id) < 0)
return NULL;
bkey = kmalloc_obj(*bkey, GFP_ATOMIC);
if (!bkey)
return NULL;
bkey->key = (struct key *)(unsigned long)id;
bkey->has_ref = false;
return bkey;
}
/**
* bpf_key_put - decrement key reference count if key is valid and free bpf_key
* @bkey: bpf_key structure
*
* Decrement the reference count of the key inside *bkey*, if the pointer
* is valid, and free *bkey*.
*/
__bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
{
if (bkey->has_ref)
key_put(bkey->key);
kfree(bkey);
}
/**
* bpf_verify_pkcs7_signature - verify a PKCS#7 signature
* @data_p: data to verify
* @sig_p: signature of the data
* @trusted_keyring: keyring with keys trusted for signature verification
*
* Verify the PKCS#7 signature *sig_ptr* against the supplied *data_ptr*
* with keys in a keyring referenced by *trusted_keyring*.
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const void *data, *sig;
u32 data_len, sig_len;
int ret;
if (trusted_keyring->has_ref) {
/*
* Do the permission check deferred in bpf_lookup_user_key().
* See bpf_lookup_user_key() for more details.
*
* A call to key_task_permission() here would be redundant, as
* it is already done by keyring_search() called by
* find_asymmetric_key().
*/
ret = key_validate(trusted_keyring->key);
if (ret < 0)
return ret;
}
data_len = __bpf_dynptr_size(data_ptr);
data = __bpf_dynptr_data(data_ptr, data_len);
sig_len = __bpf_dynptr_size(sig_ptr);
sig = __bpf_dynptr_data(sig_ptr, sig_len);
return verify_pkcs7_signature(data, data_len, sig, sig_len,
trusted_keyring->key,
VERIFYING_BPF_SIGNATURE, NULL,
NULL);
#else
return -EOPNOTSUPP;
#endif /* CONFIG_SYSTEM_DATA_VERIFICATION */
}
#endif /* CONFIG_KEYS */
typedef int (*bpf_task_work_callback_t)(struct bpf_map *map, void *key, void *value);
enum bpf_task_work_state {
/* bpf_task_work is ready to be used */
BPF_TW_STANDBY = 0,
/* irq work scheduling in progress */
BPF_TW_PENDING,
/* task work scheduling in progress */
BPF_TW_SCHEDULING,
/* task work is scheduled successfully */
BPF_TW_SCHEDULED,
/* callback is running */
BPF_TW_RUNNING,
/* associated BPF map value is deleted */
BPF_TW_FREED,
};
struct bpf_task_work_ctx {
enum bpf_task_work_state state;
refcount_t refcnt;
struct callback_head work;
struct irq_work irq_work;
/* bpf_prog that schedules task work */
struct bpf_prog *prog;
/* task for which callback is scheduled */
struct task_struct *task;
/* the map and map value associated with this context */
struct bpf_map *map;
void *map_val;
enum task_work_notify_mode mode;
bpf_task_work_callback_t callback_fn;
struct rcu_head rcu;
} __aligned(8);
/* Actual type for struct bpf_task_work */
struct bpf_task_work_kern {
struct bpf_task_work_ctx *ctx;
};
static void bpf_task_work_ctx_reset(struct bpf_task_work_ctx *ctx)
{
if (ctx->prog) {
bpf_prog_put(ctx->prog);
ctx->prog = NULL;
}
if (ctx->task) {
bpf_task_release(ctx->task);
ctx->task = NULL;
}
}
static bool bpf_task_work_ctx_tryget(struct bpf_task_work_ctx *ctx)
{
return refcount_inc_not_zero(&ctx->refcnt);
}
static void bpf_task_work_destroy(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_ctx_reset(ctx);
kfree_rcu(ctx, rcu);
}
static void bpf_task_work_ctx_put(struct bpf_task_work_ctx *ctx)
{
if (!refcount_dec_and_test(&ctx->refcnt))
return;
if (irqs_disabled()) {
ctx->irq_work = IRQ_WORK_INIT(bpf_task_work_destroy);
irq_work_queue(&ctx->irq_work);
} else {
bpf_task_work_destroy(&ctx->irq_work);
}
}
static void bpf_task_work_cancel(struct bpf_task_work_ctx *ctx)
{
/*
* Scheduled task_work callback holds ctx ref, so if we successfully
* cancelled, we put that ref on callback's behalf. If we couldn't
* cancel, callback will inevitably run or has already completed
* running, and it would have taken care of its ctx ref itself.
*/
if (task_work_cancel(ctx->task, &ctx->work))
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_callback(struct callback_head *cb)
{
struct bpf_task_work_ctx *ctx = container_of(cb, struct bpf_task_work_ctx, work);
enum bpf_task_work_state state;
u32 idx;
void *key;
/* Read lock is needed to protect ctx and map key/value access */
guard(rcu_tasks_trace)();
/*
* This callback may start running before bpf_task_work_irq() switched to
* SCHEDULED state, so handle both transition variants SCHEDULING|SCHEDULED -> RUNNING.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_RUNNING);
if (state == BPF_TW_SCHEDULED)
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULED, BPF_TW_RUNNING);
if (state == BPF_TW_FREED) {
bpf_task_work_ctx_put(ctx);
return;
}
key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx);
migrate_disable();
ctx->callback_fn(ctx->map, key, ctx->map_val);
migrate_enable();
bpf_task_work_ctx_reset(ctx);
(void)cmpxchg(&ctx->state, BPF_TW_RUNNING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_irq(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
enum bpf_task_work_state state;
int err;
guard(rcu)();
if (cmpxchg(&ctx->state, BPF_TW_PENDING, BPF_TW_SCHEDULING) != BPF_TW_PENDING) {
bpf_task_work_ctx_put(ctx);
return;
}
err = task_work_add(ctx->task, &ctx->work, ctx->mode);
if (err) {
bpf_task_work_ctx_reset(ctx);
/*
* try to switch back to STANDBY for another task_work reuse, but we might have
* gone to FREED already, which is fine as we already cleaned up after ourselves
*/
(void)cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
return;
}
/*
* It's technically possible for just scheduled task_work callback to
* complete running by now, going SCHEDULING -> RUNNING and then
* dropping its ctx refcount. Instead of capturing an extra ref just
* to protect below ctx->state access, we rely on rcu_read_lock
* above to prevent kfree_rcu from freeing ctx before we return.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_SCHEDULED);
if (state == BPF_TW_FREED)
bpf_task_work_cancel(ctx); /* clean up if we switched into FREED state */
}
static struct bpf_task_work_ctx *bpf_task_work_fetch_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_kern *twk = (void *)tw;
struct bpf_task_work_ctx *ctx, *old_ctx;
ctx = READ_ONCE(twk->ctx);
if (ctx)
return ctx;
ctx = bpf_map_kmalloc_nolock(map, sizeof(*ctx), 0, NUMA_NO_NODE);
if (!ctx)
return ERR_PTR(-ENOMEM);
memset(ctx, 0, sizeof(*ctx));
refcount_set(&ctx->refcnt, 1); /* map's own ref */
ctx->state = BPF_TW_STANDBY;
old_ctx = cmpxchg(&twk->ctx, NULL, ctx);
if (old_ctx) {
/*
* tw->ctx is set by concurrent BPF program, release allocated
* memory and try to reuse already set context.
*/
kfree_nolock(ctx);
return old_ctx;
}
return ctx; /* Success */
}
static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_ctx *ctx;
/*
* Sleepable BPF programs hold rcu_read_lock_trace but not
* regular rcu_read_lock. Since kfree_rcu waits for regular
* RCU GP, the ctx can be freed while we're between reading
* the pointer and incrementing the refcount. Take regular
* rcu_read_lock to prevent kfree_rcu from freeing the ctx
* before we can tryget it.
*/
scoped_guard(rcu) {
ctx = bpf_task_work_fetch_ctx(tw, map);
if (IS_ERR(ctx))
return ctx;
/* try to get ref for task_work callback to hold */
if (!bpf_task_work_ctx_tryget(ctx))
return ERR_PTR(-EBUSY);
}
if (cmpxchg(&ctx->state, BPF_TW_STANDBY, BPF_TW_PENDING) != BPF_TW_STANDBY) {
/* lost acquiring race or map_release_uref() stole it from us, put ref and bail */
bpf_task_work_ctx_put(ctx);
return ERR_PTR(-EBUSY);
}
/*
* If no process or bpffs is holding a reference to the map, no new callbacks should be
* scheduled. This does not address any race or correctness issue, but rather is a policy
* choice: dropping user references should stop everything.
*/
if (!atomic64_read(&map->usercnt)) {
/* drop ref we just got for task_work callback itself */
bpf_task_work_ctx_put(ctx);
/* transfer map's ref into cancel_and_free() */
bpf_task_work_cancel_and_free(tw);
return ERR_PTR(-EBUSY);
}
return ctx;
}
static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw,
struct bpf_map *map, bpf_task_work_callback_t callback_fn,
struct bpf_prog_aux *aux, enum task_work_notify_mode mode)
{
struct bpf_prog *prog;
struct bpf_task_work_ctx *ctx;
int err;
BTF_TYPE_EMIT(struct bpf_task_work);
prog = bpf_prog_inc_not_zero(aux->prog);
if (IS_ERR(prog))
return -EBADF;
task = bpf_task_acquire(task);
if (!task) {
err = -EBADF;
goto release_prog;
}
ctx = bpf_task_work_acquire_ctx(tw, map);
if (IS_ERR(ctx)) {
err = PTR_ERR(ctx);
goto release_all;
}
ctx->task = task;
ctx->callback_fn = callback_fn;
ctx->prog = prog;
ctx->mode = mode;
ctx->map = map;
ctx->map_val = (void *)tw - map->record->task_work_off;
init_task_work(&ctx->work, bpf_task_work_callback);
init_irq_work(&ctx->irq_work, bpf_task_work_irq);
irq_work_queue(&ctx->irq_work);
return 0;
release_all:
bpf_task_release(task);
release_prog:
bpf_prog_put(prog);
return err;
}
/**
* bpf_task_work_schedule_signal - Schedule BPF callback using task_work_add with TWA_SIGNAL
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL);
}
/**
* bpf_task_work_schedule_resume - Schedule BPF callback using task_work_add with TWA_RESUME
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME);
}
static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
struct bpf_dynptr_kern *ptr)
{
struct bpf_dynptr_file_impl *state;
/* flags is currently unsupported */
if (flags) {
bpf_dynptr_set_null(ptr);
return -EINVAL;
}
state = kmalloc_nolock(sizeof(*state), 0, NUMA_NO_NODE);
if (!state) {
bpf_dynptr_set_null(ptr);
return -ENOMEM;
}
state->offset = 0;
state->size = U64_MAX; /* Don't restrict size, as file may change anyways */
freader_init_from_file(&state->freader, NULL, 0, file, may_sleep);
bpf_dynptr_init(ptr, state, BPF_DYNPTR_TYPE_FILE, 0, 0);
bpf_dynptr_set_rdonly(ptr);
return 0;
}
__bpf_kfunc int bpf_dynptr_from_file(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, false, (struct bpf_dynptr_kern *)ptr__uninit);
}
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, true, (struct bpf_dynptr_kern *)ptr__uninit);
}
__bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)dynptr;
struct bpf_dynptr_file_impl *df = ptr->data;
if (!df)
return 0;
freader_cleanup(&df->freader);
kfree_nolock(df);
bpf_dynptr_set_null(ptr);
return 0;
}
/**
* bpf_timer_cancel_async - try to deactivate a timer
* @timer: bpf_timer to stop
*
* Returns:
*
* * 0 when the timer was not active
* * 1 when the timer was active
* * -1 when the timer is currently executing the callback function and
* cannot be stopped
* * -ECANCELED when the timer will be cancelled asynchronously
* * -ENOMEM when out of memory
* * -EINVAL when the timer was not initialized
* * -ENOENT when this kfunc is racing with timer deletion
*/
__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer)
{
struct bpf_async_kern *async = (void *)timer;
struct bpf_async_cb *cb;
int ret;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
/*
* Unlike hrtimer_start() it's ok to synchronously call
* hrtimer_try_to_cancel() when refcnt reached zero, but deferring to
* irq_work is not, since irq callback may execute after RCU GP and
* cb could be freed at that time. Check for refcnt zero for
* consistency.
*/
if (!refcount_inc_not_zero(&cb->refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
ret = hrtimer_try_to_cancel(&t->timer);
bpf_async_refcount_put(cb);
return ret;
} else {
ret = bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
return ret ? ret : -ECANCELED;
}
}
__bpf_kfunc_end_defs();
static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_cancel(ctx); /* this might put task_work callback's ref */
bpf_task_work_ctx_put(ctx); /* and here we put map's own ref that was transferred to us */
}
void bpf_task_work_cancel_and_free(void *val)
{
struct bpf_task_work_kern *twk = val;
struct bpf_task_work_ctx *ctx;
enum bpf_task_work_state state;
ctx = xchg(&twk->ctx, NULL);
if (!ctx)
return;
state = xchg(&ctx->state, BPF_TW_FREED);
if (state == BPF_TW_SCHEDULED) {
/* run in irq_work to avoid locks in NMI */
init_irq_work(&ctx->irq_work, bpf_task_work_cancel_scheduled);
irq_work_queue(&ctx->irq_work);
return;
}
bpf_task_work_ctx_put(ctx); /* put bpf map's ref */
}
BTF_KFUNCS_START(generic_btf_ids)
#ifdef CONFIG_CRASH_DUMP
BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
#endif
BTF_ID_FLAGS(func, bpf_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_percpu_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_refcount_acquire, KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE | KF_RET_NULL | KF_RCU)
BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_front_impl)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_back_impl)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_rbtree_add_impl)
BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_cgroup_ancestor, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU)
BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
#endif
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_send_signal_task)
#endif
#ifdef CONFIG_KEYS
BTF_ID_FLAGS(func, bpf_lookup_user_key, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_lookup_system_key, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_key_put, KF_RELEASE)
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
BTF_ID_FLAGS(func, bpf_verify_pkcs7_signature, KF_SLEEPABLE)
#endif
#endif
#ifdef CONFIG_S390
BTF_ID_FLAGS(func, bpf_get_lowcore)
#endif
BTF_KFUNCS_END(generic_btf_ids)
static const struct btf_kfunc_id_set generic_kfunc_set = {
.owner = THIS_MODULE,
.set = &generic_btf_ids,
};
BTF_ID_LIST(generic_dtor_ids)
BTF_ID(struct, task_struct)
BTF_ID(func, bpf_task_release_dtor)
#ifdef CONFIG_CGROUPS
BTF_ID(struct, cgroup)
BTF_ID(func, bpf_cgroup_release_dtor)
#endif
BTF_KFUNCS_START(common_btf_ids)
BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rcu_read_lock)
BTF_ID_FLAGS(func, bpf_rcu_read_unlock)
BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_css_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_css_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_destroy, KF_ITER_DESTROY)
#endif
BTF_ID_FLAGS(func, bpf_iter_task_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_dynptr_adjust)
BTF_ID_FLAGS(func, bpf_dynptr_is_null)
BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
BTF_ID_FLAGS(func, bpf_dynptr_size)
BTF_ID_FLAGS(func, bpf_dynptr_clone)
BTF_ID_FLAGS(func, bpf_dynptr_copy)
BTF_ID_FLAGS(func, bpf_dynptr_memset)
#ifdef CONFIG_NET
BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
#endif
BTF_ID_FLAGS(func, bpf_wq_init)
BTF_ID_FLAGS(func, bpf_wq_set_callback, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_wq_start)
BTF_ID_FLAGS(func, bpf_preempt_disable)
BTF_ID_FLAGS(func, bpf_preempt_enable)
BTF_ID_FLAGS(func, bpf_iter_bits_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_bits_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_bits_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_copy_from_user_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_get_kmem_cache)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_local_irq_save)
BTF_ID_FLAGS(func, bpf_local_irq_restore)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr)
BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE)
#endif
#ifdef CONFIG_DMA_SHARED_BUFFER
BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
#endif
BTF_ID_FLAGS(func, __bpf_trap)
BTF_ID_FLAGS(func, bpf_strcmp);
BTF_ID_FLAGS(func, bpf_strcasecmp);
BTF_ID_FLAGS(func, bpf_strncasecmp);
BTF_ID_FLAGS(func, bpf_strchr);
BTF_ID_FLAGS(func, bpf_strchrnul);
BTF_ID_FLAGS(func, bpf_strnchr);
BTF_ID_FLAGS(func, bpf_strrchr);
BTF_ID_FLAGS(func, bpf_strlen);
BTF_ID_FLAGS(func, bpf_strnlen);
BTF_ID_FLAGS(func, bpf_strspn);
BTF_ID_FLAGS(func, bpf_strcspn);
BTF_ID_FLAGS(func, bpf_strstr);
BTF_ID_FLAGS(func, bpf_strcasestr);
BTF_ID_FLAGS(func, bpf_strnstr);
BTF_ID_FLAGS(func, bpf_strncasestr);
#if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS)
BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU)
#endif
BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_dynptr_from_file)
BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
BTF_ID_FLAGS(func, bpf_timer_cancel_async)
BTF_KFUNCS_END(common_btf_ids)
static const struct btf_kfunc_id_set common_kfunc_set = {
.owner = THIS_MODULE,
.set = &common_btf_ids,
};
static int __init kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc generic_dtors[] = {
{
.btf_id = generic_dtor_ids[0],
.kfunc_btf_id = generic_dtor_ids[1]
},
#ifdef CONFIG_CGROUPS
{
.btf_id = generic_dtor_ids[2],
.kfunc_btf_id = generic_dtor_ids[3]
},
#endif
};
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &generic_kfunc_set);
ret = ret ?: register_btf_id_dtor_kfuncs(generic_dtors,
ARRAY_SIZE(generic_dtors),
THIS_MODULE);
return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
}
late_initcall(kfunc_init);
/* Get a pointer to dynptr data up to len bytes for read only access. If
* the dynptr doesn't have continuous data up to len bytes, return NULL.
*/
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u64 len)
{
const struct bpf_dynptr *p = (struct bpf_dynptr *)ptr;
return bpf_dynptr_slice(p, 0, NULL, len);
}
/* Get a pointer to dynptr data up to len bytes for read write access. If
* the dynptr doesn't have continuous data up to len bytes, or the dynptr
* is read only, return NULL.
*/
void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u64 len)
{
if (__bpf_dynptr_is_rdonly(ptr))
return NULL;
return (void *)__bpf_dynptr_data(ptr, len);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *val)
{
if (btf_record_has_field(map->record, BPF_TIMER))
bpf_obj_free_timer(map->record, val);
if (btf_record_has_field(map->record, BPF_WORKQUEUE))
bpf_obj_free_workqueue(map->record, val);
if (btf_record_has_field(map->record, BPF_TASK_WORK))
bpf_obj_free_task_work(map->record, val);
}
]
|
| 80/2 |
2026/05/18 18:43 |
llm |
debugger |
0m
Model:
Tokens:
input: 99076
output: 36
thoughts: 251
Thoughts: **Analyzing the BPF Tracepoint Formatting Logic**
Okay, so I'm looking at how `bpf_snprintf` works, and I see it delegates the core formatting to `bpf_bprintf_prepare`. That's standard, good abstraction. Let me look at that function. Aha, inside `bpf_bprintf_prepare`, I'm seeing a section dedicated to handling the format specifier `%pB`. This is interesting; I need to understand what this is. From the code, it looks like it does a `snprintf` to a temporary buffer, in the case that `tmp_buf` exists. Okay, so now I am thinking that `%pB` is going to be for `print_symbol`.
Now, the format specifier `%pB` is unfamiliar. I recall that I am going to have to look it up in `lib/vsprintf.c` to see what is going on. This is likely related to displaying something symbolic, perhaps a backtrace or some kind of symbolic representation, given the "B" and the "p" (pointer).
|
| 81/2 |
2026/05/18 18:43 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/vsprintf.c IncludeLines:true Name:pointer]
Results: map[SourceCode:2388: /*
2389: * Show a '%p' thing. A kernel extension is that the '%p' is followed
2390: * by an extra set of alphanumeric characters that are extended format
2391: * specifiers.
2392: *
2393: * Please update scripts/checkpatch.pl when adding/removing conversion
2394: * characters. (Search for "check for vsprintf extension").
2395: *
2396: * Right now we handle:
2397: *
2398: * - 'S' For symbolic direct pointers (or function descriptors) with offset
2399: * - 's' For symbolic direct pointers (or function descriptors) without offset
2400: * - '[Ss]R' as above with __builtin_extract_return_addr() translation
2401: * - 'S[R]b' as above with module build ID (for use in backtraces)
2402: * - '[Ff]' %pf and %pF were obsoleted and later removed in favor of
2403: * %ps and %pS. Be careful when re-using these specifiers.
2404: * - 'B' For backtraced symbolic direct pointers with offset
2405: * - 'Bb' as above with module build ID (for use in backtraces)
2406: * - 'R' For decoded struct resource, e.g., [mem 0x0-0x1f 64bit pref]
2407: * - 'r' For raw struct resource, e.g., [mem 0x0-0x1f flags 0x201]
2408: * - 'ra' For struct ranges, e.g., [range 0x0000000000000000 - 0x00000000000000ff]
2409: * - 'b[l]' For a bitmap, the number of bits is determined by the field
2410: * width which must be explicitly specified either as part of the
2411: * format string '%32b[l]' or through '%*b[l]', [l] selects
2412: * range-list format instead of hex format
2413: * - 'M' For a 6-byte MAC address, it prints the address in the
2414: * usual colon-separated hex notation
2415: * - 'm' For a 6-byte MAC address, it prints the hex address without colons
2416: * - 'MF' For a 6-byte MAC FDDI address, it prints the address
2417: * with a dash-separated hex notation
2418: * - '[mM]R' For a 6-byte MAC address, Reverse order (Bluetooth)
2419: * - 'I' [46] for IPv4/IPv6 addresses printed in the usual way
2420: * IPv4 uses dot-separated decimal without leading 0's (1.2.3.4)
2421: * IPv6 uses colon separated network-order 16 bit hex with leading 0's
2422: * [S][pfs]
2423: * Generic IPv4/IPv6 address (struct sockaddr *) that falls back to
2424: * [4] or [6] and is able to print port [p], flowinfo [f], scope [s]
2425: * - 'i' [46] for 'raw' IPv4/IPv6 addresses
2426: * IPv6 omits the colons (01020304...0f)
2427: * IPv4 uses dot-separated decimal with leading 0's (010.123.045.006)
2428: * [S][pfs]
2429: * Generic IPv4/IPv6 address (struct sockaddr *) that falls back to
2430: * [4] or [6] and is able to print port [p], flowinfo [f], scope [s]
2431: * - '[Ii][4S][hnbl]' IPv4 addresses in host, network, big or little endian order
2432: * - 'I[6S]c' for IPv6 addresses printed as specified by
2433: * https://tools.ietf.org/html/rfc5952
2434: * - 'E[achnops]' For an escaped buffer, where rules are defined by combination
2435: * of the following flags (see string_escape_mem() for the
2436: * details):
2437: * a - ESCAPE_ANY
2438: * c - ESCAPE_SPECIAL
2439: * h - ESCAPE_HEX
2440: * n - ESCAPE_NULL
2441: * o - ESCAPE_OCTAL
2442: * p - ESCAPE_NP
2443: * s - ESCAPE_SPACE
2444: * By default ESCAPE_ANY_NP is used.
2445: * - 'U' For a 16 byte UUID/GUID, it prints the UUID/GUID in the form
2446: * "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
2447: * Options for %pU are:
2448: * b big endian lower case hex (default)
2449: * B big endian UPPER case hex
2450: * l little endian lower case hex
2451: * L little endian UPPER case hex
2452: * big endian output byte order is:
2453: * [0][1][2][3]-[4][5]-[6][7]-[8][9]-[10][11][12][13][14][15]
2454: * little endian output byte order is:
2455: * [3][2][1][0]-[5][4]-[7][6]-[8][9]-[10][11][12][13][14][15]
2456: * - 'V' For a struct va_format which contains a format string * and va_list *,
2457: * call vsnprintf(->format, *->va_list).
2458: * Implements a "recursive vsnprintf".
2459: * Do not use this feature without some mechanism to verify the
2460: * correctness of the format string and va_list arguments.
2461: * - 'K' For a kernel pointer that should be hidden from unprivileged users.
2462: * Use only for procfs, sysfs and similar files, not printk(); please
2463: * read the documentation (path below) first.
2464: * - 'NF' For a netdev_features_t
2465: * - '4cc' V4L2 or DRM FourCC code, with endianness and raw numerical value.
2466: * - '4c[h[R]lb]' For generic FourCC code with raw numerical value. Both are
2467: * displayed in the big-endian format. This is the opposite of V4L2 or
2468: * DRM FourCCs.
2469: * The additional specifiers define what endianness is used to load
2470: * the stored bytes. The data might be interpreted using the host,
2471: * reversed host byte order, little-endian, or big-endian.
2472: * - 'h[CDN]' For a variable-length buffer, it prints it as a hex string with
2473: * a certain separator (' ' by default):
2474: * C colon
2475: * D dash
2476: * N no separator
2477: * The maximum supported length is 64 bytes of the input. Consider
2478: * to use print_hex_dump() for the larger input.
2479: * - 'a[pd]' For address types [p] phys_addr_t, [d] dma_addr_t and derivatives
2480: * (default assumed to be phys_addr_t, passed by reference)
2481: * - 'd[234]' For a dentry name (optionally 2-4 last components)
2482: * - 'D[234]' Same as 'd' but for a struct file
2483: * - 'g' For block_device name (gendisk + partition number)
2484: * - 't[RST][dt][r][s]' For time and date as represented by:
2485: * R struct rtc_time
2486: * S struct timespec64
2487: * T time64_t
2488: * - 'tSp' For time represented by struct timespec64 printed as <seconds>.<nanoseconds>
2489: * - 'C' For a clock, it prints the name (Common Clock Framework) or address
2490: * (legacy clock framework) of the clock
2491: * - 'G' For flags to be printed as a collection of symbolic strings that would
2492: * construct the specific value. Supported flags given by option:
2493: * p page flags (see struct page) given as pointer to unsigned long
2494: * g gfp flags (GFP_* and __GFP_*) given as pointer to gfp_t
2495: * v vma flags (VM_*) given as pointer to unsigned long
2496: * - 'OF[fnpPcCF]' For a device tree object
2497: * Without any optional arguments prints the full_name
2498: * f device node full_name
2499: * n device node name
2500: * p device node phandle
2501: * P device node path spec (name + @unit)
2502: * F device node flags
2503: * c major compatible string
2504: * C full compatible string
2505: * - 'fw[fP]' For a firmware node (struct fwnode_handle) pointer
2506: * Without an option prints the full name of the node
2507: * f full name
2508: * P node name, including a possible unit address
2509: * - 'x' For printing the address unmodified. Equivalent to "%lx".
2510: * Please read the documentation (path below) before using!
2511: * - '[ku]s' For a BPF/tracing related format specifier, e.g. used out of
2512: * bpf_trace_printk() where [ku] prefix specifies either kernel (k)
2513: * or user (u) memory to probe, and:
2514: * s a string, equivalent to "%s" on direct vsnprintf() use
2515: *
2516: * ** When making changes please also update:
2517: * Documentation/core-api/printk-formats.rst
2518: *
2519: * Note: The default behaviour (unadorned %p) is to hash the address,
2520: * rendering it useful as a unique identifier.
2521: *
2522: * There is also a '%pA' format specifier, but it is only intended to be used
2523: * from Rust code to format core::fmt::Arguments. Do *not* use it from C.
2524: * See rust/kernel/print.rs for details.
2525: */
2526: static noinline_for_stack
2527: char *pointer(const char *fmt, char *buf, char *end, void *ptr,
2528: struct printf_spec spec)
2529: {
2530: switch (*fmt) {
2531: case 'S':
2532: case 's':
2533: ptr = dereference_symbol_descriptor(ptr);
2534: fallthrough;
2535: case 'B':
2536: return symbol_string(buf, end, ptr, spec, fmt);
2537: case 'R':
2538: case 'r':
2539: return resource_or_range(fmt, buf, end, ptr, spec);
2540: case 'h':
2541: return hex_string(buf, end, ptr, spec, fmt);
2542: case 'b':
2543: switch (fmt[1]) {
2544: case 'l':
2545: return bitmap_list_string(buf, end, ptr, spec, fmt);
2546: default:
2547: return bitmap_string(buf, end, ptr, spec, fmt);
2548: }
2549: case 'M': /* Colon separated: 00:01:02:03:04:05 */
2550: case 'm': /* Contiguous: 000102030405 */
2551: /* [mM]F (FDDI) */
2552: /* [mM]R (Reverse order; Bluetooth) */
2553: return mac_address_string(buf, end, ptr, spec, fmt);
2554: case 'I': /* Formatted IP supported
2555: * 4: 1.2.3.4
2556: * 6: 0001:0203:...:0708
2557: * 6c: 1::708 or 1::1.2.3.4
2558: */
2559: case 'i': /* Contiguous:
2560: * 4: 001.002.003.004
2561: * 6: 000102...0f
2562: */
2563: return ip_addr_string(buf, end, ptr, spec, fmt);
2564: case 'E':
2565: return escaped_string(buf, end, ptr, spec, fmt);
2566: case 'U':
2567: return uuid_string(buf, end, ptr, spec, fmt);
2568: case 'V':
2569: return va_format(buf, end, ptr, spec);
2570: case 'K':
2571: return restricted_pointer(buf, end, ptr, spec);
2572: case 'N':
2573: return netdev_bits(buf, end, ptr, spec, fmt);
2574: case '4':
2575: return fourcc_string(buf, end, ptr, spec, fmt);
2576: case 'a':
2577: return address_val(buf, end, ptr, spec, fmt);
2578: case 'd':
2579: return dentry_name(buf, end, ptr, spec, fmt);
2580: case 't':
2581: return time_and_date(buf, end, ptr, spec, fmt);
2582: case 'C':
2583: return clock(buf, end, ptr, spec, fmt);
2584: case 'D':
2585: return file_dentry_name(buf, end, ptr, spec, fmt);
2586: #ifdef CONFIG_BLOCK
2587: case 'g':
2588: return bdev_name(buf, end, ptr, spec, fmt);
2589: #endif
2590:
2591: case 'G':
2592: return flags_string(buf, end, ptr, spec, fmt);
2593: case 'O':
2594: return device_node_string(buf, end, ptr, spec, fmt + 1);
2595: case 'f':
2596: return fwnode_string(buf, end, ptr, spec, fmt + 1);
2597: case 'A':
2598: if (!IS_ENABLED(CONFIG_RUST)) {
2599: WARN_ONCE(1, "Please remove %%pA from non-Rust code\n");
2600: return error_string(buf, end, "(%pA?)", spec);
2601: }
2602: return rust_fmt_argument(buf, end, ptr);
2603: case 'x':
2604: return pointer_string(buf, end, ptr, spec);
2605: case 'e':
2606: /* %pe with a non-ERR_PTR gets treated as plain %p */
2607: if (!IS_ERR(ptr))
2608: return default_pointer(buf, end, ptr, spec);
2609: return err_ptr(buf, end, ptr, spec);
2610: case 'u':
2611: case 'k':
2612: switch (fmt[1]) {
2613: case 's':
2614: return string(buf, end, ptr, spec);
2615: default:
2616: return error_string(buf, end, "(einval)", spec);
2617: }
2618: default:
2619: return default_pointer(buf, end, ptr, spec);
2620: }
2621: }
SourceFile:lib/vsprintf.c]
|
| 82/2 |
2026/05/18 18:43 |
llm |
debugger |
0m
Model:
Tokens:
input: 104235
output: 38
thoughts: 272
Thoughts: **Decoding the `%pB` Format Specifier**
Okay, so `%pB`... that's for backtraced symbolic direct pointers with an offset. It's calling `symbol_string(buf, end, ptr, spec, fmt);` - so, it's clearly related to symbolic representation of memory addresses. Let me think... what exactly is the flow here?
Right, `bpf_bprintf_prepare` does some work on the format string. Ah, here it is:
```c
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
```
So, if it encounters `%pB`, and `tmp_buf` isn't NULL, it's using `snprintf` with `%pB`. Interesting. It's using `raw_args[num_spec]` as the address. And, if `tmp_buf` is filled with `%pB`, it is calling `snprintf` with `%pB` using `raw_args[num_spec]`. I was almost thinking that `snprintf` with `%pB` would print the backtrace of the *current* task - but no. It's printing the symbol name of the address `raw_args[num_spec]`.
Now, does `symbol_string` use `sprint_backtrace` or `sprint_symbol`? That's the key. I should look into `symbol_string` function located in `lib/vsprintf.c` and see what it does.
|
| 83/2 |
2026/05/18 18:43 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/vsprintf.c IncludeLines:true Name:symbol_string]
Results: map[SourceCode: 990: static noinline_for_stack
991: char *symbol_string(char *buf, char *end, void *ptr,
992: struct printf_spec spec, const char *fmt)
993: {
994: unsigned long value;
995: #ifdef CONFIG_KALLSYMS
996: char sym[KSYM_SYMBOL_LEN];
997: #endif
998:
999: if (fmt[1] == 'R')
1000: ptr = __builtin_extract_return_addr(ptr);
1001: value = (unsigned long)ptr;
1002:
1003: #ifdef CONFIG_KALLSYMS
1004: if (*fmt == 'B' && fmt[1] == 'b')
1005: sprint_backtrace_build_id(sym, value);
1006: else if (*fmt == 'B')
1007: sprint_backtrace(sym, value);
1008: else if (*fmt == 'S' && (fmt[1] == 'b' || (fmt[1] == 'R' && fmt[2] == 'b')))
1009: sprint_symbol_build_id(sym, value);
1010: else if (*fmt != 's')
1011: sprint_symbol(sym, value);
1012: else
1013: sprint_symbol_no_offset(sym, value);
1014:
1015: return string_nocheck(buf, end, sym, spec);
1016: #else
1017: return special_hex_number(buf, end, value, sizeof(void *));
1018: #endif
1019: }
SourceFile:lib/vsprintf.c]
|
| 84/2 |
2026/05/18 18:43 |
llm |
debugger |
0m
Model:
Tokens:
input: 105467
output: 46
thoughts: 1918
Thoughts: **Analyzing a Potential Kernel Bug**
Okay, here's what I'm thinking as I dissect this reproducer. First, I noticed `sprint_backtrace` in the call stack and immediately considered whether it's taking a lock that might be causing a deadlock, especially if it's being called within a BPF context. The reproducer seems to be using `bpf_prog_run` or something similar to execute the BPF program. It attaches the program to a tracepoint or kprobe. The system call being used is `sendmsg` with a `RTM_NEWQDISC` message. That seems like an odd trigger.
I saw the BPF program is loaded but isn't explicitly attached in the reproducer, and that it has an `ifindex` field set even though the type is `BPF_PROG_TYPE_TRACEPOINT`. That's odd because tracepoints don't typically use `ifindex`. The reproducer doesn't seem to be using `BPF_PROG_ATTACH` and its type is explicitly set to 5, which is a tracepoint. I checked if `bpf_prog_load` can auto-attach the program, but that doesn't seem to be the case unless it's a `BPF_PROG_TYPE_TRACING` or `BPF_PROG_TYPE_EXT` program, or perhaps I'm missing something about the program loading mechanism. Tracepoints need `perf_event_open` and `ioctl(PERF_EVENT_IOC_SET_BPF)`, which are not present here.
Next, I looked at the BPF program itself, specifically the instructions. I was looking for any indication of how the program is executed. It has several `bpf_helpers_ringbuf_*` instructions and `bpf_snprintf`. It's definitely not a straightforward BPF program. I saw that the `BPF_PROG_TYPE_TRACEPOINT` isn't supposed to have an `ifindex` attached to it, but the type is `BPF_PROG_TYPE_TRACEPOINT`.
The reproducer calls `sendmsg`, which sends a `RTM_NEWQDISC` message. I'm thinking the `qdisc_create` might trigger a tracepoint. The question is, if the BPF program isn't explicitly attached, will it be executed by the tracepoint? Probably not. The type is 5, a tracepoint. So how is it being executed? Perhaps `bpf_prog_load` is somehow auto-attaching it, but that seems unlikely, and the reproducer doesn't use `perf_event_open`.
I began to wonder if the BPF program is even executed, or if the crash is caused just by loading it, by `bpf_prog_load`. It calls `bpf_check`, which simulates the execution, and that might be where the issue lies. `bpf_check` does call `bpf_snprintf`, but not directly. The crash is a soft lockup on CPU 0 in `csd_lock_wait`, and CPU 1 is in `__remove_hrtimer`. This is starting to look like a timer-related issue, and it's looking like a corrupted rbtree. I thought about how the rbtree of timers could be corrupted and if a BPF program can cause that corruption.
I checked the BPF program's helpers again, specifically the `bpf_timer_*` helpers, but the program is not directly using them. I looked at the BPF instructions. It uses `bpf_ringbuf_*` helpers, which seems odd because the program's type is a tracepoint.
Let's check `include/uapi/linux/bpf.h` for the helper IDs.
|
| 85/2 |
2026/05/18 18:44 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/uapi/linux/bpf.h IncludeLines:true Name:bpf_func_id]
Results: map[SourceCode:6132: enum bpf_func_id {
6133: ___BPF_FUNC_MAPPER(__BPF_ENUM_FN)
6134: __BPF_FUNC_MAX_ID,
6135: };
SourceFile:include/uapi/linux/bpf.h]
|
| 86/2 |
2026/05/18 18:44 |
llm |
debugger |
0m
Model:
Tokens:
input: 108342
output: 22
thoughts: 29
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Analyzing the BPF Function Mapper**
Right, let's get down to it. I need to understand how the BPF function mapper is handled, so I'm going to kick things off by grepping for `___BPF_FUNC_MAPPER` within `include/uapi/linux/bpf.h`. That macro's almost certainly the key to understanding the underlying implementation and how the available BPF helpers are defined and exposed to user-space programs. By starting with that macro, I should be able to trace through the definitions, understand how the helper function numbers are assigned, and ideally, identify any potential areas of concern or changes that I need to be aware of. I'm expecting this to lead me to other relevant macro definitions and structures – likely some complex bit-field manipulations and type definitions – that dictate the available BPF helper functions, their associated arguments, and their return types. Understanding this mapper is crucial for anything related to BPF introspection, security analysis, or custom BPF program development, so this is a good place to begin. Time to get to it.
|
| 87/2 |
2026/05/18 18:44 |
tool |
grepper |
0m
Args: map[Expression:___BPF_FUNC_MAPPER]
Results: map[Output:include/uapi/linux/bpf.h=1510=union bpf_attr {
--
include/uapi/linux/bpf.h-5903- */
include/uapi/linux/bpf.h:5904:#define ___BPF_FUNC_MAPPER(FN, ctx...) \
include/uapi/linux/bpf.h-5905- FN(unspec, 0, ##ctx) \
--
include/uapi/linux/bpf.h-6125-#define __BPF_FUNC_MAPPER_APPLY(name, value, FN) FN(name),
include/uapi/linux/bpf.h:6126:#define __BPF_FUNC_MAPPER(FN) ___BPF_FUNC_MAPPER(__BPF_FUNC_MAPPER_APPLY, FN)
include/uapi/linux/bpf.h-6127-
--
include/uapi/linux/bpf.h=6132=enum bpf_func_id {
include/uapi/linux/bpf.h:6133: ___BPF_FUNC_MAPPER(__BPF_ENUM_FN)
include/uapi/linux/bpf.h-6134- __BPF_FUNC_MAX_ID,
--
scripts/bpf_doc.py=104=class HeaderParser(object):
--
scripts/bpf_doc.py-299- def parse_define_helpers(self):
scripts/bpf_doc.py:300: # Parse FN(...) in #define ___BPF_FUNC_MAPPER to compare later with the
scripts/bpf_doc.py-301- # number of unique function names present in description and use the
--
scripts/bpf_doc.py-305- # self.define_unique_helpers.
scripts/bpf_doc.py:306: self.seek_to('#define ___BPF_FUNC_MAPPER(FN, ctx...)',
scripts/bpf_doc.py-307- 'Could not find start of eBPF helper definition list')
--
scripts/bpf_doc.py=478=class PrinterHelpersRST(PrinterRST):
--
scripts/bpf_doc.py-486- self.elements = parser.helpers
scripts/bpf_doc.py:487: self.elem_number_check(parser.desc_unique_helpers, parser.define_unique_helpers, 'helper', '___BPF_FUNC_MAPPER')
scripts/bpf_doc.py-488-
--
scripts/bpf_doc.py=692=class PrinterHelpersHeader(Printer):
--
scripts/bpf_doc.py-699- self.elements = parser.helpers
scripts/bpf_doc.py:700: self.elem_number_check(parser.desc_unique_helpers, parser.define_unique_helpers, 'helper', '___BPF_FUNC_MAPPER')
scripts/bpf_doc.py-701-
--
scripts/bpf_doc.py=915=class PrinterHelpersJSON(Printer):
--
scripts/bpf_doc.py-926- "helper",
scripts/bpf_doc.py:927: "___BPF_FUNC_MAPPER",
scripts/bpf_doc.py-928- )
--
tools/include/uapi/linux/bpf.h=1510=union bpf_attr {
--
tools/include/uapi/linux/bpf.h-5903- */
tools/include/uapi/linux/bpf.h:5904:#define ___BPF_FUNC_MAPPER(FN, ctx...) \
tools/include/uapi/linux/bpf.h-5905- FN(unspec, 0, ##ctx) \
--
tools/include/uapi/linux/bpf.h-6125-#define __BPF_FUNC_MAPPER_APPLY(name, value, FN) FN(name),
tools/include/uapi/linux/bpf.h:6126:#define __BPF_FUNC_MAPPER(FN) ___BPF_FUNC_MAPPER(__BPF_FUNC_MAPPER_APPLY, FN)
tools/include/uapi/linux/bpf.h-6127-
--
tools/include/uapi/linux/bpf.h=6132=enum bpf_func_id {
tools/include/uapi/linux/bpf.h:6133: ___BPF_FUNC_MAPPER(__BPF_ENUM_FN)
tools/include/uapi/linux/bpf.h-6134- __BPF_FUNC_MAX_ID,
]
|
| 88/2 |
2026/05/18 18:44 |
llm |
debugger |
0m
Model:
Tokens:
input: 109817
output: 20
thoughts: 30
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Examining BPF Helper IDs in `bpf.h`**
Right, let's dive into the core. I need to refresh my memory on the helper function IDs available for BPF. Specifically, I'm thinking I'll head straight to `include/uapi/linux/bpf.h` and take a look around line 5904. That's usually where the macro definitions for the helper IDs start popping up. I know there have been updates recently, and I want to make sure I'm using the latest available. I need to quickly cross-reference a few of the newer ones to see if there's any differences, especially concerning the experimental ones, and what the latest additions are. This is going to be quick; I'm expecting some specific helpers, but I know I'll likely find a few gems I've overlooked. No need for a lengthy scan; I'm after something very specific to inform a little code I'm about to write.
|
| 89/2 |
2026/05/18 18:44 |
tool |
grepper |
0m
Args: map[Expression:FN\(ringbuf_reserve]
Results: map[Output:include/uapi/linux/bpf.h=1510=union bpf_attr {
--
include/uapi/linux/bpf.h-6035- FN(ringbuf_output, 130, ##ctx) \
include/uapi/linux/bpf.h:6036: FN(ringbuf_reserve, 131, ##ctx) \
include/uapi/linux/bpf.h-6037- FN(ringbuf_submit, 132, ##ctx) \
--
include/uapi/linux/bpf.h-6102- FN(dynptr_from_mem, 197, ##ctx) \
include/uapi/linux/bpf.h:6103: FN(ringbuf_reserve_dynptr, 198, ##ctx) \
include/uapi/linux/bpf.h-6104- FN(ringbuf_submit_dynptr, 199, ##ctx) \
--
tools/include/uapi/linux/bpf.h=1510=union bpf_attr {
--
tools/include/uapi/linux/bpf.h-6035- FN(ringbuf_output, 130, ##ctx) \
tools/include/uapi/linux/bpf.h:6036: FN(ringbuf_reserve, 131, ##ctx) \
tools/include/uapi/linux/bpf.h-6037- FN(ringbuf_submit, 132, ##ctx) \
--
tools/include/uapi/linux/bpf.h-6102- FN(dynptr_from_mem, 197, ##ctx) \
tools/include/uapi/linux/bpf.h:6103: FN(ringbuf_reserve_dynptr, 198, ##ctx) \
tools/include/uapi/linux/bpf.h-6104- FN(ringbuf_submit_dynptr, 199, ##ctx) \
]
|
| 90/2 |
2026/05/18 18:44 |
llm |
debugger |
0m
Model:
Tokens:
input: 110627
output: 44
thoughts: 2594
Thoughts: **Analyzing the BPF Snprintf Reproducer**
Okay, so I'm looking at this reproducer and tracing its logic. I see the BPF helper function `bpf_snprintf` being called (0xa5, as indicated by the constants). The reproducer sets up arguments: a string buffer at `r10 - 8`, a size of 8 (R2), a format string (`bpf_frozen_const_str` which is register R3), an argument array pointer at `r10 - 16` which contains the value of `9` (R4), and a length of 8 (R5).
The crucial part is understanding what happens when `bpf_snprintf` uses this `fmt` string and the argument. I can tell that the format string `fmt` (R3) must be a map value since it is of type `bpf_frozen_const_str`. The reproducer uses `%pB`. This means the code inside `bpf_snprintf` calls `snprintf` with `%pB` and 9. Now, `%pB` is problematic. I'm pretty sure it calls the `sprint_backtrace` function.
`sprint_backtrace` itself then calls `__sprint_symbol`, which in turn calls `kallsyms_lookup` with this address 9. But does `kallsyms_lookup` take a lock? No, I believe it uses RCU or a similar mechanism. The question is, can this be exploited? If we can control the `fmt` string to `%pB` and then provide a valid address in `raw_args[num_spec]`, and what context the program executes in. What can we do with that?
I need to confirm the conditions for triggering `bpf_snprintf` with `%pB`. It seems `bpf_bprintf_prepare` does the actual `snprintf` if the `fmt` string uses `%pB`. This suggests that the format string is prepared within the `bpf_snprintf` function. So it looks like `bpf_bprintf_prepare` is safe to call from BPF.
I need to figure out *how* this BPF program gets executed. The reproducer calls `sendmsg` which sends `RTM_NEWQDISC` to netlink, triggering `tc_modify_qdisc`. Does `tc_modify_qdisc` trigger BPF execution? The reproducer loads a BPF program but doesn't attach it! This is very suspicious. `bpf_prog_load` is called using the BPF syscall (command 5). I see a `BPF_PROG_RUN` command which I need to investigate. So, how does this BPF program run? I need to analyze this further.
|
| 91/2 |
2026/05/18 18:44 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/uapi/linux/bpf.h IncludeLines:true Name:bpf_cmd]
Results: map[SourceCode: 155: /* BPF syscall commands, see bpf(2) man-page for more details. */
156: /**
157: * DOC: eBPF Syscall Preamble
158: *
159: * The operation to be performed by the **bpf**\ () system call is determined
160: * by the *cmd* argument. Each operation takes an accompanying argument,
161: * provided via *attr*, which is a pointer to a union of type *bpf_attr* (see
162: * below). The size argument is the size of the union pointed to by *attr*.
163: */
164: /**
165: * DOC: eBPF Syscall Commands
166: *
167: * BPF_MAP_CREATE
168: * Description
169: * Create a map and return a file descriptor that refers to the
170: * map. The close-on-exec file descriptor flag (see **fcntl**\ (2))
171: * is automatically enabled for the new file descriptor.
172: *
173: * Applying **close**\ (2) to the file descriptor returned by
174: * **BPF_MAP_CREATE** will delete the map (but see NOTES).
175: *
176: * Return
177: * A new file descriptor (a nonnegative integer), or -1 if an
178: * error occurred (in which case, *errno* is set appropriately).
179: *
180: * BPF_MAP_LOOKUP_ELEM
181: * Description
182: * Look up an element with a given *key* in the map referred to
183: * by the file descriptor *map_fd*.
184: *
185: * The *flags* argument may be specified as one of the
186: * following:
187: *
188: * **BPF_F_LOCK**
189: * Look up the value of a spin-locked map without
190: * returning the lock. This must be specified if the
191: * elements contain a spinlock.
192: *
193: * Return
194: * Returns zero on success. On error, -1 is returned and *errno*
195: * is set appropriately.
196: *
197: * BPF_MAP_UPDATE_ELEM
198: * Description
199: * Create or update an element (key/value pair) in a specified map.
200: *
201: * The *flags* argument should be specified as one of the
202: * following:
203: *
204: * **BPF_ANY**
205: * Create a new element or update an existing element.
206: * **BPF_NOEXIST**
207: * Create a new element only if it did not exist.
208: * **BPF_EXIST**
209: * Update an existing element.
210: * **BPF_F_LOCK**
211: * Update a spin_lock-ed map element.
212: *
213: * Return
214: * Returns zero on success. On error, -1 is returned and *errno*
215: * is set appropriately.
216: *
217: * May set *errno* to **EINVAL**, **EPERM**, **ENOMEM**,
218: * **E2BIG**, **EEXIST**, or **ENOENT**.
219: *
220: * **E2BIG**
221: * The number of elements in the map reached the
222: * *max_entries* limit specified at map creation time.
223: * **EEXIST**
224: * If *flags* specifies **BPF_NOEXIST** and the element
225: * with *key* already exists in the map.
226: * **ENOENT**
227: * If *flags* specifies **BPF_EXIST** and the element with
228: * *key* does not exist in the map.
229: *
230: * BPF_MAP_DELETE_ELEM
231: * Description
232: * Look up and delete an element by key in a specified map.
233: *
234: * Return
235: * Returns zero on success. On error, -1 is returned and *errno*
236: * is set appropriately.
237: *
238: * BPF_MAP_GET_NEXT_KEY
239: * Description
240: * Look up an element by key in a specified map and return the key
241: * of the next element. Can be used to iterate over all elements
242: * in the map.
243: *
244: * Return
245: * Returns zero on success. On error, -1 is returned and *errno*
246: * is set appropriately.
247: *
248: * The following cases can be used to iterate over all elements of
249: * the map:
250: *
251: * * If *key* is not found, the operation returns zero and sets
252: * the *next_key* pointer to the key of the first element.
253: * * If *key* is found, the operation returns zero and sets the
254: * *next_key* pointer to the key of the next element.
255: * * If *key* is the last element, returns -1 and *errno* is set
256: * to **ENOENT**.
257: *
258: * May set *errno* to **ENOMEM**, **EFAULT**, **EPERM**, or
259: * **EINVAL** on error.
260: *
261: * BPF_PROG_LOAD
262: * Description
263: * Verify and load an eBPF program, returning a new file
264: * descriptor associated with the program.
265: *
266: * Applying **close**\ (2) to the file descriptor returned by
267: * **BPF_PROG_LOAD** will unload the eBPF program (but see NOTES).
268: *
269: * The close-on-exec file descriptor flag (see **fcntl**\ (2)) is
270: * automatically enabled for the new file descriptor.
271: *
272: * Return
273: * A new file descriptor (a nonnegative integer), or -1 if an
274: * error occurred (in which case, *errno* is set appropriately).
275: *
276: * BPF_OBJ_PIN
277: * Description
278: * Pin an eBPF program or map referred by the specified *bpf_fd*
279: * to the provided *pathname* on the filesystem.
280: *
281: * The *pathname* argument must not contain a dot (".").
282: *
283: * On success, *pathname* retains a reference to the eBPF object,
284: * preventing deallocation of the object when the original
285: * *bpf_fd* is closed. This allow the eBPF object to live beyond
286: * **close**\ (\ *bpf_fd*\ ), and hence the lifetime of the parent
287: * process.
288: *
289: * Applying **unlink**\ (2) or similar calls to the *pathname*
290: * unpins the object from the filesystem, removing the reference.
291: * If no other file descriptors or filesystem nodes refer to the
292: * same object, it will be deallocated (see NOTES).
293: *
294: * The filesystem type for the parent directory of *pathname* must
295: * be **BPF_FS_MAGIC**.
296: *
297: * Return
298: * Returns zero on success. On error, -1 is returned and *errno*
299: * is set appropriately.
300: *
301: * BPF_OBJ_GET
302: * Description
303: * Open a file descriptor for the eBPF object pinned to the
304: * specified *pathname*.
305: *
306: * Return
307: * A new file descriptor (a nonnegative integer), or -1 if an
308: * error occurred (in which case, *errno* is set appropriately).
309: *
310: * BPF_PROG_ATTACH
311: * Description
312: * Attach an eBPF program to a *target_fd* at the specified
313: * *attach_type* hook.
314: *
315: * The *attach_type* specifies the eBPF attachment point to
316: * attach the program to, and must be one of *bpf_attach_type*
317: * (see below).
318: *
319: * The *attach_bpf_fd* must be a valid file descriptor for a
320: * loaded eBPF program of a cgroup, flow dissector, LIRC, sockmap
321: * or sock_ops type corresponding to the specified *attach_type*.
322: *
323: * The *target_fd* must be a valid file descriptor for a kernel
324: * object which depends on the attach type of *attach_bpf_fd*:
325: *
326: * **BPF_PROG_TYPE_CGROUP_DEVICE**,
327: * **BPF_PROG_TYPE_CGROUP_SKB**,
328: * **BPF_PROG_TYPE_CGROUP_SOCK**,
329: * **BPF_PROG_TYPE_CGROUP_SOCK_ADDR**,
330: * **BPF_PROG_TYPE_CGROUP_SOCKOPT**,
331: * **BPF_PROG_TYPE_CGROUP_SYSCTL**,
332: * **BPF_PROG_TYPE_SOCK_OPS**
333: *
334: * Control Group v2 hierarchy with the eBPF controller
335: * enabled. Requires the kernel to be compiled with
336: * **CONFIG_CGROUP_BPF**.
337: *
338: * **BPF_PROG_TYPE_FLOW_DISSECTOR**
339: *
340: * Network namespace (eg /proc/self/ns/net).
341: *
342: * **BPF_PROG_TYPE_LIRC_MODE2**
343: *
344: * LIRC device path (eg /dev/lircN). Requires the kernel
345: * to be compiled with **CONFIG_BPF_LIRC_MODE2**.
346: *
347: * **BPF_PROG_TYPE_SK_SKB**,
348: * **BPF_PROG_TYPE_SK_MSG**
349: *
350: * eBPF map of socket type (eg **BPF_MAP_TYPE_SOCKHASH**).
351: *
352: * Return
353: * Returns zero on success. On error, -1 is returned and *errno*
354: * is set appropriately.
355: *
356: * BPF_PROG_DETACH
357: * Description
358: * Detach the eBPF program associated with the *target_fd* at the
359: * hook specified by *attach_type*. The program must have been
360: * previously attached using **BPF_PROG_ATTACH**.
361: *
362: * Return
363: * Returns zero on success. On error, -1 is returned and *errno*
364: * is set appropriately.
365: *
366: * BPF_PROG_TEST_RUN
367: * Description
368: * Run the eBPF program associated with the *prog_fd* a *repeat*
369: * number of times against a provided program context *ctx_in* and
370: * data *data_in*, and return the modified program context
371: * *ctx_out*, *data_out* (for example, packet data), result of the
372: * execution *retval*, and *duration* of the test run.
373: *
374: * The sizes of the buffers provided as input and output
375: * parameters *ctx_in*, *ctx_out*, *data_in*, and *data_out* must
376: * be provided in the corresponding variables *ctx_size_in*,
377: * *ctx_size_out*, *data_size_in*, and/or *data_size_out*. If any
378: * of these parameters are not provided (ie set to NULL), the
379: * corresponding size field must be zero.
380: *
381: * Some program types have particular requirements:
382: *
383: * **BPF_PROG_TYPE_SK_LOOKUP**
384: * *data_in* and *data_out* must be NULL.
385: *
386: * **BPF_PROG_TYPE_RAW_TRACEPOINT**,
387: * **BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE**
388: *
389: * *ctx_out*, *data_in* and *data_out* must be NULL.
390: * *repeat* must be zero.
391: *
392: * BPF_PROG_RUN is an alias for BPF_PROG_TEST_RUN.
393: *
394: * Return
395: * Returns zero on success. On error, -1 is returned and *errno*
396: * is set appropriately.
397: *
398: * **ENOSPC**
399: * Either *data_size_out* or *ctx_size_out* is too small.
400: * **ENOTSUPP**
401: * This command is not supported by the program type of
402: * the program referred to by *prog_fd*.
403: *
404: * BPF_PROG_GET_NEXT_ID
405: * Description
406: * Fetch the next eBPF program currently loaded into the kernel.
407: *
408: * Looks for the eBPF program with an id greater than *start_id*
409: * and updates *next_id* on success. If no other eBPF programs
410: * remain with ids higher than *start_id*, returns -1 and sets
411: * *errno* to **ENOENT**.
412: *
413: * Return
414: * Returns zero on success. On error, or when no id remains, -1
415: * is returned and *errno* is set appropriately.
416: *
417: * BPF_MAP_GET_NEXT_ID
418: * Description
419: * Fetch the next eBPF map currently loaded into the kernel.
420: *
421: * Looks for the eBPF map with an id greater than *start_id*
422: * and updates *next_id* on success. If no other eBPF maps
423: * remain with ids higher than *start_id*, returns -1 and sets
424: * *errno* to **ENOENT**.
425: *
426: * Return
427: * Returns zero on success. On error, or when no id remains, -1
428: * is returned and *errno* is set appropriately.
429: *
430: * BPF_PROG_GET_FD_BY_ID
431: * Description
432: * Open a file descriptor for the eBPF program corresponding to
433: * *prog_id*.
434: *
435: * Return
436: * A new file descriptor (a nonnegative integer), or -1 if an
437: * error occurred (in which case, *errno* is set appropriately).
438: *
439: * BPF_MAP_GET_FD_BY_ID
440: * Description
441: * Open a file descriptor for the eBPF map corresponding to
442: * *map_id*.
443: *
444: * Return
445: * A new file descriptor (a nonnegative integer), or -1 if an
446: * error occurred (in which case, *errno* is set appropriately).
447: *
448: * BPF_OBJ_GET_INFO_BY_FD
449: * Description
450: * Obtain information about the eBPF object corresponding to
451: * *bpf_fd*.
452: *
453: * Populates up to *info_len* bytes of *info*, which will be in
454: * one of the following formats depending on the eBPF object type
455: * of *bpf_fd*:
456: *
457: * * **struct bpf_prog_info**
458: * * **struct bpf_map_info**
459: * * **struct bpf_btf_info**
460: * * **struct bpf_link_info**
461: * * **struct bpf_token_info**
462: *
463: * Return
464: * Returns zero on success. On error, -1 is returned and *errno*
465: * is set appropriately.
466: *
467: * BPF_PROG_QUERY
468: * Description
469: * Obtain information about eBPF programs associated with the
470: * specified *attach_type* hook.
471: *
472: * The *target_fd* must be a valid file descriptor for a kernel
473: * object which depends on the attach type of *attach_bpf_fd*:
474: *
475: * **BPF_PROG_TYPE_CGROUP_DEVICE**,
476: * **BPF_PROG_TYPE_CGROUP_SKB**,
477: * **BPF_PROG_TYPE_CGROUP_SOCK**,
478: * **BPF_PROG_TYPE_CGROUP_SOCK_ADDR**,
479: * **BPF_PROG_TYPE_CGROUP_SOCKOPT**,
480: * **BPF_PROG_TYPE_CGROUP_SYSCTL**,
481: * **BPF_PROG_TYPE_SOCK_OPS**
482: *
483: * Control Group v2 hierarchy with the eBPF controller
484: * enabled. Requires the kernel to be compiled with
485: * **CONFIG_CGROUP_BPF**.
486: *
487: * **BPF_PROG_TYPE_FLOW_DISSECTOR**
488: *
489: * Network namespace (eg /proc/self/ns/net).
490: *
491: * **BPF_PROG_TYPE_LIRC_MODE2**
492: *
493: * LIRC device path (eg /dev/lircN). Requires the kernel
494: * to be compiled with **CONFIG_BPF_LIRC_MODE2**.
495: *
496: * **BPF_PROG_QUERY** always fetches the number of programs
497: * attached and the *attach_flags* which were used to attach those
498: * programs. Additionally, if *prog_ids* is nonzero and the number
499: * of attached programs is less than *prog_cnt*, populates
500: * *prog_ids* with the eBPF program ids of the programs attached
501: * at *target_fd*.
502: *
503: * The following flags may alter the result:
504: *
505: * **BPF_F_QUERY_EFFECTIVE**
506: * Only return information regarding programs which are
507: * currently effective at the specified *target_fd*.
508: *
509: * Return
510: * Returns zero on success. On error, -1 is returned and *errno*
511: * is set appropriately.
512: *
513: * BPF_RAW_TRACEPOINT_OPEN
514: * Description
515: * Attach an eBPF program to a tracepoint *name* to access kernel
516: * internal arguments of the tracepoint in their raw form.
517: *
518: * The *prog_fd* must be a valid file descriptor associated with
519: * a loaded eBPF program of type **BPF_PROG_TYPE_RAW_TRACEPOINT**.
520: *
521: * No ABI guarantees are made about the content of tracepoint
522: * arguments exposed to the corresponding eBPF program.
523: *
524: * Applying **close**\ (2) to the file descriptor returned by
525: * **BPF_RAW_TRACEPOINT_OPEN** will delete the map (but see NOTES).
526: *
527: * Return
528: * A new file descriptor (a nonnegative integer), or -1 if an
529: * error occurred (in which case, *errno* is set appropriately).
530: *
531: * BPF_BTF_LOAD
532: * Description
533: * Verify and load BPF Type Format (BTF) metadata into the kernel,
534: * returning a new file descriptor associated with the metadata.
535: * BTF is described in more detail at
536: * https://www.kernel.org/doc/html/latest/bpf/btf.html.
537: *
538: * The *btf* parameter must point to valid memory providing
539: * *btf_size* bytes of BTF binary metadata.
540: *
541: * The returned file descriptor can be passed to other **bpf**\ ()
542: * subcommands such as **BPF_PROG_LOAD** or **BPF_MAP_CREATE** to
543: * associate the BTF with those objects.
544: *
545: * Similar to **BPF_PROG_LOAD**, **BPF_BTF_LOAD** has optional
546: * parameters to specify a *btf_log_buf*, *btf_log_size* and
547: * *btf_log_level* which allow the kernel to return freeform log
548: * output regarding the BTF verification process.
549: *
550: * Return
551: * A new file descriptor (a nonnegative integer), or -1 if an
552: * error occurred (in which case, *errno* is set appropriately).
553: *
554: * BPF_BTF_GET_FD_BY_ID
555: * Description
556: * Open a file descriptor for the BPF Type Format (BTF)
557: * corresponding to *btf_id*.
558: *
559: * Return
560: * A new file descriptor (a nonnegative integer), or -1 if an
561: * error occurred (in which case, *errno* is set appropriately).
562: *
563: * BPF_TASK_FD_QUERY
564: * Description
565: * Obtain information about eBPF programs associated with the
566: * target process identified by *pid* and *fd*.
567: *
568: * If the *pid* and *fd* are associated with a tracepoint, kprobe
569: * or uprobe perf event, then the *prog_id* and *fd_type* will
570: * be populated with the eBPF program id and file descriptor type
571: * of type **bpf_task_fd_type**. If associated with a kprobe or
572: * uprobe, the *probe_offset* and *probe_addr* will also be
573: * populated. Optionally, if *buf* is provided, then up to
574: * *buf_len* bytes of *buf* will be populated with the name of
575: * the tracepoint, kprobe or uprobe.
576: *
577: * The resulting *prog_id* may be introspected in deeper detail
578: * using **BPF_PROG_GET_FD_BY_ID** and **BPF_OBJ_GET_INFO_BY_FD**.
579: *
580: * Return
581: * Returns zero on success. On error, -1 is returned and *errno*
582: * is set appropriately.
583: *
584: * BPF_MAP_LOOKUP_AND_DELETE_ELEM
585: * Description
586: * Look up an element with the given *key* in the map referred to
587: * by the file descriptor *fd*, and if found, delete the element.
588: *
589: * For **BPF_MAP_TYPE_QUEUE** and **BPF_MAP_TYPE_STACK** map
590: * types, the *flags* argument needs to be set to 0, but for other
591: * map types, it may be specified as:
592: *
593: * **BPF_F_LOCK**
594: * Look up and delete the value of a spin-locked map
595: * without returning the lock. This must be specified if
596: * the elements contain a spinlock.
597: *
598: * The **BPF_MAP_TYPE_QUEUE** and **BPF_MAP_TYPE_STACK** map types
599: * implement this command as a "pop" operation, deleting the top
600: * element rather than one corresponding to *key*.
601: * The *key* and *key_len* parameters should be zeroed when
602: * issuing this operation for these map types.
603: *
604: * This command is only valid for the following map types:
605: * * **BPF_MAP_TYPE_QUEUE**
606: * * **BPF_MAP_TYPE_STACK**
607: * * **BPF_MAP_TYPE_HASH**
608: * * **BPF_MAP_TYPE_PERCPU_HASH**
609: * * **BPF_MAP_TYPE_LRU_HASH**
610: * * **BPF_MAP_TYPE_LRU_PERCPU_HASH**
611: *
612: * Return
613: * Returns zero on success. On error, -1 is returned and *errno*
614: * is set appropriately.
615: *
616: * BPF_MAP_FREEZE
617: * Description
618: * Freeze the permissions of the specified map.
619: *
620: * Write permissions may be frozen by passing zero *flags*.
621: * Upon success, no future syscall invocations may alter the
622: * map state of *map_fd*. Write operations from eBPF programs
623: * are still possible for a frozen map.
624: *
625: * Not supported for maps of type **BPF_MAP_TYPE_STRUCT_OPS**.
626: *
627: * Return
628: * Returns zero on success. On error, -1 is returned and *errno*
629: * is set appropriately.
630: *
631: * BPF_BTF_GET_NEXT_ID
632: * Description
633: * Fetch the next BPF Type Format (BTF) object currently loaded
634: * into the kernel.
635: *
636: * Looks for the BTF object with an id greater than *start_id*
637: * and updates *next_id* on success. If no other BTF objects
638: * remain with ids higher than *start_id*, returns -1 and sets
639: * *errno* to **ENOENT**.
640: *
641: * Return
642: * Returns zero on success. On error, or when no id remains, -1
643: * is returned and *errno* is set appropriately.
644: *
645: * BPF_MAP_LOOKUP_BATCH
646: * Description
647: * Iterate and fetch multiple elements in a map.
648: *
649: * Two opaque values are used to manage batch operations,
650: * *in_batch* and *out_batch*. Initially, *in_batch* must be set
651: * to NULL to begin the batched operation. After each subsequent
652: * **BPF_MAP_LOOKUP_BATCH**, the caller should pass the resultant
653: * *out_batch* as the *in_batch* for the next operation to
654: * continue iteration from the current point. Both *in_batch* and
655: * *out_batch* must point to memory large enough to hold a key,
656: * except for maps of type **BPF_MAP_TYPE_{HASH, PERCPU_HASH,
657: * LRU_HASH, LRU_PERCPU_HASH}**, for which batch parameters
658: * must be at least 4 bytes wide regardless of key size.
659: *
660: * The *keys* and *values* are output parameters which must point
661: * to memory large enough to hold *count* items based on the key
662: * and value size of the map *map_fd*. The *keys* buffer must be
663: * of *key_size* * *count*. The *values* buffer must be of
664: * *value_size* * *count*.
665: *
666: * The *elem_flags* argument may be specified as one of the
667: * following:
668: *
669: * **BPF_F_LOCK**
670: * Look up the value of a spin-locked map without
671: * returning the lock. This must be specified if the
672: * elements contain a spinlock.
673: *
674: * On success, *count* elements from the map are copied into the
675: * user buffer, with the keys copied into *keys* and the values
676: * copied into the corresponding indices in *values*.
677: *
678: * If an error is returned and *errno* is not **EFAULT**, *count*
679: * is set to the number of successfully processed elements.
680: *
681: * Return
682: * Returns zero on success. On error, -1 is returned and *errno*
683: * is set appropriately.
684: *
685: * May set *errno* to **ENOSPC** to indicate that *keys* or
686: * *values* is too small to dump an entire bucket during
687: * iteration of a hash-based map type.
688: *
689: * BPF_MAP_LOOKUP_AND_DELETE_BATCH
690: * Description
691: * Iterate and delete all elements in a map.
692: *
693: * This operation has the same behavior as
694: * **BPF_MAP_LOOKUP_BATCH** with two exceptions:
695: *
696: * * Every element that is successfully returned is also deleted
697: * from the map. This is at least *count* elements. Note that
698: * *count* is both an input and an output parameter.
699: * * Upon returning with *errno* set to **EFAULT**, up to
700: * *count* elements may be deleted without returning the keys
701: * and values of the deleted elements.
702: *
703: * Return
704: * Returns zero on success. On error, -1 is returned and *errno*
705: * is set appropriately.
706: *
707: * BPF_MAP_UPDATE_BATCH
708: * Description
709: * Update multiple elements in a map by *key*.
710: *
711: * The *keys* and *values* are input parameters which must point
712: * to memory large enough to hold *count* items based on the key
713: * and value size of the map *map_fd*. The *keys* buffer must be
714: * of *key_size* * *count*. The *values* buffer must be of
715: * *value_size* * *count*.
716: *
717: * Each element specified in *keys* is sequentially updated to the
718: * value in the corresponding index in *values*. The *in_batch*
719: * and *out_batch* parameters are ignored and should be zeroed.
720: *
721: * The *elem_flags* argument should be specified as one of the
722: * following:
723: *
724: * **BPF_ANY**
725: * Create new elements or update a existing elements.
726: * **BPF_NOEXIST**
727: * Create new elements only if they do not exist.
728: * **BPF_EXIST**
729: * Update existing elements.
730: * **BPF_F_LOCK**
731: * Update spin_lock-ed map elements. This must be
732: * specified if the map value contains a spinlock.
733: *
734: * On success, *count* elements from the map are updated.
735: *
736: * If an error is returned and *errno* is not **EFAULT**, *count*
737: * is set to the number of successfully processed elements.
738: *
739: * Return
740: * Returns zero on success. On error, -1 is returned and *errno*
741: * is set appropriately.
742: *
743: * May set *errno* to **EINVAL**, **EPERM**, **ENOMEM**, or
744: * **E2BIG**. **E2BIG** indicates that the number of elements in
745: * the map reached the *max_entries* limit specified at map
746: * creation time.
747: *
748: * May set *errno* to one of the following error codes under
749: * specific circumstances:
750: *
751: * **EEXIST**
752: * If *flags* specifies **BPF_NOEXIST** and the element
753: * with *key* already exists in the map.
754: * **ENOENT**
755: * If *flags* specifies **BPF_EXIST** and the element with
756: * *key* does not exist in the map.
757: *
758: * BPF_MAP_DELETE_BATCH
759: * Description
760: * Delete multiple elements in a map by *key*.
761: *
762: * The *keys* parameter is an input parameter which must point
763: * to memory large enough to hold *count* items based on the key
764: * size of the map *map_fd*, that is, *key_size* * *count*.
765: *
766: * Each element specified in *keys* is sequentially deleted. The
767: * *in_batch*, *out_batch*, and *values* parameters are ignored
768: * and should be zeroed.
769: *
770: * The *elem_flags* argument may be specified as one of the
771: * following:
772: *
773: * **BPF_F_LOCK**
774: * Look up the value of a spin-locked map without
775: * returning the lock. This must be specified if the
776: * elements contain a spinlock.
777: *
778: * On success, *count* elements from the map are updated.
779: *
780: * If an error is returned and *errno* is not **EFAULT**, *count*
781: * is set to the number of successfully processed elements. If
782: * *errno* is **EFAULT**, up to *count* elements may be been
783: * deleted.
784: *
785: * Return
786: * Returns zero on success. On error, -1 is returned and *errno*
787: * is set appropriately.
788: *
789: * BPF_LINK_CREATE
790: * Description
791: * Attach an eBPF program to a *target_fd* at the specified
792: * *attach_type* hook and return a file descriptor handle for
793: * managing the link.
794: *
795: * Return
796: * A new file descriptor (a nonnegative integer), or -1 if an
797: * error occurred (in which case, *errno* is set appropriately).
798: *
799: * BPF_LINK_UPDATE
800: * Description
801: * Update the eBPF program in the specified *link_fd* to
802: * *new_prog_fd*.
803: *
804: * Return
805: * Returns zero on success. On error, -1 is returned and *errno*
806: * is set appropriately.
807: *
808: * BPF_LINK_GET_FD_BY_ID
809: * Description
810: * Open a file descriptor for the eBPF Link corresponding to
811: * *link_id*.
812: *
813: * Return
814: * A new file descriptor (a nonnegative integer), or -1 if an
815: * error occurred (in which case, *errno* is set appropriately).
816: *
817: * BPF_LINK_GET_NEXT_ID
818: * Description
819: * Fetch the next eBPF link currently loaded into the kernel.
820: *
821: * Looks for the eBPF link with an id greater than *start_id*
822: * and updates *next_id* on success. If no other eBPF links
823: * remain with ids higher than *start_id*, returns -1 and sets
824: * *errno* to **ENOENT**.
825: *
826: * Return
827: * Returns zero on success. On error, or when no id remains, -1
828: * is returned and *errno* is set appropriately.
829: *
830: * BPF_ENABLE_STATS
831: * Description
832: * Enable eBPF runtime statistics gathering.
833: *
834: * Runtime statistics gathering for the eBPF runtime is disabled
835: * by default to minimize the corresponding performance overhead.
836: * This command enables statistics globally.
837: *
838: * Multiple programs may independently enable statistics.
839: * After gathering the desired statistics, eBPF runtime statistics
840: * may be disabled again by calling **close**\ (2) for the file
841: * descriptor returned by this function. Statistics will only be
842: * disabled system-wide when all outstanding file descriptors
843: * returned by prior calls for this subcommand are closed.
844: *
845: * Return
846: * A new file descriptor (a nonnegative integer), or -1 if an
847: * error occurred (in which case, *errno* is set appropriately).
848: *
849: * BPF_ITER_CREATE
850: * Description
851: * Create an iterator on top of the specified *link_fd* (as
852: * previously created using **BPF_LINK_CREATE**) and return a
853: * file descriptor that can be used to trigger the iteration.
854: *
855: * If the resulting file descriptor is pinned to the filesystem
856: * using **BPF_OBJ_PIN**, then subsequent **read**\ (2) syscalls
857: * for that path will trigger the iterator to read kernel state
858: * using the eBPF program attached to *link_fd*.
859: *
860: * Return
861: * A new file descriptor (a nonnegative integer), or -1 if an
862: * error occurred (in which case, *errno* is set appropriately).
863: *
864: * BPF_LINK_DETACH
865: * Description
866: * Forcefully detach the specified *link_fd* from its
867: * corresponding attachment point.
868: *
869: * Return
870: * Returns zero on success. On error, -1 is returned and *errno*
871: * is set appropriately.
872: *
873: * BPF_PROG_BIND_MAP
874: * Description
875: * Bind a map to the lifetime of an eBPF program.
876: *
877: * The map identified by *map_fd* is bound to the program
878: * identified by *prog_fd* and only released when *prog_fd* is
879: * released. This may be used in cases where metadata should be
880: * associated with a program which otherwise does not contain any
881: * references to the map (for example, embedded in the eBPF
882: * program instructions).
883: *
884: * Return
885: * Returns zero on success. On error, -1 is returned and *errno*
886: * is set appropriately.
887: *
888: * BPF_TOKEN_CREATE
889: * Description
890: * Create BPF token with embedded information about what
891: * BPF-related functionality it allows:
892: * - a set of allowed bpf() syscall commands;
893: * - a set of allowed BPF map types to be created with
894: * BPF_MAP_CREATE command, if BPF_MAP_CREATE itself is allowed;
895: * - a set of allowed BPF program types and BPF program attach
896: * types to be loaded with BPF_PROG_LOAD command, if
897: * BPF_PROG_LOAD itself is allowed.
898: *
899: * BPF token is created (derived) from an instance of BPF FS,
900: * assuming it has necessary delegation mount options specified.
901: * This BPF token can be passed as an extra parameter to various
902: * bpf() syscall commands to grant BPF subsystem functionality to
903: * unprivileged processes.
904: *
905: * When created, BPF token is "associated" with the owning
906: * user namespace of BPF FS instance (super block) that it was
907: * derived from, and subsequent BPF operations performed with
908: * BPF token would be performing capabilities checks (i.e.,
909: * CAP_BPF, CAP_PERFMON, CAP_NET_ADMIN, CAP_SYS_ADMIN) within
910: * that user namespace. Without BPF token, such capabilities
911: * have to be granted in init user namespace, making bpf()
912: * syscall incompatible with user namespace, for the most part.
913: *
914: * Return
915: * A new file descriptor (a nonnegative integer), or -1 if an
916: * error occurred (in which case, *errno* is set appropriately).
917: *
918: * BPF_PROG_STREAM_READ_BY_FD
919: * Description
920: * Read data of a program's BPF stream. The program is identified
921: * by *prog_fd*, and the stream is identified by the *stream_id*.
922: * The data is copied to a buffer pointed to by *stream_buf*, and
923: * filled less than or equal to *stream_buf_len* bytes.
924: *
925: * Return
926: * Number of bytes read from the stream on success, or -1 if an
927: * error occurred (in which case, *errno* is set appropriately).
928: *
929: * BPF_PROG_ASSOC_STRUCT_OPS
930: * Description
931: * Associate a BPF program with a struct_ops map. The struct_ops
932: * map is identified by *map_fd* and the BPF program is
933: * identified by *prog_fd*.
934: *
935: * Return
936: * 0 on success or -1 if an error occurred (in which case,
937: * *errno* is set appropriately).
938: *
939: * NOTES
940: * eBPF objects (maps and programs) can be shared between processes.
941: *
942: * * After **fork**\ (2), the child inherits file descriptors
943: * referring to the same eBPF objects.
944: * * File descriptors referring to eBPF objects can be transferred over
945: * **unix**\ (7) domain sockets.
946: * * File descriptors referring to eBPF objects can be duplicated in the
947: * usual way, using **dup**\ (2) and similar calls.
948: * * File descriptors referring to eBPF objects can be pinned to the
949: * filesystem using the **BPF_OBJ_PIN** command of **bpf**\ (2).
950: *
951: * An eBPF object is deallocated only after all file descriptors referring
952: * to the object have been closed and no references remain pinned to the
953: * filesystem or attached (for example, bound to a program or device).
954: */
955: enum bpf_cmd {
956: BPF_MAP_CREATE,
957: BPF_MAP_LOOKUP_ELEM,
958: BPF_MAP_UPDATE_ELEM,
959: BPF_MAP_DELETE_ELEM,
960: BPF_MAP_GET_NEXT_KEY,
961: BPF_PROG_LOAD,
962: BPF_OBJ_PIN,
963: BPF_OBJ_GET,
964: BPF_PROG_ATTACH,
965: BPF_PROG_DETACH,
966: BPF_PROG_TEST_RUN,
967: BPF_PROG_RUN = BPF_PROG_TEST_RUN,
968: BPF_PROG_GET_NEXT_ID,
969: BPF_MAP_GET_NEXT_ID,
970: BPF_PROG_GET_FD_BY_ID,
971: BPF_MAP_GET_FD_BY_ID,
972: BPF_OBJ_GET_INFO_BY_FD,
973: BPF_PROG_QUERY,
974: BPF_RAW_TRACEPOINT_OPEN,
975: BPF_BTF_LOAD,
976: BPF_BTF_GET_FD_BY_ID,
977: BPF_TASK_FD_QUERY,
978: BPF_MAP_LOOKUP_AND_DELETE_ELEM,
979: BPF_MAP_FREEZE,
980: BPF_BTF_GET_NEXT_ID,
981: BPF_MAP_LOOKUP_BATCH,
982: BPF_MAP_LOOKUP_AND_DELETE_BATCH,
983: BPF_MAP_UPDATE_BATCH,
984: BPF_MAP_DELETE_BATCH,
985: BPF_LINK_CREATE,
986: BPF_LINK_UPDATE,
987: BPF_LINK_GET_FD_BY_ID,
988: BPF_LINK_GET_NEXT_ID,
989: BPF_ENABLE_STATS,
990: BPF_ITER_CREATE,
991: BPF_LINK_DETACH,
992: BPF_PROG_BIND_MAP,
993: BPF_TOKEN_CREATE,
994: BPF_PROG_STREAM_READ_BY_FD,
995: BPF_PROG_ASSOC_STRUCT_OPS,
996: __MAX_BPF_CMD,
997: };
SourceFile:include/uapi/linux/bpf.h]
|
| 92/2 |
2026/05/18 18:44 |
llm |
debugger |
0m
Model:
Tokens:
input: 128314
output: 46
thoughts: 647
Thoughts: **Analyzing the BPF Reproducer**
Okay, let's break this down. First, `BPF_PROG_LOAD` is 5, noted. The reproducer focuses only on this. If it's *just* a load, how does the crash happen? The type is `BPF_PROG_TYPE_TRACEPOINT`, which aren't executed on load directly. That's a key observation.
Now, I'm seeing `prog_ifindex` in the reproducer, and that screams hardware offload. But then I remember that `ioctl` call:
```c
// ioctl$ifreq_SIOCGIFINDEX_team arguments: [
// fd: sock (resource)
// cmd: const = 0x8933 (4 bytes)
// arg: ptr[out, ifreq_dev_t["team0", ifindex_team]] {
// ifreq_dev_t["team0", ifindex_team] {
// ifr_ifrn: buffer: {74 65 61 6d 30 00 00 00 00 00 00 00 00 00 00 00} (length 0x10)
// elem: ifindex_team (resource)
// pad = 0x0 (20 bytes)
// }
// }
// ]
memcpy((void*)0x200000000600, "team0\000\000\000\000\000\000\000\000\000\000\000", 16);
syscall(__NR_ioctl, /*fd=*/r[0], /*cmd=*/0x8933, /*arg=*/0x200000000600ul);
```
Wait, it's getting the `ifindex` of `team0`. But wait! the reproducer does the `BPF_PROG_LOAD` *before* the `ioctl`. Then I see that `prog_ifindex` is set to 0.
So, if the program isn't *executed*, how does it crash? I'm thinking the crash is in `__remove_hrtimer`. This is starting to look like a verification issue. Could `bpf_snprintf` be executed in `bpf_check`? I know `bpf_check` doesn't *run* the program, but I need to investigate further whether it *evaluates* the format string of `bpf_snprintf`. I'll go look at `check_bpf_snprintf_call` in `kernel/bpf/verifier.c`.
|
| 93/2 |
2026/05/18 18:45 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/verifier.c IncludeLines:true Name:check_bpf_snprintf_call]
Results: map[SourceCode:10118: static int check_bpf_snprintf_call(struct bpf_verifier_env *env,
10119: struct bpf_reg_state *regs)
10120: {
10121: struct bpf_reg_state *fmt_reg = ®s[BPF_REG_3];
10122: struct bpf_reg_state *data_len_reg = ®s[BPF_REG_5];
10123: struct bpf_map *fmt_map = fmt_reg->map_ptr;
10124: struct bpf_bprintf_data data = {};
10125: int err, fmt_map_off, num_args;
10126: u64 fmt_addr;
10127: char *fmt;
10128:
10129: /* data must be an array of u64 */
10130: if (data_len_reg->var_off.value % 8)
10131: return -EINVAL;
10132: num_args = data_len_reg->var_off.value / 8;
10133:
10134: /* fmt being ARG_PTR_TO_CONST_STR guarantees that var_off is const
10135: * and map_direct_value_addr is set.
10136: */
10137: fmt_map_off = fmt_reg->var_off.value;
10138: err = fmt_map->ops->map_direct_value_addr(fmt_map, &fmt_addr,
10139: fmt_map_off);
10140: if (err) {
10141: verbose(env, "failed to retrieve map value address\n");
10142: return -EFAULT;
10143: }
10144: fmt = (char *)(long)fmt_addr + fmt_map_off;
10145:
10146: /* We are also guaranteed that fmt+fmt_map_off is NULL terminated, we
10147: * can focus on validating the format specifiers.
10148: */
10149: err = bpf_bprintf_prepare(fmt, UINT_MAX, NULL, num_args, &data);
10150: if (err < 0)
10151: verbose(env, "Invalid format string\n");
10152:
10153: return err;
10154: }
SourceFile:kernel/bpf/verifier.c]
|
| 94/2 |
2026/05/18 18:45 |
llm |
debugger |
0m
Model:
Tokens:
input: 130345
output: 42
thoughts: 961
Thoughts: **BPF, Qdiscs, and a Soft Lockup - A Thread of Inquiry**
Okay, hold on a second. I need to trace this `bpf_snprintf_call` because it's calling `bpf_bprintf_prepare` with `args = NULL`. Let's re-examine that `bpf_bprintf_prepare` function. Right, in `check_bpf_snprintf_call`, the `data` struct is initialized as empty. This means `data->get_bin_args` and `data->get_buf` are both false, making `get_buffers` false as well.
Now, within the `bpf_bprintf_prepare` function, if `tmp_buf` is NULL due to `get_buffers` being false, the code simply increments `i` and `num_spec`. That means `bpf_bprintf_prepare` *isn't* actually calling `snprintf` during the verification stage, which is crucial.
Wait, but the reproducer is doing something else. It's using `sendmsg$nl_route_sched` with arguments pointing to `RTM_NEWQDISC`. That's the key: it creates a new qdisc! Is there a BPF program attached to it?
The reproducer shows a `prog_token_fd`, and it loads a BPF program. But, does it actually attach it? No, `bpf_prog_load` by itself *doesn't* attach. Unless it's a `BPF_PROG_TYPE_STRUCT_OPS` which is not the case here. It's a `BPF_PROG_TYPE_TRACEPOINT` (type 5).
So, if that's not the issue, what's causing the soft lockup? Let's trace it out. The lockup is within `toggle_allocation_gate` which goes through `static_key_enable`, then through `arch_jump_label_transform_apply`, `smp_text_poke_batch_finish`, `smp_text_poke_sync_each_cpu`, `on_each_cpu_cond_mask`, `smp_call_function_many_cond`, finally stuck in `csd_lock_wait`. CPU 1 is in `__remove_hrtimer`. Okay, and `toggle_allocation_gate` is in `mm/kfence/core.c`. I need to inspect that `toggle_allocation_gate` function more closely now.
|
| 95/2 |
2026/05/18 18:45 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:mm/kfence/core.c IncludeLines:true Name:toggle_allocation_gate]
Results: map[SourceCode: 887: /*
888: * Set up delayed work, which will enable and disable the static key. We need to
889: * use a work queue (rather than a simple timer), since enabling and disabling a
890: * static key cannot be done from an interrupt.
891: *
892: * Note: Toggling a static branch currently causes IPIs, and here we'll end up
893: * with a total of 2 IPIs to all CPUs. If this ends up a problem in future (with
894: * more aggressive sampling intervals), we could get away with a variant that
895: * avoids IPIs, at the cost of not immediately capturing allocations if the
896: * instructions remain cached.
897: */
898: static void toggle_allocation_gate(struct work_struct *work)
899: {
900: if (!READ_ONCE(kfence_enabled))
901: return;
902:
903: atomic_set(&kfence_allocation_gate, -kfence_burst);
904: #ifdef CONFIG_KFENCE_STATIC_KEYS
905: /* Enable static key, and await allocation to happen. */
906: static_branch_enable(&kfence_allocation_key);
907:
908: wait_event_idle(allocation_wait,
909: atomic_read(&kfence_allocation_gate) > 0 ||
910: !READ_ONCE(kfence_enabled));
911:
912: /* Disable static key and reset timer. */
913: static_branch_disable(&kfence_allocation_key);
914: #endif
915: queue_delayed_work(system_dfl_wq, &kfence_timer,
916: msecs_to_jiffies(kfence_sample_interval));
917: }
SourceFile:mm/kfence/core.c]
|
| 96/2 |
2026/05/18 18:45 |
llm |
debugger |
0m
Model:
Tokens:
input: 132447
output: 182
thoughts: 685
Thoughts: **Debugging a KFENCE Inter-Processor Interrupt (IPI) Stall**
Okay, so `toggle_allocation_gate` – that’s a KFENCE delayed work item, right? It sets up a static key, and that's kicking off an IPI to all the CPUs. CPU 0 is sitting here, waiting on CPU 1 to acknowledge the interrupt, but CPU 1 is hung up in `__remove_hrtimer`. That's the core of the problem, why is CPU 1 stalling?
First thought was to re-examine the assembly of `__remove_hrtimer`. But, that assembly I pulled up earlier... that *isn’t* `__remove_hrtimer`! That’s throwing me off. I need to get the real assembly from the kernel.
Aha! `__remove_hrtimer` has `lockdep_assert_held(&cpu_base->lock);`. Could this assembly be *from* a `lockdep_assert_held` call? That seems likely.
Let's look at the crash dump to get a better look at what is going on, and re-disassemble the code there.
|
| 97/2 |
2026/05/18 18:45 |
tool |
codeexpert |
6m
Args: map[Question:Disassemble this x86_64 code snippet:
ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 45 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d
]
Results: map[Answer:The provided hex string represents a sequence of x86_64 machine code instructions. Below is the step-by-step disassembly and analysis of the code:
### Disassembly
| Offset | Hex Bytes | Instruction | Description |
| :--- | :--- | :--- | :--- |
| 0x00 | `ee` | `out %al, (%dx)` | Output byte in `AL` to I/O port in `DX`. |
| 0x01 | `e8 f6 fd 11 00` | `call 0x11fdfc` | Relative call to a function (let's call it `func1`). |
| 0x06 | `45 85 ed` | `test %r13d, %r13d` | Test the lower 32 bits of `R13`. |
| 0x09 | `48 89 5c 24 08` | `mov %rbx, 0x8(%rsp)` | Save `RBX` to the stack at `[RSP + 8]`. |
| 0x0e | `74 2a` | `je 0x3a` | Jump to offset `0x3a` if `R13D` is zero. |
| 0x10 | `48 8d 7b 18` | `lea 0x18(%rbx), %rdi` | Load effective address of `[RBX + 24]` into `RDI`. |
| 0x14 | `be ff ff ff ff` | `mov $0xffffffff, %esi` | Move `-1` (32-bit) into `ESI`. |
| 0x19 | `e8 ce a2 21 0a` | `call 0xa21a2ec` | Call another function (let's call it `func2`). |
| 0x1e | `41 89 c5` | `mov %eax, %r13d` | Move return value from `EAX` to `R13D`. |
| 0x21 | `31 ff` | `xor %edi, %edi` | Zero out `EDI` (first argument = 0). |
| 0x23 | `89 c6` | `mov %eax, %esi` | Move `EAX` to `ESI` (second argument = status). |
| 0x25 | `e8 d2 fd 11 00` | `call 0x11fdfc` | Call `func1` again with arguments `(0, status)`. |
| 0x2a | `45 85 ed` | `test %r13d, %r13d` | Test the new value of `R13D`. |
| 0x2d | `0f 84 33 02 00 00` | `je 0x266` | Long jump if `R13D` is zero (likely an error or exit path). |
| 0x33 | `e8 84 f9 11 00` | `call 0x11f9bc` | Call a third function (let's call it `func3`). |
| 0x38 | `eb 05` | `jmp 0x3f` | Jump over the next instruction. |
| 0x3a | `e8 7d f9 11 00` | `call 0x11f9bc` | Call `func3` (target of the `je` at `0x0e`). |
| 0x3f | `4d` | `rex.WRB` | REX prefix (likely part of the next instruction). |
### Analysis
1. **I/O and Synchronization**: The code starts with an `out` instruction, which is often used in low-level drivers or for hardware synchronization/signaling.
2. **Function Logic**:
* It calls `func1` (possibly a trace point or a lock acquisition).
* It checks a status in `R13D`.
* If `R13D` is non-zero, it performs an operation on a structure pointed to by `RBX`. Specifically, it passes the address at offset 24 (`0x18`) to `func2` along with `-1`. This pattern is highly characteristic of an **atomic decrement** (e.g., `atomic_dec_return` or `refcount_dec_and_test`).
* After the decrement, it calls `func1` again, this time passing the new status/refcount.
* If the resulting value is zero, it jumps to a distant cleanup/destructor block (`0x266`).
* Otherwise, it proceeds to call `func3` (possibly a "put" or "release" completion).
3. **Context**: This snippet appears to be part of a reference counting or locking mechanism in the Linux kernel, possibly related to networking (skb) or memory management, where `RBX` holds a pointer to a tracked object. The `ee` byte at the start might also be the tail end of a `VMLAUNCH` instruction (`0F 01 EE`) if this code is from a hypervisor (KVM) error path.]
|
| 98/3 |
2026/05/18 18:45 |
agent |
codeexpert |
6m
Model:
Instruction:
You are a capable Linux kernel developer tasked with researching complex questions
about kernel source code. You will be given a concrete question, and need to provide
a concrete answer.
Use tools extensively while researching the question. Don't make assumptions,
or rely on your previous knowledge about the kernel source code, use available tools
to access the actual source code.
Use all available sources of information:
- kernel source code
- documentation in the Documentation dir in the source tree
- git commits descriptions, git blame
Prefer calling several tools at the same time to save round-trips.
Prompt: Disassemble this x86_64 code snippet:
ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 45 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d
Reply: The provided hex string represents a sequence of x86_64 machine code instructions. Below is the step-by-step disassembly and analysis of the code:
### Disassembly
| Offset | Hex Bytes | Instruction | Description |
| :--- | :--- | :--- | :--- |
| 0x00 | `ee` | `out %al, (%dx)` | Output byte in `AL` to I/O port in `DX`. |
| 0x01 | `e8 f6 fd 11 00` | `call 0x11fdfc` | Relative call to a function (let's call it `func1`). |
| 0x06 | `45 85 ed` | `test %r13d, %r13d` | Test the lower 32 bits of `R13`. |
| 0x09 | `48 89 5c 24 08` | `mov %rbx, 0x8(%rsp)` | Save `RBX` to the stack at `[RSP + 8]`. |
| 0x0e | `74 2a` | `je 0x3a` | Jump to offset `0x3a` if `R13D` is zero. |
| 0x10 | `48 8d 7b 18` | `lea 0x18(%rbx), %rdi` | Load effective address of `[RBX + 24]` into `RDI`. |
| 0x14 | `be ff ff ff ff` | `mov $0xffffffff, %esi` | Move `-1` (32-bit) into `ESI`. |
| 0x19 | `e8 ce a2 21 0a` | `call 0xa21a2ec` | Call another function (let's call it `func2`). |
| 0x1e | `41 89 c5` | `mov %eax, %r13d` | Move return value from `EAX` to `R13D`. |
| 0x21 | `31 ff` | `xor %edi, %edi` | Zero out `EDI` (first argument = 0). |
| 0x23 | `89 c6` | `mov %eax, %esi` | Move `EAX` to `ESI` (second argument = status). |
| 0x25 | `e8 d2 fd 11 00` | `call 0x11fdfc` | Call `func1` again with arguments `(0, status)`. |
| 0x2a | `45 85 ed` | `test %r13d, %r13d` | Test the new value of `R13D`. |
| 0x2d | `0f 84 33 02 00 00` | `je 0x266` | Long jump if `R13D` is zero (likely an error or exit path). |
| 0x33 | `e8 84 f9 11 00` | `call 0x11f9bc` | Call a third function (let's call it `func3`). |
| 0x38 | `eb 05` | `jmp 0x3f` | Jump over the next instruction. |
| 0x3a | `e8 7d f9 11 00` | `call 0x11f9bc` | Call `func3` (target of the `je` at `0x0e`). |
| 0x3f | `4d` | `rex.WRB` | REX prefix (likely part of the next instruction). |
### Analysis
1. **I/O and Synchronization**: The code starts with an `out` instruction, which is often used in low-level drivers or for hardware synchronization/signaling.
2. **Function Logic**:
* It calls `func1` (possibly a trace point or a lock acquisition).
* It checks a status in `R13D`.
* If `R13D` is non-zero, it performs an operation on a structure pointed to by `RBX`. Specifically, it passes the address at offset 24 (`0x18`) to `func2` along with `-1`. This pattern is highly characteristic of an **atomic decrement** (e.g., `atomic_dec_return` or `refcount_dec_and_test`).
* After the decrement, it calls `func1` again, this time passing the new status/refcount.
* If the resulting value is zero, it jumps to a distant cleanup/destructor block (`0x266`).
* Otherwise, it proceeds to call `func3` (possibly a "put" or "release" completion).
3. **Context**: This snippet appears to be part of a reference counting or locking mechanism in the Linux kernel, possibly related to networking (skb) or memory management, where `RBX` holds a pointer to a tracked object. The `ee` byte at the start might also be the tail end of a `VMLAUNCH` instruction (`0F 01 EE`) if this code is from a hypervisor (KVM) error path.
|
| 99/4 |
2026/05/18 18:45 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 2420
output: 21
thoughts: 9523
Thoughts: **Dissecting the X86_64 Hex Code**
Alright, the task is to essentially reverse engineer this x86_64 hex string. It appears I need to figure out what this code does. I've been instructed to use tools, but I don't have direct access to a dedicated disassembler. So, I'm going to rely on my knowledge of x86_64 and see if I can figure this out manually by piecing together instructions and looking for recognizable patterns within the provided byte sequence. My initial thought is to try and match byte sequences to known x86_64 instructions. I'll start by breaking down the hex string into individual instructions and figuring out what they *could* represent, and then hopefully be able to recognize some higher-level patterns.
I notice the first two bytes, `ee e8`, and that's immediately throwing me for a loop. `ee` usually translates to `out dx, al`, an I/O instruction, but that seems unusual for a standard kernel code sequence. It might be a red herring, or it might indicate something specific about the system. I also see a series of relative `call` instructions, which are pretty common. There are a few `test` instructions, and a lot of `mov` operations. The `lea` instruction with a relative offset looks interesting, as it is pointing to an offset of the `rbx` register, and could be a struct field. The presence of `0xffffffff` (which is -1) as an immediate value is another clue.
I am looking for potentially common x86_64 patterns that I am familiar with. I am on the lookout for function prologues (saving registers), common library functions, and control flow structures. I also realize that the `e8` prefix is for a `call` instruction, which might be the start of a function. I'm going to look at the calls to see if I can determine if they share similar destinations.
The presence of multiple calls to the same function (relative to the offset of this snippet) seems important. The fact that the jump instruction's target seems to be within this snippet makes me think it's a loop or conditional block. I am going to see if I can match the addresses.
After some analysis, I found a possible pattern in how the calls are relative to each other:
1. `call func1`
2. `test r13d`
3. conditional jump
4. `call func2`
5. `mov`, `xor`, `mov`
6. `call func1`
This looks like a conditional block where a call happens if a certain status is true. But the code does something else if it is false.
I have come to a conclusion that `func1` is called twice, and the second call takes in `func2`'s return.
The structure looks like:
```c
status = func1(...);
if (status != 0) {
status = func2(&obj->field, -1);
func1(0, status); // Calling func1 again using the status
if (status != 0) {
func3(); // calls func3 if the status != 0
}
} else {
func3();
}
```
If the first `func1` returns a non-zero value, `func2` is called. `func1` is called with the return value of `func2`.
The functions are likely using r13d as an int. `0x18(%rbx)` is likely a member of a struct pointed by `rbx`. I am now going to focus on finding `func2`, because it seems like the most interesting part. `0x0a21a2ce` looks like a function that is located at a very distant address compared to the others. I also realize that the last instruction is a REX prefix, `4d`. This is a clue that the next instruction is also likely part of a function.
This is where I am right now. I need to figure out what `func1`, `func2`, and `func3` represent. And also what `r13d` is.
|
| 100/4 |
2026/05/18 18:46 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:thread_info OffsetBits:0 SizeBits:192] map[Name:__state OffsetBits:192 SizeBits:32] map[Name:saved_state OffsetBits:224 SizeBits:32] map[Name:stack OffsetBits:256 SizeBits:64] map[Name:usage OffsetBits:320 SizeBits:32] map[Name:flags OffsetBits:352 SizeBits:32] map[Name:ptrace OffsetBits:384 SizeBits:32] map[Name:on_cpu OffsetBits:416 SizeBits:32] map[Name:wake_entry OffsetBits:448 SizeBits:128] map[Name:wakee_flips OffsetBits:576 SizeBits:32] map[Name:wakee_flip_decay_ts OffsetBits:640 SizeBits:64] map[Name:last_wakee OffsetBits:704 SizeBits:64] map[Name:recent_used_cpu OffsetBits:768 SizeBits:32] map[Name:wake_cpu OffsetBits:800 SizeBits:32] map[Name:on_rq OffsetBits:832 SizeBits:32] map[Name:prio OffsetBits:864 SizeBits:32] map[Name:static_prio OffsetBits:896 SizeBits:32] map[Name:normal_prio OffsetBits:928 SizeBits:32] map[Name:rt_priority OffsetBits:960 SizeBits:32] map[Name:se OffsetBits:1024 SizeBits:2560] map[Name:rt OffsetBits:3584 SizeBits:384] map[Name:dl OffsetBits:3968 SizeBits:2176] map[Name:dl_server OffsetBits:6144 SizeBits:64] map[Name:sched_class OffsetBits:6208 SizeBits:64] map[Name:sched_task_group OffsetBits:6272 SizeBits:64] map[Name:sched_throttle_work OffsetBits:6336 SizeBits:128] map[Name:throttle_node OffsetBits:6464 SizeBits:128] map[Name:throttled OffsetBits:6592 SizeBits:8] map[Name:stats OffsetBits:6656 SizeBits:2048] map[Name:policy OffsetBits:8704 SizeBits:32] map[Name:max_allowed_capacity OffsetBits:8768 SizeBits:64] map[Name:nr_cpus_allowed OffsetBits:8832 SizeBits:32] map[Name:cpus_ptr OffsetBits:8896 SizeBits:64] map[Name:user_cpus_ptr OffsetBits:8960 SizeBits:64] map[Name:cpus_mask OffsetBits:9024 SizeBits:64] map[Name:migration_pending OffsetBits:9088 SizeBits:64] map[Name:migration_disabled OffsetBits:9152 SizeBits:16] map[Name:migration_flags OffsetBits:9168 SizeBits:16] map[Name:rcu_read_lock_nesting OffsetBits:9184 SizeBits:32] map[Name:rcu_read_unlock_special OffsetBits:9216 SizeBits:32] map[Name:rcu_node_entry OffsetBits:9280 SizeBits:128] map[Name:rcu_blocked_node OffsetBits:9408 SizeBits:64] map[Name:rcu_tasks_nvcsw OffsetBits:9472 SizeBits:64] map[Name:rcu_tasks_holdout OffsetBits:9536 SizeBits:8] map[Name:rcu_tasks_idx OffsetBits:9544 SizeBits:8] map[Name:rcu_tasks_idle_cpu OffsetBits:9568 SizeBits:32] map[Name:rcu_tasks_holdout_list OffsetBits:9600 SizeBits:128] map[Name:rcu_tasks_exit_cpu OffsetBits:9728 SizeBits:32] map[Name:rcu_tasks_exit_list OffsetBits:9792 SizeBits:128] map[Name:trc_reader_nesting OffsetBits:9920 SizeBits:32] map[Name:trc_reader_scp OffsetBits:9984 SizeBits:64] map[Name:sched_info OffsetBits:10048 SizeBits:512] map[Name:tasks OffsetBits:10560 SizeBits:128] map[Name:pushable_tasks OffsetBits:10688 SizeBits:320] map[Name:pushable_dl_tasks OffsetBits:11008 SizeBits:192] map[Name:mm OffsetBits:11200 SizeBits:64] map[Name:active_mm OffsetBits:11264 SizeBits:64] map[Name:exit_state OffsetBits:11328 SizeBits:32] map[Name:exit_code OffsetBits:11360 SizeBits:32] map[Name:exit_signal OffsetBits:11392 SizeBits:32] map[Name:pdeath_signal OffsetBits:11424 SizeBits:32] map[Name:jobctl OffsetBits:11456 SizeBits:64] map[Name:personality OffsetBits:11520 SizeBits:32] map[Name:sched_reset_on_fork OffsetBits:11552 SizeBits:1] map[Name:sched_contributes_to_load OffsetBits:11553 SizeBits:1] map[Name:sched_migrated OffsetBits:11554 SizeBits:1] map[Name:sched_task_hot OffsetBits:11555 SizeBits:1] map[Name: OffsetBits:11584 SizeBits:0] map[Name:sched_remote_wakeup OffsetBits:11584 SizeBits:1] map[Name:sched_rt_mutex OffsetBits:11585 SizeBits:1] map[Name:user_dumpable OffsetBits:11586 SizeBits:1] map[Name:in_execve OffsetBits:11587 SizeBits:1] map[Name:in_iowait OffsetBits:11588 SizeBits:1] map[Name:restore_sigmask OffsetBits:11589 SizeBits:1] map[Name:no_cgroup_migration OffsetBits:11590 SizeBits:1] map[Name:frozen OffsetBits:11591 SizeBits:1] map[Name:use_memdelay OffsetBits:11592 SizeBits:1] map[Name:in_memstall OffsetBits:11593 SizeBits:1] map[Name:in_page_owner OffsetBits:11594 SizeBits:1] map[Name:in_eventfd OffsetBits:11595 SizeBits:1] map[Name:reported_split_lock OffsetBits:11596 SizeBits:1] map[Name:in_nf_duplicate OffsetBits:11597 SizeBits:1] map[Name:atomic_flags OffsetBits:11648 SizeBits:64] map[Name:restart_block OffsetBits:11712 SizeBits:448] map[Name:pid OffsetBits:12160 SizeBits:32] map[Name:tgid OffsetBits:12192 SizeBits:32] map[Name:stack_canary OffsetBits:12224 SizeBits:64] map[Name:real_parent OffsetBits:12288 SizeBits:64] map[Name:parent OffsetBits:12352 SizeBits:64] map[Name:children OffsetBits:12416 SizeBits:128] map[Name:sibling OffsetBits:12544 SizeBits:128] map[Name:group_leader OffsetBits:12672 SizeBits:64] map[Name:ptraced OffsetBits:12736 SizeBits:128] map[Name:ptrace_entry OffsetBits:12864 SizeBits:128] map[Name:thread_pid OffsetBits:12992 SizeBits:64] map[Name:pid_links OffsetBits:13056 SizeBits:512] map[Name:thread_node OffsetBits:13568 SizeBits:128] map[Name:vfork_done OffsetBits:13696 SizeBits:64] map[Name:set_child_tid OffsetBits:13760 SizeBits:64] map[Name:clear_child_tid OffsetBits:13824 SizeBits:64] map[Name:worker_private OffsetBits:13888 SizeBits:64] map[Name:utime OffsetBits:13952 SizeBits:64] map[Name:stime OffsetBits:14016 SizeBits:64] map[Name:gtime OffsetBits:14080 SizeBits:64] map[Name:prev_cputime OffsetBits:14144 SizeBits:640] map[Name:vtime OffsetBits:14784 SizeBits:704] map[Name:nvcsw OffsetBits:15488 SizeBits:64] map[Name:nivcsw OffsetBits:15552 SizeBits:64] map[Name:start_time OffsetBits:15616 SizeBits:64] map[Name:start_boottime OffsetBits:15680 SizeBits:64] map[Name:min_flt OffsetBits:15744 SizeBits:64] map[Name:maj_flt OffsetBits:15808 SizeBits:64] map[Name:posix_cputimers OffsetBits:15872 SizeBits:640] map[Name:posix_cputimers_work OffsetBits:16512 SizeBits:1280] map[Name:ptracer_cred OffsetBits:17792 SizeBits:64] map[Name:real_cred OffsetBits:17856 SizeBits:64] map[Name:cred OffsetBits:17920 SizeBits:64] map[Name:cached_requested_key OffsetBits:17984 SizeBits:64] map[Name:comm OffsetBits:18048 SizeBits:128] map[Name:nameidata OffsetBits:18176 SizeBits:64] map[Name:sysvsem OffsetBits:18240 SizeBits:64] map[Name:sysvshm OffsetBits:18304 SizeBits:128] map[Name:last_switch_count OffsetBits:18432 SizeBits:64] map[Name:last_switch_time OffsetBits:18496 SizeBits:64] map[Name:fs OffsetBits:18560 SizeBits:64] map[Name:files OffsetBits:18624 SizeBits:64] map[Name:io_uring OffsetBits:18688 SizeBits:64] map[Name:io_uring_restrict OffsetBits:18752 SizeBits:64] map[Name:nsproxy OffsetBits:18816 SizeBits:64] map[Name:signal OffsetBits:18880 SizeBits:64] map[Name:sighand OffsetBits:18944 SizeBits:64] map[Name:blocked OffsetBits:19008 SizeBits:64] map[Name:real_blocked OffsetBits:19072 SizeBits:64] map[Name:saved_sigmask OffsetBits:19136 SizeBits:64] map[Name:pending OffsetBits:19200 SizeBits:192] map[Name:sas_ss_sp OffsetBits:19392 SizeBits:64] map[Name:sas_ss_size OffsetBits:19456 SizeBits:64] map[Name:sas_ss_flags OffsetBits:19520 SizeBits:32] map[Name:task_works OffsetBits:19584 SizeBits:64] map[Name:audit_context OffsetBits:19648 SizeBits:64] map[Name:loginuid OffsetBits:19712 SizeBits:32] map[Name:sessionid OffsetBits:19744 SizeBits:32] map[Name:seccomp OffsetBits:19776 SizeBits:128] map[Name:syscall_dispatch OffsetBits:19904 SizeBits:256] map[Name:parent_exec_id OffsetBits:20160 SizeBits:64] map[Name:self_exec_id OffsetBits:20224 SizeBits:64] map[Name:alloc_lock OffsetBits:20288 SizeBits:512] map[Name:pi_lock OffsetBits:20800 SizeBits:512] map[Name:wake_q OffsetBits:21312 SizeBits:64] map[Name:pi_waiters OffsetBits:21376 SizeBits:128] map[Name:pi_top_task OffsetBits:21504 SizeBits:64] map[Name:pi_blocked_on OffsetBits:21568 SizeBits:64] map[Name:blocked_on OffsetBits:21632 SizeBits:64] map[Name:blocked_lock OffsetBits:21696 SizeBits:512] map[Name:blocker OffsetBits:22208 SizeBits:64] map[Name:non_block_count OffsetBits:22272 SizeBits:32] map[Name:irqtrace OffsetBits:22336 SizeBits:448] map[Name:hardirq_threaded OffsetBits:22784 SizeBits:32] map[Name:hardirq_chain_key OffsetBits:22848 SizeBits:64] map[Name:softirqs_enabled OffsetBits:22912 SizeBits:32] map[Name:softirq_context OffsetBits:22944 SizeBits:32] map[Name:irq_config OffsetBits:22976 SizeBits:32] map[Name:curr_chain_key OffsetBits:23040 SizeBits:64] map[Name:lockdep_depth OffsetBits:23104 SizeBits:32] map[Name:lockdep_recursion OffsetBits:23136 SizeBits:32] map[Name:held_locks OffsetBits:23168 SizeBits:15360] map[Name:in_ubsan OffsetBits:38528 SizeBits:32] map[Name:journal_info OffsetBits:38592 SizeBits:64] map[Name:bio_list OffsetBits:38656 SizeBits:64] map[Name:plug OffsetBits:38720 SizeBits:64] map[Name:reclaim_state OffsetBits:38784 SizeBits:64] map[Name:io_context OffsetBits:38848 SizeBits:64] map[Name:capture_control OffsetBits:38912 SizeBits:64] map[Name:ptrace_message OffsetBits:38976 SizeBits:64] map[Name:last_siginfo OffsetBits:39040 SizeBits:64] map[Name:ioac OffsetBits:39104 SizeBits:448] map[Name:psi_flags OffsetBits:39552 SizeBits:32] map[Name:acct_rss_mem1 OffsetBits:39616 SizeBits:64] map[Name:acct_vm_mem1 OffsetBits:39680 SizeBits:64] map[Name:acct_timexpd OffsetBits:39744 SizeBits:64] map[Name:mems_allowed OffsetBits:39808 SizeBits:64] map[Name:mems_allowed_seq OffsetBits:39872 SizeBits:448] map[Name:cpuset_mem_spread_rotor OffsetBits:40320 SizeBits:32] map[Name:cgroups OffsetBits:40384 SizeBits:64] map[Name:cg_list OffsetBits:40448 SizeBits:128] map[Name:robust_list OffsetBits:40576 SizeBits:64] map[Name:pi_state_list OffsetBits:40640 SizeBits:128] map[Name:pi_state_cache OffsetBits:40768 SizeBits:64] map[Name:futex_exit_mutex OffsetBits:40832 SizeBits:1088] map[Name:futex_state OffsetBits:41920 SizeBits:32] map[Name:perf_recursion OffsetBits:41952 SizeBits:32] map[Name:perf_event_ctxp OffsetBits:41984 SizeBits:64] map[Name:perf_event_mutex OffsetBits:42048 SizeBits:1088] map[Name:perf_event_list OffsetBits:43136 SizeBits:128] map[Name:perf_ctx_data OffsetBits:43264 SizeBits:64] map[Name:preempt_disable_ip OffsetBits:43328 SizeBits:64] map[Name:mempolicy OffsetBits:43392 SizeBits:64] map[Name:il_prev OffsetBits:43456 SizeBits:16] map[Name:il_weight OffsetBits:43472 SizeBits:8] map[Name:pref_node_fork OffsetBits:43488 SizeBits:16] map[Name:numa_scan_seq OffsetBits:43520 SizeBits:32] map[Name:numa_scan_period OffsetBits:43552 SizeBits:32] map[Name:numa_scan_period_max OffsetBits:43584 SizeBits:32] map[Name:numa_preferred_nid OffsetBits:43616 SizeBits:32] map[Name:numa_migrate_retry OffsetBits:43648 SizeBits:64] map[Name:node_stamp OffsetBits:43712 SizeBits:64] map[Name:last_task_numa_placement OffsetBits:43776 SizeBits:64] map[Name:last_sum_exec_runtime OffsetBits:43840 SizeBits:64] map[Name:numa_work OffsetBits:43904 SizeBits:128] map[Name:numa_group OffsetBits:44032 SizeBits:64] map[Name:numa_faults OffsetBits:44096 SizeBits:64] map[Name:total_numa_faults OffsetBits:44160 SizeBits:64] map[Name:numa_faults_locality OffsetBits:44224 SizeBits:192] map[Name:numa_pages_migrated OffsetBits:44416 SizeBits:64] map[Name:rseq OffsetBits:44480 SizeBits:320] map[Name:mm_cid OffsetBits:44800 SizeBits:192] map[Name:tlb_ubc OffsetBits:44992 SizeBits:192] map[Name:splice_pipe OffsetBits:45184 SizeBits:64] map[Name:task_frag OffsetBits:45248 SizeBits:128] map[Name:make_it_fail OffsetBits:45376 SizeBits:32] map[Name:fail_nth OffsetBits:45408 SizeBits:32] map[Name:nr_dirtied OffsetBits:45440 SizeBits:32] map[Name:nr_dirtied_pause OffsetBits:45472 SizeBits:32] map[Name:dirty_paused_when OffsetBits:45504 SizeBits:64] map[Name:timer_slack_ns OffsetBits:45568 SizeBits:64] map[Name:default_timer_slack_ns OffsetBits:45632 SizeBits:64] map[Name:kasan_depth OffsetBits:45696 SizeBits:32] map[Name:trace_recursion OffsetBits:45760 SizeBits:64] map[Name:kcov_mode OffsetBits:45824 SizeBits:32] map[Name:kcov_size OffsetBits:45856 SizeBits:32] map[Name:kcov_area OffsetBits:45888 SizeBits:64] map[Name:kcov OffsetBits:45952 SizeBits:64] map[Name:kcov_handle OffsetBits:46016 SizeBits:64] map[Name:kcov_sequence OffsetBits:46080 SizeBits:32] map[Name:kcov_softirq OffsetBits:46112 SizeBits:32] map[Name:memcg_nr_pages_over_high OffsetBits:46144 SizeBits:32] map[Name:active_memcg OffsetBits:46208 SizeBits:64] map[Name:objcg OffsetBits:46272 SizeBits:64] map[Name:throttle_disk OffsetBits:46336 SizeBits:64] map[Name:sequential_io OffsetBits:46400 SizeBits:32] map[Name:sequential_io_avg OffsetBits:46432 SizeBits:32] map[Name:kmap_ctrl OffsetBits:46464 SizeBits:0] map[Name:task_state_change OffsetBits:46464 SizeBits:64] map[Name:rcu OffsetBits:46528 SizeBits:128] map[Name:rcu_users OffsetBits:46656 SizeBits:32] map[Name:pagefault_disabled OffsetBits:46688 SizeBits:32] map[Name:oom_reaper_list OffsetBits:46720 SizeBits:64] map[Name:oom_reaper_timer OffsetBits:46784 SizeBits:640] map[Name:stack_vm_area OffsetBits:47424 SizeBits:64] map[Name:stack_refcount OffsetBits:47488 SizeBits:32] map[Name:security OffsetBits:47552 SizeBits:64] map[Name:bpf_storage OffsetBits:47616 SizeBits:64] map[Name:bpf_ctx OffsetBits:47680 SizeBits:64] map[Name:bpf_net_context OffsetBits:47744 SizeBits:64] map[Name:mce_vaddr OffsetBits:47808 SizeBits:64] map[Name:mce_kflags OffsetBits:47872 SizeBits:64] map[Name:mce_addr OffsetBits:47936 SizeBits:64] map[Name:mce_ripv OffsetBits:48000 SizeBits:1] map[Name:mce_whole_page OffsetBits:48001 SizeBits:1] map[Name:__mce_reserved OffsetBits:48002 SizeBits:62] map[Name:mce_kill_me OffsetBits:48064 SizeBits:128] map[Name:mce_count OffsetBits:48192 SizeBits:32] map[Name:l1d_flush_kill OffsetBits:48256 SizeBits:128] map[Name:unwind_info OffsetBits:48384 SizeBits:320] map[Name:thread OffsetBits:48704 SizeBits:1216]]]
|
| 101/4 |
2026/05/18 18:46 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 19212
output: 19
thoughts: 2903
Thoughts: **Analyzing a Snippet of Assembly Code**
Okay, here's the deal. I'm looking at this assembly, and it's got me thinking in circles a bit. First off, it's immediately throwing me off with that `out %al, %dx` instruction at the beginning. It's unusual, but my immediate thought is that it could be related to some hardware interaction, maybe some kind of lock/unlock sequence, or even a very specific part of the kernel initializing a hardware state. But, it could also be padding or an `alternative` instruction - so I am not sure.
Initially, I looked at offsets and assumed something was wrong with the `task_struct`, specifically with `__state`. The address calculation was spot on, `192 bits = 24 bytes`, but `__state` is just a `unsigned int` which is only 32 bits, which felt wrong. This wasn't the correct path. So I discarded this thought.
I'm seeing `func2` being called with what looks like a pointer to a field at offset 0x18 from `rbx`, and `-1`. This looks like some sort of atomic operation or modification. I initially thought it might be something like `atomic_add_return` or something similar, maybe `atomic_dec_return` but this is unlikely because `__state` is probably modified through `set_current_state`.
Then I got fixated on figuring out what those functions, `func1` and `func2` and `func3`, are. I was trying to map these calls to some common kernel locking primitives. I considered `preempt_disable/enable` because of the `test %r13d, %r13d` and conditional jump, and that `func1` is called twice. Initially, I thought that `func1` may be a form of `__preempt_count_add/sub`, perhaps through a macro like `preempt_count_add(-val)`. However, it felt like the arguments didn't quite line up. My thoughts then pivoted to `spin_lock` and `spin_unlock`. If `func1` is `local_irq_restore`, then what's that initial call? No arguments. This felt confusing.
The offsets from the dis-assembly don't line up, so I am going to recalculate those.
* `func1` = `0x11fdfc`
* `func2` = `0x0a21a2ec`
* `func3` = `0x11f9bc`
These are my current thoughts: I see a pattern: `status = func1()` ; `if (status) { status = func2(...); func1(0, status); }` ; `func3()`
It looks like we are dealing with a nested operation. Some status is checked, and a function is called conditional, and then another function is called.
The `lea 0x18(%rbx), %rdi` is the key. I need to identify what struct `rbx` points to. Offset 0x18 is 24 bytes. I thought of `struct task_struct`, `struct file`, `struct inode`, `struct rq`, `struct lock_class`, but no certain struct felt right. Perhaps, I am looking for `__state` which is `unsigned int`.
I want to search the source for those instructions, which will hopefully reveal the struct being used. I will search for `test %r13d, %r13d` ; `mov %rbx, 0x8(%rsp)` ; `je ...` and also search for `lea 0x18(%rbx), %rdi` with `mov $-1, %esi`. Perhaps, I can use the `-1` to narrow the search. Maybe, I should try to search for `atomic_dec_return`, as that fits the call pattern with a `-1` argument.
I am not sure what the purpose of the `ee` instruction is. It is unusual.
|
| 102/4 |
2026/05/18 18:46 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:nr_running OffsetBits:0 SizeBits:32] map[Name:nr_numa_running OffsetBits:32 SizeBits:32] map[Name:nr_preferred_running OffsetBits:64 SizeBits:32] map[Name:ttwu_pending OffsetBits:96 SizeBits:32] map[Name:cpu_capacity OffsetBits:128 SizeBits:64] map[Name: OffsetBits:192 SizeBits:64] map[Name:idle OffsetBits:256 SizeBits:64] map[Name:nr_switches OffsetBits:512 SizeBits:64] map[Name:__lock OffsetBits:576 SizeBits:512] map[Name:nohz_tick_stopped OffsetBits:1088 SizeBits:32] map[Name:nohz_flags OffsetBits:1120 SizeBits:32] map[Name:has_blocked_load OffsetBits:1152 SizeBits:32] map[Name:last_blocked_load_update_tick OffsetBits:1216 SizeBits:64] map[Name:nohz_csd OffsetBits:1280 SizeBits:256] map[Name:cfs OffsetBits:1536 SizeBits:4608] map[Name:rt OffsetBits:6144 SizeBits:13376] map[Name:dl OffsetBits:19520 SizeBits:832] map[Name:fair_server OffsetBits:20352 SizeBits:2176] map[Name:leaf_cfs_rq_list OffsetBits:22528 SizeBits:128] map[Name:tmp_alone_branch OffsetBits:22656 SizeBits:64] map[Name:numa_migrate_on OffsetBits:22720 SizeBits:32] map[Name:nr_uninterruptible OffsetBits:22784 SizeBits:64] map[Name:dl_server OffsetBits:22848 SizeBits:64] map[Name:stop OffsetBits:22912 SizeBits:64] map[Name:next_class OffsetBits:22976 SizeBits:64] map[Name:next_balance OffsetBits:23040 SizeBits:64] map[Name:prev_mm OffsetBits:23104 SizeBits:64] map[Name:clock_task OffsetBits:23552 SizeBits:64] map[Name:clock_pelt OffsetBits:23616 SizeBits:64] map[Name:clock OffsetBits:23680 SizeBits:64] map[Name:lost_idle_time OffsetBits:23744 SizeBits:64] map[Name:clock_update_flags OffsetBits:23808 SizeBits:32] map[Name:clock_pelt_idle OffsetBits:23872 SizeBits:64] map[Name:clock_idle OffsetBits:23936 SizeBits:64] map[Name:last_seen_need_resched_ns OffsetBits:24000 SizeBits:64] map[Name:ticks_without_resched OffsetBits:24064 SizeBits:32] map[Name:membarrier_state OffsetBits:24096 SizeBits:32] map[Name:rd OffsetBits:24128 SizeBits:64] map[Name:sd OffsetBits:24192 SizeBits:64] map[Name:balance_callback OffsetBits:24256 SizeBits:64] map[Name:nohz_idle_balance OffsetBits:24320 SizeBits:8] map[Name:idle_balance OffsetBits:24328 SizeBits:8] map[Name:misfit_task_load OffsetBits:24384 SizeBits:64] map[Name:active_balance OffsetBits:24448 SizeBits:32] map[Name:push_cpu OffsetBits:24480 SizeBits:32] map[Name:active_balance_work OffsetBits:24512 SizeBits:384] map[Name:cpu OffsetBits:24896 SizeBits:32] map[Name:online OffsetBits:24928 SizeBits:32] map[Name:cfs_tasks OffsetBits:24960 SizeBits:128] map[Name:avg_rt OffsetBits:25088 SizeBits:512] map[Name:avg_dl OffsetBits:25600 SizeBits:512] map[Name:avg_irq OffsetBits:26112 SizeBits:512] map[Name:idle_stamp OffsetBits:26624 SizeBits:64] map[Name:avg_idle OffsetBits:26688 SizeBits:64] map[Name:max_idle_balance_cost OffsetBits:26752 SizeBits:64] map[Name:hotplug_wait OffsetBits:26816 SizeBits:64] map[Name:prev_irq_time OffsetBits:26880 SizeBits:64] map[Name:psi_irq_time OffsetBits:26944 SizeBits:64] map[Name:calc_load_update OffsetBits:27008 SizeBits:64] map[Name:calc_load_active OffsetBits:27072 SizeBits:64] map[Name:hrtick_csd OffsetBits:27136 SizeBits:256] map[Name:hrtick_timer OffsetBits:27392 SizeBits:640] map[Name:hrtick_time OffsetBits:28032 SizeBits:64] map[Name:hrtick_delay OffsetBits:28096 SizeBits:64] map[Name:hrtick_sched OffsetBits:28160 SizeBits:32] map[Name:rq_sched_info OffsetBits:28224 SizeBits:512] map[Name:rq_cpu_time OffsetBits:28736 SizeBits:64] map[Name:yld_count OffsetBits:28800 SizeBits:32] map[Name:sched_count OffsetBits:28832 SizeBits:32] map[Name:sched_goidle OffsetBits:28864 SizeBits:32] map[Name:ttwu_count OffsetBits:28896 SizeBits:32] map[Name:ttwu_local OffsetBits:28928 SizeBits:32] map[Name:idle_state OffsetBits:28992 SizeBits:64] map[Name:nr_pinned OffsetBits:29056 SizeBits:32] map[Name:push_busy OffsetBits:29088 SizeBits:32] map[Name:push_work OffsetBits:29120 SizeBits:384] map[Name:scratch_mask OffsetBits:29504 SizeBits:64] map[Name:cfsb_csd OffsetBits:29696 SizeBits:256] map[Name:cfsb_csd_list OffsetBits:29952 SizeBits:128] map[Name:nr_iowait OffsetBits:30080 SizeBits:32]]]
|
| 103/4 |
2026/05/18 18:46 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 24938
output: 21
thoughts: 67
Thoughts: Here's my summary, as the expert:
**Deep Dive into Kernel Data Structures**
Okay, so I'm digging into `struct rq` again, specifically the memory layout. I'm focusing on offsets and sizes. I'm seeing a bit of a wrinkle at offset 24, which translates to 192 bits. It's an unnamed field, so I suspect it's either padding for alignment or possibly a union. Hmm, the size of that field, as reported, is 64 bits. Now, that's interesting and I need to be sure of my interpretation. That section warrants a second look; I'll need to double check the struct definition to be 100% on the purpose of that field. Further, I see `idle` at 256 bits, which means that field begins at a different offset.
Let's switch gears and investigate `struct mm_struct` next. I need to get a broader view of how these structures relate and how memory is actually mapped. This is where the real complexity begins, and understanding the MMU is crucial. I need to understand the implications of these offsets for performance and potential cache line effects, as well as possible security considerations.
|
| 104/4 |
2026/05/18 18:46 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name: OffsetBits:0 SizeBits:22528] map[Name:flexible_array OffsetBits:22528 SizeBits:0]]]
|
| 105/4 |
2026/05/18 18:46 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 25339
output: 21
thoughts: 1847
Thoughts: **Deconstructing the Assembly**
Okay, this is interesting. I'm looking at this assembly sequence, and my initial thought is, it's not immediately obvious what this is doing. Let's break it down methodically.
First, I need to get a grip on the assembly pattern. I see `xor %edi, %edi`, followed by `mov %eax, %esi`, and then a `call` instruction. This strongly suggests a function call with arguments. Specifically, it's very likely `func(0, eax)`. If that's the case and the function is `__raw_spin_unlock_irqrestore`, which seems plausible, the code is likely related to handling interrupts. And the function after that might be `local_irq_restore`, which is itself likely a macro that calls `native_irq_restore`. Hang on... the `ee` instruction throws a wrench in this. It's `out %al, %dx`, which usually has to do with direct hardware access. That's unexpected in an interrupt-handling context, so I'll put a pin in that for now.
Wait, what if the `ee` instruction is a marker, like `0xee` for `__this_cpu_...`? No, that doesn't make sense.
I'm now focusing on a specific sequence: `0: ee`, `1: e8 f6 fd 11 00`. This looks like an `out` instruction followed by a `call`. This *could* be related to tracing, maybe a `trace_...` function, or some specific hardware interaction with `outb`.
I'm going to shift my focus and start looking at this `4d` instruction sequence, specifically `45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18` and look for this pattern inside the kernel. The `48 89 5c 24 08` is `mov %rbx, 0x8(%rsp)` – saving a value on the stack. Saving `rbx` in the middle of a function could signal something important, maybe a "cold" path, or conditional execution, perhaps with some unusual stack management.
Okay, `74 2a` is a `je +0x2a` conditional jump. If the condition is met, it skips some code and calls another function, `func3`. The jump target is at offset 58.
So, if `r13d == 0`, the code jumps. Otherwise, the code executes some instructions. Let's see: `48 8d 7b 18` is `lea 0x18(%rbx),%rdi`, then `be ff ff ff ff` is `mov $0xffffffff,%esi`, then `e8 ce a2 21 0a` is `call func2`. Then we have `mov %eax,%r13d`. This sequence suggests that the return value of `func2` is being stored in `r13d`. Following that is `xor %edi,%edi`, `mov %eax,%esi`, then a call to `func1`. Following that is a conditional branch, the `test %r13d,%r13d`, `je 0x266` instruction. So it looks like `r13d` is tested, and if `r13d == 0`, a jump happens.
Wait, `func3` is called whether the jump occurs or not.
I think I see a pattern here:
```c
if (r13d) {
r13d = func2(&rbx->field, -1);
func1(0, r13d);
if (r13d == 0) goto label_far;
}
func3();
```
Let's re-check the jump. The call at address `3a` is also `func3`! So, yes, `func3()` is called in both cases. Now, the code is roughly equivalent to:
```c
if (r13d) {
r13d = func2(&rbx->field, -1);
func1(0, r13d);
if (r13d == 0) return; // or something
}
func3();
```
Aha! `func2` is called with `-1`. This looks like a decrement function. Specifically, it very likely is `atomic_add_return
|
| 106/4 |
2026/05/18 18:46 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name: OffsetBits:0 SizeBits:192] map[Name:sk OffsetBits:192 SizeBits:64] map[Name: OffsetBits:256 SizeBits:64] map[Name:cb OffsetBits:320 SizeBits:384] map[Name: OffsetBits:704 SizeBits:128] map[Name:_nfct OffsetBits:832 SizeBits:64] map[Name:len OffsetBits:896 SizeBits:32] map[Name:data_len OffsetBits:928 SizeBits:32] map[Name:mac_len OffsetBits:960 SizeBits:16] map[Name:hdr_len OffsetBits:976 SizeBits:16] map[Name:queue_mapping OffsetBits:992 SizeBits:16] map[Name:__cloned_offset OffsetBits:1008 SizeBits:0] map[Name:cloned OffsetBits:1008 SizeBits:1] map[Name:nohdr OffsetBits:1009 SizeBits:1] map[Name:fclone OffsetBits:1010 SizeBits:2] map[Name:peeked OffsetBits:1012 SizeBits:1] map[Name:head_frag OffsetBits:1013 SizeBits:1] map[Name:pfmemalloc OffsetBits:1014 SizeBits:1] map[Name:pp_recycle OffsetBits:1015 SizeBits:1] map[Name:active_extensions OffsetBits:1016 SizeBits:8] map[Name: OffsetBits:1024 SizeBits:576] map[Name:tail OffsetBits:1600 SizeBits:32] map[Name:end OffsetBits:1632 SizeBits:32] map[Name:head OffsetBits:1664 SizeBits:64] map[Name:data OffsetBits:1728 SizeBits:64] map[Name:truesize OffsetBits:1792 SizeBits:32] map[Name:users OffsetBits:1824 SizeBits:32] map[Name:extensions OffsetBits:1856 SizeBits:64]]]
|
| 107/4 |
2026/05/18 18:46 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 28812
output: 19
thoughts: 166
Thoughts: **Decoding a Memory Access Mystery**
Okay, so I'm looking at this code and something's just not clicking. At offset 24, which translates to 192 bits, I see `sk`. Alright, that's supposed to be a pointer, probably a `struct sock`. Then there's this call to `func2(&skb->sk, -1)`. Hmmm, I can't quite make sense of that; the function call's structure doesn't seem quite right to pass that to `atomic_dec_return`.
Wait a minute, maybe I'm overlooking something about the `skb_shared_info`? That's always at the end of the `skb->head`. But the register being used, `rbx`, usually points to the main struct itself. This isn't adding up.
Okay, let's take a step back and shift focus. Let's look at `struct page` for a moment. If I remember correctly, the struct is 64 bytes in size. Offset 24 in `struct page`... what's there? Aha! I think offset 24 should be `_refcount`. But wait, *is* that actually the location of `_refcount` in `struct page`? I need to double-check that. Let's verify that's the correct offset.
|
| 108/4 |
2026/05/18 18:46 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:flags OffsetBits:0 SizeBits:64] map[Name: OffsetBits:64 SizeBits:320] map[Name: OffsetBits:384 SizeBits:32] map[Name:_refcount OffsetBits:416 SizeBits:32] map[Name:memcg_data OffsetBits:448 SizeBits:64]]]
|
| 109/4 |
2026/05/18 18:46 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 29403
output: 30
thoughts: 1670
Thoughts: **Dissecting a Kernel Code Snippet: A Journey Through Assembly and Logic**
Okay, so I'm looking at this assembly snippet, and initially, I was trying to make sense of the `_refcount` offset and comparing it to `struct bio` but quickly realized it wasn't the right track. My mind then jumped to looking at the `ee` byte, trying to figure out if it held a hidden meaning or was a prefix. I briefly entertained the thought of architecture differences but knew the prompt said x86_64 so I put that to the side. Searching for byte sequences led me to the `r13d` register and some possible function names but that seemed like it wasn't the best path. The thought process went towards possible function names, and the typical function call inlining scenarios, but nothing clicked.
Then, I went back to the `ee` byte. `out %al, %dx` was my initial interpretation, which didn't feel right. I started searching for the byte sequence `45 85 ed 48 89 5c 24 08 74 2a` in the kernel source, and thought I found a C code snippet which didn't look right. I noticed that the assembly was saving `rbx` to the stack which was odd because it was immediately used in the next instruction. I then realized the prologue would be at the start, and it was in the middle.
A glimmer of an idea struck: maybe the `ee` wasn't the *start* of anything, but the *end* of the *previous* instruction. That led me to consider `vmlaunch` which ends in `ee`. I knew that the `vmlaunch` instruction can only fail and return if it does not enter the guest and that the next instruction should be an error path!
Now I'm focusing on the KVM code. I'm checking `arch/x86/kvm/vmx/vmenter.S` and `arch/x86/kvm/vmx/vmx.c` to see if my hypothesis that this is a `vmlaunch` error handler is correct. In `vmenter.S`, I've got confirmation! `vmlaunch` is indeed `0x0f 0x01 0xee`, and the provided code snippet starts with `ee`. So, this is a handler for an error after a `vmlaunch` or `vmresume`! The `e8 f6 fd 11 00` instruction is a call to handle the error. I'm going to look deeper into `arch/x86/kvm/vmx/vmenter.S` to see the details of this error handling.
|
| 110/4 |
2026/05/18 18:47 |
tool |
codesearch-dir-index |
0m
Args: map[Dir:arch/x86/kvm/vmx/]
Results: map[Files:[capabilities.h common.h hyperv.c hyperv.h hyperv_evmcs.c hyperv_evmcs.h main.c nested.c nested.h pmu_intel.c pmu_intel.h posted_intr.c posted_intr.h run_flags.h sgx.c sgx.h tdx.c tdx.h tdx_arch.h tdx_errno.h vmcs.h vmcs12.c vmcs12.h vmcs_shadow_fields.h vmenter.S vmx.c vmx.h vmx_onhyperv.c vmx_onhyperv.h vmx_ops.h x86_ops.h] Subdirs:<nil>]
|
| 111/4 |
2026/05/18 18:47 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 31961
output: 32
thoughts: 23
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and presented as if it were my own internal monologue:
**Diving into the VMX Entry Point**
Right, let's take a look at `arch/x86/kvm/vmx/vmenter.S`. I need to refresh my mental model of the VMX entry sequence. I know the high-level architecture: the host prepares the guest state, sets up the VMCS, and then triggers the `VM-Enter` instruction. But the nitty-gritty details of the assembly, the specific registers being manipulated, the shadow page tables, interrupt handling, and the control-flow within `vmenter.S` are what I'm interested in here. I want to understand how the processor transitions from host to guest execution, including the handling of preemption, nested virtualization considerations (if any), and any specific optimizations implemented by the KVM developers. It's time to map the documented behavior to the implemented instructions. I need to refresh my knowledge of the interaction between the VMX root and non-root modes, and the role each instruction plays. Specifically, I'll be looking to understand the mechanics that set the stage for the guest’s execution; memory permissions, register loading, and state saving.
|
| 112/4 |
2026/05/18 18:47 |
tool |
read-file |
0m
Args: map[File:arch/x86/kvm/vmx/vmenter.S]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/linkage.h>
#include <asm/asm.h>
#include <asm/bitsperlong.h>
#include <asm/kvm_vcpu_regs.h>
#include <asm/nospec-branch.h>
#include <asm/percpu.h>
#include <asm/segment.h>
#include "kvm-asm-offsets.h"
#include "run_flags.h"
#define WORD_SIZE (BITS_PER_LONG / 8)
#define VCPU_RAX __VCPU_REGS_RAX * WORD_SIZE
#define VCPU_RCX __VCPU_REGS_RCX * WORD_SIZE
#define VCPU_RDX __VCPU_REGS_RDX * WORD_SIZE
#define VCPU_RBX __VCPU_REGS_RBX * WORD_SIZE
/* Intentionally omit RSP as it's context switched by hardware */
#define VCPU_RBP __VCPU_REGS_RBP * WORD_SIZE
#define VCPU_RSI __VCPU_REGS_RSI * WORD_SIZE
#define VCPU_RDI __VCPU_REGS_RDI * WORD_SIZE
#ifdef CONFIG_X86_64
#define VCPU_R8 __VCPU_REGS_R8 * WORD_SIZE
#define VCPU_R9 __VCPU_REGS_R9 * WORD_SIZE
#define VCPU_R10 __VCPU_REGS_R10 * WORD_SIZE
#define VCPU_R11 __VCPU_REGS_R11 * WORD_SIZE
#define VCPU_R12 __VCPU_REGS_R12 * WORD_SIZE
#define VCPU_R13 __VCPU_REGS_R13 * WORD_SIZE
#define VCPU_R14 __VCPU_REGS_R14 * WORD_SIZE
#define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE
#endif
.macro VMX_DO_EVENT_IRQOFF call_insn call_target
/*
* Unconditionally create a stack frame, getting the correct RSP on the
* stack (for x86-64) would take two instructions anyways, and RBP can
* be used to restore RSP to make objtool happy (see below).
*/
push %_ASM_BP
mov %_ASM_SP, %_ASM_BP
#ifdef CONFIG_X86_64
/*
* Align RSP to a 16-byte boundary (to emulate CPU behavior) before
* creating the synthetic interrupt stack frame for the IRQ/NMI.
*/
and $-16, %rsp
push $__KERNEL_DS
push %rbp
#endif
pushf
push $__KERNEL_CS
\call_insn \call_target
/*
* "Restore" RSP from RBP, even though IRET has already unwound RSP to
* the correct value. objtool doesn't know the callee will IRET and,
* without the explicit restore, thinks the stack is getting walloped.
* Using an unwind hint is problematic due to x86-64's dynamic alignment.
*/
leave
RET
.endm
.section .noinstr.text, "ax"
/**
* __vmx_vcpu_run - Run a vCPU via a transition to VMX guest mode
* @vmx: struct vcpu_vmx *
* @regs: unsigned long * (to guest registers)
* @flags: VMX_RUN_VMRESUME: use VMRESUME instead of VMLAUNCH
* VMX_RUN_SAVE_SPEC_CTRL: save guest SPEC_CTRL into vmx->spec_ctrl
* VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO: vCPU can access host MMIO
*
* Returns:
* 0 on VM-Exit, 1 on VM-Fail
*/
SYM_FUNC_START(__vmx_vcpu_run)
push %_ASM_BP
mov %_ASM_SP, %_ASM_BP
#ifdef CONFIG_X86_64
push %r15
push %r14
push %r13
push %r12
#else
push %edi
push %esi
#endif
push %_ASM_BX
/* Save @vmx for SPEC_CTRL handling */
push %_ASM_ARG1
/* Save @flags (used for VMLAUNCH vs. VMRESUME and mitigations). */
push %_ASM_ARG3
/*
* Save @regs, _ASM_ARG2 may be modified by vmx_update_host_rsp() and
* @regs is needed after VM-Exit to save the guest's register values.
*/
push %_ASM_ARG2
lea (%_ASM_SP), %_ASM_ARG2
call vmx_update_host_rsp
ALTERNATIVE "jmp .Lspec_ctrl_done", "", X86_FEATURE_MSR_SPEC_CTRL
/*
* SPEC_CTRL handling: if the guest's SPEC_CTRL value differs from the
* host's, write the MSR.
*
* IMPORTANT: To avoid RSB underflow attacks and any other nastiness,
* there must not be any returns or indirect branches between this code
* and vmentry.
*/
mov 2*WORD_SIZE(%_ASM_SP), %_ASM_DI
#ifdef CONFIG_X86_64
mov VMX_spec_ctrl(%rdi), %rdx
cmp PER_CPU_VAR(x86_spec_ctrl_current), %rdx
je .Lspec_ctrl_done
movl %edx, %eax
shr $32, %rdx
#else
mov VMX_spec_ctrl(%edi), %eax
mov PER_CPU_VAR(x86_spec_ctrl_current), %ecx
xor %eax, %ecx
mov VMX_spec_ctrl + 4(%edi), %edx
mov PER_CPU_VAR(x86_spec_ctrl_current + 4), %edi
xor %edx, %edi
or %edi, %ecx
je .Lspec_ctrl_done
#endif
mov $MSR_IA32_SPEC_CTRL, %ecx
wrmsr
.Lspec_ctrl_done:
/*
* Since vmentry is serializing on affected CPUs, there's no need for
* an LFENCE to stop speculation from skipping the wrmsr.
*/
/* Load @regs to RAX. */
mov (%_ASM_SP), %_ASM_AX
/* Load guest registers. Don't clobber flags. */
mov VCPU_RCX(%_ASM_AX), %_ASM_CX
mov VCPU_RDX(%_ASM_AX), %_ASM_DX
mov VCPU_RBX(%_ASM_AX), %_ASM_BX
mov VCPU_RBP(%_ASM_AX), %_ASM_BP
mov VCPU_RSI(%_ASM_AX), %_ASM_SI
mov VCPU_RDI(%_ASM_AX), %_ASM_DI
#ifdef CONFIG_X86_64
mov VCPU_R8 (%_ASM_AX), %r8
mov VCPU_R9 (%_ASM_AX), %r9
mov VCPU_R10(%_ASM_AX), %r10
mov VCPU_R11(%_ASM_AX), %r11
mov VCPU_R12(%_ASM_AX), %r12
mov VCPU_R13(%_ASM_AX), %r13
mov VCPU_R14(%_ASM_AX), %r14
mov VCPU_R15(%_ASM_AX), %r15
#endif
/* Load guest RAX. This kills the @regs pointer! */
mov VCPU_RAX(%_ASM_AX), %_ASM_AX
/*
* Note, ALTERNATIVE_2 works in reverse order. If CLEAR_CPU_BUF_VM is
* enabled, do VERW unconditionally. If CPU_BUF_VM_MMIO is enabled,
* check @flags to see if the vCPU has access to host MMIO, and if so,
* do VERW. Else, do nothing (no mitigations needed/enabled).
*/
ALTERNATIVE_2 "", \
__stringify(testl $VMX_RUN_CLEAR_CPU_BUFFERS_FOR_MMIO, WORD_SIZE(%_ASM_SP); \
jz .Lskip_mmio_verw; \
VERW; \
.Lskip_mmio_verw:), \
X86_FEATURE_CLEAR_CPU_BUF_VM_MMIO, \
__stringify(VERW), X86_FEATURE_CLEAR_CPU_BUF_VM
/* Check @flags to see if VMLAUNCH or VMRESUME is needed. */
testl $VMX_RUN_VMRESUME, WORD_SIZE(%_ASM_SP)
jz .Lvmlaunch
/*
* After a successful VMRESUME/VMLAUNCH, control flow "magically"
* resumes below at 'vmx_vmexit' due to the VMCS HOST_RIP setting.
* So this isn't a typical function and objtool needs to be told to
* save the unwind state here and restore it below.
*/
UNWIND_HINT_SAVE
/*
* If VMRESUME/VMLAUNCH and corresponding vmexit succeed, execution resumes at
* the 'vmx_vmexit' label below.
*/
.Lvmresume:
vmresume
jmp .Lvmfail
.Lvmlaunch:
vmlaunch
jmp .Lvmfail
_ASM_EXTABLE(.Lvmresume, .Lfixup)
_ASM_EXTABLE(.Lvmlaunch, .Lfixup)
SYM_INNER_LABEL_ALIGN(vmx_vmexit, SYM_L_GLOBAL)
/* Restore unwind state from before the VMRESUME/VMLAUNCH. */
UNWIND_HINT_RESTORE
ENDBR
/* Temporarily save guest's RAX. */
push %_ASM_AX
/* Reload @regs to RAX. */
mov WORD_SIZE(%_ASM_SP), %_ASM_AX
/* Save all guest registers, including RAX from the stack */
pop VCPU_RAX(%_ASM_AX)
mov %_ASM_CX, VCPU_RCX(%_ASM_AX)
mov %_ASM_DX, VCPU_RDX(%_ASM_AX)
mov %_ASM_BX, VCPU_RBX(%_ASM_AX)
mov %_ASM_BP, VCPU_RBP(%_ASM_AX)
mov %_ASM_SI, VCPU_RSI(%_ASM_AX)
mov %_ASM_DI, VCPU_RDI(%_ASM_AX)
#ifdef CONFIG_X86_64
mov %r8, VCPU_R8 (%_ASM_AX)
mov %r9, VCPU_R9 (%_ASM_AX)
mov %r10, VCPU_R10(%_ASM_AX)
mov %r11, VCPU_R11(%_ASM_AX)
mov %r12, VCPU_R12(%_ASM_AX)
mov %r13, VCPU_R13(%_ASM_AX)
mov %r14, VCPU_R14(%_ASM_AX)
mov %r15, VCPU_R15(%_ASM_AX)
#endif
/* Clear return value to indicate VM-Exit (as opposed to VM-Fail). */
xor %ebx, %ebx
.Lclear_regs:
/* Discard @regs. The register is irrelevant, it just can't be RBX. */
pop %_ASM_AX
/*
* Clear all general purpose registers except RSP and RBX to prevent
* speculative use of the guest's values, even those that are reloaded
* via the stack. In theory, an L1 cache miss when restoring registers
* could lead to speculative execution with the guest's values.
* Zeroing XORs are dirt cheap, i.e. the extra paranoia is essentially
* free. RSP and RBX are exempt as RSP is restored by hardware during
* VM-Exit and RBX is explicitly loaded with 0 or 1 to hold the return
* value.
*/
xor %eax, %eax
xor %ecx, %ecx
xor %edx, %edx
xor %ebp, %ebp
xor %esi, %esi
xor %edi, %edi
#ifdef CONFIG_X86_64
xor %r8d, %r8d
xor %r9d, %r9d
xor %r10d, %r10d
xor %r11d, %r11d
xor %r12d, %r12d
xor %r13d, %r13d
xor %r14d, %r14d
xor %r15d, %r15d
#endif
/*
* IMPORTANT: RSB filling and SPEC_CTRL handling must be done before
* the first unbalanced RET after vmexit!
*
* For retpoline or IBRS, RSB filling is needed to prevent poisoned RSB
* entries and (in some cases) RSB underflow.
*
* eIBRS has its own protection against poisoned RSB, so it doesn't
* need the RSB filling sequence. But it does need to be enabled, and a
* single call to retire, before the first unbalanced RET.
*/
FILL_RETURN_BUFFER %_ASM_CX, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_VMEXIT,\
X86_FEATURE_RSB_VMEXIT_LITE
pop %_ASM_ARG2 /* @flags */
pop %_ASM_ARG1 /* @vmx */
call vmx_spec_ctrl_restore_host
CLEAR_BRANCH_HISTORY_VMEXIT
/* Put return value in AX */
mov %_ASM_BX, %_ASM_AX
pop %_ASM_BX
#ifdef CONFIG_X86_64
pop %r12
pop %r13
pop %r14
pop %r15
#else
pop %esi
pop %edi
#endif
pop %_ASM_BP
RET
.Lfixup:
cmpb $0, _ASM_RIP(virt_rebooting)
jne .Lvmfail
ud2
.Lvmfail:
/* VM-Fail: set return value to 1 */
mov $1, %_ASM_BX
jmp .Lclear_regs
SYM_FUNC_END(__vmx_vcpu_run)
SYM_FUNC_START(vmx_do_nmi_irqoff)
VMX_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx
SYM_FUNC_END(vmx_do_nmi_irqoff)
#ifndef CONFIG_CC_HAS_ASM_GOTO_OUTPUT
/**
* vmread_error_trampoline - Trampoline from inline asm to vmread_error()
* @field: VMCS field encoding that failed
* @fault: %true if the VMREAD faulted, %false if it failed
*
* Save and restore volatile registers across a call to vmread_error(). Note,
* all parameters are passed on the stack.
*/
SYM_FUNC_START(vmread_error_trampoline)
push %_ASM_BP
mov %_ASM_SP, %_ASM_BP
push %_ASM_AX
push %_ASM_CX
push %_ASM_DX
#ifdef CONFIG_X86_64
push %rdi
push %rsi
push %r8
push %r9
push %r10
push %r11
#endif
/* Load @field and @fault to arg1 and arg2 respectively. */
mov 3*WORD_SIZE(%_ASM_BP), %_ASM_ARG2
mov 2*WORD_SIZE(%_ASM_BP), %_ASM_ARG1
call vmread_error_trampoline2
/* Zero out @fault, which will be popped into the result register. */
_ASM_MOV $0, 3*WORD_SIZE(%_ASM_BP)
#ifdef CONFIG_X86_64
pop %r11
pop %r10
pop %r9
pop %r8
pop %rsi
pop %rdi
#endif
pop %_ASM_DX
pop %_ASM_CX
pop %_ASM_AX
pop %_ASM_BP
RET
SYM_FUNC_END(vmread_error_trampoline)
#endif
.section .text, "ax"
#ifndef CONFIG_X86_FRED
SYM_FUNC_START(vmx_do_interrupt_irqoff)
VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1
SYM_FUNC_END(vmx_do_interrupt_irqoff)
#endif
]
|
| 113/4 |
2026/05/18 18:47 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 36561
output: 26
thoughts: 423
Thoughts: **Discrepancies in VM Entry Analysis**
Okay, so I'm looking at this assembly and something's not quite right. The snippet I have doesn't seem to align with the `vmenter.S` code. Specifically, the snippet starts with `ee`, which I know is the last byte of `vmlaunch` ( `0x0f 0x01 0xee` ). But the `vmenter.S` code *after* `vmlaunch` is just a jump, `jmp .Lvmfail`. It uses `eb`, but the snippet has something completely different following the `ee`, specifically a `call` instruction.
Wait, maybe I'm looking at the wrong version, or a totally different file. Or maybe that `ee` isn't related to `vmlaunch` at all. Time to reconsider.
Let's refocus on that `45 85 ed` sequence. That's `test %r13d, %r13d`. I know `r13` is saved and restored in `vmenter.S`, but beyond that, it's not super important. It doesn't seem *that* distinctive.
Wait! I have an idea. Let me search for the whole sequence: `45 85 ed 48 89 5c 24 08 74 2a` in the kernel source. Since I can't search hex directly, I will translate it into its assembly equivalent.
So that's `test %r13d, %r13d` followed by `mov %rbx, 0x8(%rsp)` and then a `je`.
Hang on a second. That `mov %rbx, 0x8(%rsp)` is *really* specific. I'm saving a register, `rbx`, to a particular stack slot. That suggests compiler-generated code for a function that needs to save `rbx` before using it. This is probably a function with local variables, perhaps.
Now, let's try searching the kernel for `mov %rbx, 8(%rsp)` to pinpoint the code location. Let me use `codesearch-find-references` for this task.
|
| 114/4 |
2026/05/18 18:47 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:rbx]
Results: map[References:<nil> TruncatedOutput:false]
|
| 115/4 |
2026/05/18 18:47 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 37517
output: 19
thoughts: 1449
Thoughts: **Dissecting the Assembly: A Kernel Deep Dive**
Okay, so I'm hitting a wall with my initial search terms. Trying to use `rbx` directly isn't working, so let's get creative. I'll pivot to searching for the instruction `lea 24(%rbx), %rdi` instead, and also try the slightly more general `&.*->.*` where the target field is at offset 24.
I need to quickly understand the sequence, so I'll try disassembling this snippet of code by hand.
I've identified these calls:
`call func1`
`call func2`
`call func1`
`call func3`
and noticed that `func1` is called twice with the values `(0, status)` and the second time. And `func2` is called with `(rbx + 24, -1)`.
This looks very strongly like an `atomic_dec_return` call to `func2`, given the negative one used as the second argument. I suspect that `func1` could be something like `preempt_count_sub`. However, `preempt_count_sub` doesn't take 2 arguments, so that feels unlikely.
So, let's look at possibilities. Is there a `spin_unlock_irqrestore` in the mix? That function is typically inlined, which would explain the lack of a direct call to it. The calling convention does look right for something related to tracing. Perhaps a `trace_preempt_off`? That function takes the appropriate number of arguments. I'll search for how that function is defined in a `preempt.h` file, or similar.
A better strategy might be to search for `atomic_dec_return` specifically in the kernel codebase. Then I'll check if those instances are followed by a call to the same function that was previously called. Alternatively, let me search for the common instruction sequence `lea 24(%rbx), %rdi` along with the instruction `mov $-1, %esi`.
Right, that points directly to a function like `atomic_add_return(-1, &obj->field)`, where `obj` is the register `rbx` and `field` is at offset 24 within the struct.
Now the question is, what struct is `rbx` pointing to? I was looking at `struct page` earlier, where `_refcount` is at offset 52. Let's not make that assumption and try to find structs that have fields at offset 24. It may be compiled with options that change the struct layout. Also, it might be a different struct entirely.
Okay, is it `struct kref`? No. Is it `struct rcu_head`? Usually that is at the end. Perhaps `struct list_head`, but that is 16 bytes. Let's go through the possible structs... Ah, if the register `rbx` is pointing at the beginning of a struct, then we're calculating the address of a member at offset 24.
Could `rbx` be a `struct sk_buff`? No.
How about `struct task_struct`? Checking, offset 24 is `__state`.
Is it `struct inode`? I'll check and see if that is a likely candidate.
|
| 116/4 |
2026/05/18 18:47 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:i_mode OffsetBits:0 SizeBits:16] map[Name:i_opflags OffsetBits:16 SizeBits:16] map[Name:i_flags OffsetBits:32 SizeBits:32] map[Name:i_acl OffsetBits:64 SizeBits:64] map[Name:i_default_acl OffsetBits:128 SizeBits:64] map[Name:i_uid OffsetBits:192 SizeBits:32] map[Name:i_gid OffsetBits:224 SizeBits:32] map[Name:i_op OffsetBits:256 SizeBits:64] map[Name:i_sb OffsetBits:320 SizeBits:64] map[Name:i_mapping OffsetBits:384 SizeBits:64] map[Name:i_security OffsetBits:448 SizeBits:64] map[Name:i_ino OffsetBits:512 SizeBits:64] map[Name: OffsetBits:576 SizeBits:32] map[Name:i_rdev OffsetBits:608 SizeBits:32] map[Name:i_size OffsetBits:640 SizeBits:64] map[Name:i_atime_sec OffsetBits:704 SizeBits:64] map[Name:i_mtime_sec OffsetBits:768 SizeBits:64] map[Name:i_ctime_sec OffsetBits:832 SizeBits:64] map[Name:i_atime_nsec OffsetBits:896 SizeBits:32] map[Name:i_mtime_nsec OffsetBits:928 SizeBits:32] map[Name:i_ctime_nsec OffsetBits:960 SizeBits:32] map[Name:i_generation OffsetBits:992 SizeBits:32] map[Name:i_lock OffsetBits:1024 SizeBits:512] map[Name:i_bytes OffsetBits:1536 SizeBits:16] map[Name:i_blkbits OffsetBits:1552 SizeBits:8] map[Name:i_write_hint OffsetBits:1560 SizeBits:8] map[Name:i_blocks OffsetBits:1600 SizeBits:64] map[Name:i_state OffsetBits:1664 SizeBits:32] map[Name:i_rwsem OffsetBits:1728 SizeBits:1152] map[Name:dirtied_when OffsetBits:2880 SizeBits:64] map[Name:dirtied_time_when OffsetBits:2944 SizeBits:64] map[Name:i_hash OffsetBits:3008 SizeBits:128] map[Name:i_io_list OffsetBits:3136 SizeBits:128] map[Name:i_wb OffsetBits:3264 SizeBits:64] map[Name:i_wb_frn_winner OffsetBits:3328 SizeBits:32] map[Name:i_wb_frn_avg_time OffsetBits:3360 SizeBits:16] map[Name:i_wb_frn_history OffsetBits:3376 SizeBits:16] map[Name:i_lru OffsetBits:3392 SizeBits:128] map[Name:i_sb_list OffsetBits:3520 SizeBits:128] map[Name:i_wb_list OffsetBits:3648 SizeBits:128] map[Name: OffsetBits:3776 SizeBits:128] map[Name:i_version OffsetBits:3904 SizeBits:64] map[Name:i_sequence OffsetBits:3968 SizeBits:64] map[Name:i_count OffsetBits:4032 SizeBits:32] map[Name:i_dio_count OffsetBits:4064 SizeBits:32] map[Name:i_writecount OffsetBits:4096 SizeBits:32] map[Name:i_readcount OffsetBits:4128 SizeBits:32] map[Name: OffsetBits:4160 SizeBits:64] map[Name:i_flctx OffsetBits:4224 SizeBits:64] map[Name:i_data OffsetBits:4288 SizeBits:4096] map[Name: OffsetBits:8384 SizeBits:128] map[Name: OffsetBits:8512 SizeBits:64] map[Name:i_fsnotify_mask OffsetBits:8576 SizeBits:32] map[Name:i_fsnotify_marks OffsetBits:8640 SizeBits:64] map[Name:i_private OffsetBits:8704 SizeBits:64]]]
|
| 117/4 |
2026/05/18 18:47 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 40967
output: 19
thoughts: 1861
Thoughts: **Dissecting the Assembly: A Kernel Deep Dive**
Okay, so I'm looking at this snippet of assembly, and something doesn't feel right about how `i_uid` is being handled. I see this `func2` call, and the address manipulation around `i_uid`'s memory location is making me think, "that can't be right".
My first instinct is to try and search for the byte sequence in the kernel source, but since I don't have that option, I need to try some other methods.
I get the idea to look at the assembly, so I start decoding. The `mov %rbx, 0x8(%rsp)` instruction is interesting because it means `rbx` is being saved on the stack. I then start searching for patterns, looking for `r13d` and related assembly instructions, but I get blocked.
Then I get a better handle on the assembly with `lea 24(%rbx), %rdi` and start searching for `&.*->.*` at offset 24.
I see the call to `0x0a21a2ec` and recognize this might be a call to a function in a different section of the kernel, like `.init.text` or a module.
I go back to decoding the assembly more precisely. I see this pattern:
* `call func1`
* `call func2`
* `call func1`
* `call func3`
and that `func2` is being called with `(rbx + 24, -1)`.
This looks like an `atomic_dec_return` for `func2`!
Then the light bulb goes off. It is `atomic_add_return(-1, &obj->field)` where `obj` is `rbx` and `field` is at offset 24.
Then I start thinking about which struct the offset points to. I initially consider `struct page`, but I remember that layout can change based on kernel options or even be a different struct.
I think about `struct kref`, `struct rcu_head`, and `struct list_head`, but I discard them as not matching the memory layout I see. I keep the possibility of the layout changing open.
Then I look at `struct task_struct`, and offset 24 is `__state`.
I check the layout and consider `struct inode` and `struct file`.
|
| 118/4 |
2026/05/18 18:47 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:f_lock OffsetBits:0 SizeBits:512] map[Name:f_mode OffsetBits:512 SizeBits:32] map[Name:f_op OffsetBits:576 SizeBits:64] map[Name:f_mapping OffsetBits:640 SizeBits:64] map[Name:private_data OffsetBits:704 SizeBits:64] map[Name:f_inode OffsetBits:768 SizeBits:64] map[Name:f_flags OffsetBits:832 SizeBits:32] map[Name:f_iocb_flags OffsetBits:864 SizeBits:32] map[Name:f_cred OffsetBits:896 SizeBits:64] map[Name:f_owner OffsetBits:960 SizeBits:64] map[Name: OffsetBits:1024 SizeBits:128] map[Name: OffsetBits:1152 SizeBits:1088] map[Name:f_pos OffsetBits:2240 SizeBits:64] map[Name:f_security OffsetBits:2304 SizeBits:64] map[Name:f_wb_err OffsetBits:2368 SizeBits:32] map[Name:f_sb_err OffsetBits:2400 SizeBits:32] map[Name:f_ep OffsetBits:2432 SizeBits:64] map[Name: OffsetBits:2496 SizeBits:256] map[Name:f_ref OffsetBits:2752 SizeBits:64]]]
|
| 119/4 |
2026/05/18 18:47 |
llm |
codeexpert |
4m
Model:
Tokens:
input: 43799
output: 1241
thoughts: 62915
|
| 120/2 |
2026/05/18 18:52 |
llm |
debugger |
0m
Model:
Tokens:
input: 134813
output: 40
thoughts: 294
Thoughts: **Dissecting This Code Snippet**
Okay, first things first, that initial `ee` byte is throwing me off. Could it be a remnant from a `VMLAUNCH` instruction's tail end? No, hold on, `0F 01 EE` is a `bndstx` or something similar, right? Wait, could that `ee` be the last byte of a `call` instruction?
Alright, moving on. I see `48 8d 7b 18`, which I'm fairly certain is an `lea 0x18(%rbx), %rdi` instruction. And then `be ff ff ff ff`, that's a straightforward `mov $0xffffffff, %esi`. Okay, and `e8 ce a2 21 0a` is a `call` instruction to some address... need to look that up.
Wait a second. I recognize `timerqueue_linked_del`, and that function calls `rb_erase_linked`. Aha, is there a direct `call` to `rb_erase_linked` somewhere? I should check the assembly of `__remove_hrtimer` in my local kernel or, if I had it, via objdump.
Oh, wait a minute, `timerqueue_linked_del` is inline, isn't it? Okay, that changes things. So, `rb_erase_linked` is in `lib/rbtree.c`. Is it exported, though? It's in `lib/rbtree.c`, but maybe it's not inline. Let me check. Nope, `rb_erase_linked` is not inline.
Okay, time to consult `lib/rbtree.c` and figure out what exactly `rb_erase_linked` is doing.
|
| 121/2 |
2026/05/18 18:52 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/rbtree.c IncludeLines:true Name:rb_erase_linked]
Results: map[SourceCode: 449: bool rb_erase_linked(struct rb_node_linked *node, struct rb_root_linked *root)
450: {
451: if (node->prev)
452: node->prev->next = node->next;
453: else
454: root->rb_leftmost = node->next;
455:
456: if (node->next)
457: node->next->prev = node->prev;
458:
459: rb_erase(&node->node, &root->rb_root);
460: RB_CLEAR_LINKED_NODE(node);
461:
462: return !!root->rb_leftmost;
463: }
SourceFile:lib/rbtree.c]
|
| 122/2 |
2026/05/18 18:52 |
llm |
debugger |
0m
Model:
Tokens:
input: 135773
output: 20
thoughts: 332
Thoughts: **Debugging a Kernel Crash: The Rabbit Hole of `rb_erase_linked`**
Okay, so I've got a crash, and it's happening inside `__remove_hrtimer` – specifically at `0x8e/0x3e0`. The trace is pointing me towards `lib/rbtree.c`, which makes me think this might be rbtree related. Let's dig into the `__remove_hrtimer` code. I see a `timerqueue_linked_del` call in there, and that rings a bell... I remember that function ultimately calls `rb_erase_linked` to remove an entry.
The real question is, is `rb_erase_linked` being called directly here? The assembly is the key now. I see a `call 0xa21a2ec`, and I'm betting that's `rb_erase_linked`. Let's confirm with the arguments. Okay, I see `lea 0x18(%rbx), %rdi` and `mov $0xffffffff, %esi`. Okay, the arguments to `rb_erase_linked` are `node` and `root`, respectively. `rdi` is holding the first argument, and it looks like that is `&timer->node` and the timer data itself is in `rbx`. The offset to `node` is `0x18` from the start of the `hrtimer` structure. I'll need to double-check `struct hrtimer` to be sure, but this looks promising. This could be the source of the crash! Time to go back and verify the structure layout.
|
| 123/2 |
2026/05/18 18:52 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:node OffsetBits:0 SizeBits:384] map[Name:base OffsetBits:384 SizeBits:64] map[Name:is_queued OffsetBits:448 SizeBits:8] map[Name:is_rel OffsetBits:456 SizeBits:8] map[Name:is_soft OffsetBits:464 SizeBits:8] map[Name:is_hard OffsetBits:472 SizeBits:8] map[Name:is_lazy OffsetBits:480 SizeBits:8] map[Name:_softexpires OffsetBits:512 SizeBits:64] map[Name:function OffsetBits:576 SizeBits:64]]]
|
| 124/2 |
2026/05/18 18:52 |
llm |
debugger |
0m
Model:
Tokens:
input: 136668
output: 22
thoughts: 104
Thoughts: **Dissecting the Memory Layout**
Okay, so I'm looking at this assembly, and the first thing that jumps out is that `node` member. It's at offset 0, which means `&timer->node` is essentially just `timer` itself, right? That simplifies things a bit. I need to figure out what's happening in this `lea 0x18(%rbx), %rdi` instruction.
Offset 0x18... that's 24 bytes. Now I remember we're dealing with a timerqueue. Hang on... `struct timerqueue_linked_node` has that embedded `rb_node_linked node;` member. Let me double-check the definitions of `struct timerqueue_node` and `struct timerqueue_linked_node`. I need to confirm the exact layout and sizes. I'm suspecting that instruction is grabbing a pointer to something in the timer structure, likely related to its linked node aspect. Need to clarify the relation between these two structs.
|
| 125/2 |
2026/05/18 18:52 |
tool |
codesearch-struct-layout |
0m
Args: map[Name:timerqueue_node]
Results: map[Fields:[map[Name:node OffsetBits:0 SizeBits:192] map[Name:expires OffsetBits:192 SizeBits:64]]]
|
| 126/2 |
2026/05/18 18:52 |
llm |
debugger |
0m
Model:
Tokens:
input: 137065
output: 24
thoughts: 42
Thoughts: Here's my attempt at summarizing those thoughts, tailored for an expert audience and written in the first person:
**Navigating Timer Queue Data Structures**
Okay, let's see... *struct timerqueue_node*. Right, so it starts with a `node` field at offset zero. That's a classic approach to make embedding nodes easier. Size... 24 bytes, or 192 bits. Makes sense; probably includes a few pointers for the queue linkage and some associated timer data. Now, what's this? *struct timerqueue_linked_node*? I wonder why it's using a linked node structure. It will be interesting to see if it is used with the primary timer queue or an auxiliary structure.
|
| 127/2 |
2026/05/18 18:52 |
tool |
codesearch-struct-layout |
0m
Args: map[Name:timerqueue_linked_node]
Results: map[Fields:[map[Name:node OffsetBits:0 SizeBits:320] map[Name:expires OffsetBits:320 SizeBits:64]]]
|
| 128/2 |
2026/05/18 18:52 |
llm |
debugger |
0m
Model:
Tokens:
input: 137337
output: 41
thoughts: 105
Thoughts: **Dissecting the Timer Structures**
Okay, let's break this down. First, I'm taking a look at `struct timerqueue_linked_node`. I see that the `node` field, whatever it's defining, starts at offset zero and is 40 bytes in size, which is quite significant. That gives me an immediate context. Now, I'm immediately trying to see where this all fits in a larger context, and I need to start asking some questions.
Now I'm trying to figure out where that `timerqueue_linked_node` is used within `struct hrtimer`, so, let me see... I need to find what's at offset `0x18` (which is 24 bytes) in the `struct hrtimer`. Okay, it looks like it does.
Hold on, let me double-check this. I was on the right track initially, but I need to make absolutely sure. Is `hrtimer` using a `timerqueue_node` or a `timerqueue_linked_node` within its structure? Better to confirm my original assumption and check `struct hrtimer` again to make sure I'm precise with my thinking.
|
| 129/2 |
2026/05/18 18:52 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/linux/hrtimer_defs.h IncludeLines:true Name:hrtimer]
Results: map[SourceCode: 18: /**
19: * struct hrtimer - the basic hrtimer structure
20: * @node: Linked timerqueue node, which also manages node.expires,
21: * the absolute expiry time in the hrtimers internal
22: * representation. The time is related to the clock on
23: * which the timer is based. Is setup by adding
24: * slack to the _softexpires value. For non range timers
25: * identical to _softexpires.
26: * @_softexpires: the absolute earliest expiry time of the hrtimer.
27: * The time which was given as expiry time when the timer
28: * was armed.
29: * @function: timer expiry callback function
30: * @base: pointer to the timer base (per cpu and per clock)
31: * @is_queued: Indicates whether a timer is enqueued or not
32: * @is_rel: Set if the timer was armed relative
33: * @is_soft: Set if hrtimer will be expired in soft interrupt context.
34: * @is_hard: Set if hrtimer will be expired in hard interrupt context
35: * even on RT.
36: * @is_lazy: Set if the timer is frequently rearmed to avoid updates
37: * of the clock event device
38: *
39: * The hrtimer structure must be initialized by hrtimer_setup()
40: */
41: struct hrtimer {
42: struct timerqueue_linked_node node;
43: struct hrtimer_clock_base *base;
44: bool is_queued;
45: bool is_rel;
46: bool is_soft;
47: bool is_hard;
48: bool is_lazy;
49: ktime_t _softexpires;
50: enum hrtimer_restart (*__private function)(struct hrtimer *);
51: };
SourceFile:include/linux/hrtimer_types.h]
|
| 130/2 |
2026/05/18 18:52 |
llm |
debugger |
1m
Model:
Tokens:
input: 138333
output: 23
thoughts: 38
Thoughts: Here's my thought process, as if I were considering this code:
**Diving into the Timer Queue Structure**
Okay, so I'm looking at this `struct timerqueue_linked_node`, and my initial thought is, "Aha! This thing has a `node` member, and it's of type `struct rb_node_linked`." My spider senses are tingling. Given the names, I'm guessing this might be related to a red-black tree implementation or something similar. Time to dig deeper and confirm that suspicion. Let's see what `struct rb_node_linked` *actually* is. I need to understand how this `node` fits into the overall timer queue structure and how its `rb_node_linked` is used. This is likely the key to understanding how these timers are organized and managed. I'm expecting this to be the backbone of efficient insertion, deletion, and searching of timers within this queue.
|
| 131/2 |
2026/05/18 18:53 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:node OffsetBits:0 SizeBits:192] map[Name:prev OffsetBits:192 SizeBits:64] map[Name:next OffsetBits:256 SizeBits:64]]]
|
| 132/2 |
2026/05/18 18:53 |
llm |
debugger |
0m
Model:
Tokens:
input: 138667
output: 24
thoughts: 625
Thoughts: **Dissecting a Kernel Crash: From Assembly to Lockdep**
Okay, let's break this down. I'm looking at a crash that's pointing to `__remove_hrtimer` in the kernel. The crash dump gives me a RIP at `kernel/time/hrtimer.c:1145`. I see some assembly related to memory access and calling a function, so my immediate thought is to trace the execution flow and try to see where things are failing.
First, I note the structure `rb_node_linked` and its members, realizing the offset math for `prev` and `next`. I was initially confused by the `lea` instruction, assuming it was accessing the `prev` member. I quickly corrected that thought: It's getting the *address* of something. I recall `timerqueue_linked_prev` function, which, I remember, uses `rb_entry_safe` to get the address.
I started down a path that I quickly realized was wrong. Then, I step back and examine the assembly again, noticing a `call` instruction and a parameter setup, `rdi` and `esi`. I remember that if it was calling `rb_erase_linked`, then it would call a function directly; this isn't `__remove_hrtimer`. This changes my train of thought.
My thoughts drifted toward a possible BPF JIT issue since I was looking at a call to an unknown function. But the crash dump clearly says the RIP is in `__remove_hrtimer`. So, let's step back and consider the possibility that something is being missed about the function call.
I then check `__remove_hrtimer` at the line number in the backtrace. Line 1145 is `lockdep_assert_held(&cpu_base->lock)`. I remember that this is a lock debugging assertion and calls `lock_is_held` function, which takes a `struct lockdep_map *`. Okay, I think I'm on the right track; let's check `struct hrtimer_cpu_base` to understand how the lock is used and see if there's a problem with the lock.
|
| 133/2 |
2026/05/18 18:54 |
tool |
codesearch-struct-layout |
0m
Args: map[Name:hrtimer_cpu_base]
Results: map[Fields:[map[Name:lock OffsetBits:0 SizeBits:512] map[Name:cpu OffsetBits:512 SizeBits:32] map[Name:active_bases OffsetBits:544 SizeBits:32] map[Name:clock_was_set_seq OffsetBits:576 SizeBits:32] map[Name:hres_active OffsetBits:608 SizeBits:8] map[Name:deferred_rearm OffsetBits:616 SizeBits:8] map[Name:deferred_needs_update OffsetBits:624 SizeBits:8] map[Name:hang_detected OffsetBits:632 SizeBits:8] map[Name:softirq_activated OffsetBits:640 SizeBits:8] map[Name:online OffsetBits:648 SizeBits:8] map[Name:nr_events OffsetBits:672 SizeBits:32] map[Name:nr_retries OffsetBits:704 SizeBits:16] map[Name:nr_hangs OffsetBits:720 SizeBits:16] map[Name:max_hang_time OffsetBits:736 SizeBits:32] map[Name:expires_next OffsetBits:768 SizeBits:64] map[Name:next_timer OffsetBits:832 SizeBits:64] map[Name:softirq_expires_next OffsetBits:896 SizeBits:64] map[Name:softirq_next_timer OffsetBits:960 SizeBits:64] map[Name:deferred_expires_next OffsetBits:1024 SizeBits:64] map[Name:clock_base OffsetBits:1536 SizeBits:8192] map[Name:csd OffsetBits:9728 SizeBits:256]]]
|
| 134/2 |
2026/05/18 18:54 |
llm |
debugger |
0m
Model:
Tokens:
input: 140251
output: 21
thoughts: 62
Thoughts: **Verifying Lock Structure and Offset**
Okay, first things first, the `lock` variable - that's sitting right at offset 0. No surprises there. It's defined as a `raw_spinlock_t`. Now, the important bit. Inside this `raw_spinlock_t`, I *know* there's a `struct lockdep_map dep_map` involved, and I'm pretty sure it's at offset 24, which is `0x18` in hex. But I need to confirm this absolutely. Let's double-check the structure to make sure that offset is correct. Can't trust my memory for something this critical.
|
| 135/2 |
2026/05/18 18:54 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:raw_lock OffsetBits:0 SizeBits:32] map[Name:magic OffsetBits:32 SizeBits:32] map[Name:owner_cpu OffsetBits:64 SizeBits:32] map[Name:owner OffsetBits:128 SizeBits:64] map[Name:dep_map OffsetBits:192 SizeBits:320]]]
|
| 136/2 |
2026/05/18 18:54 |
llm |
debugger |
0m
Model:
Tokens:
input: 140591
output: 45
thoughts: 1298
Thoughts: **Analyzing a Multi-CPU Stall in the Kernel**
Okay, let's break this down. First, I see an assembly snippet. Ah! The offset `0x18` in `lea 0x18(%rbx), %rdi` corresponds to `cpu_base->lock.dep_map`, which makes sense. The code's using `lock_is_held` to check a lock, with `-1` as the argument. That looks like a call to `lockdep_assert_held`. And, I know that `lockdep_assert_held` is a `WARN_ON` based on whether `lockdep_is_held` returns true or false.
Now, I understand the relevant code segment: It appears to test if a lock is currently held, and if it's not, it will jump to `WARN_ON`. I'm concerned about a multi-CPU stall, and this code segment doesn't look like it would loop; it either continues or it hits a warning.
The core issue seems to be CPU 1 hanging in `__remove_hrtimer`. The crash dump shows the RIP at `__remove_hrtimer+0x8e`, which lines up with the instruction `test r13d, r13d`. This means the NMI likely interrupted CPU 1 at that instruction! It seems like CPU 1 _wasn't_ stuck in `__remove_hrtimer` in the primary sense, but was _interrupted_ there by the NMI.
Then, from the NMI backtrace, CPU 1 was in `default_idle` when it received a timer interrupt. That interrupt triggered a sequence: `sysvec_apic_timer_interrupt` -> `hrtimer_interrupt` -> `__hrtimer_run_queues` -> `__run_hrtimer` -> `__remove_hrtimer`. So, now CPU 1 is handling the timer interrupt within `__remove_hrtimer`.
However, the problem is that CPU 0 is stuck in `csd_lock_wait` waiting for an IPI from CPU 1. If CPU 1 is in `default_idle` and got a timer interrupt, it now has IRQs disabled, so it *cannot* process the IPI from CPU 0. CPU 1 is stuck in the timer interrupt handler because it cannot receive interrupts, and CPU 0 is waiting on the IPI that will never come. But what specifically in the interrupt handler is causing CPU 1 to get stuck? Is it looping in `__hrtimer_run_queues`? I'll need to check the code in `__hrtimer_run_queues`.
|
| 137/2 |
2026/05/18 18:54 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/time/hrtimer.c IncludeLines:true Name:__hrtimer_run_queues]
Results: map[SourceCode:1968: static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now,
1969: unsigned long flags, unsigned int active_mask)
1970: {
1971: unsigned int active = cpu_base->active_bases & active_mask;
1972: struct hrtimer_clock_base *base;
1973:
1974: for_each_active_base(base, cpu_base, active) {
1975: ktime_t basenow = ktime_add(now, base->offset);
1976: struct hrtimer *timer;
1977:
1978: while ((timer = clock_base_next_timer(base))) {
1979: /*
1980: * The immediate goal for using the softexpires is
1981: * minimizing wakeups, not running timers at the
1982: * earliest interrupt after their soft expiration.
1983: * This allows us to avoid using a Priority Search
1984: * Tree, which can answer a stabbing query for
1985: * overlapping intervals and instead use the simple
1986: * BST we already have.
1987: * We don't add extra wakeups by delaying timers that
1988: * are right-of a not yet expired timer, because that
1989: * timer will have to trigger a wakeup anyway.
1990: */
1991: if (basenow < hrtimer_get_softexpires(timer))
1992: break;
1993:
1994: __run_hrtimer(cpu_base, base, timer, basenow, flags);
1995: if (active_mask == HRTIMER_ACTIVE_SOFT)
1996: hrtimer_sync_wait_running(cpu_base, flags);
1997: }
1998: }
1999: }
SourceFile:kernel/time/hrtimer.c]
|
| 138/2 |
2026/05/18 18:54 |
llm |
debugger |
0m
Model:
Tokens:
input: 143060
output: 42
thoughts: 1283
Thoughts: **Analyzing a Potential Kernel Timer Infinite Loop**
Okay, here's what I'm thinking. I'm seeing that `__hrtimer_run_queues` seems to have a vulnerability. It loops as long as the timer's soft expiry time is in the past, meaning if `__run_hrtimer` doesn't remove the timer or it's re-enqueued with a past expiration time, we're looking at an infinite loop, a classic denial-of-service vector.
The key is in `__run_hrtimer`: It calls a function pointer, `fn`, and then checks its return value. If `fn` consistently returns `HRTIMER_RESTART` *without* advancing the timer, the loop condition persists, leading to an infinite loop within `__hrtimer_run_queues`. So, what timer is it? What's the context?
Looking at the reproducer, it's setting up `sendmsg` with `RTM_NEWQDISC`. Okay, so we're creating a new queueing discipline, a qdisc. My mind immediately jumps to "is there a qdisc that has a timer in it that could be buggy?" That's a definite possibility.
The reproducer also involves loading a BPF program. But wait, it doesn't *attach* the BPF program, as far as I can see! It's loading it, which *could* be using `BPF_PROG_TYPE_STRUCT_OPS` or something similar to get it attached. But no, the reproducer sets the type to `BPF_PROG_TYPE_TRACEPOINT`.
The reproducer is doing `BPF_PROG_LOAD`, command 5. Does the load function execute the BPF program? No, not directly.
But, the BPF program uses `bpf_snprintf`. However, it may be irrelevant. What if the BPF program *isn't* the root cause?
Let's step back. What if the `RTM_NEWQDISC` itself creates a qdisc that has a built-in timer flaw? Let's analyze the qdisc that's being created.
Inspecting the `netlink_msg_route_sched`, I see `attrs` array is empty. This is interesting. If `attrs` is empty and `TCA_KIND` isn't specified, `qdisc_create` should fail, shouldn't it? Let me confirm that by checking `qdisc_create` in `net/sched/sch_api.c`.
|
| 139/2 |
2026/05/18 18:54 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:net/sched/sch_api.c IncludeLines:true Name:qdisc_create]
Results: map[SourceCode:1244: static struct Qdisc *qdisc_create(struct net_device *dev,
1245: struct netdev_queue *dev_queue,
1246: u32 parent, u32 handle,
1247: struct nlattr **tca, int *errp,
1248: struct netlink_ext_ack *extack)
1249: {
1250: int err;
1251: struct nlattr *kind = tca[TCA_KIND];
1252: struct Qdisc *sch;
1253: struct Qdisc_ops *ops;
1254: struct qdisc_size_table *stab;
1255:
1256: ops = qdisc_lookup_ops(kind);
1257: if (!ops) {
1258: err = -ENOENT;
1259: NL_SET_ERR_MSG(extack, "Specified qdisc kind is unknown");
1260: goto err_out;
1261: }
1262:
1263: sch = qdisc_alloc(dev_queue, ops, extack);
1264: if (IS_ERR(sch)) {
1265: err = PTR_ERR(sch);
1266: goto err_out2;
1267: }
1268:
1269: sch->parent = parent;
1270:
1271: if (handle == TC_H_INGRESS) {
1272: if (!(sch->flags & TCQ_F_INGRESS)) {
1273: NL_SET_ERR_MSG(extack,
1274: "Specified parent ID is reserved for ingress and clsact Qdiscs");
1275: err = -EINVAL;
1276: goto err_out3;
1277: }
1278: handle = TC_H_MAKE(TC_H_INGRESS, 0);
1279: } else {
1280: if (handle == 0) {
1281: handle = qdisc_alloc_handle(dev);
1282: if (handle == 0) {
1283: NL_SET_ERR_MSG(extack, "Maximum number of qdisc handles was exceeded");
1284: err = -ENOSPC;
1285: goto err_out3;
1286: }
1287: }
1288: if (!netif_is_multiqueue(dev))
1289: sch->flags |= TCQ_F_ONETXQUEUE;
1290: }
1291:
1292: sch->handle = handle;
1293:
1294: /* This exist to keep backward compatible with a userspace
1295: * loophole, what allowed userspace to get IFF_NO_QUEUE
1296: * facility on older kernels by setting tx_queue_len=0 (prior
1297: * to qdisc init), and then forgot to reinit tx_queue_len
1298: * before again attaching a qdisc.
1299: */
1300: if ((dev->priv_flags & IFF_NO_QUEUE) && (dev->tx_queue_len == 0)) {
1301: WRITE_ONCE(dev->tx_queue_len, DEFAULT_TX_QUEUE_LEN);
1302: netdev_info(dev, "Caught tx_queue_len zero misconfig\n");
1303: }
1304:
1305: err = qdisc_block_indexes_set(sch, tca, extack);
1306: if (err)
1307: goto err_out3;
1308:
1309: if (tca[TCA_STAB]) {
1310: stab = qdisc_get_stab(tca[TCA_STAB], extack);
1311: if (IS_ERR(stab)) {
1312: err = PTR_ERR(stab);
1313: goto err_out3;
1314: }
1315: rcu_assign_pointer(sch->stab, stab);
1316: }
1317:
1318: if (ops->init) {
1319: err = ops->init(sch, tca[TCA_OPTIONS], extack);
1320: if (err != 0)
1321: goto err_out4;
1322: }
1323:
1324: if (tca[TCA_RATE]) {
1325: err = -EOPNOTSUPP;
1326: if (sch->flags & TCQ_F_MQROOT) {
1327: NL_SET_ERR_MSG(extack, "Cannot attach rate estimator to a multi-queue root qdisc");
1328: goto err_out4;
1329: }
1330:
1331: err = gen_new_estimator(&sch->bstats,
1332: sch->cpu_bstats,
1333: &sch->rate_est,
1334: NULL,
1335: true,
1336: tca[TCA_RATE]);
1337: if (err) {
1338: NL_SET_ERR_MSG(extack, "Failed to generate new estimator");
1339: goto err_out4;
1340: }
1341: }
1342:
1343: qdisc_hash_add(sch, false);
1344: trace_qdisc_create(ops, dev, parent);
1345:
1346: return sch;
1347:
1348: err_out4:
1349: /* Even if ops->init() failed, we call ops->destroy()
1350: * like qdisc_create_dflt().
1351: */
1352: if (ops->destroy)
1353: ops->destroy(sch);
1354: qdisc_put_stab(rtnl_dereference(sch->stab));
1355: err_out3:
1356: qdisc_lock_uninit(sch, ops);
1357: netdev_put(dev, &sch->dev_tracker);
1358: qdisc_free(sch);
1359: err_out2:
1360: bpf_module_put(ops, ops->owner);
1361: err_out:
1362: *errp = err;
1363: return NULL;
1364: }
SourceFile:net/sched/sch_api.c]
|
| 140/2 |
2026/05/18 18:54 |
llm |
debugger |
0m
Model:
Tokens:
input: 146774
output: 44
thoughts: 67
Thoughts: Here's my summary, as though I'm thinking through this process:
**Diving into the Qdisc Lookup**
Alright, let's break this down. We're looking at a function call: `qdisc_lookup_ops(kind)`. Okay, and the input `kind` is actually `tca[TCA_KIND]`. My initial thought is: if this `tca[TCA_KIND]` is pointing to nothing—if it's NULL—then wouldn't `qdisc_lookup_ops` likely return NULL as well? Seems logical. But, I don't want to rely on assumptions.
Best to confirm this behavior and understand how it handles a NULL `kind` input. I'm going to have to check the implementation of `qdisc_lookup_ops` itself. That's the only way to be completely certain.
|
| 141/2 |
2026/05/18 18:55 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:net/sched/sch_api.c IncludeLines:true Name:qdisc_lookup_ops]
Results: map[SourceCode: 360: static struct Qdisc_ops *qdisc_lookup_ops(struct nlattr *kind)
361: {
362: struct Qdisc_ops *q = NULL;
363:
364: if (kind) {
365: read_lock(&qdisc_mod_lock);
366: for (q = qdisc_base; q; q = q->next) {
367: if (nla_strcmp(kind, q->id) == 0) {
368: if (!bpf_try_module_get(q, q->owner))
369: q = NULL;
370: break;
371: }
372: }
373: read_unlock(&qdisc_mod_lock);
374: }
375: return q;
376: }
SourceFile:net/sched/sch_api.c]
|
| 142/2 |
2026/05/18 18:55 |
llm |
debugger |
0m
Model:
Tokens:
input: 147349
output: 43
thoughts: 1046
Thoughts: **Analyzing the Kernel Crash: A Deep Dive into Netlink, Qdiscs, and BPF Programs**
Okay, let's break this down. My initial thought is that if the `kind` field is NULL, `qdisc_lookup_ops` returns NULL. Consequently, `qdisc_create` returns NULL with `-ENOENT`. Then, I look at the reproducer, and it defines a `netlink_msg_t` structure with an empty `attrs` array. Thus, `tca[TCA_KIND]` becomes NULL, causing `qdisc_create` to fail. Since `tc_modify_qdisc` calls `qdisc_create`, I expect an error to be returned from the `RTM_NEWQDISC` message. So, why the crash?
`RTM_NEWQDISC` is 36 (0x24). The reproducer does a `syscall(__NR_sendmsg, ...)` sending a netlink message. Is there any tracepoint that is triggered by `RTM_NEWQDISC`? Yes, `trace_qdisc_create` is called in `qdisc_create`. But `qdisc_create` fails before calling `trace_qdisc_create` because `ops` is NULL! Let's think, what other tracepoints are triggered? Perhaps `kfree_skb`?
Now, the BPF program... it's loaded as `BPF_PROG_TYPE_TRACEPOINT`. However, it's NOT attached to any tracepoint. Wait, `bpf_prog_load` *does not* attach the program. The reproducer also sets `flags = BPF_F_SLEEPABLE` in the `bpf_prog_load` call.
This is where things get interesting. `BPF_PROG_TYPE_TRACEPOINT` is used along with `BPF_F_SLEEPABLE`. Is that permitted? I'll need to double-check `bpf_prog_load` in `kernel/bpf/syscall.c` to understand the implications of this combination. The fact that `flags = 2` is set to `BPF_F_SLEEPABLE` with `BPF_PROG_TYPE_TRACEPOINT` definitely warrants further investigation. This smells like a subtle bug, possibly a race condition or an uninitialized state, triggered by the unusual combination of these BPF flags and a failed Qdisc creation.
|
| 143/2 |
2026/05/18 18:55 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/syscall.c IncludeLines:true Name:bpf_prog_load]
Results: map[SourceCode:2864: static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
2865: {
2866: enum bpf_prog_type type = attr->prog_type;
2867: struct bpf_prog *prog, *dst_prog = NULL;
2868: struct btf *attach_btf = NULL;
2869: struct bpf_token *token = NULL;
2870: bool bpf_cap;
2871: int err;
2872: char license[128];
2873:
2874: if (CHECK_ATTR(BPF_PROG_LOAD))
2875: return -EINVAL;
2876:
2877: if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT |
2878: BPF_F_ANY_ALIGNMENT |
2879: BPF_F_TEST_STATE_FREQ |
2880: BPF_F_SLEEPABLE |
2881: BPF_F_TEST_RND_HI32 |
2882: BPF_F_XDP_HAS_FRAGS |
2883: BPF_F_XDP_DEV_BOUND_ONLY |
2884: BPF_F_TEST_REG_INVARIANTS |
2885: BPF_F_TOKEN_FD))
2886: return -EINVAL;
2887:
2888: bpf_prog_load_fixup_attach_type(attr);
2889:
2890: if (attr->prog_flags & BPF_F_TOKEN_FD) {
2891: token = bpf_token_get_from_fd(attr->prog_token_fd);
2892: if (IS_ERR(token))
2893: return PTR_ERR(token);
2894: /* if current token doesn't grant prog loading permissions,
2895: * then we can't use this token, so ignore it and rely on
2896: * system-wide capabilities checks
2897: */
2898: if (!bpf_token_allow_cmd(token, BPF_PROG_LOAD) ||
2899: !bpf_token_allow_prog_type(token, attr->prog_type,
2900: attr->expected_attach_type)) {
2901: bpf_token_put(token);
2902: token = NULL;
2903: }
2904: }
2905:
2906: bpf_cap = bpf_token_capable(token, CAP_BPF);
2907: err = -EPERM;
2908:
2909: if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
2910: (attr->prog_flags & BPF_F_ANY_ALIGNMENT) &&
2911: !bpf_cap)
2912: goto put_token;
2913:
2914: /* Intent here is for unprivileged_bpf_disabled to block BPF program
2915: * creation for unprivileged users; other actions depend
2916: * on fd availability and access to bpffs, so are dependent on
2917: * object creation success. Even with unprivileged BPF disabled,
2918: * capability checks are still carried out for these
2919: * and other operations.
2920: */
2921: if (sysctl_unprivileged_bpf_disabled && !bpf_cap)
2922: goto put_token;
2923:
2924: if (attr->insn_cnt == 0 ||
2925: attr->insn_cnt > (bpf_cap ? BPF_COMPLEXITY_LIMIT_INSNS : BPF_MAXINSNS)) {
2926: err = -E2BIG;
2927: goto put_token;
2928: }
2929: if (type != BPF_PROG_TYPE_SOCKET_FILTER &&
2930: type != BPF_PROG_TYPE_CGROUP_SKB &&
2931: !bpf_cap)
2932: goto put_token;
2933:
2934: if (is_net_admin_prog_type(type) && !bpf_token_capable(token, CAP_NET_ADMIN))
2935: goto put_token;
2936: if (is_perfmon_prog_type(type) && !bpf_token_capable(token, CAP_PERFMON))
2937: goto put_token;
2938:
2939: /* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog
2940: * or btf, we need to check which one it is
2941: */
2942: if (attr->attach_prog_fd) {
2943: dst_prog = bpf_prog_get(attr->attach_prog_fd);
2944: if (IS_ERR(dst_prog)) {
2945: dst_prog = NULL;
2946: attach_btf = btf_get_by_fd(attr->attach_btf_obj_fd);
2947: if (IS_ERR(attach_btf)) {
2948: err = -EINVAL;
2949: goto put_token;
2950: }
2951: if (!btf_is_kernel(attach_btf)) {
2952: /* attaching through specifying bpf_prog's BTF
2953: * objects directly might be supported eventually
2954: */
2955: btf_put(attach_btf);
2956: err = -ENOTSUPP;
2957: goto put_token;
2958: }
2959: }
2960: } else if (attr->attach_btf_id) {
2961: /* fall back to vmlinux BTF, if BTF type ID is specified */
2962: attach_btf = bpf_get_btf_vmlinux();
2963: if (IS_ERR(attach_btf)) {
2964: err = PTR_ERR(attach_btf);
2965: goto put_token;
2966: }
2967: if (!attach_btf) {
2968: err = -EINVAL;
2969: goto put_token;
2970: }
2971: btf_get(attach_btf);
2972: }
2973:
2974: if (bpf_prog_load_check_attach(type, attr->expected_attach_type,
2975: attach_btf, attr->attach_btf_id,
2976: dst_prog)) {
2977: if (dst_prog)
2978: bpf_prog_put(dst_prog);
2979: if (attach_btf)
2980: btf_put(attach_btf);
2981: err = -EINVAL;
2982: goto put_token;
2983: }
2984:
2985: /* plain bpf_prog allocation */
2986: prog = bpf_prog_alloc(bpf_prog_size(attr->insn_cnt), GFP_USER);
2987: if (!prog) {
2988: if (dst_prog)
2989: bpf_prog_put(dst_prog);
2990: if (attach_btf)
2991: btf_put(attach_btf);
2992: err = -EINVAL;
2993: goto put_token;
2994: }
2995:
2996: prog->expected_attach_type = attr->expected_attach_type;
2997: prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE);
2998: prog->aux->attach_btf = attach_btf;
2999: prog->aux->attach_btf_id = attr->attach_btf_id;
3000: prog->aux->dst_prog = dst_prog;
3001: prog->aux->dev_bound = !!attr->prog_ifindex;
3002: prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS;
3003:
3004: /* move token into prog->aux, reuse taken refcnt */
3005: prog->aux->token = token;
3006: token = NULL;
3007:
3008: prog->aux->user = get_current_user();
3009: prog->len = attr->insn_cnt;
3010:
3011: err = -EFAULT;
3012: if (copy_from_bpfptr(prog->insns,
3013: make_bpfptr(attr->insns, uattr.is_kernel),
3014: bpf_prog_insn_size(prog)) != 0)
3015: goto free_prog;
3016: /* copy eBPF program license from user space */
3017: if (strncpy_from_bpfptr(license,
3018: make_bpfptr(attr->license, uattr.is_kernel),
3019: sizeof(license) - 1) < 0)
3020: goto free_prog;
3021: license[sizeof(license) - 1] = 0;
3022:
3023: /* eBPF programs must be GPL compatible to use GPL-ed functions */
3024: prog->gpl_compatible = license_is_gpl_compatible(license) ? 1 : 0;
3025:
3026: if (attr->signature) {
3027: err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel);
3028: if (err)
3029: goto free_prog;
3030: }
3031:
3032: prog->orig_prog = NULL;
3033: prog->jited = 0;
3034:
3035: atomic64_set(&prog->aux->refcnt, 1);
3036:
3037: if (bpf_prog_is_dev_bound(prog->aux)) {
3038: err = bpf_prog_dev_bound_init(prog, attr);
3039: if (err)
3040: goto free_prog;
3041: }
3042:
3043: if (type == BPF_PROG_TYPE_EXT && dst_prog &&
3044: bpf_prog_is_dev_bound(dst_prog->aux)) {
3045: err = bpf_prog_dev_bound_inherit(prog, dst_prog);
3046: if (err)
3047: goto free_prog;
3048: }
3049:
3050: /*
3051: * Bookkeeping for managing the program attachment chain.
3052: *
3053: * It might be tempting to set attach_tracing_prog flag at the attachment
3054: * time, but this will not prevent from loading bunch of tracing prog
3055: * first, then attach them one to another.
3056: *
3057: * The flag attach_tracing_prog is set for the whole program lifecycle, and
3058: * doesn't have to be cleared in bpf_tracing_link_release, since tracing
3059: * programs cannot change attachment target.
3060: */
3061: if (type == BPF_PROG_TYPE_TRACING && dst_prog &&
3062: dst_prog->type == BPF_PROG_TYPE_TRACING) {
3063: prog->aux->attach_tracing_prog = true;
3064: }
3065:
3066: /* find program type: socket_filter vs tracing_filter */
3067: err = find_prog_type(type, prog);
3068: if (err < 0)
3069: goto free_prog;
3070:
3071: prog->aux->load_time = ktime_get_boottime_ns();
3072: err = bpf_obj_name_cpy(prog->aux->name, attr->prog_name,
3073: sizeof(attr->prog_name));
3074: if (err < 0)
3075: goto free_prog;
3076:
3077: err = security_bpf_prog_load(prog, attr, token, uattr.is_kernel);
3078: if (err)
3079: goto free_prog_sec;
3080:
3081: /* run eBPF verifier */
3082: err = bpf_check(&prog, attr, uattr, uattr_size);
3083: if (err < 0)
3084: goto free_used_maps;
3085:
3086: err = bpf_prog_mark_insn_arrays_ready(prog);
3087: if (err < 0)
3088: goto free_used_maps;
3089:
3090: err = bpf_prog_alloc_id(prog);
3091: if (err)
3092: goto free_used_maps;
3093:
3094: /* Upon success of bpf_prog_alloc_id(), the BPF prog is
3095: * effectively publicly exposed. However, retrieving via
3096: * bpf_prog_get_fd_by_id() will take another reference,
3097: * therefore it cannot be gone underneath us.
3098: *
3099: * Only for the time /after/ successful bpf_prog_new_fd()
3100: * and before returning to userspace, we might just hold
3101: * one reference and any parallel close on that fd could
3102: * rip everything out. Hence, below notifications must
3103: * happen before bpf_prog_new_fd().
3104: *
3105: * Also, any failure handling from this point onwards must
3106: * be using bpf_prog_put() given the program is exposed.
3107: */
3108: bpf_prog_kallsyms_add(prog);
3109: perf_event_bpf_event(prog, PERF_BPF_EVENT_PROG_LOAD, 0);
3110: bpf_audit_prog(prog, BPF_AUDIT_LOAD);
3111:
3112: err = bpf_prog_new_fd(prog);
3113: if (err < 0)
3114: bpf_prog_put(prog);
3115: return err;
3116:
3117: free_used_maps:
3118: /* In case we have subprogs, we need to wait for a grace
3119: * period before we can tear down JIT memory since symbols
3120: * are already exposed under kallsyms.
3121: */
3122: __bpf_prog_put_noref(prog, prog->aux->real_func_cnt);
3123: return err;
3124:
3125: free_prog_sec:
3126: security_bpf_prog_free(prog);
3127: free_prog:
3128: free_uid(prog->aux->user);
3129: if (prog->aux->attach_btf)
3130: btf_put(prog->aux->attach_btf);
3131: bpf_prog_free(prog);
3132: put_token:
3133: bpf_token_put(token);
3134: return err;
3135: }
SourceFile:kernel/bpf/syscall.c]
|
| 144/2 |
2026/05/18 18:55 |
llm |
debugger |
0m
Model:
Tokens:
input: 153449
output: 44
thoughts: 638
Thoughts: **Analyzing the BPF Program and QDisc Interactions**
Okay, let's break this down. First, `bpf_prog_load` is the entry point, and it calls `bpf_check`. Within `bpf_check` we have `bpf_prog_mark_insn_arrays_ready`, then `bpf_prog_alloc_id`, and then `bpf_prog_kallsyms_add`. Finally, a `perf_event_bpf_event` is generated for the load. That perf event doesn't execute the program, just logs the load. The `bpf_check` itself calls `do_check`, which is static analysis. So, it appears the BPF program isn't executed during the load process.
Now, I need to figure out how it *could* be executed. Could it be triggered by `RTM_NEWQDISC` and attached to a tracepoint? But the reproducer *doesn't* attach the BPF program, so that's out. I see that `bpf_prog_load` returns `fd_bpf_prog`. However, the reproducer's system call: `syscall(__NR_bpf, /*cmd=*/5ul, /*arg=*/0x200000000c40ul, /*size=*/0x90ul)` doesn't even save that file descriptor! This means the BPF program *shouldn't* be executed at all.
This means the BPF program isn't the cause of the infinite loop. Then, could `RTM_NEWQDISC` itself be the problem? Looking at the `RTM_NEWQDISC` parameters, the `ifindex` is `ifindex_team` (for `team0`). The `tcm_parent` is 0, `tcm_handle` is 0, and the `tca` is empty. The `qdisc_create` call will fail with `-ENOENT` because `tca[TCA_KIND]` is NULL.
Let me think. With the empty `tca`, `tc_modify_qdisc` might update an existing qdisc? The `tcm_handle` is 0, so `qdisc_lookup(dev, 0)` is returning the root qdisc! It seems like there's an issue with the root qdisc. I should check `tc_modify_qdisc` in `net/sched/sch_api.c`.
|
| 145/2 |
2026/05/18 18:55 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:net/sched/sch_api.c IncludeLines:true Name:tc_modify_qdisc]
Results: map[SourceCode:1792: /*
1793: * Create/change qdisc.
1794: */
1795: static int tc_modify_qdisc(struct sk_buff *skb, struct nlmsghdr *n,
1796: struct netlink_ext_ack *extack)
1797: {
1798: struct net *net = sock_net(skb->sk);
1799: struct nlattr *tca[TCA_MAX + 1];
1800: struct net_device *dev;
1801: struct tcmsg *tcm;
1802: int err;
1803:
1804: err = nlmsg_parse_deprecated(n, sizeof(*tcm), tca, TCA_MAX,
1805: rtm_tca_policy, extack);
1806: if (err < 0)
1807: return err;
1808:
1809: request_qdisc_module(tca[TCA_KIND]);
1810:
1811: tcm = nlmsg_data(n);
1812: dev = __dev_get_by_index(net, tcm->tcm_ifindex);
1813: if (!dev)
1814: return -ENODEV;
1815:
1816: netdev_lock_ops(dev);
1817: err = __tc_modify_qdisc(skb, n, extack, dev, tca, tcm);
1818: netdev_unlock_ops(dev);
1819:
1820: return err;
1821: }
SourceFile:net/sched/sch_api.c]
|
| 146/2 |
2026/05/18 18:55 |
llm |
debugger |
0m
Model:
Tokens:
input: 155161
output: 45
thoughts: 13
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Analyzing the `__tc_modify_qdisc` Function**
Alright, let's dive into this `__tc_modify_qdisc` function. I need to refresh my mental model here. My initial thought is to quickly analyze the core purpose and how it fits into the broader traffic control landscape. This is likely the workhorse for actually modifying a queuing discipline, which means understanding what parameters it takes, what data structures it manipulates, and the locking implications are paramount. I need to think about potential race conditions. Is this a critical section? Does it handle concurrency appropriately? Are there potential deadlocks lurking? Specifically, I'm thinking about the interactions between different threads or processes modifying the same qdisc or related objects. What are the dependencies?
Next, I need to understand its relationship to the broader qdisc manipulation workflow. Where is it called from? Is it part of the user-space interaction via `tc` or is it more of a kernel-internal function? What's the context? Is it called during qdisc addition, deletion, or simply configuration updates? It must interface with the netlink socket, so I need to keep that in mind.
A quick check for potential error handling is also crucial. What are the common error codes this function can return? What are the potential failure points (memory allocation, invalid parameters, locking conflicts)? I need to understand the implications of different failure paths and if it handles these gracefully or if it leaves things in an inconsistent state. Are there potential security vulnerabilities here? Any place for a race condition for a crafted packet?
Lastly, I need to evaluate its impact on performance. Is this a heavily used function? Does it contribute to any bottlenecks in the traffic control pipeline? Profiling data could be valuable if I'm trying to optimize traffic control.
|
| 147/2 |
2026/05/18 18:55 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:net/sched/sch_api.c IncludeLines:true Name:__tc_modify_qdisc]
Results: map[SourceCode:1581: static int __tc_modify_qdisc(struct sk_buff *skb, struct nlmsghdr *n,
1582: struct netlink_ext_ack *extack,
1583: struct net_device *dev,
1584: struct nlattr *tca[TCA_MAX + 1],
1585: struct tcmsg *tcm)
1586: {
1587: struct Qdisc *q = NULL;
1588: struct Qdisc *p = NULL;
1589: u32 clid;
1590: int err;
1591:
1592: clid = tcm->tcm_parent;
1593:
1594: if (clid) {
1595: if (clid != TC_H_ROOT) {
1596: if (clid != TC_H_INGRESS) {
1597: p = qdisc_lookup(dev, TC_H_MAJ(clid));
1598: if (!p) {
1599: NL_SET_ERR_MSG(extack, "Failed to find specified qdisc");
1600: return -ENOENT;
1601: }
1602: if (p->flags & TCQ_F_INGRESS) {
1603: NL_SET_ERR_MSG(extack,
1604: "Cannot add children to ingress/clsact qdisc");
1605: return -EOPNOTSUPP;
1606: }
1607: q = qdisc_leaf(p, clid, extack);
1608: if (IS_ERR(q))
1609: return PTR_ERR(q);
1610: } else if (dev_ingress_queue_create(dev)) {
1611: q = rtnl_dereference(dev_ingress_queue(dev)->qdisc_sleeping);
1612: }
1613: } else {
1614: q = rtnl_dereference(dev->qdisc);
1615: }
1616:
1617: /* It may be default qdisc, ignore it */
1618: if (q && q->handle == 0)
1619: q = NULL;
1620:
1621: if (!q || !tcm->tcm_handle || q->handle != tcm->tcm_handle) {
1622: if (tcm->tcm_handle) {
1623: if (q && !(n->nlmsg_flags & NLM_F_REPLACE)) {
1624: NL_SET_ERR_MSG(extack, "NLM_F_REPLACE needed to override");
1625: return -EEXIST;
1626: }
1627: if (TC_H_MIN(tcm->tcm_handle)) {
1628: NL_SET_ERR_MSG(extack, "Invalid minor handle");
1629: return -EINVAL;
1630: }
1631: q = qdisc_lookup(dev, tcm->tcm_handle);
1632: if (!q)
1633: goto create_n_graft;
1634: if (q->parent != tcm->tcm_parent) {
1635: NL_SET_ERR_MSG(extack, "Cannot move an existing qdisc to a different parent");
1636: return -EINVAL;
1637: }
1638: if (n->nlmsg_flags & NLM_F_EXCL) {
1639: NL_SET_ERR_MSG(extack, "Exclusivity flag on, cannot override");
1640: return -EEXIST;
1641: }
1642: if (tca[TCA_KIND] &&
1643: nla_strcmp(tca[TCA_KIND], q->ops->id)) {
1644: NL_SET_ERR_MSG(extack, "Invalid qdisc name: must match existing qdisc");
1645: return -EINVAL;
1646: }
1647: if (q->flags & TCQ_F_INGRESS) {
1648: NL_SET_ERR_MSG(extack,
1649: "Cannot regraft ingress or clsact Qdiscs");
1650: return -EINVAL;
1651: }
1652: if (q == p ||
1653: (p && check_loop(q, p, 0))) {
1654: NL_SET_ERR_MSG(extack, "Qdisc parent/child loop detected");
1655: return -ELOOP;
1656: }
1657: if (clid == TC_H_INGRESS) {
1658: NL_SET_ERR_MSG(extack, "Ingress cannot graft directly");
1659: return -EINVAL;
1660: }
1661: qdisc_refcount_inc(q);
1662: goto graft;
1663: } else {
1664: if (!q)
1665: goto create_n_graft;
1666:
1667: /* This magic test requires explanation.
1668: *
1669: * We know, that some child q is already
1670: * attached to this parent and have choice:
1671: * 1) change it or 2) create/graft new one.
1672: * If the requested qdisc kind is different
1673: * than the existing one, then we choose graft.
1674: * If they are the same then this is "change"
1675: * operation - just let it fallthrough..
1676: *
1677: * 1. We are allowed to create/graft only
1678: * if the request is explicitly stating
1679: * "please create if it doesn't exist".
1680: *
1681: * 2. If the request is to exclusive create
1682: * then the qdisc tcm_handle is not expected
1683: * to exist, so that we choose create/graft too.
1684: *
1685: * 3. The last case is when no flags are set.
1686: * This will happen when for example tc
1687: * utility issues a "change" command.
1688: * Alas, it is sort of hole in API, we
1689: * cannot decide what to do unambiguously.
1690: * For now we select create/graft.
1691: */
1692: if (tca[TCA_KIND] &&
1693: nla_strcmp(tca[TCA_KIND], q->ops->id)) {
1694: if (req_create_or_replace(n) ||
1695: req_create_exclusive(n))
1696: goto create_n_graft;
1697: else if (req_change(n))
1698: goto create_n_graft2;
1699: }
1700: }
1701: }
1702: } else {
1703: if (!tcm->tcm_handle) {
1704: NL_SET_ERR_MSG(extack, "Handle cannot be zero");
1705: return -EINVAL;
1706: }
1707: q = qdisc_lookup(dev, tcm->tcm_handle);
1708: }
1709:
1710: /* Change qdisc parameters */
1711: if (!q) {
1712: NL_SET_ERR_MSG(extack, "Specified qdisc not found");
1713: return -ENOENT;
1714: }
1715: if (n->nlmsg_flags & NLM_F_EXCL) {
1716: NL_SET_ERR_MSG(extack, "Exclusivity flag on, cannot modify");
1717: return -EEXIST;
1718: }
1719: if (tca[TCA_KIND] && nla_strcmp(tca[TCA_KIND], q->ops->id)) {
1720: NL_SET_ERR_MSG(extack, "Invalid qdisc name: must match existing qdisc");
1721: return -EINVAL;
1722: }
1723: err = qdisc_change(q, tca, extack);
1724: if (err == 0)
1725: qdisc_notify(sock_net(skb->sk), skb, n, clid, NULL, q, extack);
1726: return err;
1727:
1728: create_n_graft:
1729: if (!(n->nlmsg_flags & NLM_F_CREATE)) {
1730: NL_SET_ERR_MSG(extack, "Qdisc not found. To create specify NLM_F_CREATE flag");
1731: return -ENOENT;
1732: }
1733: create_n_graft2:
1734: if (clid == TC_H_INGRESS) {
1735: if (dev_ingress_queue(dev)) {
1736: q = qdisc_create(dev, dev_ingress_queue(dev),
1737: tcm->tcm_parent, tcm->tcm_parent,
1738: tca, &err, extack);
1739: } else {
1740: NL_SET_ERR_MSG(extack, "Cannot find ingress queue for specified device");
1741: err = -ENOENT;
1742: }
1743: } else {
1744: struct netdev_queue *dev_queue;
1745:
1746: if (p && p->ops->cl_ops && p->ops->cl_ops->select_queue)
1747: dev_queue = p->ops->cl_ops->select_queue(p, tcm);
1748: else if (p)
1749: dev_queue = p->dev_queue;
1750: else
1751: dev_queue = netdev_get_tx_queue(dev, 0);
1752:
1753: q = qdisc_create(dev, dev_queue,
1754: tcm->tcm_parent, tcm->tcm_handle,
1755: tca, &err, extack);
1756: }
1757: if (!q)
1758: return err;
1759:
1760: graft:
1761: err = qdisc_graft(dev, p, skb, n, clid, q, NULL, extack);
1762: if (err) {
1763: if (q)
1764: qdisc_put(q);
1765: return err;
1766: }
1767:
1768: return 0;
1769: }
SourceFile:net/sched/sch_api.c]
|
| 148/2 |
2026/05/18 18:55 |
llm |
debugger |
0m
Model:
Tokens:
input: 158692
output: 22
thoughts: 896
Thoughts: **Debugging a Kernel Soft Lockup: A BPF Conundrum**
Okay, so I'm looking at this crash, and it's a soft lockup. The reproducer seems straightforward, but I'm getting confused. Initially, I saw that `tcm_parent` is zero, which means `clid` is also zero. This triggers the `else` branch, and the code checks for `tcm->tcm_handle`. If that's zero, it should return `-EINVAL`. Right? And if it returns `-EINVAL`, the `RTM_NEWQDISC` call should do nothing. So, why the crash?
Then I notice the reproducer calls `mmap` multiple times. That's unusual, but the critical part is the presence of a BPF program with `BPF_PROG_TYPE_TRACEPOINT`. Okay, I know that's something to investigate. Hold on, the `flags` field is set to 2, which is `BPF_F_SLEEPABLE`. Wait a second... `BPF_PROG_TYPE_TRACEPOINT` doesn't support the `BPF_F_SLEEPABLE` flag! I need to re-examine the `bpf_prog_load` function.
Let's see... within `bpf_prog_load`, there's a check: if `BPF_F_SLEEPABLE` is set, the program type has to be one of a limited number of types. And here's the kicker: `BPF_PROG_TYPE_TRACEPOINT` isn't one of them. So, `bpf_prog_load` *should* return `-EINVAL`! In that case, the BPF program *wouldn't* be loaded. The reproducer then calls `ioctl(SIOCGIFINDEX)` and `sendmsg(RTM_NEWQDISC)`, which also end up returning `-EINVAL`.
But... the reproducer then exits! So why the crash? Is the `-EINVAL` somehow being missed? The whole thing is being run by syzkaller. Does syzkaller run the reproducer in a loop? Could the crash be caused by something else?
The fact that the crash is a soft lockup is throwing me off. I'm starting to think that the BPF program *is* getting loaded somehow. I need to re-check `bpf_prog_load` in `kernel/bpf/syscall.c` and focus on how it handles `BPF_F_SLEEPABLE`. There's something I'm missing here, and the soft lockup is the key.
|
| 149/2 |
2026/05/18 18:56 |
tool |
grepper |
0m
Args: map[Expression:BPF_F_SLEEPABLE]
Results: map[Output:Documentation/bpf/kfuncs.rst=304=The KF_SLEEPABLE flag is used for kfuncs that may sleep. Such kfuncs can only
Documentation/bpf/kfuncs.rst:305:be called by sleepable BPF programs (BPF_F_SLEEPABLE).
Documentation/bpf/kfuncs.rst-306-
--
include/uapi/linux/bpf.h=1186=enum bpf_perf_event_type {
--
include/uapi/linux/bpf.h-1291-
include/uapi/linux/bpf.h:1292:/* If BPF_F_SLEEPABLE is used in BPF_PROG_LOAD command, the verifier will
include/uapi/linux/bpf.h-1293- * restrict map and helper usage for such programs. Sleepable BPF programs can
--
include/uapi/linux/bpf.h-1297- */
include/uapi/linux/bpf.h:1298:#define BPF_F_SLEEPABLE (1U << 4)
include/uapi/linux/bpf.h-1299-
--
kernel/bpf/syscall.c=2864=static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
--
kernel/bpf/syscall.c-2879- BPF_F_TEST_STATE_FREQ |
kernel/bpf/syscall.c:2880: BPF_F_SLEEPABLE |
kernel/bpf/syscall.c-2881- BPF_F_TEST_RND_HI32 |
--
kernel/bpf/syscall.c-2996- prog->expected_attach_type = attr->expected_attach_type;
kernel/bpf/syscall.c:2997: prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE);
kernel/bpf/syscall.c-2998- prog->aux->attach_btf = attach_btf;
--
tools/include/uapi/linux/bpf.h=1186=enum bpf_perf_event_type {
--
tools/include/uapi/linux/bpf.h-1291-
tools/include/uapi/linux/bpf.h:1292:/* If BPF_F_SLEEPABLE is used in BPF_PROG_LOAD command, the verifier will
tools/include/uapi/linux/bpf.h-1293- * restrict map and helper usage for such programs. Sleepable BPF programs can
--
tools/include/uapi/linux/bpf.h-1297- */
tools/include/uapi/linux/bpf.h:1298:#define BPF_F_SLEEPABLE (1U << 4)
tools/include/uapi/linux/bpf.h-1299-
--
tools/lib/bpf/libbpf.c=7771=static int libbpf_prepare_prog_load(struct bpf_program *prog,
--
tools/lib/bpf/libbpf.c-7780- if (def & SEC_SLEEPABLE)
tools/lib/bpf/libbpf.c:7781: opts->prog_flags |= BPF_F_SLEEPABLE;
tools/lib/bpf/libbpf.c-7782-
--
tools/lib/bpf/libbpf_probes.c=103=static int probe_prog_load(enum bpf_prog_type prog_type,
--
tools/lib/bpf/libbpf_probes.c-155- case BPF_PROG_TYPE_SYSCALL:
tools/lib/bpf/libbpf_probes.c:156: opts.prog_flags = BPF_F_SLEEPABLE;
tools/lib/bpf/libbpf_probes.c-157- break;
--
tools/lib/bpf/skel_internal.h=353=static inline int bpf_load_and_run(struct bpf_load_and_run_opts *opts)
--
tools/lib/bpf/skel_internal.h-407- attr.log_buf = opts->ctx->log_buf;
tools/lib/bpf/skel_internal.h:408: attr.prog_flags = BPF_F_SLEEPABLE;
tools/lib/bpf/skel_internal.h-409- err = prog_fd = skel_sys_bpf(BPF_PROG_LOAD, &attr, prog_load_attr_sz);
--
tools/testing/selftests/bpf/prog_tests/attach_probe.c=537=static void test_kprobe_sleepable(void)
--
tools/testing/selftests/bpf/prog_tests/attach_probe.c-546- if (!ASSERT_OK(bpf_program__set_flags(skel->progs.handle_kprobe_sleepable,
tools/testing/selftests/bpf/prog_tests/attach_probe.c:547: BPF_F_SLEEPABLE), "kprobe_sleepable_flags"))
tools/testing/selftests/bpf/prog_tests/attach_probe.c-548- goto cleanup;
--
tools/testing/selftests/bpf/prog_tests/custom_sec_handlers.c=21=static int custom_prepare_load_prog(struct bpf_program *prog,
--
tools/testing/selftests/bpf/prog_tests/custom_sec_handlers.c-24- if (cookie == COOKIE_FALLBACK)
tools/testing/selftests/bpf/prog_tests/custom_sec_handlers.c:25: opts->prog_flags |= BPF_F_SLEEPABLE;
tools/testing/selftests/bpf/prog_tests/custom_sec_handlers.c-26- else if (cookie == COOKIE_ABC1)
--
tools/testing/selftests/bpf/prog_tests/custom_sec_handlers.c=94=void test_custom_sec_handlers(void)
--
tools/testing/selftests/bpf/prog_tests/custom_sec_handlers.c-114- /* fallback treats everything as BPF_PROG_TYPE_SYSCALL program to test
tools/testing/selftests/bpf/prog_tests/custom_sec_handlers.c:115: * setting custom BPF_F_SLEEPABLE bit in preload handler
tools/testing/selftests/bpf/prog_tests/custom_sec_handlers.c-116- */
--
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c=221=static void test_attach_api_fails(void)
--
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c-364- err = bpf_program__set_flags(sl_skel->progs.handle_kprobe_multi_sleepable,
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c:365: BPF_F_SLEEPABLE);
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c-366- if (!ASSERT_OK(err, "sleep_skel_set_flags"))
--
tools/testing/selftests/bpf/progs/bpf_misc.h-114- * - BPF_F_TEST_STATE_FREQ;
tools/testing/selftests/bpf/progs/bpf_misc.h:115: * - BPF_F_SLEEPABLE;
tools/testing/selftests/bpf/progs/bpf_misc.h-116- * - BPF_F_XDP_HAS_FRAGS;
--
tools/testing/selftests/bpf/test_loader.c=399=static int parse_test_spec(struct test_loader *tester,
--
tools/testing/selftests/bpf/test_loader.c-560- update_flags(&spec->prog_flags, BPF_F_TEST_STATE_FREQ, clear);
tools/testing/selftests/bpf/test_loader.c:561: } else if (strcmp(val, "BPF_F_SLEEPABLE") == 0) {
tools/testing/selftests/bpf/test_loader.c:562: update_flags(&spec->prog_flags, BPF_F_SLEEPABLE, clear);
tools/testing/selftests/bpf/test_loader.c-563- } else if (strcmp(val, "BPF_F_XDP_HAS_FRAGS") == 0) {
--
tools/testing/selftests/bpf/verifier/sleepable.c-10- .result = ACCEPT,
tools/testing/selftests/bpf/verifier/sleepable.c:11: .flags = BPF_F_SLEEPABLE,
tools/testing/selftests/bpf/verifier/sleepable.c-12- .runs = -1,
--
tools/testing/selftests/bpf/verifier/sleepable.c-23- .result = ACCEPT,
tools/testing/selftests/bpf/verifier/sleepable.c:24: .flags = BPF_F_SLEEPABLE,
tools/testing/selftests/bpf/verifier/sleepable.c-25- .runs = -1,
--
tools/testing/selftests/bpf/verifier/sleepable.c-36- .result = ACCEPT,
tools/testing/selftests/bpf/verifier/sleepable.c:37: .flags = BPF_F_SLEEPABLE,
tools/testing/selftests/bpf/verifier/sleepable.c-38- .runs = -1,
--
tools/testing/selftests/bpf/verifier/sleepable.c-49- .result = ACCEPT,
tools/testing/selftests/bpf/verifier/sleepable.c:50: .flags = BPF_F_SLEEPABLE,
tools/testing/selftests/bpf/verifier/sleepable.c-51- .runs = -1,
--
tools/testing/selftests/bpf/verifier/sleepable.c-62- .result = ACCEPT,
tools/testing/selftests/bpf/verifier/sleepable.c:63: .flags = BPF_F_SLEEPABLE,
tools/testing/selftests/bpf/verifier/sleepable.c-64- .runs = -1,
--
tools/testing/selftests/bpf/verifier/sleepable.c-74- .result = ACCEPT,
tools/testing/selftests/bpf/verifier/sleepable.c:75: .flags = BPF_F_SLEEPABLE,
tools/testing/selftests/bpf/verifier/sleepable.c-76- .runs = -1,
--
tools/testing/selftests/bpf/verifier/sleepable.c-88- .errstr = "Only fentry/fexit/fsession/fmod_ret, lsm, iter, uprobe, and struct_ops programs can be sleepable",
tools/testing/selftests/bpf/verifier/sleepable.c:89: .flags = BPF_F_SLEEPABLE,
tools/testing/selftests/bpf/verifier/sleepable.c-90- .runs = -1,
]
|
| 150/2 |
2026/05/18 18:56 |
llm |
debugger |
0m
Model:
Tokens:
input: 163105
output: 20
thoughts: 3640
Thoughts: **The Puzzle of the Infinite hrtimer Loop**
Okay, so here's what I've got. The initial BPF program loading is a red herring. It *succeeds* in loading, despite my initial concerns about the flags or the invalid map file descriptor (-1), so it's not the root cause. This seemed suspicious, but the crash occurs *after* the BPF load, meaning it is probably not the core of the problem. Also, the BPF program's code, even if loaded, never runs as `bpf_prog_load` is successful, but the program is not attached.
My focus, initially, was the `hrtimer` corruption, the `rb_erase_linked` loop, and the map file descriptor. The code does some mmap calls which look suspicious, which could potentially explain data corruption. However, that line of inquiry seems like it's leading nowhere.
The kernel is crashing in `__remove_hrtimer`, specifically, the issue has something to do with `timerqueue_linked_del`, and by extension the rbtree. After revisiting the crash dump and the code, I see that the important part is actually that CPU 1 *appears* to be stuck in `__remove_hrtimer`, but this is wrong. CPU 1 is NOT stuck, it is just interrupted by the NMI at that exact point. It is in `default_idle`, waiting for an interrupt, and the backtrace points to a timer interrupt. That interrupt's handler, `__hrtimer_run_queues`, is looping infinitely.
The infinite loop is the real issue. `__hrtimer_run_queues` is looping, and thus, processing the *same* timer over and over again. The timer is likely returning `HRTIMER_RESTART` but *not* advancing its expiration time.
The reproducer includes the creation of a netlink socket and sending a `RTM_NEWQDISC` message. `tcm_ifindex`, the interface index in the `tcmsg` struct, is set to 0. This is the key. While the system appears to correctly handle an ifindex of 0 and return `-ENODEV`, perhaps there is something in how this affects an hrtimer associated with a QDISC? The fact that a non-existent device is specified could be the root cause.
|
| 151/2 |
2026/05/18 18:56 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:tcm_family OffsetBits:0 SizeBits:8] map[Name:tcm__pad1 OffsetBits:8 SizeBits:8] map[Name:tcm__pad2 OffsetBits:16 SizeBits:16] map[Name:tcm_ifindex OffsetBits:32 SizeBits:32] map[Name:tcm_handle OffsetBits:64 SizeBits:32] map[Name:tcm_parent OffsetBits:96 SizeBits:32] map[Name:tcm_info OffsetBits:128 SizeBits:32]]]
|
| 152/2 |
2026/05/18 18:56 |
llm |
debugger |
0m
Model:
Tokens:
input: 167427
output: 22
thoughts: 351
Thoughts: **Analyzing the Netlink Message Header**
Okay, so I'm trying to pinpoint where `tcm_ifindex` is within this Netlink message structure. I remember from somewhere that it's located at an offset of 32 bits, which translates to 4 bytes. Let me just confirm that quickly.
Right, let's look at this C code snippet. It looks like it's trying to write specific values to memory addresses. Okay, `*(uint64_t*)0x200000000780 = 0x200000000000;` - this appears to be setting up an address. Then, `*(uint32_t*)0x200000000000 = 0;`... Aha, `0x200000000000` is being used as the base address. And the subsequent lines show data being written to offsets from that base address.
Let me think… `0x200000000000`… That address looks suspiciously like the start of an `nlmsghdr` structure. I need to be sure; let me quickly double check the definition of the `struct nlmsghdr` itself.
|
| 153/2 |
2026/05/18 18:57 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name:nlmsg_len OffsetBits:0 SizeBits:32] map[Name:nlmsg_type OffsetBits:32 SizeBits:16] map[Name:nlmsg_flags OffsetBits:48 SizeBits:16] map[Name:nlmsg_seq OffsetBits:64 SizeBits:32] map[Name:nlmsg_pid OffsetBits:96 SizeBits:32]]]
|
| 154/2 |
2026/05/18 18:57 |
llm |
debugger |
0m
Model:
Tokens:
input: 168199
output: 22
thoughts: 636
Thoughts: **Analyzing the Kernel Crash: A Deep Dive**
Okay, so I'm looking at this crash and trying to piece together what's happening. `nlmsghdr` is 16 bytes, that's clear. `struct tcmsg` then starts at offset 16, which means `tcm_family` is at `0x10`. And `tcm_ifindex` should follow at `0x14`.
But then I see this C code: `*(uint32_t*)0x200000000014 = 0;` which sets `tcm_ifindex` to 0. Hmm, if `tcm_ifindex` is 0, then `__dev_get_by_index(net, 0)` will return NULL. That means `tc_modify_qdisc` will return `-ENODEV`, indicating a device not found, making this `sendmsg` potentially fail silently.
Let's look at the full syscall sequence. There's a `bpf` call, a `memcpy` to "team0", and then an `ioctl` call with `SIOCGIFINDEX` (0x8933). This part, getting the ifindex of `team0`, is where things get interesting. But, WAIT! The reproduced code **doesn't actually use** the index after getting it. That's odd. Syzkaller usually handles the resources.
The Syzkaller description indicates an `ifindex` resource, but the generated C code might be failing to pass it correctly. If `tcm_ifindex` is 0, the `sendmsg` would effectively do nothing. But then, how is the kernel crashing?
Hold on. The BPF program... Let's check `prog_ifindex` offset at 0xcd0.
Let's confirm the `union bpf_attr` structure to fully understand `BPF_PROG_LOAD`. It seems the crash must be coming from the `BPF` and not the `tc` side of this. This is where I need to focus my analysis.
|
| 155/2 |
2026/05/18 18:57 |
tool |
codesearch-struct-layout |
0m
Args:
Results: map[Fields:[map[Name: OffsetBits:0 SizeBits:768] map[Name: OffsetBits:0 SizeBits:256] map[Name:batch OffsetBits:0 SizeBits:448] map[Name: OffsetBits:0 SizeBits:1344] map[Name: OffsetBits:0 SizeBits:192] map[Name: OffsetBits:0 SizeBits:256] map[Name:test OffsetBits:0 SizeBits:640] map[Name: OffsetBits:0 SizeBits:128] map[Name:info OffsetBits:0 SizeBits:128] map[Name:query OffsetBits:0 SizeBits:512] map[Name:raw_tracepoint OffsetBits:0 SizeBits:192] map[Name: OffsetBits:0 SizeBits:320] map[Name:task_fd_query OffsetBits:0 SizeBits:384] map[Name:link_create OffsetBits:0 SizeBits:512] map[Name:link_update OffsetBits:0 SizeBits:128] map[Name:link_detach OffsetBits:0 SizeBits:32] map[Name:enable_stats OffsetBits:0 SizeBits:32] map[Name:iter_create OffsetBits:0 SizeBits:64] map[Name:prog_bind_map OffsetBits:0 SizeBits:96] map[Name:token_create OffsetBits:0 SizeBits:64] map[Name:prog_stream_read OffsetBits:0 SizeBits:192] map[Name:prog_assoc_struct_ops OffsetBits:0 SizeBits:96]]]
|
| 156/2 |
2026/05/18 18:57 |
llm |
debugger |
0m
Model:
Tokens:
input: 169746
output: 44
thoughts: 33
Thoughts: Okay, let's see...
**Deconstructing the bpf_attr Union for BPF_PROG_LOAD**
Hmm, right, *`bpf_attr` is a union.* That's important. I need to make sure I'm using the right member depending on the BPF system call I'm making. I'm focusing on `BPF_PROG_LOAD` right now. So, I need to go back and explicitly check the `bpf_attr` structure definition for how the arguments are laid out for *that specific* `BPF` command. I need to be precise here; one misaligned field and the kernel will probably have kittens. Time to dive into the kernel headers and refresh my memory on the layout. No room for assumptions, even on something I *think* I know well. Best to go straight to the source.
|
| 157/2 |
2026/05/18 18:57 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/uapi/linux/bpf.h IncludeLines:true Name:bpf_attr]
Results: map[SourceCode:1510: union bpf_attr {
1511: struct { /* anonymous struct used by BPF_MAP_CREATE command */
1512: __u32 map_type; /* one of enum bpf_map_type */
1513: __u32 key_size; /* size of key in bytes */
1514: __u32 value_size; /* size of value in bytes */
1515: __u32 max_entries; /* max number of entries in a map */
1516: __u32 map_flags; /* BPF_MAP_CREATE related
1517: * flags defined above.
1518: */
1519: __u32 inner_map_fd; /* fd pointing to the inner map */
1520: __u32 numa_node; /* numa node (effective only if
1521: * BPF_F_NUMA_NODE is set).
1522: */
1523: char map_name[BPF_OBJ_NAME_LEN];
1524: __u32 map_ifindex; /* ifindex of netdev to create on */
1525: __u32 btf_fd; /* fd pointing to a BTF type data */
1526: __u32 btf_key_type_id; /* BTF type_id of the key */
1527: __u32 btf_value_type_id; /* BTF type_id of the value */
1528: __u32 btf_vmlinux_value_type_id;/* BTF type_id of a kernel-
1529: * struct stored as the
1530: * map value
1531: */
1532: /* Any per-map-type extra fields
1533: *
1534: * BPF_MAP_TYPE_BLOOM_FILTER - the lowest 4 bits indicate the
1535: * number of hash functions (if 0, the bloom filter will default
1536: * to using 5 hash functions).
1537: *
1538: * BPF_MAP_TYPE_ARENA - contains the address where user space
1539: * is going to mmap() the arena. It has to be page aligned.
1540: */
1541: __u64 map_extra;
1542:
1543: __s32 value_type_btf_obj_fd; /* fd pointing to a BTF
1544: * type data for
1545: * btf_vmlinux_value_type_id.
1546: */
1547: /* BPF token FD to use with BPF_MAP_CREATE operation.
1548: * If provided, map_flags should have BPF_F_TOKEN_FD flag set.
1549: */
1550: __s32 map_token_fd;
1551:
1552: /* Hash of the program that has exclusive access to the map.
1553: */
1554: __aligned_u64 excl_prog_hash;
1555: /* Size of the passed excl_prog_hash. */
1556: __u32 excl_prog_hash_size;
1557: };
1558:
1559: struct { /* anonymous struct used by BPF_MAP_*_ELEM and BPF_MAP_FREEZE commands */
1560: __u32 map_fd;
1561: __aligned_u64 key;
1562: union {
1563: __aligned_u64 value;
1564: __aligned_u64 next_key;
1565: };
1566: __u64 flags;
1567: };
1568:
1569: struct { /* struct used by BPF_MAP_*_BATCH commands */
1570: __aligned_u64 in_batch; /* start batch,
1571: * NULL to start from beginning
1572: */
1573: __aligned_u64 out_batch; /* output: next start batch */
1574: __aligned_u64 keys;
1575: __aligned_u64 values;
1576: __u32 count; /* input/output:
1577: * input: # of key/value
1578: * elements
1579: * output: # of filled elements
1580: */
1581: __u32 map_fd;
1582: __u64 elem_flags;
1583: __u64 flags;
1584: } batch;
1585:
1586: struct { /* anonymous struct used by BPF_PROG_LOAD command */
1587: __u32 prog_type; /* one of enum bpf_prog_type */
1588: __u32 insn_cnt;
1589: __aligned_u64 insns;
1590: __aligned_u64 license;
1591: __u32 log_level; /* verbosity level of verifier */
1592: __u32 log_size; /* size of user buffer */
1593: __aligned_u64 log_buf; /* user supplied buffer */
1594: __u32 kern_version; /* not used */
1595: __u32 prog_flags;
1596: char prog_name[BPF_OBJ_NAME_LEN];
1597: __u32 prog_ifindex; /* ifindex of netdev to prep for */
1598: /* For some prog types expected attach type must be known at
1599: * load time to verify attach type specific parts of prog
1600: * (context accesses, allowed helpers, etc).
1601: */
1602: __u32 expected_attach_type;
1603: __u32 prog_btf_fd; /* fd pointing to BTF type data */
1604: __u32 func_info_rec_size; /* userspace bpf_func_info size */
1605: __aligned_u64 func_info; /* func info */
1606: __u32 func_info_cnt; /* number of bpf_func_info records */
1607: __u32 line_info_rec_size; /* userspace bpf_line_info size */
1608: __aligned_u64 line_info; /* line info */
1609: __u32 line_info_cnt; /* number of bpf_line_info records */
1610: __u32 attach_btf_id; /* in-kernel BTF type id to attach to */
1611: union {
1612: /* valid prog_fd to attach to bpf prog */
1613: __u32 attach_prog_fd;
1614: /* or valid module BTF object fd or 0 to attach to vmlinux */
1615: __u32 attach_btf_obj_fd;
1616: };
1617: __u32 core_relo_cnt; /* number of bpf_core_relo */
1618: __aligned_u64 fd_array; /* array of FDs */
1619: __aligned_u64 core_relos;
1620: __u32 core_relo_rec_size; /* sizeof(struct bpf_core_relo) */
1621: /* output: actual total log contents size (including termintaing zero).
1622: * It could be both larger than original log_size (if log was
1623: * truncated), or smaller (if log buffer wasn't filled completely).
1624: */
1625: __u32 log_true_size;
1626: /* BPF token FD to use with BPF_PROG_LOAD operation.
1627: * If provided, prog_flags should have BPF_F_TOKEN_FD flag set.
1628: */
1629: __s32 prog_token_fd;
1630: /* The fd_array_cnt can be used to pass the length of the
1631: * fd_array array. In this case all the [map] file descriptors
1632: * passed in this array will be bound to the program, even if
1633: * the maps are not referenced directly. The functionality is
1634: * similar to the BPF_PROG_BIND_MAP syscall, but maps can be
1635: * used by the verifier during the program load. If provided,
1636: * then the fd_array[0,...,fd_array_cnt-1] is expected to be
1637: * continuous.
1638: */
1639: __u32 fd_array_cnt;
1640: /* Pointer to a buffer containing the signature of the BPF
1641: * program.
1642: */
1643: __aligned_u64 signature;
1644: /* Size of the signature buffer in bytes. */
1645: __u32 signature_size;
1646: /* ID of the kernel keyring to be used for signature
1647: * verification.
1648: */
1649: __s32 keyring_id;
1650: };
1651:
1652: struct { /* anonymous struct used by BPF_OBJ_* commands */
1653: __aligned_u64 pathname;
1654: __u32 bpf_fd;
1655: __u32 file_flags;
1656: /* Same as dirfd in openat() syscall; see openat(2)
1657: * manpage for details of path FD and pathname semantics;
1658: * path_fd should accompanied by BPF_F_PATH_FD flag set in
1659: * file_flags field, otherwise it should be set to zero;
1660: * if BPF_F_PATH_FD flag is not set, AT_FDCWD is assumed.
1661: */
1662: __s32 path_fd;
1663: };
1664:
1665: struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
1666: union {
1667: __u32 target_fd; /* target object to attach to or ... */
1668: __u32 target_ifindex; /* target ifindex */
1669: };
1670: __u32 attach_bpf_fd;
1671: __u32 attach_type;
1672: __u32 attach_flags;
1673: __u32 replace_bpf_fd;
1674: union {
1675: __u32 relative_fd;
1676: __u32 relative_id;
1677: };
1678: __u64 expected_revision;
1679: };
1680:
1681: struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
1682: __u32 prog_fd;
1683: __u32 retval;
1684: __u32 data_size_in; /* input: len of data_in */
1685: __u32 data_size_out; /* input/output: len of data_out
1686: * returns ENOSPC if data_out
1687: * is too small.
1688: */
1689: __aligned_u64 data_in;
1690: __aligned_u64 data_out;
1691: __u32 repeat;
1692: __u32 duration;
1693: __u32 ctx_size_in; /* input: len of ctx_in */
1694: __u32 ctx_size_out; /* input/output: len of ctx_out
1695: * returns ENOSPC if ctx_out
1696: * is too small.
1697: */
1698: __aligned_u64 ctx_in;
1699: __aligned_u64 ctx_out;
1700: __u32 flags;
1701: __u32 cpu;
1702: __u32 batch_size;
1703: } test;
1704:
1705: struct { /* anonymous struct used by BPF_*_GET_*_ID */
1706: union {
1707: __u32 start_id;
1708: __u32 prog_id;
1709: __u32 map_id;
1710: __u32 btf_id;
1711: __u32 link_id;
1712: };
1713: __u32 next_id;
1714: __u32 open_flags;
1715: __s32 fd_by_id_token_fd;
1716: };
1717:
1718: struct { /* anonymous struct used by BPF_OBJ_GET_INFO_BY_FD */
1719: __u32 bpf_fd;
1720: __u32 info_len;
1721: __aligned_u64 info;
1722: } info;
1723:
1724: struct { /* anonymous struct used by BPF_PROG_QUERY command */
1725: union {
1726: __u32 target_fd; /* target object to query or ... */
1727: __u32 target_ifindex; /* target ifindex */
1728: };
1729: __u32 attach_type;
1730: __u32 query_flags;
1731: __u32 attach_flags;
1732: __aligned_u64 prog_ids;
1733: union {
1734: __u32 prog_cnt;
1735: __u32 count;
1736: };
1737: __u32 :32;
1738: /* output: per-program attach_flags.
1739: * not allowed to be set during effective query.
1740: */
1741: __aligned_u64 prog_attach_flags;
1742: __aligned_u64 link_ids;
1743: __aligned_u64 link_attach_flags;
1744: __u64 revision;
1745: } query;
1746:
1747: struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
1748: __u64 name;
1749: __u32 prog_fd;
1750: __u32 :32;
1751: __aligned_u64 cookie;
1752: } raw_tracepoint;
1753:
1754: struct { /* anonymous struct for BPF_BTF_LOAD */
1755: __aligned_u64 btf;
1756: __aligned_u64 btf_log_buf;
1757: __u32 btf_size;
1758: __u32 btf_log_size;
1759: __u32 btf_log_level;
1760: /* output: actual total log contents size (including termintaing zero).
1761: * It could be both larger than original log_size (if log was
1762: * truncated), or smaller (if log buffer wasn't filled completely).
1763: */
1764: __u32 btf_log_true_size;
1765: __u32 btf_flags;
1766: /* BPF token FD to use with BPF_BTF_LOAD operation.
1767: * If provided, btf_flags should have BPF_F_TOKEN_FD flag set.
1768: */
1769: __s32 btf_token_fd;
1770: };
1771:
1772: struct {
1773: __u32 pid; /* input: pid */
1774: __u32 fd; /* input: fd */
1775: __u32 flags; /* input: flags */
1776: __u32 buf_len; /* input/output: buf len */
1777: __aligned_u64 buf; /* input/output:
1778: * tp_name for tracepoint
1779: * symbol for kprobe
1780: * filename for uprobe
1781: */
1782: __u32 prog_id; /* output: prod_id */
1783: __u32 fd_type; /* output: BPF_FD_TYPE_* */
1784: __u64 probe_offset; /* output: probe_offset */
1785: __u64 probe_addr; /* output: probe_addr */
1786: } task_fd_query;
1787:
1788: struct { /* struct used by BPF_LINK_CREATE command */
1789: union {
1790: __u32 prog_fd; /* eBPF program to attach */
1791: __u32 map_fd; /* struct_ops to attach */
1792: };
1793: union {
1794: __u32 target_fd; /* target object to attach to or ... */
1795: __u32 target_ifindex; /* target ifindex */
1796: };
1797: __u32 attach_type; /* attach type */
1798: __u32 flags; /* extra flags */
1799: union {
1800: __u32 target_btf_id; /* btf_id of target to attach to */
1801: struct {
1802: __aligned_u64 iter_info; /* extra bpf_iter_link_info */
1803: __u32 iter_info_len; /* iter_info length */
1804: };
1805: struct {
1806: /* black box user-provided value passed through
1807: * to BPF program at the execution time and
1808: * accessible through bpf_get_attach_cookie() BPF helper
1809: */
1810: __u64 bpf_cookie;
1811: } perf_event;
1812: struct {
1813: __u32 flags;
1814: __u32 cnt;
1815: __aligned_u64 syms;
1816: __aligned_u64 addrs;
1817: __aligned_u64 cookies;
1818: } kprobe_multi;
1819: struct {
1820: /* this is overlaid with the target_btf_id above. */
1821: __u32 target_btf_id;
1822: /* black box user-provided value passed through
1823: * to BPF program at the execution time and
1824: * accessible through bpf_get_attach_cookie() BPF helper
1825: */
1826: __u64 cookie;
1827: } tracing;
1828: struct {
1829: __u32 pf;
1830: __u32 hooknum;
1831: __s32 priority;
1832: __u32 flags;
1833: } netfilter;
1834: struct {
1835: union {
1836: __u32 relative_fd;
1837: __u32 relative_id;
1838: };
1839: __u64 expected_revision;
1840: } tcx;
1841: struct {
1842: __aligned_u64 path;
1843: __aligned_u64 offsets;
1844: __aligned_u64 ref_ctr_offsets;
1845: __aligned_u64 cookies;
1846: __u32 cnt;
1847: __u32 flags;
1848: __u32 pid;
1849: } uprobe_multi;
1850: struct {
1851: union {
1852: __u32 relative_fd;
1853: __u32 relative_id;
1854: };
1855: __u64 expected_revision;
1856: } netkit;
1857: struct {
1858: union {
1859: __u32 relative_fd;
1860: __u32 relative_id;
1861: };
1862: __u64 expected_revision;
1863: } cgroup;
1864: };
1865: } link_create;
1866:
1867: struct { /* struct used by BPF_LINK_UPDATE command */
1868: __u32 link_fd; /* link fd */
1869: union {
1870: /* new program fd to update link with */
1871: __u32 new_prog_fd;
1872: /* new struct_ops map fd to update link with */
1873: __u32 new_map_fd;
1874: };
1875: __u32 flags; /* extra flags */
1876: union {
1877: /* expected link's program fd; is specified only if
1878: * BPF_F_REPLACE flag is set in flags.
1879: */
1880: __u32 old_prog_fd;
1881: /* expected link's map fd; is specified only
1882: * if BPF_F_REPLACE flag is set.
1883: */
1884: __u32 old_map_fd;
1885: };
1886: } link_update;
1887:
1888: struct {
1889: __u32 link_fd;
1890: } link_detach;
1891:
1892: struct { /* struct used by BPF_ENABLE_STATS command */
1893: __u32 type;
1894: } enable_stats;
1895:
1896: struct { /* struct used by BPF_ITER_CREATE command */
1897: __u32 link_fd;
1898: __u32 flags;
1899: } iter_create;
1900:
1901: struct { /* struct used by BPF_PROG_BIND_MAP command */
1902: __u32 prog_fd;
1903: __u32 map_fd;
1904: __u32 flags; /* extra flags */
1905: } prog_bind_map;
1906:
1907: struct { /* struct used by BPF_TOKEN_CREATE command */
1908: __u32 flags;
1909: __u32 bpffs_fd;
1910: } token_create;
1911:
1912: struct {
1913: __aligned_u64 stream_buf;
1914: __u32 stream_buf_len;
1915: __u32 stream_id;
1916: __u32 prog_fd;
1917: } prog_stream_read;
1918:
1919: struct {
1920: __u32 map_fd;
1921: __u32 prog_fd;
1922: __u32 flags;
1923: } prog_assoc_struct_ops;
1924:
1925: } __attribute__((aligned(8)));
SourceFile:include/uapi/linux/bpf.h]
|
| 158/2 |
2026/05/18 18:57 |
llm |
debugger |
0m
Model:
Tokens:
input: 176833
output: 48
thoughts: 1182
Thoughts: **Analyzing the BPF Program Loader Reproducer**
Okay, let's break this down. First thing I notice is the memory layout being defined by this reproducer code. I immediately key in on the structure offsets – especially the one for `prog_ifindex` at offset `0x28` (40 bytes). I've got to keep track of this memory layout while working. I need to be sure the reproducer is correctly crafting the `bpf_attr` structure for the `bpf_prog_load` syscall.
Looking at the C code, I see the reproducer is populating the `bpf_attr` struct, field by field. I'm focusing on the field initializations. Okay, so we're setting `prog_type`, `insn_cnt`, and providing the instructions. Then there's the `license`, and some logging parameters. We're setting `kern_version`, `prog_flags` (which is `BPF_F_ANY_ALIGNMENT` = 2, so `BPF_F_SLEEPABLE` isn't enabled), a `prog_name`, `prog_ifindex` (which is zero), and various other fields related to BTF, function information, line information, attachments, and core relocations. I'm going through each initialization to make sure it's valid.
Then I get to the end of the structure setup, and I focus on the value provided for `prog_token_fd` at `0xcd0`, or 144, which must be the struct member offset. Then I remembered the size of `sizeof(union bpf_attr)` = 192 bytes. Now I see that the reproducer has passed a size of 144 bytes, so there is something truncated here.
Okay, so I have to confirm the `bpf_prog_load` call will succeed if the program is valid. The `prog_flags` were set with `BPF_F_ANY_ALIGNMENT`. Then, it hits me that the program has `ringbuf_map_fd` set to `-1`. I recognize that as an invalid file descriptor.
Now, I understand! The reproducer's intention is to fail the verifier! I see that `-1` is in the `imm` field of `insn1` in the `ringbuf_reserve` BPF instruction. So I know this instruction is `bpf_insn_map_fd_t`. If `imm` is `-1`, then `bpf_check` should fail because `-1` is an invalid file descriptor. Therefore the verifier should have failed.
I am going to check `replace_map_fd_with_map_ptr` in the kernel at `kernel/bpf/verifier.c`. Let's see what is happening!
|
| 159/2 |
2026/05/18 18:57 |
tool |
codesearch-definition-source |
0m
Error:
Args: map[ContextFile:kernel/bpf/verifier.c IncludeLines:true Name:replace_map_fd_with_map_ptr]
Results: map[SourceCode: SourceFile:]
|
| 160/2 |
2026/05/18 18:57 |
llm |
debugger |
0m
Model:
Tokens:
input: 178693
output: 25
thoughts: 18
Thoughts: Okay, here's my attempt at summarizing the thought process, as if it were my own, assuming I'm an expert in the field:
**Initial Investigation: The `replace_map_fd_with_map_ptr` Search**
Alright, time to dig in. I need to find all instances of `replace_map_fd_with_map_ptr`. This is likely a critical function or a piece of code that’s been flagged, or that I need to understand better. Given the naming convention, it suggests a transition or refactoring relating to file descriptors and memory pointers associated with memory mapped regions. Could be a significant change involving how we're handling memory mapped files. I’ll start with a comprehensive code search – I'll probably leverage my preferred tool (grep, ripgrep, or whatever is appropriate for this project) to get a broad overview. My initial search will target the exact string, with some wildcards to find variations or partial matches, in a wide search scope. Then, I’ll refine it based on what I find.
|
| 161/2 |
2026/05/18 18:57 |
tool |
grepper |
0m
Error:
Args: map[Expression:replace_map_fd_with_map_ptr]
Results:
|
| 162/2 |
2026/05/18 18:57 |
llm |
debugger |
1m
Model:
Tokens:
input: 178973
output: 20
thoughts: 25
Thoughts: Here's my thought process, summarized as you requested:
**Navigating the BPF Verifier**
Alright, let's dive into the BPF verifier code. I need to track down how `map_ptr` is handled, and my initial instinct is to focus on `fd` because it's a common identifier related to file descriptors and, by extension, BPF maps. I'm going to start by searching for instances of `fd` specifically within `kernel/bpf/verifier.c`. My thinking is that this file is the likely spot where the validation and access control around maps, and therefore their associated file descriptors, are checked during BPF program execution. I'll need to then see how `map_ptr` is used in that context.
|
| 163/2 |
2026/05/18 18:58 |
tool |
grepper |
0m
Args: map[Expression:bpf_map_get]
Results: map[Output:
Full output is too long, showing 500 out of 1295 lines.
Use more precise expression if possible.
[Documentation/bpf/map_hash.rst=118=Userspace
--
Documentation/bpf/map_hash.rst-120-
Documentation/bpf/map_hash.rst:121:bpf_map_get_next_key()
Documentation/bpf/map_hash.rst-122-~~~~~~~~~~~~~~~~~~~~~~
--
Documentation/bpf/map_hash.rst-125-
Documentation/bpf/map_hash.rst:126: int bpf_map_get_next_key(int fd, const void *cur_key, void *next_key)
Documentation/bpf/map_hash.rst-127-
Documentation/bpf/map_hash.rst=128=In userspace, it is possible to iterate through the keys of a hash using
Documentation/bpf/map_hash.rst:129:libbpf's ``bpf_map_get_next_key()`` function. The first key can be fetched by
Documentation/bpf/map_hash.rst:130:calling ``bpf_map_get_next_key()`` with ``cur_key`` set to
Documentation/bpf/map_hash.rst-131-``NULL``. Subsequent calls will fetch the next key that follows the
Documentation/bpf/map_hash.rst:132:current key. ``bpf_map_get_next_key()`` returns 0 on success, -ENOENT if
Documentation/bpf/map_hash.rst-133-cur_key is the last key in the hash, or negative error in case of
Documentation/bpf/map_hash.rst=134=failure.
Documentation/bpf/map_hash.rst-135-
Documentation/bpf/map_hash.rst:136:Note that if ``cur_key`` gets deleted then ``bpf_map_get_next_key()``
Documentation/bpf/map_hash.rst-137-will instead return the *first* key in the hash table which is
Documentation/bpf/map_hash.rst=138=undesirable. It is recommended to use batched lookup if there is going
Documentation/bpf/map_hash.rst:139:to be key deletion intermixed with ``bpf_map_get_next_key()``.
Documentation/bpf/map_hash.rst-140-
--
Documentation/bpf/map_hash.rst=193=Userspace walking the map elements from the map declared above:
--
Documentation/bpf/map_hash.rst-207- for (;;) {
Documentation/bpf/map_hash.rst:208: err = bpf_map_get_next_key(map_fd, cur_key, &next_key);
Documentation/bpf/map_hash.rst-209- if (err)
--
Documentation/bpf/map_lpm_trie.rst=87=the map identified by ``fd``.
Documentation/bpf/map_lpm_trie.rst-88-
Documentation/bpf/map_lpm_trie.rst:89:bpf_map_get_next_key()
Documentation/bpf/map_lpm_trie.rst-90-~~~~~~~~~~~~~~~~~~~~~~
--
Documentation/bpf/map_lpm_trie.rst-93-
Documentation/bpf/map_lpm_trie.rst:94: int bpf_map_get_next_key (int fd, const void *cur_key, void *next_key)
Documentation/bpf/map_lpm_trie.rst-95-
Documentation/bpf/map_lpm_trie.rst=96=A userspace program can iterate through the entries in an LPM trie using
Documentation/bpf/map_lpm_trie.rst:97:libbpf's ``bpf_map_get_next_key()`` function. The first key can be
Documentation/bpf/map_lpm_trie.rst:98:fetched by calling ``bpf_map_get_next_key()`` with ``cur_key`` set to
Documentation/bpf/map_lpm_trie.rst-99-``NULL``. Subsequent calls will fetch the next key that follows the
Documentation/bpf/map_lpm_trie.rst:100:current key. ``bpf_map_get_next_key()`` returns ``0`` on success,
Documentation/bpf/map_lpm_trie.rst-101-``-ENOENT`` if ``cur_key`` is the last key in the trie, or negative
Documentation/bpf/map_lpm_trie.rst=102=error in case of failure.
Documentation/bpf/map_lpm_trie.rst-103-
Documentation/bpf/map_lpm_trie.rst:104:``bpf_map_get_next_key()`` will iterate through the LPM trie elements
Documentation/bpf/map_lpm_trie.rst-105-from leftmost leaf first. This means that iteration will return more
--
Documentation/bpf/map_lpm_trie.rst=171=of an LPM trie:
--
Documentation/bpf/map_lpm_trie.rst-186- for (;;) {
Documentation/bpf/map_lpm_trie.rst:187: err = bpf_map_get_next_key(map_fd, cur_key, &next_key);
Documentation/bpf/map_lpm_trie.rst-188- if (err)
--
drivers/net/ethernet/netronome/nfp/bpf/offload.c=305=static int
drivers/net/ethernet/netronome/nfp/bpf/offload.c:306:nfp_bpf_map_get_next_key(struct bpf_offloaded_map *offmap,
drivers/net/ethernet/netronome/nfp/bpf/offload.c-307- void *key, void *next_key)
--
drivers/net/ethernet/netronome/nfp/bpf/offload.c=322=static const struct bpf_map_dev_ops nfp_bpf_map_ops = {
drivers/net/ethernet/netronome/nfp/bpf/offload.c:323: .map_get_next_key = nfp_bpf_map_get_next_key,
drivers/net/ethernet/netronome/nfp/bpf/offload.c-324- .map_lookup_elem = nfp_bpf_map_lookup_entry,
--
include/linux/bpf.h=2599=void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu);
include/linux/bpf.h-2600-
include/linux/bpf.h:2601:struct bpf_map *bpf_map_get(u32 ufd);
include/linux/bpf.h:2602:struct bpf_map *bpf_map_get_with_uref(u32 ufd);
include/linux/bpf.h-2603-
include/linux/bpf.h-2604-/*
include/linux/bpf.h:2605: * The __bpf_map_get() and __btf_get_by_fd() functions parse a file
include/linux/bpf.h-2606- * descriptor and return a corresponding map or btf object.
--
include/linux/bpf.h-2608- * do not increase refcnt. To also increase refcnt use corresponding
include/linux/bpf.h:2609: * bpf_map_get() and btf_get_by_fd() functions.
include/linux/bpf.h-2610- */
include/linux/bpf.h-2611-
include/linux/bpf.h:2612:static inline struct bpf_map *__bpf_map_get(struct fd f)
include/linux/bpf.h-2613-{
--
include/linux/bpf.h=2647=int generic_map_delete_batch(struct bpf_map *map,
--
include/linux/bpf.h-2649- union bpf_attr __user *uattr);
include/linux/bpf.h:2650:struct bpf_map *bpf_map_get_curr_or_next(u32 *id);
include/linux/bpf.h-2651-struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id);
--
kernel/bpf/bpf_struct_ops.c=1311=static int bpf_struct_ops_map_link_detach(struct bpf_link *link)
--
kernel/bpf/bpf_struct_ops.c-1328- RCU_INIT_POINTER(st_link->map, NULL);
kernel/bpf/bpf_struct_ops.c:1329: /* Pair with bpf_map_get() in bpf_struct_ops_link_create() or
kernel/bpf/bpf_struct_ops.c-1330- * bpf_map_inc() in bpf_struct_ops_map_link_update().
--
kernel/bpf/bpf_struct_ops.c=1360=int bpf_struct_ops_link_create(union bpf_attr *attr)
--
kernel/bpf/bpf_struct_ops.c-1367-
kernel/bpf/bpf_struct_ops.c:1368: map = bpf_map_get(attr->link_create.map_fd);
kernel/bpf/bpf_struct_ops.c-1369- if (IS_ERR(map))
--
kernel/bpf/inode.c=71=static void *bpf_fd_probe_obj(u32 ufd, enum bpf_type *type)
--
kernel/bpf/inode.c-74-
kernel/bpf/inode.c:75: raw = bpf_map_get_with_uref(ufd);
kernel/bpf/inode.c-76- if (!IS_ERR(raw)) {
--
kernel/bpf/map_in_map.c=10=struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd)
--
kernel/bpf/map_in_map.c-15-
kernel/bpf/map_in_map.c:16: inner_map = __bpf_map_get(f);
kernel/bpf/map_in_map.c-17- if (IS_ERR(inner_map))
--
kernel/bpf/map_in_map.c=94=void *bpf_map_fd_get_ptr(struct bpf_map *map,
--
kernel/bpf/map_in_map.c-100-
kernel/bpf/map_in_map.c:101: inner_map = __bpf_map_get(f);
kernel/bpf/map_in_map.c-102- if (IS_ERR(inner_map))
--
kernel/bpf/map_iter.c=13=static void *bpf_map_seq_start(struct seq_file *seq, loff_t *pos)
--
kernel/bpf/map_iter.c-17-
kernel/bpf/map_iter.c:18: map = bpf_map_get_curr_or_next(&info->map_id);
kernel/bpf/map_iter.c-19- if (!map)
--
kernel/bpf/map_iter.c=27=static void *bpf_map_seq_next(struct seq_file *seq, void *v, loff_t *pos)
--
kernel/bpf/map_iter.c-33- bpf_map_put((struct bpf_map *)v);
kernel/bpf/map_iter.c:34: return bpf_map_get_curr_or_next(&info->map_id);
kernel/bpf/map_iter.c-35-}
--
kernel/bpf/map_iter.c=100=static int bpf_iter_attach_map(struct bpf_prog *prog,
--
kernel/bpf/map_iter.c-111-
kernel/bpf/map_iter.c:112: map = bpf_map_get_with_uref(linfo->map.map_fd);
kernel/bpf/map_iter.c-113- if (IS_ERR(map))
--
kernel/bpf/syscall.c=497=static void bpf_map_release_memcg(struct bpf_map *map)
--
kernel/bpf/syscall.c-502-
kernel/bpf/syscall.c:503:static struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map)
kernel/bpf/syscall.c-504-{
--
kernel/bpf/syscall.c=511=void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg,
--
kernel/bpf/syscall.c-513-{
kernel/bpf/syscall.c:514: *new_memcg = bpf_map_get_memcg(map);
kernel/bpf/syscall.c-515- *old_memcg = set_active_memcg(*new_memcg);
--
kernel/bpf/syscall.c=1625=EXPORT_SYMBOL_GPL(bpf_map_inc_with_uref);
kernel/bpf/syscall.c-1626-
kernel/bpf/syscall.c:1627:struct bpf_map *bpf_map_get(u32 ufd)
kernel/bpf/syscall.c-1628-{
kernel/bpf/syscall.c-1629- CLASS(fd, f)(ufd);
kernel/bpf/syscall.c:1630: struct bpf_map *map = __bpf_map_get(f);
kernel/bpf/syscall.c-1631-
--
kernel/bpf/syscall.c-1636-}
kernel/bpf/syscall.c:1637:EXPORT_SYMBOL_NS(bpf_map_get, "BPF_INTERNAL");
kernel/bpf/syscall.c-1638-
kernel/bpf/syscall.c:1639:struct bpf_map *bpf_map_get_with_uref(u32 ufd)
kernel/bpf/syscall.c-1640-{
kernel/bpf/syscall.c-1641- CLASS(fd, f)(ufd);
kernel/bpf/syscall.c:1642: struct bpf_map *map = __bpf_map_get(f);
kernel/bpf/syscall.c-1643-
--
kernel/bpf/syscall.c=1704=static int map_lookup_elem(union bpf_attr *attr)
--
kernel/bpf/syscall.c-1716- CLASS(fd, f)(attr->map_fd);
kernel/bpf/syscall.c:1717: map = __bpf_map_get(f);
kernel/bpf/syscall.c-1718- if (IS_ERR(map))
--
kernel/bpf/syscall.c=1766=static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
--
kernel/bpf/syscall.c-1778- CLASS(fd, f)(attr->map_fd);
kernel/bpf/syscall.c:1779: map = __bpf_map_get(f);
kernel/bpf/syscall.c-1780- if (IS_ERR(map))
--
kernel/bpf/syscall.c=1819=static int map_delete_elem(union bpf_attr *attr, bpfptr_t uattr)
--
kernel/bpf/syscall.c-1829- CLASS(fd, f)(attr->map_fd);
kernel/bpf/syscall.c:1830: map = __bpf_map_get(f);
kernel/bpf/syscall.c-1831- if (IS_ERR(map))
--
kernel/bpf/syscall.c=1872=static int map_get_next_key(union bpf_attr *attr)
--
kernel/bpf/syscall.c-1883- CLASS(fd, f)(attr->map_fd);
kernel/bpf/syscall.c:1884: map = __bpf_map_get(f);
kernel/bpf/syscall.c-1885- if (IS_ERR(map))
--
kernel/bpf/syscall.c=2135=static int map_lookup_and_delete_elem(union bpf_attr *attr)
--
kernel/bpf/syscall.c-2150- CLASS(fd, f)(attr->map_fd);
kernel/bpf/syscall.c:2151: map = __bpf_map_get(f);
kernel/bpf/syscall.c-2152- if (IS_ERR(map))
--
kernel/bpf/syscall.c=2226=static int map_freeze(const union bpf_attr *attr)
--
kernel/bpf/syscall.c-2234- CLASS(fd, f)(attr->map_fd);
kernel/bpf/syscall.c:2235: map = __bpf_map_get(f);
kernel/bpf/syscall.c-2236- if (IS_ERR(map))
--
kernel/bpf/syscall.c=4750=static int bpf_obj_get_next_id(const union bpf_attr *attr,
--
kernel/bpf/syscall.c-4775-
kernel/bpf/syscall.c:4776:struct bpf_map *bpf_map_get_curr_or_next(u32 *id)
kernel/bpf/syscall.c-4777-{
--
kernel/bpf/syscall.c=4833=static int bpf_prog_get_fd_by_id(const union bpf_attr *attr)
--
kernel/bpf/syscall.c-4857-
kernel/bpf/syscall.c:4858:static int bpf_map_get_fd_by_id(const union bpf_attr *attr)
kernel/bpf/syscall.c-4859-{
--
kernel/bpf/syscall.c=5011=static int bpf_prog_get_info_by_fd(struct file *file,
--
kernel/bpf/syscall.c-5299-
kernel/bpf/syscall.c:5300:static int bpf_map_get_info_by_fd(struct file *file,
kernel/bpf/syscall.c-5301- struct bpf_map *map,
--
kernel/bpf/syscall.c=5443=static int bpf_obj_get_info_by_fd(const union bpf_attr *attr,
--
kernel/bpf/syscall.c-5456- else if (fd_file(f)->f_op == &bpf_map_fops)
kernel/bpf/syscall.c:5457: return bpf_map_get_info_by_fd(fd_file(f), fd_file(f)->private_data, attr,
kernel/bpf/syscall.c-5458- uattr);
--
kernel/bpf/syscall.c=5655=static int bpf_map_do_batch(const union bpf_attr *attr,
--
kernel/bpf/syscall.c-5669-
kernel/bpf/syscall.c:5670: map = __bpf_map_get(f);
kernel/bpf/syscall.c-5671- if (IS_ERR(map))
--
kernel/bpf/syscall.c=5805=static int link_update_map(struct bpf_link *link, union bpf_attr *attr)
--
kernel/bpf/syscall.c-5809-
kernel/bpf/syscall.c:5810: new_map = bpf_map_get(attr->link_update.new_map_fd);
kernel/bpf/syscall.c-5811- if (IS_ERR(new_map))
--
kernel/bpf/syscall.c-5814- if (attr->link_update.flags & BPF_F_REPLACE) {
kernel/bpf/syscall.c:5815: old_map = bpf_map_get(attr->link_update.old_map_fd);
kernel/bpf/syscall.c-5816- if (IS_ERR(old_map)) {
--
kernel/bpf/syscall.c=6066=static int bpf_prog_bind_map(union bpf_attr *attr)
--
kernel/bpf/syscall.c-6082-
kernel/bpf/syscall.c:6083: map = bpf_map_get(attr->prog_bind_map.map_fd);
kernel/bpf/syscall.c-6084- if (IS_ERR(map)) {
--
kernel/bpf/syscall.c=6168=static int prog_assoc_struct_ops(union bpf_attr *attr)
--
kernel/bpf/syscall.c-6188-
kernel/bpf/syscall.c:6189: map = bpf_map_get(attr->prog_assoc_struct_ops.map_fd);
kernel/bpf/syscall.c-6190- if (IS_ERR(map)) {
--
kernel/bpf/syscall.c=6209=static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size)
--
kernel/bpf/syscall.c-6283- case BPF_MAP_GET_FD_BY_ID:
kernel/bpf/syscall.c:6284: err = bpf_map_get_fd_by_id(&attr);
kernel/bpf/syscall.c-6285- break;
--
kernel/bpf/verifier.c=18210=static int add_used_map(struct bpf_verifier_env *env, int fd)
--
kernel/bpf/verifier.c-18214-
kernel/bpf/verifier.c:18215: map = __bpf_map_get(f);
kernel/bpf/verifier.c-18216- if (IS_ERR(map)) {
--
kernel/bpf/verifier.c=19635=static int add_fd_from_fd_array(struct bpf_verifier_env *env, int fd)
--
kernel/bpf/verifier.c-19641-
kernel/bpf/verifier.c:19642: map = __bpf_map_get(f);
kernel/bpf/verifier.c-19643- if (!IS_ERR(map)) {
--
net/core/bpf_sk_storage.c=480=bpf_sk_storage_diag_alloc(const struct nlattr *nla_stgs)
--
net/core/bpf_sk_storage.c-506- int map_fd = nla_get_u32(nla);
net/core/bpf_sk_storage.c:507: struct bpf_map *map = bpf_map_get(map_fd);
net/core/bpf_sk_storage.c-508-
--
net/core/bpf_sk_storage.c=851=static int bpf_iter_attach_map(struct bpf_prog *prog,
--
net/core/bpf_sk_storage.c-860-
net/core/bpf_sk_storage.c:861: map = bpf_map_get_with_uref(linfo->map.map_fd);
net/core/bpf_sk_storage.c-862- if (IS_ERR(map))
--
net/core/sock_map.c=68=int sock_map_get_from_fd(const union bpf_attr *attr, struct bpf_prog *prog)
--
net/core/sock_map.c-76- CLASS(fd, f)(attr->target_fd);
net/core/sock_map.c:77: map = __bpf_map_get(f);
net/core/sock_map.c-78- if (IS_ERR(map))
--
net/core/sock_map.c=86=int sock_map_prog_detach(const union bpf_attr *attr, enum bpf_prog_type ptype)
--
net/core/sock_map.c-95- CLASS(fd, f)(attr->target_fd);
net/core/sock_map.c:96: map = __bpf_map_get(f);
net/core/sock_map.c-97- if (IS_ERR(map))
--
net/core/sock_map.c=1555=int sock_map_bpf_prog_query(const union bpf_attr *attr,
--
net/core/sock_map.c-1569- CLASS(fd, f)(attr->target_fd);
net/core/sock_map.c:1570: map = __bpf_map_get(f);
net/core/sock_map.c-1571- if (IS_ERR(map))
--
net/core/sock_map.c=1853=int sock_map_link_create(const union bpf_attr *attr, struct bpf_prog *prog)
--
net/core/sock_map.c-1863-
net/core/sock_map.c:1864: map = bpf_map_get_with_uref(attr->link_create.target_fd);
net/core/sock_map.c-1865- if (IS_ERR(map))
--
net/core/sock_map.c=1913=static int sock_map_iter_attach_target(struct bpf_prog *prog,
--
net/core/sock_map.c-1922-
net/core/sock_map.c:1923: map = bpf_map_get_with_uref(linfo->map.map_fd);
net/core/sock_map.c-1924- if (IS_ERR(map))
--
samples/bpf/cookie_uid_helper_example.c=196=static void print_table(void)
--
samples/bpf/cookie_uid_helper_example.c-202-
samples/bpf/cookie_uid_helper_example.c:203: while (bpf_map_get_next_key(map_fd, &curN, &nextN) > -1) {
samples/bpf/cookie_uid_helper_example.c-204- curN = nextN;
--
samples/bpf/lwt_len_hist_user.c=29=int main(int argc, char **argv)
--
samples/bpf/lwt_len_hist_user.c-44-
samples/bpf/lwt_len_hist_user.c:45: while (bpf_map_get_next_key(map_fd, &key, &next_key) == 0) {
samples/bpf/lwt_len_hist_user.c-46- if (next_key >= MAX_INDEX) {
--
samples/bpf/offwaketime_user.c=77=static void print_stacks(int fd)
--
samples/bpf/offwaketime_user.c-81-
samples/bpf/offwaketime_user.c:82: while (bpf_map_get_next_key(fd, &key, &next_key) == 0) {
samples/bpf/offwaketime_user.c-83- bpf_map_lookup_elem(fd, &next_key, &value);
--
samples/bpf/sampleip_user.c=89=static void print_ip_map(int fd)
--
samples/bpf/sampleip_user.c-99- key = 0, i = 0;
samples/bpf/sampleip_user.c:100: while (bpf_map_get_next_key(fd, &key, &next_key) == 0) {
samples/bpf/sampleip_user.c-101- bpf_map_lookup_elem(fd, &next_key, &value);
--
samples/bpf/sockex2_user.c=16=int main(int ac, char **argv)
--
samples/bpf/sockex2_user.c-51-
samples/bpf/sockex2_user.c:52: while (bpf_map_get_next_key(map_fd, &key, &next_key) == 0) {
samples/bpf/sockex2_user.c-53- bpf_map_lookup_elem(map_fd, &next_key, &value);
--
samples/bpf/sockex3_user.c=25=int main(int argc, char **argv)
--
samples/bpf/sockex3_user.c-84- printf("IP src.port -> dst.port bytes packets\n");
samples/bpf/sockex3_user.c:85: while (bpf_map_get_next_key(hash_map_fd, &key, &next_key) == 0) {
samples/bpf/sockex3_user.c-86- bpf_map_lookup_elem(hash_map_fd, &next_key, &value);
--
samples/bpf/spintest_user.c=10=int main(int ac, char **argv)
--
samples/bpf/spintest_user.c-57- printf("kprobing funcs:");
samples/bpf/spintest_user.c:58: while (bpf_map_get_next_key(map_fd, &key, &next_key) == 0) {
samples/bpf/spintest_user.c-59- bpf_map_lookup_elem(map_fd, &next_key, &value);
--
samples/bpf/spintest_user.c-72- key = 0;
samples/bpf/spintest_user.c:73: while (bpf_map_get_next_key(map_fd, &key, &next_key) == 0)
samples/bpf/spintest_user.c-74- bpf_map_delete_elem(map_fd, &next_key);
--
samples/bpf/test_map_in_map_user.c=35=static void check_map_id(int inner_map_fd, int map_in_map_fd, uint32_t key)
--
samples/bpf/test_map_in_map_user.c-40-
samples/bpf/test_map_in_map_user.c:41: ret = bpf_map_get_info_by_fd(inner_map_fd, &info, &info_len);
samples/bpf/test_map_in_map_user.c-42- assert(!ret);
--
samples/bpf/trace_event_user.c=100=static void print_stacks(void)
--
samples/bpf/trace_event_user.c-107- sys_read_seen = sys_write_seen = false;
samples/bpf/trace_event_user.c:108: while (bpf_map_get_next_key(fd, &key, &next_key) == 0) {
samples/bpf/trace_event_user.c-109- bpf_map_lookup_elem(fd, &next_key, &value);
--
samples/bpf/trace_event_user.c-120- /* clear stack map */
samples/bpf/trace_event_user.c:121: while (bpf_map_get_next_key(stack_map, &stackid, &next_id) == 0) {
samples/bpf/trace_event_user.c-122- bpf_map_delete_elem(stack_map, &next_id);
--
samples/bpf/tracex4_user.c=28=static void print_old_objects(int fd)
--
samples/bpf/tracex4_user.c-36- key = -1;
samples/bpf/tracex4_user.c:37: while (bpf_map_get_next_key(fd, &key, &next_key) == 0) {
samples/bpf/tracex4_user.c-38- bpf_map_lookup_elem(fd, &next_key, &v);
--
samples/bpf/tracex6_user.c=25=static void check_on_cpu(int cpu, struct perf_event_attr *attr)
--
samples/bpf/tracex6_user.c-45- /* Trigger the kprobe */
samples/bpf/tracex6_user.c:46: bpf_map_get_next_key(map_fd[1], &cpu, NULL);
samples/bpf/tracex6_user.c-47- /* Check the value */
--
tools/bpf/bpftool/btf.c=1210=build_btf_type_table(struct hashmap *tab, enum bpf_obj_type type,
--
tools/bpf/bpftool/btf.c-1230- case BPF_OBJ_MAP:
tools/bpf/bpftool/btf.c:1231: err = bpf_map_get_next_id(id, &id);
tools/bpf/bpftool/btf.c-1232- break;
--
tools/bpf/bpftool/btf.c-1253- case BPF_OBJ_MAP:
tools/bpf/bpftool/btf.c:1254: fd = bpf_map_get_fd_by_id_opts(id, &opts_ro);
tools/bpf/bpftool/btf.c-1255- break;
--
tools/bpf/bpftool/btf.c-1273- else
tools/bpf/bpftool/btf.c:1274: err = bpf_map_get_info_by_fd(fd, info, len);
tools/bpf/bpftool/btf.c-1275- close(fd);
--
tools/bpf/bpftool/common.c=973=static int map_fd_by_name(char *name, int **fds,
--
tools/bpf/bpftool/common.c-985-
tools/bpf/bpftool/common.c:986: err = bpf_map_get_next_id(id, &id);
tools/bpf/bpftool/common.c-987- if (err) {
--
tools/bpf/bpftool/common.c-996- opts_ro.open_flags = BPF_F_RDONLY;
tools/bpf/bpftool/common.c:997: fd = bpf_map_get_fd_by_id_opts(id, &opts_ro);
tools/bpf/bpftool/common.c-998- if (fd < 0) {
--
tools/bpf/bpftool/common.c-1003-
tools/bpf/bpftool/common.c:1004: err = bpf_map_get_info_by_fd(fd, &info, &len);
tools/bpf/bpftool/common.c-1005- if (err) {
--
tools/bpf/bpftool/common.c-1020- close(fd);
tools/bpf/bpftool/common.c:1021: fd = bpf_map_get_fd_by_id_opts(id, opts);
tools/bpf/bpftool/common.c-1022- if (fd < 0) {
--
tools/bpf/bpftool/common.c=1048=int map_parse_fds(int *argc, char ***argv, int **fds, __u32 open_flags)
--
tools/bpf/bpftool/common.c-1067-
tools/bpf/bpftool/common.c:1068: (*fds)[0] = bpf_map_get_fd_by_id_opts(id, &opts);
tools/bpf/bpftool/common.c-1069- if ((*fds)[0] < 0) {
--
tools/bpf/bpftool/common.c=1134=int map_parse_fd_and_info(int *argc, char ***argv, struct bpf_map_info *info,
--
tools/bpf/bpftool/common.c-1143-
tools/bpf/bpftool/common.c:1144: err = bpf_map_get_info_by_fd(fd, info, info_len);
tools/bpf/bpftool/common.c-1145- if (err) {
--
tools/bpf/bpftool/map.c=629=static int do_show_subset(int argc, char **argv)
--
tools/bpf/bpftool/map.c-648- for (i = 0; i < nb_fds; i++) {
tools/bpf/bpftool/map.c:649: err = bpf_map_get_info_by_fd(fds[i], &info, &len);
tools/bpf/bpftool/map.c-650- if (err) {
--
tools/bpf/bpftool/map.c=673=static int do_show(int argc, char **argv)
--
tools/bpf/bpftool/map.c-703- while (true) {
tools/bpf/bpftool/map.c:704: err = bpf_map_get_next_id(id, &id);
tools/bpf/bpftool/map.c-705- if (err) {
--
tools/bpf/bpftool/map.c-712-
tools/bpf/bpftool/map.c:713: fd = bpf_map_get_fd_by_id_opts(id, &opts);
tools/bpf/bpftool/map.c-714- if (fd < 0) {
--
tools/bpf/bpftool/map.c-721-
tools/bpf/bpftool/map.c:722: err = bpf_map_get_info_by_fd(fd, &info, &len);
tools/bpf/bpftool/map.c-723- if (err) {
--
tools/bpf/bpftool/map.c=771=static int maps_have_btf(int *fds, int nb_fds)
--
tools/bpf/bpftool/map.c-777- for (i = 0; i < nb_fds; i++) {
tools/bpf/bpftool/map.c:778: err = bpf_map_get_info_by_fd(fds[i], &info, &len);
tools/bpf/bpftool/map.c-779- if (err) {
--
tools/bpf/bpftool/map.c=826=map_dump(int fd, struct bpf_map_info *info, json_writer_t *wtr,
--
tools/bpf/bpftool/map.c-868- while (true) {
tools/bpf/bpftool/map.c:869: err = bpf_map_get_next_key(fd, prev_key, key);
tools/bpf/bpftool/map.c-870- if (err) {
--
tools/bpf/bpftool/map.c=898=static int do_dump(int argc, char **argv)
--
tools/bpf/bpftool/map.c-938- for (i = 0; i < nb_fds; i++) {
tools/bpf/bpftool/map.c:939: if (bpf_map_get_info_by_fd(fds[i], &info, &len)) {
tools/bpf/bpftool/map.c-940- p_err("can't get map info: %s", strerror(errno));
--
tools/bpf/bpftool/map.c=1123=static int do_getnext(int argc, char **argv)
--
tools/bpf/bpftool/map.c-1155-
tools/bpf/bpftool/map.c:1156: err = bpf_map_get_next_key(fd, key, nextkey);
tools/bpf/bpftool/map.c-1157- if (err) {
--
tools/bpf/bpftool/prog.c=220=static void *find_metadata(int prog_fd, struct bpf_map_info *map_info)
--
tools/bpf/bpftool/prog.c-256- for (i = 0; i < prog_info.nr_map_ids; i++) {
tools/bpf/bpftool/prog.c:257: map_fd = bpf_map_get_fd_by_id(map_ids[i]);
tools/bpf/bpftool/prog.c-258- if (map_fd < 0)
--
tools/bpf/bpftool/prog.c-262- map_info_len = sizeof(*map_info);
tools/bpf/bpftool/prog.c:263: ret = bpf_map_get_info_by_fd(map_fd, map_info, &map_info_len);
tools/bpf/bpftool/prog.c-264- if (ret < 0) {
--
tools/bpf/bpftool/struct_ops.c=130=static int get_next_struct_ops_map(const char *name, int *res_fd,
--
tools/bpf/bpftool/struct_ops.c-136- while (true) {
tools/bpf/bpftool/struct_ops.c:137: err = bpf_map_get_next_id(id, &id);
tools/bpf/bpftool/struct_ops.c-138- if (err) {
--
tools/bpf/bpftool/struct_ops.c-144-
tools/bpf/bpftool/struct_ops.c:145: fd = bpf_map_get_fd_by_id(id);
tools/bpf/bpftool/struct_ops.c-146- if (fd < 0) {
--
tools/bpf/bpftool/struct_ops.c-153-
tools/bpf/bpftool/struct_ops.c:154: err = bpf_map_get_info_by_fd(fd, info, &info_len);
tools/bpf/bpftool/struct_ops.c-155- if (err) {
--
tools/bpf/bpftool/struct_ops.c=235=static struct res do_one_id(const char *id_str, work_func func, void *data,
--
tools/bpf/bpftool/struct_ops.c-251-
tools/bpf/bpftool/struct_ops.c:252: fd = bpf_map_get_fd_by_id(id);
tools/bpf/bpftool/struct_ops.c-253- if (fd < 0) {
]
]
|
| 164/2 |
2026/05/18 18:58 |
llm |
debugger |
0m
Model:
Tokens:
input: 190492
output: 45
thoughts: 158
Thoughts: **BPF Verification Puzzles**
Okay, so I'm trying to wrap my head around this BPF verification process. I was under the impression that `bpf_prog_load` would fail if `ringbuf_map_fd` was set to `-1` under certain circumstances, but I'm questioning that now. Does the verifier only flag errors based on executed instructions? No, that's not right. The verifier, `bpf_check`, examines *all* instructions.
Right, I'm remembering `bpf_insn_map_fd_t`. The immediate value, `imm`, *is* supposed to hold the file descriptor. So, if `imm` is `-1`, then `replace_map_fd_with_map_ptr` in `bpf_check` should fail, shouldn't it? That makes sense.
Aha! `replace_map_fd_with_map_ptr` is defined within `kernel/bpf/verifier.c`. Let's dig deeper. The critical piece here is the `resolve_pseudo_ldimm64` function, also in that file. That's the likely key area to examine. Gotta understand how it handles the `-1` file descriptor and potential failures.
|
| 165/2 |
2026/05/18 18:59 |
tool |
codesearch-definition-source |
0m
Error:
Args: map[ContextFile:kernel/bpf/verifier.c IncludeLines:true Name:resolve_pseudo_ldimm64]
Results: map[SourceCode: SourceFile:]
|
| 166/2 |
2026/05/18 18:59 |
llm |
debugger |
0m
Model:
Tokens:
input: 190986
output: 22
thoughts: 15
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Investigating `resolve_pseudo_ldimm64`**
Right, let's get down to brass tacks. My initial thought is to zero in on `resolve_pseudo_ldimm64`. It's a common pattern, and I've likely encountered it countless times, so it's a good place to start my investigation. It has all the hallmarks of something I've dealt with before, so I feel I can apply my expertise to determine a starting point to diagnose the issue. There could be any number of reasons why I'm prompted to search this way, but this is the appropriate first line of investigation. Let's see what the code reveals.
|
| 167/2 |
2026/05/18 18:59 |
tool |
grepper |
0m
Error:
Args: map[Expression:resolve_pseudo_ldimm64]
Results:
|
| 168/2 |
2026/05/18 18:59 |
llm |
debugger |
0m
Model:
Tokens:
input: 191198
output: 24
thoughts: 37
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Investigating Potential BPF Verifier Weakness**
Right, let's get down to brass tacks. I need to dive into the BPF verifier in the kernel. My immediate focus is on potential vulnerabilities or interesting behaviors related to `pseudo_ldimm64` and, more generally, `BPF_PSEUDO_MAP_FD`. I suspect these might be key areas to examine for unintended side effects or, let's say, *creative* uses. Specifically, `kernel/bpf/verifier.c` is the obvious place to start digging. That file is the heart of the verifier, so it'll have the crucial logic. I'm thinking specifically about how these pseudo-instructions and map file descriptors interact with the control flow and data validation checks. I want to see how the verifier handles them; does it account for all possible edge cases and combinations? Does it properly track and restrict their uses? It's a hunch, but these often-overlooked components are where the juiciest issues are. My hope is to discover something new, a gap in the armor.
|
| 169/2 |
2026/05/18 18:59 |
tool |
grepper |
0m
Args: map[Expression:BPF_PSEUDO_MAP_FD]
Results: map[Output:drivers/net/ethernet/netronome/nfp/bpf/jit.c=4401=static int nfp_bpf_replace_map_ptrs(struct nfp_prog *nfp_prog)
--
drivers/net/ethernet/netronome/nfp/bpf/jit.c-4413- if (meta1->insn.code != (BPF_LD | BPF_IMM | BPF_DW) ||
drivers/net/ethernet/netronome/nfp/bpf/jit.c:4414: meta1->insn.src_reg != BPF_PSEUDO_MAP_FD)
drivers/net/ethernet/netronome/nfp/bpf/jit.c-4415- continue;
--
include/linux/filter.h=258=static inline bool insn_is_cast_user(const struct bpf_insn *insn)
--
include/linux/filter.h-284-#define BPF_LD_MAP_FD(DST, MAP_FD) \
include/linux/filter.h:285: BPF_LD_IMM64_RAW(DST, BPF_PSEUDO_MAP_FD, MAP_FD)
include/linux/filter.h-286-
--
include/uapi/linux/bpf.h=1323=enum {
--
include/uapi/linux/bpf.h-1342- */
include/uapi/linux/bpf.h:1343:#define BPF_PSEUDO_MAP_FD 1
include/uapi/linux/bpf.h-1344-#define BPF_PSEUDO_MAP_IDX 5
--
kernel/bpf/const_fold.c=50=static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *ci_out,
--
kernel/bpf/const_fold.c-153- dst->val = env->insn_aux_data[idx].map_off;
kernel/bpf/const_fold.c:154: } else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
kernel/bpf/const_fold.c-155- insn->src_reg == BPF_PSEUDO_MAP_IDX) {
--
kernel/bpf/core.c=300=int bpf_prog_calc_tag(struct bpf_prog *fp)
--
kernel/bpf/core.c-317- dst[i].code == (BPF_LD | BPF_IMM | BPF_DW) &&
kernel/bpf/core.c:318: (dst[i].src_reg == BPF_PSEUDO_MAP_FD ||
kernel/bpf/core.c-319- dst[i].src_reg == BPF_PSEUDO_MAP_VALUE)) {
--
kernel/bpf/disasm.c=186=void print_bpf_insn(const struct bpf_insn_cbs *cbs,
--
kernel/bpf/disasm.c-324- u64 imm = ((u64)(insn + 1)->imm << 32) | (u32)insn->imm;
kernel/bpf/disasm.c:325: bool is_ptr = insn->src_reg == BPF_PSEUDO_MAP_FD ||
kernel/bpf/disasm.c-326- insn->src_reg == BPF_PSEUDO_MAP_VALUE;
--
kernel/bpf/syscall.c=4894=static const struct bpf_map *bpf_map_from_imm(const struct bpf_prog *prog,
--
kernel/bpf/syscall.c-4904- if (map == (void *)addr) {
kernel/bpf/syscall.c:4905: *type = BPF_PSEUDO_MAP_FD;
kernel/bpf/syscall.c-4906- goto out;
--
kernel/bpf/verifier.c=16524=static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
--
kernel/bpf/verifier.c-16606- /* We want reg->id to be same (0) as map_value is not distinct */
kernel/bpf/verifier.c:16607: } else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
kernel/bpf/verifier.c-16608- insn->src_reg == BPF_PSEUDO_MAP_IDX) {
--
kernel/bpf/verifier.c=18416=static int check_and_resolve_insns(struct bpf_verifier_env *env)
--
kernel/bpf/verifier.c-18478- break;
kernel/bpf/verifier.c:18479: case BPF_PSEUDO_MAP_FD:
kernel/bpf/verifier.c-18480- case BPF_PSEUDO_MAP_IDX:
--
kernel/bpf/verifier.c-18513-
kernel/bpf/verifier.c:18514: if (insn[0].src_reg == BPF_PSEUDO_MAP_FD ||
kernel/bpf/verifier.c-18515- insn[0].src_reg == BPF_PSEUDO_MAP_IDX) {
--
samples/bpf/bpf_insn.h=6=struct bpf_insn;
--
samples/bpf/bpf_insn.h-97-
samples/bpf/bpf_insn.h:98:#ifndef BPF_PSEUDO_MAP_FD
samples/bpf/bpf_insn.h:99:# define BPF_PSEUDO_MAP_FD 1
samples/bpf/bpf_insn.h-100-#endif
--
samples/bpf/bpf_insn.h-103-#define BPF_LD_MAP_FD(DST, MAP_FD) \
samples/bpf/bpf_insn.h:104: BPF_LD_IMM64_RAW(DST, BPF_PSEUDO_MAP_FD, MAP_FD)
samples/bpf/bpf_insn.h-105-
--
tools/bpf/bpftool/xlated_dumper.c=194=static const char *print_imm(void *private_data,
--
tools/bpf/bpftool/xlated_dumper.c-199-
tools/bpf/bpftool/xlated_dumper.c:200: if (insn->src_reg == BPF_PSEUDO_MAP_FD)
tools/bpf/bpftool/xlated_dumper.c-201- snprintf(dd->scratch_buff, sizeof(dd->scratch_buff),
--
tools/include/linux/filter.h-341-#define BPF_LD_MAP_FD(DST, MAP_FD) \
tools/include/linux/filter.h:342: BPF_LD_IMM64_RAW_FULL(DST, BPF_PSEUDO_MAP_FD, 0, 0, \
tools/include/linux/filter.h-343- MAP_FD, 0)
--
tools/include/uapi/linux/bpf.h=1323=enum {
--
tools/include/uapi/linux/bpf.h-1342- */
tools/include/uapi/linux/bpf.h:1343:#define BPF_PSEUDO_MAP_FD 1
tools/include/uapi/linux/bpf.h-1344-#define BPF_PSEUDO_MAP_IDX 5
--
tools/lib/bpf/libbpf.c=4572=static int bpf_prog_compute_hash(struct bpf_program *prog)
--
tools/lib/bpf/libbpf.c-4586- if (purged[i].code == (BPF_LD | BPF_IMM | BPF_DW) &&
tools/lib/bpf/libbpf.c:4587: (purged[i].src_reg == BPF_PSEUDO_MAP_FD ||
tools/lib/bpf/libbpf.c-4588- purged[i].src_reg == BPF_PSEUDO_MAP_VALUE)) {
--
tools/lib/bpf/libbpf.c=6411=bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
--
tools/lib/bpf/libbpf.c-6427- } else if (map->autocreate) {
tools/lib/bpf/libbpf.c:6428: insn[0].src_reg = BPF_PSEUDO_MAP_FD;
tools/lib/bpf/libbpf.c-6429- insn[0].imm = map->fd;
--
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c=73=__naked void invalid_cgroup_storage_access_2(void)
--
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c-83- : __imm(bpf_get_local_storage),
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c:84: __imm_insn(ld_map_fd, BPF_RAW_INSN(BPF_LD | BPF_DW | BPF_IMM, BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, 1))
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c-85- : __clobber_all);
--
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c=212=__naked void cpu_cgroup_storage_access_2(void)
--
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c-222- : __imm(bpf_get_local_storage),
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c:223: __imm_insn(ld_map_fd, BPF_RAW_INSN(BPF_LD | BPF_DW | BPF_IMM, BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, 1))
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c-224- : __clobber_all);
--
tools/testing/selftests/bpf/progs/verifier_map_ret_val.c=20=__naked void map_fd_for_function_call(void)
--
tools/testing/selftests/bpf/progs/verifier_map_ret_val.c-32- : __imm(bpf_map_delete_elem),
tools/testing/selftests/bpf/progs/verifier_map_ret_val.c:33: __imm_insn(ld_map_fd, BPF_RAW_INSN(BPF_LD | BPF_DW | BPF_IMM, BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, 0))
tools/testing/selftests/bpf/progs/verifier_map_ret_val.c-34- : __clobber_all);
--
tools/testing/selftests/bpf/verifier/direct_value_access.c-310- BPF_MOV64_IMM(BPF_REG_0, 1),
tools/testing/selftests/bpf/verifier/direct_value_access.c:311: BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, ~0, 0, 0, 0),
tools/testing/selftests/bpf/verifier/direct_value_access.c-312- BPF_EXIT_INSN(),
--
tools/testing/selftests/bpf/verifier/direct_value_access.c-321- BPF_MOV64_IMM(BPF_REG_0, 1),
tools/testing/selftests/bpf/verifier/direct_value_access.c:322: BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, ~0, 0, 0),
tools/testing/selftests/bpf/verifier/direct_value_access.c-323- BPF_EXIT_INSN(),
--
tools/testing/selftests/bpf/verifier/direct_value_access.c-332- BPF_MOV64_IMM(BPF_REG_0, 1),
tools/testing/selftests/bpf/verifier/direct_value_access.c:333: BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, ~0, ~0, 0, 0),
tools/testing/selftests/bpf/verifier/direct_value_access.c-334- BPF_EXIT_INSN(),
--
tools/testing/selftests/bpf/verifier/direct_value_access.c-343- BPF_MOV64_IMM(BPF_REG_0, 1),
tools/testing/selftests/bpf/verifier/direct_value_access.c:344: BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, 0, 0, 47),
tools/testing/selftests/bpf/verifier/direct_value_access.c-345- BPF_EXIT_INSN(),
--
tools/testing/selftests/bpf/verifier/ld_imm64.c-138- BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, BPF_REG_1,
tools/testing/selftests/bpf/verifier/ld_imm64.c:139: BPF_PSEUDO_MAP_FD, 0, 0),
tools/testing/selftests/bpf/verifier/ld_imm64.c-140- BPF_RAW_INSN(0, 0, 0, 0, 0xfefefe),
]
|
| 170/2 |
2026/05/18 18:59 |
llm |
debugger |
0m
Model:
Tokens:
input: 195232
output: 48
thoughts: 307
Thoughts: **Analyzing the BPF Instruction**
Okay, first things first, I need to confirm that `BPF_PSEUDO_MAP_FD` is indeed assigned the value 1. Now, let's dive into the reproducer's `insn1`. Examining the instruction directly:
```c
// insn1: bpf_insn_map_fd_t[const[BPF_REG_1, int8:4], ringbuf_map_fd] {
// code: const = 0x18 (1 bytes)
// dst: const = 0x1 (0 bytes)
// src: const = 0x1 (1 bytes)
// off: const = 0x0 (2 bytes)
// imm: ringbuf_map_fd (resource)
// code2: const = 0x0 (1 bytes)
// regs2: const = 0x0 (1 bytes)
// off2: const = 0x0 (2 bytes)
// imm2: const = 0x0 (4 bytes)
// }
```
Okay, the crucial part here is that `src` is set to 1. Which, as I've confirmed, corresponds to `BPF_PSEUDO_MAP_FD`. Also, and this is important, `imm` is -1. This means, the `bpf_check` function will subsequently call the `replace_map_fd_with_map_ptr` function. Now, I need to check the logic behind that function. Time to consult `kernel/bpf/verifier.c`.
|
| 171/2 |
2026/05/18 18:59 |
tool |
codesearch-definition-source |
0m
Error:
Args: map[ContextFile:kernel/bpf/verifier.c IncludeLines:true Name:replace_map_fd_with_map_ptr]
Results: map[SourceCode: SourceFile:]
|
| 172/2 |
2026/05/18 18:59 |
llm |
debugger |
0m
Model:
Tokens:
input: 195969
output: 24
thoughts: 28
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Diving into BPF Verification for Pseudo-Map File Descriptors**
Alright, time to dig into the BPF verifier's handling of pseudo-map file descriptors. Specifically, I need to understand how the `BPF_PSEUDO_MAP_FD` pseudo-instruction is evaluated within the `kernel/bpf/verifier.c` file. I'm assuming there's some interesting logic there regarding how these special file descriptors are treated during program verification. I expect to see the usual checks on map types, access permissions, bounds, and potentially some unique verification rules given the pseudo-nature of this FD. Probably something to do with sanitizing the file descriptor early, and making sure that all subsequent accesses are valid. I'll need to pay close attention to the data flow analysis surrounding any uses of that pseudo-instruction to understand if there are any potential vulnerabilities or nuances in its implementation that could be exploited. Let's see what the code reveals.
|
| 173/2 |
2026/05/18 18:59 |
tool |
grepper |
0m
Args: map[Expression:BPF_PSEUDO_MAP_FD]
Results: map[Output:drivers/net/ethernet/netronome/nfp/bpf/jit.c=4401=static int nfp_bpf_replace_map_ptrs(struct nfp_prog *nfp_prog)
--
drivers/net/ethernet/netronome/nfp/bpf/jit.c-4413- if (meta1->insn.code != (BPF_LD | BPF_IMM | BPF_DW) ||
drivers/net/ethernet/netronome/nfp/bpf/jit.c:4414: meta1->insn.src_reg != BPF_PSEUDO_MAP_FD)
drivers/net/ethernet/netronome/nfp/bpf/jit.c-4415- continue;
--
include/linux/filter.h=258=static inline bool insn_is_cast_user(const struct bpf_insn *insn)
--
include/linux/filter.h-284-#define BPF_LD_MAP_FD(DST, MAP_FD) \
include/linux/filter.h:285: BPF_LD_IMM64_RAW(DST, BPF_PSEUDO_MAP_FD, MAP_FD)
include/linux/filter.h-286-
--
include/uapi/linux/bpf.h=1323=enum {
--
include/uapi/linux/bpf.h-1342- */
include/uapi/linux/bpf.h:1343:#define BPF_PSEUDO_MAP_FD 1
include/uapi/linux/bpf.h-1344-#define BPF_PSEUDO_MAP_IDX 5
--
kernel/bpf/const_fold.c=50=static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *ci_out,
--
kernel/bpf/const_fold.c-153- dst->val = env->insn_aux_data[idx].map_off;
kernel/bpf/const_fold.c:154: } else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
kernel/bpf/const_fold.c-155- insn->src_reg == BPF_PSEUDO_MAP_IDX) {
--
kernel/bpf/core.c=300=int bpf_prog_calc_tag(struct bpf_prog *fp)
--
kernel/bpf/core.c-317- dst[i].code == (BPF_LD | BPF_IMM | BPF_DW) &&
kernel/bpf/core.c:318: (dst[i].src_reg == BPF_PSEUDO_MAP_FD ||
kernel/bpf/core.c-319- dst[i].src_reg == BPF_PSEUDO_MAP_VALUE)) {
--
kernel/bpf/disasm.c=186=void print_bpf_insn(const struct bpf_insn_cbs *cbs,
--
kernel/bpf/disasm.c-324- u64 imm = ((u64)(insn + 1)->imm << 32) | (u32)insn->imm;
kernel/bpf/disasm.c:325: bool is_ptr = insn->src_reg == BPF_PSEUDO_MAP_FD ||
kernel/bpf/disasm.c-326- insn->src_reg == BPF_PSEUDO_MAP_VALUE;
--
kernel/bpf/syscall.c=4894=static const struct bpf_map *bpf_map_from_imm(const struct bpf_prog *prog,
--
kernel/bpf/syscall.c-4904- if (map == (void *)addr) {
kernel/bpf/syscall.c:4905: *type = BPF_PSEUDO_MAP_FD;
kernel/bpf/syscall.c-4906- goto out;
--
kernel/bpf/verifier.c=16524=static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
--
kernel/bpf/verifier.c-16606- /* We want reg->id to be same (0) as map_value is not distinct */
kernel/bpf/verifier.c:16607: } else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
kernel/bpf/verifier.c-16608- insn->src_reg == BPF_PSEUDO_MAP_IDX) {
--
kernel/bpf/verifier.c=18416=static int check_and_resolve_insns(struct bpf_verifier_env *env)
--
kernel/bpf/verifier.c-18478- break;
kernel/bpf/verifier.c:18479: case BPF_PSEUDO_MAP_FD:
kernel/bpf/verifier.c-18480- case BPF_PSEUDO_MAP_IDX:
--
kernel/bpf/verifier.c-18513-
kernel/bpf/verifier.c:18514: if (insn[0].src_reg == BPF_PSEUDO_MAP_FD ||
kernel/bpf/verifier.c-18515- insn[0].src_reg == BPF_PSEUDO_MAP_IDX) {
--
samples/bpf/bpf_insn.h=6=struct bpf_insn;
--
samples/bpf/bpf_insn.h-97-
samples/bpf/bpf_insn.h:98:#ifndef BPF_PSEUDO_MAP_FD
samples/bpf/bpf_insn.h:99:# define BPF_PSEUDO_MAP_FD 1
samples/bpf/bpf_insn.h-100-#endif
--
samples/bpf/bpf_insn.h-103-#define BPF_LD_MAP_FD(DST, MAP_FD) \
samples/bpf/bpf_insn.h:104: BPF_LD_IMM64_RAW(DST, BPF_PSEUDO_MAP_FD, MAP_FD)
samples/bpf/bpf_insn.h-105-
--
tools/bpf/bpftool/xlated_dumper.c=194=static const char *print_imm(void *private_data,
--
tools/bpf/bpftool/xlated_dumper.c-199-
tools/bpf/bpftool/xlated_dumper.c:200: if (insn->src_reg == BPF_PSEUDO_MAP_FD)
tools/bpf/bpftool/xlated_dumper.c-201- snprintf(dd->scratch_buff, sizeof(dd->scratch_buff),
--
tools/include/linux/filter.h-341-#define BPF_LD_MAP_FD(DST, MAP_FD) \
tools/include/linux/filter.h:342: BPF_LD_IMM64_RAW_FULL(DST, BPF_PSEUDO_MAP_FD, 0, 0, \
tools/include/linux/filter.h-343- MAP_FD, 0)
--
tools/include/uapi/linux/bpf.h=1323=enum {
--
tools/include/uapi/linux/bpf.h-1342- */
tools/include/uapi/linux/bpf.h:1343:#define BPF_PSEUDO_MAP_FD 1
tools/include/uapi/linux/bpf.h-1344-#define BPF_PSEUDO_MAP_IDX 5
--
tools/lib/bpf/libbpf.c=4572=static int bpf_prog_compute_hash(struct bpf_program *prog)
--
tools/lib/bpf/libbpf.c-4586- if (purged[i].code == (BPF_LD | BPF_IMM | BPF_DW) &&
tools/lib/bpf/libbpf.c:4587: (purged[i].src_reg == BPF_PSEUDO_MAP_FD ||
tools/lib/bpf/libbpf.c-4588- purged[i].src_reg == BPF_PSEUDO_MAP_VALUE)) {
--
tools/lib/bpf/libbpf.c=6411=bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
--
tools/lib/bpf/libbpf.c-6427- } else if (map->autocreate) {
tools/lib/bpf/libbpf.c:6428: insn[0].src_reg = BPF_PSEUDO_MAP_FD;
tools/lib/bpf/libbpf.c-6429- insn[0].imm = map->fd;
--
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c=73=__naked void invalid_cgroup_storage_access_2(void)
--
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c-83- : __imm(bpf_get_local_storage),
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c:84: __imm_insn(ld_map_fd, BPF_RAW_INSN(BPF_LD | BPF_DW | BPF_IMM, BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, 1))
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c-85- : __clobber_all);
--
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c=212=__naked void cpu_cgroup_storage_access_2(void)
--
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c-222- : __imm(bpf_get_local_storage),
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c:223: __imm_insn(ld_map_fd, BPF_RAW_INSN(BPF_LD | BPF_DW | BPF_IMM, BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, 1))
tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c-224- : __clobber_all);
--
tools/testing/selftests/bpf/progs/verifier_map_ret_val.c=20=__naked void map_fd_for_function_call(void)
--
tools/testing/selftests/bpf/progs/verifier_map_ret_val.c-32- : __imm(bpf_map_delete_elem),
tools/testing/selftests/bpf/progs/verifier_map_ret_val.c:33: __imm_insn(ld_map_fd, BPF_RAW_INSN(BPF_LD | BPF_DW | BPF_IMM, BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, 0))
tools/testing/selftests/bpf/progs/verifier_map_ret_val.c-34- : __clobber_all);
--
tools/testing/selftests/bpf/verifier/direct_value_access.c-310- BPF_MOV64_IMM(BPF_REG_0, 1),
tools/testing/selftests/bpf/verifier/direct_value_access.c:311: BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, ~0, 0, 0, 0),
tools/testing/selftests/bpf/verifier/direct_value_access.c-312- BPF_EXIT_INSN(),
--
tools/testing/selftests/bpf/verifier/direct_value_access.c-321- BPF_MOV64_IMM(BPF_REG_0, 1),
tools/testing/selftests/bpf/verifier/direct_value_access.c:322: BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, ~0, 0, 0),
tools/testing/selftests/bpf/verifier/direct_value_access.c-323- BPF_EXIT_INSN(),
--
tools/testing/selftests/bpf/verifier/direct_value_access.c-332- BPF_MOV64_IMM(BPF_REG_0, 1),
tools/testing/selftests/bpf/verifier/direct_value_access.c:333: BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, ~0, ~0, 0, 0),
tools/testing/selftests/bpf/verifier/direct_value_access.c-334- BPF_EXIT_INSN(),
--
tools/testing/selftests/bpf/verifier/direct_value_access.c-343- BPF_MOV64_IMM(BPF_REG_0, 1),
tools/testing/selftests/bpf/verifier/direct_value_access.c:344: BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_FD, 0, 0, 0, 47),
tools/testing/selftests/bpf/verifier/direct_value_access.c-345- BPF_EXIT_INSN(),
--
tools/testing/selftests/bpf/verifier/ld_imm64.c-138- BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, BPF_REG_1,
tools/testing/selftests/bpf/verifier/ld_imm64.c:139: BPF_PSEUDO_MAP_FD, 0, 0),
tools/testing/selftests/bpf/verifier/ld_imm64.c-140- BPF_RAW_INSN(0, 0, 0, 0, 0xfefefe),
]
|
| 174/2 |
2026/05/18 18:59 |
llm |
debugger |
0m
Model:
Tokens:
input: 199963
output: 25
thoughts: 54
Thoughts: Here's my thought process, as it stands:
**Tracing a Function Call Path**
Okay, so I'm trying to understand how `add_used_map` is being used, and the crucial piece is that it's called from `replace_map_fd_with_map_ptr` within `kernel/bpf/verifier.c`. My instinct is to zero in on that function, `replace_map_fd_with_map_ptr`. I need to figure out the context of that call. What's the purpose of `replace_map_fd_with_map_ptr`? What's *its* role in the broader verifier workflow? Understanding that will give me the necessary information to grasp what `add_used_map` is actually *doing* in this specific scenario. I need to refresh my memory on the interactions of these functions within the BPF verifier. Let's dig into that function and see what's what.
|
| 175/2 |
2026/05/18 18:59 |
tool |
grepper |
0m
Error:
Args: map[Expression:replace_map_fd_with_map_ptr]
Results:
|
| 176/2 |
2026/05/18 18:59 |
llm |
debugger |
1m
Model:
Tokens:
input: 200262
output: 22
thoughts: 16
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Hmm, The Function Name is Now Clear**
Right, so that function I was trying to recall... it's `resolve_pseudo_ldimm64`. That name finally clicks. It's a pseudo instruction, obviously, but now the whole thing makes more sense. I've encountered this before, and it was a small detail that I needed to remember.
|
| 177/2 |
2026/05/18 19:00 |
tool |
grepper |
0m
Error:
Args: map[Expression:resolve_pseudo_ldimm64]
Results:
|
| 178/2 |
2026/05/18 19:00 |
llm |
debugger |
0m
Model:
Tokens:
input: 200410
output: 45
thoughts: 25
Thoughts: Here's my take on those thoughts, tailored for an expert audience:
**Diving into the BPF Verifier: A Closer Look at `check_and_resolve_insns`**
Okay, so we're looking at something potentially fishy. Time to dig into `check_and_resolve_insns` within `kernel/bpf/verifier.c`. This is where the magic (or the problems) happen, right? This function's probably responsible for the heavy lifting of instruction analysis and any necessary resolutions before a BPF program gets a chance to run. I need to refresh my memory on exactly how it interacts with the instruction set and operand validation. I'm especially interested in how it handles control flow and jump targets; those are often the trickiest parts to get right securely. I'll need to figure out what are the edge-cases that it does or does not cover. I suspect this function touches the register state tracking too, it is essential to ensure that sensitive data isn't leaking. It's time to trace the control flow of this function; I wonder where that takes me.
|
| 179/2 |
2026/05/18 19:01 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/verifier.c IncludeLines:true Name:check_and_resolve_insns]
Results: map[SourceCode:18408: /*
18409: * Check that insns are sane and rewrite pseudo imm in ld_imm64 instructions:
18410: *
18411: * 1. if it accesses map FD, replace it with actual map pointer.
18412: * 2. if it accesses btf_id of a VAR, replace it with pointer to the var.
18413: *
18414: * NOTE: btf_vmlinux is required for converting pseudo btf_id.
18415: */
18416: static int check_and_resolve_insns(struct bpf_verifier_env *env)
18417: {
18418: struct bpf_insn *insn = env->prog->insnsi;
18419: int insn_cnt = env->prog->len;
18420: int i, err;
18421:
18422: err = bpf_prog_calc_tag(env->prog);
18423: if (err)
18424: return err;
18425:
18426: for (i = 0; i < insn_cnt; i++, insn++) {
18427: if (insn->dst_reg >= MAX_BPF_REG) {
18428: verbose(env, "R%d is invalid\n", insn->dst_reg);
18429: return -EINVAL;
18430: }
18431: if (insn->src_reg >= MAX_BPF_REG) {
18432: verbose(env, "R%d is invalid\n", insn->src_reg);
18433: return -EINVAL;
18434: }
18435: if (insn[0].code == (BPF_LD | BPF_IMM | BPF_DW)) {
18436: struct bpf_insn_aux_data *aux;
18437: struct bpf_map *map;
18438: int map_idx;
18439: u64 addr;
18440: u32 fd;
18441:
18442: if (i == insn_cnt - 1 || insn[1].code != 0 ||
18443: insn[1].dst_reg != 0 || insn[1].src_reg != 0 ||
18444: insn[1].off != 0) {
18445: verbose(env, "invalid bpf_ld_imm64 insn\n");
18446: return -EINVAL;
18447: }
18448:
18449: if (insn[0].off != 0) {
18450: verbose(env, "BPF_LD_IMM64 uses reserved fields\n");
18451: return -EINVAL;
18452: }
18453:
18454: if (insn[0].src_reg == 0)
18455: /* valid generic load 64-bit imm */
18456: goto next_insn;
18457:
18458: if (insn[0].src_reg == BPF_PSEUDO_BTF_ID) {
18459: aux = &env->insn_aux_data[i];
18460: err = check_pseudo_btf_id(env, insn, aux);
18461: if (err)
18462: return err;
18463: goto next_insn;
18464: }
18465:
18466: if (insn[0].src_reg == BPF_PSEUDO_FUNC) {
18467: aux = &env->insn_aux_data[i];
18468: aux->ptr_type = PTR_TO_FUNC;
18469: goto next_insn;
18470: }
18471:
18472: /* In final convert_pseudo_ld_imm64() step, this is
18473: * converted into regular 64-bit imm load insn.
18474: */
18475: switch (insn[0].src_reg) {
18476: case BPF_PSEUDO_MAP_VALUE:
18477: case BPF_PSEUDO_MAP_IDX_VALUE:
18478: break;
18479: case BPF_PSEUDO_MAP_FD:
18480: case BPF_PSEUDO_MAP_IDX:
18481: if (insn[1].imm == 0)
18482: break;
18483: fallthrough;
18484: default:
18485: verbose(env, "unrecognized bpf_ld_imm64 insn\n");
18486: return -EINVAL;
18487: }
18488:
18489: switch (insn[0].src_reg) {
18490: case BPF_PSEUDO_MAP_IDX_VALUE:
18491: case BPF_PSEUDO_MAP_IDX:
18492: if (bpfptr_is_null(env->fd_array)) {
18493: verbose(env, "fd_idx without fd_array is invalid\n");
18494: return -EPROTO;
18495: }
18496: if (copy_from_bpfptr_offset(&fd, env->fd_array,
18497: insn[0].imm * sizeof(fd),
18498: sizeof(fd)))
18499: return -EFAULT;
18500: break;
18501: default:
18502: fd = insn[0].imm;
18503: break;
18504: }
18505:
18506: map_idx = add_used_map(env, fd);
18507: if (map_idx < 0)
18508: return map_idx;
18509: map = env->used_maps[map_idx];
18510:
18511: aux = &env->insn_aux_data[i];
18512: aux->map_index = map_idx;
18513:
18514: if (insn[0].src_reg == BPF_PSEUDO_MAP_FD ||
18515: insn[0].src_reg == BPF_PSEUDO_MAP_IDX) {
18516: addr = (unsigned long)map;
18517: } else {
18518: u32 off = insn[1].imm;
18519:
18520: if (!map->ops->map_direct_value_addr) {
18521: verbose(env, "no direct value access support for this map type\n");
18522: return -EINVAL;
18523: }
18524:
18525: err = map->ops->map_direct_value_addr(map, &addr, off);
18526: if (err) {
18527: verbose(env, "invalid access to map value pointer, value_size=%u off=%u\n",
18528: map->value_size, off);
18529: return err;
18530: }
18531:
18532: aux->map_off = off;
18533: addr += off;
18534: }
18535:
18536: insn[0].imm = (u32)addr;
18537: insn[1].imm = addr >> 32;
18538:
18539: next_insn:
18540: insn++;
18541: i++;
18542: continue;
18543: }
18544:
18545: /* Basic sanity check before we invest more work here. */
18546: if (!bpf_opcode_in_insntable(insn->code)) {
18547: verbose(env, "unknown opcode %02x\n", insn->code);
18548: return -EINVAL;
18549: }
18550:
18551: err = check_insn_fields(env, insn);
18552: if (err)
18553: return err;
18554: }
18555:
18556: /* now all pseudo BPF_LD_IMM64 instructions load valid
18557: * 'struct bpf_map *' into a register instead of user map_fd.
18558: * These pointers will be used later by verifier to validate map access.
18559: */
18560: return 0;
18561: }
SourceFile:kernel/bpf/verifier.c]
|
| 180/2 |
2026/05/18 19:01 |
llm |
debugger |
0m
Model:
Tokens:
input: 203276
output: 42
thoughts: 3635
Thoughts: **Analyzing the Crash: A Deep Dive**
Okay, let's break this down. My initial thought was the BPF program, focusing on `add_used_map` and its potential for an invalid file descriptor (`fd = -1`) leading to an error path that might trigger the crash. The sequence seemed logical: `__bpf_map_get(-1)` returning `-EBADF`, which cascades through `add_used_map`, `check_and_resolve_insns`, `bpf_check`, `bpf_prog_load`, ultimately resulting in a failed program load, effectively making the BPF code irrelevant. But if the BPF program wasn't loaded, what *was* going on?
Then the focus shifted to the reproducer, specifically the `mmap` calls, followed by `socket` and `sendmsg`. My intuition honed in on `sendmsg` as a possible culprit for an infinite loop. The `tc_modify_qdisc` function came to mind, specifically examining the `tcm_ifindex`. I quickly confirmed that `tcm_ifindex` was set to zero, triggering a return of `-ENODEV` from `__dev_get_by_index`. This led to the conclusion that `sendmsg` itself would exit returning `-ENODEV`. This didn't seem to explain a soft lockup.
I started digging into the syzkaller log, and the crash dump, noticing a soft lockup on CPU 0 in a workqueue thread `kworker/u9:4` executing `toggle_allocation_gate`, while CPU 1, `swapper/1`, was stuck in `__remove_hrtimer`. This indicated a potential timer issue. The `toggle_allocation_gate` function, and the `kfence_timer` it adds, became the focus. I realized that a `hrtimer` being stuck in `__remove_hrtimer` would not be due to a workqueue timer (using `timer_list`).
The backtrace for CPU 1's NMI interrupt showed the core of the timer issue as being stuck within `__remove_hrtimer` and its reliance on the `hrtimer` tree. The function `timerqueue_linked_del` and subsequent calls to `rb_erase_linked` and `__rb_erase_augmented` made the possibility of rbtree corruption, perhaps due to concurrent modification or memory corruption, a likely explanation for the loop.
I considered a known `hrtimer` bug and went back to the BPF program, considering possible concurrent modification issues with the BPF code. I then looked at the generated C code, and I began to wonder whether the map was ever properly created given that there was a `-1` used for the `ringbuf_map_fd`. This meant the load would fail, and I would be back to wondering what was causing the soft lockup.
Then, I went back to the code, and realized that even if the `bpf_prog_load` fails, the BPF program might still be relevant if it was attached somewhere. However, since there was no `BPF_PROG_ATTACH` or `perf_event_open` it would not be. The `BPF_PROG_TYPE_TRACEPOINT` was also confirmed, but it wasn't attached. Also, the BPF program didn't have any `bpf_timer_init` calls.
This eliminated the BPF program as a direct cause, meaning the issue was external. I had to look at other code within the reproducer. The `ioctl(0x8933)` call (`SIOCGIFINDEX`) seemed important, along with `sendmsg`. I recalled that syzkaller likely handles the index and passed `team0`'s ifindex to `tcm_ifindex`. The generated C code bug would not have resulted in the `RTM_NEWQDISC` path being taken. That means `tc_modify_qdisc` would be called with `tcm_ifindex` as the ifindex of `team0`. This should work, since `tcm_parent` is 0. If `tcm_parent` is 0 then the code goes to the `else` condition, where handle is also zero, causing the return of `-EINVAL`, doing nothing. So `RTM_NEWQDISC` does nothing, and the reproducer does nothing. If this is true, then why the soft lockup?
Finally, the `Code` snippet with `lock_is_held` seemed important.
|
| 181/2 |
2026/05/18 19:01 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/locking/lockdep.c IncludeLines:true Name:lock_is_held]
Results: map[SourceCode: 247: static inline int lock_is_held(const struct lockdep_map *lock)
248: {
249: return lock_is_held_type(lock, -1);
250: }
SourceFile:include/linux/lockdep.h]
|
| 182/2 |
2026/05/18 19:01 |
llm |
debugger |
0m
Model:
Tokens:
input: 208014
output: 44
thoughts: 71
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience:
**Analyzing an Inline Function and Identifying a Call Site**
Alright, a-ha! I see that `lock_is_held` is defined as an inline function. This immediately tells me something about performance – it's likely optimized for speed. Digging deeper, it appears this inline function resolves to a call to `lock_is_held_type(lock, -1)`. Excellent. This confirms my suspicion.
So, the assembly instruction `call 0xa21a2ce`, which would be otherwise unclear, is, in fact, the invocation of `lock_is_held_type`. Now I've connected the dots between the high-level inline call and the actual runtime function call. Time to investigate `lock_is_held_type` to understand how the lock's held state is actually determined. This is where the real logic likely resides. Let's get to it.
|
| 183/2 |
2026/05/18 19:01 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/locking/lockdep.c IncludeLines:true Name:lock_is_held_type]
Results: map[SourceCode:5924: noinstr int lock_is_held_type(const struct lockdep_map *lock, int read)
5925: {
5926: unsigned long flags;
5927: int ret = LOCK_STATE_NOT_HELD;
5928:
5929: /*
5930: * Avoid false negative lockdep_assert_held() and
5931: * lockdep_assert_not_held().
5932: */
5933: if (unlikely(!lockdep_enabled()))
5934: return LOCK_STATE_UNKNOWN;
5935:
5936: raw_local_irq_save(flags);
5937: check_flags(flags);
5938:
5939: lockdep_recursion_inc();
5940: ret = __lock_is_held(lock, read);
5941: lockdep_recursion_finish();
5942: raw_local_irq_restore(flags);
5943:
5944: return ret;
5945: }
SourceFile:kernel/locking/lockdep.c]
|
| 184/2 |
2026/05/18 19:01 |
llm |
debugger |
0m
Model:
Tokens:
input: 208663
output: 43
thoughts: 862
Thoughts: **Dissecting the Assembly: A Deep Dive into RCU and Lockdep**
Okay, so I'm looking at this assembly, and it seems like `lock_is_held_type` isn't inlined, which is good to know. The initial code loads the address of `cpu_base->lock.dep_map` into `rdi` and sets `esi` to -1, which is a bit strange, but probably a flag of sorts. Then it calls `lock_is_held_type`, and the result is stored in `r13d`. Afterward, I see `edi` set to zero, and `esi` gets the value of `eax` (which is the result of `lock_is_held_type`). Then there's this call to `rcu_read_unlock`, and that's what's got me curious.
Wait a second...I am now calling `rcu_read_unlock` *after* calling `lock_is_held_type`? I need to figure out why. That makes me think of `lockdep_assert_held`. Looking at that definition, `WARN_ON` will be triggered if `debug_locks` is enabled and `lockdep_is_held(l)` returns false. `WARN_ON` is more complicated, but basically, it checks a condition and potentially calls `__WARN()`. But why the `rcu_read_unlock` call? This is likely related to RCU's mechanism, so let's look at that.
Hmm, `rcu_read_unlock` is inlined, so that's not the answer. What's this function that takes `0` and `lock_is_held_type()` as arguments? I need to figure out what's called with `edi=0` and `esi=eax`. Okay, looking at `lockdep_is_held` now, it just calls `lock_is_held`. So `lockdep_assert_held` and `debug_locks` are a factor here. `debug_locks` is a global variable; if it's set, this could be important. What about `rcu_read_unlock_strict`?
I noticed there are two nearby calls, `call 0x11fdfb` and then `call 0x11fdf9`. Now, since the calls are really close to each other, are we dealing with a lock and unlock pair? It's highly suggestive. But, `rcu_read_lock` and `rcu_read_unlock` don't take any arguments, so that can't be it, right? But the values that are being passed in as arguments are zero, and the result of the `lock_is_held_type()` call, which is passed in `edi=0` and `esi=eax`...
So, I need to figure out what function uses these arguments. Could it be `trace_rcu_utilization`? Or `rcu_lock_acquire`? Going to check `rcu_read_lock` again.
|
| 185/2 |
2026/05/18 19:01 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:include/linux/rcupdate.h IncludeLines:true Name:rcu_read_lock]
Results: map[SourceCode: 788: /**
789: * rcu_read_lock() - mark the beginning of an RCU read-side critical section
790: *
791: * When synchronize_rcu() is invoked on one CPU while other CPUs
792: * are within RCU read-side critical sections, then the
793: * synchronize_rcu() is guaranteed to block until after all the other
794: * CPUs exit their critical sections. Similarly, if call_rcu() is invoked
795: * on one CPU while other CPUs are within RCU read-side critical
796: * sections, invocation of the corresponding RCU callback is deferred
797: * until after the all the other CPUs exit their critical sections.
798: *
799: * Both synchronize_rcu() and call_rcu() also wait for regions of code
800: * with preemption disabled, including regions of code with interrupts or
801: * softirqs disabled.
802: *
803: * Note, however, that RCU callbacks are permitted to run concurrently
804: * with new RCU read-side critical sections. One way that this can happen
805: * is via the following sequence of events: (1) CPU 0 enters an RCU
806: * read-side critical section, (2) CPU 1 invokes call_rcu() to register
807: * an RCU callback, (3) CPU 0 exits the RCU read-side critical section,
808: * (4) CPU 2 enters a RCU read-side critical section, (5) the RCU
809: * callback is invoked. This is legal, because the RCU read-side critical
810: * section that was running concurrently with the call_rcu() (and which
811: * therefore might be referencing something that the corresponding RCU
812: * callback would free up) has completed before the corresponding
813: * RCU callback is invoked.
814: *
815: * RCU read-side critical sections may be nested. Any deferred actions
816: * will be deferred until the outermost RCU read-side critical section
817: * completes.
818: *
819: * You can avoid reading and understanding the next paragraph by
820: * following this rule: don't put anything in an rcu_read_lock() RCU
821: * read-side critical section that would block in a !PREEMPTION kernel.
822: * But if you want the full story, read on!
823: *
824: * In non-preemptible RCU implementations (pure TREE_RCU and TINY_RCU),
825: * it is illegal to block while in an RCU read-side critical section.
826: * In preemptible RCU implementations (PREEMPT_RCU) in CONFIG_PREEMPTION
827: * kernel builds, RCU read-side critical sections may be preempted,
828: * but explicit blocking is illegal. Finally, in preemptible RCU
829: * implementations in real-time (with -rt patchset) kernel builds, RCU
830: * read-side critical sections may be preempted and they may also block, but
831: * only when acquiring spinlocks that are subject to priority inheritance.
832: */
833: static __always_inline void rcu_read_lock(void)
834: __acquires_shared(RCU)
835: {
836: __rcu_read_lock();
837: __acquire_shared(RCU);
838: rcu_lock_acquire(&rcu_lock_map);
839: RCU_LOCKDEP_WARN(!rcu_is_watching(),
840: "rcu_read_lock() used illegally while idle");
841: }
SourceFile:include/linux/rcupdate.h]
|
| 186/2 |
2026/05/18 19:02 |
llm |
debugger |
0m
Model:
Tokens:
input: 211311
output: 22
thoughts: 3941
Thoughts: **A Deep Dive into a Soft Lockup and the Elusive Reproducer**
Okay, here's the breakdown of my thought process. I started by tracing through the assembly, attempting to understand the call stack leading up to the reported soft lockup. I see `rcu_lock_acquire`, which eventually lands me in `__remove_hrtimer`. My initial concern was a potential lockdep issue, perhaps a missing `rcu_read_lock` or misapplication of `lockdep_assert_held`. I dug into the macro definitions of `lockdep_assert_held` and `WARN_ON`, considering global variables like `debug_locks` and their implications on the execution path. I considered, what if the code is `rcu_read_lock_bh_held` or `trace_hrtimer_cancel`? Nope, it's just `__remove_hrtimer`.
I then analyzed the assembly, focusing on the `lock_is_held_type` call and the subsequent function call. I was puzzled by the jump to `WARN_ON` and tried to figure out what was happening on the call at `0x11fdf9`. I also wondered if `debug_locks` was checked before the `lock_is_held_type`.
Then came the realization: if CPU 1 is at `test r13d, r13d`, it isn't stuck in an infinite loop. It is just running `__remove_hrtimer`, but then it got an NMI! The NMI is sent by CPU 0 because CPU 0 detected a soft lockup. So, I figured CPU 1 isn't the root cause; CPU 0 is the stuck one. I shifted my focus to CPU 0's stack trace.
The stack trace of CPU 0 revealed that it was stuck in `csd_lock_wait`, waiting on CPU 1, which led me to think CPU 1 must be busy. The stack of CPU 1 shows `__remove_hrtimer` and then `__run_hrtimer`. Ah, I then thought, if CPU 1 is stuck in `__hrtimer_run_queues` for 26 seconds, it's an infinite loop in `__hrtimer_run_queues`!
I considered how the reproducer might fit, thinking a timer callback returning `HRTIMER_RESTART` without advancing the expiration time could be the problem. I had to look into how the system calls, specifically the `sendmsg` call, fit in. I went through the arguments of `sendmsg$nl_route_sched`, checking the `newqdisc` arguments and the attributes within. I identified potentially invalid attributes with a length of 0. If `tca` is empty, `nlmsg_parse_deprecated` would return `-EINVAL`. The reproducer writes only a few bytes in the `tcmsg`. The rest of the buffer is zeroed, which would make the parsing function return `-EINVAL`.
With the `sendmsg` syscall failing, I looked into the `bpf_prog_load` call. Analyzing the arguments, I noticed `license` is NULL, which means `strncpy_from_bpfptr` will fail, returning `-EFAULT`. I confirmed that the license is indeed zero in the reproducer, causing bpf load to not work. Then, I realized that, if the system calls do nothing, then the crash must be caused by something else. Since syzkaller says that the reproducer might be incomplete, I had to keep going. I considered searching for similar bugs.
|
| 187/2 |
2026/05/18 19:02 |
tool |
grepper |
0m
Args: map[Expression:__hrtimer_run_queues]
Results: map[Output:Documentation/tools/rtla/rtla-timerlat-top.rst=49=higher is found::
--
Documentation/tools/rtla/rtla-timerlat-top.rst-86- -> timerlat_irq
Documentation/tools/rtla/rtla-timerlat-top.rst:87: -> __hrtimer_run_queues
Documentation/tools/rtla/rtla-timerlat-top.rst-88- -> hrtimer_interrupt
--
Documentation/trace/timerlat-tracer.rst=142=irq disabled. For example::
--
Documentation/trace/timerlat-tracer.rst-155- => timerlat_irq
Documentation/trace/timerlat-tracer.rst:156: => __hrtimer_run_queues
Documentation/trace/timerlat-tracer.rst-157- => hrtimer_interrupt
--
drivers/gpu/drm/ci/xfails/msm-sm8350-hdk-skips.txt=20=msm/msm_mapping@ring
--
drivers/gpu/drm/ci/xfails/msm-sm8350-hdk-skips.txt-162-# [ 228.509826] watchdog_timer_fn+0x254/0x2e4
drivers/gpu/drm/ci/xfails/msm-sm8350-hdk-skips.txt:163:# [ 228.514087] __hrtimer_run_queues+0x3b0/0x40c
drivers/gpu/drm/ci/xfails/msm-sm8350-hdk-skips.txt-164-# [ 228.518612] hrtimer_interrupt+0xe8/0x248
--
kernel/time/hrtimer.c=1961=static __always_inline struct hrtimer *clock_base_next_timer_safe(struct hrtimer_clock_base *base)
--
kernel/time/hrtimer.c-1967-
kernel/time/hrtimer.c:1968:static void __hrtimer_run_queues(struct hrtimer_cpu_base *cpu_base, ktime_t now,
kernel/time/hrtimer.c-1969- unsigned long flags, unsigned int active_mask)
--
kernel/time/hrtimer.c=2001=static __latent_entropy void hrtimer_run_softirq(void)
--
kernel/time/hrtimer.c-2010- now = hrtimer_update_base(cpu_base);
kernel/time/hrtimer.c:2011: __hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_SOFT);
kernel/time/hrtimer.c-2012-
--
kernel/time/hrtimer.c=2083=void hrtimer_interrupt(struct clock_event_device *dev)
--
kernel/time/hrtimer.c-2100- * Set expires_next to KTIME_MAX, which prevents that remote CPUs queue
kernel/time/hrtimer.c:2101: * timers while __hrtimer_run_queues() is expiring the clock bases.
kernel/time/hrtimer.c-2102- * Timers which are re/enqueued on the local CPU are not affected by
--
kernel/time/hrtimer.c-2112-
kernel/time/hrtimer.c:2113: __hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
kernel/time/hrtimer.c-2114-
--
kernel/time/hrtimer.c=2147=void hrtimer_run_queues(void)
--
kernel/time/hrtimer.c-2176-
kernel/time/hrtimer.c:2177: __hrtimer_run_queues(cpu_base, now, flags, HRTIMER_ACTIVE_HARD);
kernel/time/hrtimer.c-2178- raw_spin_unlock_irqrestore(&cpu_base->lock, flags);
]
|
| 188/2 |
2026/05/18 19:02 |
llm |
debugger |
0m
Model:
Tokens:
input: 217086
output: 45
thoughts: 978
Thoughts: **Debugging a Potential Infinite Loop in `__hrtimer_run_queues`**
Okay, so I'm staring at this reproducer, and it looks like `__hrtimer_run_queues` is looping indefinitely. The first question is always, which timer is stuck? Could there be a timer callback that returns `HRTIMER_RESTART` without actually advancing the expiration time? Time to start searching for instances of `HRTIMER_RESTART` in the kernel source.
Hmm, there are a lot of those. But, wait, the reproducer uses `bpf_timer_init`, right? No, I checked the BPF instructions and didn't find `bpf_timer_init`. Ah, the reproducer did have a `bpf_ringbuf_discard` call.
Now wait, what if the BPF *actually* does use a BPF timer? I'd better double-check the BPF helper codes. Yep, `bpf_timer_init` is 170 (`0xaa`), `bpf_timer_set_callback` is 171 (`0xab`), `bpf_timer_start` is 172 (`0xac`), and `bpf_timer_cancel` is 173 (`0xad`). Let's scan the hex dump of the instructions in the reproducer: `0x83`, `0xa5`, `0x86`, `0x82`, `0x85`. Nope, none of them match the BPF timer helper codes! The BPF program *definitely* doesn't use BPF timers.
So, if it's not a BPF timer, what *else* could trigger an infinite loop in `__hrtimer_run_queues`? Wait a minute... the reproducer sends an `RTM_NEWQDISC` message. Could there be a bug *there* in `qdisc` that's messing with the hrtimer? Thinking back, I know `qdisc` uses `qdisc_watchdog`, and that *itself* uses hrtimers! Looks like I need to dive into `qdisc_watchdog_schedule` in `net/sched/sch_api.c`. Time to see what's going on there.
|
| 189/2 |
2026/05/18 19:02 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:net/sched/sch_api.c IncludeLines:true Name:qdisc_watchdog_schedule]
Results: map[SourceCode: 78: static inline void qdisc_watchdog_schedule(struct qdisc_watchdog *wd,
79: psched_time_t expires)
80: {
81: qdisc_watchdog_schedule_ns(wd, PSCHED_TICKS2NS(expires));
82: }
SourceFile:include/net/pkt_sched.h]
|
| 190/2 |
2026/05/18 19:02 |
llm |
debugger |
0m
Model:
Tokens:
input: 218724
output: 47
thoughts: 27
Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience:
**Diving into the qdisc Watchdog Scheduler**
Okay, time to get to the heart of the matter. I need to understand how the scheduling watchdog operates, and the place to start is definitely the core scheduler API. I'm going to jump into `net/sched/sch_api.c`. Specifically, I need to understand the mechanics of `qdisc_watchdog_schedule_ns`. My intuition tells me this function is the key to how the watchdog timers are set up and managed for quality-of-service disciplines. I'm anticipating this code will shed light on the timing mechanisms, the context switching, and any possible race conditions that might exist. I'm keen to dissect how it interacts with the underlying timer framework within the kernel, given its significance in maintaining predictable and timely network traffic shaping. Let's see what we find.
|
| 191/2 |
2026/05/18 19:02 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:net/sched/sch_api.c IncludeLines:true Name:qdisc_watchdog_schedule_ns]
Results: map[SourceCode: 72: static inline void qdisc_watchdog_schedule_ns(struct qdisc_watchdog *wd,
73: u64 expires)
74: {
75: return qdisc_watchdog_schedule_range_ns(wd, expires, 0ULL);
76: }
SourceFile:include/net/pkt_sched.h]
|
| 192/2 |
2026/05/18 19:02 |
llm |
debugger |
0m
Model:
Tokens:
input: 219106
output: 49
thoughts: 17
Thoughts: Okay, here's my take on those thoughts, framed as an expert's self-reflection:
**Deep Dive into `qdisc_watchdog_schedule_range_ns`**
Right, let's take a look at `qdisc_watchdog_schedule_range_ns`. I need to refresh my understanding of how this particular function fits into the overall queuing discipline watchdog mechanism. Specifically, I'm going to focus on its role in scheduling the watchdog timer within a specified time range, presumably to avoid issues with excessive timer latency or starvation. I’m thinking I'll want to quickly check the surrounding code and possibly related functions that call or are called by this one, as I'm interested in how it integrates with the rest of the scheduler core. It's likely involved in implementing some adaptive backoff or jittering strategy. Time to get the details!
|
| 193/2 |
2026/05/18 19:02 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:net/sched/sch_api.c IncludeLines:true Name:qdisc_watchdog_schedule_range_ns]
Results: map[SourceCode: 630: void qdisc_watchdog_schedule_range_ns(struct qdisc_watchdog *wd, u64 expires,
631: u64 delta_ns)
632: {
633: bool deactivated;
634:
635: rcu_read_lock();
636: deactivated = test_bit(__QDISC_STATE_DEACTIVATED,
637: &qdisc_root_sleeping(wd->qdisc)->state);
638: rcu_read_unlock();
639: if (deactivated)
640: return;
641:
642: if (hrtimer_is_queued(&wd->timer)) {
643: u64 softexpires;
644:
645: softexpires = ktime_to_ns(hrtimer_get_softexpires(&wd->timer));
646: /* If timer is already set in [expires, expires + delta_ns],
647: * do not reprogram it.
648: */
649: if (softexpires - expires <= delta_ns)
650: return;
651: }
652:
653: hrtimer_start_range_ns(&wd->timer,
654: ns_to_ktime(expires),
655: delta_ns,
656: HRTIMER_MODE_ABS_PINNED);
657: }
SourceFile:net/sched/sch_api.c]
|
| 194/2 |
2026/05/18 19:02 |
llm |
debugger |
0m
Model:
Tokens:
input: 219799
output: 43
thoughts: 28
Thoughts: Okay, here's my interpretation of those thoughts, presented as my own mental process, given my expertise:
**Diving into the `qdisc_watchdog` Mechanism**
Alright, hang on a second... my mind just took a little detour. I was deep in this networking code, and I just realized something significant. The `qdisc_watchdog` – that's the scheduler watchdog timer – it relies on `hrtimer`, doesn't it? That's a key detail! I should have known this already, but it's always good to refresh.
Now the follow up question has to be: *what* is the specific callback function that `qdisc_watchdog` actually uses? That callback is the heart of the watchdog mechanism itself, so I need to understand it. I want to trace the code. I need to figure out what happens when the timer actually fires. This will tell me how the scheduler is actually being managed in the face of possible lockups, or other such failures. I have to understand the callback's role! Let's get to the bottom of this.
|
| 195/2 |
2026/05/18 19:03 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:net/sched/sch_api.c IncludeLines:true Name:qdisc_watchdog]
Results: map[SourceCode: 604: static enum hrtimer_restart qdisc_watchdog(struct hrtimer *timer)
605: {
606: struct qdisc_watchdog *wd = container_of(timer, struct qdisc_watchdog,
607: timer);
608:
609: rcu_read_lock();
610: __netif_schedule(qdisc_root(wd->qdisc));
611: rcu_read_unlock();
612:
613: return HRTIMER_NORESTART;
614: }
SourceFile:net/sched/sch_api.c]
|
| 196/2 |
2026/05/18 19:03 |
llm |
debugger |
1m
Model:
Tokens:
input: 220294
output: 44
thoughts: 869
Thoughts: **Debugging a Potential BPF-Related Kernel Panic**
Okay, so `qdisc_watchdog` returns `HRTIMER_NORESTART`, which means it shouldn't be the direct cause of an infinite loop in `__hrtimer_run_queues`. That rules that out as the immediate problem. But... could the BPF program *indirectly* be involved? The reproducer uses `BPF_PROG_LOAD`, so let's dig deeper into what it is doing.
I was trying to consider which other BPF helpers use `hrtimer`. Hmm, let me check `bpf_timer_init` again, just in case. Wait, that's helper 170... but the reproducer is using others – `bpf_ringbuf_reserve` (131), `bpf_snprintf` (165), `bpf_ringbuf_query` (134), `bpf_ringbuf_output` (130), `bpf_ringbuf_discard` (133). Maybe the BPF program is causing some sort of memory corruption within a different helper.
Wait, `bpf_snprintf` is at helper number 165. Let's look closely at `bpf_snprintf` in `kernel/bpf/helpers.c`. Here it is:
```c
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
```
Okay, the crucial part seems to be the `bpf_bprintf_prepare` function. Let's analyze its code:
```c
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
```
Wait, `MAX_BPRINTF_BIN_ARGS` is defined as 512. Now I remember.
Let's look at the `snprintf` use:
```c
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
```
Aha! `snprintf` returns the number of characters that *would* have been written if the buffer was large enough. If `err` is larger than `tmp_buf_end - tmp_buf`, the line `tmp_buf += (err + 1)` will advance `tmp_buf` *past* `tmp_buf_end`! Then, the *next* argument will be written *out of bounds*! This could lead to a memory corruption. Let me examine `bpf_bprintf_prepare` one more time.
|
| 197/2 |
2026/05/18 19:04 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_prepare]
Results: map[SourceCode: 804: /*
805: * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
806: *
807: * Returns a negative value if fmt is an invalid format string or 0 otherwise.
808: *
809: * This can be used in two ways:
810: * - Format string verification only: when data->get_bin_args is false
811: * - Arguments preparation: in addition to the above verification, it writes in
812: * data->bin_args a binary representation of arguments usable by bstr_printf
813: * where pointers from BPF have been sanitized.
814: *
815: * In argument preparation mode, if 0 is returned, safe temporary buffers are
816: * allocated and bpf_bprintf_cleanup should be called to free them after use.
817: */
818: int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
819: u32 num_args, struct bpf_bprintf_data *data)
820: {
821: bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
822: char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
823: struct bpf_bprintf_buffers *buffers = NULL;
824: size_t sizeof_cur_arg, sizeof_cur_ip;
825: int err, i, num_spec = 0;
826: u64 cur_arg;
827: char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
828:
829: fmt_end = strnchr(fmt, fmt_size, 0);
830: if (!fmt_end)
831: return -EINVAL;
832: fmt_size = fmt_end - fmt;
833:
834: if (get_buffers && bpf_try_get_buffers(&buffers))
835: return -EBUSY;
836:
837: if (data->get_bin_args) {
838: if (num_args)
839: tmp_buf = buffers->bin_args;
840: tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
841: data->bin_args = (u32 *)tmp_buf;
842: }
843:
844: if (data->get_buf)
845: data->buf = buffers->buf;
846:
847: for (i = 0; i < fmt_size; i++) {
848: unsigned char c = fmt[i];
849:
850: /*
851: * Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
852: * through unchanged, while still rejecting ASCII control bytes.
853: */
854: if (isascii(c) && !isprint(c) && !isspace(c)) {
855: err = -EINVAL;
856: goto out;
857: }
858:
859: if (fmt[i] != '%')
860: continue;
861:
862: if (fmt[i + 1] == '%') {
863: i++;
864: continue;
865: }
866:
867: if (num_spec >= num_args) {
868: err = -EINVAL;
869: goto out;
870: }
871:
872: /* The string is zero-terminated so if fmt[i] != 0, we can
873: * always access fmt[i + 1], in the worst case it will be a 0
874: */
875: i++;
876: c = fmt[i];
877: /*
878: * The format parser below only understands ASCII conversion
879: * specifiers and modifiers, so reject non-ASCII after '%'.
880: */
881: if (!isascii(c)) {
882: err = -EINVAL;
883: goto out;
884: }
885:
886: /* skip optional "[0 +-][num]" width formatting field */
887: while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
888: fmt[i] == ' ')
889: i++;
890: if (fmt[i] >= '1' && fmt[i] <= '9') {
891: i++;
892: while (fmt[i] >= '0' && fmt[i] <= '9')
893: i++;
894: }
895:
896: if (fmt[i] == 'p') {
897: sizeof_cur_arg = sizeof(long);
898:
899: if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
900: ispunct(fmt[i + 1])) {
901: if (tmp_buf)
902: cur_arg = raw_args[num_spec];
903: goto nocopy_fmt;
904: }
905:
906: if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
907: fmt[i + 2] == 's') {
908: fmt_ptype = fmt[i + 1];
909: i += 2;
910: goto fmt_str;
911: }
912:
913: if (fmt[i + 1] == 'K' ||
914: fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
915: fmt[i + 1] == 'S') {
916: if (tmp_buf)
917: cur_arg = raw_args[num_spec];
918: i++;
919: goto nocopy_fmt;
920: }
921:
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
930:
931: i++;
932: num_spec++;
933: continue;
934: }
935:
936: /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
937: if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
938: (fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
939: err = -EINVAL;
940: goto out;
941: }
942:
943: i += 2;
944: if (!tmp_buf)
945: goto nocopy_fmt;
946:
947: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
948: if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
949: err = -ENOSPC;
950: goto out;
951: }
952:
953: unsafe_ptr = (char *)(long)raw_args[num_spec];
954: err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
955: sizeof_cur_ip);
956: if (err < 0)
957: memset(cur_ip, 0, sizeof_cur_ip);
958:
959: /* hack: bstr_printf expects IP addresses to be
960: * pre-formatted as strings, ironically, the easiest way
961: * to do that is to call snprintf.
962: */
963: ip_spec[2] = fmt[i - 1];
964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
969: num_spec++;
970:
971: continue;
972: } else if (fmt[i] == 's') {
973: fmt_ptype = fmt[i];
974: fmt_str:
975: if (fmt[i + 1] != 0 &&
976: !isspace(fmt[i + 1]) &&
977: !ispunct(fmt[i + 1])) {
978: err = -EINVAL;
979: goto out;
980: }
981:
982: if (!tmp_buf)
983: goto nocopy_fmt;
984:
985: if (tmp_buf_end == tmp_buf) {
986: err = -ENOSPC;
987: goto out;
988: }
989:
990: unsafe_ptr = (char *)(long)raw_args[num_spec];
991: err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
992: fmt_ptype,
993: tmp_buf_end - tmp_buf);
994: if (err < 0) {
995: tmp_buf[0] = '\0';
996: err = 1;
997: }
998:
999: tmp_buf += err;
1000: num_spec++;
1001:
1002: continue;
1003: } else if (fmt[i] == 'c') {
1004: if (!tmp_buf)
1005: goto nocopy_fmt;
1006:
1007: if (tmp_buf_end == tmp_buf) {
1008: err = -ENOSPC;
1009: goto out;
1010: }
1011:
1012: *tmp_buf = raw_args[num_spec];
1013: tmp_buf++;
1014: num_spec++;
1015:
1016: continue;
1017: }
1018:
1019: sizeof_cur_arg = sizeof(int);
1020:
1021: if (fmt[i] == 'l') {
1022: sizeof_cur_arg = sizeof(long);
1023: i++;
1024: }
1025: if (fmt[i] == 'l') {
1026: sizeof_cur_arg = sizeof(long long);
1027: i++;
1028: }
1029:
1030: if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
1031: fmt[i] != 'x' && fmt[i] != 'X') {
1032: err = -EINVAL;
1033: goto out;
1034: }
1035:
1036: if (tmp_buf)
1037: cur_arg = raw_args[num_spec];
1038: nocopy_fmt:
1039: if (tmp_buf) {
1040: tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
1041: if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
1042: err = -ENOSPC;
1043: goto out;
1044: }
1045:
1046: if (sizeof_cur_arg == 8) {
1047: *(u32 *)tmp_buf = *(u32 *)&cur_arg;
1048: *(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
1049: } else {
1050: *(u32 *)tmp_buf = (u32)(long)cur_arg;
1051: }
1052: tmp_buf += sizeof_cur_arg;
1053: }
1054: num_spec++;
1055: }
1056:
1057: err = 0;
1058: out:
1059: if (err)
1060: bpf_bprintf_cleanup(data);
1061: return err;
1062: }
SourceFile:kernel/bpf/helpers.c]
|
| 198/2 |
2026/05/18 19:04 |
llm |
debugger |
0m
Model:
Tokens:
input: 226006
output: 68
thoughts: 6172
Thoughts: **Analysis of the BPF Buffer Overflow Vulnerability**
Okay, so I've been digging into this kernel crash, and I've got a pretty solid handle on what's going on. It boils down to a buffer overflow in the `bpf_bprintf_prepare` function, specifically when it's handling format strings used by the `bpf_snprintf` family of helpers. The crash manifests as a soft lockup, but the root cause is a memory corruption issue that ultimately damages kernel data structures, specifically `hrtimer` related data, leading to the infinite loop. Here's the breakdown:
The first thing I realized is that `snprintf` returns the number of *characters that would have been written*. This is a critical detail. The code then uses this return value to advance the output buffer pointer, `tmp_buf`. If the formatted string is longer than the remaining space in the buffer, `tmp_buf` will be advanced beyond its allocated end, `tmp_buf_end`. This is problem #1.
Now, for problem #2, there's a sneaky signedness bug in the bounds checking. The code compares `tmp_buf_end - tmp_buf` (a signed `ptrdiff_t`) with a size requirement (`sizeof_cur_arg`, a `size_t`). The C compiler will cast the signed `ptrdiff_t` to `size_t`. If `tmp_buf` has already gone past `tmp_buf_end` (because of the `snprintf` issue above), then `tmp_buf_end - tmp_buf` becomes negative. When it's cast to `size_t`, it turns into a HUGE positive number. This means that if it encounters an unaligned access, it won't check if the buffer is big enough, and it will bypass the error check and continue writing to the buffer. If it encounters aligned access it won't notice the overflow, and it will write out of bounds.
Now, because of the overflow, let's suppose `tmp_buf` ends up past `tmp_buf_end`. When this happens, there is a `bpf_trace_copy_string` function call and `tmp_buf_end - tmp_buf` is passed to the size parameter of this function. Since `tmp_buf_end - tmp_buf` ends up being negative (due to our overflow), the signed integer will cast to unsigned, which is a HUGE positive number. Then `bpf_trace_copy_string` starts copying a large amount of memory into `tmp_buf`, corrupting memory. This corrupted the `hrtimer` structures.
The exploit works by using `%c` to consume bytes, misaligning the buffer, then a specifier such as `%d` that consumes a chunk of memory that will advance the buffer pointer out of bounds. The overflow then overwrites adjacent kernel memory. The fact that the timer code was affected makes sense because it's sensitive to even slight memory corruption.
Therefore, the vulnerability stems from these two bugs, allowing the overwrite of critical kernel data and leading to a crash. To fix this, we need to ensure the code does not advance the pointer beyond the remaining buffer size. The checks need to use the size, as opposed to the signed-to-unsigned conversion, to make sure there's enough space.
|
| 199/2 |
2026/05/18 19:05 |
tool |
codeexpert |
2m
Args: map[Question:Is `tmp_buf_end - tmp_buf < sizeof_cur_arg` an unsigned comparison in C when `tmp_buf_end` and `tmp_buf` are pointers and `sizeof_cur_arg` is `size_t`?]
Results: map[Answer:In C, the expression `tmp_buf_end - tmp_buf < sizeof_cur_arg` is indeed an **unsigned comparison** given the types specified.
Here is the detailed breakdown of why:
1. **Pointer Subtraction**: When you subtract two pointers (`tmp_buf_end - tmp_buf`), the result is of type `ptrdiff_t`. According to the C standard (e.g., C11 §6.5.6), `ptrdiff_t` is a **signed integer type**.
2. **Type of `sizeof_cur_arg`**: The variable `sizeof_cur_arg` is stated to be of type `size_t`. According to the C standard, `size_t` is an **unsigned integer type**.
3. **Usual Arithmetic Conversions**: When an operator (like the `<` comparison operator) has operands of different types, C applies "usual arithmetic conversions" (C11 §6.3.1.8).
* If the operand with the unsigned integer type has a rank greater than or equal to the rank of the type of the other operand, the operand with the signed integer type is converted to the type of the operand with the unsigned integer type.
4. **Ranks in the Linux Kernel**: In the Linux kernel, `size_t` and `ptrdiff_t` are defined to have the same width as a pointer (32 bits on 32-bit architectures and 64 bits on 64-bit architectures). Consequently, they have the same integer conversion rank.
5. **Conversion**: Because they have the same rank and one is unsigned (`size_t`), the signed operand (`ptrdiff_t`) is converted to the unsigned type (`size_t`).
### Consequences of Unsigned Comparison
This can lead to subtle bugs if `tmp_buf_end` is actually less than `tmp_buf`. In a signed comparison, `tmp_buf_end - tmp_buf` would be negative (e.g., `-1`), and `-1 < sizeof_cur_arg` would be true (assuming `sizeof_cur_arg > 0`).
However, in an **unsigned comparison**:
* The negative `ptrdiff_t` value is converted to a very large `size_t` value (e.g., `-1` becomes `SIZE_MAX`).
* The comparison `SIZE_MAX < sizeof_cur_arg` will be **false** for any typical value of `sizeof_cur_arg`.
This specific pattern has been the source of security vulnerabilities in various software (including the Linux kernel in the past), where a bounds check was bypassed because a negative offset was treated as a very large positive value during an unsigned comparison.]
|
| 200/3 |
2026/05/18 19:05 |
agent |
codeexpert |
2m
Model:
Instruction:
You are a capable Linux kernel developer tasked with researching complex questions
about kernel source code. You will be given a concrete question, and need to provide
a concrete answer.
Use tools extensively while researching the question. Don't make assumptions,
or rely on your previous knowledge about the kernel source code, use available tools
to access the actual source code.
Use all available sources of information:
- kernel source code
- documentation in the Documentation dir in the source tree
- git commits descriptions, git blame
Prefer calling several tools at the same time to save round-trips.
Prompt: Is `tmp_buf_end - tmp_buf < sizeof_cur_arg` an unsigned comparison in C when `tmp_buf_end` and `tmp_buf` are pointers and `sizeof_cur_arg` is `size_t`?
Reply: In C, the expression `tmp_buf_end - tmp_buf < sizeof_cur_arg` is indeed an **unsigned comparison** given the types specified.
Here is the detailed breakdown of why:
1. **Pointer Subtraction**: When you subtract two pointers (`tmp_buf_end - tmp_buf`), the result is of type `ptrdiff_t`. According to the C standard (e.g., C11 §6.5.6), `ptrdiff_t` is a **signed integer type**.
2. **Type of `sizeof_cur_arg`**: The variable `sizeof_cur_arg` is stated to be of type `size_t`. According to the C standard, `size_t` is an **unsigned integer type**.
3. **Usual Arithmetic Conversions**: When an operator (like the `<` comparison operator) has operands of different types, C applies "usual arithmetic conversions" (C11 §6.3.1.8).
* If the operand with the unsigned integer type has a rank greater than or equal to the rank of the type of the other operand, the operand with the signed integer type is converted to the type of the operand with the unsigned integer type.
4. **Ranks in the Linux Kernel**: In the Linux kernel, `size_t` and `ptrdiff_t` are defined to have the same width as a pointer (32 bits on 32-bit architectures and 64 bits on 64-bit architectures). Consequently, they have the same integer conversion rank.
5. **Conversion**: Because they have the same rank and one is unsigned (`size_t`), the signed operand (`ptrdiff_t`) is converted to the unsigned type (`size_t`).
### Consequences of Unsigned Comparison
This can lead to subtle bugs if `tmp_buf_end` is actually less than `tmp_buf`. In a signed comparison, `tmp_buf_end - tmp_buf` would be negative (e.g., `-1`), and `-1 < sizeof_cur_arg` would be true (assuming `sizeof_cur_arg > 0`).
However, in an **unsigned comparison**:
* The negative `ptrdiff_t` value is converted to a very large `size_t` value (e.g., `-1` becomes `SIZE_MAX`).
* The comparison `SIZE_MAX < sizeof_cur_arg` will be **false** for any typical value of `sizeof_cur_arg`.
This specific pattern has been the source of security vulnerabilities in various software (including the Linux kernel in the past), where a bounds check was bypassed because a negative offset was treated as a very large positive value during an unsigned comparison.
|
| 201/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 2306
output: 33
thoughts: 351
Thoughts: **Analyzing a C Comparison in Kernel Context**
Okay, so I'm looking at this C expression: `tmp_buf_end - tmp_buf < sizeof_cur_arg`. My immediate thought is, "Is this an unsigned comparison?" Given that `tmp_buf_end` and `tmp_buf` are pointers and `sizeof_cur_arg` is a `size_t`, which is unsigned, that's the core question.
In standard C, the difference of two pointers gives you a `ptrdiff_t`, which is signed. The problem comes from comparing that signed `ptrdiff_t` to the unsigned `size_t`. The usual arithmetic conversions kick in. If `size_t` and `ptrdiff_t` have the same rank (which they usually do, both matching the pointer size), then the `ptrdiff_t` will be converted to `size_t` for the comparison. Therefore, yes, in most standard implementations, it *should* be an unsigned comparison.
But, I know better than to just assume. This is kernel code, so I need to verify. I need to make sure the types are as I expect them to be, especially if this is related to a specific code snippet or potentially a security vulnerability. That's what makes this interesting and reminds me of the kind of work I usually do. The mention of "researching complex questions about kernel source code" makes me think this isn't just a generic C question. It sounds like this could relate to a specific bug, a subtle overflow, or some tricky memory management pattern in the kernel. Therefore, I need to search the kernel source code for this specific expression, see where it appears, and understand the context. That's the best next step.
|
| 202/4 |
2026/05/18 19:05 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:tmp_buf_end - tmp_buf < sizeof_cur_arg]
Results: map[References:<nil> TruncatedOutput:false]
|
| 203/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 3085
output: 23
thoughts: 58
Thoughts: **Navigating the Codebase**
Okay, so I'm trying to locate a specific expression within this codebase, but it's not popping up directly with a simple `codesearch-find-references`. That's a classic sign that it's probably not a standalone, self-contained unit. It's likely a composite thing, built up from several pieces. I need to be more strategic.
Alright, instead of looking for the whole thing, I'll break it down. Let's start with a focused search. Given what I'm dealing with, let's try searching for the component parts, specifically `tmp_buf_end - tmp_buf`. That should be a good starting point. Hopefully, that will lead me to the larger construct I am after.
|
| 204/4 |
2026/05/18 19:05 |
tool |
codesearch-find-references |
0m
Error:
Args:
Results: map[References:<nil> TruncatedOutput:false]
|
| 205/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 3351
output: 23
thoughts: 44
Thoughts: Okay, here's my take on those thoughts:
**Navigating the Prompt & Tool Use**
Alright, the wording of this prompt is a bit ambiguous – are they really after something specific, or just a general C programming question? The variable names are throwing me off, they might not be exactly what's used. But, the explicit instruction to use "tools" suggests I should dive into some research. Rather than guess, let's start with a focused search. Let's see what comes up if I search for something as specific as `sizeof_cur_arg`. That should at least give me a solid base to understand the intent.
|
| 206/4 |
2026/05/18 19:05 |
tool |
codesearch-find-references |
0m
Error:
Args:
Results: map[References:<nil> TruncatedOutput:false]
|
| 207/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 3574
output: 18
thoughts: 151
Thoughts: **Navigating the Codebase**
Okay, so I'm trying to locate something in this codebase, likely related to buffer management or the filesystem. My initial instinct was to reach for `grep` or a similar command-line tool, but I recall I don't have that directly available. However, I *do* have `codesearch-dir-index` and `codesearch-file-index`, which are probably sufficient for this task.
Given that I'm looking for usages of `tmp_buf`, I'm going to try to discover references to that specific string. It sounds like something related to temporary buffers in either a filesystem or memory management.
That said, before I go down that path, it might be more fruitful to get a broader understanding of the context. I think I'll look for comparisons between `ptrdiff_t` and `size_t` because those are often clues.
Wait a moment, I can probably search more efficiently using `codesearch-find-references`. But it requires an entity name! That might be more difficult than just searching the whole code base.
Okay, let's start simple. Let's see if I can find any file that even *contains* the string `tmp_buf`. That should be quick and might give me a good jumping-off point.
|
| 208/4 |
2026/05/18 19:05 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:<nil> Subdirs:[Documentation LICENSES arch block certs crypto drivers fs include init io_uring ipc kernel lib mm net rust samples scripts security sound tools usr virt]]
|
| 209/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 4134
output: 21
thoughts: 92
Thoughts: Okay, here's my thought process as I approach this.
**Investigating `tmp_buf` in Kernel Code**
Alright, so the task at hand involves this `tmp_buf` variable. My initial instinct is to dig into the `fs/` and `kernel/` directories – those are the usual suspects when dealing with low-level kernel operations and data handling. It's a pretty common pattern to find temporary buffers hanging around in those areas.
But, before I go brute-forcing it with a file system search, let me think a bit smarter. Rather than just searching for the name itself, I'll leverage my knowledge of the codebase and utilize `codesearch-find-references`. I'll try searching for references to a related, more general name, something that's likely to appear in the same function where `tmp_buf` might be used. This could help narrow down the scope significantly and identify relevant contexts.
Wait a second… the prompt is pretty specific. It zeroes in on the *expression* itself. That suggests this isn't just a generic variable name issue. It might be tied to a specific vulnerability, perhaps a recent CVE or a known bug that mentions `tmp_buf` in its description. Hmm, that's a good angle to consider.
Right, so, given that the search is expression specific, let's start by trying a more direct search for `tmp_buf` specifically as a variable name. Let's see what that yields first. I can refine my approach after I get some preliminary results.
|
| 210/4 |
2026/05/18 19:05 |
tool |
codesearch-find-references |
0m
Error:
Args:
Results: map[References:<nil> TruncatedOutput:false]
|
| 211/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 4594
output: 24
thoughts: 109
Thoughts: **Debugging a Mysterious Variable**
Okay, that's not what I expected at all. I was certain `codesearch-find-references` would flag that use of `sizeof_cur_arg`, but it's returning empty. Hmm, could this be a local variable that isn't indexed globally? `codesearch-find-references` isn't perfect, especially with scoping.
Alright, let's double-check. I'll search for `sizeof_cur_arg` again, just to be sure I didn't introduce a typo when typing it. Nope, still nothing.
Alright, let's widen the search a little. What about `tmp_buf_end`? Ah, still no joy.
Time to think this through. "cur_arg"... that screams argument parsing. Now, where is argument parsing typically handled in this codebase? My instincts are telling me to look in `fs/exec.c`. That seems like the most likely place based on the variable names. Let's start there.
|
| 212/4 |
2026/05/18 19:05 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:fs/exec.c]
Results: map[Entities:[map[Kind:function Name:__do_sys_execve] map[Kind:function Name:__do_sys_execveat] map[Kind:function Name:__register_binfmt] map[Kind:function Name:__se_sys_execve] map[Kind:function Name:__se_sys_execveat] map[Kind:function Name:__set_task_comm] map[Kind:function Name:__x64_sys_execve] map[Kind:function Name:__x64_sys_execveat] map[Kind:function Name:acct_arg_size] map[Kind:function Name:alloc_bprm] map[Kind:function Name:begin_new_exec] map[Kind:function Name:bprm_change_interp] map[Kind:function Name:bprm_creds_from_file] map[Kind:function Name:bprm_execve] map[Kind:function Name:bprm_fill_uid] map[Kind:function Name:bprm_hit_stack_limit] map[Kind:function Name:bprm_mm_init] map[Kind:function Name:bprm_set_stack_limit] map[Kind:function Name:bprm_stack_limits] map[Kind:function Name:check_unsafe_exec] map[Kind:function Name:class_bprm_constructor] map[Kind:function Name:class_bprm_destructor] map[Kind:function Name:copy_string_kernel] map[Kind:function Name:copy_strings] map[Kind:function Name:copy_strings_kernel] map[Kind:function Name:count] map[Kind:function Name:count_strings_kernel] map[Kind:function Name:de_thread] map[Kind:function Name:do_close_execat] map[Kind:function Name:do_execveat_common] map[Kind:function Name:do_open_execat] map[Kind:function Name:exec_binprm] map[Kind:function Name:exec_mmap] map[Kind:function Name:finalize_exec] map[Kind:function Name:flush_arg_page] map[Kind:function Name:free_arg_pages] map[Kind:function Name:free_bprm] map[Kind:function Name:get_arg_page] map[Kind:function Name:get_user_arg_ptr] map[Kind:function Name:init_fs_exec_sysctls] map[Kind:function Name:kernel_execve] map[Kind:function Name:native_arg] map[Kind:function Name:open_exec] map[Kind:function Name:path_noexec] map[Kind:function Name:prepare_binprm] map[Kind:function Name:prepare_bprm_creds] map[Kind:function Name:proc_dointvec_minmax_coredump] map[Kind:function Name:put_arg_page] map[Kind:function Name:put_binfmt] map[Kind:function Name:remove_arg_zero] map[Kind:function Name:search_binary_handler] map[Kind:function Name:set_binfmt] map[Kind:function Name:set_dumpable] map[Kind:function Name:setup_arg_pages] map[Kind:function Name:setup_new_exec] map[Kind:function Name:unregister_binfmt] map[Kind:function Name:unshare_sighand] map[Kind:function Name:valid_arg_len] map[Kind:function Name:would_dump] map[Kind:struct Name:user_arg_ptr] map[Kind:global_variable Name:__UNIQUE_ID_addressable___register_binfmt_885] map[Kind:global_variable Name:__UNIQUE_ID_addressable_begin_new_exec_918] map[Kind:global_variable Name:__UNIQUE_ID_addressable_bprm_change_interp_922] map[Kind:global_variable Name:__UNIQUE_ID_addressable_copy_string_kernel_903] map[Kind:global_variable Name:__UNIQUE_ID_addressable_finalize_exec_921] map[Kind:global_variable Name:__UNIQUE_ID_addressable_init_fs_exec_sysctls_928] map[Kind:global_variable Name:__UNIQUE_ID_addressable_open_exec_908] map[Kind:global_variable Name:__UNIQUE_ID_addressable_remove_arg_zero_925] map[Kind:global_variable Name:__UNIQUE_ID_addressable_set_binfmt_926] map[Kind:global_variable Name:__UNIQUE_ID_addressable_setup_arg_pages_907] map[Kind:global_variable Name:__UNIQUE_ID_addressable_setup_new_exec_920] map[Kind:global_variable Name:__UNIQUE_ID_addressable_unregister_binfmt_886] map[Kind:global_variable Name:__UNIQUE_ID_addressable_would_dump_919] map[Kind:global_variable Name:binfmt_lock] map[Kind:global_variable Name:formats] map[Kind:global_variable Name:fs_exec_sysctls] map[Kind:global_variable Name:suid_dumpable] map[Kind:typedef Name:class_bprm_t] map[Kind:typedef Name:lock_bprm_t]]]
|
| 213/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 6440
output: 27
thoughts: 1160
Thoughts: **Analyzing the Arithmetic Conversion in a C Comparison**
Okay, so I'm presented with a comparison involving `tmp_buf_end - tmp_buf` and `sizeof_cur_arg`. I don't immediately see the variables in the immediate context, so I need to figure out the types and whether the comparison is unsigned. Let's start with where these might be located, maybe in a `drivers/` directory, I can search for `tmp_buf` within it. I could try `codesearch-find-references` to see if those variable are referenced anywhere. Even try `codesearch-find-references` on the name of the buffer. If that doesn't yield anything, I'll fall back to C rules.
First, let's determine the types involved. `tmp_buf_end - tmp_buf` is a `ptrdiff_t` (signed), and `sizeof_cur_arg` is a `size_t` (unsigned), by definition, which is critical. The question hinges on if that difference results in an unsigned comparison. So, based on C99/C11/C17, Section 6.3.1.8 ("Usual arithmetic conversions"), I know that if the operands have the same rank and one is signed and the other is unsigned, the signed operand is converted to the unsigned type.
`ptrdiff_t` and `size_t` typically have the same rank on modern systems (e.g., both 64-bit on 64-bit systems, or 32-bit on 32-bit systems). If `ptrdiff_t` is converted to `size_t`, it becomes an unsigned comparison, and any negative result from the subtraction would wrap around to a very large positive `size_t` value.
The question explicitly asks if it *is* an unsigned comparison. The answer is yes. It's an unsigned comparison due to the usual arithmetic conversions.
To make absolutely sure, I'd like to see an example to confirm the types. I'll need to double check the type of `ptrdiff_t` and `size_t`, or where they are used. Even try to search for the variables in `fs/`.
So, even though I don't have the context of the code, I know the answer: yes, this *is* an unsigned comparison due to the implicit type conversion in C.
|
| 214/4 |
2026/05/18 19:05 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:fs/binfmt_elf.c]
Results: map[Entities:[map[Kind:function Name:__fill_note] map[Kind:function Name:arch_check_elf] map[Kind:function Name:arch_elf_pt_proc] map[Kind:function Name:create_elf_tables] map[Kind:function Name:do_thread_regset_writeback] map[Kind:function Name:elf_core_dump] map[Kind:function Name:elf_coredump_get_mm_eflags] map[Kind:function Name:elf_coredump_set_mm_eflags] map[Kind:function Name:elf_load] map[Kind:function Name:elf_map] map[Kind:function Name:elf_read] map[Kind:function Name:exit_elf_binfmt] map[Kind:function Name:fill_auxv_note] map[Kind:function Name:fill_elf_header] map[Kind:function Name:fill_elf_note_phdr] map[Kind:function Name:fill_extnum_info] map[Kind:function Name:fill_files_note] map[Kind:function Name:fill_note_info] map[Kind:function Name:fill_prstatus] map[Kind:function Name:fill_psinfo] map[Kind:function Name:fill_siginfo_note] map[Kind:function Name:fill_thread_core_info] map[Kind:function Name:free_note_info] map[Kind:function Name:init_elf_binfmt] map[Kind:function Name:load_elf_binary] map[Kind:function Name:load_elf_interp] map[Kind:function Name:load_elf_phdrs] map[Kind:function Name:make_prot] map[Kind:function Name:maximum_alignment] map[Kind:function Name:notesize] map[Kind:function Name:padzero] map[Kind:function Name:parse_elf_properties] map[Kind:function Name:parse_elf_property] map[Kind:function Name:total_mapping_size] map[Kind:function Name:write_note_info] map[Kind:function Name:writenote] map[Kind:struct Name:arch_elf_state] map[Kind:struct Name:elf_note_info] map[Kind:struct Name:elf_thread_core_info] map[Kind:struct Name:memelfnote] map[Kind:global_variable Name:__UNIQUE_ID_addressable_init_elf_binfmt_537] map[Kind:global_variable Name:__exitcall_exit_elf_binfmt] map[Kind:global_variable Name:elf_format]]]
|
| 215/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 8881
output: 26
thoughts: 24
Thoughts: Okay, here's my summary, tailored as if I'm an expert in the field thinking through this:
**Where Am I?**
Hmm, I'm drawing a blank. Not immediately obvious where to start this particular investigation. My mind is casting about... Let's see... thinking about file systems... Okay, my gut says start with the proc filesystem. That's a reasonable place to look for system information and process data. Perhaps there are some low-level internals to glean. Let me probe around the `/proc/` mountpoint. Specifically, `fs/proc/` sounds familiar, a good starting point to explore. And... Yes, that rings a bell! It feels like `fs/proc/base.c` is the most obvious candidate for some core proc filesystem functionality. Let's see what that file might contain, and if that gives me a better idea of the state of things.
|
| 216/4 |
2026/05/18 19:05 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:fs/proc/base.c]
Results: map[Entities:[map[Kind:function Name:__mem_open] map[Kind:function Name:__set_oom_adj] map[Kind:function Name:auxv_open] map[Kind:function Name:auxv_read] map[Kind:function Name:comm_open] map[Kind:function Name:comm_show] map[Kind:function Name:comm_write] map[Kind:function Name:dname_to_vma_addr] map[Kind:function Name:do_io_accounting] map[Kind:function Name:do_proc_readlink] map[Kind:function Name:early_proc_mem_force_override] map[Kind:function Name:environ_open] map[Kind:function Name:environ_read] map[Kind:function Name:first_tid] map[Kind:function Name:get_mm_cmdline] map[Kind:function Name:get_mm_proctitle] map[Kind:function Name:get_task_cmdline] map[Kind:function Name:get_task_root] map[Kind:function Name:has_pid_permissions] map[Kind:function Name:lock_trace] map[Kind:function Name:map_files_d_revalidate] map[Kind:function Name:map_files_get_link] map[Kind:function Name:mem_lseek] map[Kind:function Name:mem_open] map[Kind:function Name:mem_read] map[Kind:function Name:mem_release] map[Kind:function Name:mem_rw] map[Kind:function Name:mem_write] map[Kind:function Name:next_tgid] map[Kind:function Name:next_tid] map[Kind:function Name:oom_adj_read] map[Kind:function Name:oom_adj_write] map[Kind:function Name:oom_score_adj_read] map[Kind:function Name:oom_score_adj_write] map[Kind:function Name:pid_delete_dentry] map[Kind:function Name:pid_entry_nlink] map[Kind:function Name:pid_getattr] map[Kind:function Name:pid_revalidate] map[Kind:function Name:pid_update_inode] map[Kind:function Name:proc_attr_dir_lookup] map[Kind:function Name:proc_attr_dir_readdir] map[Kind:function Name:proc_coredump_filter_read] map[Kind:function Name:proc_coredump_filter_write] map[Kind:function Name:proc_cwd_link] map[Kind:function Name:proc_dir_llseek] map[Kind:function Name:proc_exe_link] map[Kind:function Name:proc_fail_nth_read] map[Kind:function Name:proc_fail_nth_write] map[Kind:function Name:proc_fault_inject_read] map[Kind:function Name:proc_fault_inject_write] map[Kind:function Name:proc_fd_access_allowed] map[Kind:function Name:proc_fill_cache] map[Kind:function Name:proc_flush_pid] map[Kind:function Name:proc_gid_map_open] map[Kind:function Name:proc_id_map_open] map[Kind:function Name:proc_id_map_release] map[Kind:function Name:proc_inode_is_dead] map[Kind:function Name:proc_loginuid_read] map[Kind:function Name:proc_loginuid_write] map[Kind:function Name:proc_map_files_get_link] map[Kind:function Name:proc_map_files_instantiate] map[Kind:function Name:proc_map_files_lookup] map[Kind:function Name:proc_map_files_readdir] map[Kind:function Name:proc_mem_foll_force] map[Kind:function Name:proc_mem_open] map[Kind:function Name:proc_nochmod_setattr] map[Kind:function Name:proc_oom_score] map[Kind:function Name:proc_pid_attr_open] map[Kind:function Name:proc_pid_attr_read] map[Kind:function Name:proc_pid_attr_write] map[Kind:function Name:proc_pid_cmdline_read] map[Kind:function Name:proc_pid_evict_inode] map[Kind:function Name:proc_pid_get_link] map[Kind:function Name:proc_pid_instantiate] map[Kind:function Name:proc_pid_ksm_merging_pages] map[Kind:function Name:proc_pid_ksm_stat] map[Kind:function Name:proc_pid_limits] map[Kind:function Name:proc_pid_lookup] map[Kind:function Name:proc_pid_make_base_inode] map[Kind:function Name:proc_pid_make_inode] map[Kind:function Name:proc_pid_permission] map[Kind:function Name:proc_pid_personality] map[Kind:function Name:proc_pid_readdir] map[Kind:function Name:proc_pid_readlink] map[Kind:function Name:proc_pid_schedstat] map[Kind:function Name:proc_pid_stack] map[Kind:function Name:proc_pid_syscall] map[Kind:function Name:proc_pid_wchan] map[Kind:function Name:proc_pident_instantiate] map[Kind:function Name:proc_pident_lookup] map[Kind:function Name:proc_pident_readdir] map[Kind:function Name:proc_projid_map_open] map[Kind:function Name:proc_root_link] map[Kind:function Name:proc_sessionid_read] map[Kind:function Name:proc_setgroups_open] map[Kind:function Name:proc_setgroups_release] map[Kind:function Name:proc_single_open] map[Kind:function Name:proc_single_show] map[Kind:function Name:proc_smack_attr_dir_iterate] map[Kind:function Name:proc_smack_attr_dir_lookup] map[Kind:function Name:proc_task_getattr] map[Kind:function Name:proc_task_instantiate] map[Kind:function Name:proc_task_lookup] map[Kind:function Name:proc_task_readdir] map[Kind:function Name:proc_tgid_base_lookup] map[Kind:function Name:proc_tgid_base_readdir] map[Kind:function Name:proc_tgid_io_accounting] map[Kind:function Name:proc_tid_base_lookup] map[Kind:function Name:proc_tid_base_readdir] map[Kind:function Name:proc_tid_comm_permission] map[Kind:function Name:proc_tid_io_accounting] map[Kind:function Name:proc_timers_open] map[Kind:function Name:proc_uid_map_open] map[Kind:function Name:sched_autogroup_open] map[Kind:function Name:sched_autogroup_show] map[Kind:function Name:sched_autogroup_write] map[Kind:function Name:sched_open] map[Kind:function Name:sched_show] map[Kind:function Name:sched_write] map[Kind:function Name:set_proc_pid_nlink] map[Kind:function Name:show_timer] map[Kind:function Name:task_dump_owner] map[Kind:function Name:tgid_pidfd_to_pid] map[Kind:function Name:timens_offsets_open] map[Kind:function Name:timens_offsets_show] map[Kind:function Name:timens_offsets_write] map[Kind:function Name:timers_next] map[Kind:function Name:timers_start] map[Kind:function Name:timers_stop] map[Kind:function Name:timerslack_ns_open] map[Kind:function Name:timerslack_ns_show] map[Kind:function Name:timerslack_ns_write] map[Kind:function Name:unlock_trace] map[Kind:struct Name:limit_names] map[Kind:struct Name:map_files_info] map[Kind:struct Name:pid_entry] map[Kind:struct Name:tgid_iter] map[Kind:struct Name:timers_private] map[Kind:global_variable Name:__setup_early_proc_mem_force_override] map[Kind:global_variable Name:__setup_str_early_proc_mem_force_override] map[Kind:global_variable Name:attr_dir_stuff] map[Kind:global_variable Name:lnames] map[Kind:global_variable Name:pid_dentry_operations] map[Kind:global_variable Name:proc_attr_dir_inode_operations] map[Kind:global_variable Name:proc_attr_dir_operations] map[Kind:global_variable Name:proc_auxv_operations] map[Kind:global_variable Name:proc_coredump_filter_operations] map[Kind:global_variable Name:proc_def_inode_operations] map[Kind:global_variable Name:proc_environ_operations] map[Kind:global_variable Name:proc_fail_nth_operations] map[Kind:global_variable Name:proc_fault_inject_operations] map[Kind:global_variable Name:proc_gid_map_operations] map[Kind:global_variable Name:proc_loginuid_operations] map[Kind:global_variable Name:proc_map_files_inode_operations] map[Kind:global_variable Name:proc_map_files_link_inode_operations] map[Kind:global_variable Name:proc_map_files_operations] map[Kind:global_variable Name:proc_mem_force_override] map[Kind:global_variable Name:proc_mem_force_table] map[Kind:global_variable Name:proc_mem_operations] map[Kind:global_variable Name:proc_oom_adj_operations] map[Kind:global_variable Name:proc_oom_score_adj_operations] map[Kind:global_variable Name:proc_pid_attr_operations] map[Kind:global_variable Name:proc_pid_cmdline_ops] map[Kind:global_variable Name:proc_pid_link_inode_operations] map[Kind:global_variable Name:proc_pid_sched_autogroup_operations] map[Kind:global_variable Name:proc_pid_sched_operations] map[Kind:global_variable Name:proc_pid_set_comm_operations] map[Kind:global_variable Name:proc_pid_set_timerslack_ns_operations] map[Kind:global_variable Name:proc_projid_map_operations] map[Kind:global_variable Name:proc_sessionid_operations] map[Kind:global_variable Name:proc_setgroups_operations] map[Kind:global_variable Name:proc_single_file_operations] map[Kind:global_variable Name:proc_smack_attr_dir_inode_ops] map[Kind:global_variable Name:proc_smack_attr_dir_ops] map[Kind:global_variable Name:proc_task_inode_operations] map[Kind:global_variable Name:proc_task_operations] map[Kind:global_variable Name:proc_tgid_base_inode_operations] map[Kind:global_variable Name:proc_tgid_base_operations] map[Kind:global_variable Name:proc_tid_base_inode_operations] map[Kind:global_variable Name:proc_tid_base_operations] map[Kind:global_variable Name:proc_tid_comm_inode_operations] map[Kind:global_variable Name:proc_timens_offsets_operations] map[Kind:global_variable Name:proc_timers_operations] map[Kind:global_variable Name:proc_timers_seq_ops] map[Kind:global_variable Name:proc_uid_map_operations] map[Kind:global_variable Name:smack_attr_dir_stuff] map[Kind:global_variable Name:tgid_base_stuff] map[Kind:global_variable Name:tid_base_stuff] map[Kind:global_variable Name:tid_map_files_dentry_operations] map[Kind:enum Name:proc_mem_force]]]
|
| 217/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 12520
output: 44
thoughts: 24
Thoughts: Here's my take on those thoughts, tailored for an expert audience:
**Diving into the Process Command Line Read**
Okay, so I need to understand how the `/proc/[pid]/cmdline` file works. My instinct is to head straight for the kernel source. Specifically, the function responsible for handling the read operation: `proc_pid_cmdline_read`. I expect it'll be defined in `fs/proc/base.c`. That file's usually the go-to for process-related procfs entries, and the naming convention suggests the right place. I'm curious about the mechanics here – how does the kernel sanitize potentially sensitive or malicious data, how's the command line formatted, and what's the access control like? Time to dig in and see how it's handled at the lowest level. I'm expecting some interesting edge cases to explore, things like how the kernel deals with argument lists exceeding the maximum length allowed, potential race conditions and so on. Let's see if the code has anything interesting or unexpected to offer.
|
| 218/4 |
2026/05/18 19:05 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:fs/proc/base.c IncludeLines:true Name:proc_pid_cmdline_read]
Results: map[SourceCode: 393: static ssize_t proc_pid_cmdline_read(struct file *file, char __user *buf,
394: size_t count, loff_t *pos)
395: {
396: struct task_struct *tsk;
397: ssize_t ret;
398:
399: BUG_ON(*pos < 0);
400:
401: tsk = get_proc_task(file_inode(file));
402: if (!tsk)
403: return -ESRCH;
404: ret = get_task_cmdline(tsk, buf, count, pos);
405: put_task_struct(tsk);
406: if (ret > 0)
407: *pos += ret;
408: return ret;
409: }
SourceFile:fs/proc/base.c]
|
| 219/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 13086
output: 42
thoughts: 16
Thoughts: Okay, here's my summarized thought process, presented as if I'm the one thinking it:
**Analyzing the Function Call**
Alright, the code's calling `get_task_cmdline`. That function name rings a bell... it's clearly crucial to understanding how a specific task's command-line is being handled. I need to dig into `get_task_cmdline` immediately. Let's start by examining the definition and where it's called from. I'm already anticipating some interesting edge cases, depending on how this is implemented.
|
| 220/4 |
2026/05/18 19:05 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:fs/proc/base.c IncludeLines:true Name:get_task_cmdline]
Results: map[SourceCode: 378: static ssize_t get_task_cmdline(struct task_struct *tsk, char __user *buf,
379: size_t count, loff_t *pos)
380: {
381: struct mm_struct *mm;
382: ssize_t ret;
383:
384: mm = get_task_mm(tsk);
385: if (!mm)
386: return 0;
387:
388: ret = get_mm_cmdline(mm, buf, count, pos);
389: mmput(mm);
390: return ret;
391: }
SourceFile:fs/proc/base.c]
|
| 221/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 13486
output: 42
thoughts: 10
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:
**Analyzing the Function Call: `get_mm_cmdline`**
Right, so the system is calling `get_mm_cmdline`. That immediately signals a focus on memory management and the command-line arguments related to that. My initial thought process revolves around the specifics of this function: What's the context? What's the target platform? Knowing those details is crucial for assessing its purpose. Is it retrieving kernel command-line parameters for memory tuning, debugging, or system configuration? Or perhaps it's concerned with user-space memory management via specific application launch options? Depending on the architecture, the retrieval method could vary greatly. Are we dealing with something reading from `/proc/cmdline`, or a more specialized interface? Understanding the underlying implementation is key to interpreting the implications of this call. The possibilities range from something as basic as reading a static configuration file, to interfacing with complex firmware or hardware registers. My brain immediately starts formulating hypotheses.
|
| 222/4 |
2026/05/18 19:05 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:fs/proc/base.c IncludeLines:true Name:get_mm_cmdline]
Results: map[SourceCode: 291: static ssize_t get_mm_cmdline(struct mm_struct *mm, char __user *buf,
292: size_t count, loff_t *ppos)
293: {
294: unsigned long arg_start, arg_end, env_start, env_end;
295: unsigned long pos, len;
296: char *page, c;
297:
298: /* Check if process spawned far enough to have cmdline. */
299: if (!mm->env_end)
300: return 0;
301:
302: spin_lock(&mm->arg_lock);
303: arg_start = mm->arg_start;
304: arg_end = mm->arg_end;
305: env_start = mm->env_start;
306: env_end = mm->env_end;
307: spin_unlock(&mm->arg_lock);
308:
309: if (arg_start >= arg_end)
310: return 0;
311:
312: /*
313: * We allow setproctitle() to overwrite the argument
314: * strings, and overflow past the original end. But
315: * only when it overflows into the environment area.
316: */
317: if (env_start != arg_end || env_end < env_start)
318: env_start = env_end = arg_end;
319: len = env_end - arg_start;
320:
321: /* We're not going to care if "*ppos" has high bits set */
322: pos = *ppos;
323: if (pos >= len)
324: return 0;
325: if (count > len - pos)
326: count = len - pos;
327: if (!count)
328: return 0;
329:
330: /*
331: * Magical special case: if the argv[] end byte is not
332: * zero, the user has overwritten it with setproctitle(3).
333: *
334: * Possible future enhancement: do this only once when
335: * pos is 0, and set a flag in the 'struct file'.
336: */
337: if (access_remote_vm(mm, arg_end-1, &c, 1, FOLL_ANON) == 1 && c)
338: return get_mm_proctitle(mm, buf, count, pos, arg_start);
339:
340: /*
341: * For the non-setproctitle() case we limit things strictly
342: * to the [arg_start, arg_end[ range.
343: */
344: pos += arg_start;
345: if (pos < arg_start || pos >= arg_end)
346: return 0;
347: if (count > arg_end - pos)
348: count = arg_end - pos;
349:
350: page = (char *)__get_free_page(GFP_KERNEL);
351: if (!page)
352: return -ENOMEM;
353:
354: len = 0;
355: while (count) {
356: int got;
357: size_t size = min_t(size_t, PAGE_SIZE, count);
358:
359: got = access_remote_vm(mm, pos, page, size, FOLL_ANON);
360: if (got <= 0)
361: break;
362: got -= copy_to_user(buf, page, got);
363: if (unlikely(!got)) {
364: if (!len)
365: len = -EFAULT;
366: break;
367: }
368: pos += got;
369: buf += got;
370: len += got;
371: count -= got;
372: }
373:
374: free_page((unsigned long)page);
375: return len;
376: }
SourceFile:fs/proc/base.c]
|
| 223/4 |
2026/05/18 19:05 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 15031
output: 20
thoughts: 68
Thoughts: Okay, here's my interpretation of those thoughts, tailored for an expert audience, and delivered as a first-person summary:
**Digging Deeper for That Elusive Expression**
Right, this initial search didn't yield the specific expression I was expecting. Seems like a dead end for now. I need to broaden my scope. Given that this is kernel code, the term isn't a likely candidate to be declared or defined in user-space code.
My next step is to initiate a deeper dive. I'll rerun the search for `tmp_buf` in the kernel source, but this time I'll make it much more comprehensive. Instead of focusing on specific files at first, I'm going to leverage `codesearch-dir-index` and apply it to the `kernel/` and `lib/` directories, to perform a global sweep. I'm aiming to identify *any* file, no matter how obscure, that might contain a reference to that variable or struct. This more aggressive search strategy should flush out any hidden definitions or uses, even if they're indirect or in a more specialized subsystem. Hopefully, this broader search will yield some new leads.
|
| 224/4 |
2026/05/18 19:06 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:[acct.c async.c audit.c audit.h audit_fsnotify.c audit_tree.c audit_watch.c auditfilter.c auditsc.c backtracetest.c bounds.c capability.c cfi.c compat.c configs.c context_tracking.c cpu.c cpu_pm.c crash_core.c crash_core_test.c crash_dump_dm_crypt.c crash_reserve.c cred.c delayacct.c dma.c elfcorehdr.c exec_domain.c exit.c exit.h extable.c fail_function.c fork.c freezer.c groups.c hung_task.c iomem.c irq_work.c jump_label.c kallsyms.c kallsyms_internal.h kallsyms_selftest.c kallsyms_selftest.h kcmp.c kcov.c kexec.c kexec_core.c kexec_elf.c kexec_file.c kexec_internal.h kheaders.c kprobes.c kstack_erase.c ksyms_common.c ksysfs.c kthread.c latencytop.c module_signature.c notifier.c nscommon.c nsproxy.c nstree.c padata.c panic.c params.c pid.c pid_namespace.c pid_sysctl.h profile.c ptrace.c range.c reboot.c regset.c relay.c resource.c resource_kunit.c rseq.c scftorture.c scs.c seccomp.c signal.c smp.c smpboot.c smpboot.h softirq.c stacktrace.c static_call.c static_call_inline.c stop_machine.c sys.c sys_ni.c sysctl-test.c sysctl.c task_work.c taskstats.c torture.c tracepoint.c tsacct.c ucount.c uid16.c uid16.h umh.c up.c user-return-notifier.c user.c user_namespace.c utsname.c utsname_sysctl.c vhost_task.c vmcore_info.c watch_queue.c watchdog.c watchdog_buddy.c watchdog_perf.c workqueue.c workqueue_internal.h] Subdirs:[bpf cgroup configs debug dma entry events futex gcov irq kcsan livepatch liveupdate locking module power printk rcu sched time trace unwind]]
|
| 225/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 16363
output: 24
thoughts: 11
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Diving into Kernel Parameters**
Alright, time to roll up my sleeves. The issue at hand is likely manifesting in system behavior governed by kernel parameters. My first thought is to zero in on `kernel/params.c`. This is the logical starting point – the heart of where tunable kernel parameters are defined, initialized, and potentially modified at runtime. I need to understand how the system's runtime configurations interact with the broader system functionality, so this file is the key to understanding all of the options that could be contributing to the specific performance issue, and how they interact. Analyzing that code will give me a better understanding of the possible sources of a particular system behavior.
|
| 226/4 |
2026/05/18 19:06 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/params.c]
Results: map[Entities:[map[Kind:function Name:__modver_version_show] map[Kind:function Name:add_sysfs_param] map[Kind:function Name:check_kparam_locked] map[Kind:function Name:dash2underscore] map[Kind:function Name:free_module_param_attrs] map[Kind:function Name:kernel_add_sysfs_param] map[Kind:function Name:kernel_param_lock] map[Kind:function Name:kernel_param_unlock] map[Kind:function Name:kmalloc_parameter] map[Kind:function Name:lookup_or_create_module_kobject] map[Kind:function Name:maybe_kfree_parameter] map[Kind:function Name:module_attr_show] map[Kind:function Name:module_attr_store] map[Kind:function Name:module_destroy_params] map[Kind:function Name:module_kobj_release] map[Kind:function Name:module_param_sysfs_remove] map[Kind:function Name:module_param_sysfs_setup] map[Kind:function Name:param_array] map[Kind:function Name:param_array_free] map[Kind:function Name:param_array_get] map[Kind:function Name:param_array_set] map[Kind:function Name:param_attr_show] map[Kind:function Name:param_attr_store] map[Kind:function Name:param_check_unsafe] map[Kind:function Name:param_free_charp] map[Kind:function Name:param_get_bool] map[Kind:function Name:param_get_byte] map[Kind:function Name:param_get_charp] map[Kind:function Name:param_get_hexint] map[Kind:function Name:param_get_int] map[Kind:function Name:param_get_invbool] map[Kind:function Name:param_get_long] map[Kind:function Name:param_get_short] map[Kind:function Name:param_get_string] map[Kind:function Name:param_get_uint] map[Kind:function Name:param_get_ullong] map[Kind:function Name:param_get_ulong] map[Kind:function Name:param_get_ushort] map[Kind:function Name:param_set_bint] map[Kind:function Name:param_set_bool] map[Kind:function Name:param_set_bool_enable_only] map[Kind:function Name:param_set_byte] map[Kind:function Name:param_set_charp] map[Kind:function Name:param_set_copystring] map[Kind:function Name:param_set_hexint] map[Kind:function Name:param_set_int] map[Kind:function Name:param_set_invbool] map[Kind:function Name:param_set_long] map[Kind:function Name:param_set_short] map[Kind:function Name:param_set_uint] map[Kind:function Name:param_set_uint_minmax] map[Kind:function Name:param_set_ullong] map[Kind:function Name:param_set_ulong] map[Kind:function Name:param_set_ushort] map[Kind:function Name:param_sysfs_builtin] map[Kind:function Name:param_sysfs_builtin_init] map[Kind:function Name:param_sysfs_init] map[Kind:function Name:parameq] map[Kind:function Name:parameqn] map[Kind:function Name:parse_args] map[Kind:function Name:parse_one] map[Kind:function Name:uevent_filter] map[Kind:function Name:version_sysfs_builtin] map[Kind:struct Name:kmalloced_param] map[Kind:struct Name:module_param_attrs] map[Kind:struct Name:param_attribute] map[Kind:global_variable Name:__UNIQUE_ID_addressable_kernel_param_lock_527] map[Kind:global_variable Name:__UNIQUE_ID_addressable_kernel_param_unlock_528] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_array_ops_523] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_free_charp_511] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_bool_514] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_byte_482] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_charp_510] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_hexint_506] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_int_491] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_invbool_519] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_long_497] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_short_485] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_string_525] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_uint_494] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_ullong_503] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_ulong_500] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_get_ushort_488] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_bint_522] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_bool_515] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_bool_enable_only_517] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_byte_483] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_charp_512] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_hexint_507] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_int_492] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_invbool_520] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_long_498] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_short_486] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_string_526] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_uint_495] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_ullong_504] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_ulong_501] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_ops_ushort_489] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_bint_521] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_bool_513] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_bool_enable_only_516] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_byte_481] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_charp_509] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_copystring_524] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_hexint_505] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_int_490] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_invbool_518] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_long_496] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_short_484] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_uint_493] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_uint_minmax_508] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_ullong_502] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_ulong_499] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_set_ushort_487] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_sysfs_builtin_init_532] map[Kind:global_variable Name:__UNIQUE_ID_addressable_param_sysfs_init_530] map[Kind:global_variable Name:kmalloced_params] map[Kind:global_variable Name:kmalloced_params_lock] map[Kind:global_variable Name:module_ktype] map[Kind:global_variable Name:module_sysfs_ops] map[Kind:global_variable Name:module_uevent_ops] map[Kind:global_variable Name:param_array_ops] map[Kind:global_variable Name:param_lock] map[Kind:global_variable Name:param_ops_bint] map[Kind:global_variable Name:param_ops_bool] map[Kind:global_variable Name:param_ops_bool_enable_only] map[Kind:global_variable Name:param_ops_byte] map[Kind:global_variable Name:param_ops_charp] map[Kind:global_variable Name:param_ops_hexint] map[Kind:global_variable Name:param_ops_int] map[Kind:global_variable Name:param_ops_invbool] map[Kind:global_variable Name:param_ops_long] map[Kind:global_variable Name:param_ops_short] map[Kind:global_variable Name:param_ops_string] map[Kind:global_variable Name:param_ops_uint] map[Kind:global_variable Name:param_ops_ullong] map[Kind:global_variable Name:param_ops_ulong] map[Kind:global_variable Name:param_ops_ushort]]]
|
| 227/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 19596
output: 37
thoughts: 61
Thoughts: **Delving into a Potential Kernel Bug**
Okay, so this question has definitely sparked something. My initial hunch is that we're dealing with a bug related to file system execution, perhaps within `fs/exec.c` or something analogous. That's a reasonable starting point, and I've seen some head-scratchers there in the past.
Wait a second... a memory jogged! I *think* I recall something specifically about `copy_strings` inside `fs/exec.c`. Yes, that function, I have a vague recall of having issues with that in a previous project or review. Time to refresh that memory.
Alright, let's circle back to `fs/exec.c` and zero in on the `copy_strings` function. It's time to dig in and see if this rings any bells or provides a clue to the root cause.
|
| 228/4 |
2026/05/18 19:06 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:fs/exec.c IncludeLines:true Name:copy_strings]
Results: map[SourceCode: 443: /*
444: * 'copy_strings()' copies argument/environment strings from the old
445: * processes's memory to the new process's stack. The call to get_user_pages()
446: * ensures the destination page is created and not swapped out.
447: */
448: static int copy_strings(int argc, struct user_arg_ptr argv,
449: struct linux_binprm *bprm)
450: {
451: struct page *kmapped_page = NULL;
452: char *kaddr = NULL;
453: unsigned long kpos = 0;
454: int ret;
455:
456: while (argc-- > 0) {
457: const char __user *str;
458: int len;
459: unsigned long pos;
460:
461: ret = -EFAULT;
462: str = get_user_arg_ptr(argv, argc);
463: if (IS_ERR(str))
464: goto out;
465:
466: len = strnlen_user(str, MAX_ARG_STRLEN);
467: if (!len)
468: goto out;
469:
470: ret = -E2BIG;
471: if (!valid_arg_len(bprm, len))
472: goto out;
473:
474: /* We're going to work our way backwards. */
475: pos = bprm->p;
476: str += len;
477: bprm->p -= len;
478: if (bprm_hit_stack_limit(bprm))
479: goto out;
480:
481: while (len > 0) {
482: int offset, bytes_to_copy;
483:
484: if (fatal_signal_pending(current)) {
485: ret = -ERESTARTNOHAND;
486: goto out;
487: }
488: cond_resched();
489:
490: offset = pos % PAGE_SIZE;
491: if (offset == 0)
492: offset = PAGE_SIZE;
493:
494: bytes_to_copy = offset;
495: if (bytes_to_copy > len)
496: bytes_to_copy = len;
497:
498: offset -= bytes_to_copy;
499: pos -= bytes_to_copy;
500: str -= bytes_to_copy;
501: len -= bytes_to_copy;
502:
503: if (!kmapped_page || kpos != (pos & PAGE_MASK)) {
504: struct page *page;
505:
506: page = get_arg_page(bprm, pos, 1);
507: if (!page) {
508: ret = -E2BIG;
509: goto out;
510: }
511:
512: if (kmapped_page) {
513: flush_dcache_page(kmapped_page);
514: kunmap_local(kaddr);
515: put_arg_page(kmapped_page);
516: }
517: kmapped_page = page;
518: kaddr = kmap_local_page(kmapped_page);
519: kpos = pos & PAGE_MASK;
520: flush_arg_page(bprm, kpos, kmapped_page);
521: }
522: if (copy_from_user(kaddr+offset, str, bytes_to_copy)) {
523: ret = -EFAULT;
524: goto out;
525: }
526: }
527: }
528: ret = 0;
529: out:
530: if (kmapped_page) {
531: flush_dcache_page(kmapped_page);
532: kunmap_local(kaddr);
533: put_arg_page(kmapped_page);
534: }
535: return ret;
536: }
SourceFile:fs/exec.c]
|
| 229/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 21177
output: 25
thoughts: 197
Thoughts: **Navigating the Labyrinth of Variable Names**
Okay, still hitting a wall trying to pinpoint these elusive variable names. I'm going to revisit that `tmp_buf` concept. Instead of just searching for the variable itself, I'll use `codesearch-find-references` again, but this time I'll think about searching for it as a string, too. It's often the case.
Maybe I'll limit the search to the `drivers/staging/` directory. That seems like a logical place to start digging deeper. Hmm, on second thought, I was considering searching for the expression `tmp_buf_end - tmp_buf`, but I can't do that with the current tools. Time to rethink this approach.
Back to square one; let me revisit `sizeof_cur_arg`. Maybe it's a variation, like `sizeof_arg`?
Wait a second... hold on... I remember a similar pattern, wasn't there a known vulnerability, CVE-2023-31248 or something like that? No, that's not right. Let me pivot and see what I can find in `kernel/auditsc.c`. I have a feeling that file has some complex argument handling that might reveal the answer.
|
| 230/4 |
2026/05/18 19:06 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/auditsc.c]
Results: map[Entities:[map[Kind:function Name:__audit_bprm] map[Kind:function Name:__audit_fanotify] map[Kind:function Name:__audit_fd_pair] map[Kind:function Name:__audit_file] map[Kind:function Name:__audit_filter_op] map[Kind:function Name:__audit_free] map[Kind:function Name:__audit_getname] map[Kind:function Name:__audit_inode] map[Kind:function Name:__audit_inode_child] map[Kind:function Name:__audit_ipc_obj] map[Kind:function Name:__audit_ipc_set_perm] map[Kind:function Name:__audit_log_bprm_fcaps] map[Kind:function Name:__audit_log_capset] map[Kind:function Name:__audit_log_kern_module] map[Kind:function Name:__audit_log_nfcfg] map[Kind:function Name:__audit_mmap_fd] map[Kind:function Name:__audit_mq_getsetattr] map[Kind:function Name:__audit_mq_notify] map[Kind:function Name:__audit_mq_open] map[Kind:function Name:__audit_mq_sendrecv] map[Kind:function Name:__audit_ntp_log] map[Kind:function Name:__audit_openat2_how] map[Kind:function Name:__audit_ptrace] map[Kind:function Name:__audit_sockaddr] map[Kind:function Name:__audit_socketcall] map[Kind:function Name:__audit_syscall_entry] map[Kind:function Name:__audit_syscall_exit] map[Kind:function Name:__audit_tk_injoffset] map[Kind:function Name:__audit_uring_entry] map[Kind:function Name:__audit_uring_exit] map[Kind:function Name:audit_alloc] map[Kind:function Name:audit_alloc_context] map[Kind:function Name:audit_alloc_name] map[Kind:function Name:audit_compare_gid] map[Kind:function Name:audit_compare_uid] map[Kind:function Name:audit_copy_fcaps] map[Kind:function Name:audit_copy_inode] map[Kind:function Name:audit_core_dumps] map[Kind:function Name:audit_field_compare] map[Kind:function Name:audit_filter_inode_name] map[Kind:function Name:audit_filter_inodes] map[Kind:function Name:audit_filter_rules] map[Kind:function Name:audit_filter_syscall] map[Kind:function Name:audit_filter_task] map[Kind:function Name:audit_filter_uring] map[Kind:function Name:audit_free_aux] map[Kind:function Name:audit_free_context] map[Kind:function Name:audit_free_module] map[Kind:function Name:audit_free_names] map[Kind:function Name:audit_in_mask] map[Kind:function Name:audit_killed_trees] map[Kind:function Name:audit_log_cap] map[Kind:function Name:audit_log_execve_info] map[Kind:function Name:audit_log_exit] map[Kind:function Name:audit_log_fcaps] map[Kind:function Name:audit_log_name] map[Kind:function Name:audit_log_pid_context] map[Kind:function Name:audit_log_proctitle] map[Kind:function Name:audit_log_task] map[Kind:function Name:audit_log_time] map[Kind:function Name:audit_log_uring] map[Kind:function Name:audit_match_filetype] map[Kind:function Name:audit_match_perm] map[Kind:function Name:audit_proctitle_free] map[Kind:function Name:audit_proctitle_rtrim] map[Kind:function Name:audit_reset_context] map[Kind:function Name:audit_return_fixup] map[Kind:function Name:audit_seccomp] map[Kind:function Name:audit_seccomp_actions_logged] map[Kind:function Name:audit_set_auditable] map[Kind:function Name:audit_signal_info_syscall] map[Kind:function Name:auditsc_get_stamp] map[Kind:function Name:free_tree_refs] map[Kind:function Name:grow_tree_refs] map[Kind:function Name:handle_one] map[Kind:function Name:handle_path] map[Kind:function Name:match_tree_refs] map[Kind:function Name:put_tree_ref] map[Kind:function Name:show_special] map[Kind:function Name:unroll_tree_refs] map[Kind:struct Name:audit_aux_data] map[Kind:struct Name:audit_aux_data_bprm_fcaps] map[Kind:struct Name:audit_aux_data_pids] map[Kind:struct Name:audit_nfcfgop_tab] map[Kind:struct Name:audit_tree_refs] map[Kind:global_variable Name:__UNIQUE_ID_addressable___audit_inode_child_531] map[Kind:global_variable Name:__UNIQUE_ID_addressable___audit_log_nfcfg_537] map[Kind:global_variable Name:audit_nfcfgs]]]
|
| 231/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 23189
output: 43
thoughts: 23
Thoughts: Here's my thought process, summarized as you requested:
**Diving into `audit_log_execve_info`**
Okay, time to get to the core of this. My initial thought is to navigate to the `audit_log_execve_info` function within `kernel/auditsc.c`. That's where I would expect to find the meat of the execution event logging, specifically for `execve` calls. I need to understand exactly what information is being captured during those system calls, and how the audit subsystem handles this crucial data. From there, I can analyze the audit data structure and decide how to deal with this data.
|
| 232/4 |
2026/05/18 19:06 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/auditsc.c IncludeLines:true Name:audit_log_execve_info]
Results: map[SourceCode:1120: static void audit_log_execve_info(struct audit_context *context,
1121: struct audit_buffer **ab)
1122: {
1123: long len_max;
1124: long len_rem;
1125: long len_full;
1126: long len_buf;
1127: long len_abuf = 0;
1128: long len_tmp;
1129: bool require_data;
1130: bool encode;
1131: unsigned int iter;
1132: unsigned int arg;
1133: char *buf_head;
1134: char *buf;
1135: const char __user *p = (const char __user *)current->mm->arg_start;
1136:
1137: /* NOTE: this buffer needs to be large enough to hold all the non-arg
1138: * data we put in the audit record for this argument (see the
1139: * code below) ... at this point in time 96 is plenty */
1140: char abuf[96];
1141:
1142: /* NOTE: we set MAX_EXECVE_AUDIT_LEN to a rather arbitrary limit, the
1143: * current value of 7500 is not as important as the fact that it
1144: * is less than 8k, a setting of 7500 gives us plenty of wiggle
1145: * room if we go over a little bit in the logging below */
1146: WARN_ON_ONCE(MAX_EXECVE_AUDIT_LEN > 7500);
1147: len_max = MAX_EXECVE_AUDIT_LEN;
1148:
1149: /* scratch buffer to hold the userspace args */
1150: buf_head = kmalloc(MAX_EXECVE_AUDIT_LEN + 1, GFP_KERNEL);
1151: if (!buf_head) {
1152: audit_panic("out of memory for argv string");
1153: return;
1154: }
1155: buf = buf_head;
1156:
1157: audit_log_format(*ab, "argc=%d", context->execve.argc);
1158:
1159: len_rem = len_max;
1160: len_buf = 0;
1161: len_full = 0;
1162: require_data = true;
1163: encode = false;
1164: iter = 0;
1165: arg = 0;
1166: do {
1167: /* NOTE: we don't ever want to trust this value for anything
1168: * serious, but the audit record format insists we
1169: * provide an argument length for really long arguments,
1170: * e.g. > MAX_EXECVE_AUDIT_LEN, so we have no choice but
1171: * to use strncpy_from_user() to obtain this value for
1172: * recording in the log, although we don't use it
1173: * anywhere here to avoid a double-fetch problem */
1174: if (len_full == 0)
1175: len_full = strnlen_user(p, MAX_ARG_STRLEN) - 1;
1176:
1177: /* read more data from userspace */
1178: if (require_data) {
1179: /* can we make more room in the buffer? */
1180: if (buf != buf_head) {
1181: memmove(buf_head, buf, len_buf);
1182: buf = buf_head;
1183: }
1184:
1185: /* fetch as much as we can of the argument */
1186: len_tmp = strncpy_from_user(&buf_head[len_buf], p,
1187: len_max - len_buf);
1188: if (len_tmp == -EFAULT) {
1189: /* unable to copy from userspace */
1190: send_sig(SIGKILL, current, 0);
1191: goto out;
1192: } else if (len_tmp == (len_max - len_buf)) {
1193: /* buffer is not large enough */
1194: require_data = true;
1195: /* NOTE: if we are going to span multiple
1196: * buffers force the encoding so we stand
1197: * a chance at a sane len_full value and
1198: * consistent record encoding */
1199: encode = true;
1200: len_full = len_full * 2;
1201: p += len_tmp;
1202: } else {
1203: require_data = false;
1204: if (!encode)
1205: encode = audit_string_contains_control(
1206: buf, len_tmp);
1207: /* try to use a trusted value for len_full */
1208: if (len_full < len_max)
1209: len_full = (encode ?
1210: len_tmp * 2 : len_tmp);
1211: p += len_tmp + 1;
1212: }
1213: len_buf += len_tmp;
1214: buf_head[len_buf] = '\0';
1215:
1216: /* length of the buffer in the audit record? */
1217: len_abuf = (encode ? len_buf * 2 : len_buf + 2);
1218: }
1219:
1220: /* write as much as we can to the audit log */
1221: if (len_buf >= 0) {
1222: /* NOTE: some magic numbers here - basically if we
1223: * can't fit a reasonable amount of data into the
1224: * existing audit buffer, flush it and start with
1225: * a new buffer */
1226: if ((sizeof(abuf) + 8) > len_rem) {
1227: len_rem = len_max;
1228: audit_log_end(*ab);
1229: *ab = audit_log_start(context,
1230: GFP_KERNEL, AUDIT_EXECVE);
1231: if (!*ab)
1232: goto out;
1233: }
1234:
1235: /* create the non-arg portion of the arg record */
1236: len_tmp = 0;
1237: if (require_data || (iter > 0) ||
1238: ((len_abuf + sizeof(abuf)) > len_rem)) {
1239: if (iter == 0) {
1240: len_tmp += snprintf(&abuf[len_tmp],
1241: sizeof(abuf) - len_tmp,
1242: " a%d_len=%lu",
1243: arg, len_full);
1244: }
1245: len_tmp += snprintf(&abuf[len_tmp],
1246: sizeof(abuf) - len_tmp,
1247: " a%d[%d]=", arg, iter++);
1248: } else
1249: len_tmp += snprintf(&abuf[len_tmp],
1250: sizeof(abuf) - len_tmp,
1251: " a%d=", arg);
1252: WARN_ON(len_tmp >= sizeof(abuf));
1253: abuf[sizeof(abuf) - 1] = '\0';
1254:
1255: /* log the arg in the audit record */
1256: audit_log_format(*ab, "%s", abuf);
1257: len_rem -= len_tmp;
1258: len_tmp = len_buf;
1259: if (encode) {
1260: if (len_abuf > len_rem)
1261: len_tmp = len_rem / 2; /* encoding */
1262: audit_log_n_hex(*ab, buf, len_tmp);
1263: len_rem -= len_tmp * 2;
1264: len_abuf -= len_tmp * 2;
1265: } else {
1266: if (len_abuf > len_rem)
1267: len_tmp = len_rem - 2; /* quotes */
1268: audit_log_n_string(*ab, buf, len_tmp);
1269: len_rem -= len_tmp + 2;
1270: /* don't subtract the "2" because we still need
1271: * to add quotes to the remaining string */
1272: len_abuf -= len_tmp;
1273: }
1274: len_buf -= len_tmp;
1275: buf += len_tmp;
1276: }
1277:
1278: /* ready to move to the next argument? */
1279: if ((len_buf == 0) && !require_data) {
1280: arg++;
1281: iter = 0;
1282: len_full = 0;
1283: require_data = true;
1284: encode = false;
1285: }
1286: } while (arg < context->execve.argc);
1287:
1288: /* NOTE: the caller handles the final audit_log_end() call */
1289:
1290: out:
1291: kfree(buf_head);
1292: }
SourceFile:kernel/auditsc.c]
|
| 233/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 26153
output: 23
thoughts: 160
Thoughts: **Analyzing the Code Snippet**
Okay, so this code snippet... it's not quite right. Back to the drawing board, let's re-examine the original question. I see a comparison: `tmp_buf_end - tmp_buf < sizeof_cur_arg`. Alright, `tmp_buf_end` and `tmp_buf` are pointers, and `sizeof_cur_arg` is a `size_t`. I'm going to base my answer on C's language rules, if all else fails. Although, I have this nagging suspicion this is related to a specific patch or a known bug. Hold on, let me search for `sizeof_cur_arg` throughout the entire code tree again. Perhaps I should use `codesearch-dir-index` on the include files. Actually, scrap that, I'm going to search for `tmp_buf_end` within the `fs/` directory first. I'll get to the bottom of this.
|
| 234/4 |
2026/05/18 19:06 |
tool |
codesearch-find-references |
0m
Error:
Args:
Results: map[References:<nil> TruncatedOutput:false]
|
| 235/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 26566
output: 22
thoughts: 129
Thoughts: **Analyzing C Code in a Linux Kernel Context**
Alright, let's break this down. It seems we're dealing with a general C question, likely focused on the Linux kernel's coding environment. My initial thought is to make sure we understand the types involved and potential pitfalls in their interactions.
Specifically, I'm thinking about `size_t` and `ptrdiff_t`. I know, in this environment, `size_t` is *unsigned*, and `ptrdiff_t` is *signed*. So, we're talking about differences between memory addresses – the classic pointer arithmetic scenario. That subtraction, `tmp_buf_end - tmp_buf`, is going to result in a `ptrdiff_t` value. Makes perfect sense.
Now comes the tricky part: the comparison. Since we're comparing `ptrdiff_t` (signed) to `size_t` (unsigned), we're going to trigger the standard arithmetic conversions. That's where things can get interesting, and potentially problematic, due to how the compiler handles signed/unsigned comparisons.
My next step would be to verify these type definitions within the kernel's header files. I'd typically look in a familiar place like `include/linux/types.h` or something similar. Confirming these definitions is critical. Once I have the definitions right in front of me, I can be certain in my assessments and I can start writing out the code that I need.
|
| 236/4 |
2026/05/18 19:06 |
tool |
read-file |
0m
Args: map[File:include/linux/types.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_TYPES_H
#define _LINUX_TYPES_H
#include <uapi/linux/types.h>
#ifndef __ASSEMBLY__
#define DECLARE_BITMAP(name,bits) \
unsigned long name[BITS_TO_LONGS(bits)]
#ifdef __SIZEOF_INT128__
typedef __s128 s128;
typedef __u128 u128;
#endif
typedef u32 __kernel_dev_t;
typedef __kernel_fd_set fd_set;
typedef __kernel_dev_t dev_t;
typedef __kernel_ulong_t ino_t;
typedef __kernel_mode_t mode_t;
typedef unsigned short umode_t;
typedef u32 nlink_t;
typedef __kernel_off_t off_t;
typedef __kernel_pid_t pid_t;
typedef __kernel_daddr_t daddr_t;
typedef __kernel_key_t key_t;
typedef __kernel_suseconds_t suseconds_t;
typedef __kernel_timer_t timer_t;
typedef __kernel_clockid_t clockid_t;
typedef __kernel_mqd_t mqd_t;
typedef _Bool bool;
typedef __kernel_uid32_t uid_t;
typedef __kernel_gid32_t gid_t;
typedef __kernel_uid16_t uid16_t;
typedef __kernel_gid16_t gid16_t;
typedef unsigned long uintptr_t;
typedef long intptr_t;
#ifdef CONFIG_HAVE_UID16
/* This is defined by arch/{arch}/include/asm/posix_types.h */
typedef __kernel_old_uid_t old_uid_t;
typedef __kernel_old_gid_t old_gid_t;
#endif /* CONFIG_UID16 */
#if defined(__GNUC__)
typedef __kernel_loff_t loff_t;
typedef __kernel_uoff_t uoff_t;
#endif
/*
* The following typedefs are also protected by individual ifdefs for
* historical reasons:
*/
#ifndef _SIZE_T
#define _SIZE_T
typedef __kernel_size_t size_t;
#endif
#ifndef _SSIZE_T
#define _SSIZE_T
typedef __kernel_ssize_t ssize_t;
#endif
#ifndef _PTRDIFF_T
#define _PTRDIFF_T
typedef __kernel_ptrdiff_t ptrdiff_t;
#endif
#ifndef _CLOCK_T
#define _CLOCK_T
typedef __kernel_clock_t clock_t;
#endif
#ifndef _CADDR_T
#define _CADDR_T
typedef __kernel_caddr_t caddr_t;
#endif
/* bsd */
typedef unsigned char u_char;
typedef unsigned short u_short;
typedef unsigned int u_int;
typedef unsigned long u_long;
/* sysv */
typedef unsigned char unchar;
typedef unsigned short ushort;
typedef unsigned int uint;
typedef unsigned long ulong;
typedef unsigned long long ullong;
#ifndef __BIT_TYPES_DEFINED__
#define __BIT_TYPES_DEFINED__
typedef u8 u_int8_t;
typedef s8 int8_t;
typedef u16 u_int16_t;
typedef s16 int16_t;
typedef u32 u_int32_t;
typedef s32 int32_t;
#endif /* !(__BIT_TYPES_DEFINED__) */
typedef u8 uint8_t;
typedef u16 uint16_t;
typedef u32 uint32_t;
#if defined(__GNUC__)
typedef u64 uint64_t;
typedef u64 u_int64_t;
typedef s64 int64_t;
#endif
/* These are the special 64-bit data types that are 8-byte aligned */
#define aligned_u64 __aligned_u64
#define aligned_s64 __aligned_s64
#define aligned_be64 __aligned_be64
#define aligned_le64 __aligned_le64
/* Nanosecond scalar representation for kernel time values */
typedef s64 ktime_t;
/**
* The type used for indexing onto a disc or disc partition.
*
* Linux always considers sectors to be 512 bytes long independently
* of the devices real block size.
*
* blkcnt_t is the type of the inode's block count.
*/
typedef u64 sector_t;
typedef u64 blkcnt_t;
/* generic data direction definitions */
#define READ 0
#define WRITE 1
/*
* The type of an index into the pagecache.
*/
#define pgoff_t unsigned long
/*
* A dma_addr_t can hold any valid DMA address, i.e., any address returned
* by the DMA API.
*
* If the DMA API only uses 32-bit addresses, dma_addr_t need only be 32
* bits wide. Bus addresses, e.g., PCI BARs, may be wider than 32 bits,
* but drivers do memory-mapped I/O to ioremapped kernel virtual addresses,
* so they don't care about the size of the actual bus addresses.
*/
#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
typedef u64 dma_addr_t;
#else
typedef u32 dma_addr_t;
#endif
typedef unsigned int __bitwise gfp_t;
typedef unsigned int __bitwise slab_flags_t;
typedef unsigned int __bitwise fmode_t;
#ifdef CONFIG_PHYS_ADDR_T_64BIT
typedef u64 phys_addr_t;
#else
typedef u32 phys_addr_t;
#endif
struct phys_vec {
phys_addr_t paddr;
size_t len;
};
typedef phys_addr_t resource_size_t;
/*
* This type is the placeholder for a hardware interrupt number. It has to be
* big enough to enclose whatever representation is used by a given platform.
*/
typedef unsigned long irq_hw_number_t;
typedef struct {
int __aligned(sizeof(int)) counter;
} atomic_t;
#define ATOMIC_INIT(i) { (i) }
#ifdef CONFIG_64BIT
typedef struct {
s64 counter;
} atomic64_t;
#endif
typedef struct {
atomic_t refcnt;
} rcuref_t;
#define RCUREF_INIT(i) { .refcnt = ATOMIC_INIT(i - 1) }
struct list_head {
struct list_head *next, *prev;
};
struct hlist_head {
struct hlist_node *first;
};
struct hlist_node {
struct hlist_node *next, **pprev;
};
struct ustat {
__kernel_daddr_t f_tfree;
#ifdef CONFIG_ARCH_32BIT_USTAT_F_TINODE
unsigned int f_tinode;
#else
unsigned long f_tinode;
#endif
char f_fname[6];
char f_fpack[6];
};
/**
* struct callback_head - callback structure for use with RCU and task_work
* @next: next update requests in a list
* @func: actual update function to call after the grace period.
*
* The struct is aligned to size of pointer. On most architectures it happens
* naturally due ABI requirements, but some architectures (like CRIS) have
* weird ABI and we need to ask it explicitly.
*
* The alignment is required to guarantee that bit 0 of @next will be
* clear under normal conditions -- as long as we use call_rcu() or
* call_srcu() to queue the callback.
*
* This guarantee is important for few reasons:
* - future call_rcu_lazy() will make use of lower bits in the pointer;
* - the structure shares storage space in struct page with @compound_info,
* which encode PageTail() in bit 0. The guarantee is needed to avoid
* false-positive PageTail().
*/
struct callback_head {
struct callback_head *next;
void (*func)(struct callback_head *head);
} __attribute__((aligned(sizeof(void *))));
#define rcu_head callback_head
typedef void (*rcu_callback_t)(struct rcu_head *head);
typedef void (*call_rcu_func_t)(struct rcu_head *head, rcu_callback_t func);
typedef void (*swap_r_func_t)(void *a, void *b, int size, const void *priv);
typedef void (*swap_func_t)(void *a, void *b, int size);
typedef int (*cmp_r_func_t)(const void *a, const void *b, const void *priv);
typedef int (*cmp_func_t)(const void *a, const void *b);
/*
* rcuwait provides a way of blocking and waking up a single
* task in an rcu-safe manner.
*
* The only time @task is non-nil is when a user is blocked (or
* checking if it needs to) on a condition, and reset as soon as we
* know that the condition has succeeded and are awoken.
*/
struct rcuwait {
struct task_struct __rcu *task;
};
#endif /* __ASSEMBLY__ */
#endif /* _LINUX_TYPES_H */
]
|
| 237/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 29409
output: 37
thoughts: 86
Thoughts: Here's my thought process, as it stands:
**Navigating Kernel Data Type Definitions**
Okay, so I'm looking at how `size_t` and `ptrdiff_t` are defined within the kernel. I remember seeing these as `__kernel_size_t` and `__kernel_ptrdiff_t`, respectively. Right, and those are going to be defined somewhere in the `uapi/linux/types.h` header, or perhaps an architecture-specific header. Let me confirm that by taking a look at a common example, say, `arch/x86/include/uapi/asm/posix_types_64.h`. I'll want to check this file (or a similar one, for another architecture if I need to) to see precisely how these essential types are mapped to their underlying representations, especially for 64-bit x86 systems. It's important to understand the kernel's definitions to ensure portability and avoid any subtle type-related issues.
|
| 238/4 |
2026/05/18 19:06 |
tool |
read-file |
0m
Args: map[File:arch/x86/include/uapi/asm/posix_types_64.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
#ifndef _ASM_X86_POSIX_TYPES_64_H
#define _ASM_X86_POSIX_TYPES_64_H
/*
* This file is generally used by user-level software, so you need to
* be a little careful about namespace pollution etc. Also, we cannot
* assume GCC is being used.
*/
typedef unsigned short __kernel_old_uid_t;
typedef unsigned short __kernel_old_gid_t;
#define __kernel_old_uid_t __kernel_old_uid_t
typedef unsigned long __kernel_old_dev_t;
#define __kernel_old_dev_t __kernel_old_dev_t
#include <asm-generic/posix_types.h>
#endif /* _ASM_X86_POSIX_TYPES_64_H */
]
|
| 239/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 29978
output: 30
thoughts: 14
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience and presented in the first person:
**Navigating the Headers**
Alright, let's see. My mind immediately recognizes the inclusion of `asm-generic/posix_types.h`. That's interesting. It signals a dive into the fundamental type definitions, the bedrock upon which our system calls and data structures are built. This is where the core definitions for things like `__kernel_long_t`, `__kernel_ulong_t`, and other primitive types reside. It suggests that we're dealing with code that either needs to interact directly with the kernel's data representations or perhaps is defining interfaces that have to be carefully aligned with kernel expectations. The presence of this header automatically flags the possibility of architecture-specific adaptations. I'll need to keep an eye out for potential platform-dependent nuances in how these types are used and interpreted downstream. Also it is important to remember that since this is part of the `asm-generic` it means it is platform agnostic and will most likely need to use more specific types or calls further down. It is likely that this code might interact with a kernel module or some system level call.
|
| 240/4 |
2026/05/18 19:06 |
tool |
read-file |
0m
Args: map[File:include/uapi/asm-generic/posix_types.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
#ifndef __ASM_GENERIC_POSIX_TYPES_H
#define __ASM_GENERIC_POSIX_TYPES_H
#include <asm/bitsperlong.h>
/*
* This file is generally used by user-level software, so you need to
* be a little careful about namespace pollution etc.
*
* First the types that are often defined in different ways across
* architectures, so that you can override them.
*/
#ifndef __kernel_long_t
typedef long __kernel_long_t;
typedef unsigned long __kernel_ulong_t;
#endif
#ifndef __kernel_ino_t
typedef __kernel_ulong_t __kernel_ino_t;
#endif
#ifndef __kernel_mode_t
typedef unsigned int __kernel_mode_t;
#endif
#ifndef __kernel_pid_t
typedef int __kernel_pid_t;
#endif
#ifndef __kernel_ipc_pid_t
typedef int __kernel_ipc_pid_t;
#endif
#ifndef __kernel_uid_t
typedef unsigned int __kernel_uid_t;
typedef unsigned int __kernel_gid_t;
#endif
#ifndef __kernel_suseconds_t
typedef __kernel_long_t __kernel_suseconds_t;
#endif
#ifndef __kernel_daddr_t
typedef int __kernel_daddr_t;
#endif
#ifndef __kernel_uid32_t
typedef unsigned int __kernel_uid32_t;
typedef unsigned int __kernel_gid32_t;
#endif
#ifndef __kernel_old_uid_t
typedef __kernel_uid_t __kernel_old_uid_t;
typedef __kernel_gid_t __kernel_old_gid_t;
#endif
#ifndef __kernel_old_dev_t
typedef unsigned int __kernel_old_dev_t;
#endif
/*
* Most 32 bit architectures use "unsigned int" size_t,
* and all 64 bit architectures use "unsigned long" size_t.
*/
#ifndef __kernel_size_t
#if __BITS_PER_LONG != 64
typedef unsigned int __kernel_size_t;
typedef int __kernel_ssize_t;
typedef int __kernel_ptrdiff_t;
#else
typedef __kernel_ulong_t __kernel_size_t;
typedef __kernel_long_t __kernel_ssize_t;
typedef __kernel_long_t __kernel_ptrdiff_t;
#endif
#endif
#ifndef __kernel_fsid_t
typedef struct {
int val[2];
} __kernel_fsid_t;
#endif
/*
* anything below here should be completely generic
*/
typedef __kernel_long_t __kernel_off_t;
typedef long long __kernel_loff_t;
typedef unsigned long long __kernel_uoff_t;
typedef __kernel_long_t __kernel_old_time_t;
#ifndef __KERNEL__
typedef __kernel_long_t __kernel_time_t;
#endif
typedef long long __kernel_time64_t;
typedef __kernel_long_t __kernel_clock_t;
typedef int __kernel_timer_t;
typedef int __kernel_clockid_t;
typedef char * __kernel_caddr_t;
typedef unsigned short __kernel_uid16_t;
typedef unsigned short __kernel_gid16_t;
#endif /* __ASM_GENERIC_POSIX_TYPES_H */
]
|
| 241/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 31154
output: 33
thoughts: 534
Thoughts: **Analyzing Potential Integer Overflow in Kernel Code**
Okay, let's break this down. My initial thought is about how `__kernel_size_t` and `__kernel_ptrdiff_t` behave across different architectures, specifically 32-bit versus 64-bit. On a 64-bit system, they're `unsigned long` and `long`, respectively, and on 32-bit, they're `unsigned int` and `int`. Crucially, in both scenarios, the *rank* (conversion hierarchy) of these types is the same.
The real crux of the matter comes when comparing `ptrdiff_t` (which is what `__kernel_ptrdiff_t` resolves to) with `size_t`. Since `ptrdiff_t` is signed and `size_t` is unsigned, the `ptrdiff_t` will always be converted to unsigned for the comparison. Now, I need to consider a potential problem: could `tmp_buf_end - tmp_buf` ever be negative? If it is, that subtraction results in a huge unsigned value. If `sizeof_cur_arg` is relatively small, this means the comparison will be false. Had it been signed, `negative_value < small_value` *would* have been true. That behavior difference is important. This is a classic integer overflow situation.
The user's query is pretty specific, mentioning `tmp_buf_end`, `tmp_buf`, and `sizeof_cur_arg`, which is very helpful. I need to find where this potential issue exists in the kernel. I tried a previous tool but I did not find anything, so I'm going to try to search again. I can't search for substrings directly, so I need to refine my search.
I'm thinking where I might have seen similar code, and it's making me think of `fs/exec.c` and other files in the `fs/` directory, so I'm going to search for `tmp_buf` in those, looking particularly at any code related to execution or arguments (`*exec*`, `*arg*`).
Wait, I think I found it! It looks like it might be in `fs/exec.c`, but maybe a slightly older version, or in a specific branch of the kernel.
Hmm, I've checked `fs/exec.c`, I don't see it, it's not here.
Let me try another possibility - I've worked on USB gadget drivers, and I recall similar buffer management in that area. I'll search for `tmp_buf` in `drivers/usb/gadget/function/f_fs.c`. Let's see if this yields any results!
|
| 242/4 |
2026/05/18 19:06 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:drivers/usb/gadget/function/f_fs.c]
Results: map[Entities:[map[Kind:function Name:__ffs_data_do_entity] map[Kind:function Name:__ffs_data_do_os_desc] map[Kind:function Name:__ffs_data_got_descs] map[Kind:function Name:__ffs_data_got_strings] map[Kind:function Name:__ffs_do_os_desc_header] map[Kind:function Name:__ffs_ep0_queue_wait] map[Kind:function Name:__ffs_ep0_read_events] map[Kind:function Name:__ffs_ep0_stall] map[Kind:function Name:__ffs_epfile_read_buffer_free] map[Kind:function Name:__ffs_epfile_read_buffered] map[Kind:function Name:__ffs_epfile_read_data] map[Kind:function Name:__ffs_event_add] map[Kind:function Name:__ffs_func_bind_do_descs] map[Kind:function Name:__ffs_func_bind_do_nums] map[Kind:function Name:__ffs_func_bind_do_os_desc] map[Kind:function Name:_ffs_alloc_dev] map[Kind:function Name:_ffs_do_find_dev] map[Kind:function Name:_ffs_find_dev] map[Kind:function Name:_ffs_free_dev] map[Kind:function Name:_ffs_func_bind] map[Kind:function Name:_ffs_get_single_dev] map[Kind:function Name:clear_one] map[Kind:function Name:f_fs_opts_ready_show] map[Kind:function Name:ffs_acquire_dev] map[Kind:function Name:ffs_aio_cancel] map[Kind:function Name:ffs_alloc] map[Kind:function Name:ffs_alloc_buffer] map[Kind:function Name:ffs_alloc_inst] map[Kind:function Name:ffs_attr_release] map[Kind:function Name:ffs_build_sg_list] map[Kind:function Name:ffs_closed] map[Kind:function Name:ffs_copy_to_iter] map[Kind:function Name:ffs_data_clear] map[Kind:function Name:ffs_data_closed] map[Kind:function Name:ffs_data_get] map[Kind:function Name:ffs_data_new] map[Kind:function Name:ffs_data_put] map[Kind:function Name:ffs_data_reset] map[Kind:function Name:ffs_dma_resv_lock] map[Kind:function Name:ffs_dmabuf_attach] map[Kind:function Name:ffs_dmabuf_cleanup] map[Kind:function Name:ffs_dmabuf_detach] map[Kind:function Name:ffs_dmabuf_fence_release] map[Kind:function Name:ffs_dmabuf_find_attachment] map[Kind:function Name:ffs_dmabuf_get] map[Kind:function Name:ffs_dmabuf_get_driver_name] map[Kind:function Name:ffs_dmabuf_get_timeline_name] map[Kind:function Name:ffs_dmabuf_put] map[Kind:function Name:ffs_dmabuf_release] map[Kind:function Name:ffs_dmabuf_signal_done] map[Kind:function Name:ffs_dmabuf_transfer] map[Kind:function Name:ffs_do_descs] map[Kind:function Name:ffs_do_functionfs_bind] map[Kind:function Name:ffs_do_os_descs] map[Kind:function Name:ffs_do_single_desc] map[Kind:function Name:ffs_do_single_os_desc] map[Kind:function Name:ffs_ep0_complete] map[Kind:function Name:ffs_ep0_ioctl] map[Kind:function Name:ffs_ep0_open] map[Kind:function Name:ffs_ep0_poll] map[Kind:function Name:ffs_ep0_read] map[Kind:function Name:ffs_ep0_release] map[Kind:function Name:ffs_ep0_write] map[Kind:function Name:ffs_ep_addr2idx] map[Kind:function Name:ffs_epfile_async_io_complete] map[Kind:function Name:ffs_epfile_dmabuf_io_complete] map[Kind:function Name:ffs_epfile_io] map[Kind:function Name:ffs_epfile_io_complete] map[Kind:function Name:ffs_epfile_ioctl] map[Kind:function Name:ffs_epfile_open] map[Kind:function Name:ffs_epfile_read_iter] map[Kind:function Name:ffs_epfile_release] map[Kind:function Name:ffs_epfile_wait_ep] map[Kind:function Name:ffs_epfile_write_iter] map[Kind:function Name:ffs_epfiles_create] map[Kind:function Name:ffs_epfiles_destroy] map[Kind:function Name:ffs_event_add] map[Kind:function Name:ffs_free] map[Kind:function Name:ffs_free_buffer] map[Kind:function Name:ffs_free_inst] map[Kind:function Name:ffs_fs_free_fc] map[Kind:function Name:ffs_fs_get_tree] map[Kind:function Name:ffs_fs_init_fs_context] map[Kind:function Name:ffs_fs_kill_sb] map[Kind:function Name:ffs_fs_parse_param] map[Kind:function Name:ffs_func_bind] map[Kind:function Name:ffs_func_disable] map[Kind:function Name:ffs_func_eps_disable] map[Kind:function Name:ffs_func_eps_enable] map[Kind:function Name:ffs_func_from_usb] map[Kind:function Name:ffs_func_get_alt] map[Kind:function Name:ffs_func_req_match] map[Kind:function Name:ffs_func_resume] map[Kind:function Name:ffs_func_revmap_ep] map[Kind:function Name:ffs_func_revmap_intf] map[Kind:function Name:ffs_func_set_alt] map[Kind:function Name:ffs_func_setup] map[Kind:function Name:ffs_func_suspend] map[Kind:function Name:ffs_func_unbind] map[Kind:function Name:ffs_mutex_lock] map[Kind:function Name:ffs_name_dev] map[Kind:function Name:ffs_prepare_buffer] map[Kind:function Name:ffs_ready] map[Kind:function Name:ffs_release_dev] map[Kind:function Name:ffs_reset_work] map[Kind:function Name:ffs_sb_create_file] map[Kind:function Name:ffs_sb_fill] map[Kind:function Name:ffs_sb_make_inode] map[Kind:function Name:ffs_set_inst_name] map[Kind:function Name:ffs_setup_state_clear_cancelled] map[Kind:function Name:ffs_single_dev] map[Kind:function Name:ffs_user_copy_worker] map[Kind:function Name:ffsmod_exit] map[Kind:function Name:ffsmod_init] map[Kind:function Name:functionfs_bind] map[Kind:function Name:functionfs_cleanup] map[Kind:function Name:functionfs_init] map[Kind:function Name:functionfs_unbind] map[Kind:function Name:to_ffs_opts] map[Kind:struct Name:ffs_buffer] map[Kind:struct Name:ffs_desc_helper] map[Kind:struct Name:ffs_dma_fence] map[Kind:struct Name:ffs_dmabuf_priv] map[Kind:struct Name:ffs_ep] map[Kind:struct Name:ffs_epfile] map[Kind:struct Name:ffs_function] map[Kind:struct Name:ffs_io_data] map[Kind:struct Name:ffs_sb_fill_data] map[Kind:global_variable Name:__UNIQUE_ID_addressable_ffs_lock_529] map[Kind:global_variable Name:__UNIQUE_ID_addressable_ffs_name_dev_566] map[Kind:global_variable Name:__UNIQUE_ID_addressable_ffs_single_dev_567] map[Kind:global_variable Name:__UNIQUE_ID_addressable_ffsmod_init_570] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_528] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_542] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_568] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_571] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_572] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_573] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_574] map[Kind:global_variable Name:__exitcall_ffsmod_exit] map[Kind:global_variable Name:f_fs_opts_attr_ready] map[Kind:global_variable Name:ffs_attrs] map[Kind:global_variable Name:ffs_devices] map[Kind:global_variable Name:ffs_dmabuf_fence_ops] map[Kind:global_variable Name:ffs_ep0_operations] map[Kind:global_variable Name:ffs_epfile_operations] map[Kind:global_variable Name:ffs_fs_context_ops] map[Kind:global_variable Name:ffs_fs_fs_parameters] map[Kind:global_variable Name:ffs_fs_type] map[Kind:global_variable Name:ffs_func_type] map[Kind:global_variable Name:ffs_item_ops] map[Kind:global_variable Name:ffs_lock] map[Kind:global_variable Name:ffs_sb_operations] map[Kind:global_variable Name:ffsusb_func] map[Kind:enum Name:ffs_entity_type] map[Kind:enum Name:ffs_os_desc_type] map[Kind:typedef Name:ffs_entity_callback] map[Kind:typedef Name:ffs_os_desc_callback]]]
|
| 243/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 35355
output: 52
thoughts: 32
Thoughts: Okay, here's my take on those thoughts, framed as an expert's internal monologue:
**Investigating `f_fs.c` String Handling**
Alright, time to dive into this. The task at hand is to understand how strings are being handled within the `f_fs.c` file, specifically within the USB gadget function driver framework. I need to refresh my memory on the internal data structures used to store these strings. Let's start by scrutinizing the `__ffs_data_got_strings` variable within that file. I'm assuming it's a critical component, likely the key data structure or variable holding pointers to the string descriptors used by the function filesystem. I expect there will be a section that handles their allocation, initialization, and deallocation. I need to clarify if these strings are static or dynamically allocated and, if dynamic, the mechanisms of their memory management. I'm looking for potential vulnerabilities, like buffer overflows, or unexpected side effects caused by improper string handling. My goal is to grasp the overall string handling strategy employed within this specific function driver and make sure it's robust.
|
| 244/4 |
2026/05/18 19:06 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:drivers/usb/gadget/function/f_fs.c IncludeLines:true Name:__ffs_data_got_strings]
Results: map[SourceCode:3035: static int __ffs_data_got_strings(struct ffs_data *ffs,
3036: char *const _data, size_t len)
3037: {
3038: u32 str_count, needed_count, lang_count;
3039: struct usb_gadget_strings **stringtabs, *t;
3040: const char *data = _data;
3041: struct usb_string *s;
3042:
3043: if (len < 16 ||
3044: get_unaligned_le32(data) != FUNCTIONFS_STRINGS_MAGIC ||
3045: get_unaligned_le32(data + 4) != len)
3046: goto error;
3047: str_count = get_unaligned_le32(data + 8);
3048: lang_count = get_unaligned_le32(data + 12);
3049:
3050: /* if one is zero the other must be zero */
3051: if (!str_count != !lang_count)
3052: goto error;
3053:
3054: /* Do we have at least as many strings as descriptors need? */
3055: needed_count = ffs->strings_count;
3056: if (str_count < needed_count)
3057: goto error;
3058:
3059: /*
3060: * If we don't need any strings just return and free all
3061: * memory.
3062: */
3063: if (!needed_count) {
3064: kfree(_data);
3065: return 0;
3066: }
3067:
3068: /* Allocate everything in one chunk so there's less maintenance. */
3069: {
3070: unsigned i = 0;
3071: vla_group(d);
3072: vla_item(d, struct usb_gadget_strings *, stringtabs,
3073: size_add(lang_count, 1));
3074: vla_item(d, struct usb_gadget_strings, stringtab, lang_count);
3075: vla_item(d, struct usb_string, strings,
3076: size_mul(lang_count, (needed_count + 1)));
3077:
3078: char *vlabuf = kmalloc(vla_group_size(d), GFP_KERNEL);
3079:
3080: if (!vlabuf) {
3081: kfree(_data);
3082: return -ENOMEM;
3083: }
3084:
3085: /* Initialize the VLA pointers */
3086: stringtabs = vla_ptr(vlabuf, d, stringtabs);
3087: t = vla_ptr(vlabuf, d, stringtab);
3088: i = lang_count;
3089: do {
3090: *stringtabs++ = t++;
3091: } while (--i);
3092: *stringtabs = NULL;
3093:
3094: /* stringtabs = vlabuf = d_stringtabs for later kfree */
3095: stringtabs = vla_ptr(vlabuf, d, stringtabs);
3096: t = vla_ptr(vlabuf, d, stringtab);
3097: s = vla_ptr(vlabuf, d, strings);
3098: }
3099:
3100: /* For each language */
3101: data += 16;
3102: len -= 16;
3103:
3104: do { /* lang_count > 0 so we can use do-while */
3105: unsigned needed = needed_count;
3106: u32 str_per_lang = str_count;
3107:
3108: if (len < 3)
3109: goto error_free;
3110: t->language = get_unaligned_le16(data);
3111: t->strings = s;
3112: ++t;
3113:
3114: data += 2;
3115: len -= 2;
3116:
3117: /* For each string */
3118: do { /* str_count > 0 so we can use do-while */
3119: size_t length = strnlen(data, len);
3120:
3121: if (length == len)
3122: goto error_free;
3123:
3124: /*
3125: * User may provide more strings then we need,
3126: * if that's the case we simply ignore the
3127: * rest
3128: */
3129: if (needed) {
3130: /*
3131: * s->id will be set while adding
3132: * function to configuration so for
3133: * now just leave garbage here.
3134: */
3135: s->s = data;
3136: --needed;
3137: ++s;
3138: }
3139:
3140: data += length + 1;
3141: len -= length + 1;
3142: } while (--str_per_lang);
3143:
3144: s->id = 0; /* terminator */
3145: s->s = NULL;
3146: ++s;
3147:
3148: } while (--lang_count);
3149:
3150: /* Some garbage left? */
3151: if (len)
3152: goto error_free;
3153:
3154: /* Done! */
3155: ffs->stringtabs = stringtabs;
3156: ffs->raw_strings = _data;
3157:
3158: return 0;
3159:
3160: error_free:
3161: kfree(stringtabs);
3162: error:
3163: kfree(_data);
3164: return -EINVAL;
3165: }
SourceFile:drivers/usb/gadget/function/f_fs.c]
|
| 245/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 37504
output: 26
thoughts: 206
Thoughts: **Kernel Debugging Deep Dive**
Okay, still no luck. I'm going to revisit `tmp_buf` in `kernel/auditsc.c` but this time I'll dig into the history – see if I can catch something I missed. Or, better yet, I'll try a broader search using `codesearch-find-references`. I'll look for `tmp_buf_end - tmp_buf`, but with a different variable name in the search query.
Hold on, `codesearch-find-references` might be clever enough to return nothing if `tmp_buf` is a local variable. That's actually useful – if it *is* local, I'm going to have a hard time finding it without knowing the file. But a negative result would be telling.
Right, back to the size issue. Let me try that global search for `sizeof_cur_arg` again. Wait... did I misread that variable name? Could it be `sizeof_arg` instead? I'm missing something, clearly.
Alright, I'll switch gears and try another file: let me search for `tmp_buf` in `fs/proc/array.c`.
|
| 246/4 |
2026/05/18 19:06 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:fs/proc/array.c]
Results: map[Entities:[map[Kind:function Name:arch_proc_pid_thread_features] map[Kind:function Name:children_seq_next] map[Kind:function Name:children_seq_open] map[Kind:function Name:children_seq_show] map[Kind:function Name:children_seq_start] map[Kind:function Name:children_seq_stop] map[Kind:function Name:collect_sigign_sigcatch] map[Kind:function Name:do_task_stat] map[Kind:function Name:get_children_pid] map[Kind:function Name:get_task_state] map[Kind:function Name:proc_pid_statm] map[Kind:function Name:proc_pid_status] map[Kind:function Name:proc_task_name] map[Kind:function Name:proc_tgid_stat] map[Kind:function Name:proc_tid_stat] map[Kind:function Name:render_cap_t] map[Kind:function Name:render_sigset_t] map[Kind:function Name:task_cap] map[Kind:function Name:task_context_switch_counts] map[Kind:function Name:task_core_dumping] map[Kind:function Name:task_cpus_allowed] map[Kind:function Name:task_seccomp] map[Kind:function Name:task_sig] map[Kind:function Name:task_state] map[Kind:function Name:task_thp_status] map[Kind:function Name:task_untag_mask] map[Kind:global_variable Name:children_seq_ops] map[Kind:global_variable Name:proc_tid_children_operations] map[Kind:global_variable Name:task_state_array]]]
|
| 247/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 38486
output: 25
thoughts: 25
Thoughts: Okay, here's my thought process laid out, assuming I'm an expert debugging a system:
**Lost in the Code, Time to Search**
Okay, I'm drawing a blank here. I'm not immediately seeing the root of the problem. Time for a targeted approach. Let's see... what was that variable again? Ah, yes, `tmp_buf`. I vaguely recall something using it. Right, I need to understand where it's being used. Now, based on the context, and my familiarity with the usual suspects, `lib/vsprintf.c` is probably the right place to start looking. I'll search for all instances of `tmp_buf` within that file and see what comes up. That should give me the contextual clues I need.
|
| 248/4 |
2026/05/18 19:06 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:lib/vsprintf.c]
Results: map[Entities:[map[Kind:function Name:__ptr_to_hashval] map[Kind:function Name:address_val] map[Kind:function Name:bdev_name] map[Kind:function Name:bitmap_list_string] map[Kind:function Name:bitmap_string] map[Kind:function Name:bstr_printf] map[Kind:function Name:check_pointer] map[Kind:function Name:check_pointer_msg] map[Kind:function Name:clock] map[Kind:function Name:convert_num_spec] map[Kind:function Name:date_str] map[Kind:function Name:debug_boot_weak_hash_enable] map[Kind:function Name:default_pointer] map[Kind:function Name:dentry_name] map[Kind:function Name:device_node_string] map[Kind:function Name:err_ptr] map[Kind:function Name:error_string] map[Kind:function Name:escaped_string] map[Kind:function Name:file_dentry_name] map[Kind:function Name:fill_ptr_key] map[Kind:function Name:flags_string] map[Kind:function Name:format_decode] map[Kind:function Name:format_flags] map[Kind:function Name:format_page_flags] map[Kind:function Name:fourcc_string] map[Kind:function Name:fwnode_full_name_string] map[Kind:function Name:fwnode_string] map[Kind:function Name:hash_pointers_finalize] map[Kind:function Name:hash_pointers_mode_parse] map[Kind:function Name:hex_range] map[Kind:function Name:hex_string] map[Kind:function Name:ip4_addr_string] map[Kind:function Name:ip4_addr_string_sa] map[Kind:function Name:ip4_string] map[Kind:function Name:ip6_addr_string] map[Kind:function Name:ip6_addr_string_sa] map[Kind:function Name:ip6_compressed_string] map[Kind:function Name:ip6_string] map[Kind:function Name:ip_addr_string] map[Kind:function Name:mac_address_string] map[Kind:function Name:move_right] map[Kind:function Name:netdev_bits] map[Kind:function Name:no_hash_pointers_enable] map[Kind:function Name:num_to_str] map[Kind:function Name:number] map[Kind:function Name:pointer] map[Kind:function Name:pointer_string] map[Kind:function Name:ptr_to_hashval] map[Kind:function Name:ptr_to_id] map[Kind:function Name:put_dec] map[Kind:function Name:put_dec_full8] map[Kind:function Name:put_dec_trunc8] map[Kind:function Name:range_string] map[Kind:function Name:resource_or_range] map[Kind:function Name:resource_string] map[Kind:function Name:restricted_pointer] map[Kind:function Name:rtc_str] map[Kind:function Name:scnprintf] map[Kind:function Name:set_field_width] map[Kind:function Name:set_precision] map[Kind:function Name:simple_strntoll] map[Kind:function Name:simple_strntoul] map[Kind:function Name:simple_strntoull] map[Kind:function Name:simple_strtol] map[Kind:function Name:simple_strtoll] map[Kind:function Name:simple_strtoul] map[Kind:function Name:simple_strtoull] map[Kind:function Name:skip_atoi] map[Kind:function Name:snprintf] map[Kind:function Name:spec_flag] map[Kind:function Name:special_hex_number] map[Kind:function Name:sprintf] map[Kind:function Name:sscanf] map[Kind:function Name:string] map[Kind:function Name:string_nocheck] map[Kind:function Name:symbol_string] map[Kind:function Name:time64_str] map[Kind:function Name:time_and_date] map[Kind:function Name:time_str] map[Kind:function Name:timespec64_str] map[Kind:function Name:uuid_string] map[Kind:function Name:va_format] map[Kind:function Name:vbin_printf] map[Kind:function Name:vscnprintf] map[Kind:function Name:vsnprintf] map[Kind:function Name:vsprintf] map[Kind:function Name:vsprintf_init_hashval] map[Kind:function Name:vsscanf] map[Kind:function Name:widen_string] map[Kind:struct Name:fmt] map[Kind:struct Name:format_state] map[Kind:struct Name:page_flags_fields] map[Kind:struct Name:printf_spec] map[Kind:global_variable Name:__UNIQUE_ID_addressable_bstr_printf_858] map[Kind:global_variable Name:__UNIQUE_ID_addressable_no_hash_pointers_819] map[Kind:global_variable Name:__UNIQUE_ID_addressable_scnprintf_854] map[Kind:global_variable Name:__UNIQUE_ID_addressable_simple_strntoul_822] map[Kind:global_variable Name:__UNIQUE_ID_addressable_simple_strtol_823] map[Kind:global_variable Name:__UNIQUE_ID_addressable_simple_strtoll_824] map[Kind:global_variable Name:__UNIQUE_ID_addressable_simple_strtoul_821] map[Kind:global_variable Name:__UNIQUE_ID_addressable_simple_strtoull_820] map[Kind:global_variable Name:__UNIQUE_ID_addressable_snprintf_853] map[Kind:global_variable Name:__UNIQUE_ID_addressable_sprintf_856] map[Kind:global_variable Name:__UNIQUE_ID_addressable_sscanf_860] map[Kind:global_variable Name:__UNIQUE_ID_addressable_vbin_printf_857] map[Kind:global_variable Name:__UNIQUE_ID_addressable_vscnprintf_852] map[Kind:global_variable Name:__UNIQUE_ID_addressable_vsnprintf_851] map[Kind:global_variable Name:__UNIQUE_ID_addressable_vsprintf_855] map[Kind:global_variable Name:__UNIQUE_ID_addressable_vsprintf_init_hashval_827] map[Kind:global_variable Name:__UNIQUE_ID_addressable_vsscanf_859] map[Kind:global_variable Name:__setup_debug_boot_weak_hash_enable] map[Kind:global_variable Name:__setup_hash_pointers_mode_parse] map[Kind:global_variable Name:__setup_no_hash_pointers_enable] map[Kind:global_variable Name:__setup_str_debug_boot_weak_hash_enable] map[Kind:global_variable Name:__setup_str_hash_pointers_mode_parse] map[Kind:global_variable Name:__setup_str_no_hash_pointers_enable] map[Kind:global_variable Name:decpair] map[Kind:global_variable Name:default_dec02_spec] map[Kind:global_variable Name:default_dec04_spec] map[Kind:global_variable Name:default_dec_spec] map[Kind:global_variable Name:default_flag_spec] map[Kind:global_variable Name:default_str_spec] map[Kind:global_variable Name:pff] map[Kind:enum Name:format_state] map[Kind:enum Name:hash_pointers_policy]]]
|
| 249/4 |
2026/05/18 19:06 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 40917
output: 38
thoughts: 18
Thoughts: Okay, here's my take on those thoughts, framed as an expert's internal monologue:
**Diving into the `vsprintf` Implementation**
Right, so I need to understand how this is working under the hood, and my first instinct is to go straight to the heart of it. `vsnprintf`...that's the key variable argument formatting function. Let's see what's happening internally. I'm heading directly to `lib/vsprintf.c`. That's where I expect to find the core logic for the variable argument processing and formatting that `vsnprintf` relies on. I'll need to trace how it handles the format string, how it pulls the arguments off the stack (or registers, depending on the architecture), and how it ultimately constructs the output string. I expect to see the typical parse-and-format loop in there, dealing with things like field width, precision, flags, and the different conversion specifiers (`%d`, `%s`, `%x`, etc.). I will be particularly interested in memory management within this function to make sure that it deals with potential overflows and other security aspects properly. I'll need to confirm that this version of `vsprintf` is safe, and I will be looking closely at the implementation of the string formatting and overflow handling.
|
| 250/4 |
2026/05/18 19:07 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/vsprintf.c IncludeLines:true Name:vsnprintf]
Results: map[SourceCode:2836: /**
2837: * vsnprintf - Format a string and place it in a buffer
2838: * @buf: The buffer to place the result into
2839: * @size: The size of the buffer, including the trailing null space
2840: * @fmt_str: The format string to use
2841: * @args: Arguments for the format string
2842: *
2843: * This function generally follows C99 vsnprintf, but has some
2844: * extensions and a few limitations:
2845: *
2846: * - ``%n`` is unsupported
2847: * - ``%p*`` is handled by pointer()
2848: *
2849: * See pointer() or Documentation/core-api/printk-formats.rst for more
2850: * extensive description.
2851: *
2852: * **Please update the documentation in both places when making changes**
2853: *
2854: * The return value is the number of characters which would
2855: * be generated for the given input, excluding the trailing
2856: * '\0', as per ISO C99. If you want to have the exact
2857: * number of characters written into @buf as return value
2858: * (not including the trailing '\0'), use vscnprintf(). If the
2859: * return is greater than or equal to @size, the resulting
2860: * string is truncated.
2861: *
2862: * If you're not already dealing with a va_list consider using snprintf().
2863: */
2864: int vsnprintf(char *buf, size_t size, const char *fmt_str, va_list args)
2865: {
2866: char *str, *end;
2867: struct printf_spec spec = {0};
2868: struct fmt fmt = {
2869: .str = fmt_str,
2870: .state = FORMAT_STATE_NONE,
2871: };
2872:
2873: /* Reject out-of-range values early. Large positive sizes are
2874: used for unknown buffer sizes. */
2875: if (WARN_ON_ONCE(size > INT_MAX))
2876: return 0;
2877:
2878: str = buf;
2879: end = buf + size;
2880:
2881: /* Make sure end is always >= buf */
2882: if (end < buf) {
2883: end = ((void *)-1);
2884: size = end - buf;
2885: }
2886:
2887: while (*fmt.str) {
2888: const char *old_fmt = fmt.str;
2889:
2890: fmt = format_decode(fmt, &spec);
2891:
2892: switch (fmt.state) {
2893: case FORMAT_STATE_NONE: {
2894: int read = fmt.str - old_fmt;
2895: if (str < end) {
2896: int copy = read;
2897: if (copy > end - str)
2898: copy = end - str;
2899: memcpy(str, old_fmt, copy);
2900: }
2901: str += read;
2902: continue;
2903: }
2904:
2905: case FORMAT_STATE_NUM: {
2906: unsigned long long num;
2907:
2908: if (fmt.size > sizeof(int))
2909: num = va_arg(args, long long);
2910: else
2911: num = convert_num_spec(va_arg(args, int), fmt.size, spec);
2912: str = number(str, end, num, spec);
2913: continue;
2914: }
2915:
2916: case FORMAT_STATE_WIDTH:
2917: set_field_width(&spec, va_arg(args, int));
2918: continue;
2919:
2920: case FORMAT_STATE_PRECISION:
2921: set_precision(&spec, va_arg(args, int));
2922: continue;
2923:
2924: case FORMAT_STATE_CHAR: {
2925: char c;
2926:
2927: if (!(spec.flags & LEFT)) {
2928: while (--spec.field_width > 0) {
2929: if (str < end)
2930: *str = ' ';
2931: ++str;
2932:
2933: }
2934: }
2935: c = (unsigned char) va_arg(args, int);
2936: if (str < end)
2937: *str = c;
2938: ++str;
2939: while (--spec.field_width > 0) {
2940: if (str < end)
2941: *str = ' ';
2942: ++str;
2943: }
2944: continue;
2945: }
2946:
2947: case FORMAT_STATE_STR:
2948: str = string(str, end, va_arg(args, char *), spec);
2949: continue;
2950:
2951: case FORMAT_STATE_PTR:
2952: str = pointer(fmt.str, str, end, va_arg(args, void *),
2953: spec);
2954: while (isalnum(*fmt.str))
2955: fmt.str++;
2956: continue;
2957:
2958: case FORMAT_STATE_PERCENT_CHAR:
2959: if (str < end)
2960: *str = '%';
2961: ++str;
2962: continue;
2963:
2964: default:
2965: /*
2966: * Presumably the arguments passed gcc's type
2967: * checking, but there is no safe or sane way
2968: * for us to continue parsing the format and
2969: * fetching from the va_list; the remaining
2970: * specifiers and arguments would be out of
2971: * sync.
2972: */
2973: goto out;
2974: }
2975: }
2976:
2977: out:
2978: if (size > 0) {
2979: if (str < end)
2980: *str = '\0';
2981: else
2982: end[-1] = '\0';
2983: }
2984:
2985: /* the trailing null byte doesn't count towards the total */
2986: return str-buf;
2987:
2988: }
SourceFile:lib/vsprintf.c]
|
| 251/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 43280
output: 29
thoughts: 126
Thoughts: **Analyzing the Function's Return Value**
Okay, so I'm looking at this code and I've spotted `str-buf` at the tail end. Right, that's a `ptrdiff_t` that's been type-cast to an `int` for the return. But hang on, I'm not seeing *exactly* where that calculation is being done. It's a bit unclear without the direct expression.
Let me take another crack at this. I'll search for `tmp_buf` again in `drivers/usb/gadget/function/f_fs.c`. This time, though, I'll broaden the search to the entire file to make sure I don't miss anything. Might be I need to search the entire `drivers/usb/gadget/function/f_fs.c` file using the `read-file` command, just to be absolutely certain I'm not overlooking some subtle reference. I'm sure it's in there somewhere.
|
| 252/4 |
2026/05/18 19:07 |
tool |
read-file |
0m
Args: map[File:drivers/usb/gadget/function/f_fs.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0+
/*
* f_fs.c -- user mode file system API for USB composite function controllers
*
* Copyright (C) 2010 Samsung Electronics
* Author: Michal Nazarewicz <mina86@mina86.com>
*
* Based on inode.c (GadgetFS) which was:
* Copyright (C) 2003-2004 David Brownell
* Copyright (C) 2003 Agilent Technologies
*/
/* #define DEBUG */
/* #define VERBOSE_DEBUG */
#include <linux/blkdev.h>
#include <linux/dma-buf.h>
#include <linux/dma-fence.h>
#include <linux/dma-resv.h>
#include <linux/pagemap.h>
#include <linux/export.h>
#include <linux/fs_parser.h>
#include <linux/hid.h>
#include <linux/mm.h>
#include <linux/module.h>
#include <linux/scatterlist.h>
#include <linux/sched/signal.h>
#include <linux/uio.h>
#include <linux/vmalloc.h>
#include <linux/unaligned.h>
#include <linux/usb/ccid.h>
#include <linux/usb/composite.h>
#include <linux/usb/functionfs.h>
#include <linux/usb/func_utils.h>
#include <linux/aio.h>
#include <linux/kthread.h>
#include <linux/poll.h>
#include <linux/eventfd.h>
#include "u_fs.h"
#include "u_os_desc.h"
#include "configfs.h"
#define FUNCTIONFS_MAGIC 0xa647361 /* Chosen by a honest dice roll ;) */
#define MAX_ALT_SETTINGS 2 /* Allow up to 2 alt settings to be set. */
#define DMABUF_ENQUEUE_TIMEOUT_MS 5000
MODULE_IMPORT_NS("DMA_BUF");
/* Reference counter handling */
static void ffs_data_get(struct ffs_data *ffs);
static void ffs_data_put(struct ffs_data *ffs);
/* Creates new ffs_data object. */
static struct ffs_data *__must_check ffs_data_new(const char *dev_name)
__attribute__((malloc));
/* Opened counter handling. */
static void ffs_data_closed(struct ffs_data *ffs);
/* Called with ffs->mutex held; take over ownership of data. */
static int __must_check
__ffs_data_got_descs(struct ffs_data *ffs, char *data, size_t len);
static int __must_check
__ffs_data_got_strings(struct ffs_data *ffs, char *data, size_t len);
/* The function structure ***************************************************/
struct ffs_ep;
struct ffs_function {
struct usb_configuration *conf;
struct usb_gadget *gadget;
struct ffs_data *ffs;
struct ffs_ep *eps;
u8 eps_revmap[16];
short *interfaces_nums;
struct usb_function function;
int cur_alt[MAX_CONFIG_INTERFACES];
};
static struct ffs_function *ffs_func_from_usb(struct usb_function *f)
{
return container_of(f, struct ffs_function, function);
}
static inline enum ffs_setup_state
ffs_setup_state_clear_cancelled(struct ffs_data *ffs)
{
return (enum ffs_setup_state)
cmpxchg(&ffs->setup_state, FFS_SETUP_CANCELLED, FFS_NO_SETUP);
}
static void ffs_func_eps_disable(struct ffs_function *func);
static int __must_check ffs_func_eps_enable(struct ffs_function *func);
static int ffs_func_bind(struct usb_configuration *,
struct usb_function *);
static int ffs_func_set_alt(struct usb_function *, unsigned, unsigned);
static int ffs_func_get_alt(struct usb_function *f, unsigned int intf);
static void ffs_func_disable(struct usb_function *);
static int ffs_func_setup(struct usb_function *,
const struct usb_ctrlrequest *);
static bool ffs_func_req_match(struct usb_function *,
const struct usb_ctrlrequest *,
bool config0);
static void ffs_func_suspend(struct usb_function *);
static void ffs_func_resume(struct usb_function *);
static int ffs_func_revmap_ep(struct ffs_function *func, u8 num);
static int ffs_func_revmap_intf(struct ffs_function *func, u8 intf);
/* The endpoints structures *************************************************/
struct ffs_ep {
struct usb_ep *ep; /* P: ffs->eps_lock */
struct usb_request *req; /* P: epfile->mutex */
/* [0]: full speed, [1]: high speed, [2]: super speed */
struct usb_endpoint_descriptor *descs[3];
u8 num;
};
struct ffs_dmabuf_priv {
struct list_head entry;
struct kref ref;
struct ffs_data *ffs;
struct dma_buf_attachment *attach;
struct sg_table *sgt;
enum dma_data_direction dir;
spinlock_t lock;
u64 context;
struct usb_request *req; /* P: ffs->eps_lock */
struct usb_ep *ep; /* P: ffs->eps_lock */
};
struct ffs_dma_fence {
struct dma_fence base;
struct ffs_dmabuf_priv *priv;
struct work_struct work;
};
struct ffs_epfile {
/* Protects ep->ep and ep->req. */
struct mutex mutex;
struct ffs_data *ffs;
struct ffs_ep *ep; /* P: ffs->eps_lock */
/*
* Buffer for holding data from partial reads which may happen since
* we’re rounding user read requests to a multiple of a max packet size.
*
* The pointer is initialised with NULL value and may be set by
* __ffs_epfile_read_data function to point to a temporary buffer.
*
* In normal operation, calls to __ffs_epfile_read_buffered will consume
* data from said buffer and eventually free it. Importantly, while the
* function is using the buffer, it sets the pointer to NULL. This is
* all right since __ffs_epfile_read_data and __ffs_epfile_read_buffered
* can never run concurrently (they are synchronised by epfile->mutex)
* so the latter will not assign a new value to the pointer.
*
* Meanwhile ffs_func_eps_disable frees the buffer (if the pointer is
* valid) and sets the pointer to READ_BUFFER_DROP value. This special
* value is crux of the synchronisation between ffs_func_eps_disable and
* __ffs_epfile_read_data.
*
* Once __ffs_epfile_read_data is about to finish it will try to set the
* pointer back to its old value (as described above), but seeing as the
* pointer is not-NULL (namely READ_BUFFER_DROP) it will instead free
* the buffer.
*
* == State transitions ==
*
* • ptr == NULL: (initial state)
* ◦ __ffs_epfile_read_buffer_free: go to ptr == DROP
* ◦ __ffs_epfile_read_buffered: nop
* ◦ __ffs_epfile_read_data allocates temp buffer: go to ptr == buf
* ◦ reading finishes: n/a, not in ‘and reading’ state
* • ptr == DROP:
* ◦ __ffs_epfile_read_buffer_free: nop
* ◦ __ffs_epfile_read_buffered: go to ptr == NULL
* ◦ __ffs_epfile_read_data allocates temp buffer: free buf, nop
* ◦ reading finishes: n/a, not in ‘and reading’ state
* • ptr == buf:
* ◦ __ffs_epfile_read_buffer_free: free buf, go to ptr == DROP
* ◦ __ffs_epfile_read_buffered: go to ptr == NULL and reading
* ◦ __ffs_epfile_read_data: n/a, __ffs_epfile_read_buffered
* is always called first
* ◦ reading finishes: n/a, not in ‘and reading’ state
* • ptr == NULL and reading:
* ◦ __ffs_epfile_read_buffer_free: go to ptr == DROP and reading
* ◦ __ffs_epfile_read_buffered: n/a, mutex is held
* ◦ __ffs_epfile_read_data: n/a, mutex is held
* ◦ reading finishes and …
* … all data read: free buf, go to ptr == NULL
* … otherwise: go to ptr == buf and reading
* • ptr == DROP and reading:
* ◦ __ffs_epfile_read_buffer_free: nop
* ◦ __ffs_epfile_read_buffered: n/a, mutex is held
* ◦ __ffs_epfile_read_data: n/a, mutex is held
* ◦ reading finishes: free buf, go to ptr == DROP
*/
struct ffs_buffer *read_buffer;
#define READ_BUFFER_DROP ((struct ffs_buffer *)ERR_PTR(-ESHUTDOWN))
char name[5];
unsigned char in; /* P: ffs->eps_lock */
unsigned char isoc; /* P: ffs->eps_lock */
unsigned char _pad;
/* Protects dmabufs */
struct mutex dmabufs_mutex;
struct list_head dmabufs; /* P: dmabufs_mutex */
atomic_t seqno;
};
struct ffs_buffer {
size_t length;
char *data;
char storage[] __counted_by(length);
};
/* ffs_io_data structure ***************************************************/
struct ffs_io_data {
bool aio;
bool read;
struct kiocb *kiocb;
struct iov_iter data;
const void *to_free;
char *buf;
struct mm_struct *mm;
struct work_struct work;
struct usb_ep *ep;
struct usb_request *req;
struct sg_table sgt;
bool use_sg;
struct ffs_data *ffs;
int status;
struct completion done;
};
struct ffs_desc_helper {
struct ffs_data *ffs;
unsigned interfaces_count;
unsigned eps_count;
};
static int __must_check ffs_epfiles_create(struct ffs_data *ffs);
static void ffs_epfiles_destroy(struct super_block *sb,
struct ffs_epfile *epfiles, unsigned count);
static int ffs_sb_create_file(struct super_block *sb, const char *name,
void *data, const struct file_operations *fops);
/* Devices management *******************************************************/
DEFINE_MUTEX(ffs_lock);
EXPORT_SYMBOL_GPL(ffs_lock);
static struct ffs_dev *_ffs_find_dev(const char *name);
static struct ffs_dev *_ffs_alloc_dev(void);
static void _ffs_free_dev(struct ffs_dev *dev);
static int ffs_acquire_dev(const char *dev_name, struct ffs_data *ffs_data);
static void ffs_release_dev(struct ffs_dev *ffs_dev);
static int ffs_ready(struct ffs_data *ffs);
static void ffs_closed(struct ffs_data *ffs);
/* Misc helper functions ****************************************************/
static int ffs_mutex_lock(struct mutex *mutex, unsigned nonblock)
__attribute__((warn_unused_result, nonnull));
static char *ffs_prepare_buffer(const char __user *buf, size_t len)
__attribute__((warn_unused_result, nonnull));
/* Control file aka ep0 *****************************************************/
static void ffs_ep0_complete(struct usb_ep *ep, struct usb_request *req)
{
struct ffs_data *ffs = req->context;
complete(&ffs->ep0req_completion);
}
static int __ffs_ep0_queue_wait(struct ffs_data *ffs, char *data, size_t len)
__releases(&ffs->ev.waitq.lock)
{
struct usb_request *req = ffs->ep0req;
int ret;
if (!req) {
spin_unlock_irq(&ffs->ev.waitq.lock);
return -EINVAL;
}
req->zero = len < le16_to_cpu(ffs->ev.setup.wLength);
spin_unlock_irq(&ffs->ev.waitq.lock);
req->buf = data;
req->length = len;
/*
* UDC layer requires to provide a buffer even for ZLP, but should
* not use it at all. Let's provide some poisoned pointer to catch
* possible bug in the driver.
*/
if (req->buf == NULL)
req->buf = (void *)0xDEADBABE;
reinit_completion(&ffs->ep0req_completion);
ret = usb_ep_queue(ffs->gadget->ep0, req, GFP_ATOMIC);
if (ret < 0)
return ret;
ret = wait_for_completion_interruptible(&ffs->ep0req_completion);
if (ret) {
usb_ep_dequeue(ffs->gadget->ep0, req);
return -EINTR;
}
ffs->setup_state = FFS_NO_SETUP;
return req->status ? req->status : req->actual;
}
static int __ffs_ep0_stall(struct ffs_data *ffs)
{
if (ffs->ev.can_stall) {
pr_vdebug("ep0 stall\n");
usb_ep_set_halt(ffs->gadget->ep0);
ffs->setup_state = FFS_NO_SETUP;
return -EL2HLT;
} else {
pr_debug("bogus ep0 stall!\n");
return -ESRCH;
}
}
static ssize_t ffs_ep0_write(struct file *file, const char __user *buf,
size_t len, loff_t *ptr)
{
struct ffs_data *ffs = file->private_data;
ssize_t ret;
char *data;
/* Fast check if setup was canceled */
if (ffs_setup_state_clear_cancelled(ffs) == FFS_SETUP_CANCELLED)
return -EIDRM;
/* Acquire mutex */
ret = ffs_mutex_lock(&ffs->mutex, file->f_flags & O_NONBLOCK);
if (ret < 0)
return ret;
/* Check state */
switch (ffs->state) {
case FFS_READ_DESCRIPTORS:
case FFS_READ_STRINGS:
/* Copy data */
if (len < 16) {
ret = -EINVAL;
break;
}
data = ffs_prepare_buffer(buf, len);
if (IS_ERR(data)) {
ret = PTR_ERR(data);
break;
}
/* Handle data */
if (ffs->state == FFS_READ_DESCRIPTORS) {
pr_info("read descriptors\n");
ret = __ffs_data_got_descs(ffs, data, len);
if (ret < 0)
break;
ffs->state = FFS_READ_STRINGS;
ret = len;
} else {
pr_info("read strings\n");
ret = __ffs_data_got_strings(ffs, data, len);
if (ret < 0)
break;
ret = ffs_epfiles_create(ffs);
if (ret) {
ffs->state = FFS_CLOSING;
break;
}
ffs->state = FFS_ACTIVE;
mutex_unlock(&ffs->mutex);
ret = ffs_ready(ffs);
if (ret < 0) {
ffs->state = FFS_CLOSING;
return ret;
}
return len;
}
break;
case FFS_ACTIVE:
data = NULL;
/*
* We're called from user space, we can use _irq
* rather then _irqsave
*/
spin_lock_irq(&ffs->ev.waitq.lock);
switch (ffs_setup_state_clear_cancelled(ffs)) {
case FFS_SETUP_CANCELLED:
ret = -EIDRM;
goto done_spin;
case FFS_NO_SETUP:
ret = -ESRCH;
goto done_spin;
case FFS_SETUP_PENDING:
break;
}
/* FFS_SETUP_PENDING */
if (!(ffs->ev.setup.bRequestType & USB_DIR_IN)) {
spin_unlock_irq(&ffs->ev.waitq.lock);
ret = __ffs_ep0_stall(ffs);
break;
}
/* FFS_SETUP_PENDING and not stall */
len = min_t(size_t, len, le16_to_cpu(ffs->ev.setup.wLength));
spin_unlock_irq(&ffs->ev.waitq.lock);
data = ffs_prepare_buffer(buf, len);
if (IS_ERR(data)) {
ret = PTR_ERR(data);
break;
}
spin_lock_irq(&ffs->ev.waitq.lock);
/*
* We are guaranteed to be still in FFS_ACTIVE state
* but the state of setup could have changed from
* FFS_SETUP_PENDING to FFS_SETUP_CANCELLED so we need
* to check for that. If that happened we copied data
* from user space in vain but it's unlikely.
*
* For sure we are not in FFS_NO_SETUP since this is
* the only place FFS_SETUP_PENDING -> FFS_NO_SETUP
* transition can be performed and it's protected by
* mutex.
*/
if (ffs_setup_state_clear_cancelled(ffs) ==
FFS_SETUP_CANCELLED) {
ret = -EIDRM;
done_spin:
spin_unlock_irq(&ffs->ev.waitq.lock);
} else {
/* unlocks spinlock */
ret = __ffs_ep0_queue_wait(ffs, data, len);
}
kfree(data);
break;
default:
ret = -EBADFD;
break;
}
mutex_unlock(&ffs->mutex);
return ret;
}
/* Called with ffs->ev.waitq.lock and ffs->mutex held, both released on exit. */
static ssize_t __ffs_ep0_read_events(struct ffs_data *ffs, char __user *buf,
size_t n)
__releases(&ffs->ev.waitq.lock)
{
/*
* n cannot be bigger than ffs->ev.count, which cannot be bigger than
* size of ffs->ev.types array (which is four) so that's how much space
* we reserve.
*/
struct usb_functionfs_event events[ARRAY_SIZE(ffs->ev.types)];
const size_t size = n * sizeof *events;
unsigned i = 0;
memset(events, 0, size);
do {
events[i].type = ffs->ev.types[i];
if (events[i].type == FUNCTIONFS_SETUP) {
events[i].u.setup = ffs->ev.setup;
ffs->setup_state = FFS_SETUP_PENDING;
}
} while (++i < n);
ffs->ev.count -= n;
if (ffs->ev.count)
memmove(ffs->ev.types, ffs->ev.types + n,
ffs->ev.count * sizeof *ffs->ev.types);
spin_unlock_irq(&ffs->ev.waitq.lock);
mutex_unlock(&ffs->mutex);
return copy_to_user(buf, events, size) ? -EFAULT : size;
}
static ssize_t ffs_ep0_read(struct file *file, char __user *buf,
size_t len, loff_t *ptr)
{
struct ffs_data *ffs = file->private_data;
char *data = NULL;
size_t n;
int ret;
/* Fast check if setup was canceled */
if (ffs_setup_state_clear_cancelled(ffs) == FFS_SETUP_CANCELLED)
return -EIDRM;
/* Acquire mutex */
ret = ffs_mutex_lock(&ffs->mutex, file->f_flags & O_NONBLOCK);
if (ret < 0)
return ret;
/* Check state */
if (ffs->state != FFS_ACTIVE) {
ret = -EBADFD;
goto done_mutex;
}
/*
* We're called from user space, we can use _irq rather then
* _irqsave
*/
spin_lock_irq(&ffs->ev.waitq.lock);
switch (ffs_setup_state_clear_cancelled(ffs)) {
case FFS_SETUP_CANCELLED:
ret = -EIDRM;
break;
case FFS_NO_SETUP:
n = len / sizeof(struct usb_functionfs_event);
if (!n) {
ret = -EINVAL;
break;
}
if ((file->f_flags & O_NONBLOCK) && !ffs->ev.count) {
ret = -EAGAIN;
break;
}
if (wait_event_interruptible_exclusive_locked_irq(ffs->ev.waitq,
ffs->ev.count)) {
ret = -EINTR;
break;
}
/* unlocks spinlock */
return __ffs_ep0_read_events(ffs, buf,
min_t(size_t, n, ffs->ev.count));
case FFS_SETUP_PENDING:
if (ffs->ev.setup.bRequestType & USB_DIR_IN) {
spin_unlock_irq(&ffs->ev.waitq.lock);
ret = __ffs_ep0_stall(ffs);
goto done_mutex;
}
len = min_t(size_t, len, le16_to_cpu(ffs->ev.setup.wLength));
spin_unlock_irq(&ffs->ev.waitq.lock);
if (len) {
data = kmalloc(len, GFP_KERNEL);
if (!data) {
ret = -ENOMEM;
goto done_mutex;
}
}
spin_lock_irq(&ffs->ev.waitq.lock);
/* See ffs_ep0_write() */
if (ffs_setup_state_clear_cancelled(ffs) ==
FFS_SETUP_CANCELLED) {
ret = -EIDRM;
break;
}
/* unlocks spinlock */
ret = __ffs_ep0_queue_wait(ffs, data, len);
if ((ret > 0) && (copy_to_user(buf, data, len)))
ret = -EFAULT;
goto done_mutex;
default:
ret = -EBADFD;
break;
}
spin_unlock_irq(&ffs->ev.waitq.lock);
done_mutex:
mutex_unlock(&ffs->mutex);
kfree(data);
return ret;
}
static void ffs_data_reset(struct ffs_data *ffs);
static int ffs_ep0_open(struct inode *inode, struct file *file)
{
struct ffs_data *ffs = inode->i_sb->s_fs_info;
spin_lock_irq(&ffs->eps_lock);
if (ffs->state == FFS_CLOSING) {
spin_unlock_irq(&ffs->eps_lock);
return -EBUSY;
}
if (!ffs->opened++ && ffs->state == FFS_DEACTIVATED) {
ffs->state = FFS_CLOSING;
spin_unlock_irq(&ffs->eps_lock);
ffs_data_reset(ffs);
} else {
spin_unlock_irq(&ffs->eps_lock);
}
file->private_data = ffs;
return stream_open(inode, file);
}
static int ffs_ep0_release(struct inode *inode, struct file *file)
{
struct ffs_data *ffs = file->private_data;
ffs_data_closed(ffs);
return 0;
}
static long ffs_ep0_ioctl(struct file *file, unsigned code, unsigned long value)
{
struct ffs_data *ffs = file->private_data;
struct usb_gadget *gadget = ffs->gadget;
long ret;
if (code == FUNCTIONFS_INTERFACE_REVMAP) {
struct ffs_function *func = ffs->func;
ret = func ? ffs_func_revmap_intf(func, value) : -ENODEV;
} else if (gadget && gadget->ops->ioctl) {
ret = gadget->ops->ioctl(gadget, code, value);
} else {
ret = -ENOTTY;
}
return ret;
}
static __poll_t ffs_ep0_poll(struct file *file, poll_table *wait)
{
struct ffs_data *ffs = file->private_data;
__poll_t mask = EPOLLWRNORM;
int ret;
poll_wait(file, &ffs->ev.waitq, wait);
ret = ffs_mutex_lock(&ffs->mutex, file->f_flags & O_NONBLOCK);
if (ret < 0)
return mask;
switch (ffs->state) {
case FFS_READ_DESCRIPTORS:
case FFS_READ_STRINGS:
mask |= EPOLLOUT;
break;
case FFS_ACTIVE:
switch (ffs->setup_state) {
case FFS_NO_SETUP:
if (ffs->ev.count)
mask |= EPOLLIN;
break;
case FFS_SETUP_PENDING:
case FFS_SETUP_CANCELLED:
mask |= (EPOLLIN | EPOLLOUT);
break;
}
break;
case FFS_CLOSING:
break;
case FFS_DEACTIVATED:
break;
}
mutex_unlock(&ffs->mutex);
return mask;
}
static const struct file_operations ffs_ep0_operations = {
.open = ffs_ep0_open,
.write = ffs_ep0_write,
.read = ffs_ep0_read,
.release = ffs_ep0_release,
.unlocked_ioctl = ffs_ep0_ioctl,
.poll = ffs_ep0_poll,
};
/* "Normal" endpoints operations ********************************************/
static void ffs_epfile_io_complete(struct usb_ep *_ep, struct usb_request *req)
{
struct ffs_io_data *io_data = req->context;
if (req->status)
io_data->status = req->status;
else
io_data->status = req->actual;
complete(&io_data->done);
}
static ssize_t ffs_copy_to_iter(void *data, int data_len, struct iov_iter *iter)
{
ssize_t ret = copy_to_iter(data, data_len, iter);
if (ret == data_len)
return ret;
if (iov_iter_count(iter))
return -EFAULT;
/*
* Dear user space developer!
*
* TL;DR: To stop getting below error message in your kernel log, change
* user space code using functionfs to align read buffers to a max
* packet size.
*
* Some UDCs (e.g. dwc3) require request sizes to be a multiple of a max
* packet size. When unaligned buffer is passed to functionfs, it
* internally uses a larger, aligned buffer so that such UDCs are happy.
*
* Unfortunately, this means that host may send more data than was
* requested in read(2) system call. f_fs doesn’t know what to do with
* that excess data so it simply drops it.
*
* Was the buffer aligned in the first place, no such problem would
* happen.
*
* Data may be dropped only in AIO reads. Synchronous reads are handled
* by splitting a request into multiple parts. This splitting may still
* be a problem though so it’s likely best to align the buffer
* regardless of it being AIO or not..
*
* This only affects OUT endpoints, i.e. reading data with a read(2),
* aio_read(2) etc. system calls. Writing data to an IN endpoint is not
* affected.
*/
pr_err("functionfs read size %d > requested size %zd, dropping excess data. "
"Align read buffer size to max packet size to avoid the problem.\n",
data_len, ret);
return ret;
}
/*
* allocate a virtually contiguous buffer and create a scatterlist describing it
* @sg_table - pointer to a place to be filled with sg_table contents
* @size - required buffer size
*/
static void *ffs_build_sg_list(struct sg_table *sgt, size_t sz)
{
struct page **pages;
void *vaddr, *ptr;
unsigned int n_pages;
int i;
vaddr = vmalloc(sz);
if (!vaddr)
return NULL;
n_pages = PAGE_ALIGN(sz) >> PAGE_SHIFT;
pages = kvmalloc_objs(struct page *, n_pages);
if (!pages) {
vfree(vaddr);
return NULL;
}
for (i = 0, ptr = vaddr; i < n_pages; ++i, ptr += PAGE_SIZE)
pages[i] = vmalloc_to_page(ptr);
if (sg_alloc_table_from_pages(sgt, pages, n_pages, 0, sz, GFP_KERNEL)) {
kvfree(pages);
vfree(vaddr);
return NULL;
}
kvfree(pages);
return vaddr;
}
static inline void *ffs_alloc_buffer(struct ffs_io_data *io_data,
size_t data_len)
{
if (io_data->use_sg)
return ffs_build_sg_list(&io_data->sgt, data_len);
return kmalloc(data_len, GFP_KERNEL);
}
static inline void ffs_free_buffer(struct ffs_io_data *io_data)
{
if (!io_data->buf)
return;
if (io_data->use_sg) {
sg_free_table(&io_data->sgt);
vfree(io_data->buf);
} else {
kfree(io_data->buf);
}
}
static void ffs_user_copy_worker(struct work_struct *work)
{
struct ffs_io_data *io_data = container_of(work, struct ffs_io_data,
work);
int ret = io_data->status;
bool kiocb_has_eventfd = io_data->kiocb->ki_flags & IOCB_EVENTFD;
if (io_data->read && ret > 0) {
kthread_use_mm(io_data->mm);
ret = ffs_copy_to_iter(io_data->buf, ret, &io_data->data);
kthread_unuse_mm(io_data->mm);
}
io_data->kiocb->ki_complete(io_data->kiocb, ret);
if (io_data->ffs->ffs_eventfd && !kiocb_has_eventfd)
eventfd_signal(io_data->ffs->ffs_eventfd);
usb_ep_free_request(io_data->ep, io_data->req);
if (io_data->read)
kfree(io_data->to_free);
ffs_free_buffer(io_data);
kfree(io_data);
}
static void ffs_epfile_async_io_complete(struct usb_ep *_ep,
struct usb_request *req)
{
struct ffs_io_data *io_data = req->context;
struct ffs_data *ffs = io_data->ffs;
io_data->status = req->status ? req->status : req->actual;
INIT_WORK(&io_data->work, ffs_user_copy_worker);
queue_work(ffs->io_completion_wq, &io_data->work);
}
static void __ffs_epfile_read_buffer_free(struct ffs_epfile *epfile)
{
/*
* See comment in struct ffs_epfile for full read_buffer pointer
* synchronisation story.
*/
struct ffs_buffer *buf = xchg(&epfile->read_buffer, READ_BUFFER_DROP);
if (buf && buf != READ_BUFFER_DROP)
kfree(buf);
}
/* Assumes epfile->mutex is held. */
static ssize_t __ffs_epfile_read_buffered(struct ffs_epfile *epfile,
struct iov_iter *iter)
{
/*
* Null out epfile->read_buffer so ffs_func_eps_disable does not free
* the buffer while we are using it. See comment in struct ffs_epfile
* for full read_buffer pointer synchronisation story.
*/
struct ffs_buffer *buf = xchg(&epfile->read_buffer, NULL);
ssize_t ret;
if (!buf || buf == READ_BUFFER_DROP)
return 0;
ret = copy_to_iter(buf->data, buf->length, iter);
if (buf->length == ret) {
kfree(buf);
return ret;
}
if (iov_iter_count(iter)) {
ret = -EFAULT;
} else {
buf->length -= ret;
buf->data += ret;
}
if (cmpxchg(&epfile->read_buffer, NULL, buf))
kfree(buf);
return ret;
}
/* Assumes epfile->mutex is held. */
static ssize_t __ffs_epfile_read_data(struct ffs_epfile *epfile,
void *data, int data_len,
struct iov_iter *iter)
{
struct ffs_buffer *buf;
ssize_t ret = copy_to_iter(data, data_len, iter);
if (data_len == ret)
return ret;
if (iov_iter_count(iter))
return -EFAULT;
/* See ffs_copy_to_iter for more context. */
pr_warn("functionfs read size %d > requested size %zd, splitting request into multiple reads.",
data_len, ret);
data_len -= ret;
buf = kmalloc_flex(*buf, storage, data_len);
if (!buf)
return -ENOMEM;
buf->length = data_len;
buf->data = buf->storage;
memcpy(buf->storage, data + ret, flex_array_size(buf, storage, data_len));
/*
* At this point read_buffer is NULL or READ_BUFFER_DROP (if
* ffs_func_eps_disable has been called in the meanwhile). See comment
* in struct ffs_epfile for full read_buffer pointer synchronisation
* story.
*/
if (cmpxchg(&epfile->read_buffer, NULL, buf))
kfree(buf);
return ret;
}
static struct ffs_ep *ffs_epfile_wait_ep(struct file *file)
{
struct ffs_epfile *epfile = file->private_data;
struct ffs_ep *ep;
int ret;
/* Wait for endpoint to be enabled */
ep = epfile->ep;
if (!ep) {
if (file->f_flags & O_NONBLOCK)
return ERR_PTR(-EAGAIN);
ret = wait_event_interruptible(
epfile->ffs->wait, (ep = epfile->ep));
if (ret)
return ERR_PTR(-EINTR);
}
return ep;
}
static ssize_t ffs_epfile_io(struct file *file, struct ffs_io_data *io_data)
{
struct ffs_epfile *epfile = file->private_data;
struct usb_request *req;
struct ffs_ep *ep;
char *data = NULL;
ssize_t ret, data_len = -EINVAL;
int halt;
/* Are we still active? */
if (WARN_ON(epfile->ffs->state != FFS_ACTIVE))
return -ENODEV;
ep = ffs_epfile_wait_ep(file);
if (IS_ERR(ep))
return PTR_ERR(ep);
/* Do we halt? */
halt = (!io_data->read == !epfile->in);
if (halt && epfile->isoc)
return -EINVAL;
/* We will be using request and read_buffer */
ret = ffs_mutex_lock(&epfile->mutex, file->f_flags & O_NONBLOCK);
if (ret)
goto error;
/* Allocate & copy */
if (!halt) {
struct usb_gadget *gadget;
/*
* Do we have buffered data from previous partial read? Check
* that for synchronous case only because we do not have
* facility to ‘wake up’ a pending asynchronous read and push
* buffered data to it which we would need to make things behave
* consistently.
*/
if (!io_data->aio && io_data->read) {
ret = __ffs_epfile_read_buffered(epfile, &io_data->data);
if (ret)
goto error_mutex;
}
/*
* if we _do_ wait above, the epfile->ffs->gadget might be NULL
* before the waiting completes, so do not assign to 'gadget'
* earlier
*/
gadget = epfile->ffs->gadget;
spin_lock_irq(&epfile->ffs->eps_lock);
/* In the meantime, endpoint got disabled or changed. */
if (epfile->ep != ep) {
ret = -ESHUTDOWN;
goto error_lock;
}
data_len = iov_iter_count(&io_data->data);
/*
* Controller may require buffer size to be aligned to
* maxpacketsize of an out endpoint.
*/
if (io_data->read)
data_len = usb_ep_align_maybe(gadget, ep->ep, data_len);
io_data->use_sg = gadget->sg_supported && data_len > PAGE_SIZE;
spin_unlock_irq(&epfile->ffs->eps_lock);
data = ffs_alloc_buffer(io_data, data_len);
if (!data) {
ret = -ENOMEM;
goto error_mutex;
}
if (!io_data->read &&
!copy_from_iter_full(data, data_len, &io_data->data)) {
ret = -EFAULT;
goto error_mutex;
}
}
spin_lock_irq(&epfile->ffs->eps_lock);
if (epfile->ep != ep) {
/* In the meantime, endpoint got disabled or changed. */
ret = -ESHUTDOWN;
} else if (halt) {
ret = usb_ep_set_halt(ep->ep);
if (!ret)
ret = -EBADMSG;
} else if (data_len == -EINVAL) {
/*
* Sanity Check: even though data_len can't be used
* uninitialized at the time I write this comment, some
* compilers complain about this situation.
* In order to keep the code clean from warnings, data_len is
* being initialized to -EINVAL during its declaration, which
* means we can't rely on compiler anymore to warn no future
* changes won't result in data_len being used uninitialized.
* For such reason, we're adding this redundant sanity check
* here.
*/
WARN(1, "%s: data_len == -EINVAL\n", __func__);
ret = -EINVAL;
} else if (!io_data->aio) {
bool interrupted = false;
req = ep->req;
if (io_data->use_sg) {
req->buf = NULL;
req->sg = io_data->sgt.sgl;
req->num_sgs = io_data->sgt.nents;
} else {
req->buf = data;
req->num_sgs = 0;
}
req->length = data_len;
io_data->buf = data;
init_completion(&io_data->done);
req->context = io_data;
req->complete = ffs_epfile_io_complete;
ret = usb_ep_queue(ep->ep, req, GFP_ATOMIC);
if (ret < 0)
goto error_lock;
spin_unlock_irq(&epfile->ffs->eps_lock);
if (wait_for_completion_interruptible(&io_data->done)) {
spin_lock_irq(&epfile->ffs->eps_lock);
if (epfile->ep != ep) {
ret = -ESHUTDOWN;
goto error_lock;
}
/*
* To avoid race condition with ffs_epfile_io_complete,
* dequeue the request first then check
* status. usb_ep_dequeue API should guarantee no race
* condition with req->complete callback.
*/
usb_ep_dequeue(ep->ep, req);
spin_unlock_irq(&epfile->ffs->eps_lock);
wait_for_completion(&io_data->done);
interrupted = io_data->status < 0;
}
if (interrupted)
ret = -EINTR;
else if (io_data->read && io_data->status > 0)
ret = __ffs_epfile_read_data(epfile, data, io_data->status,
&io_data->data);
else
ret = io_data->status;
goto error_mutex;
} else if (!(req = usb_ep_alloc_request(ep->ep, GFP_ATOMIC))) {
ret = -ENOMEM;
} else {
if (io_data->use_sg) {
req->buf = NULL;
req->sg = io_data->sgt.sgl;
req->num_sgs = io_data->sgt.nents;
} else {
req->buf = data;
req->num_sgs = 0;
}
req->length = data_len;
io_data->buf = data;
io_data->ep = ep->ep;
io_data->req = req;
io_data->ffs = epfile->ffs;
req->context = io_data;
req->complete = ffs_epfile_async_io_complete;
ret = usb_ep_queue(ep->ep, req, GFP_ATOMIC);
if (ret) {
io_data->req = NULL;
usb_ep_free_request(ep->ep, req);
goto error_lock;
}
ret = -EIOCBQUEUED;
/*
* Do not kfree the buffer in this function. It will be freed
* by ffs_user_copy_worker.
*/
data = NULL;
}
error_lock:
spin_unlock_irq(&epfile->ffs->eps_lock);
error_mutex:
mutex_unlock(&epfile->mutex);
error:
if (ret != -EIOCBQUEUED) /* don't free if there is iocb queued */
ffs_free_buffer(io_data);
return ret;
}
static int
ffs_epfile_open(struct inode *inode, struct file *file)
{
struct ffs_data *ffs = inode->i_sb->s_fs_info;
struct ffs_epfile *epfile;
spin_lock_irq(&ffs->eps_lock);
if (!ffs->opened) {
spin_unlock_irq(&ffs->eps_lock);
return -ENODEV;
}
/*
* we want the state to be FFS_ACTIVE; FFS_ACTIVE alone is
* not enough, though - we might have been through FFS_CLOSING
* and back to FFS_ACTIVE, with our file already removed.
*/
epfile = smp_load_acquire(&inode->i_private);
if (unlikely(ffs->state != FFS_ACTIVE || !epfile)) {
spin_unlock_irq(&ffs->eps_lock);
return -ENODEV;
}
ffs->opened++;
spin_unlock_irq(&ffs->eps_lock);
file->private_data = epfile;
return stream_open(inode, file);
}
static int ffs_aio_cancel(struct kiocb *kiocb)
{
struct ffs_io_data *io_data = kiocb->private;
int value;
if (io_data && io_data->ep && io_data->req)
value = usb_ep_dequeue(io_data->ep, io_data->req);
else
value = -EINVAL;
return value;
}
static ssize_t ffs_epfile_write_iter(struct kiocb *kiocb, struct iov_iter *from)
{
struct ffs_io_data io_data, *p = &io_data;
ssize_t res;
if (!is_sync_kiocb(kiocb)) {
p = kzalloc_obj(io_data);
if (!p)
return -ENOMEM;
p->aio = true;
} else {
memset(p, 0, sizeof(*p));
p->aio = false;
}
p->read = false;
p->kiocb = kiocb;
p->data = *from;
p->mm = current->mm;
kiocb->private = p;
if (p->aio)
kiocb_set_cancel_fn(kiocb, ffs_aio_cancel);
res = ffs_epfile_io(kiocb->ki_filp, p);
if (res == -EIOCBQUEUED)
return res;
if (p->aio)
kfree(p);
else
*from = p->data;
return res;
}
static ssize_t ffs_epfile_read_iter(struct kiocb *kiocb, struct iov_iter *to)
{
struct ffs_io_data io_data, *p = &io_data;
ssize_t res;
if (!is_sync_kiocb(kiocb)) {
p = kzalloc_obj(io_data);
if (!p)
return -ENOMEM;
p->aio = true;
} else {
memset(p, 0, sizeof(*p));
p->aio = false;
}
p->read = true;
p->kiocb = kiocb;
if (p->aio) {
p->to_free = dup_iter(&p->data, to, GFP_KERNEL);
if (!iter_is_ubuf(&p->data) && !p->to_free) {
kfree(p);
return -ENOMEM;
}
} else {
p->data = *to;
p->to_free = NULL;
}
p->mm = current->mm;
kiocb->private = p;
if (p->aio)
kiocb_set_cancel_fn(kiocb, ffs_aio_cancel);
res = ffs_epfile_io(kiocb->ki_filp, p);
if (res == -EIOCBQUEUED)
return res;
if (p->aio) {
kfree(p->to_free);
kfree(p);
} else {
*to = p->data;
}
return res;
}
static void ffs_dmabuf_release(struct kref *ref)
{
struct ffs_dmabuf_priv *priv = container_of(ref, struct ffs_dmabuf_priv, ref);
struct dma_buf_attachment *attach = priv->attach;
struct dma_buf *dmabuf = attach->dmabuf;
pr_vdebug("FFS DMABUF release\n");
dma_buf_unmap_attachment_unlocked(attach, priv->sgt, priv->dir);
dma_buf_detach(attach->dmabuf, attach);
dma_buf_put(dmabuf);
kfree(priv);
}
static void ffs_dmabuf_get(struct dma_buf_attachment *attach)
{
struct ffs_dmabuf_priv *priv = attach->importer_priv;
kref_get(&priv->ref);
}
static void ffs_dmabuf_put(struct dma_buf_attachment *attach)
{
struct ffs_dmabuf_priv *priv = attach->importer_priv;
kref_put(&priv->ref, ffs_dmabuf_release);
}
static int
ffs_epfile_release(struct inode *inode, struct file *file)
{
struct ffs_epfile *epfile = file->private_data;
struct ffs_dmabuf_priv *priv, *tmp;
struct ffs_data *ffs = epfile->ffs;
mutex_lock(&epfile->dmabufs_mutex);
/* Close all attached DMABUFs */
list_for_each_entry_safe(priv, tmp, &epfile->dmabufs, entry) {
/* Cancel any pending transfer */
spin_lock_irq(&ffs->eps_lock);
if (priv->ep && priv->req)
usb_ep_dequeue(priv->ep, priv->req);
spin_unlock_irq(&ffs->eps_lock);
list_del(&priv->entry);
ffs_dmabuf_put(priv->attach);
}
mutex_unlock(&epfile->dmabufs_mutex);
__ffs_epfile_read_buffer_free(epfile);
ffs_data_closed(epfile->ffs);
return 0;
}
static void ffs_dmabuf_cleanup(struct work_struct *work)
{
struct ffs_dma_fence *dma_fence =
container_of(work, struct ffs_dma_fence, work);
struct ffs_dmabuf_priv *priv = dma_fence->priv;
struct dma_buf_attachment *attach = priv->attach;
struct dma_fence *fence = &dma_fence->base;
ffs_dmabuf_put(attach);
dma_fence_put(fence);
}
static void ffs_dmabuf_signal_done(struct ffs_dma_fence *dma_fence, int ret)
{
struct ffs_dmabuf_priv *priv = dma_fence->priv;
struct dma_fence *fence = &dma_fence->base;
bool cookie = dma_fence_begin_signalling();
dma_fence_get(fence);
fence->error = ret;
dma_fence_signal(fence);
dma_fence_end_signalling(cookie);
/*
* The fence will be unref'd in ffs_dmabuf_cleanup.
* It can't be done here, as the unref functions might try to lock
* the resv object, which would deadlock.
*/
INIT_WORK(&dma_fence->work, ffs_dmabuf_cleanup);
queue_work(priv->ffs->io_completion_wq, &dma_fence->work);
}
static void ffs_epfile_dmabuf_io_complete(struct usb_ep *ep,
struct usb_request *req)
{
pr_vdebug("FFS: DMABUF transfer complete, status=%d\n", req->status);
ffs_dmabuf_signal_done(req->context, req->status);
usb_ep_free_request(ep, req);
}
static const char *ffs_dmabuf_get_driver_name(struct dma_fence *fence)
{
return "functionfs";
}
static const char *ffs_dmabuf_get_timeline_name(struct dma_fence *fence)
{
return "";
}
static void ffs_dmabuf_fence_release(struct dma_fence *fence)
{
struct ffs_dma_fence *dma_fence =
container_of(fence, struct ffs_dma_fence, base);
kfree(dma_fence);
}
static const struct dma_fence_ops ffs_dmabuf_fence_ops = {
.get_driver_name = ffs_dmabuf_get_driver_name,
.get_timeline_name = ffs_dmabuf_get_timeline_name,
.release = ffs_dmabuf_fence_release,
};
static int ffs_dma_resv_lock(struct dma_buf *dmabuf, bool nonblock)
{
if (!nonblock)
return dma_resv_lock_interruptible(dmabuf->resv, NULL);
if (!dma_resv_trylock(dmabuf->resv))
return -EBUSY;
return 0;
}
static struct dma_buf_attachment *
ffs_dmabuf_find_attachment(struct ffs_epfile *epfile, struct dma_buf *dmabuf)
{
struct device *dev = epfile->ffs->gadget->dev.parent;
struct dma_buf_attachment *attach = NULL;
struct ffs_dmabuf_priv *priv;
mutex_lock(&epfile->dmabufs_mutex);
list_for_each_entry(priv, &epfile->dmabufs, entry) {
if (priv->attach->dev == dev
&& priv->attach->dmabuf == dmabuf) {
attach = priv->attach;
break;
}
}
if (attach)
ffs_dmabuf_get(attach);
mutex_unlock(&epfile->dmabufs_mutex);
return attach ?: ERR_PTR(-EPERM);
}
static int ffs_dmabuf_attach(struct file *file, int fd)
{
bool nonblock = file->f_flags & O_NONBLOCK;
struct ffs_epfile *epfile = file->private_data;
struct usb_gadget *gadget = epfile->ffs->gadget;
struct dma_buf_attachment *attach;
struct ffs_dmabuf_priv *priv;
enum dma_data_direction dir;
struct sg_table *sg_table;
struct dma_buf *dmabuf;
int err;
if (!gadget || !gadget->sg_supported)
return -EPERM;
dmabuf = dma_buf_get(fd);
if (IS_ERR(dmabuf))
return PTR_ERR(dmabuf);
attach = dma_buf_attach(dmabuf, gadget->dev.parent);
if (IS_ERR(attach)) {
err = PTR_ERR(attach);
goto err_dmabuf_put;
}
priv = kzalloc_obj(*priv);
if (!priv) {
err = -ENOMEM;
goto err_dmabuf_detach;
}
dir = epfile->in ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
err = ffs_dma_resv_lock(dmabuf, nonblock);
if (err)
goto err_free_priv;
sg_table = dma_buf_map_attachment(attach, dir);
dma_resv_unlock(dmabuf->resv);
if (IS_ERR(sg_table)) {
err = PTR_ERR(sg_table);
goto err_free_priv;
}
attach->importer_priv = priv;
priv->sgt = sg_table;
priv->dir = dir;
priv->ffs = epfile->ffs;
priv->attach = attach;
spin_lock_init(&priv->lock);
kref_init(&priv->ref);
priv->context = dma_fence_context_alloc(1);
mutex_lock(&epfile->dmabufs_mutex);
list_add(&priv->entry, &epfile->dmabufs);
mutex_unlock(&epfile->dmabufs_mutex);
return 0;
err_free_priv:
kfree(priv);
err_dmabuf_detach:
dma_buf_detach(dmabuf, attach);
err_dmabuf_put:
dma_buf_put(dmabuf);
return err;
}
static int ffs_dmabuf_detach(struct file *file, int fd)
{
struct ffs_epfile *epfile = file->private_data;
struct ffs_data *ffs = epfile->ffs;
struct device *dev = ffs->gadget->dev.parent;
struct ffs_dmabuf_priv *priv, *tmp;
struct dma_buf *dmabuf;
int ret = -EPERM;
dmabuf = dma_buf_get(fd);
if (IS_ERR(dmabuf))
return PTR_ERR(dmabuf);
mutex_lock(&epfile->dmabufs_mutex);
list_for_each_entry_safe(priv, tmp, &epfile->dmabufs, entry) {
if (priv->attach->dev == dev
&& priv->attach->dmabuf == dmabuf) {
/* Cancel any pending transfer */
spin_lock_irq(&ffs->eps_lock);
if (priv->ep && priv->req)
usb_ep_dequeue(priv->ep, priv->req);
spin_unlock_irq(&ffs->eps_lock);
list_del(&priv->entry);
/* Unref the reference from ffs_dmabuf_attach() */
ffs_dmabuf_put(priv->attach);
ret = 0;
break;
}
}
mutex_unlock(&epfile->dmabufs_mutex);
dma_buf_put(dmabuf);
return ret;
}
static int ffs_dmabuf_transfer(struct file *file,
const struct usb_ffs_dmabuf_transfer_req *req)
{
bool nonblock = file->f_flags & O_NONBLOCK;
struct ffs_epfile *epfile = file->private_data;
struct dma_buf_attachment *attach;
struct ffs_dmabuf_priv *priv;
struct ffs_dma_fence *fence;
struct usb_request *usb_req;
enum dma_resv_usage resv_dir;
struct dma_buf *dmabuf;
unsigned long timeout;
struct ffs_ep *ep;
bool cookie;
u32 seqno;
long retl;
int ret;
if (req->flags & ~USB_FFS_DMABUF_TRANSFER_MASK)
return -EINVAL;
dmabuf = dma_buf_get(req->fd);
if (IS_ERR(dmabuf))
return PTR_ERR(dmabuf);
if (req->length > dmabuf->size || req->length == 0) {
ret = -EINVAL;
goto err_dmabuf_put;
}
attach = ffs_dmabuf_find_attachment(epfile, dmabuf);
if (IS_ERR(attach)) {
ret = PTR_ERR(attach);
goto err_dmabuf_put;
}
priv = attach->importer_priv;
ep = ffs_epfile_wait_ep(file);
if (IS_ERR(ep)) {
ret = PTR_ERR(ep);
goto err_attachment_put;
}
ret = ffs_dma_resv_lock(dmabuf, nonblock);
if (ret)
goto err_attachment_put;
/* Make sure we don't have writers */
timeout = nonblock ? 0 : msecs_to_jiffies(DMABUF_ENQUEUE_TIMEOUT_MS);
retl = dma_resv_wait_timeout(dmabuf->resv,
dma_resv_usage_rw(!epfile->in),
true, timeout);
if (retl == 0)
retl = -EBUSY;
if (retl < 0) {
ret = (int)retl;
goto err_resv_unlock;
}
ret = dma_resv_reserve_fences(dmabuf->resv, 1);
if (ret)
goto err_resv_unlock;
fence = kmalloc_obj(*fence);
if (!fence) {
ret = -ENOMEM;
goto err_resv_unlock;
}
fence->priv = priv;
spin_lock_irq(&epfile->ffs->eps_lock);
/* In the meantime, endpoint got disabled or changed. */
if (epfile->ep != ep) {
ret = -ESHUTDOWN;
goto err_fence_put;
}
usb_req = usb_ep_alloc_request(ep->ep, GFP_ATOMIC);
if (!usb_req) {
ret = -ENOMEM;
goto err_fence_put;
}
/*
* usb_ep_queue() guarantees that all transfers are processed in the
* order they are enqueued, so we can use a simple incrementing
* sequence number for the dma_fence.
*/
seqno = atomic_add_return(1, &epfile->seqno);
dma_fence_init(&fence->base, &ffs_dmabuf_fence_ops,
&priv->lock, priv->context, seqno);
resv_dir = epfile->in ? DMA_RESV_USAGE_READ : DMA_RESV_USAGE_WRITE;
dma_resv_add_fence(dmabuf->resv, &fence->base, resv_dir);
dma_resv_unlock(dmabuf->resv);
/* Now that the dma_fence is in place, queue the transfer. */
usb_req->length = req->length;
usb_req->buf = NULL;
usb_req->sg = priv->sgt->sgl;
usb_req->num_sgs = sg_nents_for_len(priv->sgt->sgl, req->length);
usb_req->sg_was_mapped = true;
usb_req->context = fence;
usb_req->complete = ffs_epfile_dmabuf_io_complete;
cookie = dma_fence_begin_signalling();
ret = usb_ep_queue(ep->ep, usb_req, GFP_ATOMIC);
dma_fence_end_signalling(cookie);
if (!ret) {
priv->req = usb_req;
priv->ep = ep->ep;
} else {
pr_warn("FFS: Failed to queue DMABUF: %d\n", ret);
ffs_dmabuf_signal_done(fence, ret);
usb_ep_free_request(ep->ep, usb_req);
}
spin_unlock_irq(&epfile->ffs->eps_lock);
dma_buf_put(dmabuf);
return ret;
err_fence_put:
spin_unlock_irq(&epfile->ffs->eps_lock);
dma_fence_put(&fence->base);
err_resv_unlock:
dma_resv_unlock(dmabuf->resv);
err_attachment_put:
ffs_dmabuf_put(attach);
err_dmabuf_put:
dma_buf_put(dmabuf);
return ret;
}
static long ffs_epfile_ioctl(struct file *file, unsigned code,
unsigned long value)
{
struct ffs_epfile *epfile = file->private_data;
struct ffs_ep *ep;
int ret;
if (WARN_ON(epfile->ffs->state != FFS_ACTIVE))
return -ENODEV;
switch (code) {
case FUNCTIONFS_DMABUF_ATTACH:
{
int fd;
if (copy_from_user(&fd, (void __user *)value, sizeof(fd)))
return -EFAULT;
return ffs_dmabuf_attach(file, fd);
}
case FUNCTIONFS_DMABUF_DETACH:
{
int fd;
if (copy_from_user(&fd, (void __user *)value, sizeof(fd)))
return -EFAULT;
return ffs_dmabuf_detach(file, fd);
}
case FUNCTIONFS_DMABUF_TRANSFER:
{
struct usb_ffs_dmabuf_transfer_req req;
if (copy_from_user(&req, (void __user *)value, sizeof(req)))
return -EFAULT;
return ffs_dmabuf_transfer(file, &req);
}
default:
break;
}
/* Wait for endpoint to be enabled */
ep = ffs_epfile_wait_ep(file);
if (IS_ERR(ep))
return PTR_ERR(ep);
spin_lock_irq(&epfile->ffs->eps_lock);
/* In the meantime, endpoint got disabled or changed. */
if (epfile->ep != ep) {
spin_unlock_irq(&epfile->ffs->eps_lock);
return -ESHUTDOWN;
}
switch (code) {
case FUNCTIONFS_FIFO_STATUS:
ret = usb_ep_fifo_status(epfile->ep->ep);
break;
case FUNCTIONFS_FIFO_FLUSH:
usb_ep_fifo_flush(epfile->ep->ep);
ret = 0;
break;
case FUNCTIONFS_CLEAR_HALT:
ret = usb_ep_clear_halt(epfile->ep->ep);
break;
case FUNCTIONFS_ENDPOINT_REVMAP:
ret = epfile->ep->num;
break;
case FUNCTIONFS_ENDPOINT_DESC:
{
int desc_idx;
struct usb_endpoint_descriptor desc1, *desc;
switch (epfile->ffs->gadget->speed) {
case USB_SPEED_SUPER:
case USB_SPEED_SUPER_PLUS:
desc_idx = 2;
break;
case USB_SPEED_HIGH:
desc_idx = 1;
break;
default:
desc_idx = 0;
}
desc = epfile->ep->descs[desc_idx];
memcpy(&desc1, desc, desc->bLength);
spin_unlock_irq(&epfile->ffs->eps_lock);
ret = copy_to_user((void __user *)value, &desc1, desc1.bLength);
if (ret)
ret = -EFAULT;
return ret;
}
default:
ret = -ENOTTY;
}
spin_unlock_irq(&epfile->ffs->eps_lock);
return ret;
}
static const struct file_operations ffs_epfile_operations = {
.open = ffs_epfile_open,
.write_iter = ffs_epfile_write_iter,
.read_iter = ffs_epfile_read_iter,
.release = ffs_epfile_release,
.unlocked_ioctl = ffs_epfile_ioctl,
.compat_ioctl = compat_ptr_ioctl,
};
/* File system and super block operations ***********************************/
/*
* Mounting the file system creates a controller file, used first for
* function configuration then later for event monitoring.
*/
static struct inode *__must_check
ffs_sb_make_inode(struct super_block *sb, void *data,
const struct file_operations *fops,
const struct inode_operations *iops,
struct ffs_file_perms *perms)
{
struct inode *inode;
inode = new_inode(sb);
if (inode) {
struct timespec64 ts = inode_set_ctime_current(inode);
inode->i_ino = get_next_ino();
inode->i_mode = perms->mode;
inode->i_uid = perms->uid;
inode->i_gid = perms->gid;
inode_set_atime_to_ts(inode, ts);
inode_set_mtime_to_ts(inode, ts);
inode->i_private = data;
if (fops)
inode->i_fop = fops;
if (iops)
inode->i_op = iops;
}
return inode;
}
/* Create "regular" file */
static int ffs_sb_create_file(struct super_block *sb, const char *name,
void *data, const struct file_operations *fops)
{
struct ffs_data *ffs = sb->s_fs_info;
struct dentry *dentry;
struct inode *inode;
inode = ffs_sb_make_inode(sb, data, fops, NULL, &ffs->file_perms);
if (!inode)
return -ENOMEM;
dentry = simple_start_creating(sb->s_root, name);
if (IS_ERR(dentry)) {
iput(inode);
return PTR_ERR(dentry);
}
d_make_persistent(dentry, inode);
simple_done_creating(dentry);
return 0;
}
/* Super block */
static const struct super_operations ffs_sb_operations = {
.statfs = simple_statfs,
.drop_inode = inode_just_drop,
};
struct ffs_sb_fill_data {
struct ffs_file_perms perms;
umode_t root_mode;
const char *dev_name;
bool no_disconnect;
struct ffs_data *ffs_data;
};
static int ffs_sb_fill(struct super_block *sb, struct fs_context *fc)
{
struct ffs_sb_fill_data *data = fc->fs_private;
struct inode *inode;
struct ffs_data *ffs = data->ffs_data;
ffs->sb = sb;
data->ffs_data = NULL;
sb->s_fs_info = ffs;
sb->s_blocksize = PAGE_SIZE;
sb->s_blocksize_bits = PAGE_SHIFT;
sb->s_magic = FUNCTIONFS_MAGIC;
sb->s_op = &ffs_sb_operations;
sb->s_time_gran = 1;
/* Root inode */
data->perms.mode = data->root_mode;
inode = ffs_sb_make_inode(sb, NULL,
&simple_dir_operations,
&simple_dir_inode_operations,
&data->perms);
sb->s_root = d_make_root(inode);
if (!sb->s_root)
return -ENOMEM;
/* EP0 file */
return ffs_sb_create_file(sb, "ep0", ffs, &ffs_ep0_operations);
}
enum {
Opt_no_disconnect,
Opt_rmode,
Opt_fmode,
Opt_mode,
Opt_uid,
Opt_gid,
};
static const struct fs_parameter_spec ffs_fs_fs_parameters[] = {
fsparam_bool ("no_disconnect", Opt_no_disconnect),
fsparam_u32 ("rmode", Opt_rmode),
fsparam_u32 ("fmode", Opt_fmode),
fsparam_u32 ("mode", Opt_mode),
fsparam_u32 ("uid", Opt_uid),
fsparam_u32 ("gid", Opt_gid),
{}
};
static int ffs_fs_parse_param(struct fs_context *fc, struct fs_parameter *param)
{
struct ffs_sb_fill_data *data = fc->fs_private;
struct fs_parse_result result;
int opt;
opt = fs_parse(fc, ffs_fs_fs_parameters, param, &result);
if (opt < 0)
return opt;
switch (opt) {
case Opt_no_disconnect:
data->no_disconnect = result.boolean;
break;
case Opt_rmode:
data->root_mode = (result.uint_32 & 0555) | S_IFDIR;
break;
case Opt_fmode:
data->perms.mode = (result.uint_32 & 0666) | S_IFREG;
break;
case Opt_mode:
data->root_mode = (result.uint_32 & 0555) | S_IFDIR;
data->perms.mode = (result.uint_32 & 0666) | S_IFREG;
break;
case Opt_uid:
data->perms.uid = make_kuid(current_user_ns(), result.uint_32);
if (!uid_valid(data->perms.uid))
goto unmapped_value;
break;
case Opt_gid:
data->perms.gid = make_kgid(current_user_ns(), result.uint_32);
if (!gid_valid(data->perms.gid))
goto unmapped_value;
break;
default:
return -ENOPARAM;
}
return 0;
unmapped_value:
return invalf(fc, "%s: unmapped value: %u", param->key, result.uint_32);
}
/*
* Set up the superblock for a mount.
*/
static int ffs_fs_get_tree(struct fs_context *fc)
{
struct ffs_sb_fill_data *ctx = fc->fs_private;
struct ffs_data *ffs;
int ret;
if (!fc->source)
return invalf(fc, "No source specified");
ffs = ffs_data_new(fc->source);
if (!ffs)
return -ENOMEM;
ffs->file_perms = ctx->perms;
ffs->no_disconnect = ctx->no_disconnect;
ffs->dev_name = kstrdup(fc->source, GFP_KERNEL);
if (!ffs->dev_name) {
ffs_data_put(ffs);
return -ENOMEM;
}
ret = ffs_acquire_dev(ffs->dev_name, ffs);
if (ret) {
ffs_data_put(ffs);
return ret;
}
ctx->ffs_data = ffs;
return get_tree_nodev(fc, ffs_sb_fill);
}
static void ffs_fs_free_fc(struct fs_context *fc)
{
struct ffs_sb_fill_data *ctx = fc->fs_private;
if (ctx) {
if (ctx->ffs_data) {
ffs_data_put(ctx->ffs_data);
}
kfree(ctx);
}
}
static const struct fs_context_operations ffs_fs_context_ops = {
.free = ffs_fs_free_fc,
.parse_param = ffs_fs_parse_param,
.get_tree = ffs_fs_get_tree,
};
static int ffs_fs_init_fs_context(struct fs_context *fc)
{
struct ffs_sb_fill_data *ctx;
ctx = kzalloc_obj(struct ffs_sb_fill_data);
if (!ctx)
return -ENOMEM;
ctx->perms.mode = S_IFREG | 0600;
ctx->perms.uid = GLOBAL_ROOT_UID;
ctx->perms.gid = GLOBAL_ROOT_GID;
ctx->root_mode = S_IFDIR | 0500;
ctx->no_disconnect = false;
fc->fs_private = ctx;
fc->ops = &ffs_fs_context_ops;
return 0;
}
static void
ffs_fs_kill_sb(struct super_block *sb)
{
kill_anon_super(sb);
if (sb->s_fs_info) {
struct ffs_data *ffs = sb->s_fs_info;
ffs->state = FFS_CLOSING;
ffs_data_reset(ffs);
// no configfs accesses from that point on,
// so no further schedule_work() is possible
cancel_work_sync(&ffs->reset_work);
ffs_data_put(ffs);
}
}
static struct file_system_type ffs_fs_type = {
.owner = THIS_MODULE,
.name = "functionfs",
.init_fs_context = ffs_fs_init_fs_context,
.parameters = ffs_fs_fs_parameters,
.kill_sb = ffs_fs_kill_sb,
};
MODULE_ALIAS_FS("functionfs");
/* Driver's main init/cleanup functions *************************************/
static int functionfs_init(void)
{
int ret;
ret = register_filesystem(&ffs_fs_type);
if (!ret)
pr_info("file system registered\n");
else
pr_err("failed registering file system (%d)\n", ret);
return ret;
}
static void functionfs_cleanup(void)
{
pr_info("unloading\n");
unregister_filesystem(&ffs_fs_type);
}
/* ffs_data and ffs_function construction and destruction code **************/
static void ffs_data_clear(struct ffs_data *ffs);
static void ffs_data_get(struct ffs_data *ffs)
{
refcount_inc(&ffs->ref);
}
static void ffs_data_put(struct ffs_data *ffs)
{
if (refcount_dec_and_test(&ffs->ref)) {
pr_info("%s(): freeing\n", __func__);
ffs_data_clear(ffs);
ffs_release_dev(ffs->private_data);
BUG_ON(waitqueue_active(&ffs->ev.waitq) ||
swait_active(&ffs->ep0req_completion.wait) ||
waitqueue_active(&ffs->wait));
destroy_workqueue(ffs->io_completion_wq);
kfree(ffs->dev_name);
kfree(ffs);
}
}
static void ffs_data_closed(struct ffs_data *ffs)
{
spin_lock_irq(&ffs->eps_lock);
if (--ffs->opened) { // not the last opener?
spin_unlock_irq(&ffs->eps_lock);
return;
}
if (ffs->no_disconnect) {
struct ffs_epfile *epfiles;
ffs->state = FFS_DEACTIVATED;
epfiles = ffs->epfiles;
ffs->epfiles = NULL;
spin_unlock_irq(&ffs->eps_lock);
if (epfiles)
ffs_epfiles_destroy(ffs->sb, epfiles,
ffs->eps_count);
if (ffs->setup_state == FFS_SETUP_PENDING)
__ffs_ep0_stall(ffs);
} else {
ffs->state = FFS_CLOSING;
spin_unlock_irq(&ffs->eps_lock);
ffs_data_reset(ffs);
}
}
static struct ffs_data *ffs_data_new(const char *dev_name)
{
struct ffs_data *ffs = kzalloc_obj(*ffs);
if (!ffs)
return NULL;
ffs->io_completion_wq = alloc_ordered_workqueue("%s", 0, dev_name);
if (!ffs->io_completion_wq) {
kfree(ffs);
return NULL;
}
refcount_set(&ffs->ref, 1);
ffs->opened = 0;
ffs->state = FFS_READ_DESCRIPTORS;
mutex_init(&ffs->mutex);
spin_lock_init(&ffs->eps_lock);
init_waitqueue_head(&ffs->ev.waitq);
init_waitqueue_head(&ffs->wait);
init_completion(&ffs->ep0req_completion);
/* XXX REVISIT need to update it in some places, or do we? */
ffs->ev.can_stall = 1;
return ffs;
}
static void ffs_data_clear(struct ffs_data *ffs)
{
struct ffs_epfile *epfiles;
unsigned long flags;
ffs_closed(ffs);
BUG_ON(ffs->gadget);
spin_lock_irqsave(&ffs->eps_lock, flags);
epfiles = ffs->epfiles;
ffs->epfiles = NULL;
spin_unlock_irqrestore(&ffs->eps_lock, flags);
/*
* potential race possible between ffs_func_eps_disable
* & ffs_epfile_release therefore maintaining a local
* copy of epfile will save us from use-after-free.
*/
if (epfiles) {
ffs_epfiles_destroy(ffs->sb, epfiles, ffs->eps_count);
ffs->epfiles = NULL;
}
if (ffs->ffs_eventfd) {
eventfd_ctx_put(ffs->ffs_eventfd);
ffs->ffs_eventfd = NULL;
}
kfree(ffs->raw_descs_data);
kfree(ffs->raw_strings);
kfree(ffs->stringtabs);
}
static void ffs_data_reset(struct ffs_data *ffs)
{
ffs_data_clear(ffs);
spin_lock_irq(&ffs->eps_lock);
ffs->raw_descs_data = NULL;
ffs->raw_descs = NULL;
ffs->raw_strings = NULL;
ffs->stringtabs = NULL;
ffs->raw_descs_length = 0;
ffs->fs_descs_count = 0;
ffs->hs_descs_count = 0;
ffs->ss_descs_count = 0;
ffs->strings_count = 0;
ffs->interfaces_count = 0;
ffs->eps_count = 0;
ffs->ev.count = 0;
ffs->state = FFS_READ_DESCRIPTORS;
ffs->setup_state = FFS_NO_SETUP;
ffs->flags = 0;
ffs->ms_os_descs_ext_prop_count = 0;
ffs->ms_os_descs_ext_prop_name_len = 0;
ffs->ms_os_descs_ext_prop_data_len = 0;
spin_unlock_irq(&ffs->eps_lock);
}
static int functionfs_bind(struct ffs_data *ffs, struct usb_composite_dev *cdev)
{
struct usb_gadget_strings **lang;
int first_id;
if ((ffs->state != FFS_ACTIVE
|| test_and_set_bit(FFS_FL_BOUND, &ffs->flags)))
return -EBADFD;
first_id = usb_string_ids_n(cdev, ffs->strings_count);
if (first_id < 0)
return first_id;
ffs->ep0req = usb_ep_alloc_request(cdev->gadget->ep0, GFP_KERNEL);
if (!ffs->ep0req)
return -ENOMEM;
ffs->ep0req->complete = ffs_ep0_complete;
ffs->ep0req->context = ffs;
lang = ffs->stringtabs;
if (lang) {
for (; *lang; ++lang) {
struct usb_string *str = (*lang)->strings;
int id = first_id;
for (; str->s; ++id, ++str)
str->id = id;
}
}
ffs->gadget = cdev->gadget;
ffs_data_get(ffs);
return 0;
}
static void functionfs_unbind(struct ffs_data *ffs)
{
if (!WARN_ON(!ffs->gadget)) {
/* dequeue before freeing ep0req */
usb_ep_dequeue(ffs->gadget->ep0, ffs->ep0req);
mutex_lock(&ffs->mutex);
usb_ep_free_request(ffs->gadget->ep0, ffs->ep0req);
ffs->ep0req = NULL;
ffs->gadget = NULL;
clear_bit(FFS_FL_BOUND, &ffs->flags);
mutex_unlock(&ffs->mutex);
ffs_data_put(ffs);
}
}
static int ffs_epfiles_create(struct ffs_data *ffs)
{
struct ffs_epfile *epfile, *epfiles;
unsigned i, count;
int err;
count = ffs->eps_count;
epfiles = kzalloc_objs(*epfiles, count);
if (!epfiles)
return -ENOMEM;
epfile = epfiles;
for (i = 1; i <= count; ++i, ++epfile) {
epfile->ffs = ffs;
mutex_init(&epfile->mutex);
mutex_init(&epfile->dmabufs_mutex);
INIT_LIST_HEAD(&epfile->dmabufs);
if (ffs->user_flags & FUNCTIONFS_VIRTUAL_ADDR)
sprintf(epfile->name, "ep%02x", ffs->eps_addrmap[i]);
else
sprintf(epfile->name, "ep%u", i);
err = ffs_sb_create_file(ffs->sb, epfile->name,
epfile, &ffs_epfile_operations);
if (err) {
ffs_epfiles_destroy(ffs->sb, epfiles, i - 1);
return err;
}
}
ffs->epfiles = epfiles;
return 0;
}
static void clear_one(struct dentry *dentry)
{
smp_store_release(&dentry->d_inode->i_private, NULL);
}
static void ffs_epfiles_destroy(struct super_block *sb,
struct ffs_epfile *epfiles, unsigned count)
{
struct ffs_epfile *epfile = epfiles;
struct dentry *root = sb->s_root;
for (; count; --count, ++epfile) {
BUG_ON(mutex_is_locked(&epfile->mutex));
simple_remove_by_name(root, epfile->name, clear_one);
}
kfree(epfiles);
}
static void ffs_func_eps_disable(struct ffs_function *func)
{
struct ffs_ep *ep;
struct ffs_epfile *epfile;
unsigned short count;
unsigned long flags;
spin_lock_irqsave(&func->ffs->eps_lock, flags);
count = func->ffs->eps_count;
epfile = func->ffs->epfiles;
ep = func->eps;
while (count--) {
/* pending requests get nuked */
if (ep->ep)
usb_ep_disable(ep->ep);
++ep;
if (epfile) {
epfile->ep = NULL;
__ffs_epfile_read_buffer_free(epfile);
++epfile;
}
}
spin_unlock_irqrestore(&func->ffs->eps_lock, flags);
}
static int ffs_func_eps_enable(struct ffs_function *func)
{
struct ffs_data *ffs;
struct ffs_ep *ep;
struct ffs_epfile *epfile;
unsigned short count;
unsigned long flags;
int ret = 0;
spin_lock_irqsave(&func->ffs->eps_lock, flags);
ffs = func->ffs;
ep = func->eps;
epfile = ffs->epfiles;
count = ffs->eps_count;
if (!epfile) {
ret = -ENOMEM;
goto done;
}
while (count--) {
ep->ep->driver_data = ep;
ret = config_ep_by_speed(func->gadget, &func->function, ep->ep);
if (ret) {
pr_err("%s: config_ep_by_speed(%s) returned %d\n",
__func__, ep->ep->name, ret);
break;
}
ret = usb_ep_enable(ep->ep);
if (!ret) {
epfile->ep = ep;
epfile->in = usb_endpoint_dir_in(ep->ep->desc);
epfile->isoc = usb_endpoint_xfer_isoc(ep->ep->desc);
} else {
break;
}
++ep;
++epfile;
}
wake_up_interruptible(&ffs->wait);
done:
spin_unlock_irqrestore(&func->ffs->eps_lock, flags);
return ret;
}
/* Parsing and building descriptors and strings *****************************/
/*
* This validates if data pointed by data is a valid USB descriptor as
* well as record how many interfaces, endpoints and strings are
* required by given configuration. Returns address after the
* descriptor or NULL if data is invalid.
*/
enum ffs_entity_type {
FFS_DESCRIPTOR, FFS_INTERFACE, FFS_STRING, FFS_ENDPOINT
};
enum ffs_os_desc_type {
FFS_OS_DESC, FFS_OS_DESC_EXT_COMPAT, FFS_OS_DESC_EXT_PROP
};
typedef int (*ffs_entity_callback)(enum ffs_entity_type entity,
u8 *valuep,
struct usb_descriptor_header *desc,
void *priv);
typedef int (*ffs_os_desc_callback)(enum ffs_os_desc_type entity,
struct usb_os_desc_header *h, void *data,
unsigned len, void *priv);
static int __must_check ffs_do_single_desc(char *data, unsigned len,
ffs_entity_callback entity,
void *priv, int *current_class, int *current_subclass)
{
struct usb_descriptor_header *_ds = (void *)data;
u8 length;
int ret;
/* At least two bytes are required: length and type */
if (len < 2) {
pr_vdebug("descriptor too short\n");
return -EINVAL;
}
/* If we have at least as many bytes as the descriptor takes? */
length = _ds->bLength;
if (len < length) {
pr_vdebug("descriptor longer then available data\n");
return -EINVAL;
}
#define __entity_check_INTERFACE(val) 1
#define __entity_check_STRING(val) (val)
#define __entity_check_ENDPOINT(val) ((val) & USB_ENDPOINT_NUMBER_MASK)
#define __entity(type, val) do { \
pr_vdebug("entity " #type "(%02x)\n", (val)); \
if (!__entity_check_ ##type(val)) { \
pr_vdebug("invalid entity's value\n"); \
return -EINVAL; \
} \
ret = entity(FFS_ ##type, &val, _ds, priv); \
if (ret < 0) { \
pr_debug("entity " #type "(%02x); ret = %d\n", \
(val), ret); \
return ret; \
} \
} while (0)
/* Parse descriptor depending on type. */
switch (_ds->bDescriptorType) {
case USB_DT_DEVICE:
case USB_DT_CONFIG:
case USB_DT_STRING:
case USB_DT_DEVICE_QUALIFIER:
/* function can't have any of those */
pr_vdebug("descriptor reserved for gadget: %d\n",
_ds->bDescriptorType);
return -EINVAL;
case USB_DT_INTERFACE: {
struct usb_interface_descriptor *ds = (void *)_ds;
pr_vdebug("interface descriptor\n");
if (length != sizeof *ds)
goto inv_length;
__entity(INTERFACE, ds->bInterfaceNumber);
if (ds->iInterface)
__entity(STRING, ds->iInterface);
*current_class = ds->bInterfaceClass;
*current_subclass = ds->bInterfaceSubClass;
}
break;
case USB_DT_ENDPOINT: {
struct usb_endpoint_descriptor *ds = (void *)_ds;
pr_vdebug("endpoint descriptor\n");
if (length != USB_DT_ENDPOINT_SIZE &&
length != USB_DT_ENDPOINT_AUDIO_SIZE)
goto inv_length;
__entity(ENDPOINT, ds->bEndpointAddress);
}
break;
case USB_TYPE_CLASS | 0x01:
if (*current_class == USB_INTERFACE_CLASS_HID) {
pr_vdebug("hid descriptor\n");
if (length != sizeof(struct hid_descriptor))
goto inv_length;
break;
} else if (*current_class == USB_INTERFACE_CLASS_CCID) {
pr_vdebug("ccid descriptor\n");
if (length != sizeof(struct ccid_descriptor))
goto inv_length;
break;
} else if (*current_class == USB_CLASS_APP_SPEC &&
*current_subclass == USB_SUBCLASS_DFU) {
pr_vdebug("dfu functional descriptor\n");
if (length != sizeof(struct usb_dfu_functional_descriptor))
goto inv_length;
break;
} else {
pr_vdebug("unknown descriptor: %d for class %d\n",
_ds->bDescriptorType, *current_class);
return -EINVAL;
}
case USB_DT_OTG:
if (length != sizeof(struct usb_otg_descriptor))
goto inv_length;
break;
case USB_DT_INTERFACE_ASSOCIATION: {
struct usb_interface_assoc_descriptor *ds = (void *)_ds;
pr_vdebug("interface association descriptor\n");
if (length != sizeof *ds)
goto inv_length;
if (ds->iFunction)
__entity(STRING, ds->iFunction);
}
break;
case USB_DT_SS_ENDPOINT_COMP:
pr_vdebug("EP SS companion descriptor\n");
if (length != sizeof(struct usb_ss_ep_comp_descriptor))
goto inv_length;
break;
case USB_DT_OTHER_SPEED_CONFIG:
case USB_DT_INTERFACE_POWER:
case USB_DT_DEBUG:
case USB_DT_SECURITY:
case USB_DT_CS_RADIO_CONTROL:
/* TODO */
pr_vdebug("unimplemented descriptor: %d\n", _ds->bDescriptorType);
return -EINVAL;
default:
/* We should never be here */
pr_vdebug("unknown descriptor: %d\n", _ds->bDescriptorType);
return -EINVAL;
inv_length:
pr_vdebug("invalid length: %d (descriptor %d)\n",
_ds->bLength, _ds->bDescriptorType);
return -EINVAL;
}
#undef __entity
#undef __entity_check_DESCRIPTOR
#undef __entity_check_INTERFACE
#undef __entity_check_STRING
#undef __entity_check_ENDPOINT
return length;
}
static int __must_check ffs_do_descs(unsigned count, char *data, unsigned len,
ffs_entity_callback entity, void *priv)
{
const unsigned _len = len;
unsigned long num = 0;
int current_class = -1;
int current_subclass = -1;
for (;;) {
int ret;
if (num == count)
data = NULL;
/* Record "descriptor" entity */
ret = entity(FFS_DESCRIPTOR, (u8 *)num, (void *)data, priv);
if (ret < 0) {
pr_debug("entity DESCRIPTOR(%02lx); ret = %d\n",
num, ret);
return ret;
}
if (!data)
return _len - len;
ret = ffs_do_single_desc(data, len, entity, priv,
¤t_class, ¤t_subclass);
if (ret < 0) {
pr_debug("%s returns %d\n", __func__, ret);
return ret;
}
len -= ret;
data += ret;
++num;
}
}
static int __ffs_data_do_entity(enum ffs_entity_type type,
u8 *valuep, struct usb_descriptor_header *desc,
void *priv)
{
struct ffs_desc_helper *helper = priv;
struct usb_endpoint_descriptor *d;
switch (type) {
case FFS_DESCRIPTOR:
break;
case FFS_INTERFACE:
/*
* Interfaces are indexed from zero so if we
* encountered interface "n" then there are at least
* "n+1" interfaces.
*/
if (*valuep >= helper->interfaces_count)
helper->interfaces_count = *valuep + 1;
break;
case FFS_STRING:
/*
* Strings are indexed from 1 (0 is reserved
* for languages list)
*/
if (*valuep > helper->ffs->strings_count)
helper->ffs->strings_count = *valuep;
break;
case FFS_ENDPOINT:
d = (void *)desc;
helper->eps_count++;
if (helper->eps_count >= FFS_MAX_EPS_COUNT)
return -EINVAL;
/* Check if descriptors for any speed were already parsed */
if (!helper->ffs->eps_count && !helper->ffs->interfaces_count)
helper->ffs->eps_addrmap[helper->eps_count] =
d->bEndpointAddress;
else if (helper->ffs->eps_addrmap[helper->eps_count] !=
d->bEndpointAddress)
return -EINVAL;
break;
}
return 0;
}
static int __ffs_do_os_desc_header(enum ffs_os_desc_type *next_type,
struct usb_os_desc_header *desc)
{
u16 bcd_version = le16_to_cpu(desc->bcdVersion);
u16 w_index = le16_to_cpu(desc->wIndex);
if (bcd_version == 0x1) {
pr_warn("bcdVersion must be 0x0100, stored in Little Endian order. "
"Userspace driver should be fixed, accepting 0x0001 for compatibility.\n");
} else if (bcd_version != 0x100) {
pr_vdebug("unsupported os descriptors version: 0x%x\n",
bcd_version);
return -EINVAL;
}
switch (w_index) {
case 0x4:
*next_type = FFS_OS_DESC_EXT_COMPAT;
break;
case 0x5:
*next_type = FFS_OS_DESC_EXT_PROP;
break;
default:
pr_vdebug("unsupported os descriptor type: %d", w_index);
return -EINVAL;
}
return sizeof(*desc);
}
/*
* Process all extended compatibility/extended property descriptors
* of a feature descriptor
*/
static int __must_check ffs_do_single_os_desc(char *data, unsigned len,
enum ffs_os_desc_type type,
u16 feature_count,
ffs_os_desc_callback entity,
void *priv,
struct usb_os_desc_header *h)
{
int ret;
const unsigned _len = len;
/* loop over all ext compat/ext prop descriptors */
while (feature_count--) {
ret = entity(type, h, data, len, priv);
if (ret < 0) {
pr_debug("bad OS descriptor, type: %d\n", type);
return ret;
}
data += ret;
len -= ret;
}
return _len - len;
}
/* Process a number of complete Feature Descriptors (Ext Compat or Ext Prop) */
static int __must_check ffs_do_os_descs(unsigned count,
char *data, unsigned len,
ffs_os_desc_callback entity, void *priv)
{
const unsigned _len = len;
unsigned long num = 0;
for (num = 0; num < count; ++num) {
int ret;
enum ffs_os_desc_type type;
u16 feature_count;
struct usb_os_desc_header *desc = (void *)data;
if (len < sizeof(*desc))
return -EINVAL;
/*
* Record "descriptor" entity.
* Process dwLength, bcdVersion, wIndex, get b/wCount.
* Move the data pointer to the beginning of extended
* compatibilities proper or extended properties proper
* portions of the data
*/
if (le32_to_cpu(desc->dwLength) > len)
return -EINVAL;
ret = __ffs_do_os_desc_header(&type, desc);
if (ret < 0) {
pr_debug("entity OS_DESCRIPTOR(%02lx); ret = %d\n",
num, ret);
return ret;
}
/*
* 16-bit hex "?? 00" Little Endian looks like 8-bit hex "??"
*/
feature_count = le16_to_cpu(desc->wCount);
if (type == FFS_OS_DESC_EXT_COMPAT &&
(feature_count > 255 || desc->Reserved))
return -EINVAL;
len -= ret;
data += ret;
/*
* Process all function/property descriptors
* of this Feature Descriptor
*/
ret = ffs_do_single_os_desc(data, len, type,
feature_count, entity, priv, desc);
if (ret < 0) {
pr_debug("%s returns %d\n", __func__, ret);
return ret;
}
len -= ret;
data += ret;
}
return _len - len;
}
/*
* Validate contents of the buffer from userspace related to OS descriptors.
*/
static int __ffs_data_do_os_desc(enum ffs_os_desc_type type,
struct usb_os_desc_header *h, void *data,
unsigned len, void *priv)
{
struct ffs_data *ffs = priv;
u8 length;
switch (type) {
case FFS_OS_DESC_EXT_COMPAT: {
struct usb_ext_compat_desc *d = data;
int i;
if (len < sizeof(*d) ||
d->bFirstInterfaceNumber >= ffs->interfaces_count)
return -EINVAL;
if (d->Reserved1 != 1) {
/*
* According to the spec, Reserved1 must be set to 1
* but older kernels incorrectly rejected non-zero
* values. We fix it here to avoid returning EINVAL
* in response to values we used to accept.
*/
pr_debug("usb_ext_compat_desc::Reserved1 forced to 1\n");
d->Reserved1 = 1;
}
for (i = 0; i < ARRAY_SIZE(d->Reserved2); ++i)
if (d->Reserved2[i])
return -EINVAL;
length = sizeof(struct usb_ext_compat_desc);
}
break;
case FFS_OS_DESC_EXT_PROP: {
struct usb_ext_prop_desc *d = data;
u32 type, pdl;
u16 pnl;
if (len < sizeof(*d) || h->interface >= ffs->interfaces_count)
return -EINVAL;
length = le32_to_cpu(d->dwSize);
if (len < length)
return -EINVAL;
type = le32_to_cpu(d->dwPropertyDataType);
if (type < USB_EXT_PROP_UNICODE ||
type > USB_EXT_PROP_UNICODE_MULTI) {
pr_vdebug("unsupported os descriptor property type: %d",
type);
return -EINVAL;
}
pnl = le16_to_cpu(d->wPropertyNameLength);
if (length < 14 + pnl) {
pr_vdebug("invalid os descriptor length: %d pnl:%d (descriptor %d)\n",
length, pnl, type);
return -EINVAL;
}
pdl = le32_to_cpu(*(__le32 *)((u8 *)data + 10 + pnl));
if (length != 14 + pnl + pdl) {
pr_vdebug("invalid os descriptor length: %d pnl:%d pdl:%d (descriptor %d)\n",
length, pnl, pdl, type);
return -EINVAL;
}
++ffs->ms_os_descs_ext_prop_count;
/* property name reported to the host as "WCHAR"s */
ffs->ms_os_descs_ext_prop_name_len += pnl * 2;
ffs->ms_os_descs_ext_prop_data_len += pdl;
}
break;
default:
pr_vdebug("unknown descriptor: %d\n", type);
return -EINVAL;
}
return length;
}
static int __ffs_data_got_descs(struct ffs_data *ffs,
char *const _data, size_t len)
{
char *data = _data, *raw_descs;
unsigned os_descs_count = 0, counts[3], flags;
int ret = -EINVAL, i;
struct ffs_desc_helper helper;
if (get_unaligned_le32(data + 4) != len)
goto error;
switch (get_unaligned_le32(data)) {
case FUNCTIONFS_DESCRIPTORS_MAGIC:
flags = FUNCTIONFS_HAS_FS_DESC | FUNCTIONFS_HAS_HS_DESC;
data += 8;
len -= 8;
break;
case FUNCTIONFS_DESCRIPTORS_MAGIC_V2:
flags = get_unaligned_le32(data + 8);
ffs->user_flags = flags;
if (flags & ~(FUNCTIONFS_HAS_FS_DESC |
FUNCTIONFS_HAS_HS_DESC |
FUNCTIONFS_HAS_SS_DESC |
FUNCTIONFS_HAS_MS_OS_DESC |
FUNCTIONFS_VIRTUAL_ADDR |
FUNCTIONFS_EVENTFD |
FUNCTIONFS_ALL_CTRL_RECIP |
FUNCTIONFS_CONFIG0_SETUP)) {
ret = -ENOSYS;
goto error;
}
data += 12;
len -= 12;
break;
default:
goto error;
}
if (flags & FUNCTIONFS_EVENTFD) {
if (len < 4)
goto error;
ffs->ffs_eventfd =
eventfd_ctx_fdget((int)get_unaligned_le32(data));
if (IS_ERR(ffs->ffs_eventfd)) {
ret = PTR_ERR(ffs->ffs_eventfd);
ffs->ffs_eventfd = NULL;
goto error;
}
data += 4;
len -= 4;
}
/* Read fs_count, hs_count and ss_count (if present) */
for (i = 0; i < 3; ++i) {
if (!(flags & (1 << i))) {
counts[i] = 0;
} else if (len < 4) {
goto error;
} else {
counts[i] = get_unaligned_le32(data);
data += 4;
len -= 4;
}
}
if (flags & (1 << i)) {
if (len < 4) {
goto error;
}
os_descs_count = get_unaligned_le32(data);
data += 4;
len -= 4;
}
/* Read descriptors */
raw_descs = data;
helper.ffs = ffs;
for (i = 0; i < 3; ++i) {
if (!counts[i])
continue;
helper.interfaces_count = 0;
helper.eps_count = 0;
ret = ffs_do_descs(counts[i], data, len,
__ffs_data_do_entity, &helper);
if (ret < 0)
goto error;
if (!ffs->eps_count && !ffs->interfaces_count) {
ffs->eps_count = helper.eps_count;
ffs->interfaces_count = helper.interfaces_count;
} else {
if (ffs->eps_count != helper.eps_count) {
ret = -EINVAL;
goto error;
}
if (ffs->interfaces_count != helper.interfaces_count) {
ret = -EINVAL;
goto error;
}
}
data += ret;
len -= ret;
}
if (os_descs_count) {
ret = ffs_do_os_descs(os_descs_count, data, len,
__ffs_data_do_os_desc, ffs);
if (ret < 0)
goto error;
data += ret;
len -= ret;
}
if (raw_descs == data || len) {
ret = -EINVAL;
goto error;
}
ffs->raw_descs_data = _data;
ffs->raw_descs = raw_descs;
ffs->raw_descs_length = data - raw_descs;
ffs->fs_descs_count = counts[0];
ffs->hs_descs_count = counts[1];
ffs->ss_descs_count = counts[2];
ffs->ms_os_descs_count = os_descs_count;
return 0;
error:
kfree(_data);
return ret;
}
static int __ffs_data_got_strings(struct ffs_data *ffs,
char *const _data, size_t len)
{
u32 str_count, needed_count, lang_count;
struct usb_gadget_strings **stringtabs, *t;
const char *data = _data;
struct usb_string *s;
if (len < 16 ||
get_unaligned_le32(data) != FUNCTIONFS_STRINGS_MAGIC ||
get_unaligned_le32(data + 4) != len)
goto error;
str_count = get_unaligned_le32(data + 8);
lang_count = get_unaligned_le32(data + 12);
/* if one is zero the other must be zero */
if (!str_count != !lang_count)
goto error;
/* Do we have at least as many strings as descriptors need? */
needed_count = ffs->strings_count;
if (str_count < needed_count)
goto error;
/*
* If we don't need any strings just return and free all
* memory.
*/
if (!needed_count) {
kfree(_data);
return 0;
}
/* Allocate everything in one chunk so there's less maintenance. */
{
unsigned i = 0;
vla_group(d);
vla_item(d, struct usb_gadget_strings *, stringtabs,
size_add(lang_count, 1));
vla_item(d, struct usb_gadget_strings, stringtab, lang_count);
vla_item(d, struct usb_string, strings,
size_mul(lang_count, (needed_count + 1)));
char *vlabuf = kmalloc(vla_group_size(d), GFP_KERNEL);
if (!vlabuf) {
kfree(_data);
return -ENOMEM;
}
/* Initialize the VLA pointers */
stringtabs = vla_ptr(vlabuf, d, stringtabs);
t = vla_ptr(vlabuf, d, stringtab);
i = lang_count;
do {
*stringtabs++ = t++;
} while (--i);
*stringtabs = NULL;
/* stringtabs = vlabuf = d_stringtabs for later kfree */
stringtabs = vla_ptr(vlabuf, d, stringtabs);
t = vla_ptr(vlabuf, d, stringtab);
s = vla_ptr(vlabuf, d, strings);
}
/* For each language */
data += 16;
len -= 16;
do { /* lang_count > 0 so we can use do-while */
unsigned needed = needed_count;
u32 str_per_lang = str_count;
if (len < 3)
goto error_free;
t->language = get_unaligned_le16(data);
t->strings = s;
++t;
data += 2;
len -= 2;
/* For each string */
do { /* str_count > 0 so we can use do-while */
size_t length = strnlen(data, len);
if (length == len)
goto error_free;
/*
* User may provide more strings then we need,
* if that's the case we simply ignore the
* rest
*/
if (needed) {
/*
* s->id will be set while adding
* function to configuration so for
* now just leave garbage here.
*/
s->s = data;
--needed;
++s;
}
data += length + 1;
len -= length + 1;
} while (--str_per_lang);
s->id = 0; /* terminator */
s->s = NULL;
++s;
} while (--lang_count);
/* Some garbage left? */
if (len)
goto error_free;
/* Done! */
ffs->stringtabs = stringtabs;
ffs->raw_strings = _data;
return 0;
error_free:
kfree(stringtabs);
error:
kfree(_data);
return -EINVAL;
}
/* Events handling and management *******************************************/
static void __ffs_event_add(struct ffs_data *ffs,
enum usb_functionfs_event_type type)
{
enum usb_functionfs_event_type rem_type1, rem_type2 = type;
int neg = 0;
/*
* Abort any unhandled setup
*
* We do not need to worry about some cmpxchg() changing value
* of ffs->setup_state without holding the lock because when
* state is FFS_SETUP_PENDING cmpxchg() in several places in
* the source does nothing.
*/
if (ffs->setup_state == FFS_SETUP_PENDING)
ffs->setup_state = FFS_SETUP_CANCELLED;
/*
* Logic of this function guarantees that there are at most four pending
* evens on ffs->ev.types queue. This is important because the queue
* has space for four elements only and __ffs_ep0_read_events function
* depends on that limit as well. If more event types are added, those
* limits have to be revisited or guaranteed to still hold.
*/
switch (type) {
case FUNCTIONFS_RESUME:
rem_type2 = FUNCTIONFS_SUSPEND;
fallthrough;
case FUNCTIONFS_SUSPEND:
case FUNCTIONFS_SETUP:
rem_type1 = type;
/* Discard all similar events */
break;
case FUNCTIONFS_BIND:
case FUNCTIONFS_UNBIND:
case FUNCTIONFS_DISABLE:
case FUNCTIONFS_ENABLE:
/* Discard everything other then power management. */
rem_type1 = FUNCTIONFS_SUSPEND;
rem_type2 = FUNCTIONFS_RESUME;
neg = 1;
break;
default:
WARN(1, "%d: unknown event, this should not happen\n", type);
return;
}
{
u8 *ev = ffs->ev.types, *out = ev;
unsigned n = ffs->ev.count;
for (; n; --n, ++ev)
if ((*ev == rem_type1 || *ev == rem_type2) == neg)
*out++ = *ev;
else
pr_vdebug("purging event %d\n", *ev);
ffs->ev.count = out - ffs->ev.types;
}
pr_vdebug("adding event %d\n", type);
ffs->ev.types[ffs->ev.count++] = type;
wake_up_locked(&ffs->ev.waitq);
if (ffs->ffs_eventfd)
eventfd_signal(ffs->ffs_eventfd);
}
static void ffs_event_add(struct ffs_data *ffs,
enum usb_functionfs_event_type type)
{
unsigned long flags;
spin_lock_irqsave(&ffs->ev.waitq.lock, flags);
__ffs_event_add(ffs, type);
spin_unlock_irqrestore(&ffs->ev.waitq.lock, flags);
}
/* Bind/unbind USB function hooks *******************************************/
static int ffs_ep_addr2idx(struct ffs_data *ffs, u8 endpoint_address)
{
int i;
for (i = 1; i < ARRAY_SIZE(ffs->eps_addrmap); ++i)
if (ffs->eps_addrmap[i] == endpoint_address)
return i;
return -ENOENT;
}
static int __ffs_func_bind_do_descs(enum ffs_entity_type type, u8 *valuep,
struct usb_descriptor_header *desc,
void *priv)
{
struct usb_endpoint_descriptor *ds = (void *)desc;
struct ffs_function *func = priv;
struct ffs_ep *ffs_ep;
unsigned ep_desc_id;
int idx;
static const char *speed_names[] = { "full", "high", "super" };
if (type != FFS_DESCRIPTOR)
return 0;
/*
* If ss_descriptors is not NULL, we are reading super speed
* descriptors; if hs_descriptors is not NULL, we are reading high
* speed descriptors; otherwise, we are reading full speed
* descriptors.
*/
if (func->function.ss_descriptors) {
ep_desc_id = 2;
func->function.ss_descriptors[(long)valuep] = desc;
} else if (func->function.hs_descriptors) {
ep_desc_id = 1;
func->function.hs_descriptors[(long)valuep] = desc;
} else {
ep_desc_id = 0;
func->function.fs_descriptors[(long)valuep] = desc;
}
if (!desc || desc->bDescriptorType != USB_DT_ENDPOINT)
return 0;
idx = ffs_ep_addr2idx(func->ffs, ds->bEndpointAddress) - 1;
if (idx < 0)
return idx;
ffs_ep = func->eps + idx;
if (ffs_ep->descs[ep_desc_id]) {
pr_err("two %sspeed descriptors for EP %d\n",
speed_names[ep_desc_id],
usb_endpoint_num(ds));
return -EINVAL;
}
ffs_ep->descs[ep_desc_id] = ds;
ffs_dump_mem(": Original ep desc", ds, ds->bLength);
if (ffs_ep->ep) {
ds->bEndpointAddress = ffs_ep->descs[0]->bEndpointAddress;
if (!ds->wMaxPacketSize)
ds->wMaxPacketSize = ffs_ep->descs[0]->wMaxPacketSize;
} else {
struct usb_request *req;
struct usb_ep *ep;
u8 bEndpointAddress;
u16 wMaxPacketSize;
/*
* We back up bEndpointAddress because autoconfig overwrites
* it with physical endpoint address.
*/
bEndpointAddress = ds->bEndpointAddress;
/*
* We back up wMaxPacketSize because autoconfig treats
* endpoint descriptors as if they were full speed.
*/
wMaxPacketSize = ds->wMaxPacketSize;
pr_vdebug("autoconfig\n");
ep = usb_ep_autoconfig(func->gadget, ds);
if (!ep)
return -ENOTSUPP;
ep->driver_data = func->eps + idx;
req = usb_ep_alloc_request(ep, GFP_KERNEL);
if (!req)
return -ENOMEM;
ffs_ep->ep = ep;
ffs_ep->req = req;
func->eps_revmap[ds->bEndpointAddress &
USB_ENDPOINT_NUMBER_MASK] = idx + 1;
/*
* If we use virtual address mapping, we restore
* original bEndpointAddress value.
*/
if (func->ffs->user_flags & FUNCTIONFS_VIRTUAL_ADDR)
ds->bEndpointAddress = bEndpointAddress;
/*
* Restore wMaxPacketSize which was potentially
* overwritten by autoconfig.
*/
ds->wMaxPacketSize = wMaxPacketSize;
}
ffs_dump_mem(": Rewritten ep desc", ds, ds->bLength);
return 0;
}
static int __ffs_func_bind_do_nums(enum ffs_entity_type type, u8 *valuep,
struct usb_descriptor_header *desc,
void *priv)
{
struct ffs_function *func = priv;
unsigned idx;
u8 newValue;
switch (type) {
default:
case FFS_DESCRIPTOR:
/* Handled in previous pass by __ffs_func_bind_do_descs() */
return 0;
case FFS_INTERFACE:
idx = *valuep;
if (func->interfaces_nums[idx] < 0) {
int id = usb_interface_id(func->conf, &func->function);
if (id < 0)
return id;
func->interfaces_nums[idx] = id;
}
newValue = func->interfaces_nums[idx];
break;
case FFS_STRING:
/* String' IDs are allocated when fsf_data is bound to cdev */
newValue = func->ffs->stringtabs[0]->strings[*valuep - 1].id;
break;
case FFS_ENDPOINT:
/*
* USB_DT_ENDPOINT are handled in
* __ffs_func_bind_do_descs().
*/
if (desc->bDescriptorType == USB_DT_ENDPOINT)
return 0;
idx = (*valuep & USB_ENDPOINT_NUMBER_MASK) - 1;
if (!func->eps[idx].ep)
return -EINVAL;
{
struct usb_endpoint_descriptor **descs;
descs = func->eps[idx].descs;
newValue = descs[descs[0] ? 0 : 1]->bEndpointAddress;
}
break;
}
pr_vdebug("%02x -> %02x\n", *valuep, newValue);
*valuep = newValue;
return 0;
}
static int __ffs_func_bind_do_os_desc(enum ffs_os_desc_type type,
struct usb_os_desc_header *h, void *data,
unsigned len, void *priv)
{
struct ffs_function *func = priv;
u8 length = 0;
switch (type) {
case FFS_OS_DESC_EXT_COMPAT: {
struct usb_ext_compat_desc *desc = data;
struct usb_os_desc_table *t;
t = &func->function.os_desc_table[desc->bFirstInterfaceNumber];
t->if_id = func->interfaces_nums[desc->bFirstInterfaceNumber];
memcpy(t->os_desc->ext_compat_id, &desc->IDs,
sizeof_field(struct usb_ext_compat_desc, IDs));
length = sizeof(*desc);
}
break;
case FFS_OS_DESC_EXT_PROP: {
struct usb_ext_prop_desc *desc = data;
struct usb_os_desc_table *t;
struct usb_os_desc_ext_prop *ext_prop;
char *ext_prop_name;
char *ext_prop_data;
t = &func->function.os_desc_table[h->interface];
t->if_id = func->interfaces_nums[h->interface];
ext_prop = func->ffs->ms_os_descs_ext_prop_avail;
func->ffs->ms_os_descs_ext_prop_avail += sizeof(*ext_prop);
ext_prop->type = le32_to_cpu(desc->dwPropertyDataType);
ext_prop->name_len = le16_to_cpu(desc->wPropertyNameLength);
ext_prop->data_len = le32_to_cpu(*(__le32 *)
usb_ext_prop_data_len_ptr(data, ext_prop->name_len));
length = ext_prop->name_len + ext_prop->data_len + 14;
ext_prop_name = func->ffs->ms_os_descs_ext_prop_name_avail;
func->ffs->ms_os_descs_ext_prop_name_avail +=
ext_prop->name_len;
ext_prop_data = func->ffs->ms_os_descs_ext_prop_data_avail;
func->ffs->ms_os_descs_ext_prop_data_avail +=
ext_prop->data_len;
memcpy(ext_prop_data,
usb_ext_prop_data_ptr(data, ext_prop->name_len),
ext_prop->data_len);
/* unicode data reported to the host as "WCHAR"s */
switch (ext_prop->type) {
case USB_EXT_PROP_UNICODE:
case USB_EXT_PROP_UNICODE_ENV:
case USB_EXT_PROP_UNICODE_LINK:
case USB_EXT_PROP_UNICODE_MULTI:
ext_prop->data_len *= 2;
break;
}
ext_prop->data = ext_prop_data;
memcpy(ext_prop_name, usb_ext_prop_name_ptr(data),
ext_prop->name_len);
/* property name reported to the host as "WCHAR"s */
ext_prop->name_len *= 2;
ext_prop->name = ext_prop_name;
t->os_desc->ext_prop_len +=
ext_prop->name_len + ext_prop->data_len + 14;
++t->os_desc->ext_prop_count;
list_add_tail(&ext_prop->entry, &t->os_desc->ext_prop);
}
break;
default:
pr_vdebug("unknown descriptor: %d\n", type);
}
return length;
}
static inline struct f_fs_opts *ffs_do_functionfs_bind(struct usb_function *f,
struct usb_configuration *c)
{
struct ffs_function *func = ffs_func_from_usb(f);
struct f_fs_opts *ffs_opts =
container_of(f->fi, struct f_fs_opts, func_inst);
struct ffs_data *ffs_data;
int ret;
/*
* Legacy gadget triggers binding in functionfs_ready_callback,
* which already uses locking; taking the same lock here would
* cause a deadlock.
*
* Configfs-enabled gadgets however do need ffs_dev_lock.
*/
if (!ffs_opts->no_configfs)
ffs_dev_lock();
ret = ffs_opts->dev->desc_ready ? 0 : -ENODEV;
ffs_data = ffs_opts->dev->ffs_data;
if (!ffs_opts->no_configfs)
ffs_dev_unlock();
if (ret)
return ERR_PTR(ret);
func->ffs = ffs_data;
func->conf = c;
func->gadget = c->cdev->gadget;
/*
* in drivers/usb/gadget/configfs.c:configfs_composite_bind()
* configurations are bound in sequence with list_for_each_entry,
* in each configuration its functions are bound in sequence
* with list_for_each_entry, so we assume no race condition
* with regard to ffs_opts->bound access
*/
if (!ffs_opts->refcnt) {
ret = functionfs_bind(func->ffs, c->cdev);
if (ret)
return ERR_PTR(ret);
}
ffs_opts->refcnt++;
func->function.strings = func->ffs->stringtabs;
return ffs_opts;
}
static int _ffs_func_bind(struct usb_configuration *c,
struct usb_function *f)
{
struct ffs_function *func = ffs_func_from_usb(f);
struct ffs_data *ffs = func->ffs;
const int full = !!func->ffs->fs_descs_count;
const int high = !!func->ffs->hs_descs_count;
const int super = !!func->ffs->ss_descs_count;
int fs_len, hs_len, ss_len, ret, i;
struct ffs_ep *eps_ptr;
/* Make it a single chunk, less management later on */
vla_group(d);
vla_item_with_sz(d, struct ffs_ep, eps, ffs->eps_count);
vla_item_with_sz(d, struct usb_descriptor_header *, fs_descs,
full ? ffs->fs_descs_count + 1 : 0);
vla_item_with_sz(d, struct usb_descriptor_header *, hs_descs,
high ? ffs->hs_descs_count + 1 : 0);
vla_item_with_sz(d, struct usb_descriptor_header *, ss_descs,
super ? ffs->ss_descs_count + 1 : 0);
vla_item_with_sz(d, short, inums, ffs->interfaces_count);
vla_item_with_sz(d, struct usb_os_desc_table, os_desc_table,
c->cdev->use_os_string ? ffs->interfaces_count : 0);
vla_item_with_sz(d, char[16], ext_compat,
c->cdev->use_os_string ? ffs->interfaces_count : 0);
vla_item_with_sz(d, struct usb_os_desc, os_desc,
c->cdev->use_os_string ? ffs->interfaces_count : 0);
vla_item_with_sz(d, struct usb_os_desc_ext_prop, ext_prop,
ffs->ms_os_descs_ext_prop_count);
vla_item_with_sz(d, char, ext_prop_name,
ffs->ms_os_descs_ext_prop_name_len);
vla_item_with_sz(d, char, ext_prop_data,
ffs->ms_os_descs_ext_prop_data_len);
vla_item_with_sz(d, char, raw_descs, ffs->raw_descs_length);
char *vlabuf;
/* Has descriptors only for speeds gadget does not support */
if (!(full | high | super))
return -ENOTSUPP;
/* Allocate a single chunk, less management later on */
vlabuf = kzalloc(vla_group_size(d), GFP_KERNEL);
if (!vlabuf)
return -ENOMEM;
ffs->ms_os_descs_ext_prop_avail = vla_ptr(vlabuf, d, ext_prop);
ffs->ms_os_descs_ext_prop_name_avail =
vla_ptr(vlabuf, d, ext_prop_name);
ffs->ms_os_descs_ext_prop_data_avail =
vla_ptr(vlabuf, d, ext_prop_data);
/* Copy descriptors */
memcpy(vla_ptr(vlabuf, d, raw_descs), ffs->raw_descs,
ffs->raw_descs_length);
memset(vla_ptr(vlabuf, d, inums), 0xff, d_inums__sz);
eps_ptr = vla_ptr(vlabuf, d, eps);
for (i = 0; i < ffs->eps_count; i++)
eps_ptr[i].num = -1;
/* Save pointers
* d_eps == vlabuf, func->eps used to kfree vlabuf later
*/
func->eps = vla_ptr(vlabuf, d, eps);
func->interfaces_nums = vla_ptr(vlabuf, d, inums);
/*
* Go through all the endpoint descriptors and allocate
* endpoints first, so that later we can rewrite the endpoint
* numbers without worrying that it may be described later on.
*/
if (full) {
func->function.fs_descriptors = vla_ptr(vlabuf, d, fs_descs);
fs_len = ffs_do_descs(ffs->fs_descs_count,
vla_ptr(vlabuf, d, raw_descs),
d_raw_descs__sz,
__ffs_func_bind_do_descs, func);
if (fs_len < 0) {
ret = fs_len;
goto error;
}
} else {
fs_len = 0;
}
if (high) {
func->function.hs_descriptors = vla_ptr(vlabuf, d, hs_descs);
hs_len = ffs_do_descs(ffs->hs_descs_count,
vla_ptr(vlabuf, d, raw_descs) + fs_len,
d_raw_descs__sz - fs_len,
__ffs_func_bind_do_descs, func);
if (hs_len < 0) {
ret = hs_len;
goto error;
}
} else {
hs_len = 0;
}
if (super) {
func->function.ss_descriptors = func->function.ssp_descriptors =
vla_ptr(vlabuf, d, ss_descs);
ss_len = ffs_do_descs(ffs->ss_descs_count,
vla_ptr(vlabuf, d, raw_descs) + fs_len + hs_len,
d_raw_descs__sz - fs_len - hs_len,
__ffs_func_bind_do_descs, func);
if (ss_len < 0) {
ret = ss_len;
goto error;
}
} else {
ss_len = 0;
}
/*
* Now handle interface numbers allocation and interface and
* endpoint numbers rewriting. We can do that in one go
* now.
*/
ret = ffs_do_descs(ffs->fs_descs_count +
(high ? ffs->hs_descs_count : 0) +
(super ? ffs->ss_descs_count : 0),
vla_ptr(vlabuf, d, raw_descs), d_raw_descs__sz,
__ffs_func_bind_do_nums, func);
if (ret < 0)
goto error;
func->function.os_desc_table = vla_ptr(vlabuf, d, os_desc_table);
if (c->cdev->use_os_string) {
for (i = 0; i < ffs->interfaces_count; ++i) {
struct usb_os_desc *desc;
desc = func->function.os_desc_table[i].os_desc =
vla_ptr(vlabuf, d, os_desc) +
i * sizeof(struct usb_os_desc);
desc->ext_compat_id =
vla_ptr(vlabuf, d, ext_compat) + i * 16;
INIT_LIST_HEAD(&desc->ext_prop);
}
ret = ffs_do_os_descs(ffs->ms_os_descs_count,
vla_ptr(vlabuf, d, raw_descs) +
fs_len + hs_len + ss_len,
d_raw_descs__sz - fs_len - hs_len -
ss_len,
__ffs_func_bind_do_os_desc, func);
if (ret < 0)
goto error;
}
func->function.os_desc_n =
c->cdev->use_os_string ? ffs->interfaces_count : 0;
/* And we're done */
ffs_event_add(ffs, FUNCTIONFS_BIND);
return 0;
error:
/* XXX Do we need to release all claimed endpoints here? */
return ret;
}
static int ffs_func_bind(struct usb_configuration *c,
struct usb_function *f)
{
struct f_fs_opts *ffs_opts = ffs_do_functionfs_bind(f, c);
struct ffs_function *func = ffs_func_from_usb(f);
int ret;
if (IS_ERR(ffs_opts))
return PTR_ERR(ffs_opts);
ret = _ffs_func_bind(c, f);
if (ret && !--ffs_opts->refcnt)
functionfs_unbind(func->ffs);
return ret;
}
/* Other USB function hooks *************************************************/
static void ffs_reset_work(struct work_struct *work)
{
struct ffs_data *ffs = container_of(work,
struct ffs_data, reset_work);
ffs_data_reset(ffs);
}
static int ffs_func_get_alt(struct usb_function *f,
unsigned int interface)
{
struct ffs_function *func = ffs_func_from_usb(f);
int intf = ffs_func_revmap_intf(func, interface);
return (intf < 0) ? intf : func->cur_alt[interface];
}
static int ffs_func_set_alt(struct usb_function *f,
unsigned interface, unsigned alt)
{
struct ffs_function *func = ffs_func_from_usb(f);
struct ffs_data *ffs = func->ffs;
unsigned long flags;
int ret = 0, intf;
if (alt > MAX_ALT_SETTINGS)
return -EINVAL;
intf = ffs_func_revmap_intf(func, interface);
if (intf < 0)
return intf;
if (ffs->func)
ffs_func_eps_disable(ffs->func);
spin_lock_irqsave(&ffs->eps_lock, flags);
if (ffs->state == FFS_DEACTIVATED) {
ffs->state = FFS_CLOSING;
spin_unlock_irqrestore(&ffs->eps_lock, flags);
INIT_WORK(&ffs->reset_work, ffs_reset_work);
schedule_work(&ffs->reset_work);
return -ENODEV;
}
spin_unlock_irqrestore(&ffs->eps_lock, flags);
if (ffs->state != FFS_ACTIVE)
return -ENODEV;
ffs->func = func;
ret = ffs_func_eps_enable(func);
if (ret >= 0) {
ffs_event_add(ffs, FUNCTIONFS_ENABLE);
func->cur_alt[interface] = alt;
}
return ret;
}
static void ffs_func_disable(struct usb_function *f)
{
struct ffs_function *func = ffs_func_from_usb(f);
struct ffs_data *ffs = func->ffs;
unsigned long flags;
if (ffs->func)
ffs_func_eps_disable(ffs->func);
spin_lock_irqsave(&ffs->eps_lock, flags);
if (ffs->state == FFS_DEACTIVATED) {
ffs->state = FFS_CLOSING;
spin_unlock_irqrestore(&ffs->eps_lock, flags);
INIT_WORK(&ffs->reset_work, ffs_reset_work);
schedule_work(&ffs->reset_work);
return;
}
spin_unlock_irqrestore(&ffs->eps_lock, flags);
if (ffs->state == FFS_ACTIVE) {
ffs->func = NULL;
ffs_event_add(ffs, FUNCTIONFS_DISABLE);
}
}
static int ffs_func_setup(struct usb_function *f,
const struct usb_ctrlrequest *creq)
{
struct ffs_function *func = ffs_func_from_usb(f);
struct ffs_data *ffs = func->ffs;
unsigned long flags;
int ret;
pr_vdebug("creq->bRequestType = %02x\n", creq->bRequestType);
pr_vdebug("creq->bRequest = %02x\n", creq->bRequest);
pr_vdebug("creq->wValue = %04x\n", le16_to_cpu(creq->wValue));
pr_vdebug("creq->wIndex = %04x\n", le16_to_cpu(creq->wIndex));
pr_vdebug("creq->wLength = %04x\n", le16_to_cpu(creq->wLength));
/*
* Most requests directed to interface go through here
* (notable exceptions are set/get interface) so we need to
* handle them. All other either handled by composite or
* passed to usb_configuration->setup() (if one is set). No
* matter, we will handle requests directed to endpoint here
* as well (as it's straightforward). Other request recipient
* types are only handled when the user flag FUNCTIONFS_ALL_CTRL_RECIP
* is being used.
*/
if (ffs->state != FFS_ACTIVE)
return -ENODEV;
switch (creq->bRequestType & USB_RECIP_MASK) {
case USB_RECIP_INTERFACE:
ret = ffs_func_revmap_intf(func, le16_to_cpu(creq->wIndex));
if (ret < 0)
return ret;
break;
case USB_RECIP_ENDPOINT:
ret = ffs_func_revmap_ep(func, le16_to_cpu(creq->wIndex));
if (ret < 0)
return ret;
if (func->ffs->user_flags & FUNCTIONFS_VIRTUAL_ADDR)
ret = func->ffs->eps_addrmap[ret];
break;
default:
if (func->ffs->user_flags & FUNCTIONFS_ALL_CTRL_RECIP)
ret = le16_to_cpu(creq->wIndex);
else
return -EOPNOTSUPP;
}
spin_lock_irqsave(&ffs->ev.waitq.lock, flags);
ffs->ev.setup = *creq;
ffs->ev.setup.wIndex = cpu_to_le16(ret);
__ffs_event_add(ffs, FUNCTIONFS_SETUP);
spin_unlock_irqrestore(&ffs->ev.waitq.lock, flags);
return ffs->ev.setup.wLength == 0 ? USB_GADGET_DELAYED_STATUS : 0;
}
static bool ffs_func_req_match(struct usb_function *f,
const struct usb_ctrlrequest *creq,
bool config0)
{
struct ffs_function *func = ffs_func_from_usb(f);
if (config0 && !(func->ffs->user_flags & FUNCTIONFS_CONFIG0_SETUP))
return false;
switch (creq->bRequestType & USB_RECIP_MASK) {
case USB_RECIP_INTERFACE:
return (ffs_func_revmap_intf(func,
le16_to_cpu(creq->wIndex)) >= 0);
case USB_RECIP_ENDPOINT:
return (ffs_func_revmap_ep(func,
le16_to_cpu(creq->wIndex)) >= 0);
default:
return (bool) (func->ffs->user_flags &
FUNCTIONFS_ALL_CTRL_RECIP);
}
}
static void ffs_func_suspend(struct usb_function *f)
{
ffs_event_add(ffs_func_from_usb(f)->ffs, FUNCTIONFS_SUSPEND);
}
static void ffs_func_resume(struct usb_function *f)
{
ffs_event_add(ffs_func_from_usb(f)->ffs, FUNCTIONFS_RESUME);
}
/* Endpoint and interface numbers reverse mapping ***************************/
static int ffs_func_revmap_ep(struct ffs_function *func, u8 num)
{
num = func->eps_revmap[num & USB_ENDPOINT_NUMBER_MASK];
return num ? num : -EDOM;
}
static int ffs_func_revmap_intf(struct ffs_function *func, u8 intf)
{
short *nums = func->interfaces_nums;
unsigned count = func->ffs->interfaces_count;
for (; count; --count, ++nums) {
if (*nums >= 0 && *nums == intf)
return nums - func->interfaces_nums;
}
return -EDOM;
}
/* Devices management *******************************************************/
static LIST_HEAD(ffs_devices);
static struct ffs_dev *_ffs_do_find_dev(const char *name)
{
struct ffs_dev *dev;
if (!name)
return NULL;
list_for_each_entry(dev, &ffs_devices, entry) {
if (strcmp(dev->name, name) == 0)
return dev;
}
return NULL;
}
/*
* ffs_lock must be taken by the caller of this function
*/
static struct ffs_dev *_ffs_get_single_dev(void)
{
struct ffs_dev *dev;
if (list_is_singular(&ffs_devices)) {
dev = list_first_entry(&ffs_devices, struct ffs_dev, entry);
if (dev->single)
return dev;
}
return NULL;
}
/*
* ffs_lock must be taken by the caller of this function
*/
static struct ffs_dev *_ffs_find_dev(const char *name)
{
struct ffs_dev *dev;
dev = _ffs_get_single_dev();
if (dev)
return dev;
return _ffs_do_find_dev(name);
}
/* Configfs support *********************************************************/
static inline struct f_fs_opts *to_ffs_opts(struct config_item *item)
{
return container_of(to_config_group(item), struct f_fs_opts,
func_inst.group);
}
static ssize_t f_fs_opts_ready_show(struct config_item *item, char *page)
{
struct f_fs_opts *opts = to_ffs_opts(item);
int ready;
ffs_dev_lock();
ready = opts->dev->desc_ready;
ffs_dev_unlock();
return sprintf(page, "%d\n", ready);
}
CONFIGFS_ATTR_RO(f_fs_opts_, ready);
static struct configfs_attribute *ffs_attrs[] = {
&f_fs_opts_attr_ready,
NULL,
};
static void ffs_attr_release(struct config_item *item)
{
struct f_fs_opts *opts = to_ffs_opts(item);
usb_put_function_instance(&opts->func_inst);
}
static const struct configfs_item_operations ffs_item_ops = {
.release = ffs_attr_release,
};
static const struct config_item_type ffs_func_type = {
.ct_item_ops = &ffs_item_ops,
.ct_attrs = ffs_attrs,
.ct_owner = THIS_MODULE,
};
/* Function registration interface ******************************************/
static void ffs_free_inst(struct usb_function_instance *f)
{
struct f_fs_opts *opts;
opts = to_f_fs_opts(f);
ffs_release_dev(opts->dev);
ffs_dev_lock();
_ffs_free_dev(opts->dev);
ffs_dev_unlock();
kfree(opts);
}
static int ffs_set_inst_name(struct usb_function_instance *fi, const char *name)
{
if (strlen(name) >= sizeof_field(struct ffs_dev, name))
return -ENAMETOOLONG;
return ffs_name_dev(to_f_fs_opts(fi)->dev, name);
}
static struct usb_function_instance *ffs_alloc_inst(void)
{
struct f_fs_opts *opts;
struct ffs_dev *dev;
opts = kzalloc_obj(*opts);
if (!opts)
return ERR_PTR(-ENOMEM);
opts->func_inst.set_inst_name = ffs_set_inst_name;
opts->func_inst.free_func_inst = ffs_free_inst;
ffs_dev_lock();
dev = _ffs_alloc_dev();
ffs_dev_unlock();
if (IS_ERR(dev)) {
kfree(opts);
return ERR_CAST(dev);
}
opts->dev = dev;
dev->opts = opts;
config_group_init_type_name(&opts->func_inst.group, "",
&ffs_func_type);
return &opts->func_inst;
}
static void ffs_free(struct usb_function *f)
{
kfree(ffs_func_from_usb(f));
}
static void ffs_func_unbind(struct usb_configuration *c,
struct usb_function *f)
{
struct ffs_function *func = ffs_func_from_usb(f);
struct ffs_data *ffs = func->ffs;
struct f_fs_opts *opts =
container_of(f->fi, struct f_fs_opts, func_inst);
struct ffs_ep *ep = func->eps;
unsigned count = ffs->eps_count;
unsigned long flags;
if (ffs->func == func) {
ffs_func_eps_disable(func);
ffs->func = NULL;
}
/* Drain any pending AIO completions */
drain_workqueue(ffs->io_completion_wq);
ffs_event_add(ffs, FUNCTIONFS_UNBIND);
if (!--opts->refcnt)
functionfs_unbind(ffs);
/* cleanup after autoconfig */
spin_lock_irqsave(&func->ffs->eps_lock, flags);
while (count--) {
if (ep->ep && ep->req)
usb_ep_free_request(ep->ep, ep->req);
ep->req = NULL;
++ep;
}
spin_unlock_irqrestore(&func->ffs->eps_lock, flags);
kfree(func->eps);
func->eps = NULL;
/*
* eps, descriptors and interfaces_nums are allocated in the
* same chunk so only one free is required.
*/
func->function.fs_descriptors = NULL;
func->function.hs_descriptors = NULL;
func->function.ss_descriptors = NULL;
func->function.ssp_descriptors = NULL;
func->interfaces_nums = NULL;
}
static struct usb_function *ffs_alloc(struct usb_function_instance *fi)
{
struct ffs_function *func;
func = kzalloc_obj(*func);
if (!func)
return ERR_PTR(-ENOMEM);
func->function.name = "Function FS Gadget";
func->function.bind = ffs_func_bind;
func->function.unbind = ffs_func_unbind;
func->function.set_alt = ffs_func_set_alt;
func->function.get_alt = ffs_func_get_alt;
func->function.disable = ffs_func_disable;
func->function.setup = ffs_func_setup;
func->function.req_match = ffs_func_req_match;
func->function.suspend = ffs_func_suspend;
func->function.resume = ffs_func_resume;
func->function.free_func = ffs_free;
return &func->function;
}
/*
* ffs_lock must be taken by the caller of this function
*/
static struct ffs_dev *_ffs_alloc_dev(void)
{
struct ffs_dev *dev;
int ret;
if (_ffs_get_single_dev())
return ERR_PTR(-EBUSY);
dev = kzalloc_obj(*dev);
if (!dev)
return ERR_PTR(-ENOMEM);
if (list_empty(&ffs_devices)) {
ret = functionfs_init();
if (ret) {
kfree(dev);
return ERR_PTR(ret);
}
}
list_add(&dev->entry, &ffs_devices);
return dev;
}
int ffs_name_dev(struct ffs_dev *dev, const char *name)
{
struct ffs_dev *existing;
int ret = 0;
ffs_dev_lock();
existing = _ffs_do_find_dev(name);
if (!existing)
strscpy(dev->name, name, ARRAY_SIZE(dev->name));
else if (existing != dev)
ret = -EBUSY;
ffs_dev_unlock();
return ret;
}
EXPORT_SYMBOL_GPL(ffs_name_dev);
int ffs_single_dev(struct ffs_dev *dev)
{
int ret;
ret = 0;
ffs_dev_lock();
if (!list_is_singular(&ffs_devices))
ret = -EBUSY;
else
dev->single = true;
ffs_dev_unlock();
return ret;
}
EXPORT_SYMBOL_GPL(ffs_single_dev);
/*
* ffs_lock must be taken by the caller of this function
*/
static void _ffs_free_dev(struct ffs_dev *dev)
{
list_del(&dev->entry);
kfree(dev);
if (list_empty(&ffs_devices))
functionfs_cleanup();
}
static int ffs_acquire_dev(const char *dev_name, struct ffs_data *ffs_data)
{
int ret = 0;
struct ffs_dev *ffs_dev;
ffs_dev_lock();
ffs_dev = _ffs_find_dev(dev_name);
if (!ffs_dev) {
ret = -ENOENT;
} else if (ffs_dev->mounted) {
ret = -EBUSY;
} else if (ffs_dev->ffs_acquire_dev_callback &&
ffs_dev->ffs_acquire_dev_callback(ffs_dev)) {
ret = -ENOENT;
} else {
ffs_dev->mounted = true;
ffs_dev->ffs_data = ffs_data;
ffs_data->private_data = ffs_dev;
}
ffs_dev_unlock();
return ret;
}
static void ffs_release_dev(struct ffs_dev *ffs_dev)
{
ffs_dev_lock();
if (ffs_dev && ffs_dev->mounted) {
ffs_dev->mounted = false;
if (ffs_dev->ffs_data) {
ffs_dev->ffs_data->private_data = NULL;
ffs_dev->ffs_data = NULL;
}
if (ffs_dev->ffs_release_dev_callback)
ffs_dev->ffs_release_dev_callback(ffs_dev);
}
ffs_dev_unlock();
}
static int ffs_ready(struct ffs_data *ffs)
{
struct ffs_dev *ffs_obj;
int ret = 0;
ffs_dev_lock();
ffs_obj = ffs->private_data;
if (!ffs_obj) {
ret = -EINVAL;
goto done;
}
if (WARN_ON(ffs_obj->desc_ready)) {
ret = -EBUSY;
goto done;
}
ffs_obj->desc_ready = true;
if (ffs_obj->ffs_ready_callback) {
ret = ffs_obj->ffs_ready_callback(ffs);
if (ret)
goto done;
}
set_bit(FFS_FL_CALL_CLOSED_CALLBACK, &ffs->flags);
done:
ffs_dev_unlock();
return ret;
}
static void ffs_closed(struct ffs_data *ffs)
{
struct ffs_dev *ffs_obj;
struct f_fs_opts *opts;
struct config_item *ci;
ffs_dev_lock();
ffs_obj = ffs->private_data;
if (!ffs_obj)
goto done;
ffs_obj->desc_ready = false;
if (test_and_clear_bit(FFS_FL_CALL_CLOSED_CALLBACK, &ffs->flags) &&
ffs_obj->ffs_closed_callback)
ffs_obj->ffs_closed_callback(ffs);
if (ffs_obj->opts)
opts = ffs_obj->opts;
else
goto done;
if (opts->no_configfs || !opts->func_inst.group.cg_item.ci_parent
|| !kref_read(&opts->func_inst.group.cg_item.ci_kref))
goto done;
ci = opts->func_inst.group.cg_item.ci_parent->ci_parent;
ffs_dev_unlock();
if (test_bit(FFS_FL_BOUND, &ffs->flags))
unregister_gadget_item(ci);
return;
done:
ffs_dev_unlock();
}
/* Misc helper functions ****************************************************/
static int ffs_mutex_lock(struct mutex *mutex, unsigned nonblock)
{
return nonblock
? mutex_trylock(mutex) ? 0 : -EAGAIN
: mutex_lock_interruptible(mutex);
}
static char *ffs_prepare_buffer(const char __user *buf, size_t len)
{
char *data;
if (!len)
return NULL;
data = memdup_user(buf, len);
if (IS_ERR(data))
return data;
pr_vdebug("Buffer from user space:\n");
ffs_dump_mem("", data, len);
return data;
}
DECLARE_USB_FUNCTION_INIT(ffs, ffs_alloc_inst, ffs_alloc);
MODULE_DESCRIPTION("user mode file system API for USB composite function controllers");
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Michal Nazarewicz");
]
|
| 253/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 85322
output: 20
thoughts: 1176
Thoughts: **Analyzing the Unsigned Comparison in C Pointer Arithmetic**
Okay, so the question is about whether the comparison `tmp_buf_end - tmp_buf < sizeof_cur_arg` is unsigned in C. First things first, I couldn't find this specific expression directly in `drivers/usb/gadget/function/f_fs.c`, but that's not the point. We need to dissect the C semantics.
`tmp_buf_end` and `tmp_buf` are pointers, so their difference results in a `ptrdiff_t` (signed), and `sizeof_cur_arg` is a `size_t` (unsigned). My immediate thought is, based on C rules, we're dealing with a comparison between a signed and an unsigned integer of the same rank. In the kernel, both `ptrdiff_t` and `size_t` have the same rank (either 32-bit or 64-bit). The C standard dictates that when comparing these types, the signed integer (`ptrdiff_t`) is *implicitly* converted to the unsigned type (`size_t`) for the comparison.
However, I need to be thorough. I need to make sure I am not missing some obscure edge case. So, I checked the standard again. And verified how `ptrdiff_t` and `size_t` are defined in the kernel's `include/uapi/asm-generic/posix_types.h`. Even in the 32-bit case (`__BITS_PER_LONG != 64`), `ptrdiff_t` and `size_t` are defined as `int` and `unsigned int`, so they are still of the same rank. The same applies for 64-bit systems.
The standard says that if the operand with the unsigned type has a greater rank, the signed type is converted. Otherwise, it is converted to the signed type. In this case, `ptrdiff_t` and `size_t` always have the same rank, and we're dealing with the `ptrdiff_t` being implicitly converted to `size_t`. So, there are no special cases in the kernel, at least related to the definitions.
The question explicitly states that the expressions use pointers and `size_t`, so I am certain the comparison *is* unsigned. The classic C pitfall! If `tmp_buf_end - tmp_buf` evaluates to a negative number, it will be converted to a very large unsigned value, causing unexpected behavior. For example, if the result is -1, the expression becomes `(size_t)-1 < 10` (if `sizeof_cur_arg` is 10), which is false in an unsigned comparison. In a signed comparison `-1 < 10` would be true.
I remembered a similar type of problem might be related to a buffer overflow issue from the past, so I went searching for it.
I searched `tmp_buf_end - tmp_buf` but didn't come up with anything related to this *specific* code. I did remember `audit_log_execve_info` from an earlier search. The comparison in that function `(len_abuf + sizeof(abuf)) > len_rem` *does* involve mixed signed/unsigned types (`long`, `size_t`) and it's also an unsigned comparison. Although that is not the same expression, it highlights a related pattern in the kernel. In the end, it is an **unsigned comparison**, due to C's implicit type conversion rules when comparing a signed integer (`ptrdiff_t`) to an unsigned integer (`size_t`) of the same rank.
|
| 254/4 |
2026/05/18 19:07 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:[commoncap.c commoncap_test.c device_cgroup.c inode.c lsm.h lsm_audit.c lsm_init.c lsm_notifier.c lsm_syscalls.c min_addr.c security.c] Subdirs:[apparmor bpf integrity ipe keys landlock loadpin lockdown safesetid selinux smack tomoyo yama]]
|
| 255/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 87479
output: 27
thoughts: 14
Thoughts: Okay, here's my summary, taking the role of the expert:
**Diving into `security/lsm_audit.c`**
Alright, time to dig in. Let's start with `security/lsm_audit.c`. I need to refresh my memory on the audit logging within the LSM framework. This file is probably the core of how the Linux Security Modules (LSM) handle audit events, specifically for the logging and reporting side of things. I'm expecting to find the key functions responsible for formatting audit records, potentially interfacing with the audit subsystem, and managing any related buffers or data structures. I'll be looking for where hooks are called, how events are classified, and how much context is included. Probably lots of interesting stuff regarding security event context as it pertains to the various LSM's and overall system audit policies. It's a foundational piece, so understanding this is vital for deep-level security analysis and auditing. Let's see what we've got!
|
| 256/4 |
2026/05/18 19:07 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:security/lsm_audit.c]
Results: map[Entities:[map[Kind:function Name:audit_log_lsm_data] map[Kind:function Name:common_lsm_audit] map[Kind:function Name:dump_common_audit_data] map[Kind:function Name:ipv4_skb_to_auditdata] map[Kind:function Name:ipv6_skb_to_auditdata] map[Kind:function Name:print_ipv4_addr] map[Kind:function Name:print_ipv6_addr]]]
|
| 257/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 87873
output: 27
thoughts: 17
Thoughts: Okay, here's my attempt at summarizing those thoughts from a first-person, expert perspective:
**Initial Assessment: A Disjointed Search**
Ugh, I'm drawing a blank. Not quite feeling it today. The problem I'm working on, the relevant code... it's just not coming to mind immediately. I'm feeling disconnected from the relevant parts of the codebase. My usual mental map isn't quite clicking.
Wait... a flicker... maybe? Now, where would that kind of functionality *typically* live in a Linux kernel? The kernel module for miscellaneous binary formats, that's it! Let's try `fs/binfmt_misc.c`. That's likely the right area for exploring this.
|
| 258/4 |
2026/05/18 19:07 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:fs/binfmt_misc.c]
Results: map[Entities:[map[Kind:function Name:add_entry] map[Kind:function Name:bm_entry_read] map[Kind:function Name:bm_entry_write] map[Kind:function Name:bm_evict_inode] map[Kind:function Name:bm_fill_super] map[Kind:function Name:bm_free] map[Kind:function Name:bm_get_inode] map[Kind:function Name:bm_get_tree] map[Kind:function Name:bm_init_fs_context] map[Kind:function Name:bm_put_super] map[Kind:function Name:bm_register_write] map[Kind:function Name:bm_status_read] map[Kind:function Name:bm_status_write] map[Kind:function Name:check_special_flags] map[Kind:function Name:create_entry] map[Kind:function Name:entry_status] map[Kind:function Name:exit_misc_binfmt] map[Kind:function Name:get_binfmt_handler] map[Kind:function Name:i_binfmt_misc] map[Kind:function Name:init_misc_binfmt] map[Kind:function Name:load_binfmt_misc] map[Kind:function Name:load_misc_binary] map[Kind:function Name:parse_command] map[Kind:function Name:put_binfmt_handler] map[Kind:function Name:remove_binfmt_handler] map[Kind:function Name:scanarg] map[Kind:function Name:search_binfmt_handler] map[Kind:global_variable Name:__UNIQUE_ID_addressable_init_misc_binfmt_568] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_566] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_569] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_570] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_571] map[Kind:global_variable Name:__exitcall_exit_misc_binfmt] map[Kind:global_variable Name:bm_context_ops] map[Kind:global_variable Name:bm_entry_operations] map[Kind:global_variable Name:bm_fs_type] map[Kind:global_variable Name:bm_register_operations] map[Kind:global_variable Name:bm_status_operations] map[Kind:global_variable Name:misc_format] map[Kind:global_variable Name:s_ops] map[Kind:typedef Name:Node]]]
|
| 259/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 88802
output: 39
thoughts: 19
Thoughts: Here's my take on those thoughts:
**Analyzing `scanarg` in `fs/binfmt_misc.c`**
Alright, let's dive into this. My first thought is to zero in on `scanarg` within `fs/binfmt_misc.c`. I need to understand exactly how this function is handling arguments. I suspect it's critical to the binary format registration process, likely involved in parsing and validating data provided during the registration of a new interpreter or binary format. My immediate goal is to figure out the exact mechanisms this function uses to accomplish that. I need to understand any vulnerabilities, potential gotchas, or unexpected behaviors in the argument parsing. It would be good to understand any edge cases the author accounted for and any they didn't. I'm especially interested in how it deals with malformed arguments, overflows, or injection possibilities. I'll need to look at its use within the broader context of the `binfmt_misc` module to fully grasp its impact and significance.
|
| 260/4 |
2026/05/18 19:07 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:fs/binfmt_misc.c IncludeLines:true Name:scanarg]
Results: map[SourceCode: 283: /*
284: * parses and copies one argument enclosed in del from *sp to *dp,
285: * recognising the \x special.
286: * returns pointer to the copied argument or NULL in case of an
287: * error (and sets err) or null argument length.
288: */
289: static char *scanarg(char *s, char del)
290: {
291: char c;
292:
293: while ((c = *s++) != del) {
294: if (c == '\\' && *s == 'x') {
295: s++;
296: if (!isxdigit(*s++))
297: return NULL;
298: if (!isxdigit(*s++))
299: return NULL;
300: }
301: }
302: s[-1] ='\0';
303: return s;
304: }
SourceFile:fs/binfmt_misc.c]
|
| 261/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 89389
output: 21
thoughts: 523
Thoughts: **Debugging a Kernel Memory Issue**
Okay, so I'm on the trail of something... I was expecting to find `tmp_buf` in `fs/proc/base.c` and then maybe `tmp_buf_end`, but it seems like this isn't quite where the problem lies. I'll search for "tmp_buf" everywhere in the kernel source, but since I can't do that now, I'm going to have to make some educated guesses.
Aha! There it is, I think. `tmp_buf` *was* in `fs/proc/base.c` in some versions, so maybe it has a different name here. Actually, let's revisit `get_mm_cmdline`. Looking at this line:
```c
347: if (count > arg_end - pos)
348: count = arg_end - pos;
```
It looks like we've got an unsigned comparison. `count` is `size_t`, and `arg_end` and `pos` are `unsigned long`. But the expression I'm looking at has something like `tmp_buf_end - tmp_buf`, which looks suspiciously like a patch for a vulnerability.
I'll search for `tmp_buf_end` again, let's try `kernel/auditsc.c`. Wait, let's go back and carefully re-read the `audit_log_execve_info` function.
Hmm, I see `buf_head` and `buf` here:
```c
1150: buf_head = kmalloc(MAX_EXECVE_AUDIT_LEN + 1, GFP_KERNEL);
...
1155: buf = buf_head;
```
And then later, `buf += len_tmp;` but still no `tmp_buf_end`.
Okay, what about `fs/exec.c`?
```c
475: pos = bprm->p;
476: str += len;
477: bprm->p -= len;
```
That's not it.
Let's try a broader search for `sizeof_cur_arg`, or perhaps just `sizeof_arg`. Actually, I'll broaden the search further and look for the string `cur_arg` itself.
|
| 262/4 |
2026/05/18 19:07 |
tool |
codesearch-find-references |
0m
Error:
Args:
Results: map[References:<nil> TruncatedOutput:false]
|
| 263/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 90475
output: 24
thoughts: 788
Thoughts: **Analyzing the C Code Snippet for Potential Integer Overflow Vulnerability**
Alright, let's break down this C comparison, assuming it's related to some auditing or logging mechanism in the kernel, and the user likely needs it to be a correct analysis of potential vulnerabilities. The expression is `tmp_buf_end - tmp_buf < sizeof_cur_arg`. I immediately recognize that `tmp_buf_end` and `tmp_buf` are pointers, so their difference, `tmp_buf_end - tmp_buf`, is going to be a `ptrdiff_t`. Meanwhile, `sizeof_cur_arg` is of type `size_t`. This is where it gets interesting...
My understanding of C's type conversion rules kicks in. Whenever you compare a signed integer like `ptrdiff_t` to an unsigned integer like `size_t`, the compiler performs the usual arithmetic conversions. Crucially, when both types have the same rank – which is the case for `ptrdiff_t` and `size_t` on Linux (both are "pointer-sized" integers, 32-bit or 64-bit depending on architecture) – the signed type (`ptrdiff_t`) is *converted* to the unsigned type (`size_t`).
So the comparison boils down to an *unsigned* comparison. The underlying question that comes to mind is: Are there any architectures where `ptrdiff_t` and `size_t` *don't* have the same rank? Though the C standard makes this very unlikely, since they are closely tied to pointer sizes, it's worth a second. Even if they had different ranks, the conversion rules would still be, in essence, to either promote the signed type to the unsigned, or to convert both to the corresponding unsigned type.
Let's look at the practice in Linux. On x86_64, `size_t` is `unsigned long` (64-bit) and `ptrdiff_t` is `long` (64-bit). They're of the same rank, and so `long` gets implicitly converted to `unsigned long` before the comparison. Similarly, on x86 (32-bit), `size_t` is `unsigned int` (32-bit) and `ptrdiff_t` is `int` (32-bit). Again, they have the same rank, and `int` becomes `unsigned int`. The outcome in both cases is an unsigned comparison.
Okay, let's step back and consider potential edge cases. What happens if `tmp_buf_end` is actually *less than* `tmp_buf`? That would mean `tmp_buf_end - tmp_buf` would be a negative value. When this negative `ptrdiff_t` is converted to `size_t` (an unsigned type), it becomes a *very* large positive number, a wrapping around of the value. If `sizeof_cur_arg` is a regular, smaller value, then the comparison effectively becomes `large_positive_value < smaller_value`, which would be false (when it *should* be true if the calculation were interpreted signed). So, this would be an integer overflow!
So, the critical insight is: the comparison *is* an unsigned comparison due to the type conversions. I'll need to remember that when checking for potential vulnerabilities like integer overflows.
Just confirming that `sizeof_cur_arg` *is* actually `size_t`, as the user mentioned and the types suggest. Excellent.
Now, my mind goes back to the audit logging context and the type of bug. Is there a similar issue in the auditing code? Let me see. I'll search for `tmp_buf` again in `kernel/audit.c`. I might need to look at `fs/proc/base.c` as well, or `kernel/auditsc.c`.
|
| 264/4 |
2026/05/18 19:07 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/audit.c]
Results: map[Entities:[map[Kind:function Name:__audit_log_end] map[Kind:function Name:audit_backlog_limit_set] map[Kind:function Name:audit_buffer_alloc] map[Kind:function Name:audit_buffer_aux_end] map[Kind:function Name:audit_buffer_aux_new] map[Kind:function Name:audit_buffer_free] map[Kind:function Name:audit_cfg_lsm] map[Kind:function Name:audit_ctl_lock] map[Kind:function Name:audit_ctl_owner_current] map[Kind:function Name:audit_ctl_unlock] map[Kind:function Name:audit_do_config_change] map[Kind:function Name:audit_enable] map[Kind:function Name:audit_expand] map[Kind:function Name:audit_free_reply] map[Kind:function Name:audit_get_feature] map[Kind:function Name:audit_get_sk] map[Kind:function Name:audit_get_stamp] map[Kind:function Name:audit_get_tty] map[Kind:function Name:audit_init] map[Kind:function Name:audit_log] map[Kind:function Name:audit_log_common_recv_msg] map[Kind:function Name:audit_log_config_change] map[Kind:function Name:audit_log_d_path] map[Kind:function Name:audit_log_d_path_exe] map[Kind:function Name:audit_log_end] map[Kind:function Name:audit_log_feature_change] map[Kind:function Name:audit_log_format] map[Kind:function Name:audit_log_key] map[Kind:function Name:audit_log_lost] map[Kind:function Name:audit_log_multicast] map[Kind:function Name:audit_log_n_hex] map[Kind:function Name:audit_log_n_string] map[Kind:function Name:audit_log_n_untrustedstring] map[Kind:function Name:audit_log_nf_skb] map[Kind:function Name:audit_log_obj_ctx] map[Kind:function Name:audit_log_path_denied] map[Kind:function Name:audit_log_session_info] map[Kind:function Name:audit_log_set_loginuid] map[Kind:function Name:audit_log_start] map[Kind:function Name:audit_log_subj_ctx] map[Kind:function Name:audit_log_task_context] map[Kind:function Name:audit_log_task_info] map[Kind:function Name:audit_log_untrustedstring] map[Kind:function Name:audit_log_user_recv_msg] map[Kind:function Name:audit_log_vformat] map[Kind:function Name:audit_make_reply] map[Kind:function Name:audit_multicast_bind] map[Kind:function Name:audit_multicast_unbind] map[Kind:function Name:audit_net_exit] map[Kind:function Name:audit_net_init] map[Kind:function Name:audit_netlink_ok] map[Kind:function Name:audit_panic] map[Kind:function Name:audit_put_tty] map[Kind:function Name:audit_rate_check] map[Kind:function Name:audit_receive] map[Kind:function Name:audit_receive_msg] map[Kind:function Name:audit_replace] map[Kind:function Name:audit_send_list_thread] map[Kind:function Name:audit_send_reply] map[Kind:function Name:audit_send_reply_thread] map[Kind:function Name:audit_serial] map[Kind:function Name:audit_set_backlog_limit] map[Kind:function Name:audit_set_backlog_wait_time] map[Kind:function Name:audit_set_enabled] map[Kind:function Name:audit_set_failure] map[Kind:function Name:audit_set_feature] map[Kind:function Name:audit_set_loginuid] map[Kind:function Name:audit_set_loginuid_perm] map[Kind:function Name:audit_set_rate_limit] map[Kind:function Name:audit_signal_info] map[Kind:function Name:audit_string_contains_control] map[Kind:function Name:auditd_conn_free] map[Kind:function Name:auditd_pid_vnr] map[Kind:function Name:auditd_reset] map[Kind:function Name:auditd_send_unicast_skb] map[Kind:function Name:auditd_set] map[Kind:function Name:auditd_test_task] map[Kind:function Name:is_audit_feature_set] map[Kind:function Name:kauditd_hold_skb] map[Kind:function Name:kauditd_printk_skb] map[Kind:function Name:kauditd_rehold_skb] map[Kind:function Name:kauditd_retry_skb] map[Kind:function Name:kauditd_send_multicast_skb] map[Kind:function Name:kauditd_send_queue] map[Kind:function Name:kauditd_thread] map[Kind:struct Name:audit_buffer] map[Kind:struct Name:audit_ctl_mutex] map[Kind:struct Name:audit_net] map[Kind:struct Name:audit_reply] map[Kind:struct Name:auditd_connection] map[Kind:global_variable Name:__UNIQUE_ID_addressable_audit_enabled_820] map[Kind:global_variable Name:__UNIQUE_ID_addressable_audit_init_844] map[Kind:global_variable Name:__UNIQUE_ID_addressable_audit_log_857] map[Kind:global_variable Name:__UNIQUE_ID_addressable_audit_log_end_855] map[Kind:global_variable Name:__UNIQUE_ID_addressable_audit_log_format_856] map[Kind:global_variable Name:__UNIQUE_ID_addressable_audit_log_nf_skb_851] map[Kind:global_variable Name:__UNIQUE_ID_addressable_audit_log_start_854] map[Kind:global_variable Name:__UNIQUE_ID_addressable_audit_log_subj_ctx_847] map[Kind:global_variable Name:__UNIQUE_ID_addressable_audit_log_task_context_848] map[Kind:global_variable Name:__UNIQUE_ID_addressable_audit_log_task_info_850] map[Kind:global_variable Name:__setup_audit_backlog_limit_set] map[Kind:global_variable Name:__setup_audit_enable] map[Kind:global_variable Name:__setup_str_audit_backlog_limit_set] map[Kind:global_variable Name:__setup_str_audit_enable] map[Kind:global_variable Name:af] map[Kind:global_variable Name:audit_backlog_limit] map[Kind:global_variable Name:audit_backlog_wait] map[Kind:global_variable Name:audit_backlog_wait_time] map[Kind:global_variable Name:audit_backlog_wait_time_actual] map[Kind:global_variable Name:audit_default] map[Kind:global_variable Name:audit_enabled] map[Kind:global_variable Name:audit_ever_enabled] map[Kind:global_variable Name:audit_failure] map[Kind:global_variable Name:audit_feature_names] map[Kind:global_variable Name:audit_initialized] map[Kind:global_variable Name:audit_lost] map[Kind:global_variable Name:audit_net_ops] map[Kind:global_variable Name:audit_sig_pid] map[Kind:global_variable Name:audit_sig_uid] map[Kind:global_variable Name:auditd_conn_lock] map[Kind:global_variable Name:kauditd_wait] map[Kind:global_variable Name:session_id]]]
|
| 265/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 94391
output: 41
thoughts: 21
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and presented as if it were my own internal monologue:
**Diving into the Audit Log Implementation**
Right, so I need to understand how `audit_log_n_string` functions, and my best bet is to dig directly into the source. That means I'm going to navigate over to `kernel/audit.c`. That's where I expect to find the core implementation and any related data structures or helper functions that it depends on. I anticipate that this function is critical for logging string data within the audit subsystem and probably used extensively throughout the kernel. I'll be looking for things like buffer management, any potential limitations on string length, and how it interacts with the audit event structures. From experience, I anticipate this will involve a call to `audit_log_format` or something similar, and I'll need to understand how the arguments are passed to it. Let's get to it.
|
| 266/4 |
2026/05/18 19:07 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/audit.c IncludeLines:true Name:audit_log_n_string]
Results: map[SourceCode:2109: /*
2110: * Format a string of no more than slen characters into the audit buffer,
2111: * enclosed in quote marks.
2112: */
2113: void audit_log_n_string(struct audit_buffer *ab, const char *string,
2114: size_t slen)
2115: {
2116: int avail, new_len;
2117: unsigned char *ptr;
2118: struct sk_buff *skb;
2119:
2120: if (!ab)
2121: return;
2122:
2123: BUG_ON(!ab->skb);
2124: skb = ab->skb;
2125: avail = skb_tailroom(skb);
2126: new_len = slen + 3; /* enclosing quotes + null terminator */
2127: if (new_len > avail) {
2128: avail = audit_expand(ab, new_len);
2129: if (!avail)
2130: return;
2131: }
2132: ptr = skb_tail_pointer(skb);
2133: *ptr++ = '"';
2134: memcpy(ptr, string, slen);
2135: ptr += slen;
2136: *ptr++ = '"';
2137: *ptr = 0;
2138: skb_put(skb, slen + 2); /* don't include null terminator */
2139: }
SourceFile:kernel/audit.c]
|
| 267/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 95126
output: 43
thoughts: 123
Thoughts: **Navigating the Labyrinth of Kernel Code**
Okay, I'm stuck. Not making any headway on this. Hmm, let me circle back and search for `tmp_buf` again, this time in `fs/exec.c`, but this time I'll broaden my search within that file for any occurrence of it.
Actually, scratch that. I'm going to take a more aggressive approach. Instead of a targeted search, I'll search the *entire* kernel source for the string "tmp_buf". That should turn up everything, right? I'll probably be able to pin down its usage that way.
Damn, I can't do that. I'm forgetting my current environment constraints... Okay, back to a more targeted approach then. Let's revisit `fs/proc/base.c` and search for `tmp_buf` there again.
Wait, before I do that, let me just check `get_mm_proctitle`. It might be relevant.
|
| 268/4 |
2026/05/18 19:07 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:fs/proc/base.c IncludeLines:true Name:get_mm_proctitle]
Results: map[SourceCode: 250: /*
251: * If the user used setproctitle(), we just get the string from
252: * user space at arg_start, and limit it to a maximum of one page.
253: */
254: static ssize_t get_mm_proctitle(struct mm_struct *mm, char __user *buf,
255: size_t count, unsigned long pos,
256: unsigned long arg_start)
257: {
258: char *page;
259: int ret, got;
260:
261: if (pos >= PAGE_SIZE)
262: return 0;
263:
264: page = (char *)__get_free_page(GFP_KERNEL);
265: if (!page)
266: return -ENOMEM;
267:
268: ret = 0;
269: got = access_remote_vm(mm, arg_start, page, PAGE_SIZE, FOLL_ANON);
270: if (got > 0) {
271: int len = strnlen(page, got);
272:
273: /* Include the NUL character if it was found */
274: if (len < got)
275: len++;
276:
277: if (len > pos) {
278: len -= pos;
279: if (len > count)
280: len = count;
281: len -= copy_to_user(buf, page+pos, len);
282: if (!len)
283: len = -EFAULT;
284: ret = len;
285: }
286: }
287: free_page((unsigned long)page);
288: return ret;
289: }
SourceFile:fs/proc/base.c]
|
| 269/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 96073
output: 35
thoughts: 892
Thoughts: **Analyzing C Type Comparisons**
Okay, the question is straightforward – it's about C types and how they interact in a comparison. I'm looking at the expression `tmp_buf_end - tmp_buf < sizeof_cur_arg`.
Let's break down the types involved: `tmp_buf_end` and `tmp_buf` are pointers, so their difference, `tmp_buf_end - tmp_buf`, is a `ptrdiff_t` (signed), while `sizeof_cur_arg` is a `size_t` (unsigned).
Now, the C rules kick in (Usual Arithmetic Conversions). When comparing a signed integer (`ptrdiff_t`) and an unsigned integer (`size_t`), the standard dictates that there's a conversion. Specifically, since `size_t` and `ptrdiff_t` have the same rank in the Linux kernel (both are the same width as a pointer), `ptrdiff_t` gets converted to `size_t`. This makes the comparison *unsigned*.
I need to make sure my understanding is correct and that this isn't platform dependent. The implication is significant. If `tmp_buf_end - tmp_buf` is, say, -1 (meaning `tmp_buf_end` is one byte before `tmp_buf`), and `sizeof_cur_arg` is 10, then -1, when interpreted as a `size_t`, becomes a massive positive number (e.g., `0xFFFFFFFF` on a 32-bit system or `0xFFFFFFFFFFFFFFFF` on 64-bit). The comparison will yield *false*, even though -1 is mathematically less than 10. That’s counterintuitive if you're expecting a signed comparison!
Wait, let me double-check. The crux of the issue relies on the relative ranks of `size_t` and `ptrdiff_t`. I'm pretty certain, but I'll confirm in the C11 standard. Ah yes, 6.3.1.8 confirms it: the signed type is converted to the unsigned type if the unsigned type's rank is greater than or equal to the signed type's rank. And the standard also states that the rank of the signed and unsigned of same bit-width are the same. So, the rule applies.
So yes, the subtraction of two pointers gives you `ptrdiff_t`, and `sizeof` always gives you `size_t`. So the comparison is `ptrdiff_t < size_t`, which is *indeed* an unsigned comparison.
Okay, let's think. Does it matter if `sizeof_cur_arg` is a constant? No, `sizeof(something)` is always of type `size_t`, so `tmp_buf_end - tmp_buf < sizeof(something)` is also an unsigned comparison. I am confident in my analysis.
Just to be absolutely sure, I'll search the kernel code base again to see the actual usage. I'll see if I can find code that makes this comparison using a different approach. I'll search for `tmp_buf_end`, but this time focus my search on `drivers/usb/gadget/function/f_mass_storage.c`, maybe I missed something. I'll see what I can find. Wait, I already checked that and I remember it's not there. Looks like I've got a definitive answer.
|
| 270/4 |
2026/05/18 19:07 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:drivers/usb/gadget/function/f_mass_storage.c]
Results: map[Entities:[map[Kind:function Name:__fsg_is_set] map[Kind:function Name:__raise_exception] map[Kind:function Name:_fsg_common_free_buffers] map[Kind:function Name:_fsg_common_get_max_lun] map[Kind:function Name:_fsg_common_remove_luns] map[Kind:function Name:alloc_request] map[Kind:function Name:bulk_in_complete] map[Kind:function Name:bulk_out_complete] map[Kind:function Name:check_command] map[Kind:function Name:check_command_size_in_blocks] map[Kind:function Name:do_inquiry] map[Kind:function Name:do_mode_select] map[Kind:function Name:do_mode_sense] map[Kind:function Name:do_prevent_allow] map[Kind:function Name:do_read] map[Kind:function Name:do_read_capacity] map[Kind:function Name:do_read_capacity_16] map[Kind:function Name:do_read_format_capacities] map[Kind:function Name:do_read_header] map[Kind:function Name:do_read_toc] map[Kind:function Name:do_request_sense] map[Kind:function Name:do_scsi_command] map[Kind:function Name:do_set_interface] map[Kind:function Name:do_start_stop] map[Kind:function Name:do_synchronize_cache] map[Kind:function Name:do_verify] map[Kind:function Name:do_write] map[Kind:function Name:ep0_queue] map[Kind:function Name:exception_in_progress] map[Kind:function Name:file_show] map[Kind:function Name:file_store] map[Kind:function Name:finish_reply] map[Kind:function Name:forced_eject_store] map[Kind:function Name:fsg_alloc] map[Kind:function Name:fsg_alloc_inst] map[Kind:function Name:fsg_attr_release] map[Kind:function Name:fsg_bind] map[Kind:function Name:fsg_common_create_lun] map[Kind:function Name:fsg_common_create_luns] map[Kind:function Name:fsg_common_free_buffers] map[Kind:function Name:fsg_common_release] map[Kind:function Name:fsg_common_remove_lun] map[Kind:function Name:fsg_common_remove_luns] map[Kind:function Name:fsg_common_set_cdev] map[Kind:function Name:fsg_common_set_inquiry_string] map[Kind:function Name:fsg_common_set_num_buffers] map[Kind:function Name:fsg_common_set_sysfs] map[Kind:function Name:fsg_common_setup] map[Kind:function Name:fsg_config_from_params] map[Kind:function Name:fsg_disable] map[Kind:function Name:fsg_free] map[Kind:function Name:fsg_free_inst] map[Kind:function Name:fsg_from_func] map[Kind:function Name:fsg_lun_attr_release] map[Kind:function Name:fsg_lun_dev_is_visible] map[Kind:function Name:fsg_lun_drop] map[Kind:function Name:fsg_lun_make] map[Kind:function Name:fsg_lun_opts_cdrom_show] map[Kind:function Name:fsg_lun_opts_cdrom_store] map[Kind:function Name:fsg_lun_opts_file_show] map[Kind:function Name:fsg_lun_opts_file_store] map[Kind:function Name:fsg_lun_opts_forced_eject_store] map[Kind:function Name:fsg_lun_opts_inquiry_string_show] map[Kind:function Name:fsg_lun_opts_inquiry_string_store] map[Kind:function Name:fsg_lun_opts_nofua_show] map[Kind:function Name:fsg_lun_opts_nofua_store] map[Kind:function Name:fsg_lun_opts_removable_show] map[Kind:function Name:fsg_lun_opts_removable_store] map[Kind:function Name:fsg_lun_opts_ro_show] map[Kind:function Name:fsg_lun_opts_ro_store] map[Kind:function Name:fsg_lun_release] map[Kind:function Name:fsg_main_thread] map[Kind:function Name:fsg_opts_num_buffers_show] map[Kind:function Name:fsg_opts_num_buffers_store] map[Kind:function Name:fsg_opts_stall_show] map[Kind:function Name:fsg_opts_stall_store] map[Kind:function Name:fsg_set_alt] map[Kind:function Name:fsg_set_halt] map[Kind:function Name:fsg_setup] map[Kind:function Name:fsg_unbind] map[Kind:function Name:get_next_command] map[Kind:function Name:halt_bulk_in_endpoint] map[Kind:function Name:handle_exception] map[Kind:function Name:invalidate_sub] map[Kind:function Name:mass_storagemod_exit] map[Kind:function Name:mass_storagemod_init] map[Kind:function Name:nofua_show] map[Kind:function Name:nofua_store] map[Kind:function Name:raise_exception] map[Kind:function Name:received_cbw] map[Kind:function Name:ro_show] map[Kind:function Name:ro_store] map[Kind:function Name:send_status] map[Kind:function Name:set_bulk_out_req_length] map[Kind:function Name:sleep_thread] map[Kind:function Name:start_in_transfer] map[Kind:function Name:start_out_transfer] map[Kind:function Name:start_transfer] map[Kind:function Name:throw_away_data] map[Kind:function Name:to_fsg_lun_opts] map[Kind:function Name:to_fsg_opts] map[Kind:function Name:wedge_bulk_in_endpoint] map[Kind:struct Name:fsg_common] map[Kind:struct Name:fsg_dev] map[Kind:global_variable Name:__UNIQUE_ID_addressable_fsg_common_create_lun_530] map[Kind:global_variable Name:__UNIQUE_ID_addressable_fsg_common_create_luns_531] map[Kind:global_variable Name:__UNIQUE_ID_addressable_fsg_common_free_buffers_528] map[Kind:global_variable Name:__UNIQUE_ID_addressable_fsg_common_remove_lun_526] map[Kind:global_variable Name:__UNIQUE_ID_addressable_fsg_common_remove_luns_527] map[Kind:global_variable Name:__UNIQUE_ID_addressable_fsg_common_set_cdev_529] map[Kind:global_variable Name:__UNIQUE_ID_addressable_fsg_common_set_inquiry_string_532] map[Kind:global_variable Name:__UNIQUE_ID_addressable_fsg_common_set_num_buffers_525] map[Kind:global_variable Name:__UNIQUE_ID_addressable_fsg_common_set_sysfs_524] map[Kind:global_variable Name:__UNIQUE_ID_addressable_fsg_config_from_params_551] map[Kind:global_variable Name:__UNIQUE_ID_addressable_mass_storagemod_init_543] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_541] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_544] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_545] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_546] map[Kind:global_variable Name:__UNIQUE_ID_modinfo_547] map[Kind:global_variable Name:__exitcall_mass_storagemod_exit] map[Kind:global_variable Name:dev_attr_file] map[Kind:global_variable Name:dev_attr_forced_eject] map[Kind:global_variable Name:dev_attr_nofua] map[Kind:global_variable Name:dev_attr_ro] map[Kind:global_variable Name:fsg_attrs] map[Kind:global_variable Name:fsg_func_type] map[Kind:global_variable Name:fsg_group_ops] map[Kind:global_variable Name:fsg_item_ops] map[Kind:global_variable Name:fsg_lun_attrs] map[Kind:global_variable Name:fsg_lun_dev_attrs] map[Kind:global_variable Name:fsg_lun_dev_group] map[Kind:global_variable Name:fsg_lun_dev_groups] map[Kind:global_variable Name:fsg_lun_item_ops] map[Kind:global_variable Name:fsg_lun_opts_attr_cdrom] map[Kind:global_variable Name:fsg_lun_opts_attr_file] map[Kind:global_variable Name:fsg_lun_opts_attr_forced_eject] map[Kind:global_variable Name:fsg_lun_opts_attr_inquiry_string] map[Kind:global_variable Name:fsg_lun_opts_attr_nofua] map[Kind:global_variable Name:fsg_lun_opts_attr_removable] map[Kind:global_variable Name:fsg_lun_opts_attr_ro] map[Kind:global_variable Name:fsg_lun_type] map[Kind:global_variable Name:fsg_opts_attr_num_buffers] map[Kind:global_variable Name:fsg_opts_attr_stall] map[Kind:global_variable Name:fsg_string_interface] map[Kind:global_variable Name:fsg_strings] map[Kind:global_variable Name:fsg_strings_array] map[Kind:global_variable Name:fsg_stringtab] map[Kind:global_variable Name:mass_storageusb_func]]]
|
| 271/4 |
2026/05/18 19:07 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 100674
output: 595
thoughts: 186
Thoughts: **Unsigned Comparison in C**
Okay, I'm not entirely focused right now, but I can definitely address this C rule. I'm feeling good about my grasp of the core concepts here.
Let's break down this comparison: `tmp_buf_end - tmp_buf < sizeof_cur_arg`. It seems straightforward, but we need to be precise about how C handles these different integer types. First, `tmp_buf_end - tmp_buf` evaluates to a `ptrdiff_t`. I know that `ptrdiff_t` is a *signed* integer type. On the other hand, `sizeof_cur_arg` is a `size_t`, which is, without question, an *unsigned* integer.
Now, here's the crucial C rule: when we compare a signed and an unsigned integer of the same "rank," which, as an expert, I *know* is the case here as `ptrdiff_t` and `size_t` are the same size as a pointer in the Linux kernel. Therefore, the signed integer is *implicitly converted* to the unsigned type for the comparison. So, while it *looks* like a signed comparison, it's actually an unsigned one under the hood. Thus, this expression resolves to an unsigned comparison. I can now formulate a complete and accurate answer.
|
| 272/2 |
2026/05/18 19:08 |
llm |
debugger |
0m
Model:
Tokens:
input: 233562
output: 1180
thoughts: 12
Thoughts: Okay, here's the summary, formatted as requested:
**Assessing the Explanation**
Alright, after reviewing the material, I'm confident in the foundational explanation provided. It's well-structured and covers the necessary elements. There's no need for further clarification or alteration; it's ready for the final response. My expertise allows me to integrate this into a compelling conclusion, so let's get to it.
|
| 273/1 |
2026/05/18 19:08 |
action |
kernel-scratch-checkouter |
0m
Results: map[KernelScratchSrc:/app/workdir/cache/tmp/tmp2211551186]
|
| 274/1 |
2026/05/18 19:08 |
loop |
|
1h54m
|
| 275/2 |
2026/05/18 19:08 |
iteration |
0 |
1h13m
|
| 276/3 |
2026/05/18 19:08 |
agent |
patch-generator |
27m
Model:
Instruction:
You are an experienced Linux kernel developer tasked with creating a fix for a kernel bug.
You will be given a crash report, and an initial explanation of the root cause done by another
kernel expert.
Use the codeeditor tool to do code edits.
Note: you will not see your changes when looking at the code using codesearch tools.
Use the patch-diff tool to review the modifications you applied.
Your final reply should contain explanation of what you did in the patch and why
(details not present in the initial explanation of the bug).
Your fix must not just prevent the given crash, but also be the best fix for the underlying
root cause from the software engineering point of view. There can be several ways to fix the
same bug. Consider alternatives, and pick the best one. For example, additional checks may be
added at different locations/functions, it's usually better to place them earlier in the
execution to avoid multiple checks at various locations later.
Frequently the same coding mistake is done in several locations in the source code.
Check if your fix should be extended/applied to similar cases around to fix other similar bugs.
But don't go too wide, don't try to fix problems kernel-wide, fix similar issues
in the same file only.
If you are changing post-conditions of a function, consider all callers of the functions,
and if they need to be updated to handle new post-conditions. For example, if you make
a function that previously never returned a NULL, return NULL, consider if callers
need to be updated to handle NULL return value.
Prefer calling several tools at the same time to save round-trips.
Prompt:
The crash that corresponds to the bug is:
watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/u9:4:1179]
CPU#0 Utilization every 4000ms during lockup:
#1: 100% system, 1% softirq, 1% hardirq, 0% idle
#2: 100% system, 1% softirq, 0% hardirq, 0% idle
#3: 100% system, 1% softirq, 1% hardirq, 0% idle
#4: 100% system, 1% softirq, 1% hardirq, 0% idle
#5: 100% system, 1% softirq, 1% hardirq, 0% idle
Modules linked in:
irq event stamp: 386488
hardirqs last enabled at (386487): [<ffffffff8100148a>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
hardirqs last disabled at (386488): [<ffffffff8b9eb11e>] sysvec_apic_timer_interrupt+0xe/0xb0 arch/x86/kernel/apic/apic.c:1061
softirqs last enabled at (386486): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
softirqs last disabled at (386409): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
CPU: 0 UID: 0 PID: 1179 Comm: kworker/u9:4 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:342 [inline]
RIP: 0010:smp_call_function_many_cond+0xfcb/0x13b0 kernel/smp.c:892
Code: 79 45 8b 2e 44 89 ee 83 e6 01 31 ff e8 be c6 0b 00 41 83 e5 01 49 bd 00 00 00 00 00 fc ff df 75 07 e8 69 c2 0b 00 eb 37 f3 90 <43> 0f b6 04 2c 84 c0 75 10 41 f7 06 01 00 00 00 74 1e e8 4e c2 0b
RSP: 0018:ffffc90005fcf720 EFLAGS: 00000293
RAX: ffffffff81834d62 RBX: ffff88806943ad08 RCX: ffff88801d361c00
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90005fcf860 R08: ffffffff908e5af7 R09: 1ffffffff211cb5e
R10: dffffc0000000000 R11: fffffbfff211cb5f R12: 1ffff1100d2a7edd
R13: dffffc0000000000 R14: ffff88806953f6e8 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff8880d5abe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555578641490 CR3: 000000000eaf4000 CR4: 00000000000006f0
Call Trace:
<TASK>
on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1057
on_each_cpu include/linux/smp.h:72 [inline]
smp_text_poke_sync_each_cpu arch/x86/kernel/alternative.c:2773 [inline]
smp_text_poke_batch_finish+0x5e6/0x1090 arch/x86/kernel/alternative.c:2983
arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
static_key_enable_cpuslocked+0x128/0x240 kernel/jump_label.c:210
static_key_enable+0x1a/0x20 kernel/jump_label.c:223
toggle_allocation_gate+0xab/0x290 mm/kfence/core.c:906
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xad1/0x1740 kernel/workqueue.c:3397
worker_thread+0xa59/0xfc0 kernel/workqueue.c:3478
kthread+0x388/0x470 kernel/kthread.c:436
ret_from_fork+0x437/0x9d0 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:__remove_hrtimer+0x8e/0x3e0 kernel/time/hrtimer.c:1145
Code: ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d
RSP: 0018:ffffc900001b8d90 EFLAGS: 00000006
RAX: ffffffff817d161e RBX: ffff888069527f40 RCX: 0000000000010000
RDX: ffff8880160c8000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: dffffc0000000000 R08: ffffffff94f48153 R09: 1ffffffff29e902a
R10: dffffc0000000000 R11: fffffbfff29e902b R12: 0000000000000000
R13: 0000000000000001 R14: ffff888024042300 R15: ffff888069528000
FS: 0000000000000000(0000) GS:ffff8880d5bbe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f83db077470 CR3: 000000001f132000 CR4: 00000000000006f0
Call Trace:
<IRQ>
__run_hrtimer kernel/time/hrtimer.c:1910 [inline]
__hrtimer_run_queues+0x299/0xa10 kernel/time/hrtimer.c:1994
hrtimer_interrupt+0x44b/0x950 kernel/time/hrtimer.c:2113
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1050 [inline]
__sysvec_apic_timer_interrupt+0x102/0x420 arch/x86/kernel/apic/apic.c:1067
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1061 [inline]
sysvec_apic_timer_interrupt+0x8b/0xb0 arch/x86/kernel/apic/apic.c:1061
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:native_irq_disable arch/x86/include/asm/irqflags.h:37 [inline]
RIP: 0010:arch_local_irq_disable arch/x86/include/asm/irqflags.h:114 [inline]
RIP: 0010:default_idle+0xf/0x20 arch/x86/kernel/process.c:768
Code: 4c 01 c7 4c 29 c2 e9 6e ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 13 a4 23 00 fb f4 <fa> e9 8b 8d 02 00 cc 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
RSP: 0018:ffffc9000014fe28 EFLAGS: 00000202
RAX: 00000000006a8f01 RBX: ffffffff81654d3a RCX: 0000000080000001
RDX: 0000000000000001 RSI: ffffffff8e3ef513 RDI: ffffffff8c21df20
RBP: ffffc9000014ff20 R08: ffff888069532993 R09: 1ffff1100d2a6532
R10: dffffc0000000000 R11: ffffed100d2a6533 R12: 0000000000000001
R13: 1ffff11002c19000 R14: 0000000000000001 R15: 1ffff11002c19000
default_idle_call+0x70/0xb0 kernel/sched/idle.c:122
cpuidle_idle_call kernel/sched/idle.c:199 [inline]
do_idle+0x36a/0x5e0 kernel/sched/idle.c:352
cpu_startup_entry+0x43/0x60 kernel/sched/idle.c:451
start_secondary+0x101/0x110 arch/x86/kernel/smpboot.c:312
common_startup_64+0x12c/0x137
</TASK>
The explanation of the root cause of the bug is:
The root cause of this crash is a massive out-of-bounds memory write (buffer overflow) in the BPF helper `bpf_snprintf` (and other `bprintf`-like helpers) due to integer signedness and unchecked return values in `bpf_bprintf_prepare()`.
This memory corruption overwrites adjacent kernel memory, which in this case included `hrtimer` structures and their `timerqueue` rbtree nodes. When CPU 1 later attempted to remove a corrupted timer via `__remove_hrtimer` -> `timerqueue_linked_del` -> `rb_erase_linked`, it got stuck in an infinite loop traversing a cyclic rbtree. CPU 0 then detected a soft lockup because it was waiting for CPU 1 to acknowledge an IPI (`csd_lock_wait`), but CPU 1 had interrupts disabled while processing the timer queue.
### Detailed Analysis
In `kernel/bpf/helpers.c`, the `bpf_bprintf_prepare()` function formats arguments into a temporary 512-byte buffer (`buffers->bin_args`). There are two distinct bugs in this function that allow `tmp_buf` to advance past `tmp_buf_end`, bypassing bounds checks.
#### Bug 1: Unchecked `snprintf` return value for `%pB`
When handling the `%pB` format specifier, the code does:
```c
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
```
`snprintf` returns the number of characters that *would* have been written if the buffer was large enough, excluding the null terminator. If the remaining buffer space (`tmp_buf_end - tmp_buf`) is smaller than the formatted string, `err` will be greater than or equal to the remaining space. Consequently, `tmp_buf += (err + 1)` will advance `tmp_buf` **past** `tmp_buf_end`.
#### Bug 2: Signedness bug in bounds checking
When handling standard format specifiers (like `%d`, `%x`), the code aligns `tmp_buf` and checks if there is enough space:
```c
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
```
Here, `tmp_buf_end - tmp_buf` is of type `ptrdiff_t` (signed), and `sizeof_cur_arg` is of type `size_t` (unsigned). In C, this comparison forces the signed `ptrdiff_t` to be cast to an unsigned `size_t`.
If `tmp_buf` has already been advanced past `tmp_buf_end` (e.g., via the `%pB` bug above, or by using `%c` to make `tmp_buf` unaligned such that `PTR_ALIGN` pushes it past `tmp_buf_end`), `tmp_buf_end - tmp_buf` becomes negative. When cast to `size_t`, this negative value becomes a huge positive number (e.g., `0xFFFFFFFFFFFFFFF0`).
The condition `HUGE_NUMBER < sizeof_cur_arg` evaluates to `false`, bypassing the `-ENOSPC` check. The code then writes to `tmp_buf`, corrupting memory out of bounds.
Furthermore, if a `%s` specifier is used after `tmp_buf` is out of bounds:
```c
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
```
The equality check `tmp_buf_end == tmp_buf` fails to catch that `tmp_buf > tmp_buf_end`. Then `bpf_trace_copy_string` is called with a negative `bufsz` (`tmp_buf_end - tmp_buf`), which is cast to a huge `size_t`. This results in copying an arbitrarily large string into kernel memory, causing massive corruption.
### Fix
To fix this vulnerability:
1. For `%pB`, `tmp_buf` should only be advanced by the actual number of bytes written, and it must not exceed `tmp_buf_end`.
2. The bounds checks must correctly handle the case where `tmp_buf >= tmp_buf_end`. A safe way is to cast `tmp_buf_end - tmp_buf` to a signed type and check if it's less than the required size, or simply check `if (tmp_buf + sizeof_cur_arg > tmp_buf_end)`.
Reply: I have analyzed the crash report and the root cause explanation, and implemented a comprehensive fix for the vulnerabilities in `bpf_bprintf_prepare()`.
### What I did in the patch:
1. **Fixed the `snprintf` return value handling for `%pB` and `%pI`:**
`snprintf` returns the number of characters that *would* have been written if the buffer was large enough. Blindly advancing `tmp_buf` by `err + 1` causes the pointer to exceed `tmp_buf_end` when truncation occurs. I introduced a `limit` variable to represent the remaining buffer space and used `min_t(size_t, err + 1, limit)` to advance `tmp_buf`. This guarantees that `tmp_buf` never exceeds `tmp_buf_end` under any circumstances.
2. **Fixed the integer signedness bug in bounds checking:**
The original bounds checks used `tmp_buf_end - tmp_buf < sizeof_cur_arg`. Because `sizeof_cur_arg` is an unsigned `size_t`, the signed `ptrdiff_t` result of the pointer subtraction was implicitly cast to unsigned. If `tmp_buf` ever exceeded `tmp_buf_end`, the negative difference became a huge positive number, bypassing the check. I rewrote these checks as `tmp_buf + sizeof_cur_arg > tmp_buf_end`. This uses safe pointer arithmetic and completely eliminates the signedness issue.
3. **Hardened the equality checks for `%s` and `%c`:**
The original code used `if (tmp_buf_end == tmp_buf)` to check for a full buffer. While the fixes above prevent `tmp_buf` from exceeding `tmp_buf_end`, it is a better software engineering practice to use `>=` for bounds checking. I updated these checks to `if (tmp_buf >= tmp_buf_end)` to make the code robust against any future modifications that might inadvertently advance `tmp_buf` out of bounds.
4. **Applied the fix consistently:**
The initial explanation only mentioned the `%pB` bug, but a careful review revealed that the exact same `snprintf` bug existed for IP address formatting (`%pI4`, `%pI6`, etc.). I applied the same `limit` and `min_t` fix to the IP address formatting block to ensure all similar vulnerabilities in the file are addressed.
### Why this is the best fix:
Instead of just patching the specific `%pB` case, this fix addresses the underlying architectural fragility. By changing the bounds checks to use pointer addition (`tmp_buf + size > tmp_buf_end`) and hardening the equality checks (`>=`), the function is now structurally immune to signedness bypasses. Furthermore, by strictly capping the pointer advancement using `min_t`, we ensure that `tmp_buf` remains a valid pointer within the allocated buffer at all times, preventing any cascading failures in subsequent loop iterations.
|
| 277/4 |
2026/05/18 19:08 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 8936
output: 44
thoughts: 201
Thoughts: **Analyzing a Soft Lockup Bug in BPF Printing**
Okay, so we've got a crash report, and it looks like a soft lockup. Time to dig in. The root cause is pinpointed to an out-of-bounds memory write, specifically within `bpf_bprintf_prepare()` function in `kernel/bpf/helpers.c`. Interesting. Let's see what's going on there.
It appears there are two distinct, but related, bugs in the code. The first is about `snprintf` not being properly handled. The function returns how many characters *would* have been written, but isn't checked. It looks like if `snprintf` fails and the `tmp_buf` is advanced by `err + 1`, we can potentially write past the end of `tmp_buf_end`. That seems pretty straightforward.
The second bug is a classic signedness issue in the bounds checking. The check `tmp_buf_end - tmp_buf < sizeof_cur_arg` is the culprit. If `tmp_buf` somehow becomes greater than `tmp_buf_end` then the subtraction becomes negative. The negative value is interpreted as a huge unsigned number, thus bypassing the intended bounds check. Clever, but ultimately bad.
Time to pull up `kernel/bpf/helpers.c` and take a look at `bpf_bprintf_prepare()` and confirm these suspicions. Let's trace the execution and see how we can trigger this condition.
|
| 278/4 |
2026/05/18 19:08 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_prepare]
Results: map[SourceCode: 804: /*
805: * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
806: *
807: * Returns a negative value if fmt is an invalid format string or 0 otherwise.
808: *
809: * This can be used in two ways:
810: * - Format string verification only: when data->get_bin_args is false
811: * - Arguments preparation: in addition to the above verification, it writes in
812: * data->bin_args a binary representation of arguments usable by bstr_printf
813: * where pointers from BPF have been sanitized.
814: *
815: * In argument preparation mode, if 0 is returned, safe temporary buffers are
816: * allocated and bpf_bprintf_cleanup should be called to free them after use.
817: */
818: int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
819: u32 num_args, struct bpf_bprintf_data *data)
820: {
821: bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
822: char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
823: struct bpf_bprintf_buffers *buffers = NULL;
824: size_t sizeof_cur_arg, sizeof_cur_ip;
825: int err, i, num_spec = 0;
826: u64 cur_arg;
827: char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
828:
829: fmt_end = strnchr(fmt, fmt_size, 0);
830: if (!fmt_end)
831: return -EINVAL;
832: fmt_size = fmt_end - fmt;
833:
834: if (get_buffers && bpf_try_get_buffers(&buffers))
835: return -EBUSY;
836:
837: if (data->get_bin_args) {
838: if (num_args)
839: tmp_buf = buffers->bin_args;
840: tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
841: data->bin_args = (u32 *)tmp_buf;
842: }
843:
844: if (data->get_buf)
845: data->buf = buffers->buf;
846:
847: for (i = 0; i < fmt_size; i++) {
848: unsigned char c = fmt[i];
849:
850: /*
851: * Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
852: * through unchanged, while still rejecting ASCII control bytes.
853: */
854: if (isascii(c) && !isprint(c) && !isspace(c)) {
855: err = -EINVAL;
856: goto out;
857: }
858:
859: if (fmt[i] != '%')
860: continue;
861:
862: if (fmt[i + 1] == '%') {
863: i++;
864: continue;
865: }
866:
867: if (num_spec >= num_args) {
868: err = -EINVAL;
869: goto out;
870: }
871:
872: /* The string is zero-terminated so if fmt[i] != 0, we can
873: * always access fmt[i + 1], in the worst case it will be a 0
874: */
875: i++;
876: c = fmt[i];
877: /*
878: * The format parser below only understands ASCII conversion
879: * specifiers and modifiers, so reject non-ASCII after '%'.
880: */
881: if (!isascii(c)) {
882: err = -EINVAL;
883: goto out;
884: }
885:
886: /* skip optional "[0 +-][num]" width formatting field */
887: while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
888: fmt[i] == ' ')
889: i++;
890: if (fmt[i] >= '1' && fmt[i] <= '9') {
891: i++;
892: while (fmt[i] >= '0' && fmt[i] <= '9')
893: i++;
894: }
895:
896: if (fmt[i] == 'p') {
897: sizeof_cur_arg = sizeof(long);
898:
899: if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
900: ispunct(fmt[i + 1])) {
901: if (tmp_buf)
902: cur_arg = raw_args[num_spec];
903: goto nocopy_fmt;
904: }
905:
906: if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
907: fmt[i + 2] == 's') {
908: fmt_ptype = fmt[i + 1];
909: i += 2;
910: goto fmt_str;
911: }
912:
913: if (fmt[i + 1] == 'K' ||
914: fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
915: fmt[i + 1] == 'S') {
916: if (tmp_buf)
917: cur_arg = raw_args[num_spec];
918: i++;
919: goto nocopy_fmt;
920: }
921:
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
930:
931: i++;
932: num_spec++;
933: continue;
934: }
935:
936: /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
937: if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
938: (fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
939: err = -EINVAL;
940: goto out;
941: }
942:
943: i += 2;
944: if (!tmp_buf)
945: goto nocopy_fmt;
946:
947: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
948: if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
949: err = -ENOSPC;
950: goto out;
951: }
952:
953: unsafe_ptr = (char *)(long)raw_args[num_spec];
954: err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
955: sizeof_cur_ip);
956: if (err < 0)
957: memset(cur_ip, 0, sizeof_cur_ip);
958:
959: /* hack: bstr_printf expects IP addresses to be
960: * pre-formatted as strings, ironically, the easiest way
961: * to do that is to call snprintf.
962: */
963: ip_spec[2] = fmt[i - 1];
964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
969: num_spec++;
970:
971: continue;
972: } else if (fmt[i] == 's') {
973: fmt_ptype = fmt[i];
974: fmt_str:
975: if (fmt[i + 1] != 0 &&
976: !isspace(fmt[i + 1]) &&
977: !ispunct(fmt[i + 1])) {
978: err = -EINVAL;
979: goto out;
980: }
981:
982: if (!tmp_buf)
983: goto nocopy_fmt;
984:
985: if (tmp_buf_end == tmp_buf) {
986: err = -ENOSPC;
987: goto out;
988: }
989:
990: unsafe_ptr = (char *)(long)raw_args[num_spec];
991: err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
992: fmt_ptype,
993: tmp_buf_end - tmp_buf);
994: if (err < 0) {
995: tmp_buf[0] = '\0';
996: err = 1;
997: }
998:
999: tmp_buf += err;
1000: num_spec++;
1001:
1002: continue;
1003: } else if (fmt[i] == 'c') {
1004: if (!tmp_buf)
1005: goto nocopy_fmt;
1006:
1007: if (tmp_buf_end == tmp_buf) {
1008: err = -ENOSPC;
1009: goto out;
1010: }
1011:
1012: *tmp_buf = raw_args[num_spec];
1013: tmp_buf++;
1014: num_spec++;
1015:
1016: continue;
1017: }
1018:
1019: sizeof_cur_arg = sizeof(int);
1020:
1021: if (fmt[i] == 'l') {
1022: sizeof_cur_arg = sizeof(long);
1023: i++;
1024: }
1025: if (fmt[i] == 'l') {
1026: sizeof_cur_arg = sizeof(long long);
1027: i++;
1028: }
1029:
1030: if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
1031: fmt[i] != 'x' && fmt[i] != 'X') {
1032: err = -EINVAL;
1033: goto out;
1034: }
1035:
1036: if (tmp_buf)
1037: cur_arg = raw_args[num_spec];
1038: nocopy_fmt:
1039: if (tmp_buf) {
1040: tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
1041: if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
1042: err = -ENOSPC;
1043: goto out;
1044: }
1045:
1046: if (sizeof_cur_arg == 8) {
1047: *(u32 *)tmp_buf = *(u32 *)&cur_arg;
1048: *(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
1049: } else {
1050: *(u32 *)tmp_buf = (u32)(long)cur_arg;
1051: }
1052: tmp_buf += sizeof_cur_arg;
1053: }
1054: num_spec++;
1055: }
1056:
1057: err = 0;
1058: out:
1059: if (err)
1060: bpf_bprintf_cleanup(data);
1061: return err;
1062: }
SourceFile:kernel/bpf/helpers.c]
|
| 279/4 |
2026/05/18 19:08 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 13339
output: 45
thoughts: 531
Thoughts: **Analyzing the Format String Bug**
Okay, let's dive into this `%pB` format specifier issue. Looking at the code, I see a potential buffer overflow vulnerability. Specifically, when we're using `snprintf` to write the output to `tmp_buf`, there's a critical check that's missing. If `tmp_buf` ever catches up to or exceeds `tmp_buf_end`, the `tmp_buf_end - tmp_buf` calculation becomes either zero or, worse, a negative number.
That negative result, as a `size_t`, turns into an absurdly large value, which gets passed as the buffer size to `snprintf`. Obviously, this can lead to all sorts of memory corruption problems. The immediate fix is clear: we need to *absolutely* check if `tmp_buf` is greater or equal to `tmp_buf_end` *before* calling `snprintf`. We can either perform an explicit comparison `tmp_buf >= tmp_buf_end` or, use a min operation, or perform the subtraction and check the result of the `tmp_buf_end - tmp_buf` operation, as the result could be a negative value.
Further, `snprintf` returns the number of characters that would have been written, including the null terminator, if the buffer was large enough. If that return value `err` is greater or equal to the available space `tmp_buf_end - tmp_buf`, we know truncation occurred. So, how should we handle truncation? Advance `tmp_buf` by the correct amount? Or, return an error? Let's see...If `err >= tmp_buf_end - tmp_buf`, it means it's truncating, so should we advance the `tmp_buf` to `tmp_buf_end`, or should we return `-ENOSPC`? How does that interact with the behavior of other similar functions in the code, like `bpf_trace_copy_string`?
Thinking about truncation... I need to check how the rest of this code handles truncation. Does `bpf_bprintf_prepare` signal `-ENOSPC` if the buffer is full and truncation occurs? Or does it just silently truncate? I've seen that `bpf_trace_copy_string` truncates strings but still returns the number of bytes written. Importantly, does this include the null terminator in that returned length? I'd better check `bpf_trace_copy_string` to confirm.
|
| 280/4 |
2026/05/18 19:09 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_trace_copy_string]
Results: map[SourceCode: 744: static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,
745: size_t bufsz)
746: {
747: void __user *user_ptr = (__force void __user *)unsafe_ptr;
748:
749: buf[0] = 0;
750:
751: switch (fmt_ptype) {
752: case 's':
753: #ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
754: if ((unsigned long)unsafe_ptr < TASK_SIZE)
755: return strncpy_from_user_nofault(buf, user_ptr, bufsz);
756: fallthrough;
757: #endif
758: case 'k':
759: return strncpy_from_kernel_nofault(buf, unsafe_ptr, bufsz);
760: case 'u':
761: return strncpy_from_user_nofault(buf, user_ptr, bufsz);
762: }
763:
764: return -EINVAL;
765: }
SourceFile:kernel/bpf/helpers.c]
|
| 281/4 |
2026/05/18 19:09 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 14823
output: 40
thoughts: 72
Thoughts: Here's my understanding of the situation, presented from my perspective as an expert:
**Understanding strncpy_from_kernel_nofault**
Okay, so I'm looking at `strncpy_from_kernel_nofault`. The behavior I *think* I recall is that the function returns the length of the string, excluding the null terminator, *but* it copies the null terminator if there's room. Now, let me just confirm that. Wait a second, did I misremember? Doesn't `strncpy_from_kernel_nofault` actually return the *number of bytes copied*, including the null byte if it's copied? This requires immediate verification. I need to consult the documentation or the source code to be absolutely certain I understand its exact return value and what is copied. Let's dig into the implementation to clarify this.
|
| 282/4 |
2026/05/18 19:09 |
tool |
codesearch-definition-comment |
0m
Args: map[ContextFile:kernel/bpf/helpers.c Name:strncpy_from_kernel_nofault]
Results: map[Comment: Kind:function]
|
| 283/4 |
2026/05/18 19:09 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 15136
output: 45
thoughts: 16
Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience:
**Diving into `strncpy_from_kernel_nofault`**
Okay, time to get a bit deeper into this. The discussion has steered us towards `strncpy_from_kernel_nofault`, so let's crack open the source code for that function. I need to refresh my memory on the implementation details and, more importantly, how it handles potential kernel faults. I'm primarily interested in the fault handling, naturally, and any subtle corner cases in the copy itself that might be relevant to the issue we're debugging. I'll be looking for things like buffer overruns, null termination behavior under failure conditions, and any race conditions the implementation might introduce. It's also worth confirming whether it plays nice with ASLR and KASLR on the target. Let's see what we can find.
|
| 284/4 |
2026/05/18 19:09 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:strncpy_from_kernel_nofault]
Results: map[SourceCode: 86: long strncpy_from_kernel_nofault(char *dst, const void *unsafe_addr, long count)
87: {
88: const void *src = unsafe_addr;
89:
90: if (unlikely(count <= 0))
91: return 0;
92: if (!copy_from_kernel_nofault_allowed(unsafe_addr, count))
93: return -ERANGE;
94:
95: pagefault_disable();
96: do {
97: __get_kernel_nofault(dst, src, u8, Efault);
98: dst++;
99: src++;
100: } while (dst[-1] && src - unsafe_addr < count);
101: pagefault_enable();
102:
103: dst[-1] = '\0';
104: return src - unsafe_addr;
105: Efault:
106: pagefault_enable();
107: dst[0] = '\0';
108: return -EFAULT;
109: }
SourceFile:mm/maccess.c]
|
| 285/4 |
2026/05/18 19:09 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 15729
output: 24
thoughts: 2275
Thoughts: **Analyzing Potential Buffer Overflows in `bpf_bprintf_prepare`**
Okay, so I'm digging into this `bpf_bprintf_prepare` function, and I'm seeing some potential buffer overflow vulnerabilities. It starts with the return value of `bpf_trace_copy_string`, which returns the number of bytes read, potentially including the null byte, or `count` if the buffer is full. The key here is that if `count` is returned, the buffer is *filled*, and the last byte should be a null terminator written by `dst[-1] = '\0'`. This is important context for how the `tmp_buf` is advanced later.
Looking at the code, in the `bpf_bprintf_prepare` function, when `fmt[i] == 's'`, the `tmp_buf` is advanced by the return value of `bpf_trace_copy_string`. This means `tmp_buf` increments by at most `tmp_buf_end - tmp_buf` bytes, so it should not exceed `tmp_buf_end` here.
However, when I get to `%pB`, I'm seeing a possible issue. Here's what's going on: `snprintf` is used to format the argument into the `tmp_buf`. If the return value (`err`) of `snprintf` is greater than or equal to `tmp_buf_end - tmp_buf`, `tmp_buf` is advanced past `tmp_buf_end`. To fix this, I thought of clamping the advancement of `tmp_buf`. However, this also does not deal with the case of `snprintf` returning a negative value, which is possible. In that case, should `tmp_buf` advance?
Now, let's look at the `%pI4` case. It's similar. `snprintf` is used, and the buffer is advanced by `err + 1`. The current check uses `tmp_buf_end - tmp_buf < sizeof_cur_ip`. However, the IP address can be formatted to a length which will result in `tmp_buf` exceeding `tmp_buf_end`. Specifically, the code copies the IP address using `copy_from_kernel_nofault` and checks for 4 or 16 bytes for IPv4/IPv6, but `snprintf` can write up to 16 or 40 bytes.
So, the existing bounds checks are insufficient. The fix would involve making sure the increment to `tmp_buf` never pushes it past `tmp_buf_end`. I think the best check is `tmp_buf_end - tmp_buf < (ptrdiff_t)sizeof_cur_arg`, this also works well for the case of a negative return value from `snprintf`. I'll have to make sure to review all places in this function that advance `tmp_buf`, to verify this is consistently handled.
|
| 286/4 |
2026/05/18 19:09 |
tool |
grepper |
0m
Args: map[Expression:tmp_buf_end - tmp_buf <]
Results: map[Output:kernel/bpf/helpers.c=818=int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
--
kernel/bpf/helpers.c-947- sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
kernel/bpf/helpers.c:948: if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
kernel/bpf/helpers.c-949- err = -ENOSPC;
--
kernel/bpf/helpers.c-1040- tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
kernel/bpf/helpers.c:1041: if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
kernel/bpf/helpers.c-1042- err = -ENOSPC;
]
|
| 287/4 |
2026/05/18 19:09 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 18906
output: 23
thoughts: 35
Thoughts: Okay, here's my interpretation of those thoughts, framed as my own:
**Analyzing Potential Buffer Overflows**
Alright, I see a couple of spots in the code where they're checking `tmp_buf_end - tmp_buf` against some value. My immediate concern is, what happens if `tmp_buf_end` and `tmp_buf` are actually *equal*? That subtraction would result in zero, and depending on the comparison, that could lead to unexpected behavior, potentially a buffer overflow or an off-by-one error being missed. I need to carefully consider that edge case. Zero isn't always handled as intuitively as one might think. I'll need to dig into the exact context of those comparisons to determine if this is a genuine vulnerability or an expected condition. I need to make sure to check all possible logic paths to cover that.
|
| 288/4 |
2026/05/18 19:09 |
tool |
grepper |
0m
Args: map[Expression:tmp_buf_end == tmp_buf]
Results: map[Output:kernel/bpf/helpers.c=818=int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
--
kernel/bpf/helpers.c-984-
kernel/bpf/helpers.c:985: if (tmp_buf_end == tmp_buf) {
kernel/bpf/helpers.c-986- err = -ENOSPC;
--
kernel/bpf/helpers.c-1006-
kernel/bpf/helpers.c:1007: if (tmp_buf_end == tmp_buf) {
kernel/bpf/helpers.c-1008- err = -ENOSPC;
]
|
| 289/4 |
2026/05/18 19:09 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 19339
output: 22
thoughts: 1671
Thoughts: **Addressing Potential Out-of-Bounds Writes in `snprintf` Calls**
Okay, so I'm thinking about this `tmp_buf` and `tmp_buf_end` stuff in the context of `snprintf` calls. Initially, my instinct was to change the comparison `tmp_buf_end == tmp_buf` to `tmp_buf >= tmp_buf_end` to catch out-of-bounds situations. But then, I realized the better approach is to *prevent* `tmp_buf` from ever exceeding `tmp_buf_end` in the first place. That's the key.
Let's look at this `%pB` case specifically. The code currently does this:
```c
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
```
The problem is that if `snprintf` returns `err` which is greater than or equal to `tmp_buf_end - tmp_buf`, we have a truncation situation. But, there is a serious issue where if `tmp_buf_end - tmp_buf` evaluates to 0, `snprintf` will still return the required length and in the current code, `tmp_buf` would advance past `tmp_buf_end`.
My initial thought was to correct the advance logic:
```c
err = snprintf(tmp_buf,
(tmp_buf_end > tmp_buf ? tmp_buf_end - tmp_buf : 0),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err < 0) ? 0 :
(err < tmp_buf_end - tmp_buf) ? (err + 1) : (tmp_buf_end - tmp_buf);
```
I was thinking a helper or macro could simplify it, or I could just do the simpler:
```c
err = snprintf(tmp_buf,
(tmp_buf_end > tmp_buf ? tmp_buf_end - tmp_buf : 0),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += err + 1;
```
I thought about just fixing the bounds check to handle `tmp_buf >= tmp_buf_end` and returning `-ENOSPC` when it happened. But here's the crucial point: if `tmp_buf > tmp_buf_end`, `tmp_buf_end - tmp_buf` becomes negative! Since `snprintf` takes a `size_t` argument, this negative value becomes a huge unsigned value, and `snprintf` *will* write out of bounds, because it thinks it has a huge buffer. **This is a critical vulnerability that must be avoided.**
So, the fundamental principle is this: **We MUST NOT call `snprintf` with a negative size, and we MUST NOT advance `tmp_buf` past `tmp_buf_end`.**
Here's the fix for the `%pB` case, ensuring the write bounds are correct:
```c
if (tmp_buf) {
size_t limit = tmp_buf_end > tmp_buf ? tmp_buf_end - tmp_buf : 0;
err = snprintf(tmp_buf, limit,
"%pB",
(void *)(long)raw_args[num_spec]);
if (err < 0)
err = 0;
tmp_buf += min_t(size_t, err + 1, limit);
}
```
Here's why this works: if the `limit == 0`, the `min_t` evaluates to 0 and `tmp_buf` does not move. If `limit > 0` and the write is truncated (`err >= limit`), we still move up to the limit of `tmp_buf_end`. This guarantees safety.
Now, let's turn to `%pI4` and `%pI6` where:
```c
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto
|
| 290/4 |
2026/05/18 19:09 |
tool |
codesearch-find-references |
0m
Args:
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:____bpf_snprintf SourceFile:kernel/bpf/helpers.c SourceLine:1084] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_stream_vprintk SourceFile:kernel/bpf/stream.c SourceLine:240] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:seq_bprintf SourceFile:fs/seq_file.c SourceLine:418] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:seq_buf_bprintf SourceFile:lib/seq_buf.c SourceLine:162]] TruncatedOutput:false]
|
| 291/4 |
2026/05/18 19:09 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 22279
output: 43
thoughts: 21
Thoughts: Here's a summary of my thought process, keeping in mind my expert understanding:
**Deep Dive into `kernel/bpf/helpers.c` - Line 1084**
Okay, time to get specific. I need to understand what's happening at a particular point within the BPF helper functions. My mental map immediately goes to `kernel/bpf/helpers.c`. I know that's the canonical location for those core helper functions. And based on what I'm looking into, I suspect line 1084 is a critical point. I’m zeroing in on that specific line, assuming it holds some essential logic related to the issue I'm troubleshooting, or the feature I'm working with. This line number is a starting point, a pinpoint in the code that I need to dissect. I'm ready to dig in and understand the context surrounding line 1084, because that's where the answer likely resides. I'm focusing my expertise and experience on this segment of the code.
|
| 292/4 |
2026/05/18 19:10 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:____bpf_snprintf]
Results: map[SourceCode:1064: BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
1065: const void *, args, u32, data_len)
1066: {
1067: struct bpf_bprintf_data data = {
1068: .get_bin_args = true,
1069: };
1070: int err, num_args;
1071:
1072: if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
1073: (data_len && !args))
1074: return -EINVAL;
1075: num_args = data_len / 8;
1076:
1077: /* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
1078: * can safely give an unbounded size.
1079: */
1080: err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
1081: if (err < 0)
1082: return err;
1083:
1084: err = bstr_printf(str, str_size, fmt, data.bin_args);
1085:
1086: bpf_bprintf_cleanup(&data);
1087:
1088: return err + 1;
1089: }
SourceFile:kernel/bpf/helpers.c]
|
| 293/4 |
2026/05/18 19:10 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 22999
output: 159
thoughts: 3437
Thoughts: **Analysis of bpf_bprintf_prepare and Potential Buffer Overflows**
Okay, so I've been digging into this `bpf_bprintf_prepare` function, specifically how it handles formatting and preparing data for `bstr_printf`. My immediate thought is that `bstr_printf` does indeed use `data.bin_args`, so we need to ensure the data is properly formatted and safe within this context.
The most concerning area is the IP address handling, specifically within the `%pI` specifiers. The existing code uses `snprintf` to format IP addresses as strings, which is fine in principle. The code correctly copies the raw IP address into `cur_ip`, but that is 4 or 16 bytes depending on whether the format is IPv4 or IPv6. The problem, as I see it, comes from how the buffer size is determined and how `snprintf` is called. It appears there is a potential for writing beyond the allocated buffer, `tmp_buf`, if the return value from `snprintf` exceeds the available space.
The original check, `if (tmp_buf_end - tmp_buf < sizeof_cur_ip)`, checks if there's enough space for the raw IP address, not the *formatted* string. This leads to the fundamental flaw: the code allocates space based on the *raw* IP address size but then uses `snprintf`, which potentially writes a *longer* string to that buffer. We need to reserve space for the *formatted* string, not the raw binary data.
Moreover, if `snprintf` writes more than the remaining space, it truncates. If this happens, `err` holds the number of bytes that would have been written, but `tmp_buf` advances by `err + 1`, potentially exceeding the buffer's bounds. The bounds checks are not sufficient because they are based on the *raw* IP size, not the size of the formatted string.
So, I think we need to rethink this approach. I'm leaning toward using the same `limit` logic for `snprintf` that prevents the buffer from exceeding the allocated space.
```c
size_t limit = tmp_buf_end > tmp_buf ? tmp_buf_end - tmp_buf : 0;
err = snprintf(tmp_buf, limit, ip_spec, &cur_ip);
if (err < 0)
err = 0;
tmp_buf += min_t(size_t, err + 1, limit);
```
That should be safe! Using `min_t` will ensure `tmp_buf` will never exceed the end of the buffer. The resulting formatted IP string might be truncated, but this is acceptable behavior, and consistent with the truncation in `bpf_trace_copy_string`.
I've also got to confirm that the code is safely handling all the buffer increments with the `%s` format specifier. We're using `bpf_trace_copy_string`, which copies strings from kernel space. In this case, `bpf_trace_copy_string` correctly returns the written length including the null byte. However, there is a potentially dangerous case, if `tmp_buf_end - tmp_buf` is negative. This would create a large `size_t bufsz` which leads to a write out-of-bounds in `bpf_trace_copy_string`. I also see that there *is* a check that must handle the negative case. So if `tmp_buf_end - tmp_buf` *is* negative, the existing code should return `-ENOSPC`, which is the correct response.
And then I've confirmed that the alignment with `PTR_ALIGN` when formatting integer args is safe because the buffer is a multiple of the alignment size. Even if `tmp_buf == tmp_buf_end`, `PTR_ALIGN` will align to exactly `tmp_buf_end`.
Finally, the most general thing is that any pointer arithmetic comparison should correctly handle the corner cases. Instead of trying to cast to different signed types, a better, clearer, and safer solution is to check:
`if (tmp_buf + sizeof_cur_arg > tmp_buf_end)`
This will avoid any signedness errors that could arise.
So, to summarize, I need to focus on these points to ensure the code is robust:
* **Fix `%pI` formatting**: Ensure `snprintf` doesn't write out of bounds. The safest approach is to use the `limit` with `min_t` logic that I showed above.
* **Carefully review all potential out-of-bounds writes**: Make sure that the buffer is properly checked before calling potentially problematic functions and ensure the bounds checks cover all corner cases.
*
|
| 294/4 |
2026/05/18 19:10 |
tool |
codeeditor |
0m
Error:
Args: map[CurrentCode: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
} NewCode: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
} SourceFile:kernel/bpf/helpers.c]
Results:
|
| 295/4 |
2026/05/18 19:10 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 27640
output: 159
thoughts: 7
Thoughts: Okay, here's the summary, formatted as requested:
**Internal Review and Refinement**
Wait, something's not quite right. I have a feeling I need to adjust this. I need to make some changes to the current approach. It needs further refinement. Time to go back and reassess the situation.
|
| 296/4 |
2026/05/18 19:10 |
tool |
codeeditor |
0m
Args: map[CurrentCode: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
} NewCode: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf + sizeof_cur_ip > tmp_buf_end) {
err = -ENOSPC;
goto out;
} SourceFile:kernel/bpf/helpers.c]
Results:
|
| 297/4 |
2026/05/18 19:10 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 27880
output: 167
thoughts: 8
Thoughts: Okay, here's the summary, assuming I'm the expert and the "other one" refers to something technical we're working on:
**Re-evaluating the Current Approach**
Alright, I'm looking at this current implementation...and I'm thinking, perhaps it's not the optimal path. The constraints suggest we could potentially derive greater efficiencies if we were to adapt the parameters of the adjacent system. We have a solid baseline here, but something is telling me that we could go further. It would require modifying the adjacent system. If we can reconfigure that, then we can take advantage of the improvements it brings. Let's dig in and see what's possible.
|
| 298/4 |
2026/05/18 19:10 |
tool |
codeeditor |
0m
Args: map[CurrentCode: if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
} NewCode: if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
} SourceFile:kernel/bpf/helpers.c]
Results:
|
| 299/4 |
2026/05/18 19:10 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 28207
output: 219
thoughts: 28
Thoughts: Here's a summary of my thinking, expressed in the first person:
**Refining a Comparison for Buffer Handling**
Okay, so I'm looking at this condition: `tmp_buf_end == tmp_buf`. It's used in this buffer management logic, and it's making me pause. While it might seem logically correct in some contexts, I'm pretty sure it's susceptible to a subtle bug. It feels too specific.
What I'm envisioning is a scenario where `tmp_buf_end` and `tmp_buf` might be valid, *but* the comparison logic will not behave how I'm intending.
I'm starting to think a more robust approach would be to use a greater-than-or-equal-to comparison. Switching to `tmp_buf >= tmp_buf_end` would provide a better signal of that state, handling edge cases more gracefully, specifically when those buffer pointers are equal, which, in the context I'm considering, likely signifies something meaningful, such as an empty buffer or a boundary condition. This change should make the code more resilient and less prone to unexpected behavior if there are even minor shifts to buffer memory allocation or initialization.
|
| 300/4 |
2026/05/18 19:10 |
tool |
codeeditor |
0m
Args: map[CurrentCode: if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf); NewCode: if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf); SourceFile:kernel/bpf/helpers.c]
Results:
|
| 301/4 |
2026/05/18 19:10 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 28715
output: 159
thoughts: 20
Thoughts: Here's my take on those thoughts, tailored for an expert audience:
**Deduction on Buffer Boundary Condition**
Okay, so I'm processing this code, and I see a potentially tricky boundary condition in this `%c` formatting scenario. The critical piece jumps out: `tmp_buf_end == tmp_buf`. It's a precise check, and it's reminding me that we have to be *absolutely* sure we handle the case where the temporary buffer is, in fact, empty *before* writing even a single character. This isn't just a stylistic quirk; this hints at something critical about buffer management during this specific format operation. It’s making me think about edge cases. Are we initializing the `tmp_buf` correctly? What happens if that buffer is very small? I'm quickly reviewing how we handle overflow elsewhere, because a direct `tmp_buf_end == tmp_buf` check on its own would likely require further verification to prevent out-of-bounds writes. The subtlety of a character write, requiring this specific check, is raising concerns and makes me want to dig deeper into the actual buffer sizing and related protection mechanisms.
|
| 302/4 |
2026/05/18 19:10 |
tool |
codeeditor |
0m
Args: map[CurrentCode: } else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
} NewCode: } else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
} SourceFile:kernel/bpf/helpers.c]
Results:
|
| 303/4 |
2026/05/18 19:10 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 29144
output: 262
thoughts: 18
Thoughts: Okay, let's get down to brass tacks.
**Fixing Those Pesky `snprintf` Calls**
Alright, time to get this sorted. My immediate focus is on the `snprintf` calls, and specifically, that `%pB` format specifier. I've got to ensure these are handled correctly and securely. Let's dig in.
|
| 304/4 |
2026/05/18 19:10 |
tool |
codeeditor |
0m
Args: map[CurrentCode: if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++; NewCode: if (fmt[i + 1] == 'B') {
if (tmp_buf) {
size_t limit = tmp_buf < tmp_buf_end ? tmp_buf_end - tmp_buf : 0;
err = snprintf(tmp_buf, limit,
"%pB",
(void *)(long)raw_args[num_spec]);
if (err < 0)
err = 0;
tmp_buf += min_t(size_t, err + 1, limit);
}
i++; SourceFile:kernel/bpf/helpers.c]
Results:
|
| 305/4 |
2026/05/18 19:10 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 29509
output: 17
thoughts: 52
Thoughts: Okay, let me think this through.
**Checking the Availability of `min_t` in the BPF Helper Functions**
Right, I've just written something using `min_t`, and now I need to make sure I haven't jumped the gun. Specifically, I'm using `min_t` in my code, but is it actually defined and available within the `kernel/bpf/helpers.c` context? I need to quickly verify that `min_t` is, in fact, implemented or declared within that particular file. I can't assume its existence; this is critical to avoid any compilation or runtime errors down the line. It's a fundamental check before I proceed any further. So, let me go have a look in that file.
|
| 306/4 |
2026/05/18 19:11 |
tool |
grepper |
0m
Args:
Results: map[Output:
Full output is too long, showing 500 out of 34960 lines.
Use more precise expression if possible.
[Documentation/ABI/testing/sysfs-bus-iio=2281=Description:
--
Documentation/ABI/testing/sysfs-bus-iio-2284-
Documentation/ABI/testing/sysfs-bus-iio:2285:What: /sys/.../events/in_accel_gesture_tap_maxtomin_time
Documentation/ABI/testing/sysfs-bus-iio-2286-KernelVersion: 6.1
--
Documentation/ABI/testing/sysfs-bus-iio=2288=Description:
--
Documentation/ABI/testing/sysfs-bus-iio-2292-
Documentation/ABI/testing/sysfs-bus-iio:2293:What: /sys/.../events/in_accel_gesture_tap_maxtomin_time_available
Documentation/ABI/testing/sysfs-bus-iio-2294-KernelVersion: 6.1
--
Documentation/ABI/testing/sysfs-class-net-cdc_ncm:1:What: /sys/class/net/<iface>/cdc_ncm/min_tx_pkt
Documentation/ABI/testing/sysfs-class-net-cdc_ncm-2-Date: May 2014
--
Documentation/ABI/testing/sysfs-class-powercap=134=Description:
--
Documentation/ABI/testing/sysfs-class-powercap-137-
Documentation/ABI/testing/sysfs-class-powercap:138:What: /sys/class/powercap/.../<power zone>/constraint_X_min_time_window_us
Documentation/ABI/testing/sysfs-class-powercap-139-Date: September 2013
--
Documentation/ABI/testing/sysfs-fs-ext4=21=Description:
--
Documentation/ABI/testing/sysfs-fs-ext4-24-
Documentation/ABI/testing/sysfs-fs-ext4:25:What: /sys/fs/ext4/<disk>/mb_min_to_scan
Documentation/ABI/testing/sysfs-fs-ext4-26-Date: March 2008
--
Documentation/admin-guide/ext4.rst=448=Files in /sys/fs/ext4/<devname>:
--
Documentation/admin-guide/ext4.rst-484-
Documentation/admin-guide/ext4.rst:485: mb_min_to_scan
Documentation/admin-guide/ext4.rst-486- The minimum number of extents the multiblock allocator will search to
--
Documentation/admin-guide/mm/multigen_lru.rst=69=majority of laptop and desktop users who do not have ``oomd``.
Documentation/admin-guide/mm/multigen_lru.rst-70-
Documentation/admin-guide/mm/multigen_lru.rst:71:Users can write ``N`` to ``min_ttl_ms`` to prevent the working set of
Documentation/admin-guide/mm/multigen_lru.rst-72-``N`` milliseconds from getting evicted. The OOM killer is triggered
--
Documentation/arch/arm/stm32/stm32-dma-mdma-chaining.rst=210=Driver updates for STM32 DMA-MDMA chaining support in foo driver
--
Documentation/arch/arm/stm32/stm32-dma-mdma-chaining.rst-271- for_each_sg(mdma_sgt.sgl, s, mdma_sgt.nents, i) {
Documentation/arch/arm/stm32/stm32-dma-mdma-chaining.rst:272: size_t bytes = min_t(size_t, len, sram_period);
Documentation/arch/arm/stm32/stm32-dma-mdma-chaining.rst-273-
--
Documentation/driver-api/media/drivers/ccs/ccs-regs.asc=890=PHY_ctrl_capability_3 0x165c v1.1
Documentation/driver-api/media/drivers/ccs/ccs-regs.asc-891-- b dphy_timing_not_multiple 0
Documentation/driver-api/media/drivers/ccs/ccs-regs.asc:892:- b dphy_min_timing_value_1 1
Documentation/driver-api/media/drivers/ccs/ccs-regs.asc-893-- b twakeup_supported 2
--
Documentation/driver-api/media/drivers/ccs/ccs-regs.asc-896-- b cphy_timing_not_multiple 5
Documentation/driver-api/media/drivers/ccs/ccs-regs.asc:897:- b cphy_min_timing_value_1 6
Documentation/driver-api/media/drivers/ccs/ccs-regs.asc-898-dphy_sf 0x165d v1.1
--
Documentation/driver-api/media/drivers/ccs/ccs-regs.asc=1013=reset_max_delay 0x1a10 v1.1
Documentation/driver-api/media/drivers/ccs/ccs-regs.asc:1014:reset_min_time 0x1a11 v1.1
Documentation/driver-api/media/drivers/ccs/ccs-regs.asc-1015-
--
Documentation/driver-api/memory-devices/ti-emif.rst=39=information can be passed through platform data (struct emif_platform_data)
--
Documentation/driver-api/memory-devices/ti-emif.rst-41-- DDR device details: 'struct ddr_device_info'
Documentation/driver-api/memory-devices/ti-emif.rst:42:- Device AC timings: 'struct lpddr2_timings' and 'struct lpddr2_min_tck'
Documentation/driver-api/memory-devices/ti-emif.rst-43-- Custom configurations: customizable policy options through
--
Documentation/driver-api/usb/writing_usb_driver.rst=188=subsystem. This can be seen in the following code::
--
Documentation/driver-api/usb/writing_usb_driver.rst-190- /* we can only write as much as 1 urb will hold */
Documentation/driver-api/usb/writing_usb_driver.rst:191: size_t writesize = min_t(size_t, count, MAX_TRANSFER);
Documentation/driver-api/usb/writing_usb_driver.rst-192-
--
Documentation/mm/multigen_lru.rst=145=Working set protection
Documentation/mm/multigen_lru.rst-146-----------------------
Documentation/mm/multigen_lru.rst:147:Each generation is timestamped at birth. If ``lru_gen_min_ttl`` is
Documentation/mm/multigen_lru.rst-148-set, an ``lruvec`` is protected from the eviction when its oldest
Documentation/mm/multigen_lru.rst:149:generation was born within ``lru_gen_min_ttl`` milliseconds. In other
Documentation/mm/multigen_lru.rst:150:words, it prevents the working set of ``lru_gen_min_ttl`` milliseconds
Documentation/mm/multigen_lru.rst-151-from getting evicted. The OOM killer is triggered if this working set
--
Documentation/networking/ip-sysctl.rst=1087=tcp_timestamps - INTEGER
--
Documentation/networking/ip-sysctl.rst-1096-
Documentation/networking/ip-sysctl.rst:1097:tcp_min_tso_segs - INTEGER
Documentation/networking/ip-sysctl.rst-1098- Minimal number of segments per TSO frame.
--
Documentation/networking/net_cachelines/inet_sock.rst=24=u8 tos read_mostly ip_queue_xmit
Documentation/networking/net_cachelines/inet_sock.rst:25:u8 min_ttl
Documentation/networking/net_cachelines/inet_sock.rst-26-u8 mc_ttl
--
Documentation/networking/net_cachelines/netns_ipv4_sysctl.rst=113=int sysctl_tcp_min_rtt_wlen read_mostly tcp_ack_update_rtt
Documentation/networking/net_cachelines/netns_ipv4_sysctl.rst:114:u8 sysctl_tcp_min_tso_segs unlikely(icsk_ca_ops-written)
Documentation/networking/net_cachelines/netns_ipv4_sysctl.rst-115-u8 sysctl_tcp_tso_rtt_log read_mostly tcp_tso_autosize
--
Documentation/power/powercap/powercap.rst=234=constraint_X_max_time_window_us(ro)
--
Documentation/power/powercap/powercap.rst-236-
Documentation/power/powercap/powercap.rst:237:constraint_X_min_time_window_us(ro)
Documentation/power/powercap/powercap.rst-238- Minimum allowed time window in micro seconds.
--
Documentation/userspace-api/media/rc/lirc-get-timeout.rst-3-
Documentation/userspace-api/media/rc/lirc-get-timeout.rst:4:.. _lirc_get_min_timeout:
Documentation/userspace-api/media/rc/lirc-get-timeout.rst-5-.. _lirc_get_max_timeout:
--
Documentation/watchdog/convert_drivers_to_kernel_api.rst=21=Here is a overview of the functions and probably needed actions:
--
Documentation/watchdog/convert_drivers_to_kernel_api.rst-57- and a set_timeout-callback has to be defined. The core will also
Documentation/watchdog/convert_drivers_to_kernel_api.rst:58: do limit-checking, if min_timeout and max_timeout in the watchdog
Documentation/watchdog/convert_drivers_to_kernel_api.rst-59- device are set. All is optional.
--
Documentation/watchdog/watchdog-kernel-api.rst=45=The watchdog device structure looks like this::
--
Documentation/watchdog/watchdog-kernel-api.rst-56- unsigned int pretimeout;
Documentation/watchdog/watchdog-kernel-api.rst:57: unsigned int min_timeout;
Documentation/watchdog/watchdog-kernel-api.rst-58- unsigned int max_timeout;
--
Documentation/watchdog/watchdog-kernel-api.rst=69=It contains following fields:
--
Documentation/watchdog/watchdog-kernel-api.rst-86-* pretimeout: the watchdog timer's pretimeout value (in seconds).
Documentation/watchdog/watchdog-kernel-api.rst:87:* min_timeout: the watchdog timer's minimum timeout value (in seconds).
Documentation/watchdog/watchdog-kernel-api.rst-88- If set, the minimum configurable value for 'timeout'.
--
arch/alpha/kernel/ptrace.c=280=long arch_ptrace(struct task_struct *child, long request,
--
arch/alpha/kernel/ptrace.c-336- regs = task_pt_regs(child);
arch/alpha/kernel/ptrace.c:337: len = min_t(size_t, iov.iov_len, sizeof(*regs));
arch/alpha/kernel/ptrace.c-338-
--
arch/alpha/kernel/srmcons.c=95=srmcons_do_write(struct tty_port *port, const u8 *buf, size_t count)
--
arch/alpha/kernel/srmcons.c-105- */
arch/alpha/kernel/srmcons.c:106: for (c = 0; c < min_t(size_t, 128U, count) && !need_cr; c++)
arch/alpha/kernel/srmcons.c-107- if (buf[c] == '\n')
--
arch/arc/kernel/unwind.c=888=int arc_unwind(struct unwind_frame_info *frame)
--
arch/arc/kernel/unwind.c-1173- cfa = FRAME_REG(state.cfa.reg, unsigned long) + state.cfa.offs;
arch/arc/kernel/unwind.c:1174: startLoc = min_t(unsigned long, UNW_SP(frame), cfa);
arch/arc/kernel/unwind.c-1175- endLoc = max_t(unsigned long, UNW_SP(frame), cfa);
--
arch/arc/mm/cache.c=787=void flush_icache_range(unsigned long kstart, unsigned long kend)
--
arch/arc/mm/cache.c-829- phy = (pfn << PAGE_SHIFT) + off;
arch/arc/mm/cache.c:830: sz = min_t(unsigned int, tot_sz, PAGE_SIZE - off);
arch/arc/mm/cache.c-831- __sync_icache_dcache(phy, kstart, sz);
--
arch/arm/mach-davinci/sram.c=49=static int __init sram_init(void)
--
arch/arm/mach-davinci/sram.c-56- if (len) {
arch/arm/mach-davinci/sram.c:57: len = min_t(unsigned, len, SRAM_SIZE);
arch/arm/mach-davinci/sram.c-58- sram_pool = gen_pool_create(ilog2(SRAM_GRANULARITY), -1);
--
arch/arm/mach-orion5x/ts78xx-setup.c=152=static void ts78xx_ts_nand_write_buf(struct nand_chip *chip,
--
arch/arm/mach-orion5x/ts78xx-setup.c-159- if (off) {
arch/arm/mach-orion5x/ts78xx-setup.c:160: sz = min_t(int, 4 - off, len);
arch/arm/mach-orion5x/ts78xx-setup.c-161- writesb(io_base, buf, sz);
--
arch/arm/mach-orion5x/ts78xx-setup.c=178=static void ts78xx_ts_nand_read_buf(struct nand_chip *chip,
--
arch/arm/mach-orion5x/ts78xx-setup.c-185- if (off) {
arch/arm/mach-orion5x/ts78xx-setup.c:186: sz = min_t(int, 4 - off, len);
arch/arm/mach-orion5x/ts78xx-setup.c-187- readsb(io_base, buf, sz);
--
arch/arm/mm/cache-uniphier.c=179=static void __uniphier_cache_maint_range(struct uniphier_cache_data *data,
--
arch/arm/mm/cache-uniphier.c-205- while (size) {
arch/arm/mm/cache-uniphier.c:206: unsigned long chunk_size = min_t(unsigned long, size,
arch/arm/mm/cache-uniphier.c-207- data->range_op_max_size);
--
arch/arm/mm/mmu.c=1178=void __init adjust_lowmem_bounds(void)
--
arch/arm/mm/mmu.c-1216- */
arch/arm/mm/mmu.c:1217: lowmem_limit = min_t(u64,
arch/arm/mm/mmu.c-1218- vmalloc_limit,
--
arch/arm64/include/asm/archrandom.h=82=static inline size_t __must_check arch_get_random_seed_longs(unsigned long *v, size_t max_longs)
--
arch/arm64/include/asm/archrandom.h-95-
arch/arm64/include/asm/archrandom.h:96: max_longs = min_t(size_t, 3, max_longs);
arch/arm64/include/asm/archrandom.h-97- arm_smccc_1_1_invoke(ARM_SMCCC_TRNG_RND64, max_longs * 64, &res);
--
arch/arm64/kernel/patching.c=107=static void *__text_poke(text_poke_f func, void *addr, void *src, size_t len)
--
arch/arm64/kernel/patching.c-118- ptr = addr + patched;
arch/arm64/kernel/patching.c:119: size = min_t(size_t, PAGE_SIZE - offset_in_page(ptr),
arch/arm64/kernel/patching.c-120- len - patched);
--
arch/arm64/kernel/topology.c=117=static void amu_scale_freq_tick(void)
--
arch/arm64/kernel/topology.c-151-
arch/arm64/kernel/topology.c:152: scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE);
arch/arm64/kernel/topology.c-153- this_cpu_write(arch_freq_scale, (unsigned long)scale);
--
arch/arm64/kvm/arm.c=363=int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
--
arch/arm64/kvm/arm.c-408- */
arch/arm64/kvm/arm.c:409: r = min_t(unsigned int, num_online_cpus(),
arch/arm64/kvm/arm.c-410- kvm_arm_default_max_vcpus());
--
arch/arm64/kvm/mmu.c=1745=static short kvm_s2_resolve_vma_size(const struct kvm_s2_fault_desc *s2fd,
--
arch/arm64/kvm/mmu.c-1804- s2vi->max_map_size = max_map_size;
arch/arm64/kvm/mmu.c:1805: vma_shift = min_t(short, vma_shift, __ffs(max_map_size));
arch/arm64/kvm/mmu.c-1806- }
--
arch/arm64/kvm/vgic/vgic-its.c=2011=static u32 compute_next_devid_offset(struct list_head *h,
--
arch/arm64/kvm/vgic/vgic-its.c-2021-
arch/arm64/kvm/vgic/vgic-its.c:2022: return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
arch/arm64/kvm/vgic/vgic-its.c-2023-}
--
arch/arm64/kvm/vgic/vgic-its.c=2025=static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
--
arch/arm64/kvm/vgic/vgic-its.c-2034-
arch/arm64/kvm/vgic/vgic-its.c:2035: return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
arch/arm64/kvm/vgic/vgic-its.c-2036-}
--
arch/arm64/mm/gcs.c=18=static unsigned long gcs_size(unsigned long size)
--
arch/arm64/mm/gcs.c-23- /* Allocate RLIMIT_STACK/2 with limits of PAGE_SIZE..2G */
arch/arm64/mm/gcs.c:24: size = PAGE_ALIGN(min_t(unsigned long long,
arch/arm64/mm/gcs.c-25- rlimit(RLIMIT_STACK) / 2, SZ_2G));
--
arch/arm64/mm/init.c=193=void __init arm64_memblock_init(void)
--
arch/arm64/mm/init.c-207- pr_info("Capping linear region to 51 bits for KVM in nVHE mode on LVA capable hardware.\n");
arch/arm64/mm/init.c:208: linear_region_size = min_t(u64, linear_region_size, BIT(51));
arch/arm64/mm/init.c-209- }
--
arch/loongarch/kernel/kdebugfs.c=20=static ssize_t sfb_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
--
arch/loongarch/kernel/kdebugfs.c-32- s -= *ppos;
arch/loongarch/kernel/kdebugfs.c:33: s = min_t(u32, s, count);
arch/loongarch/kernel/kdebugfs.c-34-
--
arch/loongarch/kernel/kdebugfs.c=93=static ssize_t tso_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
--
arch/loongarch/kernel/kdebugfs.c-111- s -= *ppos;
arch/loongarch/kernel/kdebugfs.c:112: s = min_t(u32, s, count);
arch/loongarch/kernel/kdebugfs.c-113-
--
arch/loongarch/kernel/ptrace.c=308=static int simd_set(struct task_struct *target,
--
arch/loongarch/kernel/ptrace.c-325- /* Copy as much context as possible */
arch/loongarch/kernel/ptrace.c:326: cp_sz = min_t(unsigned int, regset->size,
arch/loongarch/kernel/ptrace.c-327- sizeof(target->thread.fpu.fpr[0]));
--
arch/loongarch/kvm/trace.h=173=TRACE_EVENT(kvm_iocsr,
--
arch/loongarch/kvm/trace.h-190- memcpy(&__entry->val, val,
arch/loongarch/kvm/trace.h:191: min_t(u32, sizeof(__entry->val), len));
arch/loongarch/kvm/trace.h-192- ),
--
arch/loongarch/kvm/vm.c=112=int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
--
arch/loongarch/kvm/vm.c-127- case KVM_CAP_NR_VCPUS:
arch/loongarch/kvm/vm.c:128: r = min_t(unsigned int, num_online_cpus(), KVM_MAX_VCPUS);
arch/loongarch/kvm/vm.c-129- break;
--
arch/loongarch/net/bpf_jit.c=1809=static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
--
arch/loongarch/net/bpf_jit.c-2020- if (flags & BPF_TRAMP_F_CALL_ORIG) {
arch/loongarch/net/bpf_jit.c:2021: restore_args(ctx, min_t(int, nr_arg_slots, LOONGARCH_MAX_REG_ARGS), args_off);
arch/loongarch/net/bpf_jit.c-2022- restore_stk_args(ctx, nr_arg_slots - LOONGARCH_MAX_REG_ARGS, args_off, stk_args_off);
--
arch/loongarch/net/bpf_jit.c-2063- if (flags & BPF_TRAMP_F_RESTORE_REGS)
arch/loongarch/net/bpf_jit.c:2064: restore_args(ctx, min_t(int, nr_arg_slots, LOONGARCH_MAX_REG_ARGS), args_off);
arch/loongarch/net/bpf_jit.c-2065-
--
arch/m68k/atari/time.c=98=static void mste_read(struct MSTE_RTC *val)
--
arch/m68k/atari/time.c-102- COPY(sec_ones) ; COPY(sec_tens) ; COPY(min_ones) ;
arch/m68k/atari/time.c:103: COPY(min_tens) ; COPY(hr_ones) ; COPY(hr_tens) ;
arch/m68k/atari/time.c-104- COPY(weekday) ; COPY(day_ones) ; COPY(day_tens) ;
--
arch/m68k/atari/time.c=112=static void mste_write(struct MSTE_RTC *val)
--
arch/m68k/atari/time.c-116- COPY(sec_ones) ; COPY(sec_tens) ; COPY(min_ones) ;
arch/m68k/atari/time.c:117: COPY(min_tens) ; COPY(hr_ones) ; COPY(hr_tens) ;
arch/m68k/atari/time.c-118- COPY(weekday) ; COPY(day_ones) ; COPY(day_tens) ;
--
arch/m68k/atari/time.c=142=int atari_mste_hwclk( int op, struct rtc_time *t )
--
arch/m68k/atari/time.c-157- val.min_ones = t->tm_min % 10;
arch/m68k/atari/time.c:158: val.min_tens = t->tm_min / 10;
arch/m68k/atari/time.c-159- hour = t->tm_hour;
--
arch/m68k/atari/time.c-183- t->tm_sec = val.sec_ones + val.sec_tens * 10;
arch/m68k/atari/time.c:184: t->tm_min = val.min_ones + val.min_tens * 10;
arch/m68k/atari/time.c-185- hour = val.hr_ones + val.hr_tens * 10;
--
arch/m68k/include/asm/atarihw.h=770=struct MSTE_RTC {
--
arch/m68k/include/asm/atarihw.h-776- u_char dummy3;
arch/m68k/include/asm/atarihw.h:777: u_char min_tens;
arch/m68k/include/asm/atarihw.h-778- u_char dummy4;
--
arch/mips/generic/yamon-dt.c=43=static unsigned int __init gen_fdt_mem_array(
--
arch/mips/generic/yamon-dt.c-60- /* How much of the remaining RAM fits in the next region? */
arch/mips/generic/yamon-dt.c:61: size = min_t(unsigned long, memsize, mr->size);
arch/mips/generic/yamon-dt.c-62- memsize -= size;
--
arch/mips/generic/yamon-dt.c-69- /* Discard the next mr->discard bytes */
arch/mips/generic/yamon-dt.c:70: memsize -= min_t(unsigned long, memsize, mr->discard);
arch/mips/generic/yamon-dt.c-71- }
--
arch/mips/include/asm/octeon/cvmx-pow-defs.h=583=union cvmx_pow_qos_thrx {
--
arch/mips/include/asm/octeon/cvmx-pow-defs.h-593- uint64_t reserved_11_11:1;
arch/mips/include/asm/octeon/cvmx-pow-defs.h:594: uint64_t min_thr:11;
arch/mips/include/asm/octeon/cvmx-pow-defs.h-595-#else
arch/mips/include/asm/octeon/cvmx-pow-defs.h:596: uint64_t min_thr:11;
arch/mips/include/asm/octeon/cvmx-pow-defs.h-597- uint64_t reserved_11_11:1;
--
arch/mips/include/asm/octeon/cvmx-pow-defs.h-616- uint64_t reserved_6_11:6;
arch/mips/include/asm/octeon/cvmx-pow-defs.h:617: uint64_t min_thr:6;
arch/mips/include/asm/octeon/cvmx-pow-defs.h-618-#else
arch/mips/include/asm/octeon/cvmx-pow-defs.h:619: uint64_t min_thr:6;
arch/mips/include/asm/octeon/cvmx-pow-defs.h-620- uint64_t reserved_6_11:6;
--
arch/mips/include/asm/octeon/cvmx-pow-defs.h-641- uint64_t reserved_8_11:4;
arch/mips/include/asm/octeon/cvmx-pow-defs.h:642: uint64_t min_thr:8;
arch/mips/include/asm/octeon/cvmx-pow-defs.h-643-#else
arch/mips/include/asm/octeon/cvmx-pow-defs.h:644: uint64_t min_thr:8;
arch/mips/include/asm/octeon/cvmx-pow-defs.h-645- uint64_t reserved_8_11:4;
--
arch/mips/include/asm/octeon/cvmx-pow-defs.h-666- uint64_t reserved_9_11:3;
arch/mips/include/asm/octeon/cvmx-pow-defs.h:667: uint64_t min_thr:9;
arch/mips/include/asm/octeon/cvmx-pow-defs.h-668-#else
arch/mips/include/asm/octeon/cvmx-pow-defs.h:669: uint64_t min_thr:9;
arch/mips/include/asm/octeon/cvmx-pow-defs.h-670- uint64_t reserved_9_11:3;
--
arch/mips/include/asm/octeon/cvmx-pow-defs.h-691- uint64_t reserved_10_11:2;
arch/mips/include/asm/octeon/cvmx-pow-defs.h:692: uint64_t min_thr:10;
arch/mips/include/asm/octeon/cvmx-pow-defs.h-693-#else
arch/mips/include/asm/octeon/cvmx-pow-defs.h:694: uint64_t min_thr:10;
arch/mips/include/asm/octeon/cvmx-pow-defs.h-695- uint64_t reserved_10_11:2;
--
arch/mips/kernel/cevt-r4k.c=49=static unsigned int calculate_min_delta(void)
--
arch/mips/kernel/cevt-r4k.c-71- if (cnt < buf1[k]) {
arch/mips/kernel/cevt-r4k.c:72: l = min_t(unsigned int,
arch/mips/kernel/cevt-r4k.c-73- j, ARRAY_SIZE(buf1) - 1);
--
arch/mips/kernel/cevt-r4k.c-85- if (buf1[ARRAY_SIZE(buf1) - 1] < buf2[k]) {
arch/mips/kernel/cevt-r4k.c:86: l = min_t(unsigned int,
arch/mips/kernel/cevt-r4k.c-87- i, ARRAY_SIZE(buf2) - 1);
--
arch/mips/kernel/ptrace.c=652=static int msa_set(struct task_struct *target,
--
arch/mips/kernel/ptrace.c-670- /* Copy as much context as possible */
arch/mips/kernel/ptrace.c:671: cp_sz = min_t(unsigned int, regset->size,
arch/mips/kernel/ptrace.c-672- sizeof(target->thread.fpu.fpr[0]));
--
arch/mips/kernel/rtlx.c=284=ssize_t rtlx_write(int index, const void __user *buffer, size_t count)
--
arch/mips/kernel/rtlx.c-300- /* total number of bytes to copy */
arch/mips/kernel/rtlx.c:301: count = min_t(size_t, count, write_spacefree(rt_read, rt->rt_write,
arch/mips/kernel/rtlx.c-302- rt->buffer_size));
--
arch/mips/kernel/smp-cps.c=211=static void __init cps_smp_setup(void)
--
arch/mips/kernel/smp-cps.c-240-
arch/mips/kernel/smp-cps.c:241: for (v = 0; v < min_t(int, core_vpes, NR_CPUS - nvpes); v++) {
arch/mips/kernel/smp-cps.c-242- cpu_set_cluster(&cpu_data[nvpes + v], cl);
--
arch/mips/kernel/smp-cps.c-254- /* Indicate present CPUs (CPU being synonymous with VPE) */
arch/mips/kernel/smp-cps.c:255: for (v = 0; v < min_t(unsigned, nvpes, NR_CPUS); v++) {
arch/mips/kernel/smp-cps.c-256- set_cpu_possible(v, true);
--
arch/mips/kvm/mips.c=1030=int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
--
arch/mips/kvm/mips.c-1041- case KVM_CAP_NR_VCPUS:
arch/mips/kvm/mips.c:1042: r = min_t(unsigned int, num_online_cpus(), KVM_MAX_VCPUS);
arch/mips/kvm/mips.c-1043- break;
--
arch/mips/mm/tlbex.c=2494=static void check_pabits(void)
--
arch/mips/mm/tlbex.c-2520- /* minus the RI & XI bits */
arch/mips/mm/tlbex.c:2521: fillbits -= min_t(unsigned, fillbits, 2);
arch/mips/mm/tlbex.c-2522-
--
arch/mips/mti-malta/malta-dtshim.c=70=static unsigned __init gen_fdt_mem_array(__be32 *mem_array, unsigned long size,
--
arch/mips/mti-malta/malta-dtshim.c-88-
arch/mips/mti-malta/malta-dtshim.c:89: size_preio = min_t(unsigned long, size, SZ_256M);
arch/mips/mti-malta/malta-dtshim.c-90- mem_array[1] = cpu_to_be32(size_preio);
--
arch/parisc/kernel/firmware.c=1361=int pdc_iodc_print(const unsigned char *str, unsigned count)
--
arch/parisc/kernel/firmware.c-1365-
arch/parisc/kernel/firmware.c:1366: count = min_t(unsigned int, count, sizeof(iodc_dbuf));
arch/parisc/kernel/firmware.c-1367-
--
arch/powerpc/boot/types.h=26=typedef s64 int64_t;
--
arch/powerpc/boot/types.h-39-
arch/powerpc/boot/types.h:40:#define min_t(type, a, b) min(((type) a), ((type) b))
arch/powerpc/boot/types.h-41-#define max_t(type, a, b) max(((type) a), ((type) b))
--
arch/powerpc/crypto/aes-spe-glue.c=132=static int ppc_ecb_crypt(struct skcipher_request *req, bool enc)
--
arch/powerpc/crypto/aes-spe-glue.c-142- while ((nbytes = walk.nbytes) != 0) {
arch/powerpc/crypto/aes-spe-glue.c:143: nbytes = min_t(unsigned int, nbytes, MAX_BYTES);
arch/powerpc/crypto/aes-spe-glue.c-144- nbytes = round_down(nbytes, AES_BLOCK_SIZE);
--
arch/powerpc/crypto/aes-spe-glue.c=171=static int ppc_cbc_crypt(struct skcipher_request *req, bool enc)
--
arch/powerpc/crypto/aes-spe-glue.c-181- while ((nbytes = walk.nbytes) != 0) {
arch/powerpc/crypto/aes-spe-glue.c:182: nbytes = min_t(unsigned int, nbytes, MAX_BYTES);
arch/powerpc/crypto/aes-spe-glue.c-183- nbytes = round_down(nbytes, AES_BLOCK_SIZE);
--
arch/powerpc/crypto/aes-spe-glue.c=212=static int ppc_ctr_crypt(struct skcipher_request *req)
--
arch/powerpc/crypto/aes-spe-glue.c-222- while ((nbytes = walk.nbytes) != 0) {
arch/powerpc/crypto/aes-spe-glue.c:223: nbytes = min_t(unsigned int, nbytes, MAX_BYTES);
arch/powerpc/crypto/aes-spe-glue.c-224- if (nbytes < walk.total)
--
arch/powerpc/crypto/aes-spe-glue.c=238=static int ppc_xts_crypt(struct skcipher_request *req, bool enc)
--
arch/powerpc/crypto/aes-spe-glue.c-250- while ((nbytes = walk.nbytes) != 0) {
arch/powerpc/crypto/aes-spe-glue.c:251: nbytes = min_t(unsigned int, nbytes, MAX_BYTES);
arch/powerpc/crypto/aes-spe-glue.c-252- nbytes = round_down(nbytes, AES_BLOCK_SIZE);
--
arch/powerpc/kernel/crash_dump.c=72=ssize_t copy_oldmem_page(struct iov_iter *iter, unsigned long pfn,
--
arch/powerpc/kernel/crash_dump.c-80-
arch/powerpc/kernel/crash_dump.c:81: csize = min_t(size_t, csize, PAGE_SIZE);
arch/powerpc/kernel/crash_dump.c-82- paddr = pfn << PAGE_SHIFT;
--
arch/powerpc/kernel/fadump.c=243=static bool is_fadump_mem_area_contiguous(u64 d_start, u64 d_end)
--
arch/powerpc/kernel/fadump.c-250- start = max_t(u64, d_start, reg_start);
arch/powerpc/kernel/fadump.c:251: end = min_t(u64, d_end, reg_end);
arch/powerpc/kernel/fadump.c-252- if (d_start < end) {
--
arch/powerpc/kernel/fadump.c=1189=static void fadump_release_reserved_area(u64 start, u64 end)
--
arch/powerpc/kernel/fadump.c-1199- tstart = max_t(u64, spfn, reg_spfn);
arch/powerpc/kernel/fadump.c:1200: tend = min_t(u64, epfn, reg_epfn);
arch/powerpc/kernel/fadump.c-1201-
--
arch/powerpc/kernel/prom.c=787=void __init early_init_devtree(void *params)
--
arch/powerpc/kernel/prom.c-840- if (memory_limit)
arch/powerpc/kernel/prom.c:841: first_memblock_size = min_t(u64, first_memblock_size, memory_limit);
arch/powerpc/kernel/prom.c-842- setup_initial_memory_limit(memstart_addr, first_memblock_size);
--
arch/powerpc/kexec/core.c=67=static unsigned long long __init get_crash_base(unsigned long long crash_base)
--
arch/powerpc/kexec/core.c-90- if (firmware_has_feature(FW_FEATURE_LPAR))
arch/powerpc/kexec/core.c:91: crash_base = min_t(u64, ppc64_rma_size / 2, SZ_512M);
arch/powerpc/kexec/core.c-92- else
arch/powerpc/kexec/core.c:93: crash_base = min_t(u64, ppc64_rma_size / 2, SZ_128M);
arch/powerpc/kexec/core.c-94-#else
--
arch/powerpc/kvm/book3s_hv_ras.c=37=static void reload_slb(struct kvm_vcpu *vcpu)
--
arch/powerpc/kvm/book3s_hv_ras.c-50- /* Sanity check */
arch/powerpc/kvm/book3s_hv_ras.c:51: n = min_t(u32, be32_to_cpu(slb->persistent), SLB_MIN_SIZE);
arch/powerpc/kvm/book3s_hv_ras.c-52- if ((void *) &slb->save_area[n] > vcpu->arch.slb_shadow.pinned_end)
--
arch/powerpc/kvm/booke.c=578=static unsigned long watchdog_next_timeout(struct kvm_vcpu *vcpu)
--
arch/powerpc/kvm/booke.c-600-
arch/powerpc/kvm/booke.c:601: return min_t(unsigned long long, nr_jiffies, TIMER_NEXT_MAX_DELTA);
arch/powerpc/kvm/booke.c-602-}
--
arch/powerpc/kvm/powerpc.c=502=int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
--
arch/powerpc/kvm/powerpc.c-637- if (hv_enabled)
arch/powerpc/kvm/powerpc.c:638: r = min_t(unsigned int, num_present_cpus(), KVM_MAX_VCPUS);
arch/powerpc/kvm/powerpc.c-639- else
arch/powerpc/kvm/powerpc.c:640: r = min_t(unsigned int, num_online_cpus(), KVM_MAX_VCPUS);
arch/powerpc/kvm/powerpc.c-641- break;
--
arch/powerpc/lib/code-patching.c=550=int patch_instructions(u32 *addr, u32 *code, size_t len, bool repeat_instr)
--
arch/powerpc/lib/code-patching.c-556-
arch/powerpc/lib/code-patching.c:557: plen = min_t(size_t, PAGE_SIZE - offset_in_page(addr), len);
arch/powerpc/lib/code-patching.c-558-
--
arch/powerpc/lib/sstep.c=665=static nokprobe_inline int do_vec_load(int rn, unsigned long ea,
--
arch/powerpc/lib/sstep.c-685- if (unlikely(cross_endian))
arch/powerpc/lib/sstep.c:686: do_byte_reverse(&u.b[ea & 0xf], min_t(size_t, size, sizeof(u)));
arch/powerpc/lib/sstep.c-687- preempt_disable();
--
arch/powerpc/lib/sstep.c=696=static nokprobe_inline int do_vec_store(int rn, unsigned long ea,
--
arch/powerpc/lib/sstep.c-719- if (unlikely(cross_endian))
arch/powerpc/lib/sstep.c:720: do_byte_reverse(&u.b[ea & 0xf], min_t(size_t, size, sizeof(u)));
arch/powerpc/lib/sstep.c-721- return copy_mem_out(&u.b[ea & 0xf], ea, size, regs);
--
arch/powerpc/mm/book3s32/mmu.c=424=void setup_initial_memory_limit(phys_addr_t first_memblock_base,
--
arch/powerpc/mm/book3s32/mmu.c-431-
arch/powerpc/mm/book3s32/mmu.c:432: memblock_set_current_limit(min_t(u64, first_memblock_size, SZ_256M));
arch/powerpc/mm/book3s32/mmu.c-433-}
--
arch/powerpc/mm/book3s64/hash_utils.c=2390=void hash__setup_initial_memory_limit(phys_addr_t first_memblock_base,
]
]
|
| 307/4 |
2026/05/18 19:11 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 41876
output: 334
thoughts: 289
Thoughts: **Reviewing the Buffer Allocation Logic**
Okay, `min_t` is definitely available, so that's good. Now, let's go over these code changes. It looks like they've refactored the buffer overflow checks. I see:
1. `tmp_buf_end - tmp_buf < sizeof_cur_ip` is now `tmp_buf + sizeof_cur_ip > tmp_buf_end`. Right, that's just a simple arithmetic transformation to avoid subtraction, which can be prone to integer overflow issues, and it's cleaner to read this way.
2. Same thing with `tmp_buf_end - tmp_buf < sizeof_cur_arg` becoming `tmp_buf + sizeof_cur_arg > tmp_buf_end`. Makes sense. Keeps the logic consistent.
3. The check `tmp_buf_end == tmp_buf` has been changed to `tmp_buf >= tmp_buf_end`. Ah, it's safer to avoid equality tests on pointer arithmetic. This avoids integer overflow if `tmp_buf` has wrapped around, although I think it is an edge case.
4. Now, looking at the `%pB` `snprintf` advance logic... I see they're using `snprintf` to format a pointer. They are carefully checking for buffer overflow issues. Okay, `limit` calculates the available space, and then the `snprintf` call happens. They're using `min_t` to determine the actual number of bytes written, which seems redundant since the return value from `snprintf` will always be less than or equal to `limit`. Okay, they're ensuring that the buffer pointer is never incremented beyond the end of the allocation, and there is a +1 for a null terminator. That's a reasonable safeguard.
Wait, what about `%pI`? I need to check how that's handled. Is it using the same buffer protection logic? Let's check the code and be absolutely certain.
|
| 308/4 |
2026/05/18 19:11 |
tool |
codeeditor |
0m
Args: map[CurrentCode: /* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
num_spec++; NewCode: /* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
if (tmp_buf) {
size_t limit = tmp_buf < tmp_buf_end ? tmp_buf_end - tmp_buf : 0;
err = snprintf(tmp_buf, limit, ip_spec, &cur_ip);
if (err < 0)
err = 0;
tmp_buf += min_t(size_t, err + 1, limit);
}
num_spec++; SourceFile:kernel/bpf/helpers.c]
Results:
|
| 309/4 |
2026/05/18 19:11 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 42937
output: 44
thoughts: 32
Thoughts: Okay, here's my interpretation of that thought process, from an expert's perspective:
**Debugging a Format String Vulnerability**
Alright, hang on a second. I see `%pI` being processed *within* the `if (fmt[i] == 'p')` block. That's immediately interesting, and potentially problematic. The `'p'` format specifier typically signifies a pointer, which often leads to an address being printed. But, the actual vulnerability depends on the *context*. I need to drill down and understand the code surrounding `%pI` in this `if` statement. Where is `fmt` defined? How is it populated? What other processing happens inside that conditional? Is there any bounds checking? The devil's definitely in the details here, and my gut is telling me this could be a potential format string vulnerability, but I can't say for sure without more information. Let's get the wider picture.
|
| 310/4 |
2026/05/18 19:11 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_prepare]
Results: map[SourceCode: 804: /*
805: * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
806: *
807: * Returns a negative value if fmt is an invalid format string or 0 otherwise.
808: *
809: * This can be used in two ways:
810: * - Format string verification only: when data->get_bin_args is false
811: * - Arguments preparation: in addition to the above verification, it writes in
812: * data->bin_args a binary representation of arguments usable by bstr_printf
813: * where pointers from BPF have been sanitized.
814: *
815: * In argument preparation mode, if 0 is returned, safe temporary buffers are
816: * allocated and bpf_bprintf_cleanup should be called to free them after use.
817: */
818: int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
819: u32 num_args, struct bpf_bprintf_data *data)
820: {
821: bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
822: char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
823: struct bpf_bprintf_buffers *buffers = NULL;
824: size_t sizeof_cur_arg, sizeof_cur_ip;
825: int err, i, num_spec = 0;
826: u64 cur_arg;
827: char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
828:
829: fmt_end = strnchr(fmt, fmt_size, 0);
830: if (!fmt_end)
831: return -EINVAL;
832: fmt_size = fmt_end - fmt;
833:
834: if (get_buffers && bpf_try_get_buffers(&buffers))
835: return -EBUSY;
836:
837: if (data->get_bin_args) {
838: if (num_args)
839: tmp_buf = buffers->bin_args;
840: tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
841: data->bin_args = (u32 *)tmp_buf;
842: }
843:
844: if (data->get_buf)
845: data->buf = buffers->buf;
846:
847: for (i = 0; i < fmt_size; i++) {
848: unsigned char c = fmt[i];
849:
850: /*
851: * Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
852: * through unchanged, while still rejecting ASCII control bytes.
853: */
854: if (isascii(c) && !isprint(c) && !isspace(c)) {
855: err = -EINVAL;
856: goto out;
857: }
858:
859: if (fmt[i] != '%')
860: continue;
861:
862: if (fmt[i + 1] == '%') {
863: i++;
864: continue;
865: }
866:
867: if (num_spec >= num_args) {
868: err = -EINVAL;
869: goto out;
870: }
871:
872: /* The string is zero-terminated so if fmt[i] != 0, we can
873: * always access fmt[i + 1], in the worst case it will be a 0
874: */
875: i++;
876: c = fmt[i];
877: /*
878: * The format parser below only understands ASCII conversion
879: * specifiers and modifiers, so reject non-ASCII after '%'.
880: */
881: if (!isascii(c)) {
882: err = -EINVAL;
883: goto out;
884: }
885:
886: /* skip optional "[0 +-][num]" width formatting field */
887: while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
888: fmt[i] == ' ')
889: i++;
890: if (fmt[i] >= '1' && fmt[i] <= '9') {
891: i++;
892: while (fmt[i] >= '0' && fmt[i] <= '9')
893: i++;
894: }
895:
896: if (fmt[i] == 'p') {
897: sizeof_cur_arg = sizeof(long);
898:
899: if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
900: ispunct(fmt[i + 1])) {
901: if (tmp_buf)
902: cur_arg = raw_args[num_spec];
903: goto nocopy_fmt;
904: }
905:
906: if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
907: fmt[i + 2] == 's') {
908: fmt_ptype = fmt[i + 1];
909: i += 2;
910: goto fmt_str;
911: }
912:
913: if (fmt[i + 1] == 'K' ||
914: fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
915: fmt[i + 1] == 'S') {
916: if (tmp_buf)
917: cur_arg = raw_args[num_spec];
918: i++;
919: goto nocopy_fmt;
920: }
921:
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
930:
931: i++;
932: num_spec++;
933: continue;
934: }
935:
936: /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
937: if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
938: (fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
939: err = -EINVAL;
940: goto out;
941: }
942:
943: i += 2;
944: if (!tmp_buf)
945: goto nocopy_fmt;
946:
947: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
948: if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
949: err = -ENOSPC;
950: goto out;
951: }
952:
953: unsafe_ptr = (char *)(long)raw_args[num_spec];
954: err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
955: sizeof_cur_ip);
956: if (err < 0)
957: memset(cur_ip, 0, sizeof_cur_ip);
958:
959: /* hack: bstr_printf expects IP addresses to be
960: * pre-formatted as strings, ironically, the easiest way
961: * to do that is to call snprintf.
962: */
963: ip_spec[2] = fmt[i - 1];
964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
969: num_spec++;
970:
971: continue;
972: } else if (fmt[i] == 's') {
973: fmt_ptype = fmt[i];
974: fmt_str:
975: if (fmt[i + 1] != 0 &&
976: !isspace(fmt[i + 1]) &&
977: !ispunct(fmt[i + 1])) {
978: err = -EINVAL;
979: goto out;
980: }
981:
982: if (!tmp_buf)
983: goto nocopy_fmt;
984:
985: if (tmp_buf_end == tmp_buf) {
986: err = -ENOSPC;
987: goto out;
988: }
989:
990: unsafe_ptr = (char *)(long)raw_args[num_spec];
991: err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
992: fmt_ptype,
993: tmp_buf_end - tmp_buf);
994: if (err < 0) {
995: tmp_buf[0] = '\0';
996: err = 1;
997: }
998:
999: tmp_buf += err;
1000: num_spec++;
1001:
1002: continue;
1003: } else if (fmt[i] == 'c') {
1004: if (!tmp_buf)
1005: goto nocopy_fmt;
1006:
1007: if (tmp_buf_end == tmp_buf) {
1008: err = -ENOSPC;
1009: goto out;
1010: }
1011:
1012: *tmp_buf = raw_args[num_spec];
1013: tmp_buf++;
1014: num_spec++;
1015:
1016: continue;
1017: }
1018:
1019: sizeof_cur_arg = sizeof(int);
1020:
1021: if (fmt[i] == 'l') {
1022: sizeof_cur_arg = sizeof(long);
1023: i++;
1024: }
1025: if (fmt[i] == 'l') {
1026: sizeof_cur_arg = sizeof(long long);
1027: i++;
1028: }
1029:
1030: if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
1031: fmt[i] != 'x' && fmt[i] != 'X') {
1032: err = -EINVAL;
1033: goto out;
1034: }
1035:
1036: if (tmp_buf)
1037: cur_arg = raw_args[num_spec];
1038: nocopy_fmt:
1039: if (tmp_buf) {
1040: tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
1041: if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
1042: err = -ENOSPC;
1043: goto out;
1044: }
1045:
1046: if (sizeof_cur_arg == 8) {
1047: *(u32 *)tmp_buf = *(u32 *)&cur_arg;
1048: *(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
1049: } else {
1050: *(u32 *)tmp_buf = (u32)(long)cur_arg;
1051: }
1052: tmp_buf += sizeof_cur_arg;
1053: }
1054: num_spec++;
1055: }
1056:
1057: err = 0;
1058: out:
1059: if (err)
1060: bpf_bprintf_cleanup(data);
1061: return err;
1062: }
SourceFile:kernel/bpf/helpers.c]
|
| 311/4 |
2026/05/18 19:11 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 47050
output: 23
thoughts: 24
Thoughts: Here's my thought process, as if I were the one thinking it:
**Checking My Work on Patch Application**
Okay, hold on a second... didn't I just tweak `%pB` and `%pI`? I could have sworn I did. Best practice is to double-check these things, especially after making even minor edits. Let me just run a quick check on the `patch-diff` utility to be sure. It's too easy to lose track of what I've done, and a quick verification now will save me potential headaches down the line. A clean `patch-diff` output confirms my changes, which is what I expect. Excellent.
|
| 312/4 |
2026/05/18 19:11 |
tool |
patch-diff |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Output:diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2bb60200c..18e382a24 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -804,259 +804,266 @@ void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
- err = snprintf(tmp_buf,
- (tmp_buf_end - tmp_buf),
+ size_t limit = tmp_buf < tmp_buf_end ? tmp_buf_end - tmp_buf : 0;
+
+ err = snprintf(tmp_buf, limit,
"%pB",
(void *)(long)raw_args[num_spec]);
- tmp_buf += (err + 1);
+ if (err < 0)
+ err = 0;
+ tmp_buf += min_t(size_t, err + 1, limit);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
- if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
+ if (tmp_buf + sizeof_cur_ip > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
- err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
- ip_spec, &cur_ip);
+ if (tmp_buf) {
+ size_t limit = tmp_buf < tmp_buf_end ? tmp_buf_end - tmp_buf : 0;
- tmp_buf += err + 1;
+ err = snprintf(tmp_buf, limit, ip_spec, &cur_ip);
+ if (err < 0)
+ err = 0;
+ tmp_buf += min_t(size_t, err + 1, limit);
+ }
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
- if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
]
|
| 313/4 |
2026/05/18 19:11 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 50346
output: 385
thoughts: 264
Thoughts: **My Analysis of the Code Changes**
Okay, the diff seems solid at first glance. It looks like we've made some nice improvements to the buffer boundary checks. Replacing those subtractions with additions is a good move; it's generally clearer and less prone to overflow issues, especially when dealing with potential size calculations. Specifically, the replacements of `tmp_buf_end - tmp_buf < sizeof_cur_ip` and `tmp_buf_end - tmp_buf < sizeof_cur_arg` with `tmp_buf + sizeof_cur_ip > tmp_buf_end` and `tmp_buf + sizeof_cur_arg > tmp_buf_end` respectively look perfect. And that substitution of `tmp_buf_end == tmp_buf` with `tmp_buf >= tmp_buf_end` for the empty buffer condition is spot-on. Finally, the adjustments to how we handle the `snprintf` calls within the `%pB` and `%pI` specifiers, using `limit` and the `min_t` function to control buffer advancement, look sound.
Wait a second, though... when I look at the `%pI` code again, I see that I added an `if (tmp_buf)` check within that block. But earlier within that exact block, there is this:
```c
if (!tmp_buf)
goto nocopy_fmt;
```
That means that `tmp_buf` can *never* be NULL at that point in the code, because if it was, we would have jumped to `nocopy_fmt`. So, that redundant `if (tmp_buf)` check is actually unnecessary. It's a bit of clutter that doesn't add anything. I should remove it to keep the code as clean and concise as possible. It's a minor thing, but good practice to be efficient.
|
| 314/4 |
2026/05/18 19:11 |
tool |
codeeditor |
0m
Args: map[CurrentCode: /* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
if (tmp_buf) {
size_t limit = tmp_buf < tmp_buf_end ? tmp_buf_end - tmp_buf : 0;
err = snprintf(tmp_buf, limit, ip_spec, &cur_ip);
if (err < 0)
err = 0;
tmp_buf += min_t(size_t, err + 1, limit);
}
num_spec++; NewCode: /* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
{
size_t limit = tmp_buf < tmp_buf_end ? tmp_buf_end - tmp_buf : 0;
err = snprintf(tmp_buf, limit, ip_spec, &cur_ip);
if (err < 0)
err = 0;
tmp_buf += min_t(size_t, err + 1, limit);
}
num_spec++; SourceFile:kernel/bpf/helpers.c]
Results:
|
| 315/4 |
2026/05/18 19:11 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 51400
output: 16
thoughts: 55
Thoughts: Okay, here's my summary, considering my expertise:
**Refactoring `bpf_bprintf_prepare` - A Quick Optimization**
Right, so I was looking at that `bpf_bprintf_prepare` function, and I was initially thinking about where to define the `limit` variable, to keep it as neat as possible. Then it hit me: I can just declare `limit` right at the top of the function, or I could stick it inside a block scope. Simple as that. A block scope is more than sufficient. Now, before I jump to a solution, let's take a quick look and make sure that there isn't another place in `bpf_bprintf_prepare` where `snprintf` is used. Always good to double check, I'm sure I've missed one once or twice.
|
| 316/4 |
2026/05/18 19:11 |
tool |
grepper |
0m
Args:
Results: map[Output:
Full output is too long, showing 500 out of 43682 lines.
Use more precise expression if possible.
[Documentation/admin-guide/media/visl.rst=146=integer that denotes the buffer sequence, i.e.:
--
Documentation/admin-guide/media/visl.rst-149-
Documentation/admin-guide/media/visl.rst:150: snprintf(name, 32, "bitstream%d", run->src->sequence);
Documentation/admin-guide/media/visl.rst-151-
--
Documentation/core-api/printk-formats.rst=53=unsupported specifier or length qualifier results in a WARN and early
Documentation/core-api/printk-formats.rst:54:return from vsnprintf().
Documentation/core-api/printk-formats.rst-55-
--
Documentation/core-api/printk-formats.rst=150=either kernel memory (k) or user memory (u). The subsequent ``s`` specifier
Documentation/core-api/printk-formats.rst:151:results in printing a string. For direct use in regular vsnprintf() the (k)
Documentation/core-api/printk-formats.rst-152-and (u) annotation is ignored, however, when used out of BPF's bpf_trace_printk(),
--
Documentation/core-api/printk-formats.rst=467=and va_list as follows::
--
Documentation/core-api/printk-formats.rst-473-
Documentation/core-api/printk-formats.rst:474:Implements a "recursive vsnprintf".
Documentation/core-api/printk-formats.rst-475-
--
Documentation/driver-api/gpio/board.rst=57=with the string passed in con_id to get the resulting string
Documentation/driver-api/gpio/board.rst:58:(``snprintf(... "%s-%s", con_id, gpio_suffixes[]``).
Documentation/driver-api/gpio/board.rst-59-
--
Documentation/driver-api/nvdimm/nvdimm.rst=529=internally with a static identifier::
--
Documentation/driver-api/nvdimm/nvdimm.rst-536-
Documentation/driver-api/nvdimm/nvdimm.rst:537: snprintf(devname, sizeof(devname), "namespace%d.%d",
Documentation/driver-api/nvdimm/nvdimm.rst-538- ndctl_region_get_id(region), parameters->id);
--
Documentation/filesystems/sysfs.rst=261=A very simple (and naive) implementation of a device attribute is::
--
Documentation/filesystems/sysfs.rst-271- {
Documentation/filesystems/sysfs.rst:272: snprintf(dev->name, sizeof(dev->name), "%.*s",
Documentation/filesystems/sysfs.rst-273- (int)min(count, sizeof(dev->name) - 1), buf);
--
Documentation/i2c/dev-interface.rst=35=Next thing, open the device file, as follows::
--
Documentation/i2c/dev-interface.rst-40-
Documentation/i2c/dev-interface.rst:41: snprintf(filename, 19, "/dev/i2c-%d", adapter_nr);
Documentation/i2c/dev-interface.rst-42- file = open(filename, O_RDWR);
--
Documentation/input/uinput.rst=185=but interact with uinput via ioctl calls, or use libevdev.
--
Documentation/input/uinput.rst-213- memset(&uud, 0, sizeof(uud));
Documentation/input/uinput.rst:214: snprintf(uud.name, UINPUT_MAX_NAME_SIZE, "uinput old interface");
Documentation/input/uinput.rst-215- write(fd, &uud, sizeof(uud));
--
Documentation/livepatch/module-elf-format.rst=146=module that patches vmlinux and modules 9p, btrfs, ext4:**
--
Documentation/livepatch/module-elf-format.rst-178- 0000000000000036 0000003b00000002 R_X86_64_PC32 0000000000000000 .klp.sym.btrfs.can_modify_feature.isra.3,0 - 4
Documentation/livepatch/module-elf-format.rst:179: 000000000000004c 0000004900000002 R_X86_64_PC32 0000000000000000 .klp.sym.vmlinux.snprintf,0 - 4
Documentation/livepatch/module-elf-format.rst-180- [ snip ] ^
--
Documentation/livepatch/module-elf-format.rst=267=Examples:
--
Documentation/livepatch/module-elf-format.rst-273-
Documentation/livepatch/module-elf-format.rst:274: .klp.sym.vmlinux.snprintf,0
Documentation/livepatch/module-elf-format.rst-275- .klp.sym.vmlinux.printk,0
--
Documentation/livepatch/module-elf-format.rst-284- [ snip ]
Documentation/livepatch/module-elf-format.rst:285: 73: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.snprintf,0
Documentation/livepatch/module-elf-format.rst-286- 74: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.capable,0
--
Documentation/scsi/ChangeLog.lpfc=1399=Changes from 20040409 to 20040416
--
Documentation/scsi/ChangeLog.lpfc-1430- * Fix RHEL 3 build by #defining EXPORT_SYMTAB.
Documentation/scsi/ChangeLog.lpfc:1431: * Replace sprintf with snprintf in lpfc_proc_info.
Documentation/scsi/ChangeLog.lpfc-1432- * Fix build warnings on 2.6 kernels - remove no longer used calls
--
Documentation/trace/ftrace.rst=90=of ftrace. Here is a list of some of the key files:
--
Documentation/trace/ftrace.rst-633- va_start(ap, fmt);
Documentation/trace/ftrace.rst:634: n = vsnprintf(buf, 256, fmt, ap);
Documentation/trace/ftrace.rst-635- va_end(ap);
--
Documentation/trace/ftrace.rst=2515=something like this simple program.
--
Documentation/trace/ftrace.rst-2567- static char trace_file[MAX_PATH+1];
Documentation/trace/ftrace.rst:2568: snprintf(trace_file, MAX_PATH, "%s/%s", find_tracefs(), file_name);
Documentation/trace/ftrace.rst-2569- return trace_file;
--
Documentation/trace/timerlat-tracer.rst=226=Here is an basic example of user-space code for timerlat::
--
Documentation/trace/timerlat-tracer.rst-241-
Documentation/trace/timerlat-tracer.rst:242: snprintf(buffer, sizeof(buffer),
Documentation/trace/timerlat-tracer.rst-243- "/sys/kernel/tracing/osnoise/per_cpu/cpu%ld/timerlat_fd",
--
Documentation/translations/zh_CN/core-api/printk-formats.rst-63-内核的printf不支持%n。显而易见,浮点格式(%e, %f, %g, %a)也不被识别。使用任何不
Documentation/translations/zh_CN/core-api/printk-formats.rst:64:支持的占位符或长度限定符都会导致一个WARN并且终止vsnprintf()执行。
Documentation/translations/zh_CN/core-api/printk-formats.rst-65-
--
Documentation/translations/zh_CN/core-api/printk-formats.rst=96=seq_printf(),而不是printk())由用户空间进程读取,使用下面描述的%pK修饰符,不
--
Documentation/translations/zh_CN/core-api/printk-formats.rst-147-``k`` 和 ``u`` 指定符用于打印来自内核内存(k)或用户内存(u)的先前探测的内存。后面的 ``s`` 指
Documentation/translations/zh_CN/core-api/printk-formats.rst:148:定符的结果是打印一个字符串。对于直接在常规的vsnprintf()中使用时,(k)和(u)注释被忽略,但是,当
Documentation/translations/zh_CN/core-api/printk-formats.rst-149-在BPF的bpf_trace_printk()之外使用时,它会读取它所指向的内存,不会出现错误。
--
Documentation/translations/zh_CN/core-api/printk-formats.rst=407=va_format结构体
--
Documentation/translations/zh_CN/core-api/printk-formats.rst-423-
Documentation/translations/zh_CN/core-api/printk-formats.rst:424:实现 "递归vsnprintf"。
Documentation/translations/zh_CN/core-api/printk-formats.rst-425-
--
Documentation/translations/zh_CN/filesystems/sysfs.txt=190=Sysfs 将会为每次读写操作调用一次这个方法。这使得这些方法在执行时
--
Documentation/translations/zh_CN/filesystems/sysfs.txt-217-
Documentation/translations/zh_CN/filesystems/sysfs.txt:218:- show() 方法在将格式化返回值返回用户空间的时候,禁止使用snprintf()。
Documentation/translations/zh_CN/filesystems/sysfs.txt-219- 如果可以保证不会发生缓冲区溢出,可以使用sprintf(),否则必须使用
--
Documentation/translations/zh_CN/filesystems/sysfs.txt=240=static ssize_t store_name(struct device *dev, struct device_attribute *attr,
--
Documentation/translations/zh_CN/filesystems/sysfs.txt-242-{
Documentation/translations/zh_CN/filesystems/sysfs.txt:243: snprintf(dev->name, sizeof(dev->name), "%.*s",
Documentation/translations/zh_CN/filesystems/sysfs.txt-244- (int)min(count, sizeof(dev->name) - 1), buf);
--
Documentation/translations/zh_TW/filesystems/sysfs.txt=193=Sysfs 將會爲每次讀寫操作調用一次這個方法。這使得這些方法在執行時
--
Documentation/translations/zh_TW/filesystems/sysfs.txt-220-
Documentation/translations/zh_TW/filesystems/sysfs.txt:221:- show() 方法在將格式化返回值返回用戶空間的時候,禁止使用snprintf()。
Documentation/translations/zh_TW/filesystems/sysfs.txt-222- 如果可以保證不會發生緩衝區溢出,可以使用sprintf(),否則必須使用
--
Documentation/translations/zh_TW/filesystems/sysfs.txt=243=static ssize_t store_name(struct device *dev, struct device_attribute *attr,
--
Documentation/translations/zh_TW/filesystems/sysfs.txt-245-{
Documentation/translations/zh_TW/filesystems/sysfs.txt:246: snprintf(dev->name, sizeof(dev->name), "%.*s",
Documentation/translations/zh_TW/filesystems/sysfs.txt-247- (int)min(count, sizeof(dev->name) - 1), buf);
--
Documentation/userspace-api/media/v4l/dev-osd.rst=57=Example: Finding a framebuffer device for OSD
--
Documentation/userspace-api/media/v4l/dev-osd.rst-76-
Documentation/userspace-api/media/v4l/dev-osd.rst:77: snprintf(dev_name, sizeof(dev_name), "/dev/fb%u", i);
Documentation/userspace-api/media/v4l/dev-osd.rst-78-
--
arch/arm/kernel/process.c=112=void __show_regs(struct pt_regs *regs)
--
arch/arm/kernel/process.c-188- : "=r" (transbase));
arch/arm/kernel/process.c:189: snprintf(buf, sizeof(buf), " Table: %08x DAC: %08x",
arch/arm/kernel/process.c-190- transbase, domain);
--
arch/arm/kernel/setup.c=358=void __init early_print(const char *str, ...)
--
arch/arm/kernel/setup.c-364- va_start(ap, str);
arch/arm/kernel/setup.c:365: vsnprintf(buf, sizeof(buf), str, ap);
arch/arm/kernel/setup.c-366- va_end(ap);
--
arch/arm/kernel/setup.c=700=static void __init setup_processor(void)
--
arch/arm/kernel/setup.c-722-
arch/arm/kernel/setup.c:723: snprintf(init_utsname()->machine, __NEW_UTS_LEN + 1, "%s%c",
arch/arm/kernel/setup.c-724- list->arch_name, ENDIANNESS);
arch/arm/kernel/setup.c:725: snprintf(elf_platform, ELF_PLATFORM_SIZE, "%s%c",
arch/arm/kernel/setup.c-726- list->elf_name, ENDIANNESS);
--
arch/arm/mach-dove/pcie.c=35=static int __init dove_pcie_setup(int nr, struct pci_sys_data *sys)
--
arch/arm/mach-dove/pcie.c-61- */
arch/arm/mach-dove/pcie.c:62: snprintf(pp->mem_space_name, sizeof(pp->mem_space_name),
arch/arm/mach-dove/pcie.c-63- "PCIe %d MEM", pp->index);
--
arch/arm/mach-mv78xx0/pcie.c=54=static void __init mv78xx0_pcie_preinit(void)
--
arch/arm/mach-mv78xx0/pcie.c-76-
arch/arm/mach-mv78xx0/pcie.c:77: snprintf(pp->mem_space_name, sizeof(pp->mem_space_name),
arch/arm/mach-mv78xx0/pcie.c-78- "PCIe %d.%d MEM", pp->maj, pp->min);
--
arch/arm/mm/cache-l2x0-pmu.c=340=static ssize_t l2x0_pmu_event_show(struct device *dev,
--
arch/arm/mm/cache-l2x0-pmu.c-345- lattr = container_of(attr, typeof(*lattr), attr);
arch/arm/mm/cache-l2x0-pmu.c:346: return snprintf(buf, PAGE_SIZE, "config=0x%x\n", lattr->config);
arch/arm/mm/cache-l2x0-pmu.c-347-}
--
arch/arm/plat-orion/gpio.c=520=void __init orion_gpio_init(int gpio_base, int ngpio,
--
arch/arm/plat-orion/gpio.c-533-
arch/arm/plat-orion/gpio.c:534: snprintf(gc_label, sizeof(gc_label), "orion_gpio%d",
arch/arm/plat-orion/gpio.c-535- orion_gpio_chip_count);
--
arch/arm64/kvm/ptdump.c=93=static int kvm_ptdump_build_levels(struct ptdump_pg_level *level, u32 start_lvl)
--
arch/arm64/kvm/ptdump.c-105- for (i = start_lvl; i < KVM_PGTABLE_MAX_LEVELS; i++) {
arch/arm64/kvm/ptdump.c:106: snprintf(level[i].name, sizeof(level[i].name), "%u", i);
arch/arm64/kvm/ptdump.c-107-
--
arch/arm64/kvm/ptdump.c=276=void kvm_nested_s2_ptdump_create_debugfs(struct kvm_s2_mmu *mmu)
--
arch/arm64/kvm/ptdump.c-280-
arch/arm64/kvm/ptdump.c:281: snprintf(file_name, sizeof(file_name), "0x%016llx-0x%016llx-s2-%sabled",
arch/arm64/kvm/ptdump.c-282- mmu->tlb_vttbr,
--
arch/loongarch/kernel/kdebugfs.c=20=static ssize_t sfb_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
--
arch/loongarch/kernel/kdebugfs.c-26-
arch/loongarch/kernel/kdebugfs.c:27: s = snprintf(str, sizeof(str), "Boot State: %x\nCurrent State: %x\n", sfb_state, state);
arch/loongarch/kernel/kdebugfs.c-28-
--
arch/loongarch/kernel/kdebugfs.c=93=static ssize_t tso_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
--
arch/loongarch/kernel/kdebugfs.c-99-
arch/loongarch/kernel/kdebugfs.c:100: s = snprintf(str, sizeof(str), "Boot State: %d (%s)\n"
arch/loongarch/kernel/kdebugfs.c-101- "Current State: %d (%s)\n\n"
--
arch/m68k/emu/natfeat.c=55=void nfprint(const char *fmt, ...)
--
arch/m68k/emu/natfeat.c-60- va_start(ap, fmt);
arch/m68k/emu/natfeat.c:61: vsnprintf(buf, 256, fmt, ap);
arch/m68k/emu/natfeat.c-62- nf_call(nf_get_id("NF_STDERR"), virt_to_phys(buf));
--
arch/m68k/emu/nfblock.c=97=static int __init nfhd_init_one(int id, u32 blocks, u32 bsize)
--
arch/m68k/emu/nfblock.c-134- dev->disk->private_data = dev;
arch/m68k/emu/nfblock.c:135: snprintf(dev->disk->disk_name, sizeof(dev->disk->disk_name), "nfhd%u",
arch/m68k/emu/nfblock.c-136- dev_id);
--
arch/m68k/sun3/prom/printf.c=24=prom_printf(char *fmt, ...)
--
arch/m68k/sun3/prom/printf.c-32- ppbuf[0] = 'O';
arch/m68k/sun3/prom/printf.c:33: vsnprintf(ppbuf + 1, sizeof(ppbuf) - 1, fmt, args);
arch/m68k/sun3/prom/printf.c-34-#else
arch/m68k/sun3/prom/printf.c:35: vsnprintf(ppbuf, sizeof(ppbuf), fmt, args);
arch/m68k/sun3/prom/printf.c-36-#endif
--
arch/m68k/virt/config.c=80=void __init config_virt(void)
--
arch/m68k/virt/config.c-83-
arch/m68k/virt/config.c:84: snprintf(earlycon, sizeof(earlycon), "early_gf_tty,0x%08x",
arch/m68k/virt/config.c-85- virt_bi_data.tty.mmio);
--
arch/mips/bcm47xx/prom.c=48=__init void bcm47xx_set_system_type(u16 chip_id)
arch/mips/bcm47xx/prom.c-49-{
arch/mips/bcm47xx/prom.c:50: snprintf(bcm47xx_system_type, sizeof(bcm47xx_system_type),
arch/mips/bcm47xx/prom.c-51- (chip_id > 0x9999) ? "Broadcom BCM%d" :
--
arch/mips/bcm63xx/boards/board_bcm963xx.c=728=void __init board_prom_init(void)
--
arch/mips/bcm63xx/boards/board_bcm963xx.c-752- if(cfe[5] == 'd')
arch/mips/bcm63xx/boards/board_bcm963xx.c:753: snprintf(cfe_version, 11, "%s",
arch/mips/bcm63xx/boards/board_bcm963xx.c-754- (char *) &cfe[5]);
arch/mips/bcm63xx/boards/board_bcm963xx.c-755- else if (cfe[10] > 0)
arch/mips/bcm63xx/boards/board_bcm963xx.c:756: snprintf(cfe_version, sizeof(cfe_version),
arch/mips/bcm63xx/boards/board_bcm963xx.c-757- "%u.%u.%u-%u.%u-%u", cfe[5], cfe[6],
--
arch/mips/bcm63xx/boards/board_bcm963xx.c-759- else
arch/mips/bcm63xx/boards/board_bcm963xx.c:760: snprintf(cfe_version, sizeof(cfe_version),
arch/mips/bcm63xx/boards/board_bcm963xx.c-761- "%u.%u.%u-%u.%u", cfe[5], cfe[6],
--
arch/mips/bcm63xx/boards/board_bcm963xx.c-763- } else {
arch/mips/bcm63xx/boards/board_bcm963xx.c:764: snprintf(cfe_version, 12, "%s", (char *) &cfe[4]);
arch/mips/bcm63xx/boards/board_bcm963xx.c-765- }
--
arch/mips/bcm63xx/setup.c=141=const char *get_system_type(void)
--
arch/mips/bcm63xx/setup.c-143- static char buf[128];
arch/mips/bcm63xx/setup.c:144: snprintf(buf, sizeof(buf), "bcm63xx/%s (0x%04x/0x%02X)",
arch/mips/bcm63xx/setup.c-145- board_get_name(),
--
arch/mips/cavium-octeon/octeon-platform.c=483=static void __init octeon_fdt_set_phy(int eth, int phy_addr)
--
arch/mips/cavium-octeon/octeon-platform.c-555-
arch/mips/cavium-octeon/octeon-platform.c:556: snprintf(new_name, sizeof(new_name), "ethernet-phy@%x", phy_addr);
arch/mips/cavium-octeon/octeon-platform.c-557-
--
arch/mips/cavium-octeon/octeon-platform.c=649=static void __init octeon_fdt_pip_port(int iface, int i, int p, int max)
--
arch/mips/cavium-octeon/octeon-platform.c-656-
arch/mips/cavium-octeon/octeon-platform.c:657: snprintf(name_buffer, sizeof(name_buffer), "ethernet@%x", p);
arch/mips/cavium-octeon/octeon-platform.c-658- eth = fdt_subnode_offset(initial_boot_params, iface, name_buffer);
--
arch/mips/cavium-octeon/octeon-platform.c=682=static void __init octeon_fdt_pip_iface(int pip, int idx)
--
arch/mips/cavium-octeon/octeon-platform.c-688-
arch/mips/cavium-octeon/octeon-platform.c:689: snprintf(name_buffer, sizeof(name_buffer), "interface@%d", idx);
arch/mips/cavium-octeon/octeon-platform.c-690- iface = fdt_subnode_offset(initial_boot_params, pip, name_buffer);
--
arch/mips/cavium-octeon/octeon-platform.c=701=void __init octeon_fill_mac_addresses(void)
--
arch/mips/cavium-octeon/octeon-platform.c-724-
arch/mips/cavium-octeon/octeon-platform.c:725: snprintf(name_buffer, sizeof(name_buffer), "mix%d", i);
arch/mips/cavium-octeon/octeon-platform.c-726- alias_prop = fdt_getprop(initial_boot_params, aliases,
--
arch/mips/cavium-octeon/octeon-platform.c-747-
arch/mips/cavium-octeon/octeon-platform.c:748: snprintf(name_buffer, sizeof(name_buffer), "interface@%d", i);
arch/mips/cavium-octeon/octeon-platform.c-749- iface = fdt_subnode_offset(initial_boot_params, pip,
--
arch/mips/cavium-octeon/octeon-platform.c-755-
arch/mips/cavium-octeon/octeon-platform.c:756: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-757- "ethernet@%x", p);
--
arch/mips/cavium-octeon/octeon-platform.c=767=int __init octeon_prune_device_tree(void)
--
arch/mips/cavium-octeon/octeon-platform.c-800-
arch/mips/cavium-octeon/octeon-platform.c:801: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-802- "mix%d", i);
--
arch/mips/cavium-octeon/octeon-platform.c-842-
arch/mips/cavium-octeon/octeon-platform.c:843: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-844- "twsi%d", i);
--
arch/mips/cavium-octeon/octeon-platform.c-873-
arch/mips/cavium-octeon/octeon-platform.c:874: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-875- "smi%d", i);
--
arch/mips/cavium-octeon/octeon-platform.c-900-
arch/mips/cavium-octeon/octeon-platform.c:901: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-902- "uart%d", i);
--
arch/mips/cavium-octeon/setup.c=493=static void __init init_octeon_system_type(void)
--
arch/mips/cavium-octeon/setup.c-508-
arch/mips/cavium-octeon/setup.c:509: snprintf(octeon_system_type, sizeof(octeon_system_type), "%s (%s)",
arch/mips/cavium-octeon/setup.c-510- board_type, octeon_model_get_string(read_c0_prid()));
--
arch/mips/dec/prom/identify.c=44=const char *get_system_type(void)
--
arch/mips/dec/prom/identify.c-51- called = 1;
arch/mips/dec/prom/identify.c:52: snprintf(system, STR_BUF_LEN, "Digital %s",
arch/mips/dec/prom/identify.c-53- dec_system_strings[mips_machtype]);
--
arch/mips/generic/yamon-dt.c=75=__init int yamon_dt_append_memory(void *fdt,
--
arch/mips/generic/yamon-dt.c-109- for (i = 0; i < ARRAY_SIZE(var_names); i++) {
arch/mips/generic/yamon-dt.c:110: snprintf(param_name, sizeof(param_name), "%s=", var_names[i]);
arch/mips/generic/yamon-dt.c-111- var = strstr(arcs_cmdline, param_name);
--
arch/mips/generic/yamon-dt.c=157=__init int yamon_dt_serial_config(void *fdt)
--
arch/mips/generic/yamon-dt.c-179-
arch/mips/generic/yamon-dt.c:180: snprintf(mode_var_name, sizeof(mode_var_name), "modetty%u", uart);
arch/mips/generic/yamon-dt.c-181- mode_var = fw_getenv(mode_var_name);
--
arch/mips/generic/yamon-dt.c-210-
arch/mips/generic/yamon-dt.c:211: WARN_ON(snprintf(path, sizeof(path), "serial%u:%u%c%u%s",
arch/mips/generic/yamon-dt.c-212- uart, baud, parity, stop_bits,
--
arch/mips/kernel/mips-cm.c=393=void mips_cm_error_report(void)
--
arch/mips/kernel/mips-cm.c-421-
arch/mips/kernel/mips-cm.c:422: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-423- "CCA=%lu TR=%s MCmd=%s STag=%lu "
--
arch/mips/kernel/mips-cm.c-435-
arch/mips/kernel/mips-cm.c:436: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-437- "C3=%s C2=%s C1=%s C0=%s SC=%s "
--
arch/mips/kernel/mips-cm.c-451-
arch/mips/kernel/mips-cm.c:452: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-453- "Type=%s%s Instr=%s DW=%lu Way=%lu "
--
arch/mips/kernel/mips-cm.c-490- unsigned long bank_bit = (cm_error >> 28) & 0x1;
arch/mips/kernel/mips-cm.c:491: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-492- "%s ECC Error: Way=%lu (DWORD=%lu, Sets=%lu)"
--
arch/mips/kernel/mips-cm.c-510-
arch/mips/kernel/mips-cm.c:511: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-512- "Decode Request Error: Type=%lu, Command=%lu"
--
arch/mips/kernel/traps.c=1991=void __noreturn nmi_exception_handler(struct pt_regs *regs)
--
arch/mips/kernel/traps.c-1997- bust_spinlocks(1);
arch/mips/kernel/traps.c:1998: snprintf(str, 100, "CPU%d NMI taken, CP0_EPC=%lx\n",
arch/mips/kernel/traps.c-1999- smp_processor_id(), regs->cp0_epc);
--
arch/mips/lantiq/prom.c=100=void __init prom_init(void)
--
arch/mips/lantiq/prom.c-103- ltq_soc_detect(&soc_info);
arch/mips/lantiq/prom.c:104: snprintf(soc_info.sys_type, LTQ_SYS_TYPE_LEN - 1, "%s rev %s",
arch/mips/lantiq/prom.c-105- soc_info.name, soc_info.rev_type);
--
arch/mips/mti-malta/malta-dtshim.c=131=static void __init append_memory(void *fdt, int root_off)
--
arch/mips/mti-malta/malta-dtshim.c-179- for (i = 0; i < ARRAY_SIZE(var_names); i++) {
arch/mips/mti-malta/malta-dtshim.c:180: snprintf(param_name, sizeof(param_name), "%s=", var_names[i]);
arch/mips/mti-malta/malta-dtshim.c-181- var = strstr(arcs_cmdline, param_name);
--
arch/mips/pci/pci-xtalk-bridge.c=568=static int bridge_get_partnum(u64 baddr, char *partnum)
--
arch/mips/pci/pci-xtalk-bridge.c-575-
arch/mips/pci/pci-xtalk-bridge.c:576: snprintf(prefix, sizeof(prefix), "bridge-%012llx-0b-", baddr);
arch/mips/pci/pci-xtalk-bridge.c-577-
--
arch/mips/ralink/mt7620.c=220=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/mt7620.c-226-
arch/mips/ralink/mt7620.c:227: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/mt7620.c-228- "MediaTek %s ver:%u eco:%u",
--
arch/mips/ralink/mt7621.c=167=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/mt7621.c-198-
arch/mips/ralink/mt7621.c:199: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/mt7621.c-200- "MediaTek %s ver:%u eco:%u",
--
arch/mips/ralink/rt288x.c=90=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/rt288x.c-97-
arch/mips/ralink/rt288x.c:98: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/rt288x.c-99- "Ralink %s id:%u rev:%u",
--
arch/mips/ralink/rt305x.c=193=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/rt305x.c-196-
arch/mips/ralink/rt305x.c:197: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/rt305x.c-198- "Ralink %s id:%u rev:%u",
--
arch/mips/ralink/rt3883.c=90=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/rt3883.c-97-
arch/mips/ralink/rt3883.c:98: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/rt3883.c-99- "Ralink %s ver:%u eco:%u",
--
arch/mips/sgi-ip27/ip27-xtalk.c=26=static void bridge_platform_create(nasid_t nasid, int widget, int masterwid)
--
arch/mips/sgi-ip27/ip27-xtalk.c-42-
arch/mips/sgi-ip27/ip27-xtalk.c:43: snprintf(wd->dev_id, sizeof(wd->dev_id), "bridge-%012lx",
arch/mips/sgi-ip27/ip27-xtalk.c-44- offset + (widget << SWIN_SIZE_BITS));
--
arch/mips/sgi-ip30/ip30-xtalk.c=39=static void bridge_platform_create(int widget, int masterwid)
--
arch/mips/sgi-ip30/ip30-xtalk.c-52-
arch/mips/sgi-ip30/ip30-xtalk.c:53: snprintf(wd->dev_id, sizeof(wd->dev_id), "bridge-%012lx",
arch/mips/sgi-ip30/ip30-xtalk.c-54- IP30_SWIN_BASE(widget));
--
arch/mips/txx9/generic/setup.c=635=void __init txx9_iocled_init(unsigned long baseaddr,
--
arch/mips/txx9/generic/setup.c-675- struct gpio_led *led = &iocled->leds[i];
arch/mips/txx9/generic/setup.c:676: snprintf(iocled->names[i], sizeof(iocled->names[i]),
arch/mips/txx9/generic/setup.c-677- "iocled:%s:%u", color, i);
--
arch/parisc/kernel/cache.c=135=show_cache_info(struct seq_file *m)
--
arch/parisc/kernel/cache.c-141- if (cache_info.dc_loop != 1)
arch/parisc/kernel/cache.c:142: snprintf(buf, 32, "%lu-way associative", cache_info.dc_loop);
arch/parisc/kernel/cache.c-143- seq_printf(m, "D-cache\t\t: %ld KB (%s%s, %s, alias=%d)\n",
--
arch/parisc/kernel/drivers.c=493=alloc_pa_dev(unsigned long hpa, struct hardware_path *mod_path)
--
arch/parisc/kernel/drivers.c-536- name = parisc_hardware_description(&dev->id) ? : "unknown";
arch/parisc/kernel/drivers.c:537: snprintf(dev->name, sizeof(dev->name), "%s [%s]",
arch/parisc/kernel/drivers.c-538- name, parisc_pathname(dev));
--
arch/powerpc/kernel/fadump.c=149=void __init fadump_append_bootargs(void)
--
arch/powerpc/kernel/fadump.c-175- pr_debug("Cmdline: %s\n", boot_command_line);
arch/powerpc/kernel/fadump.c:176: snprintf(boot_command_line + len, COMMAND_LINE_SIZE - len, " %s", append_args);
arch/powerpc/kernel/fadump.c-177- pr_info("Updated cmdline: %s\n", boot_command_line);
--
arch/powerpc/kernel/rtas.c=1644=void rtas_os_term(char *str)
--
arch/powerpc/kernel/rtas.c-1659-
arch/powerpc/kernel/rtas.c:1660: snprintf(rtas_os_term_buf, 2048, "OS panic: %s", str);
arch/powerpc/kernel/rtas.c-1661-
--
arch/powerpc/kernel/rtas_flash.c=449=static int get_validate_flash_msg(struct rtas_validate_flash_t *args_buf,
--
arch/powerpc/kernel/rtas_flash.c-457- (args_buf->update_results == VALIDATE_TMP_UPDATE))
arch/powerpc/kernel/rtas_flash.c:458: n += snprintf(msg + n, msglen - n, "%s\n",
arch/powerpc/kernel/rtas_flash.c-459- args_buf->buf);
--
arch/powerpc/kernel/udbg.c=114=void udbg_printf(const char *fmt, ...)
--
arch/powerpc/kernel/udbg.c-120- va_start(args, fmt);
arch/powerpc/kernel/udbg.c:121: vsnprintf(buf, UDBG_BUFSIZE, fmt, args);
arch/powerpc/kernel/udbg.c-122- udbg_puts(buf);
--
arch/powerpc/kexec/file_load_64.c=194=static int add_usable_mem_property(void *fdt, struct device_node *dn,
--
arch/powerpc/kexec/file_load_64.c-203-
arch/powerpc/kexec/file_load_64.c:204: if (snprintf(path, NODE_PATH_LEN, "%pOF", dn) > (NODE_PATH_LEN - 1)) {
arch/powerpc/kexec/file_load_64.c-205- pr_err("Buffer (%d) too small for memory node: %pOF\n",
--
arch/powerpc/kvm/book3s_hv.c=2864=static ssize_t debugfs_timings_read(struct file *file, char __user *buf,
--
arch/powerpc/kvm/book3s_hv.c-2899- if (!ok)
arch/powerpc/kvm/book3s_hv.c:2900: snprintf(s, buf_end - s, "%s: stuck\n",
arch/powerpc/kvm/book3s_hv.c-2901- timings[i].name);
arch/powerpc/kvm/book3s_hv.c-2902- else
arch/powerpc/kvm/book3s_hv.c:2903: snprintf(s, buf_end - s,
arch/powerpc/kvm/book3s_hv.c-2904- "%s: %llu %llu %llu %llu\n",
--
arch/powerpc/perf/hv-24x7.c=388=static char *event_fmt(struct hv_24x7_event_data *event, unsigned int domain)
--
arch/powerpc/perf/hv-24x7.c-396- case HV_PERF_DOMAIN_PHYS_CHIP:
arch/powerpc/perf/hv-24x7.c:397: snprintf(buf, sizeof(buf), "%d", domain);
arch/powerpc/perf/hv-24x7.c-398- domain_str = buf;
--
arch/powerpc/platforms/512x/clock-commonclk.c=704=static void __init mpc512x_clk_setup_clock_tree(struct device_node *np, int busfreq)
--
arch/powerpc/platforms/512x/clock-commonclk.c-822- char name[12];
arch/powerpc/platforms/512x/clock-commonclk.c:823: snprintf(name, sizeof(name), "psc%d", mclk_idx);
arch/powerpc/platforms/512x/clock-commonclk.c-824- clks[MPC512x_CLK_PSC0 + mclk_idx] = mpc512x_clk_gated(
--
arch/powerpc/platforms/512x/clock-commonclk.c=951=static void __init mpc5121_clk_provide_migration_support(void)
--
arch/powerpc/platforms/512x/clock-commonclk.c-988- of_address_to_resource(np, 0, &res); \
arch/powerpc/platforms/512x/clock-commonclk.c:989: snprintf(devname, sizeof(devname), "%pa.%s", &res.start, np->name); \
arch/powerpc/platforms/512x/clock-commonclk.c-990-} while (0)
--
arch/powerpc/platforms/82xx/ep8248e.c=109=static int ep8248e_mdio_probe(struct platform_device *ofdev)
--
arch/powerpc/platforms/82xx/ep8248e.c-130- bus->parent = &ofdev->dev;
]
]
|
| 317/4 |
2026/05/18 19:11 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 63942
output: 16
thoughts: 51
Thoughts: Okay, here's my understanding, considering I'm thoroughly familiar with these kernel contexts:
**Checking `snprintf` Usage in `kernel/bpf/helpers.c`**
Alright, I've just had a scan through `kernel/bpf/helpers.c`, and I *believe* I can confidently say that I've accounted for all the instances of `snprintf` in that file. I don't see any other calls that require my immediate attention. However, given the criticality of this, let me just double-check that location one more time, just to be absolutely sure that I haven't missed anything. A quick, thorough re-examination of `kernel/bpf/helpers.c` is warranted. Better safe than sorry when dealing with potentially vulnerable string formatting functions.
|
| 318/4 |
2026/05/18 19:11 |
tool |
grepper |
0m
Args:
Results: map[Output:
Full output is too long, showing 500 out of 43682 lines.
Use more precise expression if possible.
[Documentation/admin-guide/media/visl.rst=146=integer that denotes the buffer sequence, i.e.:
--
Documentation/admin-guide/media/visl.rst-149-
Documentation/admin-guide/media/visl.rst:150: snprintf(name, 32, "bitstream%d", run->src->sequence);
Documentation/admin-guide/media/visl.rst-151-
--
Documentation/core-api/printk-formats.rst=53=unsupported specifier or length qualifier results in a WARN and early
Documentation/core-api/printk-formats.rst:54:return from vsnprintf().
Documentation/core-api/printk-formats.rst-55-
--
Documentation/core-api/printk-formats.rst=150=either kernel memory (k) or user memory (u). The subsequent ``s`` specifier
Documentation/core-api/printk-formats.rst:151:results in printing a string. For direct use in regular vsnprintf() the (k)
Documentation/core-api/printk-formats.rst-152-and (u) annotation is ignored, however, when used out of BPF's bpf_trace_printk(),
--
Documentation/core-api/printk-formats.rst=467=and va_list as follows::
--
Documentation/core-api/printk-formats.rst-473-
Documentation/core-api/printk-formats.rst:474:Implements a "recursive vsnprintf".
Documentation/core-api/printk-formats.rst-475-
--
Documentation/driver-api/gpio/board.rst=57=with the string passed in con_id to get the resulting string
Documentation/driver-api/gpio/board.rst:58:(``snprintf(... "%s-%s", con_id, gpio_suffixes[]``).
Documentation/driver-api/gpio/board.rst-59-
--
Documentation/driver-api/nvdimm/nvdimm.rst=529=internally with a static identifier::
--
Documentation/driver-api/nvdimm/nvdimm.rst-536-
Documentation/driver-api/nvdimm/nvdimm.rst:537: snprintf(devname, sizeof(devname), "namespace%d.%d",
Documentation/driver-api/nvdimm/nvdimm.rst-538- ndctl_region_get_id(region), parameters->id);
--
Documentation/filesystems/sysfs.rst=261=A very simple (and naive) implementation of a device attribute is::
--
Documentation/filesystems/sysfs.rst-271- {
Documentation/filesystems/sysfs.rst:272: snprintf(dev->name, sizeof(dev->name), "%.*s",
Documentation/filesystems/sysfs.rst-273- (int)min(count, sizeof(dev->name) - 1), buf);
--
Documentation/i2c/dev-interface.rst=35=Next thing, open the device file, as follows::
--
Documentation/i2c/dev-interface.rst-40-
Documentation/i2c/dev-interface.rst:41: snprintf(filename, 19, "/dev/i2c-%d", adapter_nr);
Documentation/i2c/dev-interface.rst-42- file = open(filename, O_RDWR);
--
Documentation/input/uinput.rst=185=but interact with uinput via ioctl calls, or use libevdev.
--
Documentation/input/uinput.rst-213- memset(&uud, 0, sizeof(uud));
Documentation/input/uinput.rst:214: snprintf(uud.name, UINPUT_MAX_NAME_SIZE, "uinput old interface");
Documentation/input/uinput.rst-215- write(fd, &uud, sizeof(uud));
--
Documentation/livepatch/module-elf-format.rst=146=module that patches vmlinux and modules 9p, btrfs, ext4:**
--
Documentation/livepatch/module-elf-format.rst-178- 0000000000000036 0000003b00000002 R_X86_64_PC32 0000000000000000 .klp.sym.btrfs.can_modify_feature.isra.3,0 - 4
Documentation/livepatch/module-elf-format.rst:179: 000000000000004c 0000004900000002 R_X86_64_PC32 0000000000000000 .klp.sym.vmlinux.snprintf,0 - 4
Documentation/livepatch/module-elf-format.rst-180- [ snip ] ^
--
Documentation/livepatch/module-elf-format.rst=267=Examples:
--
Documentation/livepatch/module-elf-format.rst-273-
Documentation/livepatch/module-elf-format.rst:274: .klp.sym.vmlinux.snprintf,0
Documentation/livepatch/module-elf-format.rst-275- .klp.sym.vmlinux.printk,0
--
Documentation/livepatch/module-elf-format.rst-284- [ snip ]
Documentation/livepatch/module-elf-format.rst:285: 73: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.snprintf,0
Documentation/livepatch/module-elf-format.rst-286- 74: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.capable,0
--
Documentation/scsi/ChangeLog.lpfc=1399=Changes from 20040409 to 20040416
--
Documentation/scsi/ChangeLog.lpfc-1430- * Fix RHEL 3 build by #defining EXPORT_SYMTAB.
Documentation/scsi/ChangeLog.lpfc:1431: * Replace sprintf with snprintf in lpfc_proc_info.
Documentation/scsi/ChangeLog.lpfc-1432- * Fix build warnings on 2.6 kernels - remove no longer used calls
--
Documentation/trace/ftrace.rst=90=of ftrace. Here is a list of some of the key files:
--
Documentation/trace/ftrace.rst-633- va_start(ap, fmt);
Documentation/trace/ftrace.rst:634: n = vsnprintf(buf, 256, fmt, ap);
Documentation/trace/ftrace.rst-635- va_end(ap);
--
Documentation/trace/ftrace.rst=2515=something like this simple program.
--
Documentation/trace/ftrace.rst-2567- static char trace_file[MAX_PATH+1];
Documentation/trace/ftrace.rst:2568: snprintf(trace_file, MAX_PATH, "%s/%s", find_tracefs(), file_name);
Documentation/trace/ftrace.rst-2569- return trace_file;
--
Documentation/trace/timerlat-tracer.rst=226=Here is an basic example of user-space code for timerlat::
--
Documentation/trace/timerlat-tracer.rst-241-
Documentation/trace/timerlat-tracer.rst:242: snprintf(buffer, sizeof(buffer),
Documentation/trace/timerlat-tracer.rst-243- "/sys/kernel/tracing/osnoise/per_cpu/cpu%ld/timerlat_fd",
--
Documentation/translations/zh_CN/core-api/printk-formats.rst-63-内核的printf不支持%n。显而易见,浮点格式(%e, %f, %g, %a)也不被识别。使用任何不
Documentation/translations/zh_CN/core-api/printk-formats.rst:64:支持的占位符或长度限定符都会导致一个WARN并且终止vsnprintf()执行。
Documentation/translations/zh_CN/core-api/printk-formats.rst-65-
--
Documentation/translations/zh_CN/core-api/printk-formats.rst=96=seq_printf(),而不是printk())由用户空间进程读取,使用下面描述的%pK修饰符,不
--
Documentation/translations/zh_CN/core-api/printk-formats.rst-147-``k`` 和 ``u`` 指定符用于打印来自内核内存(k)或用户内存(u)的先前探测的内存。后面的 ``s`` 指
Documentation/translations/zh_CN/core-api/printk-formats.rst:148:定符的结果是打印一个字符串。对于直接在常规的vsnprintf()中使用时,(k)和(u)注释被忽略,但是,当
Documentation/translations/zh_CN/core-api/printk-formats.rst-149-在BPF的bpf_trace_printk()之外使用时,它会读取它所指向的内存,不会出现错误。
--
Documentation/translations/zh_CN/core-api/printk-formats.rst=407=va_format结构体
--
Documentation/translations/zh_CN/core-api/printk-formats.rst-423-
Documentation/translations/zh_CN/core-api/printk-formats.rst:424:实现 "递归vsnprintf"。
Documentation/translations/zh_CN/core-api/printk-formats.rst-425-
--
Documentation/translations/zh_CN/filesystems/sysfs.txt=190=Sysfs 将会为每次读写操作调用一次这个方法。这使得这些方法在执行时
--
Documentation/translations/zh_CN/filesystems/sysfs.txt-217-
Documentation/translations/zh_CN/filesystems/sysfs.txt:218:- show() 方法在将格式化返回值返回用户空间的时候,禁止使用snprintf()。
Documentation/translations/zh_CN/filesystems/sysfs.txt-219- 如果可以保证不会发生缓冲区溢出,可以使用sprintf(),否则必须使用
--
Documentation/translations/zh_CN/filesystems/sysfs.txt=240=static ssize_t store_name(struct device *dev, struct device_attribute *attr,
--
Documentation/translations/zh_CN/filesystems/sysfs.txt-242-{
Documentation/translations/zh_CN/filesystems/sysfs.txt:243: snprintf(dev->name, sizeof(dev->name), "%.*s",
Documentation/translations/zh_CN/filesystems/sysfs.txt-244- (int)min(count, sizeof(dev->name) - 1), buf);
--
Documentation/translations/zh_TW/filesystems/sysfs.txt=193=Sysfs 將會爲每次讀寫操作調用一次這個方法。這使得這些方法在執行時
--
Documentation/translations/zh_TW/filesystems/sysfs.txt-220-
Documentation/translations/zh_TW/filesystems/sysfs.txt:221:- show() 方法在將格式化返回值返回用戶空間的時候,禁止使用snprintf()。
Documentation/translations/zh_TW/filesystems/sysfs.txt-222- 如果可以保證不會發生緩衝區溢出,可以使用sprintf(),否則必須使用
--
Documentation/translations/zh_TW/filesystems/sysfs.txt=243=static ssize_t store_name(struct device *dev, struct device_attribute *attr,
--
Documentation/translations/zh_TW/filesystems/sysfs.txt-245-{
Documentation/translations/zh_TW/filesystems/sysfs.txt:246: snprintf(dev->name, sizeof(dev->name), "%.*s",
Documentation/translations/zh_TW/filesystems/sysfs.txt-247- (int)min(count, sizeof(dev->name) - 1), buf);
--
Documentation/userspace-api/media/v4l/dev-osd.rst=57=Example: Finding a framebuffer device for OSD
--
Documentation/userspace-api/media/v4l/dev-osd.rst-76-
Documentation/userspace-api/media/v4l/dev-osd.rst:77: snprintf(dev_name, sizeof(dev_name), "/dev/fb%u", i);
Documentation/userspace-api/media/v4l/dev-osd.rst-78-
--
arch/arm/kernel/process.c=112=void __show_regs(struct pt_regs *regs)
--
arch/arm/kernel/process.c-188- : "=r" (transbase));
arch/arm/kernel/process.c:189: snprintf(buf, sizeof(buf), " Table: %08x DAC: %08x",
arch/arm/kernel/process.c-190- transbase, domain);
--
arch/arm/kernel/setup.c=358=void __init early_print(const char *str, ...)
--
arch/arm/kernel/setup.c-364- va_start(ap, str);
arch/arm/kernel/setup.c:365: vsnprintf(buf, sizeof(buf), str, ap);
arch/arm/kernel/setup.c-366- va_end(ap);
--
arch/arm/kernel/setup.c=700=static void __init setup_processor(void)
--
arch/arm/kernel/setup.c-722-
arch/arm/kernel/setup.c:723: snprintf(init_utsname()->machine, __NEW_UTS_LEN + 1, "%s%c",
arch/arm/kernel/setup.c-724- list->arch_name, ENDIANNESS);
arch/arm/kernel/setup.c:725: snprintf(elf_platform, ELF_PLATFORM_SIZE, "%s%c",
arch/arm/kernel/setup.c-726- list->elf_name, ENDIANNESS);
--
arch/arm/mach-dove/pcie.c=35=static int __init dove_pcie_setup(int nr, struct pci_sys_data *sys)
--
arch/arm/mach-dove/pcie.c-61- */
arch/arm/mach-dove/pcie.c:62: snprintf(pp->mem_space_name, sizeof(pp->mem_space_name),
arch/arm/mach-dove/pcie.c-63- "PCIe %d MEM", pp->index);
--
arch/arm/mach-mv78xx0/pcie.c=54=static void __init mv78xx0_pcie_preinit(void)
--
arch/arm/mach-mv78xx0/pcie.c-76-
arch/arm/mach-mv78xx0/pcie.c:77: snprintf(pp->mem_space_name, sizeof(pp->mem_space_name),
arch/arm/mach-mv78xx0/pcie.c-78- "PCIe %d.%d MEM", pp->maj, pp->min);
--
arch/arm/mm/cache-l2x0-pmu.c=340=static ssize_t l2x0_pmu_event_show(struct device *dev,
--
arch/arm/mm/cache-l2x0-pmu.c-345- lattr = container_of(attr, typeof(*lattr), attr);
arch/arm/mm/cache-l2x0-pmu.c:346: return snprintf(buf, PAGE_SIZE, "config=0x%x\n", lattr->config);
arch/arm/mm/cache-l2x0-pmu.c-347-}
--
arch/arm/plat-orion/gpio.c=520=void __init orion_gpio_init(int gpio_base, int ngpio,
--
arch/arm/plat-orion/gpio.c-533-
arch/arm/plat-orion/gpio.c:534: snprintf(gc_label, sizeof(gc_label), "orion_gpio%d",
arch/arm/plat-orion/gpio.c-535- orion_gpio_chip_count);
--
arch/arm64/kvm/ptdump.c=93=static int kvm_ptdump_build_levels(struct ptdump_pg_level *level, u32 start_lvl)
--
arch/arm64/kvm/ptdump.c-105- for (i = start_lvl; i < KVM_PGTABLE_MAX_LEVELS; i++) {
arch/arm64/kvm/ptdump.c:106: snprintf(level[i].name, sizeof(level[i].name), "%u", i);
arch/arm64/kvm/ptdump.c-107-
--
arch/arm64/kvm/ptdump.c=276=void kvm_nested_s2_ptdump_create_debugfs(struct kvm_s2_mmu *mmu)
--
arch/arm64/kvm/ptdump.c-280-
arch/arm64/kvm/ptdump.c:281: snprintf(file_name, sizeof(file_name), "0x%016llx-0x%016llx-s2-%sabled",
arch/arm64/kvm/ptdump.c-282- mmu->tlb_vttbr,
--
arch/loongarch/kernel/kdebugfs.c=20=static ssize_t sfb_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
--
arch/loongarch/kernel/kdebugfs.c-26-
arch/loongarch/kernel/kdebugfs.c:27: s = snprintf(str, sizeof(str), "Boot State: %x\nCurrent State: %x\n", sfb_state, state);
arch/loongarch/kernel/kdebugfs.c-28-
--
arch/loongarch/kernel/kdebugfs.c=93=static ssize_t tso_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
--
arch/loongarch/kernel/kdebugfs.c-99-
arch/loongarch/kernel/kdebugfs.c:100: s = snprintf(str, sizeof(str), "Boot State: %d (%s)\n"
arch/loongarch/kernel/kdebugfs.c-101- "Current State: %d (%s)\n\n"
--
arch/m68k/emu/natfeat.c=55=void nfprint(const char *fmt, ...)
--
arch/m68k/emu/natfeat.c-60- va_start(ap, fmt);
arch/m68k/emu/natfeat.c:61: vsnprintf(buf, 256, fmt, ap);
arch/m68k/emu/natfeat.c-62- nf_call(nf_get_id("NF_STDERR"), virt_to_phys(buf));
--
arch/m68k/emu/nfblock.c=97=static int __init nfhd_init_one(int id, u32 blocks, u32 bsize)
--
arch/m68k/emu/nfblock.c-134- dev->disk->private_data = dev;
arch/m68k/emu/nfblock.c:135: snprintf(dev->disk->disk_name, sizeof(dev->disk->disk_name), "nfhd%u",
arch/m68k/emu/nfblock.c-136- dev_id);
--
arch/m68k/sun3/prom/printf.c=24=prom_printf(char *fmt, ...)
--
arch/m68k/sun3/prom/printf.c-32- ppbuf[0] = 'O';
arch/m68k/sun3/prom/printf.c:33: vsnprintf(ppbuf + 1, sizeof(ppbuf) - 1, fmt, args);
arch/m68k/sun3/prom/printf.c-34-#else
arch/m68k/sun3/prom/printf.c:35: vsnprintf(ppbuf, sizeof(ppbuf), fmt, args);
arch/m68k/sun3/prom/printf.c-36-#endif
--
arch/m68k/virt/config.c=80=void __init config_virt(void)
--
arch/m68k/virt/config.c-83-
arch/m68k/virt/config.c:84: snprintf(earlycon, sizeof(earlycon), "early_gf_tty,0x%08x",
arch/m68k/virt/config.c-85- virt_bi_data.tty.mmio);
--
arch/mips/bcm47xx/prom.c=48=__init void bcm47xx_set_system_type(u16 chip_id)
arch/mips/bcm47xx/prom.c-49-{
arch/mips/bcm47xx/prom.c:50: snprintf(bcm47xx_system_type, sizeof(bcm47xx_system_type),
arch/mips/bcm47xx/prom.c-51- (chip_id > 0x9999) ? "Broadcom BCM%d" :
--
arch/mips/bcm63xx/boards/board_bcm963xx.c=728=void __init board_prom_init(void)
--
arch/mips/bcm63xx/boards/board_bcm963xx.c-752- if(cfe[5] == 'd')
arch/mips/bcm63xx/boards/board_bcm963xx.c:753: snprintf(cfe_version, 11, "%s",
arch/mips/bcm63xx/boards/board_bcm963xx.c-754- (char *) &cfe[5]);
arch/mips/bcm63xx/boards/board_bcm963xx.c-755- else if (cfe[10] > 0)
arch/mips/bcm63xx/boards/board_bcm963xx.c:756: snprintf(cfe_version, sizeof(cfe_version),
arch/mips/bcm63xx/boards/board_bcm963xx.c-757- "%u.%u.%u-%u.%u-%u", cfe[5], cfe[6],
--
arch/mips/bcm63xx/boards/board_bcm963xx.c-759- else
arch/mips/bcm63xx/boards/board_bcm963xx.c:760: snprintf(cfe_version, sizeof(cfe_version),
arch/mips/bcm63xx/boards/board_bcm963xx.c-761- "%u.%u.%u-%u.%u", cfe[5], cfe[6],
--
arch/mips/bcm63xx/boards/board_bcm963xx.c-763- } else {
arch/mips/bcm63xx/boards/board_bcm963xx.c:764: snprintf(cfe_version, 12, "%s", (char *) &cfe[4]);
arch/mips/bcm63xx/boards/board_bcm963xx.c-765- }
--
arch/mips/bcm63xx/setup.c=141=const char *get_system_type(void)
--
arch/mips/bcm63xx/setup.c-143- static char buf[128];
arch/mips/bcm63xx/setup.c:144: snprintf(buf, sizeof(buf), "bcm63xx/%s (0x%04x/0x%02X)",
arch/mips/bcm63xx/setup.c-145- board_get_name(),
--
arch/mips/cavium-octeon/octeon-platform.c=483=static void __init octeon_fdt_set_phy(int eth, int phy_addr)
--
arch/mips/cavium-octeon/octeon-platform.c-555-
arch/mips/cavium-octeon/octeon-platform.c:556: snprintf(new_name, sizeof(new_name), "ethernet-phy@%x", phy_addr);
arch/mips/cavium-octeon/octeon-platform.c-557-
--
arch/mips/cavium-octeon/octeon-platform.c=649=static void __init octeon_fdt_pip_port(int iface, int i, int p, int max)
--
arch/mips/cavium-octeon/octeon-platform.c-656-
arch/mips/cavium-octeon/octeon-platform.c:657: snprintf(name_buffer, sizeof(name_buffer), "ethernet@%x", p);
arch/mips/cavium-octeon/octeon-platform.c-658- eth = fdt_subnode_offset(initial_boot_params, iface, name_buffer);
--
arch/mips/cavium-octeon/octeon-platform.c=682=static void __init octeon_fdt_pip_iface(int pip, int idx)
--
arch/mips/cavium-octeon/octeon-platform.c-688-
arch/mips/cavium-octeon/octeon-platform.c:689: snprintf(name_buffer, sizeof(name_buffer), "interface@%d", idx);
arch/mips/cavium-octeon/octeon-platform.c-690- iface = fdt_subnode_offset(initial_boot_params, pip, name_buffer);
--
arch/mips/cavium-octeon/octeon-platform.c=701=void __init octeon_fill_mac_addresses(void)
--
arch/mips/cavium-octeon/octeon-platform.c-724-
arch/mips/cavium-octeon/octeon-platform.c:725: snprintf(name_buffer, sizeof(name_buffer), "mix%d", i);
arch/mips/cavium-octeon/octeon-platform.c-726- alias_prop = fdt_getprop(initial_boot_params, aliases,
--
arch/mips/cavium-octeon/octeon-platform.c-747-
arch/mips/cavium-octeon/octeon-platform.c:748: snprintf(name_buffer, sizeof(name_buffer), "interface@%d", i);
arch/mips/cavium-octeon/octeon-platform.c-749- iface = fdt_subnode_offset(initial_boot_params, pip,
--
arch/mips/cavium-octeon/octeon-platform.c-755-
arch/mips/cavium-octeon/octeon-platform.c:756: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-757- "ethernet@%x", p);
--
arch/mips/cavium-octeon/octeon-platform.c=767=int __init octeon_prune_device_tree(void)
--
arch/mips/cavium-octeon/octeon-platform.c-800-
arch/mips/cavium-octeon/octeon-platform.c:801: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-802- "mix%d", i);
--
arch/mips/cavium-octeon/octeon-platform.c-842-
arch/mips/cavium-octeon/octeon-platform.c:843: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-844- "twsi%d", i);
--
arch/mips/cavium-octeon/octeon-platform.c-873-
arch/mips/cavium-octeon/octeon-platform.c:874: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-875- "smi%d", i);
--
arch/mips/cavium-octeon/octeon-platform.c-900-
arch/mips/cavium-octeon/octeon-platform.c:901: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-902- "uart%d", i);
--
arch/mips/cavium-octeon/setup.c=493=static void __init init_octeon_system_type(void)
--
arch/mips/cavium-octeon/setup.c-508-
arch/mips/cavium-octeon/setup.c:509: snprintf(octeon_system_type, sizeof(octeon_system_type), "%s (%s)",
arch/mips/cavium-octeon/setup.c-510- board_type, octeon_model_get_string(read_c0_prid()));
--
arch/mips/dec/prom/identify.c=44=const char *get_system_type(void)
--
arch/mips/dec/prom/identify.c-51- called = 1;
arch/mips/dec/prom/identify.c:52: snprintf(system, STR_BUF_LEN, "Digital %s",
arch/mips/dec/prom/identify.c-53- dec_system_strings[mips_machtype]);
--
arch/mips/generic/yamon-dt.c=75=__init int yamon_dt_append_memory(void *fdt,
--
arch/mips/generic/yamon-dt.c-109- for (i = 0; i < ARRAY_SIZE(var_names); i++) {
arch/mips/generic/yamon-dt.c:110: snprintf(param_name, sizeof(param_name), "%s=", var_names[i]);
arch/mips/generic/yamon-dt.c-111- var = strstr(arcs_cmdline, param_name);
--
arch/mips/generic/yamon-dt.c=157=__init int yamon_dt_serial_config(void *fdt)
--
arch/mips/generic/yamon-dt.c-179-
arch/mips/generic/yamon-dt.c:180: snprintf(mode_var_name, sizeof(mode_var_name), "modetty%u", uart);
arch/mips/generic/yamon-dt.c-181- mode_var = fw_getenv(mode_var_name);
--
arch/mips/generic/yamon-dt.c-210-
arch/mips/generic/yamon-dt.c:211: WARN_ON(snprintf(path, sizeof(path), "serial%u:%u%c%u%s",
arch/mips/generic/yamon-dt.c-212- uart, baud, parity, stop_bits,
--
arch/mips/kernel/mips-cm.c=393=void mips_cm_error_report(void)
--
arch/mips/kernel/mips-cm.c-421-
arch/mips/kernel/mips-cm.c:422: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-423- "CCA=%lu TR=%s MCmd=%s STag=%lu "
--
arch/mips/kernel/mips-cm.c-435-
arch/mips/kernel/mips-cm.c:436: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-437- "C3=%s C2=%s C1=%s C0=%s SC=%s "
--
arch/mips/kernel/mips-cm.c-451-
arch/mips/kernel/mips-cm.c:452: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-453- "Type=%s%s Instr=%s DW=%lu Way=%lu "
--
arch/mips/kernel/mips-cm.c-490- unsigned long bank_bit = (cm_error >> 28) & 0x1;
arch/mips/kernel/mips-cm.c:491: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-492- "%s ECC Error: Way=%lu (DWORD=%lu, Sets=%lu)"
--
arch/mips/kernel/mips-cm.c-510-
arch/mips/kernel/mips-cm.c:511: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-512- "Decode Request Error: Type=%lu, Command=%lu"
--
arch/mips/kernel/traps.c=1991=void __noreturn nmi_exception_handler(struct pt_regs *regs)
--
arch/mips/kernel/traps.c-1997- bust_spinlocks(1);
arch/mips/kernel/traps.c:1998: snprintf(str, 100, "CPU%d NMI taken, CP0_EPC=%lx\n",
arch/mips/kernel/traps.c-1999- smp_processor_id(), regs->cp0_epc);
--
arch/mips/lantiq/prom.c=100=void __init prom_init(void)
--
arch/mips/lantiq/prom.c-103- ltq_soc_detect(&soc_info);
arch/mips/lantiq/prom.c:104: snprintf(soc_info.sys_type, LTQ_SYS_TYPE_LEN - 1, "%s rev %s",
arch/mips/lantiq/prom.c-105- soc_info.name, soc_info.rev_type);
--
arch/mips/mti-malta/malta-dtshim.c=131=static void __init append_memory(void *fdt, int root_off)
--
arch/mips/mti-malta/malta-dtshim.c-179- for (i = 0; i < ARRAY_SIZE(var_names); i++) {
arch/mips/mti-malta/malta-dtshim.c:180: snprintf(param_name, sizeof(param_name), "%s=", var_names[i]);
arch/mips/mti-malta/malta-dtshim.c-181- var = strstr(arcs_cmdline, param_name);
--
arch/mips/pci/pci-xtalk-bridge.c=568=static int bridge_get_partnum(u64 baddr, char *partnum)
--
arch/mips/pci/pci-xtalk-bridge.c-575-
arch/mips/pci/pci-xtalk-bridge.c:576: snprintf(prefix, sizeof(prefix), "bridge-%012llx-0b-", baddr);
arch/mips/pci/pci-xtalk-bridge.c-577-
--
arch/mips/ralink/mt7620.c=220=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/mt7620.c-226-
arch/mips/ralink/mt7620.c:227: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/mt7620.c-228- "MediaTek %s ver:%u eco:%u",
--
arch/mips/ralink/mt7621.c=167=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/mt7621.c-198-
arch/mips/ralink/mt7621.c:199: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/mt7621.c-200- "MediaTek %s ver:%u eco:%u",
--
arch/mips/ralink/rt288x.c=90=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/rt288x.c-97-
arch/mips/ralink/rt288x.c:98: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/rt288x.c-99- "Ralink %s id:%u rev:%u",
--
arch/mips/ralink/rt305x.c=193=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/rt305x.c-196-
arch/mips/ralink/rt305x.c:197: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/rt305x.c-198- "Ralink %s id:%u rev:%u",
--
arch/mips/ralink/rt3883.c=90=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/rt3883.c-97-
arch/mips/ralink/rt3883.c:98: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/rt3883.c-99- "Ralink %s ver:%u eco:%u",
--
arch/mips/sgi-ip27/ip27-xtalk.c=26=static void bridge_platform_create(nasid_t nasid, int widget, int masterwid)
--
arch/mips/sgi-ip27/ip27-xtalk.c-42-
arch/mips/sgi-ip27/ip27-xtalk.c:43: snprintf(wd->dev_id, sizeof(wd->dev_id), "bridge-%012lx",
arch/mips/sgi-ip27/ip27-xtalk.c-44- offset + (widget << SWIN_SIZE_BITS));
--
arch/mips/sgi-ip30/ip30-xtalk.c=39=static void bridge_platform_create(int widget, int masterwid)
--
arch/mips/sgi-ip30/ip30-xtalk.c-52-
arch/mips/sgi-ip30/ip30-xtalk.c:53: snprintf(wd->dev_id, sizeof(wd->dev_id), "bridge-%012lx",
arch/mips/sgi-ip30/ip30-xtalk.c-54- IP30_SWIN_BASE(widget));
--
arch/mips/txx9/generic/setup.c=635=void __init txx9_iocled_init(unsigned long baseaddr,
--
arch/mips/txx9/generic/setup.c-675- struct gpio_led *led = &iocled->leds[i];
arch/mips/txx9/generic/setup.c:676: snprintf(iocled->names[i], sizeof(iocled->names[i]),
arch/mips/txx9/generic/setup.c-677- "iocled:%s:%u", color, i);
--
arch/parisc/kernel/cache.c=135=show_cache_info(struct seq_file *m)
--
arch/parisc/kernel/cache.c-141- if (cache_info.dc_loop != 1)
arch/parisc/kernel/cache.c:142: snprintf(buf, 32, "%lu-way associative", cache_info.dc_loop);
arch/parisc/kernel/cache.c-143- seq_printf(m, "D-cache\t\t: %ld KB (%s%s, %s, alias=%d)\n",
--
arch/parisc/kernel/drivers.c=493=alloc_pa_dev(unsigned long hpa, struct hardware_path *mod_path)
--
arch/parisc/kernel/drivers.c-536- name = parisc_hardware_description(&dev->id) ? : "unknown";
arch/parisc/kernel/drivers.c:537: snprintf(dev->name, sizeof(dev->name), "%s [%s]",
arch/parisc/kernel/drivers.c-538- name, parisc_pathname(dev));
--
arch/powerpc/kernel/fadump.c=149=void __init fadump_append_bootargs(void)
--
arch/powerpc/kernel/fadump.c-175- pr_debug("Cmdline: %s\n", boot_command_line);
arch/powerpc/kernel/fadump.c:176: snprintf(boot_command_line + len, COMMAND_LINE_SIZE - len, " %s", append_args);
arch/powerpc/kernel/fadump.c-177- pr_info("Updated cmdline: %s\n", boot_command_line);
--
arch/powerpc/kernel/rtas.c=1644=void rtas_os_term(char *str)
--
arch/powerpc/kernel/rtas.c-1659-
arch/powerpc/kernel/rtas.c:1660: snprintf(rtas_os_term_buf, 2048, "OS panic: %s", str);
arch/powerpc/kernel/rtas.c-1661-
--
arch/powerpc/kernel/rtas_flash.c=449=static int get_validate_flash_msg(struct rtas_validate_flash_t *args_buf,
--
arch/powerpc/kernel/rtas_flash.c-457- (args_buf->update_results == VALIDATE_TMP_UPDATE))
arch/powerpc/kernel/rtas_flash.c:458: n += snprintf(msg + n, msglen - n, "%s\n",
arch/powerpc/kernel/rtas_flash.c-459- args_buf->buf);
--
arch/powerpc/kernel/udbg.c=114=void udbg_printf(const char *fmt, ...)
--
arch/powerpc/kernel/udbg.c-120- va_start(args, fmt);
arch/powerpc/kernel/udbg.c:121: vsnprintf(buf, UDBG_BUFSIZE, fmt, args);
arch/powerpc/kernel/udbg.c-122- udbg_puts(buf);
--
arch/powerpc/kexec/file_load_64.c=194=static int add_usable_mem_property(void *fdt, struct device_node *dn,
--
arch/powerpc/kexec/file_load_64.c-203-
arch/powerpc/kexec/file_load_64.c:204: if (snprintf(path, NODE_PATH_LEN, "%pOF", dn) > (NODE_PATH_LEN - 1)) {
arch/powerpc/kexec/file_load_64.c-205- pr_err("Buffer (%d) too small for memory node: %pOF\n",
--
arch/powerpc/kvm/book3s_hv.c=2864=static ssize_t debugfs_timings_read(struct file *file, char __user *buf,
--
arch/powerpc/kvm/book3s_hv.c-2899- if (!ok)
arch/powerpc/kvm/book3s_hv.c:2900: snprintf(s, buf_end - s, "%s: stuck\n",
arch/powerpc/kvm/book3s_hv.c-2901- timings[i].name);
arch/powerpc/kvm/book3s_hv.c-2902- else
arch/powerpc/kvm/book3s_hv.c:2903: snprintf(s, buf_end - s,
arch/powerpc/kvm/book3s_hv.c-2904- "%s: %llu %llu %llu %llu\n",
--
arch/powerpc/perf/hv-24x7.c=388=static char *event_fmt(struct hv_24x7_event_data *event, unsigned int domain)
--
arch/powerpc/perf/hv-24x7.c-396- case HV_PERF_DOMAIN_PHYS_CHIP:
arch/powerpc/perf/hv-24x7.c:397: snprintf(buf, sizeof(buf), "%d", domain);
arch/powerpc/perf/hv-24x7.c-398- domain_str = buf;
--
arch/powerpc/platforms/512x/clock-commonclk.c=704=static void __init mpc512x_clk_setup_clock_tree(struct device_node *np, int busfreq)
--
arch/powerpc/platforms/512x/clock-commonclk.c-822- char name[12];
arch/powerpc/platforms/512x/clock-commonclk.c:823: snprintf(name, sizeof(name), "psc%d", mclk_idx);
arch/powerpc/platforms/512x/clock-commonclk.c-824- clks[MPC512x_CLK_PSC0 + mclk_idx] = mpc512x_clk_gated(
--
arch/powerpc/platforms/512x/clock-commonclk.c=951=static void __init mpc5121_clk_provide_migration_support(void)
--
arch/powerpc/platforms/512x/clock-commonclk.c-988- of_address_to_resource(np, 0, &res); \
arch/powerpc/platforms/512x/clock-commonclk.c:989: snprintf(devname, sizeof(devname), "%pa.%s", &res.start, np->name); \
arch/powerpc/platforms/512x/clock-commonclk.c-990-} while (0)
--
arch/powerpc/platforms/82xx/ep8248e.c=109=static int ep8248e_mdio_probe(struct platform_device *ofdev)
--
arch/powerpc/platforms/82xx/ep8248e.c-130- bus->parent = &ofdev->dev;
]
]
|
| 319/4 |
2026/05/18 19:11 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 76470
output: 16
thoughts: 12
Thoughts: Okay, here's my interpretation of those thoughts, crafted with the understanding that I'm speaking to a fellow expert:
**Navigating the BPF Landscape**
Alright, so the discussion has moved on, and my mind is immediately drawn to the heart of the matter when we talk about BPF helpers. Specifically, I'm thinking about the implementation details. When someone mentions "helpers," the first file that pops into my head is `kernel/bpf/helpers.c`. That's where all the magic happens, right? It's the central repository for those crucial functions that BPF programs leverage to interact with the kernel. I'm imagining us having this conversation and going directly to that file to dive into the specifics!
|
| 320/4 |
2026/05/18 19:11 |
tool |
grepper |
0m
Args:
Results: map[Output:
Full output is too long, showing 500 out of 43682 lines.
Use more precise expression if possible.
[Documentation/admin-guide/media/visl.rst=146=integer that denotes the buffer sequence, i.e.:
--
Documentation/admin-guide/media/visl.rst-149-
Documentation/admin-guide/media/visl.rst:150: snprintf(name, 32, "bitstream%d", run->src->sequence);
Documentation/admin-guide/media/visl.rst-151-
--
Documentation/core-api/printk-formats.rst=53=unsupported specifier or length qualifier results in a WARN and early
Documentation/core-api/printk-formats.rst:54:return from vsnprintf().
Documentation/core-api/printk-formats.rst-55-
--
Documentation/core-api/printk-formats.rst=150=either kernel memory (k) or user memory (u). The subsequent ``s`` specifier
Documentation/core-api/printk-formats.rst:151:results in printing a string. For direct use in regular vsnprintf() the (k)
Documentation/core-api/printk-formats.rst-152-and (u) annotation is ignored, however, when used out of BPF's bpf_trace_printk(),
--
Documentation/core-api/printk-formats.rst=467=and va_list as follows::
--
Documentation/core-api/printk-formats.rst-473-
Documentation/core-api/printk-formats.rst:474:Implements a "recursive vsnprintf".
Documentation/core-api/printk-formats.rst-475-
--
Documentation/driver-api/gpio/board.rst=57=with the string passed in con_id to get the resulting string
Documentation/driver-api/gpio/board.rst:58:(``snprintf(... "%s-%s", con_id, gpio_suffixes[]``).
Documentation/driver-api/gpio/board.rst-59-
--
Documentation/driver-api/nvdimm/nvdimm.rst=529=internally with a static identifier::
--
Documentation/driver-api/nvdimm/nvdimm.rst-536-
Documentation/driver-api/nvdimm/nvdimm.rst:537: snprintf(devname, sizeof(devname), "namespace%d.%d",
Documentation/driver-api/nvdimm/nvdimm.rst-538- ndctl_region_get_id(region), parameters->id);
--
Documentation/filesystems/sysfs.rst=261=A very simple (and naive) implementation of a device attribute is::
--
Documentation/filesystems/sysfs.rst-271- {
Documentation/filesystems/sysfs.rst:272: snprintf(dev->name, sizeof(dev->name), "%.*s",
Documentation/filesystems/sysfs.rst-273- (int)min(count, sizeof(dev->name) - 1), buf);
--
Documentation/i2c/dev-interface.rst=35=Next thing, open the device file, as follows::
--
Documentation/i2c/dev-interface.rst-40-
Documentation/i2c/dev-interface.rst:41: snprintf(filename, 19, "/dev/i2c-%d", adapter_nr);
Documentation/i2c/dev-interface.rst-42- file = open(filename, O_RDWR);
--
Documentation/input/uinput.rst=185=but interact with uinput via ioctl calls, or use libevdev.
--
Documentation/input/uinput.rst-213- memset(&uud, 0, sizeof(uud));
Documentation/input/uinput.rst:214: snprintf(uud.name, UINPUT_MAX_NAME_SIZE, "uinput old interface");
Documentation/input/uinput.rst-215- write(fd, &uud, sizeof(uud));
--
Documentation/livepatch/module-elf-format.rst=146=module that patches vmlinux and modules 9p, btrfs, ext4:**
--
Documentation/livepatch/module-elf-format.rst-178- 0000000000000036 0000003b00000002 R_X86_64_PC32 0000000000000000 .klp.sym.btrfs.can_modify_feature.isra.3,0 - 4
Documentation/livepatch/module-elf-format.rst:179: 000000000000004c 0000004900000002 R_X86_64_PC32 0000000000000000 .klp.sym.vmlinux.snprintf,0 - 4
Documentation/livepatch/module-elf-format.rst-180- [ snip ] ^
--
Documentation/livepatch/module-elf-format.rst=267=Examples:
--
Documentation/livepatch/module-elf-format.rst-273-
Documentation/livepatch/module-elf-format.rst:274: .klp.sym.vmlinux.snprintf,0
Documentation/livepatch/module-elf-format.rst-275- .klp.sym.vmlinux.printk,0
--
Documentation/livepatch/module-elf-format.rst-284- [ snip ]
Documentation/livepatch/module-elf-format.rst:285: 73: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.snprintf,0
Documentation/livepatch/module-elf-format.rst-286- 74: 0000000000000000 0 NOTYPE GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.capable,0
--
Documentation/scsi/ChangeLog.lpfc=1399=Changes from 20040409 to 20040416
--
Documentation/scsi/ChangeLog.lpfc-1430- * Fix RHEL 3 build by #defining EXPORT_SYMTAB.
Documentation/scsi/ChangeLog.lpfc:1431: * Replace sprintf with snprintf in lpfc_proc_info.
Documentation/scsi/ChangeLog.lpfc-1432- * Fix build warnings on 2.6 kernels - remove no longer used calls
--
Documentation/trace/ftrace.rst=90=of ftrace. Here is a list of some of the key files:
--
Documentation/trace/ftrace.rst-633- va_start(ap, fmt);
Documentation/trace/ftrace.rst:634: n = vsnprintf(buf, 256, fmt, ap);
Documentation/trace/ftrace.rst-635- va_end(ap);
--
Documentation/trace/ftrace.rst=2515=something like this simple program.
--
Documentation/trace/ftrace.rst-2567- static char trace_file[MAX_PATH+1];
Documentation/trace/ftrace.rst:2568: snprintf(trace_file, MAX_PATH, "%s/%s", find_tracefs(), file_name);
Documentation/trace/ftrace.rst-2569- return trace_file;
--
Documentation/trace/timerlat-tracer.rst=226=Here is an basic example of user-space code for timerlat::
--
Documentation/trace/timerlat-tracer.rst-241-
Documentation/trace/timerlat-tracer.rst:242: snprintf(buffer, sizeof(buffer),
Documentation/trace/timerlat-tracer.rst-243- "/sys/kernel/tracing/osnoise/per_cpu/cpu%ld/timerlat_fd",
--
Documentation/translations/zh_CN/core-api/printk-formats.rst-63-内核的printf不支持%n。显而易见,浮点格式(%e, %f, %g, %a)也不被识别。使用任何不
Documentation/translations/zh_CN/core-api/printk-formats.rst:64:支持的占位符或长度限定符都会导致一个WARN并且终止vsnprintf()执行。
Documentation/translations/zh_CN/core-api/printk-formats.rst-65-
--
Documentation/translations/zh_CN/core-api/printk-formats.rst=96=seq_printf(),而不是printk())由用户空间进程读取,使用下面描述的%pK修饰符,不
--
Documentation/translations/zh_CN/core-api/printk-formats.rst-147-``k`` 和 ``u`` 指定符用于打印来自内核内存(k)或用户内存(u)的先前探测的内存。后面的 ``s`` 指
Documentation/translations/zh_CN/core-api/printk-formats.rst:148:定符的结果是打印一个字符串。对于直接在常规的vsnprintf()中使用时,(k)和(u)注释被忽略,但是,当
Documentation/translations/zh_CN/core-api/printk-formats.rst-149-在BPF的bpf_trace_printk()之外使用时,它会读取它所指向的内存,不会出现错误。
--
Documentation/translations/zh_CN/core-api/printk-formats.rst=407=va_format结构体
--
Documentation/translations/zh_CN/core-api/printk-formats.rst-423-
Documentation/translations/zh_CN/core-api/printk-formats.rst:424:实现 "递归vsnprintf"。
Documentation/translations/zh_CN/core-api/printk-formats.rst-425-
--
Documentation/translations/zh_CN/filesystems/sysfs.txt=190=Sysfs 将会为每次读写操作调用一次这个方法。这使得这些方法在执行时
--
Documentation/translations/zh_CN/filesystems/sysfs.txt-217-
Documentation/translations/zh_CN/filesystems/sysfs.txt:218:- show() 方法在将格式化返回值返回用户空间的时候,禁止使用snprintf()。
Documentation/translations/zh_CN/filesystems/sysfs.txt-219- 如果可以保证不会发生缓冲区溢出,可以使用sprintf(),否则必须使用
--
Documentation/translations/zh_CN/filesystems/sysfs.txt=240=static ssize_t store_name(struct device *dev, struct device_attribute *attr,
--
Documentation/translations/zh_CN/filesystems/sysfs.txt-242-{
Documentation/translations/zh_CN/filesystems/sysfs.txt:243: snprintf(dev->name, sizeof(dev->name), "%.*s",
Documentation/translations/zh_CN/filesystems/sysfs.txt-244- (int)min(count, sizeof(dev->name) - 1), buf);
--
Documentation/translations/zh_TW/filesystems/sysfs.txt=193=Sysfs 將會爲每次讀寫操作調用一次這個方法。這使得這些方法在執行時
--
Documentation/translations/zh_TW/filesystems/sysfs.txt-220-
Documentation/translations/zh_TW/filesystems/sysfs.txt:221:- show() 方法在將格式化返回值返回用戶空間的時候,禁止使用snprintf()。
Documentation/translations/zh_TW/filesystems/sysfs.txt-222- 如果可以保證不會發生緩衝區溢出,可以使用sprintf(),否則必須使用
--
Documentation/translations/zh_TW/filesystems/sysfs.txt=243=static ssize_t store_name(struct device *dev, struct device_attribute *attr,
--
Documentation/translations/zh_TW/filesystems/sysfs.txt-245-{
Documentation/translations/zh_TW/filesystems/sysfs.txt:246: snprintf(dev->name, sizeof(dev->name), "%.*s",
Documentation/translations/zh_TW/filesystems/sysfs.txt-247- (int)min(count, sizeof(dev->name) - 1), buf);
--
Documentation/userspace-api/media/v4l/dev-osd.rst=57=Example: Finding a framebuffer device for OSD
--
Documentation/userspace-api/media/v4l/dev-osd.rst-76-
Documentation/userspace-api/media/v4l/dev-osd.rst:77: snprintf(dev_name, sizeof(dev_name), "/dev/fb%u", i);
Documentation/userspace-api/media/v4l/dev-osd.rst-78-
--
arch/arm/kernel/process.c=112=void __show_regs(struct pt_regs *regs)
--
arch/arm/kernel/process.c-188- : "=r" (transbase));
arch/arm/kernel/process.c:189: snprintf(buf, sizeof(buf), " Table: %08x DAC: %08x",
arch/arm/kernel/process.c-190- transbase, domain);
--
arch/arm/kernel/setup.c=358=void __init early_print(const char *str, ...)
--
arch/arm/kernel/setup.c-364- va_start(ap, str);
arch/arm/kernel/setup.c:365: vsnprintf(buf, sizeof(buf), str, ap);
arch/arm/kernel/setup.c-366- va_end(ap);
--
arch/arm/kernel/setup.c=700=static void __init setup_processor(void)
--
arch/arm/kernel/setup.c-722-
arch/arm/kernel/setup.c:723: snprintf(init_utsname()->machine, __NEW_UTS_LEN + 1, "%s%c",
arch/arm/kernel/setup.c-724- list->arch_name, ENDIANNESS);
arch/arm/kernel/setup.c:725: snprintf(elf_platform, ELF_PLATFORM_SIZE, "%s%c",
arch/arm/kernel/setup.c-726- list->elf_name, ENDIANNESS);
--
arch/arm/mach-dove/pcie.c=35=static int __init dove_pcie_setup(int nr, struct pci_sys_data *sys)
--
arch/arm/mach-dove/pcie.c-61- */
arch/arm/mach-dove/pcie.c:62: snprintf(pp->mem_space_name, sizeof(pp->mem_space_name),
arch/arm/mach-dove/pcie.c-63- "PCIe %d MEM", pp->index);
--
arch/arm/mach-mv78xx0/pcie.c=54=static void __init mv78xx0_pcie_preinit(void)
--
arch/arm/mach-mv78xx0/pcie.c-76-
arch/arm/mach-mv78xx0/pcie.c:77: snprintf(pp->mem_space_name, sizeof(pp->mem_space_name),
arch/arm/mach-mv78xx0/pcie.c-78- "PCIe %d.%d MEM", pp->maj, pp->min);
--
arch/arm/mm/cache-l2x0-pmu.c=340=static ssize_t l2x0_pmu_event_show(struct device *dev,
--
arch/arm/mm/cache-l2x0-pmu.c-345- lattr = container_of(attr, typeof(*lattr), attr);
arch/arm/mm/cache-l2x0-pmu.c:346: return snprintf(buf, PAGE_SIZE, "config=0x%x\n", lattr->config);
arch/arm/mm/cache-l2x0-pmu.c-347-}
--
arch/arm/plat-orion/gpio.c=520=void __init orion_gpio_init(int gpio_base, int ngpio,
--
arch/arm/plat-orion/gpio.c-533-
arch/arm/plat-orion/gpio.c:534: snprintf(gc_label, sizeof(gc_label), "orion_gpio%d",
arch/arm/plat-orion/gpio.c-535- orion_gpio_chip_count);
--
arch/arm64/kvm/ptdump.c=93=static int kvm_ptdump_build_levels(struct ptdump_pg_level *level, u32 start_lvl)
--
arch/arm64/kvm/ptdump.c-105- for (i = start_lvl; i < KVM_PGTABLE_MAX_LEVELS; i++) {
arch/arm64/kvm/ptdump.c:106: snprintf(level[i].name, sizeof(level[i].name), "%u", i);
arch/arm64/kvm/ptdump.c-107-
--
arch/arm64/kvm/ptdump.c=276=void kvm_nested_s2_ptdump_create_debugfs(struct kvm_s2_mmu *mmu)
--
arch/arm64/kvm/ptdump.c-280-
arch/arm64/kvm/ptdump.c:281: snprintf(file_name, sizeof(file_name), "0x%016llx-0x%016llx-s2-%sabled",
arch/arm64/kvm/ptdump.c-282- mmu->tlb_vttbr,
--
arch/loongarch/kernel/kdebugfs.c=20=static ssize_t sfb_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
--
arch/loongarch/kernel/kdebugfs.c-26-
arch/loongarch/kernel/kdebugfs.c:27: s = snprintf(str, sizeof(str), "Boot State: %x\nCurrent State: %x\n", sfb_state, state);
arch/loongarch/kernel/kdebugfs.c-28-
--
arch/loongarch/kernel/kdebugfs.c=93=static ssize_t tso_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
--
arch/loongarch/kernel/kdebugfs.c-99-
arch/loongarch/kernel/kdebugfs.c:100: s = snprintf(str, sizeof(str), "Boot State: %d (%s)\n"
arch/loongarch/kernel/kdebugfs.c-101- "Current State: %d (%s)\n\n"
--
arch/m68k/emu/natfeat.c=55=void nfprint(const char *fmt, ...)
--
arch/m68k/emu/natfeat.c-60- va_start(ap, fmt);
arch/m68k/emu/natfeat.c:61: vsnprintf(buf, 256, fmt, ap);
arch/m68k/emu/natfeat.c-62- nf_call(nf_get_id("NF_STDERR"), virt_to_phys(buf));
--
arch/m68k/emu/nfblock.c=97=static int __init nfhd_init_one(int id, u32 blocks, u32 bsize)
--
arch/m68k/emu/nfblock.c-134- dev->disk->private_data = dev;
arch/m68k/emu/nfblock.c:135: snprintf(dev->disk->disk_name, sizeof(dev->disk->disk_name), "nfhd%u",
arch/m68k/emu/nfblock.c-136- dev_id);
--
arch/m68k/sun3/prom/printf.c=24=prom_printf(char *fmt, ...)
--
arch/m68k/sun3/prom/printf.c-32- ppbuf[0] = 'O';
arch/m68k/sun3/prom/printf.c:33: vsnprintf(ppbuf + 1, sizeof(ppbuf) - 1, fmt, args);
arch/m68k/sun3/prom/printf.c-34-#else
arch/m68k/sun3/prom/printf.c:35: vsnprintf(ppbuf, sizeof(ppbuf), fmt, args);
arch/m68k/sun3/prom/printf.c-36-#endif
--
arch/m68k/virt/config.c=80=void __init config_virt(void)
--
arch/m68k/virt/config.c-83-
arch/m68k/virt/config.c:84: snprintf(earlycon, sizeof(earlycon), "early_gf_tty,0x%08x",
arch/m68k/virt/config.c-85- virt_bi_data.tty.mmio);
--
arch/mips/bcm47xx/prom.c=48=__init void bcm47xx_set_system_type(u16 chip_id)
arch/mips/bcm47xx/prom.c-49-{
arch/mips/bcm47xx/prom.c:50: snprintf(bcm47xx_system_type, sizeof(bcm47xx_system_type),
arch/mips/bcm47xx/prom.c-51- (chip_id > 0x9999) ? "Broadcom BCM%d" :
--
arch/mips/bcm63xx/boards/board_bcm963xx.c=728=void __init board_prom_init(void)
--
arch/mips/bcm63xx/boards/board_bcm963xx.c-752- if(cfe[5] == 'd')
arch/mips/bcm63xx/boards/board_bcm963xx.c:753: snprintf(cfe_version, 11, "%s",
arch/mips/bcm63xx/boards/board_bcm963xx.c-754- (char *) &cfe[5]);
arch/mips/bcm63xx/boards/board_bcm963xx.c-755- else if (cfe[10] > 0)
arch/mips/bcm63xx/boards/board_bcm963xx.c:756: snprintf(cfe_version, sizeof(cfe_version),
arch/mips/bcm63xx/boards/board_bcm963xx.c-757- "%u.%u.%u-%u.%u-%u", cfe[5], cfe[6],
--
arch/mips/bcm63xx/boards/board_bcm963xx.c-759- else
arch/mips/bcm63xx/boards/board_bcm963xx.c:760: snprintf(cfe_version, sizeof(cfe_version),
arch/mips/bcm63xx/boards/board_bcm963xx.c-761- "%u.%u.%u-%u.%u", cfe[5], cfe[6],
--
arch/mips/bcm63xx/boards/board_bcm963xx.c-763- } else {
arch/mips/bcm63xx/boards/board_bcm963xx.c:764: snprintf(cfe_version, 12, "%s", (char *) &cfe[4]);
arch/mips/bcm63xx/boards/board_bcm963xx.c-765- }
--
arch/mips/bcm63xx/setup.c=141=const char *get_system_type(void)
--
arch/mips/bcm63xx/setup.c-143- static char buf[128];
arch/mips/bcm63xx/setup.c:144: snprintf(buf, sizeof(buf), "bcm63xx/%s (0x%04x/0x%02X)",
arch/mips/bcm63xx/setup.c-145- board_get_name(),
--
arch/mips/cavium-octeon/octeon-platform.c=483=static void __init octeon_fdt_set_phy(int eth, int phy_addr)
--
arch/mips/cavium-octeon/octeon-platform.c-555-
arch/mips/cavium-octeon/octeon-platform.c:556: snprintf(new_name, sizeof(new_name), "ethernet-phy@%x", phy_addr);
arch/mips/cavium-octeon/octeon-platform.c-557-
--
arch/mips/cavium-octeon/octeon-platform.c=649=static void __init octeon_fdt_pip_port(int iface, int i, int p, int max)
--
arch/mips/cavium-octeon/octeon-platform.c-656-
arch/mips/cavium-octeon/octeon-platform.c:657: snprintf(name_buffer, sizeof(name_buffer), "ethernet@%x", p);
arch/mips/cavium-octeon/octeon-platform.c-658- eth = fdt_subnode_offset(initial_boot_params, iface, name_buffer);
--
arch/mips/cavium-octeon/octeon-platform.c=682=static void __init octeon_fdt_pip_iface(int pip, int idx)
--
arch/mips/cavium-octeon/octeon-platform.c-688-
arch/mips/cavium-octeon/octeon-platform.c:689: snprintf(name_buffer, sizeof(name_buffer), "interface@%d", idx);
arch/mips/cavium-octeon/octeon-platform.c-690- iface = fdt_subnode_offset(initial_boot_params, pip, name_buffer);
--
arch/mips/cavium-octeon/octeon-platform.c=701=void __init octeon_fill_mac_addresses(void)
--
arch/mips/cavium-octeon/octeon-platform.c-724-
arch/mips/cavium-octeon/octeon-platform.c:725: snprintf(name_buffer, sizeof(name_buffer), "mix%d", i);
arch/mips/cavium-octeon/octeon-platform.c-726- alias_prop = fdt_getprop(initial_boot_params, aliases,
--
arch/mips/cavium-octeon/octeon-platform.c-747-
arch/mips/cavium-octeon/octeon-platform.c:748: snprintf(name_buffer, sizeof(name_buffer), "interface@%d", i);
arch/mips/cavium-octeon/octeon-platform.c-749- iface = fdt_subnode_offset(initial_boot_params, pip,
--
arch/mips/cavium-octeon/octeon-platform.c-755-
arch/mips/cavium-octeon/octeon-platform.c:756: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-757- "ethernet@%x", p);
--
arch/mips/cavium-octeon/octeon-platform.c=767=int __init octeon_prune_device_tree(void)
--
arch/mips/cavium-octeon/octeon-platform.c-800-
arch/mips/cavium-octeon/octeon-platform.c:801: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-802- "mix%d", i);
--
arch/mips/cavium-octeon/octeon-platform.c-842-
arch/mips/cavium-octeon/octeon-platform.c:843: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-844- "twsi%d", i);
--
arch/mips/cavium-octeon/octeon-platform.c-873-
arch/mips/cavium-octeon/octeon-platform.c:874: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-875- "smi%d", i);
--
arch/mips/cavium-octeon/octeon-platform.c-900-
arch/mips/cavium-octeon/octeon-platform.c:901: snprintf(name_buffer, sizeof(name_buffer),
arch/mips/cavium-octeon/octeon-platform.c-902- "uart%d", i);
--
arch/mips/cavium-octeon/setup.c=493=static void __init init_octeon_system_type(void)
--
arch/mips/cavium-octeon/setup.c-508-
arch/mips/cavium-octeon/setup.c:509: snprintf(octeon_system_type, sizeof(octeon_system_type), "%s (%s)",
arch/mips/cavium-octeon/setup.c-510- board_type, octeon_model_get_string(read_c0_prid()));
--
arch/mips/dec/prom/identify.c=44=const char *get_system_type(void)
--
arch/mips/dec/prom/identify.c-51- called = 1;
arch/mips/dec/prom/identify.c:52: snprintf(system, STR_BUF_LEN, "Digital %s",
arch/mips/dec/prom/identify.c-53- dec_system_strings[mips_machtype]);
--
arch/mips/generic/yamon-dt.c=75=__init int yamon_dt_append_memory(void *fdt,
--
arch/mips/generic/yamon-dt.c-109- for (i = 0; i < ARRAY_SIZE(var_names); i++) {
arch/mips/generic/yamon-dt.c:110: snprintf(param_name, sizeof(param_name), "%s=", var_names[i]);
arch/mips/generic/yamon-dt.c-111- var = strstr(arcs_cmdline, param_name);
--
arch/mips/generic/yamon-dt.c=157=__init int yamon_dt_serial_config(void *fdt)
--
arch/mips/generic/yamon-dt.c-179-
arch/mips/generic/yamon-dt.c:180: snprintf(mode_var_name, sizeof(mode_var_name), "modetty%u", uart);
arch/mips/generic/yamon-dt.c-181- mode_var = fw_getenv(mode_var_name);
--
arch/mips/generic/yamon-dt.c-210-
arch/mips/generic/yamon-dt.c:211: WARN_ON(snprintf(path, sizeof(path), "serial%u:%u%c%u%s",
arch/mips/generic/yamon-dt.c-212- uart, baud, parity, stop_bits,
--
arch/mips/kernel/mips-cm.c=393=void mips_cm_error_report(void)
--
arch/mips/kernel/mips-cm.c-421-
arch/mips/kernel/mips-cm.c:422: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-423- "CCA=%lu TR=%s MCmd=%s STag=%lu "
--
arch/mips/kernel/mips-cm.c-435-
arch/mips/kernel/mips-cm.c:436: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-437- "C3=%s C2=%s C1=%s C0=%s SC=%s "
--
arch/mips/kernel/mips-cm.c-451-
arch/mips/kernel/mips-cm.c:452: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-453- "Type=%s%s Instr=%s DW=%lu Way=%lu "
--
arch/mips/kernel/mips-cm.c-490- unsigned long bank_bit = (cm_error >> 28) & 0x1;
arch/mips/kernel/mips-cm.c:491: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-492- "%s ECC Error: Way=%lu (DWORD=%lu, Sets=%lu)"
--
arch/mips/kernel/mips-cm.c-510-
arch/mips/kernel/mips-cm.c:511: snprintf(buf, sizeof(buf),
arch/mips/kernel/mips-cm.c-512- "Decode Request Error: Type=%lu, Command=%lu"
--
arch/mips/kernel/traps.c=1991=void __noreturn nmi_exception_handler(struct pt_regs *regs)
--
arch/mips/kernel/traps.c-1997- bust_spinlocks(1);
arch/mips/kernel/traps.c:1998: snprintf(str, 100, "CPU%d NMI taken, CP0_EPC=%lx\n",
arch/mips/kernel/traps.c-1999- smp_processor_id(), regs->cp0_epc);
--
arch/mips/lantiq/prom.c=100=void __init prom_init(void)
--
arch/mips/lantiq/prom.c-103- ltq_soc_detect(&soc_info);
arch/mips/lantiq/prom.c:104: snprintf(soc_info.sys_type, LTQ_SYS_TYPE_LEN - 1, "%s rev %s",
arch/mips/lantiq/prom.c-105- soc_info.name, soc_info.rev_type);
--
arch/mips/mti-malta/malta-dtshim.c=131=static void __init append_memory(void *fdt, int root_off)
--
arch/mips/mti-malta/malta-dtshim.c-179- for (i = 0; i < ARRAY_SIZE(var_names); i++) {
arch/mips/mti-malta/malta-dtshim.c:180: snprintf(param_name, sizeof(param_name), "%s=", var_names[i]);
arch/mips/mti-malta/malta-dtshim.c-181- var = strstr(arcs_cmdline, param_name);
--
arch/mips/pci/pci-xtalk-bridge.c=568=static int bridge_get_partnum(u64 baddr, char *partnum)
--
arch/mips/pci/pci-xtalk-bridge.c-575-
arch/mips/pci/pci-xtalk-bridge.c:576: snprintf(prefix, sizeof(prefix), "bridge-%012llx-0b-", baddr);
arch/mips/pci/pci-xtalk-bridge.c-577-
--
arch/mips/ralink/mt7620.c=220=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/mt7620.c-226-
arch/mips/ralink/mt7620.c:227: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/mt7620.c-228- "MediaTek %s ver:%u eco:%u",
--
arch/mips/ralink/mt7621.c=167=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/mt7621.c-198-
arch/mips/ralink/mt7621.c:199: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/mt7621.c-200- "MediaTek %s ver:%u eco:%u",
--
arch/mips/ralink/rt288x.c=90=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/rt288x.c-97-
arch/mips/ralink/rt288x.c:98: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/rt288x.c-99- "Ralink %s id:%u rev:%u",
--
arch/mips/ralink/rt305x.c=193=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/rt305x.c-196-
arch/mips/ralink/rt305x.c:197: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/rt305x.c-198- "Ralink %s id:%u rev:%u",
--
arch/mips/ralink/rt3883.c=90=void __init prom_soc_init(struct ralink_soc_info *soc_info)
--
arch/mips/ralink/rt3883.c-97-
arch/mips/ralink/rt3883.c:98: snprintf(soc_info->sys_type, RAMIPS_SYS_TYPE_LEN,
arch/mips/ralink/rt3883.c-99- "Ralink %s ver:%u eco:%u",
--
arch/mips/sgi-ip27/ip27-xtalk.c=26=static void bridge_platform_create(nasid_t nasid, int widget, int masterwid)
--
arch/mips/sgi-ip27/ip27-xtalk.c-42-
arch/mips/sgi-ip27/ip27-xtalk.c:43: snprintf(wd->dev_id, sizeof(wd->dev_id), "bridge-%012lx",
arch/mips/sgi-ip27/ip27-xtalk.c-44- offset + (widget << SWIN_SIZE_BITS));
--
arch/mips/sgi-ip30/ip30-xtalk.c=39=static void bridge_platform_create(int widget, int masterwid)
--
arch/mips/sgi-ip30/ip30-xtalk.c-52-
arch/mips/sgi-ip30/ip30-xtalk.c:53: snprintf(wd->dev_id, sizeof(wd->dev_id), "bridge-%012lx",
arch/mips/sgi-ip30/ip30-xtalk.c-54- IP30_SWIN_BASE(widget));
--
arch/mips/txx9/generic/setup.c=635=void __init txx9_iocled_init(unsigned long baseaddr,
--
arch/mips/txx9/generic/setup.c-675- struct gpio_led *led = &iocled->leds[i];
arch/mips/txx9/generic/setup.c:676: snprintf(iocled->names[i], sizeof(iocled->names[i]),
arch/mips/txx9/generic/setup.c-677- "iocled:%s:%u", color, i);
--
arch/parisc/kernel/cache.c=135=show_cache_info(struct seq_file *m)
--
arch/parisc/kernel/cache.c-141- if (cache_info.dc_loop != 1)
arch/parisc/kernel/cache.c:142: snprintf(buf, 32, "%lu-way associative", cache_info.dc_loop);
arch/parisc/kernel/cache.c-143- seq_printf(m, "D-cache\t\t: %ld KB (%s%s, %s, alias=%d)\n",
--
arch/parisc/kernel/drivers.c=493=alloc_pa_dev(unsigned long hpa, struct hardware_path *mod_path)
--
arch/parisc/kernel/drivers.c-536- name = parisc_hardware_description(&dev->id) ? : "unknown";
arch/parisc/kernel/drivers.c:537: snprintf(dev->name, sizeof(dev->name), "%s [%s]",
arch/parisc/kernel/drivers.c-538- name, parisc_pathname(dev));
--
arch/powerpc/kernel/fadump.c=149=void __init fadump_append_bootargs(void)
--
arch/powerpc/kernel/fadump.c-175- pr_debug("Cmdline: %s\n", boot_command_line);
arch/powerpc/kernel/fadump.c:176: snprintf(boot_command_line + len, COMMAND_LINE_SIZE - len, " %s", append_args);
arch/powerpc/kernel/fadump.c-177- pr_info("Updated cmdline: %s\n", boot_command_line);
--
arch/powerpc/kernel/rtas.c=1644=void rtas_os_term(char *str)
--
arch/powerpc/kernel/rtas.c-1659-
arch/powerpc/kernel/rtas.c:1660: snprintf(rtas_os_term_buf, 2048, "OS panic: %s", str);
arch/powerpc/kernel/rtas.c-1661-
--
arch/powerpc/kernel/rtas_flash.c=449=static int get_validate_flash_msg(struct rtas_validate_flash_t *args_buf,
--
arch/powerpc/kernel/rtas_flash.c-457- (args_buf->update_results == VALIDATE_TMP_UPDATE))
arch/powerpc/kernel/rtas_flash.c:458: n += snprintf(msg + n, msglen - n, "%s\n",
arch/powerpc/kernel/rtas_flash.c-459- args_buf->buf);
--
arch/powerpc/kernel/udbg.c=114=void udbg_printf(const char *fmt, ...)
--
arch/powerpc/kernel/udbg.c-120- va_start(args, fmt);
arch/powerpc/kernel/udbg.c:121: vsnprintf(buf, UDBG_BUFSIZE, fmt, args);
arch/powerpc/kernel/udbg.c-122- udbg_puts(buf);
--
arch/powerpc/kexec/file_load_64.c=194=static int add_usable_mem_property(void *fdt, struct device_node *dn,
--
arch/powerpc/kexec/file_load_64.c-203-
arch/powerpc/kexec/file_load_64.c:204: if (snprintf(path, NODE_PATH_LEN, "%pOF", dn) > (NODE_PATH_LEN - 1)) {
arch/powerpc/kexec/file_load_64.c-205- pr_err("Buffer (%d) too small for memory node: %pOF\n",
--
arch/powerpc/kvm/book3s_hv.c=2864=static ssize_t debugfs_timings_read(struct file *file, char __user *buf,
--
arch/powerpc/kvm/book3s_hv.c-2899- if (!ok)
arch/powerpc/kvm/book3s_hv.c:2900: snprintf(s, buf_end - s, "%s: stuck\n",
arch/powerpc/kvm/book3s_hv.c-2901- timings[i].name);
arch/powerpc/kvm/book3s_hv.c-2902- else
arch/powerpc/kvm/book3s_hv.c:2903: snprintf(s, buf_end - s,
arch/powerpc/kvm/book3s_hv.c-2904- "%s: %llu %llu %llu %llu\n",
--
arch/powerpc/perf/hv-24x7.c=388=static char *event_fmt(struct hv_24x7_event_data *event, unsigned int domain)
--
arch/powerpc/perf/hv-24x7.c-396- case HV_PERF_DOMAIN_PHYS_CHIP:
arch/powerpc/perf/hv-24x7.c:397: snprintf(buf, sizeof(buf), "%d", domain);
arch/powerpc/perf/hv-24x7.c-398- domain_str = buf;
--
arch/powerpc/platforms/512x/clock-commonclk.c=704=static void __init mpc512x_clk_setup_clock_tree(struct device_node *np, int busfreq)
--
arch/powerpc/platforms/512x/clock-commonclk.c-822- char name[12];
arch/powerpc/platforms/512x/clock-commonclk.c:823: snprintf(name, sizeof(name), "psc%d", mclk_idx);
arch/powerpc/platforms/512x/clock-commonclk.c-824- clks[MPC512x_CLK_PSC0 + mclk_idx] = mpc512x_clk_gated(
--
arch/powerpc/platforms/512x/clock-commonclk.c=951=static void __init mpc5121_clk_provide_migration_support(void)
--
arch/powerpc/platforms/512x/clock-commonclk.c-988- of_address_to_resource(np, 0, &res); \
arch/powerpc/platforms/512x/clock-commonclk.c:989: snprintf(devname, sizeof(devname), "%pa.%s", &res.start, np->name); \
arch/powerpc/platforms/512x/clock-commonclk.c-990-} while (0)
--
arch/powerpc/platforms/82xx/ep8248e.c=109=static int ep8248e_mdio_probe(struct platform_device *ofdev)
--
arch/powerpc/platforms/82xx/ep8248e.c-130- bus->parent = &ofdev->dev;
]
]
|
| 321/4 |
2026/05/18 19:11 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 88943
output: 25
thoughts: 20
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Streamlining the BPF Helper Analysis**
Right, let's cut through the noise. I need to quickly pinpoint where `snprintf` is being used within the BPF helper functions in the kernel. No need for a convoluted manual search. The most efficient approach here is clearly `git grep snprintf kernel/bpf/helpers.c`. That gives me a precise, targeted scan, focusing exactly on the potential areas of interest within that specific helper file. No need to waste time or go any deeper at this stage. That should get me what I need.
|
| 322/4 |
2026/05/18 19:11 |
tool |
grepper |
0m
Error:
Args: map[Expression:snprintf.*kernel/bpf/helpers.c]
Results:
|
| 323/4 |
2026/05/18 19:11 |
llm |
patch-generator |
1m
Model:
Tokens:
input: 89137
output: 16
thoughts: 20
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**BPF Helper Investigation: Snprintf Deep Dive**
Alright, so I'm thinking, where are we with `snprintf` and the BPF helpers? The most direct approach, as always, seems to be a quick `grep` within the kernel's BPF helper implementation file, specifically `kernel/bpf/helpers.c`. I expect to find a use of the system `snprintf` and want to understand how it is called and used, whether directly or in some indirect wrapper function, to allow for the use of dynamic string formatting in the BPF context. I'm keen to see the implementation details, especially any considerations given to security or resource limitations within the BPF environment. It could be directly exposed, perhaps under tight constraints, or wrapped for some security reasons. I'm hoping to get a clearer picture of the function signature, any limitations on the format string (e.g., restricted format specifiers), and if there's any sophisticated error handling in place. Furthermore, I'll pay attention to any performance considerations; BPF is all about efficiency, so I expect they've optimized it to the nth degree. Ultimately, I'm aiming to grasp the integration of `snprintf` with the BPF subsystem and understand the nuances.
|
| 324/4 |
2026/05/18 19:12 |
tool |
grepper |
0m
Error:
Args:
|
| 325/4 |
2026/05/18 19:12 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 89491
output: 23
thoughts: 17
Thoughts: Okay, here's the summary, rendered as your thoughts:
**The Familiar Comfort of `patch-diff`**
Right, let me think... I need to review those recent modifications I made. Hmm, no need to overcomplicate things here. I can just reach for `patch-diff` again. It's a tried-and-true method, really efficient for quickly highlighting the specific differences between the current version and the previous one. Why reinvent the wheel when a perfectly good tool is already at my disposal? It's a simple command, but it gets the job done reliably. And frankly, the speed of its execution feels like a good use of my time.
|
| 326/4 |
2026/05/18 19:13 |
tool |
patch-diff |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Output:diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2bb60200c..cc4994804 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -804,259 +804,266 @@ void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
- err = snprintf(tmp_buf,
- (tmp_buf_end - tmp_buf),
+ size_t limit = tmp_buf < tmp_buf_end ? tmp_buf_end - tmp_buf : 0;
+
+ err = snprintf(tmp_buf, limit,
"%pB",
(void *)(long)raw_args[num_spec]);
- tmp_buf += (err + 1);
+ if (err < 0)
+ err = 0;
+ tmp_buf += min_t(size_t, err + 1, limit);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
- if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
+ if (tmp_buf + sizeof_cur_ip > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
- err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
- ip_spec, &cur_ip);
+ {
+ size_t limit = tmp_buf < tmp_buf_end ? tmp_buf_end - tmp_buf : 0;
- tmp_buf += err + 1;
+ err = snprintf(tmp_buf, limit, ip_spec, &cur_ip);
+ if (err < 0)
+ err = 0;
+ tmp_buf += min_t(size_t, err + 1, limit);
+ }
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
- if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
]
|
| 327/4 |
2026/05/18 19:13 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 92772
output: 50
thoughts: 1420
Thoughts: **Code Review: Addressing Potential Buffer Overflows**
Okay, let's break this down. The initial code seems sound, but I need to make absolutely sure there aren't any subtle buffer overflows. I'm looking closely at how `tmp_buf` is managed within the kernel.
First, I examined `tmp_buf + sizeof_cur_arg > tmp_buf_end`, making sure that pointer arithmetic isn't going to cause an overflow, especially since `sizeof_cur_arg` is small. No problems there; pointer arithmetic is safe given the size of the buffer and the size of the arguments. `tmp_buf_end` is essentially a fixed offset from `tmp_buf`, so that check is sound.
Next, I scrutinized `min_t(size_t, err + 1, limit)`. The potential for negative `err` values worried me, but the code already handles those by setting `err = 0` if `err < 0`, so `err + 1` is always greater than or equal to 1. If `limit` is zero, `min_t` is zero; otherwise it's `err + 1` or the specified `limit`. I made sure if `err >= limit`, the `min_t` will never let me go past `limit`, so I know `tmp_buf` will advance by at most `limit` - that's safe. Then, I considered the edge case of `tmp_buf` *already* being beyond `tmp_buf_end`. In that scenario, `limit` would effectively be zero, preventing any further advance. Safe.
Now, I systematically checked every place `tmp_buf` gets incremented:
* **`%pB` and `%pI`:** `tmp_buf += min_t(size_t, err + 1, limit);` Perfect. The `min_t` function has been validated.
* **`%s`:** `tmp_buf += err;` This depends on `bpf_trace_copy_string`. My immediate concern was that if `tmp_buf > tmp_buf_end`, `tmp_buf_end - tmp_buf` would be a huge `size_t` value. However, the code *now* includes a check:
```c
if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
```
Before calling `bpf_trace_copy_string`, it now exits if the bounds check is exceeded. Thus `bpf_trace_copy_string` receives a positive `bufsz` and returns the number of bytes that have been copied which is at most `bufsz`, so `tmp_buf` remains within bounds.
* **`%c`:**
```c
if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
```
The bounds check is now `tmp_buf >= tmp_buf_end`. If the check fails the function exits, and if it passes `tmp_buf` gets incremented, which will be valid if `tmp_buf < tmp_buf_end`.
* **`nocopy_fmt`:**
```c
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
...
tmp_buf += sizeof_cur_arg;
```
I confirmed that `PTR_ALIGN` aligns `tmp_buf`. If `tmp_buf <= tmp_buf_end`, `PTR_ALIGN` ensures that after alignment it will still be `<= tmp_buf_end`. If the check `tmp_buf + sizeof_cur_arg > tmp_buf_end` fails, it exits. Otherwise, incrementing `tmp_buf` will maintain the bound.
I can now confidently say that `tmp_buf` *will* never exceed `tmp_buf_end` in any circumstance. This absolutely and completely resolves the potential out-of-bounds write and the subtle signedness bug.
Now, let's summarize the root cause and the fix. The original problem arose because `snprintf` (used by `%pB` and `%pI`) could return the number of bytes *required* to write the
|
| 328/4 |
2026/05/18 19:13 |
tool |
codeexpert |
22m
Args: map[Question:Is there any other place in `kernel/bpf/helpers.c` where `snprintf` is used to format strings into a buffer without checking the return value properly?]
Results: map[Answer:Based on a comprehensive review of `kernel/bpf/helpers.c`, the improper use of `snprintf` is localized to exactly one function: `bpf_bprintf_prepare`.
### Findings
1. **Vulnerable Function:** `bpf_bprintf_prepare`
This function is responsible for pre-formatting complex arguments (like symbol names `%pB` and IP addresses `%pI4`/`%pI6`) into a binary buffer (`bin_args`) before they are processed by the final formatting engine (`bstr_printf`).
2. **Specific Vulnerable Instances:**
There are exactly two calls to `snprintf` in `kernel/bpf/helpers.c`, both within this function:
* **Instance 1 (Line 924):** Used for symbol formatting (`%pB`).
```c
err = snprintf(tmp_buf, (tmp_buf_end - tmp_buf), "%pB", (void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1); // BUG: err can exceed remaining space
```
* **Instance 2 (Line 965):** Used for IP address formatting.
```c
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf, ip_spec, &cur_ip);
tmp_buf += err + 1; // BUG: err can exceed remaining space
```
3. **The Vulnerability Mechanism:**
In both cases, the code assumes `snprintf` returns the number of bytes actually written. However, according to standard C and kernel semantics, `snprintf` returns the number of bytes that *would* have been written if the buffer were large enough. If truncation occurs, `err` becomes greater than or equal to the remaining space (`tmp_buf_end - tmp_buf`). Advancing `tmp_buf` by `err + 1` pushes the pointer beyond the allocated buffer boundary (`tmp_buf_end`), leading to out-of-bounds writes in subsequent loop iterations.
4. **Exhaustive Search for Other Formatting Functions:**
* **`scnprintf` / `vsnprintf` / `vscnprintf` / `sprintf`:** None of these functions are called directly within `kernel/bpf/helpers.c`.
* **`bstr_printf`:** This is used in `____bpf_snprintf` (the implementation of the `bpf_snprintf` helper). Its usage there is safe because it does not use the return value to perform pointer arithmetic on an internal buffer; it simply returns the value (plus 1) to the BPF program as intended.
* **String Copying Helpers:** Functions like `bpf_trace_copy_string` use `strncpy_from_kernel_nofault`, which correctly respects the buffer size and returns the actual number of bytes copied, avoiding the `snprintf` pitfall.
### Conclusion
There are **no other places** in `kernel/bpf/helpers.c` where `snprintf` (or any similar formatting function) is used to format strings into a buffer without proper return value checking. The issue is entirely contained within the `bpf_bprintf_prepare` function, although its impact is significant as it is a shared utility used by multiple BPF helpers including `bpf_trace_printk`, `bpf_snprintf`, and `bpf_seq_printf`.]
|
| 329/5 |
2026/05/18 19:13 |
agent |
codeexpert |
22m
Model:
Instruction:
You are a capable Linux kernel developer tasked with researching complex questions
about kernel source code. You will be given a concrete question, and need to provide
a concrete answer.
Use tools extensively while researching the question. Don't make assumptions,
or rely on your previous knowledge about the kernel source code, use available tools
to access the actual source code.
Use all available sources of information:
- kernel source code
- documentation in the Documentation dir in the source tree
- git commits descriptions, git blame
Prefer calling several tools at the same time to save round-trips.
Prompt: Is there any other place in `kernel/bpf/helpers.c` where `snprintf` is used to format strings into a buffer without checking the return value properly?
Reply: Based on a comprehensive review of `kernel/bpf/helpers.c`, the improper use of `snprintf` is localized to exactly one function: `bpf_bprintf_prepare`.
### Findings
1. **Vulnerable Function:** `bpf_bprintf_prepare`
This function is responsible for pre-formatting complex arguments (like symbol names `%pB` and IP addresses `%pI4`/`%pI6`) into a binary buffer (`bin_args`) before they are processed by the final formatting engine (`bstr_printf`).
2. **Specific Vulnerable Instances:**
There are exactly two calls to `snprintf` in `kernel/bpf/helpers.c`, both within this function:
* **Instance 1 (Line 924):** Used for symbol formatting (`%pB`).
```c
err = snprintf(tmp_buf, (tmp_buf_end - tmp_buf), "%pB", (void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1); // BUG: err can exceed remaining space
```
* **Instance 2 (Line 965):** Used for IP address formatting.
```c
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf, ip_spec, &cur_ip);
tmp_buf += err + 1; // BUG: err can exceed remaining space
```
3. **The Vulnerability Mechanism:**
In both cases, the code assumes `snprintf` returns the number of bytes actually written. However, according to standard C and kernel semantics, `snprintf` returns the number of bytes that *would* have been written if the buffer were large enough. If truncation occurs, `err` becomes greater than or equal to the remaining space (`tmp_buf_end - tmp_buf`). Advancing `tmp_buf` by `err + 1` pushes the pointer beyond the allocated buffer boundary (`tmp_buf_end`), leading to out-of-bounds writes in subsequent loop iterations.
4. **Exhaustive Search for Other Formatting Functions:**
* **`scnprintf` / `vsnprintf` / `vscnprintf` / `sprintf`:** None of these functions are called directly within `kernel/bpf/helpers.c`.
* **`bstr_printf`:** This is used in `____bpf_snprintf` (the implementation of the `bpf_snprintf` helper). Its usage there is safe because it does not use the return value to perform pointer arithmetic on an internal buffer; it simply returns the value (plus 1) to the BPF program as intended.
* **String Copying Helpers:** Functions like `bpf_trace_copy_string` use `strncpy_from_kernel_nofault`, which correctly respects the buffer size and returns the actual number of bytes copied, avoiding the `snprintf` pitfall.
### Conclusion
There are **no other places** in `kernel/bpf/helpers.c` where `snprintf` (or any similar formatting function) is used to format strings into a buffer without proper return value checking. The issue is entirely contained within the `bpf_bprintf_prepare` function, although its impact is significant as it is a shared utility used by multiple BPF helpers including `bpf_trace_printk`, `bpf_snprintf`, and `bpf_seq_printf`.
|
| 330/6 |
2026/05/18 19:13 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 2288
output: 41
thoughts: 144
Thoughts: **Analyzing `snprintf` Usage in `kernel/bpf/helpers.c`**
Okay, so the user's got a question about `snprintf` within `kernel/bpf/helpers.c`, and they're wondering if there are any potential vulnerabilities stemming from improper return value checks. Makes sense; always a good idea to scrutinize string formatting functions, especially in kernel space.
Alright, so my immediate plan of action is pretty straightforward:
1. **Locate the instances:** I'm going to kick things off by grepping for `snprintf` directly within `kernel/bpf/helpers.c`. Gotta get a list of all the call sites first.
2. **Inspect Each One:** For each occurrence, I need to do a code review. I'll need to meticulously analyze the surrounding code to determine if the return value of `snprintf` is being checked, and more importantly, *how* it's being checked.
3. **Define "Proper":** And here's where my expertise comes into play. A "proper" check means that the return value check needs to account for both negative return values (indicating an error) and potentially truncation (when the return value is greater than or equal to the buffer size). Those are the primary failure modes I'll be looking for.
Let's start by getting that list of occurrences. Time to get to work!
|
| 331/6 |
2026/05/18 19:13 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:5 Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924 SourceSnippet: 919: goto nocopy_fmt;
920: }
921:
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965 SourceSnippet: 960: * pre-formatted as strings, ironically, the easiest way
961: * to do that is to call snprintf.
962: */
963: ip_spec[2] = fmt[i - 1];
964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
969: num_spec++;
970:
]] TruncatedOutput:false]
|
| 332/6 |
2026/05/18 19:13 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 3232
output: 44
thoughts: 505
Thoughts: **Analyzing Potential Buffer Overflows in `bpf_bprintf_prepare`**
Okay, so I've spotted two uses of `snprintf` within `kernel/bpf/helpers.c`, specifically inside the `bpf_bprintf_prepare` function, and that's immediately caught my attention. I need to dig deeper.
The first instance, line 923, looks concerning. `snprintf` is called to format a pointer representation (`%pB`) into `tmp_buf`. The return value, `err`, which represents the number of characters written (or would have been written), is then directly *added* to `tmp_buf`. My initial thought is that if `snprintf` truncates the output—which is perfectly possible given the buffer size `(tmp_buf_end - tmp_buf)`—then `err` will be greater than or equal to `(tmp_buf_end - tmp_buf)`. Adding that potentially large `err` to `tmp_buf` without any bounds check could easily cause `tmp_buf` to point outside the allocated buffer, leading to a classic buffer overflow.
The second occurrence, on line 965, has the same vulnerability pattern. `snprintf` is used with a different format string (`ip_spec` and `&cur_ip`), and again, the return value `err` is directly added to `tmp_buf`, creating a similar risk of overflow if the output is truncated or an error is encountered.
Now, wait a minute... I need to be precise. I need to make sure I am accounting for `snprintf`'s return value semantics. `snprintf` returns the *number of characters* that *would* have been written, including the null terminator, if the buffer had been large enough. If the return value (`err`) is *greater than or equal to* `(tmp_buf_end - tmp_buf)`, truncation *has* occurred. If `err` is negative, an error occurred. And in either of these situations, the `tmp_buf += err + 1;` line causes a real problem. No error-checking, no bounds-checking, just raw pointer arithmetic. My Spidey-sense is tingling.
The crucial next step is understanding the context. I must examine the *entire* `bpf_bprintf_prepare` function. I need to know how `tmp_buf` and `tmp_buf_end` are initialized, how the buffer is managed *before* and *after* these `snprintf` calls. Are there any other checks? Is this buffer allocated dynamically, or is it a fixed-size buffer? The answers to these questions are going to tell me the real severity of this potential overflow. I need to see the complete picture to assess the real risk.
|
| 333/6 |
2026/05/18 19:13 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_prepare]
Results: map[SourceCode: 804: /*
805: * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
806: *
807: * Returns a negative value if fmt is an invalid format string or 0 otherwise.
808: *
809: * This can be used in two ways:
810: * - Format string verification only: when data->get_bin_args is false
811: * - Arguments preparation: in addition to the above verification, it writes in
812: * data->bin_args a binary representation of arguments usable by bstr_printf
813: * where pointers from BPF have been sanitized.
814: *
815: * In argument preparation mode, if 0 is returned, safe temporary buffers are
816: * allocated and bpf_bprintf_cleanup should be called to free them after use.
817: */
818: int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
819: u32 num_args, struct bpf_bprintf_data *data)
820: {
821: bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
822: char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
823: struct bpf_bprintf_buffers *buffers = NULL;
824: size_t sizeof_cur_arg, sizeof_cur_ip;
825: int err, i, num_spec = 0;
826: u64 cur_arg;
827: char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
828:
829: fmt_end = strnchr(fmt, fmt_size, 0);
830: if (!fmt_end)
831: return -EINVAL;
832: fmt_size = fmt_end - fmt;
833:
834: if (get_buffers && bpf_try_get_buffers(&buffers))
835: return -EBUSY;
836:
837: if (data->get_bin_args) {
838: if (num_args)
839: tmp_buf = buffers->bin_args;
840: tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
841: data->bin_args = (u32 *)tmp_buf;
842: }
843:
844: if (data->get_buf)
845: data->buf = buffers->buf;
846:
847: for (i = 0; i < fmt_size; i++) {
848: unsigned char c = fmt[i];
849:
850: /*
851: * Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
852: * through unchanged, while still rejecting ASCII control bytes.
853: */
854: if (isascii(c) && !isprint(c) && !isspace(c)) {
855: err = -EINVAL;
856: goto out;
857: }
858:
859: if (fmt[i] != '%')
860: continue;
861:
862: if (fmt[i + 1] == '%') {
863: i++;
864: continue;
865: }
866:
867: if (num_spec >= num_args) {
868: err = -EINVAL;
869: goto out;
870: }
871:
872: /* The string is zero-terminated so if fmt[i] != 0, we can
873: * always access fmt[i + 1], in the worst case it will be a 0
874: */
875: i++;
876: c = fmt[i];
877: /*
878: * The format parser below only understands ASCII conversion
879: * specifiers and modifiers, so reject non-ASCII after '%'.
880: */
881: if (!isascii(c)) {
882: err = -EINVAL;
883: goto out;
884: }
885:
886: /* skip optional "[0 +-][num]" width formatting field */
887: while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
888: fmt[i] == ' ')
889: i++;
890: if (fmt[i] >= '1' && fmt[i] <= '9') {
891: i++;
892: while (fmt[i] >= '0' && fmt[i] <= '9')
893: i++;
894: }
895:
896: if (fmt[i] == 'p') {
897: sizeof_cur_arg = sizeof(long);
898:
899: if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
900: ispunct(fmt[i + 1])) {
901: if (tmp_buf)
902: cur_arg = raw_args[num_spec];
903: goto nocopy_fmt;
904: }
905:
906: if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
907: fmt[i + 2] == 's') {
908: fmt_ptype = fmt[i + 1];
909: i += 2;
910: goto fmt_str;
911: }
912:
913: if (fmt[i + 1] == 'K' ||
914: fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
915: fmt[i + 1] == 'S') {
916: if (tmp_buf)
917: cur_arg = raw_args[num_spec];
918: i++;
919: goto nocopy_fmt;
920: }
921:
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
930:
931: i++;
932: num_spec++;
933: continue;
934: }
935:
936: /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
937: if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
938: (fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
939: err = -EINVAL;
940: goto out;
941: }
942:
943: i += 2;
944: if (!tmp_buf)
945: goto nocopy_fmt;
946:
947: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
948: if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
949: err = -ENOSPC;
950: goto out;
951: }
952:
953: unsafe_ptr = (char *)(long)raw_args[num_spec];
954: err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
955: sizeof_cur_ip);
956: if (err < 0)
957: memset(cur_ip, 0, sizeof_cur_ip);
958:
959: /* hack: bstr_printf expects IP addresses to be
960: * pre-formatted as strings, ironically, the easiest way
961: * to do that is to call snprintf.
962: */
963: ip_spec[2] = fmt[i - 1];
964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
969: num_spec++;
970:
971: continue;
972: } else if (fmt[i] == 's') {
973: fmt_ptype = fmt[i];
974: fmt_str:
975: if (fmt[i + 1] != 0 &&
976: !isspace(fmt[i + 1]) &&
977: !ispunct(fmt[i + 1])) {
978: err = -EINVAL;
979: goto out;
980: }
981:
982: if (!tmp_buf)
983: goto nocopy_fmt;
984:
985: if (tmp_buf_end == tmp_buf) {
986: err = -ENOSPC;
987: goto out;
988: }
989:
990: unsafe_ptr = (char *)(long)raw_args[num_spec];
991: err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
992: fmt_ptype,
993: tmp_buf_end - tmp_buf);
994: if (err < 0) {
995: tmp_buf[0] = '\0';
996: err = 1;
997: }
998:
999: tmp_buf += err;
1000: num_spec++;
1001:
1002: continue;
1003: } else if (fmt[i] == 'c') {
1004: if (!tmp_buf)
1005: goto nocopy_fmt;
1006:
1007: if (tmp_buf_end == tmp_buf) {
1008: err = -ENOSPC;
1009: goto out;
1010: }
1011:
1012: *tmp_buf = raw_args[num_spec];
1013: tmp_buf++;
1014: num_spec++;
1015:
1016: continue;
1017: }
1018:
1019: sizeof_cur_arg = sizeof(int);
1020:
1021: if (fmt[i] == 'l') {
1022: sizeof_cur_arg = sizeof(long);
1023: i++;
1024: }
1025: if (fmt[i] == 'l') {
1026: sizeof_cur_arg = sizeof(long long);
1027: i++;
1028: }
1029:
1030: if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
1031: fmt[i] != 'x' && fmt[i] != 'X') {
1032: err = -EINVAL;
1033: goto out;
1034: }
1035:
1036: if (tmp_buf)
1037: cur_arg = raw_args[num_spec];
1038: nocopy_fmt:
1039: if (tmp_buf) {
1040: tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
1041: if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
1042: err = -ENOSPC;
1043: goto out;
1044: }
1045:
1046: if (sizeof_cur_arg == 8) {
1047: *(u32 *)tmp_buf = *(u32 *)&cur_arg;
1048: *(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
1049: } else {
1050: *(u32 *)tmp_buf = (u32)(long)cur_arg;
1051: }
1052: tmp_buf += sizeof_cur_arg;
1053: }
1054: num_spec++;
1055: }
1056:
1057: err = 0;
1058: out:
1059: if (err)
1060: bpf_bprintf_cleanup(data);
1061: return err;
1062: }
SourceFile:kernel/bpf/helpers.c]
|
| 334/6 |
2026/05/18 19:13 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 8214
output: 41
thoughts: 833
Thoughts: **Analyzing Potential Buffer Overflows in `snprintf` Calls**
Okay, let's dissect these `snprintf` calls within this context. I need to be thorough. First, that initial one, lines 924-928, demands attention:
```c
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
```
The core issue I'm seeing is this: if `snprintf` truncates, `err` will be greater than or equal to the remaining buffer space, which is `tmp_buf_end - tmp_buf`. The update `tmp_buf += (err + 1)` *will* push `tmp_buf` past `tmp_buf_end`, right? That's a classic overflow scenario! I need to be sure.
My immediate thought is, what's downstream from this? Ah, line 948...
```c
948: if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
```
If my earlier hypothesis is correct, then `tmp_buf` *is* beyond `tmp_buf_end`. That difference will be a potentially large negative value since `tmp_buf` and `tmp_buf_end` are `char *`. This is a pointer difference, which is a `ptrdiff_t`. If `tmp_buf > tmp_buf_end`, then the result of the subtraction, `tmp_buf_end - tmp_buf`, is negative. When compared to the `size_t` `sizeof_cur_ip`, the compiler will do an implicit conversion to `size_t`, which will make the negative value wrap around to a very large positive number! That's it!
So, the comparison becomes effectively `(a very large size_t) < sizeof_cur_ip`, which will almost always be *false*. Therefore, the check *won't* catch the overflow. It will write where it shouldn't.
Now, let's verify my assumptions. Let's see... `tmp_buf` and `tmp_buf_end` are `char *` and `sizeof_cur_ip` is `size_t`. That's important. With that, a negative `ptrdiff_t` in the comparison becomes a large `size_t`. The comparison's evaluation is the root of the vulnerability. This needs fixing.
The second call, lines 965-968:
```c
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
```
This one has the exact same problem! Same root cause, same potential for buffer overrun.
And, what about negative return values? That would be a failure. The standard says `snprintf` can return negative values upon error (though it's unlikely here). If `err` is negative, `tmp_buf += err + 1` could move `tmp_buf` *backwards* or keep it at the same location. That needs to be addressed as a separate concern.
I better scan the file for other `snprintf` calls. I want to be 100% sure this isn't a systemic issue. I'll search for it again. There's a lot on the line here.
|
| 335/6 |
2026/05/18 19:13 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924 SourceSnippet: 923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965 SourceSnippet: 964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
]] TruncatedOutput:false]
|
| 336/6 |
2026/05/18 19:13 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 10161
output: 46
thoughts: 1426
Thoughts: **Vulnerability Analysis of `bpf_bprintf_prepare`**
Okay, let's break this down. First, I'm noting the use of `bpf_trace_copy_string`. It's a potentially safer approach to string copying because of its error handling. That's good.
Now, I'm focusing on those `snprintf` calls. The first one is concerning because there's no check that ensures `tmp_buf` is actually *before* `tmp_buf_end` before the function is called. If the buffer pointers are somehow skewed, the size calculation (`tmp_buf_end - tmp_buf`) could result in a huge `size_t` value. Passing a huge size to `snprintf` when the buffer is actually smaller leads to a buffer overflow. It's using `%pB`, which seems to be a pointer in a binary format, so it's probably using the raw argument.
The second `snprintf` call, dealing with IP addresses, is more interesting. There's a check here, thankfully, at line 948. But the check is against the binary size of the IP address (`sizeof_cur_ip`), not the string representation that `snprintf` will produce. This is a critical oversight. The IP address can take up a string of 15 or 39 characters plus null termination. If the available buffer space is less than the string's length but greater than the binary representation, `snprintf` will write past the allocated space. And after the call, line 968 increments `tmp_buf`, likely by more than the remaining buffer space. So, the check is inadequate.
The note about `bstr_printf` raises another point. It looks like `bstr_printf` expects *strings*. This means it's calling `snprintf` to format these IP addresses. The comments are explaining it's a "hack" to use `snprintf` to get this formatting. The `bstr_printf` function takes binary arguments, and in this case, the `bpf_bprintf_prepare` function is putting string representations of IP addresses into a binary argument buffer.
The `tmp_buf` variable is likely `buffers->bin_args`, meaning it's using a buffer of size `MAX_BPRINTF_BIN_ARGS`. I need to verify that buffer size and how it is being initialized, to confirm the vulnerability.
|
| 337/6 |
2026/05/18 19:13 |
tool |
codesearch-definition-source |
0m
Error:
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:MAX_BPRINTF_BIN_ARGS]
Results: map[SourceCode: SourceFile:]
|
| 338/6 |
2026/05/18 19:13 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 12176
output: 33
thoughts: 12
Thoughts: Okay, here's the summary, expressed as my internal thought process, assuming I'm a seasoned professional in a field where macros are common:
**Investigating the Potential for a Macro**
Hmm, based on this pattern or behavior I'm seeing, my initial instinct is that this could be driven by a macro. I should investigate that possibility first. It's a very common approach in these types of situations, and often the simplest explanation is the most likely. Right, let's initiate a search and see if a relevant macro exists within the relevant system, code, or documentation. Let's see what turns up.
|
| 339/6 |
2026/05/18 19:13 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:MAX_BPRINTF_BIN_ARGS]
Results: map[References:<nil> TruncatedOutput:false]
|
| 340/6 |
2026/05/18 19:13 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 12373
output: 45
thoughts: 111
Thoughts: **Investigating the Elusive `MAX_BPRINTF_BIN_ARGS`**
Okay, so I encountered this undefined term, `MAX_BPRINTF_BIN_ARGS`, and my first thought was, "Could it be a simple typo, or is it lurking in a header somewhere?" That's usually the first thing I check when dealing with these sorts of issues. Time to revisit `kernel/bpf/helpers.c`, as that's often a good starting point for BPF helper function definitions. I'll scan that file again, maybe I missed it initially. Wait a second, I think I just saw something! Ah, yes, there it is in the `bpf_bprintf_prepare` function, line 840 to be exact: `tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;`. Okay, so it *is* used. Now, I need to know what it *means*. Let's do a quick global search through the code tree for that string to find its declaration and usage context. Hopefully, that will give me the clue I need to understand what's going on.
|
| 341/6 |
2026/05/18 19:13 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:MAX_BPRINTF_BIN_ARGS SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 342/6 |
2026/05/18 19:13 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 12796
output: 23
thoughts: 23
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:
**Initial Observation and Immediate Action**
That's an unexpected observation, and I need to dive into this immediately. Given my expertise, I'll start with the obvious starting point and read the beginning of `kernel/bpf/helpers.c`. I need to get a high-level view and refresh my memory on the initial implementation and the overall structure of that file. This is the foundation I build on.
|
| 343/6 |
2026/05/18 19:13 |
tool |
read-file |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*/
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/bpf-cgroup.h>
#include <linux/cgroup.h>
#include <linux/rcupdate.h>
#include <linux/random.h>
#include <linux/smp.h>
#include <linux/topology.h>
#include <linux/ktime.h>
#include <linux/sched.h>
#include <linux/uidgid.h>
#include <linux/filter.h>
#include <linux/ctype.h>
#include <linux/jiffies.h>
#include <linux/pid_namespace.h>
#include <linux/poison.h>
#include <linux/proc_ns.h>
#include <linux/sched/task.h>
#include <linux/security.h>
#include <linux/btf_ids.h>
#include <linux/bpf_mem_alloc.h>
#include <linux/kasan.h>
#include <linux/bpf_verifier.h>
#include <linux/uaccess.h>
#include <linux/verification.h>
#include <linux/task_work.h>
#include <linux/irq_work.h>
#include <linux/buildid.h>
#include "../../lib/kstrtox.h"
/* If kernel subsystem is allowing eBPF programs to call this function,
* inside its own verifier_ops->get_func_proto() callback it should return
* bpf_map_lookup_elem_proto, so that verifier can properly check the arguments
*
* Different map implementations will rely on rcu in map methods
* lookup/update/delete, therefore eBPF programs must run under rcu lock
* if program is allowed to access maps, so check rcu_read_lock_held() or
* rcu_read_lock_trace_held() in all three functions.
*/
BPF_CALL_2(bpf_map_lookup_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_elem(map, key);
}
const struct bpf_func_proto bpf_map_lookup_elem_proto = {
.func = bpf_map_lookup_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_4(bpf_map_update_elem, struct bpf_map *, map, void *, key,
void *, value, u64, flags)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_update_elem(map, key, value, flags);
}
const struct bpf_func_proto bpf_map_update_elem_proto = {
.func = bpf_map_update_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_PTR_TO_MAP_VALUE,
.arg4_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_delete_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_delete_elem(map, key);
}
const struct bpf_func_proto bpf_map_delete_elem_proto = {
.func = bpf_map_delete_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_3(bpf_map_push_elem, struct bpf_map *, map, void *, value, u64, flags)
{
return map->ops->map_push_elem(map, value, flags);
}
const struct bpf_func_proto bpf_map_push_elem_proto = {
.func = bpf_map_push_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_pop_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_pop_elem(map, value);
}
const struct bpf_func_proto bpf_map_pop_elem_proto = {
.func = bpf_map_pop_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_2(bpf_map_peek_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_peek_elem(map, value);
}
const struct bpf_func_proto bpf_map_peek_elem_proto = {
.func = bpf_map_peek_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_3(bpf_map_lookup_percpu_elem, struct bpf_map *, map, void *, key, u32, cpu)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_percpu_elem(map, key, cpu);
}
const struct bpf_func_proto bpf_map_lookup_percpu_elem_proto = {
.func = bpf_map_lookup_percpu_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_ANYTHING,
};
const struct bpf_func_proto bpf_get_prandom_u32_proto = {
.func = bpf_user_rnd_u32,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_smp_processor_id)
{
return smp_processor_id();
}
const struct bpf_func_proto bpf_get_smp_processor_id_proto = {
.func = bpf_get_smp_processor_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.allow_fastcall = true,
};
BPF_CALL_0(bpf_get_numa_node_id)
{
return numa_node_id();
}
const struct bpf_func_proto bpf_get_numa_node_id_proto = {
.func = bpf_get_numa_node_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_ns)
{
/* NMI safe access to clock monotonic */
return ktime_get_mono_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_ns_proto = {
.func = bpf_ktime_get_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_boot_ns)
{
/* NMI safe access to clock boottime */
return ktime_get_boot_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_boot_ns_proto = {
.func = bpf_ktime_get_boot_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_coarse_ns)
{
return ktime_get_coarse_ns();
}
const struct bpf_func_proto bpf_ktime_get_coarse_ns_proto = {
.func = bpf_ktime_get_coarse_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_tai_ns)
{
/* NMI safe access to clock tai */
return ktime_get_tai_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_tai_ns_proto = {
.func = bpf_ktime_get_tai_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_pid_tgid)
{
struct task_struct *task = current;
if (unlikely(!task))
return -EINVAL;
return (u64) task->tgid << 32 | task->pid;
}
const struct bpf_func_proto bpf_get_current_pid_tgid_proto = {
.func = bpf_get_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_uid_gid)
{
struct task_struct *task = current;
kuid_t uid;
kgid_t gid;
if (unlikely(!task))
return -EINVAL;
current_uid_gid(&uid, &gid);
return (u64) from_kgid(&init_user_ns, gid) << 32 |
from_kuid(&init_user_ns, uid);
}
const struct bpf_func_proto bpf_get_current_uid_gid_proto = {
.func = bpf_get_current_uid_gid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_2(bpf_get_current_comm, char *, buf, u32, size)
{
struct task_struct *task = current;
if (unlikely(!task))
goto err_clear;
/* Verifier guarantees that size > 0 */
strscpy_pad(buf, task->comm, size);
return 0;
err_clear:
memset(buf, 0, size);
return -EINVAL;
}
const struct bpf_func_proto bpf_get_current_comm_proto = {
.func = bpf_get_current_comm,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE,
};
#if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK)
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
union {
__u32 val;
arch_spinlock_t lock;
} u = { .lock = __ARCH_SPIN_LOCK_UNLOCKED };
compiletime_assert(u.val == 0, "__ARCH_SPIN_LOCK_UNLOCKED not 0");
BUILD_BUG_ON(sizeof(*l) != sizeof(__u32));
BUILD_BUG_ON(sizeof(*lock) != sizeof(__u32));
preempt_disable();
arch_spin_lock(l);
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
arch_spin_unlock(l);
preempt_enable();
}
#else
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
BUILD_BUG_ON(sizeof(*l) != sizeof(*lock));
do {
atomic_cond_read_relaxed(l, !VAL);
} while (atomic_xchg(l, 1));
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
atomic_set_release(l, 0);
}
#endif
static DEFINE_PER_CPU(unsigned long, irqsave_flags);
static inline void __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
{
unsigned long flags;
local_irq_save(flags);
__bpf_spin_lock(lock);
__this_cpu_write(irqsave_flags, flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
{
__bpf_spin_lock_irqsave(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_lock_proto = {
.func = bpf_spin_lock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
{
unsigned long flags;
flags = __this_cpu_read(irqsave_flags);
__bpf_spin_unlock(lock);
local_irq_restore(flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
{
__bpf_spin_unlock_irqrestore(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_unlock_proto = {
.func = bpf_spin_unlock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
bool lock_src)
{
struct bpf_spin_lock *lock;
if (lock_src)
lock = src + map->record->spin_lock_off;
else
lock = dst + map->record->spin_lock_off;
preempt_disable();
__bpf_spin_lock_irqsave(lock);
copy_map_value(map, dst, src);
__bpf_spin_unlock_irqrestore(lock);
preempt_enable();
}
BPF_CALL_0(bpf_jiffies64)
{
return get_jiffies_64();
}
const struct bpf_func_proto bpf_jiffies64_proto = {
.func = bpf_jiffies64,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
#ifdef CONFIG_CGROUPS
BPF_CALL_0(bpf_get_current_cgroup_id)
{
struct cgroup *cgrp;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
cgrp_id = cgroup_id(cgrp);
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_cgroup_id_proto = {
.func = bpf_get_current_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_1(bpf_get_current_ancestor_cgroup_id, int, ancestor_level)
{
struct cgroup *cgrp;
struct cgroup *ancestor;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
ancestor = cgroup_ancestor(cgrp, ancestor_level);
cgrp_id = ancestor ? cgroup_id(ancestor) : 0;
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto = {
.func = bpf_get_current_ancestor_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
#endif /* CONFIG_CGROUPS */
#define BPF_STRTOX_BASE_MASK 0x1F
static int __bpf_strtoull(const char *buf, size_t buf_len, u64 flags,
unsigned long long *res, bool *is_negative)
{
unsigned int base = flags & BPF_STRTOX_BASE_MASK;
const char *cur_buf = buf;
size_t cur_len = buf_len;
unsigned int consumed;
size_t val_len;
char str[64];
if (!buf || !buf_len || !res || !is_negative)
return -EINVAL;
if (base != 0 && base != 8 && base != 10 && base != 16)
return -EINVAL;
if (flags & ~BPF_STRTOX_BASE_MASK)
return -EINVAL;
while (cur_buf < buf + buf_len && isspace(*cur_buf))
++cur_buf;
*is_negative = (cur_buf < buf + buf_len && *cur_buf == '-');
if (*is_negative)
++cur_buf;
consumed = cur_buf - buf;
cur_len -= consumed;
if (!cur_len)
return -EINVAL;
cur_len = min(cur_len, sizeof(str) - 1);
memcpy(str, cur_buf, cur_len);
str[cur_len] = '\0';
cur_buf = str;
cur_buf = _parse_integer_fixup_radix(cur_buf, &base);
val_len = _parse_integer(cur_buf, base, res);
if (val_len & KSTRTOX_OVERFLOW)
return -ERANGE;
if (val_len == 0)
return -EINVAL;
cur_buf += val_len;
consumed += cur_buf - str;
return consumed;
}
static int __bpf_strtoll(const char *buf, size_t buf_len, u64 flags,
long long *res)
{
unsigned long long _res;
bool is_negative;
int err;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative) {
if ((long long)-_res > 0)
return -ERANGE;
*res = -_res;
} else {
if ((long long)_res < 0)
return -ERANGE;
*res = _res;
}
return err;
}
BPF_CALL_4(bpf_strtol, const char *, buf, size_t, buf_len, u64, flags,
s64 *, res)
{
long long _res;
int err;
*res = 0;
err = __bpf_strtoll(buf, buf_len, flags, &_res);
if (err < 0)
return err;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtol_proto = {
.func = bpf_strtol,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(s64),
};
BPF_CALL_4(bpf_strtoul, const char *, buf, size_t, buf_len, u64, flags,
u64 *, res)
{
unsigned long long _res;
bool is_negative;
int err;
*res = 0;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative)
return -EINVAL;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtoul_proto = {
.func = bpf_strtoul,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(u64),
};
BPF_CALL_3(bpf_strncmp, const char *, s1, u32, s1_sz, const char *, s2)
{
return strncmp(s1, s2, s1_sz);
}
static const struct bpf_func_proto bpf_strncmp_proto = {
.func = bpf_strncmp,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_CONST_STR,
};
BPF_CALL_4(bpf_get_ns_current_pid_tgid, u64, dev, u64, ino,
struct bpf_pidns_info *, nsdata, u32, size)
{
struct task_struct *task = current;
struct pid_namespace *pidns;
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_pidns_info)))
goto clear;
if (unlikely((u64)(dev_t)dev != dev))
goto clear;
if (unlikely(!task))
goto clear;
pidns = task_active_pid_ns(task);
if (unlikely(!pidns)) {
err = -ENOENT;
goto clear;
}
if (!ns_match(&pidns->ns, (dev_t)dev, ino))
goto clear;
nsdata->pid = task_pid_nr_ns(task, pidns);
nsdata->tgid = task_tgid_nr_ns(task, pidns);
return 0;
clear:
memset((void *)nsdata, 0, (size_t) size);
return err;
}
const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = {
.func = bpf_get_ns_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
};
static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
.func = bpf_get_raw_cpu_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
}
const struct bpf_func_proto bpf_event_output_data_proto = {
.func = bpf_event_output_data,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
const void __user *, user_ptr)
{
int ret = copy_from_user(dst, user_ptr, size);
if (unlikely(ret)) {
memset(dst, 0, size);
ret = -EFAULT;
}
return ret;
}
const struct bpf_func_proto bpf_copy_from_user_proto = {
.func = bpf_copy_from_user,
.gpl_only = false,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_5(bpf_copy_from_user_task, void *, dst, u32, size,
const void __user *, user_ptr, struct task_struct *, tsk, u64, flags)
{
int ret;
/* flags is not used yet */
if (unlikely(flags))
return -EINVAL;
if (unlikely(!size))
return 0;
ret = access_process_vm(tsk, (unsigned long)user_ptr, dst, size, 0);
if (ret == size)
return 0;
memset(dst, 0, size);
/* Return -EFAULT for partial read */
return ret < 0 ? ret : -EFAULT;
}
const struct bpf_func_proto bpf_copy_from_user_task_proto = {
.func = bpf_copy_from_user_task,
.gpl_only = true,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_BTF_ID,
.arg4_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.arg5_type = ARG_ANYTHING
};
BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
{
if (cpu >= nr_cpu_ids)
return (unsigned long)NULL;
return (unsigned long)per_cpu_ptr((const void __percpu *)(const uintptr_t)ptr, cpu);
}
const struct bpf_func_proto bpf_per_cpu_ptr_proto = {
.func = bpf_per_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | PTR_MAYBE_NULL | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
.arg2_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_this_cpu_ptr, const void *, percpu_ptr)
{
return (unsigned long)this_cpu_ptr((const void __percpu *)(const uintptr_t)percpu_ptr);
}
const struct bpf_func_proto bpf_this_cpu_ptr_proto = {
.func = bpf_this_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
};
static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,
size_t bufsz)
{
void __user *user_ptr = (__force void __user *)unsafe_ptr;
buf[0] = 0;
switch (fmt_ptype) {
case 's':
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
if ((unsigned long)unsafe_ptr < TASK_SIZE)
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
fallthrough;
#endif
case 'k':
return strncpy_from_kernel_nofault(buf, unsafe_ptr, bufsz);
case 'u':
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
}
return -EINVAL;
}
/* Support executing three nested bprintf helper calls on a given CPU */
#define MAX_BPRINTF_NEST_LEVEL 3
static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs);
static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
{
int nest_level;
preempt_disable();
nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
return -EBUSY;
}
*bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
return 0;
}
void bpf_put_buffers(void)
{
if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
return;
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
}
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
{
if (!data->bin_args && !data->buf)
return;
bpf_put_buffers();
}
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)
{
if (map->map_type == BPF_MAP_TYPE_ARRAY) {
struct bpf_array *array = container_of(map, struct bpf_array, map);
*arr_idx = ((char *)value - array->value) / array->elem_size;
return arr_idx;
}
return (void *)value - round_up(map->key_size, 8);
}
enum bpf_async_type {
BPF_ASYNC_TYPE_TIMER = 0,
BPF_ASYNC_TYPE_WQ,
};
enum bpf_async_op {
BPF_ASYNC_START,
BPF_ASYNC_CANCEL
};
struct bpf_async_cmd {
struct llist_node node;
u64 nsec;
u32 mode;
enum bpf_async_op op;
};
struct bpf_async_cb {
struct bpf_map *map;
struct bpf_prog *prog;
void __rcu *callback_fn;
void *value;
struct rcu_head rcu;
u64 flags;
struct irq_work worker;
refcount_t refcnt;
enum bpf_async_type type;
struct llist_head async_cmds;
};
/* BPF map elements can contain 'struct bpf_timer'.
* Such map owns all of its BPF timers.
* 'struct bpf_timer' is allocated as part of map element allocation
* and it's zero initialized.
* That space is used to keep 'struct bpf_async_kern'.
* bpf_timer_init() allocates 'struct bpf_hrtimer', inits hrtimer, and
* remembers 'struct bpf_map *' pointer it's part of.
* bpf_timer_set_callback() increments prog refcnt and assign bpf callback_fn.
* bpf_timer_start() arms the timer.
* If user space reference to a map goes to zero at this point
* ops->map_release_uref callback is responsible for cancelling the timers,
* freeing their memory, and decrementing prog's refcnts.
* bpf_timer_cancel() cancels the timer and decrements prog's refcnt.
* Inner maps can contain bpf timers as well. ops->map_release_uref is
* freeing the timers when inner map is replaced or deleted by user space.
*/
struct bpf_hrtimer {
struct bpf_async_cb cb;
struct hrtimer timer;
atomic_t cancelling;
};
struct bpf_work {
struct bpf_async_cb cb;
struct work_struct work;
};
/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */
struct bpf_async_kern {
union {
struct bpf_async_cb *cb;
struct bpf_hrtimer *timer;
struct bpf_work *work;
};
} __attribute__((aligned(8)));
static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);
static void bpf_async_refcount_put(struct bpf_async_cb *cb);
static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer)
{
struct bpf_hrtimer *t = container_of(hrtimer, struct bpf_hrtimer, timer);
struct bpf_map *map = t->cb.map;
void *value = t->cb.value;
bpf_callback_t callback_fn;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_timer);
callback_fn = rcu_dereference_check(t->cb.callback_fn, rcu_read_lock_bh_held());
if (!callback_fn)
goto out;
/* bpf_timer_cb() runs in hrtimer_run_softirq. It doesn't migrate and
* cannot be preempted by another bpf_timer_cb() on the same cpu.
* Remember the timer this callback is servicing to prevent
* deadlock if callback_fn() calls bpf_timer_cancel() or
* bpf_map_delete_elem() on the same timer.
*/
this_cpu_write(hrtimer_running, t);
key = map_key_from_value(map, value, &idx);
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
/* The verifier checked that return value is zero. */
this_cpu_write(hrtimer_running, NULL);
out:
return HRTIMER_NORESTART;
}
static void bpf_wq_work(struct work_struct *work)
{
struct bpf_work *w = container_of(work, struct bpf_work, work);
struct bpf_async_cb *cb = &w->cb;
struct bpf_map *map = cb->map;
bpf_callback_t callback_fn;
void *value = cb->value;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_wq);
callback_fn = READ_ONCE(cb->callback_fn);
if (!callback_fn)
return;
key = map_key_from_value(map, value, &idx);
rcu_read_lock_trace();
migrate_disable();
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
migrate_enable();
rcu_read_unlock_trace();
}
static void bpf_async_cb_rcu_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
/*
* Drop the last reference to prog only after RCU GP, as set_callback()
* may race with cancel_and_free()
*/
if (cb->prog)
bpf_prog_put(cb->prog);
kfree_nolock(cb);
}
/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */
static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
bool retry = false;
/*
* bpf_async_cancel_and_free() tried to cancel timer/wq, but it
* could have raced with timer/wq_start. Now refcnt is zero and
* srcu/rcu GP completed. Cancel timer/wq again.
*/
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER:
if (hrtimer_try_to_cancel(&t->timer) < 0)
retry = true;
break;
case BPF_ASYNC_TYPE_WQ:
if (!cancel_work(&w->work) && work_busy(&w->work))
retry = true;
break;
}
if (retry) {
/*
* hrtimer or wq callback may still be running. It must be
* in rcu_tasks_trace or rcu CS, so wait for GP again.
* It won't retry forever, since refcnt zero prevents all
* operations on timer/wq.
*/
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
return;
}
/* RCU Tasks Trace grace period implies RCU grace period. */
bpf_async_cb_rcu_free(rcu);
}
static void worker_for_call_rcu(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
static void bpf_async_refcount_put(struct bpf_async_cb *cb)
{
if (!refcount_dec_and_test(&cb->refcnt))
return;
if (irqs_disabled()) {
cb->worker = IRQ_WORK_INIT(worker_for_call_rcu);
irq_work_queue(&cb->worker);
} else {
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async);
static void bpf_async_irq_worker(struct irq_work *work);
static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags,
enum bpf_async_type type)
{
struct bpf_async_cb *cb, *old_cb;
struct bpf_hrtimer *t;
struct bpf_work *w;
clockid_t clockid;
size_t size;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
size = sizeof(struct bpf_hrtimer);
break;
case BPF_ASYNC_TYPE_WQ:
size = sizeof(struct bpf_work);
break;
default:
return -EINVAL;
}
old_cb = READ_ONCE(async->cb);
if (old_cb)
return -EBUSY;
cb = bpf_map_kmalloc_nolock(map, size, 0, map->numa_node);
if (!cb)
return -ENOMEM;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
clockid = flags & (MAX_CLOCKS - 1);
t = (struct bpf_hrtimer *)cb;
atomic_set(&t->cancelling, 0);
hrtimer_setup(&t->timer, bpf_timer_cb, clockid, HRTIMER_MODE_REL_SOFT);
cb->value = (void *)async - map->record->timer_off;
break;
case BPF_ASYNC_TYPE_WQ:
w = (struct bpf_work *)cb;
INIT_WORK(&w->work, bpf_wq_work);
cb->value = (void *)async - map->record->wq_off;
break;
}
cb->map = map;
cb->prog = NULL;
cb->flags = flags;
cb->worker = IRQ_WORK_INIT(bpf_async_irq_worker);
init_llist_head(&cb->async_cmds);
refcount_set(&cb->refcnt, 1); /* map's reference */
cb->type = type;
rcu_assign_pointer(cb->callback_fn, NULL);
old_cb = cmpxchg(&async->cb, NULL, cb);
if (old_cb) {
/* Lost the race to initialize this bpf_async_kern, drop the allocated object */
kfree_nolock(cb);
return -EBUSY;
}
/* Guarantee the order between async->cb and map->usercnt. So
* when there are concurrent uref release and bpf timer init, either
* bpf_timer_cancel_and_free() called by uref release reads a no-NULL
* timer or atomic64_read() below returns a zero usercnt.
*/
smp_mb();
if (!atomic64_read(&map->usercnt)) {
/* maps with timers must be either held by user space
* or pinned in bpffs.
*/
bpf_async_cancel_and_free(async);
return -EPERM;
}
return 0;
}
BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
u64, flags)
{
clock_t clockid = flags & (MAX_CLOCKS - 1);
BUILD_BUG_ON(MAX_CLOCKS != 16);
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer));
if (flags >= MAX_CLOCKS ||
/* similar to timerfd except _ALARM variants are not supported */
(clockid != CLOCK_MONOTONIC &&
clockid != CLOCK_REALTIME &&
clockid != CLOCK_BOOTTIME))
return -EINVAL;
return __bpf_async_init(timer, map, flags, BPF_ASYNC_TYPE_TIMER);
}
static const struct bpf_func_proto bpf_timer_init_proto = {
.func = bpf_timer_init,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
static int bpf_async_update_prog_callback(struct bpf_async_cb *cb,
struct bpf_prog *prog,
void *callback_fn)
{
struct bpf_prog *prev;
/* Acquire a guard reference on prog to prevent it from being freed during the loop */
if (prog) {
prog = bpf_prog_inc_not_zero(prog);
if (IS_ERR(prog))
return PTR_ERR(prog);
}
do {
if (prog)
prog = bpf_prog_inc_not_zero(prog);
prev = xchg(&cb->prog, prog);
rcu_assign_pointer(cb->callback_fn, callback_fn);
/*
* Release previous prog, make sure that if other CPU is contending,
* to set bpf_prog, references are not leaked as each iteration acquires and
* releases one reference.
*/
if (prev)
bpf_prog_put(prev);
} while (READ_ONCE(cb->prog) != prog ||
(void __force *)READ_ONCE(cb->callback_fn) != callback_fn);
if (prog)
bpf_prog_put(prog);
return 0;
}
static DEFINE_PER_CPU(struct bpf_async_cb *, async_cb_running);
static int bpf_async_schedule_op(struct bpf_async_cb *cb, enum bpf_async_op op,
u64 nsec, u32 timer_mode)
{
/*
* Do not schedule another operation on this cpu if it's in irq_work
* callback that is processing async_cmds queue. Otherwise the following
* loop is possible:
* bpf_timer_start() -> bpf_async_schedule_op() -> irq_work_queue().
* irqrestore -> bpf_async_irq_worker() -> tracepoint -> bpf_timer_start().
*/
if (this_cpu_read(async_cb_running) == cb) {
bpf_async_refcount_put(cb);
return -EDEADLK;
}
struct bpf_async_cmd *cmd = kmalloc_nolock(sizeof(*cmd), 0, NUMA_NO_NODE);
if (!cmd) {
bpf_async_refcount_put(cb);
return -ENOMEM;
}
init_llist_node(&cmd->node);
cmd->nsec = nsec;
cmd->mode = timer_mode;
cmd->op = op;
if (llist_add(&cmd->node, &cb->async_cmds))
irq_work_queue(&cb->worker);
return 0;
}
static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback_fn,
struct bpf_prog *prog)
{
struct bpf_async_cb *cb;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
return bpf_async_update_prog_callback(cb, prog, callback_fn);
}
BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
struct bpf_prog_aux *, aux)
{
return __bpf_async_set_callback(timer, callback_fn, aux->prog);
}
static const struct bpf_func_proto bpf_timer_set_callback_proto = {
.func = bpf_timer_set_callback,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_PTR_TO_FUNC,
};
static bool defer_timer_wq_op(void)
{
return in_hardirq() || irqs_disabled();
}
BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags)
{
struct bpf_hrtimer *t;
u32 mode;
if (flags & ~(BPF_F_TIMER_ABS | BPF_F_TIMER_CPU_PIN))
return -EINVAL;
t = READ_ONCE(async->timer);
if (!t || !READ_ONCE(t->cb.prog))
return -EINVAL;
if (flags & BPF_F_TIMER_ABS)
mode = HRTIMER_MODE_ABS_SOFT;
else
mode = HRTIMER_MODE_REL_SOFT;
if (flags & BPF_F_TIMER_CPU_PIN)
mode |= HRTIMER_MODE_PINNED;
/*
* bpf_async_cancel_and_free() could have dropped refcnt to zero. In
* such case BPF progs are not allowed to arm the timer to prevent UAF.
*/
if (!refcount_inc_not_zero(&t->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
hrtimer_start(&t->timer, ns_to_ktime(nsecs), mode);
bpf_async_refcount_put(&t->cb);
return 0;
} else {
return bpf_async_schedule_op(&t->cb, BPF_ASYNC_START, nsecs, mode);
}
}
static const struct bpf_func_proto bpf_timer_start_proto = {
.func = bpf_timer_start,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async)
{
struct bpf_hrtimer *t, *cur_t;
bool inc = false;
int ret = 0;
if (defer_timer_wq_op())
return -EOPNOTSUPP;
t = READ_ONCE(async->timer);
if (!t)
return -EINVAL;
cur_t = this_cpu_read(hrtimer_running);
if (cur_t == t) {
/* If bpf callback_fn is trying to bpf_timer_cancel()
* its own timer the hrtimer_cancel() will deadlock
* since it waits for callback_fn to finish.
*/
return -EDEADLK;
}
/* Only account in-flight cancellations when invoked from a timer
* callback, since we want to avoid waiting only if other _callbacks_
* are waiting on us, to avoid introducing lockups. Non-callback paths
* are ok, since nobody would synchronously wait for their completion.
*/
if (!cur_t)
goto drop;
atomic_inc(&t->cancelling);
/* Need full barrier after relaxed atomic_inc */
smp_mb__after_atomic();
inc = true;
if (atomic_read(&cur_t->cancelling)) {
/* We're cancelling timer t, while some other timer callback is
* attempting to cancel us. In such a case, it might be possible
* that timer t belongs to the other callback, or some other
* callback waiting upon it (creating transitive dependencies
* upon us), and we will enter a deadlock if we continue
* cancelling and waiting for it synchronously, since it might
* do the same. Bail!
*/
atomic_dec(&t->cancelling);
return -EDEADLK;
}
drop:
bpf_async_update_prog_callback(&t->cb, NULL, NULL);
/* Cancel the timer and wait for associated callback to finish
* if it was running.
*/
ret = hrtimer_cancel(&t->timer);
if (inc)
atomic_dec(&t->cancelling);
return ret;
}
static const struct bpf_func_proto bpf_timer_cancel_proto = {
.func = bpf_timer_cancel,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
};
static void bpf_async_process_op(struct bpf_async_cb *cb, u32 op,
u64 timer_nsec, u32 timer_mode)
{
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER: {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
switch (op) {
case BPF_ASYNC_START:
hrtimer_start(&t->timer, ns_to_ktime(timer_nsec), timer_mode);
break;
case BPF_ASYNC_CANCEL:
hrtimer_try_to_cancel(&t->timer);
break;
}
break;
}
case BPF_ASYNC_TYPE_WQ: {
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
switch (op) {
case BPF_ASYNC_START:
schedule_work(&w->work);
break;
case BPF_ASYNC_CANCEL:
cancel_work(&w->work);
break;
}
break;
}
}
bpf_async_refcount_put(cb);
}
static void bpf_async_irq_worker(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
struct llist_node *pos, *n, *list;
list = llist_del_all(&cb->async_cmds);
if (!list)
return;
list = llist_reverse_order(list);
this_cpu_write(async_cb_running, cb);
llist_for_each_safe(pos, n, list) {
struct bpf_async_cmd *cmd;
cmd = container_of(pos, struct bpf_async_cmd, node);
bpf_async_process_op(cb, cmd->op, cmd->nsec, cmd->mode);
kfree_nolock(cmd);
}
this_cpu_write(async_cb_running, NULL);
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async)
{
struct bpf_async_cb *cb;
if (!READ_ONCE(async->cb))
return;
cb = xchg(&async->cb, NULL);
if (!cb)
return;
bpf_async_update_prog_callback(cb, NULL, NULL);
/*
* No refcount_inc_not_zero(&cb->refcnt) here. Dropping the last
* refcnt. Either synchronously or asynchronously in irq_work.
*/
if (!defer_timer_wq_op()) {
bpf_async_process_op(cb, BPF_ASYNC_CANCEL, 0, 0);
} else {
(void)bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
/*
* bpf_async_schedule_op() either enqueues allocated cmd into llist
* or fails with ENOMEM and drop the last refcnt.
* This is unlikely, but safe, since bpf_async_cb_rcu_tasks_trace_free()
* callback will do additional timer/wq_cancel due to races anyway.
*/
}
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_timer_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_wq_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
{
unsigned long *kptr = dst;
/* This helper may be inlined by verifier. */
return xchg(kptr, (unsigned long)ptr);
}
/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg()
* helper is determined dynamically by the verifier. Use BPF_PTR_POISON to
* denote type that verifier will determine.
*/
static const struct bpf_func_proto bpf_kptr_xchg_proto = {
.func = bpf_kptr_xchg,
.gpl_only = false,
.ret_type = RET_PTR_TO_BTF_ID_OR_NULL,
.ret_btf_id = BPF_PTR_POISON,
.arg1_type = ARG_KPTR_XCHG_DEST,
.arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL | OBJ_RELEASE,
.arg2_btf_id = BPF_PTR_POISON,
};
struct bpf_dynptr_file_impl {
struct freader freader;
/* 64 bit offset and size overriding 32 bit ones in bpf_dynptr_kern */
u64 offset;
u64 size;
};
/* Since the upper 8 bits of dynptr->size is reserved, the
* maximum supported size is 2^24 - 1.
*/
#define DYNPTR_MAX_SIZE ((1UL << 24) - 1)
#define DYNPTR_TYPE_SHIFT 28
#define DYNPTR_SIZE_MASK 0xFFFFFF
#define DYNPTR_RDONLY_BIT BIT(31)
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr)
{
return ptr->size & DYNPTR_RDONLY_BIT;
}
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
{
ptr->size |= DYNPTR_RDONLY_BIT;
}
static void bpf_dynptr_set_type(struct bpf_dynptr_kern *ptr, enum bpf_dynptr_type type)
{
ptr->size |= type << DYNPTR_TYPE_SHIFT;
}
static enum bpf_dynptr_type bpf_dynptr_get_type(const struct bpf_dynptr_kern *ptr)
{
return (ptr->size & ~(DYNPTR_RDONLY_BIT)) >> DYNPTR_TYPE_SHIFT;
}
u64 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
return df->size;
}
return ptr->size & DYNPTR_SIZE_MASK;
}
static void bpf_dynptr_advance_offset(struct bpf_dynptr_kern *ptr, u64 off)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->offset += off;
return;
}
ptr->offset += off;
}
static void bpf_dynptr_set_size(struct bpf_dynptr_kern *ptr, u64 new_size)
{
u32 metadata = ptr->size & ~DYNPTR_SIZE_MASK;
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->size = new_size;
return;
}
ptr->size = (u32)new_size | metadata;
}
int bpf_dynptr_check_size(u64 size)
{
return size > DYNPTR_MAX_SIZE ? -E2BIG : 0;
}
static int bpf_file_fetch_bytes(struct bpf_dynptr_file_impl *df, u64 offset, void *buf, u64 len)
{
const void *ptr;
if (!buf)
return -EINVAL;
df->freader.buf = buf;
df->freader.buf_sz = len;
ptr = freader_fetch(&df->freader, offset + df->offset, len);
if (!ptr)
return df->freader.err;
if (ptr != buf) /* Force copying into the buffer */
memcpy(buf, ptr, len);
return 0;
}
void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size)
{
ptr->data = data;
ptr->offset = offset;
ptr->size = size;
bpf_dynptr_set_type(ptr, type);
}
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)
{
memset(ptr, 0, sizeof(*ptr));
}
BPF_CALL_4(bpf_dynptr_from_mem, void *, data, u64, size, u64, flags, struct bpf_dynptr_kern *, ptr)
{
int err;
BTF_TYPE_EMIT(struct bpf_dynptr);
err = bpf_dynptr_check_size(size);
if (err)
goto error;
/* flags is currently unsupported */
if (flags) {
err = -EINVAL;
goto error;
}
bpf_dynptr_init(ptr, data, BPF_DYNPTR_TYPE_LOCAL, 0, size);
return 0;
error:
bpf_dynptr_set_null(ptr);
return err;
}
static const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
.func = bpf_dynptr_from_mem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL | MEM_UNINIT | MEM_WRITE,
};
static int __bpf_dynptr_read(void *dst, u64 len, const struct bpf_dynptr_kern *src,
u64 offset, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!src->data || flags)
return -EINVAL;
err = bpf_dynptr_check_off_len(src, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(src);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst, src->data + src->offset + offset, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_XDP:
return __bpf_xdp_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_SKB_META:
memmove(dst, bpf_skb_meta_pointer(src->data, src->offset + offset), len);
return 0;
case BPF_DYNPTR_TYPE_FILE:
return bpf_file_fetch_bytes(src->data, offset, dst, len);
default:
WARN_ONCE(true, "bpf_dynptr_read: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_read, void *, dst, u64, len, const struct bpf_dynptr_kern *, src,
u64, offset, u64, flags)
{
return __bpf_dynptr_read(dst, len, src, offset, flags);
}
static const struct bpf_func_proto bpf_dynptr_read_proto = {
.func = bpf_dynptr_read,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
int __bpf_dynptr_write(const struct bpf_dynptr_kern *dst, u64 offset, void *src,
u64 len, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!dst->data || __bpf_dynptr_is_rdonly(dst))
return -EINVAL;
err = bpf_dynptr_check_off_len(dst, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(dst);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
if (flags)
return -EINVAL;
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst->data + dst->offset + offset, src, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_store_bytes(dst->data, dst->offset + offset, src, len,
flags);
case BPF_DYNPTR_TYPE_XDP:
if (flags)
return -EINVAL;
return __bpf_xdp_store_bytes(dst->data, dst->offset + offset, src, len);
case BPF_DYNPTR_TYPE_SKB_META:
return __bpf_skb_meta_store_bytes(dst->data, dst->offset + offset, src,
len, flags);
default:
WARN_ONCE(true, "bpf_dynptr_write: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_write, const struct bpf_dynptr_kern *, dst, u64, offset, void *, src,
u64, len, u64, flags)
{
return __bpf_dynptr_write(dst, offset, src, len, flags);
}
static const struct bpf_func_proto bpf_dynptr_write_proto = {
.func = bpf_dynptr_write,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_dynptr_data, const struct bpf_dynptr_kern *, ptr, u64, offset, u64, len)
{
enum bpf_dynptr_type type;
int err;
if (!ptr->data)
return 0;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return 0;
if (__bpf_dynptr_is_rdonly(ptr))
return 0;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return (unsigned long)(ptr->data + ptr->offset + offset);
case BPF_DYNPTR_TYPE_SKB:
case BPF_DYNPTR_TYPE_XDP:
case BPF_DYNPTR_TYPE_SKB_META:
/* skb and xdp dynptrs should use bpf_dynptr_slice / bpf_dynptr_slice_rdwr */
return 0;
default:
WARN_ONCE(true, "bpf_dynptr_data: unknown dynptr type %d\n", type);
return 0;
}
}
static const struct bpf_func_proto bpf_dynptr_data_proto = {
.func = bpf_dynptr_data,
.gpl_only = false,
.ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
};
const struct bpf_func_proto bpf_get_current_task_proto __weak;
const struct bpf_func_proto bpf_get_current_task_btf_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_str_proto __weak;
const struct bpf_func_proto bpf_task_pt_regs_proto __weak;
const struct bpf_func_proto bpf_perf_event_read_proto __weak;
const struct bpf_func_proto bpf_send_signal_proto __weak;
const struct bpf_func_proto bpf_send_signal_thread_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_sleepable_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_proto __weak;
const struct bpf_func_proto bpf_get_branch_snapshot_proto __weak;
const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_map_lookup_elem:
return &bpf_map_lookup_elem_proto;
case BPF_FUNC_map_update_elem:
return &bpf_map_update_elem_proto;
case BPF_FUNC_map_delete_elem:
return &bpf_map_delete_elem_proto;
case BPF_FUNC_map_push_elem:
return &bpf_map_push_elem_proto;
case BPF_FUNC_map_pop_elem:
return &bpf_map_pop_elem_proto;
case BPF_FUNC_map_peek_elem:
return &bpf_map_peek_elem_proto;
case BPF_FUNC_map_lookup_percpu_elem:
return &bpf_map_lookup_percpu_elem_proto;
case BPF_FUNC_get_prandom_u32:
return &bpf_get_prandom_u32_proto;
case BPF_FUNC_get_smp_processor_id:
return &bpf_get_raw_smp_processor_id_proto;
case BPF_FUNC_get_numa_node_id:
return &bpf_get_numa_node_id_proto;
case BPF_FUNC_tail_call:
return &bpf_tail_call_proto;
case BPF_FUNC_ktime_get_ns:
return &bpf_ktime_get_ns_proto;
case BPF_FUNC_ktime_get_boot_ns:
return &bpf_ktime_get_boot_ns_proto;
case BPF_FUNC_ktime_get_tai_ns:
return &bpf_ktime_get_tai_ns_proto;
case BPF_FUNC_ringbuf_output:
return &bpf_ringbuf_output_proto;
case BPF_FUNC_ringbuf_reserve:
return &bpf_ringbuf_reserve_proto;
case BPF_FUNC_ringbuf_submit:
return &bpf_ringbuf_submit_proto;
case BPF_FUNC_ringbuf_discard:
return &bpf_ringbuf_discard_proto;
case BPF_FUNC_ringbuf_query:
return &bpf_ringbuf_query_proto;
case BPF_FUNC_strncmp:
return &bpf_strncmp_proto;
case BPF_FUNC_strtol:
return &bpf_strtol_proto;
case BPF_FUNC_strtoul:
return &bpf_strtoul_proto;
case BPF_FUNC_get_current_pid_tgid:
return &bpf_get_current_pid_tgid_proto;
case BPF_FUNC_get_ns_current_pid_tgid:
return &bpf_get_ns_current_pid_tgid_proto;
case BPF_FUNC_get_current_uid_gid:
return &bpf_get_current_uid_gid_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_BPF))
return NULL;
switch (func_id) {
case BPF_FUNC_spin_lock:
return &bpf_spin_lock_proto;
case BPF_FUNC_spin_unlock:
return &bpf_spin_unlock_proto;
case BPF_FUNC_jiffies64:
return &bpf_jiffies64_proto;
case BPF_FUNC_per_cpu_ptr:
return &bpf_per_cpu_ptr_proto;
case BPF_FUNC_this_cpu_ptr:
return &bpf_this_cpu_ptr_proto;
case BPF_FUNC_timer_init:
return &bpf_timer_init_proto;
case BPF_FUNC_timer_set_callback:
return &bpf_timer_set_callback_proto;
case BPF_FUNC_timer_start:
return &bpf_timer_start_proto;
case BPF_FUNC_timer_cancel:
return &bpf_timer_cancel_proto;
case BPF_FUNC_kptr_xchg:
return &bpf_kptr_xchg_proto;
case BPF_FUNC_for_each_map_elem:
return &bpf_for_each_map_elem_proto;
case BPF_FUNC_loop:
return &bpf_loop_proto;
case BPF_FUNC_user_ringbuf_drain:
return &bpf_user_ringbuf_drain_proto;
case BPF_FUNC_ringbuf_reserve_dynptr:
return &bpf_ringbuf_reserve_dynptr_proto;
case BPF_FUNC_ringbuf_submit_dynptr:
return &bpf_ringbuf_submit_dynptr_proto;
case BPF_FUNC_ringbuf_discard_dynptr:
return &bpf_ringbuf_discard_dynptr_proto;
case BPF_FUNC_dynptr_from_mem:
return &bpf_dynptr_from_mem_proto;
case BPF_FUNC_dynptr_read:
return &bpf_dynptr_read_proto;
case BPF_FUNC_dynptr_write:
return &bpf_dynptr_write_proto;
case BPF_FUNC_dynptr_data:
return &bpf_dynptr_data_proto;
#ifdef CONFIG_CGROUPS
case BPF_FUNC_cgrp_storage_get:
return &bpf_cgrp_storage_get_proto;
case BPF_FUNC_cgrp_storage_delete:
return &bpf_cgrp_storage_delete_proto;
case BPF_FUNC_get_current_cgroup_id:
return &bpf_get_current_cgroup_id_proto;
case BPF_FUNC_get_current_ancestor_cgroup_id:
return &bpf_get_current_ancestor_cgroup_id_proto;
case BPF_FUNC_current_task_under_cgroup:
return &bpf_current_task_under_cgroup_proto;
#endif
#ifdef CONFIG_CGROUP_NET_CLASSID
case BPF_FUNC_get_cgroup_classid:
return &bpf_get_cgroup_classid_curr_proto;
#endif
case BPF_FUNC_task_storage_get:
return &bpf_task_storage_get_proto;
case BPF_FUNC_task_storage_delete:
return &bpf_task_storage_delete_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_PERFMON))
return NULL;
switch (func_id) {
case BPF_FUNC_trace_printk:
return bpf_get_trace_printk_proto();
case BPF_FUNC_get_current_task:
return &bpf_get_current_task_proto;
case BPF_FUNC_get_current_task_btf:
return &bpf_get_current_task_btf_proto;
case BPF_FUNC_get_current_comm:
return &bpf_get_current_comm_proto;
case BPF_FUNC_probe_read_user:
return &bpf_probe_read_user_proto;
case BPF_FUNC_probe_read_kernel:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_proto;
case BPF_FUNC_probe_read_user_str:
return &bpf_probe_read_user_str_proto;
case BPF_FUNC_probe_read_kernel_str:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_str_proto;
case BPF_FUNC_copy_from_user:
return &bpf_copy_from_user_proto;
case BPF_FUNC_copy_from_user_task:
return &bpf_copy_from_user_task_proto;
case BPF_FUNC_snprintf_btf:
return &bpf_snprintf_btf_proto;
case BPF_FUNC_snprintf:
return &bpf_snprintf_proto;
case BPF_FUNC_task_pt_regs:
return &bpf_task_pt_regs_proto;
case BPF_FUNC_trace_vprintk:
return bpf_get_trace_vprintk_proto();
case BPF_FUNC_perf_event_read_value:
return bpf_get_perf_event_read_value_proto();
case BPF_FUNC_perf_event_read:
return &bpf_perf_event_read_proto;
case BPF_FUNC_send_signal:
return &bpf_send_signal_proto;
case BPF_FUNC_send_signal_thread:
return &bpf_send_signal_thread_proto;
case BPF_FUNC_get_task_stack:
return prog->sleepable ? &bpf_get_task_stack_sleepable_proto
: &bpf_get_task_stack_proto;
case BPF_FUNC_get_branch_snapshot:
return &bpf_get_branch_snapshot_proto;
case BPF_FUNC_find_vma:
return &bpf_find_vma_proto;
default:
return NULL;
}
}
EXPORT_SYMBOL_GPL(bpf_base_func_proto);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock)
{
struct list_head *head = list_head, *orig_head = list_head;
BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
/* Do the actual list draining outside the lock to not hold the lock for
* too long, and also prevent deadlocks if tracing programs end up
* executing on entry/exit of functions called inside the critical
* section, and end up doing map ops that call bpf_list_head_free for
* the same map value again.
*/
__bpf_spin_lock_irqsave(spin_lock);
if (!head->next || list_empty(head))
goto unlock;
head = head->next;
unlock:
INIT_LIST_HEAD(orig_head);
__bpf_spin_unlock_irqrestore(spin_lock);
while (head != orig_head) {
void *obj = head;
obj -= field->graph_root.node_offset;
head = head->next;
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
/* Like rbtree_postorder_for_each_entry_safe, but 'pos' and 'n' are
* 'rb_node *', so field name of rb_node within containing struct is not
* needed.
*
* Since bpf_rb_tree's node type has a corresponding struct btf_field with
* graph_root.node_offset, it's not necessary to know field name
* or type of node struct
*/
#define bpf_rbtree_postorder_for_each_entry_safe(pos, n, root) \
for (pos = rb_first_postorder(root); \
pos && ({ n = rb_next_postorder(pos); 1; }); \
pos = n)
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock)
{
struct rb_root_cached orig_root, *root = rb_root;
struct rb_node *pos, *n;
void *obj;
BUILD_BUG_ON(sizeof(struct rb_root_cached) > sizeof(struct bpf_rb_root));
BUILD_BUG_ON(__alignof__(struct rb_root_cached) > __alignof__(struct bpf_rb_root));
__bpf_spin_lock_irqsave(spin_lock);
orig_root = *root;
*root = RB_ROOT_CACHED;
__bpf_spin_unlock_irqrestore(spin_lock);
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
obj = pos;
obj -= field->graph_root.node_offset;
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
__bpf_kfunc_start_defs();
/**
* bpf_obj_new() - allocate an object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate an object of the type identified by @local_type_id__k and
* initialize its special fields. BPF programs can use
* bpf_core_type_id_local() to provide @local_type_id__k. The verifier
* rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
void *p;
p = bpf_mem_alloc(&bpf_global_ma, size);
if (!p)
return NULL;
if (meta)
bpf_obj_init(meta->record, p);
return p;
}
__bpf_kfunc void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_obj_new(local_type_id__k, meta__ign);
}
/**
* bpf_percpu_obj_new() - allocate a percpu object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate a percpu object of the type identified by @local_type_id__k. BPF
* programs can use bpf_core_type_id_local() to provide @local_type_id__k.
* The verifier rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated percpu object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_percpu_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
/* The verifier has ensured that meta must be NULL */
return bpf_mem_alloc(&bpf_global_percpu_ma, size);
}
__bpf_kfunc void *bpf_percpu_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_percpu_obj_new(local_type_id__k, meta__ign);
}
/* Must be called under migrate_disable(), as required by bpf_mem_free */
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu)
{
struct bpf_mem_alloc *ma;
if (rec && rec->refcount_off >= 0 &&
!refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) {
/* Object is refcounted and refcount_dec didn't result in 0
* refcount. Return without freeing the object
*/
return;
}
if (rec)
bpf_obj_free_fields(rec, p);
if (percpu)
ma = &bpf_global_percpu_ma;
else
ma = &bpf_global_ma;
bpf_mem_free_rcu(ma, p);
}
/**
* bpf_obj_drop() - drop a previously allocated object
* @p__alloc: object to free
* @meta: verifier-supplied struct metadata
*
* Destroy special fields in @p__alloc as needed and free the object. The
* verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
void *p = p__alloc;
__bpf_obj_drop_impl(p, meta ? meta->record : NULL, false);
}
__bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
{
return bpf_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_percpu_obj_drop() - drop a previously allocated percpu object
* @p__alloc: percpu object to free
* @meta: verifier-supplied struct metadata
*
* Free @p__alloc. The verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_percpu_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
/* The verifier has ensured that meta must be NULL */
bpf_mem_free_rcu(&bpf_global_percpu_ma, p__alloc);
}
__bpf_kfunc void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign)
{
bpf_percpu_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_refcount_acquire() - turn a local kptr into an owning reference
* @p__refcounted_kptr: non-owning local kptr
* @meta: verifier-supplied struct metadata
*
* Increment the refcount for @p__refcounted_kptr. The verifier rewrites
* @meta; BPF programs do not set it.
*
* Return: Owning reference to @p__refcounted_kptr, or %NULL on failure.
*/
__bpf_kfunc void *bpf_refcount_acquire(void *p__refcounted_kptr, struct btf_struct_meta *meta)
{
struct bpf_refcount *ref;
/* Could just cast directly to refcount_t *, but need some code using
* bpf_refcount type so that it is emitted in vmlinux BTF
*/
ref = (struct bpf_refcount *)(p__refcounted_kptr + meta->record->refcount_off);
if (!refcount_inc_not_zero((refcount_t *)ref))
return NULL;
/* Verifier strips KF_RET_NULL if input is owned ref, see is_kfunc_ret_null
* in verifier.c
*/
return (void *)p__refcounted_kptr;
}
__bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta__ign)
{
return bpf_refcount_acquire(p__refcounted_kptr, meta__ign);
}
static int __bpf_list_add(struct bpf_list_node_kern *node,
struct bpf_list_head *head,
bool tail, struct btf_record *rec, u64 off)
{
struct list_head *n = &node->list_head, *h = (void *)head;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
/* node->owner != NULL implies !list_empty(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
tail ? list_add_tail(n, h) : list_add(n, h);
WRITE_ONCE(node->owner, head);
return 0;
}
/**
* bpf_list_push_front() - add a node to the front of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the front of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_front(head, node, meta__ign, off);
}
/**
* bpf_list_push_back() - add a node to the back of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the back of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_back(head, node, meta__ign, off);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
{
struct list_head *n, *h = (void *)head;
struct bpf_list_node_kern *node;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
if (list_empty(h))
return NULL;
n = tail ? h->prev : h->next;
node = container_of(n, struct bpf_list_node_kern, list_head);
if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))
return NULL;
list_del_init(n);
WRITE_ONCE(node->owner, NULL);
return (struct bpf_list_node *)n;
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
{
return __bpf_list_del(head, false);
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
{
return __bpf_list_del(head, true);
}
__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->next;
}
__bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->prev;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
struct rb_root_cached *r = (struct rb_root_cached *)root;
struct rb_node *n = &node_internal->rb_node;
/* node_internal->owner != root implies either RB_EMPTY_NODE(n) or
* n is owned by some other tree. No need to check RB_EMPTY_NODE(n)
*/
if (READ_ONCE(node_internal->owner) != root)
return NULL;
rb_erase_cached(n, r);
RB_CLEAR_NODE(n);
WRITE_ONCE(node_internal->owner, NULL);
return (struct bpf_rb_node *)n;
}
/* Need to copy rbtree_add_cached's logic here because our 'less' is a BPF
* program
*/
static int __bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node_kern *node,
void *less, struct btf_record *rec, u64 off)
{
struct rb_node **link = &((struct rb_root_cached *)root)->rb_root.rb_node;
struct rb_node *parent = NULL, *n = &node->rb_node;
bpf_callback_t cb = (bpf_callback_t)less;
bool leftmost = true;
/* node->owner != NULL implies !RB_EMPTY_NODE(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
while (*link) {
parent = *link;
if (cb((uintptr_t)node, (uintptr_t)parent, 0, 0, 0)) {
link = &parent->rb_left;
} else {
link = &parent->rb_right;
leftmost = false;
}
}
rb_link_node(n, parent, link);
rb_insert_color_cached(n, (struct rb_root_cached *)root, leftmost);
WRITE_ONCE(node->owner, root);
return 0;
}
/**
* bpf_rbtree_add() - add a node to a BPF rbtree
* @root: tree root
* @node: node to insert
* @less: comparator used to order nodes
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node into @root using @less. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked in a tree.
*/
__bpf_kfunc int bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_rb_node_kern *n = (void *)node;
return __bpf_rbtree_add(root, n, (void *)less, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
void *meta__ign, u64 off)
{
return bpf_rbtree_add(root, node, less, meta__ign, off);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)rb_first_cached(r);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_root(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)r->rb_root.rb_node;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_left(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_left;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_right(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_right;
}
/**
* bpf_task_acquire - Acquire a reference to a task. A task acquired by this
* kfunc which is not stored in a map as a kptr, must be released by calling
* bpf_task_release().
* @p: The task on which a reference is being acquired.
*/
__bpf_kfunc struct task_struct *bpf_task_acquire(struct task_struct *p)
{
if (refcount_inc_not_zero(&p->rcu_users))
return p;
return NULL;
}
/**
* bpf_task_release - Release the reference acquired on a task.
* @p: The task on which a reference is being released.
*/
__bpf_kfunc void bpf_task_release(struct task_struct *p)
{
put_task_struct_rcu_user(p);
}
__bpf_kfunc void bpf_task_release_dtor(void *p)
{
put_task_struct_rcu_user(p);
}
CFI_NOSEAL(bpf_task_release_dtor);
#ifdef CONFIG_CGROUPS
/**
* bpf_cgroup_acquire - Acquire a reference to a cgroup. A cgroup acquired by
* this kfunc which is not stored in a map as a kptr, must be released by
* calling bpf_cgroup_release().
* @cgrp: The cgroup on which a reference is being acquired.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)
{
return cgroup_tryget(cgrp) ? cgrp : NULL;
}
/**
* bpf_cgroup_release - Release the reference acquired on a cgroup.
* If this kfunc is invoked in an RCU read region, the cgroup is guaranteed to
* not be freed until the current grace period has ended, even if its refcount
* drops to 0.
* @cgrp: The cgroup on which a reference is being released.
*/
__bpf_kfunc void bpf_cgroup_release(struct cgroup *cgrp)
{
cgroup_put(cgrp);
}
__bpf_kfunc void bpf_cgroup_release_dtor(void *cgrp)
{
cgroup_put(cgrp);
}
CFI_NOSEAL(bpf_cgroup_release_dtor);
/**
* bpf_cgroup_ancestor - Perform a lookup on an entry in a cgroup's ancestor
* array. A cgroup returned by this kfunc which is not subsequently stored in a
* map, must be released by calling bpf_cgroup_release().
* @cgrp: The cgroup for which we're performing a lookup.
* @level: The level of ancestor to look up.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)
{
struct cgroup *ancestor;
if (level > cgrp->level || level < 0)
return NULL;
/* cgrp's refcnt could be 0 here, but ancestors can still be accessed */
ancestor = cgrp->ancestors[level];
if (!cgroup_tryget(ancestor))
return NULL;
return ancestor;
}
/**
* bpf_cgroup_from_id - Find a cgroup from its ID. A cgroup returned by this
* kfunc which is not subsequently stored in a map, must be released by calling
* bpf_cgroup_release().
* @cgid: cgroup id.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_from_id(u64 cgid)
{
struct cgroup *cgrp;
cgrp = __cgroup_get_from_id(cgid);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
/**
* bpf_task_under_cgroup - wrap task_under_cgroup_hierarchy() as a kfunc, test
* task's membership of cgroup ancestry.
* @task: the task to be tested
* @ancestor: possible ancestor of @task's cgroup
*
* Tests whether @task's default cgroup hierarchy is a descendant of @ancestor.
* It follows all the same rules as cgroup_is_descendant, and only applies
* to the default hierarchy.
*/
__bpf_kfunc long bpf_task_under_cgroup(struct task_struct *task,
struct cgroup *ancestor)
{
long ret;
rcu_read_lock();
ret = task_under_cgroup_hierarchy(task, ancestor);
rcu_read_unlock();
return ret;
}
BPF_CALL_2(bpf_current_task_under_cgroup, struct bpf_map *, map, u32, idx)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
struct cgroup *cgrp;
if (unlikely(idx >= array->map.max_entries))
return -E2BIG;
cgrp = READ_ONCE(array->ptrs[idx]);
if (unlikely(!cgrp))
return -EAGAIN;
return task_under_cgroup_hierarchy(current, cgrp);
}
const struct bpf_func_proto bpf_current_task_under_cgroup_proto = {
.func = bpf_current_task_under_cgroup,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
};
/**
* bpf_task_get_cgroup1 - Acquires the associated cgroup of a task within a
* specific cgroup1 hierarchy. The cgroup1 hierarchy is identified by its
* hierarchy ID.
* @task: The target task
* @hierarchy_id: The ID of a cgroup1 hierarchy
*
* On success, the cgroup is returen. On failure, NULL is returned.
*/
__bpf_kfunc struct cgroup *
bpf_task_get_cgroup1(struct task_struct *task, int hierarchy_id)
{
struct cgroup *cgrp = task_get_cgroup1(task, hierarchy_id);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
#endif /* CONFIG_CGROUPS */
/**
* bpf_task_from_pid - Find a struct task_struct from its pid by looking it up
* in the root pid namespace idr. If a task is returned, it must either be
* stored in a map, or released with bpf_task_release().
* @pid: The pid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_pid_ns(pid, &init_pid_ns);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_task_from_vpid - Find a struct task_struct from its vpid by looking it up
* in the pid namespace of the current task. If a task is returned, it must
* either be stored in a map, or released with bpf_task_release().
* @vpid: The vpid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_vpid(vpid);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_dynptr_slice() - Obtain a read-only pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* If the intention is to write to the data slice, please use
* bpf_dynptr_slice_rdwr.
*
* The user must check that the returned pointer is not null before using it.
*
* Please note that in the case of skb and xdp dynptrs, bpf_dynptr_slice
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a read-only
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
enum bpf_dynptr_type type;
u64 len = buffer__szk;
int err;
if (!ptr->data)
return NULL;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return NULL;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return ptr->data + ptr->offset + offset;
case BPF_DYNPTR_TYPE_SKB:
if (buffer__nullable)
return skb_header_pointer(ptr->data, ptr->offset + offset, len, buffer__nullable);
else
return skb_pointer_if_linear(ptr->data, ptr->offset + offset, len);
case BPF_DYNPTR_TYPE_XDP:
{
void *xdp_ptr = bpf_xdp_pointer(ptr->data, ptr->offset + offset, len);
if (!IS_ERR_OR_NULL(xdp_ptr))
return xdp_ptr;
if (!buffer__nullable)
return NULL;
bpf_xdp_copy_buf(ptr->data, ptr->offset + offset, buffer__nullable, len, false);
return buffer__nullable;
}
case BPF_DYNPTR_TYPE_SKB_META:
return bpf_skb_meta_pointer(ptr->data, ptr->offset + offset);
case BPF_DYNPTR_TYPE_FILE:
err = bpf_file_fetch_bytes(ptr->data, offset, buffer__nullable, buffer__szk);
return err ? NULL : buffer__nullable;
default:
WARN_ONCE(true, "unknown dynptr type %d\n", type);
return NULL;
}
}
/**
* bpf_dynptr_slice_rdwr() - Obtain a writable pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* The returned pointer is writable and may point to either directly the dynptr
* data at the requested offset or to the buffer if unable to obtain a direct
* data pointer to (example: the requested slice is to the paged area of an skb
* packet). In the case where the returned pointer is to the buffer, the user
* is responsible for persisting writes through calling bpf_dynptr_write(). This
* usually looks something like this pattern:
*
* struct eth_hdr *eth = bpf_dynptr_slice_rdwr(&dynptr, 0, buffer, sizeof(buffer));
* if (!eth)
* return TC_ACT_SHOT;
*
* // mutate eth header //
*
* if (eth == buffer)
* bpf_dynptr_write(&ptr, 0, buffer, sizeof(buffer), 0);
*
* Please note that, as in the example above, the user must check that the
* returned pointer is not null before using it.
*
* Please also note that in the case of skb and xdp dynptrs, bpf_dynptr_slice_rdwr
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice_rdwr will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data || __bpf_dynptr_is_rdonly(ptr))
return NULL;
/* bpf_dynptr_slice_rdwr is the same logic as bpf_dynptr_slice.
*
* For skb-type dynptrs, it is safe to write into the returned pointer
* if the bpf program allows skb data writes. There are two possibilities
* that may occur when calling bpf_dynptr_slice_rdwr:
*
* 1) The requested slice is in the head of the skb. In this case, the
* returned pointer is directly to skb data, and if the skb is cloned, the
* verifier will have uncloned it (see bpf_unclone_prologue()) already.
* The pointer can be directly written into.
*
* 2) Some portion of the requested slice is in the paged buffer area.
* In this case, the requested data will be copied out into the buffer
* and the returned pointer will be a pointer to the buffer. The skb
* will not be pulled. To persist the write, the user will need to call
* bpf_dynptr_write(), which will pull the skb and commit the write.
*
* Similarly for xdp programs, if the requested slice is not across xdp
* fragments, then a direct pointer will be returned, otherwise the data
* will be copied out into the buffer and the user will need to call
* bpf_dynptr_write() to commit changes.
*/
return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
}
__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 size;
if (!ptr->data || start > end)
return -EINVAL;
size = __bpf_dynptr_size(ptr);
if (start > size || end > size)
return -ERANGE;
bpf_dynptr_advance_offset(ptr, start);
bpf_dynptr_set_size(ptr, end - start);
return 0;
}
__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
return !ptr->data;
}
__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return false;
return __bpf_dynptr_is_rdonly(ptr);
}
__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return -EINVAL;
return __bpf_dynptr_size(ptr);
}
__bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
struct bpf_dynptr *clone__uninit)
{
struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data) {
bpf_dynptr_set_null(clone);
return -EINVAL;
}
*clone = *ptr;
return 0;
}
/**
* bpf_dynptr_copy() - Copy data from one dynptr to another.
* @dst_ptr: Destination dynptr - where data should be copied to
* @dst_off: Offset into the destination dynptr
* @src_ptr: Source dynptr - where data should be copied from
* @src_off: Offset into the source dynptr
* @size: Length of the data to copy from source to destination
*
* Copies data from source dynptr to destination dynptr.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
{
struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
void *src_slice, *dst_slice;
char buf[256];
u64 off;
src_slice = bpf_dynptr_slice(src_ptr, src_off, NULL, size);
dst_slice = bpf_dynptr_slice_rdwr(dst_ptr, dst_off, NULL, size);
if (src_slice && dst_slice) {
memmove(dst_slice, src_slice, size);
return 0;
}
if (src_slice)
return __bpf_dynptr_write(dst, dst_off, src_slice, size, 0);
if (dst_slice)
return __bpf_dynptr_read(dst_slice, size, src, src_off, 0);
if (bpf_dynptr_check_off_len(dst, dst_off, size) ||
bpf_dynptr_check_off_len(src, src_off, size))
return -E2BIG;
off = 0;
while (off < size) {
u64 chunk_sz = min_t(u64, sizeof(buf), size - off);
int err;
err = __bpf_dynptr_read(buf, chunk_sz, src, src_off + off, 0);
if (err)
return err;
err = __bpf_dynptr_write(dst, dst_off + off, buf, chunk_sz, 0);
if (err)
return err;
off += chunk_sz;
}
return 0;
}
/**
* bpf_dynptr_memset() - Fill dynptr memory with a constant byte.
* @p: Destination dynptr - where data will be filled
* @offset: Offset into the dynptr to start filling from
* @size: Number of bytes to fill
* @val: Constant byte to fill the memory with
*
* Fills the @size bytes of the memory area pointed to by @p
* at @offset with the constant byte @val.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 chunk_sz, write_off;
char buf[256];
void* slice;
int err;
slice = bpf_dynptr_slice_rdwr(p, offset, NULL, size);
if (likely(slice)) {
memset(slice, val, size);
return 0;
}
if (__bpf_dynptr_is_rdonly(ptr))
return -EINVAL;
err = bpf_dynptr_check_off_len(ptr, offset, size);
if (err)
return err;
/* Non-linear data under the dynptr, write from a local buffer */
chunk_sz = min_t(u64, sizeof(buf), size);
memset(buf, val, chunk_sz);
for (write_off = 0; write_off < size; write_off += chunk_sz) {
chunk_sz = min_t(u64, sizeof(buf), size - write_off);
err = __bpf_dynptr_write(ptr, offset + write_off, buf, chunk_sz, 0);
if (err)
return err;
}
return 0;
}
__bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
{
return obj;
}
__bpf_kfunc void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k)
{
return (void *)obj__ign;
}
__bpf_kfunc void bpf_rcu_read_lock(void)
{
rcu_read_lock();
}
__bpf_kfunc void bpf_rcu_read_unlock(void)
{
rcu_read_unlock();
}
struct bpf_throw_ctx {
struct bpf_prog_aux *aux;
u64 sp;
u64 bp;
int cnt;
};
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
{
struct bpf_throw_ctx *ctx = cookie;
struct bpf_prog *prog;
/*
* The RCU read lock is held to safely traverse the latch tree, but we
* don't need its protection when accessing the prog, since it has an
* active stack frame on the current stack trace, and won't disappear.
*/
rcu_read_lock();
prog = bpf_prog_ksym_find(ip);
rcu_read_unlock();
if (!prog)
return !ctx->cnt;
ctx->cnt++;
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
ctx->sp = sp;
ctx->bp = bp;
return false;
}
__bpf_kfunc void bpf_throw(u64 cookie)
{
struct bpf_throw_ctx ctx = {};
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
WARN_ON_ONCE(!ctx.aux->exception_boundary);
WARN_ON_ONCE(!ctx.bp);
WARN_ON_ONCE(!ctx.cnt);
/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
* deeper stack depths than ctx.sp as we do not return from bpf_throw,
* which skips compiler generated instrumentation to do the same.
*/
kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_map *map = p__map;
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq));
if (flags)
return -EINVAL;
return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ);
}
__bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_work *w;
if (flags)
return -EINVAL;
w = READ_ONCE(async->work);
if (!w || !READ_ONCE(w->cb.prog))
return -EINVAL;
if (!refcount_inc_not_zero(&w->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
schedule_work(&w->work);
bpf_async_refcount_put(&w->cb);
return 0;
} else {
return bpf_async_schedule_op(&w->cb, BPF_ASYNC_START, 0, 0);
}
}
__bpf_kfunc int bpf_wq_set_callback(struct bpf_wq *wq,
int (callback_fn)(void *map, int *key, void *value),
unsigned int flags,
struct bpf_prog_aux *aux)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
if (flags)
return -EINVAL;
return __bpf_async_set_callback(async, callback_fn, aux->prog);
}
__bpf_kfunc void bpf_preempt_disable(void)
{
preempt_disable();
}
__bpf_kfunc void bpf_preempt_enable(void)
{
preempt_enable();
}
struct bpf_iter_bits {
__u64 __opaque[2];
} __aligned(8);
#define BITS_ITER_NR_WORDS_MAX 511
struct bpf_iter_bits_kern {
union {
__u64 *bits;
__u64 bits_copy;
};
int nr_bits;
int bit;
} __aligned(8);
/* On 64-bit hosts, unsigned long and u64 have the same size, so passing
* a u64 pointer and an unsigned long pointer to find_next_bit() will
* return the same result, as both point to the same 8-byte area.
*
* For 32-bit little-endian hosts, using a u64 pointer or unsigned long
* pointer also makes no difference. This is because the first iterated
* unsigned long is composed of bits 0-31 of the u64 and the second unsigned
* long is composed of bits 32-63 of the u64.
*
* However, for 32-bit big-endian hosts, this is not the case. The first
* iterated unsigned long will be bits 32-63 of the u64, so swap these two
* ulong values within the u64.
*/
static void swap_ulong_in_u64(u64 *bits, unsigned int nr)
{
#if (BITS_PER_LONG == 32) && defined(__BIG_ENDIAN)
unsigned int i;
for (i = 0; i < nr; i++)
bits[i] = (bits[i] >> 32) | ((u64)(u32)bits[i] << 32);
#endif
}
/**
* bpf_iter_bits_new() - Initialize a new bits iterator for a given memory area
* @it: The new bpf_iter_bits to be created
* @unsafe_ptr__ign: A pointer pointing to a memory area to be iterated over
* @nr_words: The size of the specified memory area, measured in 8-byte units.
* The maximum value of @nr_words is @BITS_ITER_NR_WORDS_MAX. This limit may be
* further reduced by the BPF memory allocator implementation.
*
* This function initializes a new bpf_iter_bits structure for iterating over
* a memory area which is specified by the @unsafe_ptr__ign and @nr_words. It
* copies the data of the memory area to the newly created bpf_iter_bits @it for
* subsequent iteration operations.
*
* On success, 0 is returned. On failure, ERR is returned.
*/
__bpf_kfunc int
bpf_iter_bits_new(struct bpf_iter_bits *it, const u64 *unsafe_ptr__ign, u32 nr_words)
{
struct bpf_iter_bits_kern *kit = (void *)it;
u32 nr_bytes = nr_words * sizeof(u64);
u32 nr_bits = BYTES_TO_BITS(nr_bytes);
int err;
BUILD_BUG_ON(sizeof(struct bpf_iter_bits_kern) != sizeof(struct bpf_iter_bits));
BUILD_BUG_ON(__alignof__(struct bpf_iter_bits_kern) !=
__alignof__(struct bpf_iter_bits));
kit->nr_bits = 0;
kit->bits_copy = 0;
kit->bit = -1;
if (!unsafe_ptr__ign || !nr_words)
return -EINVAL;
if (nr_words > BITS_ITER_NR_WORDS_MAX)
return -E2BIG;
/* Optimization for u64 mask */
if (nr_bits == 64) {
err = bpf_probe_read_kernel_common(&kit->bits_copy, nr_bytes, unsafe_ptr__ign);
if (err)
return -EFAULT;
swap_ulong_in_u64(&kit->bits_copy, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
if (bpf_mem_alloc_check_size(false, nr_bytes))
return -E2BIG;
/* Fallback to memalloc */
kit->bits = bpf_mem_alloc(&bpf_global_ma, nr_bytes);
if (!kit->bits)
return -ENOMEM;
err = bpf_probe_read_kernel_common(kit->bits, nr_bytes, unsafe_ptr__ign);
if (err) {
bpf_mem_free(&bpf_global_ma, kit->bits);
return err;
}
swap_ulong_in_u64(kit->bits, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
/**
* bpf_iter_bits_next() - Get the next bit in a bpf_iter_bits
* @it: The bpf_iter_bits to be checked
*
* This function returns a pointer to a number representing the value of the
* next bit in the bits.
*
* If there are no further bits available, it returns NULL.
*/
__bpf_kfunc int *bpf_iter_bits_next(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
int bit = kit->bit, nr_bits = kit->nr_bits;
const void *bits;
if (!nr_bits || bit >= nr_bits)
return NULL;
bits = nr_bits == 64 ? &kit->bits_copy : kit->bits;
bit = find_next_bit(bits, nr_bits, bit + 1);
if (bit >= nr_bits) {
kit->bit = bit;
return NULL;
}
kit->bit = bit;
return &kit->bit;
}
/**
* bpf_iter_bits_destroy() - Destroy a bpf_iter_bits
* @it: The bpf_iter_bits to be destroyed
*
* Destroy the resource associated with the bpf_iter_bits.
*/
__bpf_kfunc void bpf_iter_bits_destroy(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
if (kit->nr_bits <= 64)
return;
bpf_mem_free(&bpf_global_ma, kit->bits);
}
/**
* bpf_copy_from_user_str() - Copy a string from an unsafe user address
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address, in user space.
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL-terminated string from userspace to BPF space. If user string is
* too long this will still ensure zero termination in the dst buffer unless
* buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst to 0 on success and
* memset all of @dst on failure.
*/
__bpf_kfunc int bpf_copy_from_user_str(void *dst, u32 dst__sz, const void __user *unsafe_ptr__ign, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(!dst__sz))
return 0;
ret = strncpy_from_user(dst, unsafe_ptr__ign, dst__sz - 1);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst + ret, 0, dst__sz - ret);
else
((char *)dst)[ret] = '\0';
return ret + 1;
}
/**
* bpf_copy_from_user_task_str() - Copy a string from an task's address space
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address in the task's address space.
* @tsk: The task whose address space will be used
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL terminated string from a task's address space to @dst__sz
* buffer. If user string is too long this will still ensure zero termination
* in the @dst__sz buffer unless buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst__sz to 0 on success
* and memset all of @dst__sz on failure.
*
* Return: The number of copied bytes on success including the NUL terminator.
* A negative error code on failure.
*/
__bpf_kfunc int bpf_copy_from_user_task_str(void *dst, u32 dst__sz,
const void __user *unsafe_ptr__ign,
struct task_struct *tsk, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(dst__sz == 0))
return 0;
ret = copy_remote_vm_str(tsk, (unsigned long)unsafe_ptr__ign, dst, dst__sz, 0);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset(dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset(dst + ret, 0, dst__sz - ret);
return ret + 1;
}
/* Keep unsinged long in prototype so that kfunc is usable when emitted to
* vmlinux.h in BPF programs directly, but note that while in BPF prog, the
* unsigned long always points to 8-byte region on stack, the kernel may only
* read and write the 4-bytes on 32-bit.
*/
__bpf_kfunc void bpf_local_irq_save(unsigned long *flags__irq_flag)
{
local_irq_save(*flags__irq_flag);
}
__bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)
{
local_irq_restore(*flags__irq_flag);
}
__bpf_kfunc void __bpf_trap(void)
{
}
/*
* Kfuncs for string operations.
*
* Since strings are not necessarily %NUL-terminated, we cannot directly call
* in-kernel implementations. Instead, we open-code the implementations using
* __get_kernel_nofault instead of plain dereference to make them safe.
*/
static int __bpf_strncasecmp(const char *s1, const char *s2, bool ignore_case, size_t len)
{
char c1, c2;
int i;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < len && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c1, s1, char, err_out);
__get_kernel_nofault(&c2, s2, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 != c2)
return c1 < c2 ? -1 : 1;
if (c1 == '\0')
return 0;
s1++;
s2++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : 0;
err_out:
return -EFAULT;
}
/**
* bpf_strcmp - Compare two strings
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, false, XATTR_SIZE_MAX);
}
/**
* bpf_strcasecmp - Compare two strings, ignoring the case of the characters
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasecmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, XATTR_SIZE_MAX);
}
/*
* bpf_strncasecmp - Compare two length-limited strings, ignoring case
* @s1__ign: One string
* @s2__ign: Another string
* @len: The maximum number of characters to compare
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasecmp(const char *s1__ign, const char *s2__ign, size_t len)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, len);
}
/**
* bpf_strnchr - Find a character in a length limited string
* @s__ign: The string to be searched
* @count: The number of characters to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in the first @count characters of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnchr(const char *s__ign, size_t count, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
return i;
if (sc == '\0')
return -ENOENT;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : -ENOENT;
err_out:
return -EFAULT;
}
/**
* bpf_strchr - Find the first occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - The index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchr(const char *s__ign, char c)
{
return bpf_strnchr(s__ign, XATTR_SIZE_MAX, c);
}
/**
* bpf_strchrnul - Find and return a character in a string, or end of string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign or index of
* the null byte at the end of @s__ign when @c is not found
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchrnul(const char *s__ign, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == '\0' || sc == c)
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strrchr - Find the last occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the last occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strrchr(const char *s__ign, int c)
{
char sc;
int i, last = -ENOENT;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
last = i;
if (sc == '\0')
return last;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strnlen - Calculate the length of a length-limited string
* @s__ign: The string
* @count: The maximum number of characters to count
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnlen(const char *s__ign, size_t count)
{
char c;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c, s__ign, char, err_out);
if (c == '\0')
return i;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : i;
err_out:
return -EFAULT;
}
/**
* bpf_strlen - Calculate the length of a string
* @s__ign: The string
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strlen(const char *s__ign)
{
return bpf_strnlen(s__ign, XATTR_SIZE_MAX);
}
/**
* bpf_strspn - Calculate the length of the initial substring of @s__ign which
* only contains letters in @accept__ign
* @s__ign: The string to be searched
* @accept__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which only
* contains letters from @accept__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strspn(const char *s__ign, const char *accept__ign)
{
char cs, ca;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(accept__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&ca, accept__ign + j, char, err_out);
if (cs == ca || ca == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (ca == '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strcspn - Calculate the length of the initial substring of @s__ign which
* does not contain letters in @reject__ign
* @s__ign: The string to be searched
* @reject__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which does not
* contain letters from @reject__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcspn(const char *s__ign, const char *reject__ign)
{
char cs, cr;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(reject__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&cr, reject__ign + j, char, err_out);
if (cs == cr || cr == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (cr != '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
static int __bpf_strnstr(const char *s1, const char *s2, size_t len,
bool ignore_case)
{
char c1, c2;
int i, j;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
for (j = 0; i + j <= len && j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&c2, s2 + j, char, err_out);
if (c2 == '\0')
return i;
/*
* We allow reading an extra byte from s2 (note the
* `i + j <= len` above) to cover the case when s2 is
* a suffix of the first len chars of s1.
*/
if (i + j == len)
break;
__get_kernel_nofault(&c1, s1 + j, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 == '\0')
return -ENOENT;
if (c1 != c2)
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (i + j == len)
return -ENOENT;
s1++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strstr - Find the first substring in a string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strstr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, false);
}
/**
* bpf_strcasestr - Find the first substring in a string, ignoring the case of
* the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasestr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, true);
}
/**
* bpf_strnstr - Find the first substring in a length-limited string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strnstr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, false);
}
/**
* bpf_strncasestr - Find the first substring in a length-limited string,
* ignoring the case of the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasestr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, true);
}
#ifdef CONFIG_KEYS
/**
* bpf_lookup_user_key - lookup a key by its serial
* @serial: key handle serial number
* @flags: lookup-specific flags
*
* Search a key with a given *serial* and the provided *flags*.
* If found, increment the reference count of the key by one, and
* return it in the bpf_key structure.
*
* The bpf_key structure must be passed to bpf_key_put() when done
* with it, so that the key reference count is decremented and the
* bpf_key structure is freed.
*
* Permission checks are deferred to the time the key is used by
* one of the available key-specific kfuncs.
*
* Set *flags* with KEY_LOOKUP_CREATE, to attempt creating a requested
* special keyring (e.g. session keyring), if it doesn't yet exist.
* Set *flags* with KEY_LOOKUP_PARTIAL, to lookup a key without waiting
* for the key construction, and to retrieve uninstantiated keys (keys
* without data attached to them).
*
* Return: a bpf_key pointer with a valid key pointer if the key is found, a
* NULL pointer otherwise.
*/
__bpf_kfunc struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags)
{
key_ref_t key_ref;
struct bpf_key *bkey;
if (flags & ~KEY_LOOKUP_ALL)
return NULL;
/*
* Permission check is deferred until the key is used, as the
* intent of the caller is unknown here.
*/
key_ref = lookup_user_key(serial, flags, KEY_DEFER_PERM_CHECK);
if (IS_ERR(key_ref))
return NULL;
bkey = kmalloc_obj(*bkey);
if (!bkey) {
key_put(key_ref_to_ptr(key_ref));
return NULL;
}
bkey->key = key_ref_to_ptr(key_ref);
bkey->has_ref = true;
return bkey;
}
/**
* bpf_lookup_system_key - lookup a key by a system-defined ID
* @id: key ID
*
* Obtain a bpf_key structure with a key pointer set to the passed key ID.
* The key pointer is marked as invalid, to prevent bpf_key_put() from
* attempting to decrement the key reference count on that pointer. The key
* pointer set in such way is currently understood only by
* verify_pkcs7_signature().
*
* Set *id* to one of the values defined in include/linux/verification.h:
* 0 for the primary keyring (immutable keyring of system keys);
* VERIFY_USE_SECONDARY_KEYRING for both the primary and secondary keyring
* (where keys can be added only if they are vouched for by existing keys
* in those keyrings); VERIFY_USE_PLATFORM_KEYRING for the platform
* keyring (primarily used by the integrity subsystem to verify a kexec'ed
* kerned image and, possibly, the initramfs signature).
*
* Return: a bpf_key pointer with an invalid key pointer set from the
* pre-determined ID on success, a NULL pointer otherwise
*/
__bpf_kfunc struct bpf_key *bpf_lookup_system_key(u64 id)
{
struct bpf_key *bkey;
if (system_keyring_id_check(id) < 0)
return NULL;
bkey = kmalloc_obj(*bkey, GFP_ATOMIC);
if (!bkey)
return NULL;
bkey->key = (struct key *)(unsigned long)id;
bkey->has_ref = false;
return bkey;
}
/**
* bpf_key_put - decrement key reference count if key is valid and free bpf_key
* @bkey: bpf_key structure
*
* Decrement the reference count of the key inside *bkey*, if the pointer
* is valid, and free *bkey*.
*/
__bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
{
if (bkey->has_ref)
key_put(bkey->key);
kfree(bkey);
}
/**
* bpf_verify_pkcs7_signature - verify a PKCS#7 signature
* @data_p: data to verify
* @sig_p: signature of the data
* @trusted_keyring: keyring with keys trusted for signature verification
*
* Verify the PKCS#7 signature *sig_ptr* against the supplied *data_ptr*
* with keys in a keyring referenced by *trusted_keyring*.
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const void *data, *sig;
u32 data_len, sig_len;
int ret;
if (trusted_keyring->has_ref) {
/*
* Do the permission check deferred in bpf_lookup_user_key().
* See bpf_lookup_user_key() for more details.
*
* A call to key_task_permission() here would be redundant, as
* it is already done by keyring_search() called by
* find_asymmetric_key().
*/
ret = key_validate(trusted_keyring->key);
if (ret < 0)
return ret;
}
data_len = __bpf_dynptr_size(data_ptr);
data = __bpf_dynptr_data(data_ptr, data_len);
sig_len = __bpf_dynptr_size(sig_ptr);
sig = __bpf_dynptr_data(sig_ptr, sig_len);
return verify_pkcs7_signature(data, data_len, sig, sig_len,
trusted_keyring->key,
VERIFYING_BPF_SIGNATURE, NULL,
NULL);
#else
return -EOPNOTSUPP;
#endif /* CONFIG_SYSTEM_DATA_VERIFICATION */
}
#endif /* CONFIG_KEYS */
typedef int (*bpf_task_work_callback_t)(struct bpf_map *map, void *key, void *value);
enum bpf_task_work_state {
/* bpf_task_work is ready to be used */
BPF_TW_STANDBY = 0,
/* irq work scheduling in progress */
BPF_TW_PENDING,
/* task work scheduling in progress */
BPF_TW_SCHEDULING,
/* task work is scheduled successfully */
BPF_TW_SCHEDULED,
/* callback is running */
BPF_TW_RUNNING,
/* associated BPF map value is deleted */
BPF_TW_FREED,
};
struct bpf_task_work_ctx {
enum bpf_task_work_state state;
refcount_t refcnt;
struct callback_head work;
struct irq_work irq_work;
/* bpf_prog that schedules task work */
struct bpf_prog *prog;
/* task for which callback is scheduled */
struct task_struct *task;
/* the map and map value associated with this context */
struct bpf_map *map;
void *map_val;
enum task_work_notify_mode mode;
bpf_task_work_callback_t callback_fn;
struct rcu_head rcu;
} __aligned(8);
/* Actual type for struct bpf_task_work */
struct bpf_task_work_kern {
struct bpf_task_work_ctx *ctx;
};
static void bpf_task_work_ctx_reset(struct bpf_task_work_ctx *ctx)
{
if (ctx->prog) {
bpf_prog_put(ctx->prog);
ctx->prog = NULL;
}
if (ctx->task) {
bpf_task_release(ctx->task);
ctx->task = NULL;
}
}
static bool bpf_task_work_ctx_tryget(struct bpf_task_work_ctx *ctx)
{
return refcount_inc_not_zero(&ctx->refcnt);
}
static void bpf_task_work_destroy(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_ctx_reset(ctx);
kfree_rcu(ctx, rcu);
}
static void bpf_task_work_ctx_put(struct bpf_task_work_ctx *ctx)
{
if (!refcount_dec_and_test(&ctx->refcnt))
return;
if (irqs_disabled()) {
ctx->irq_work = IRQ_WORK_INIT(bpf_task_work_destroy);
irq_work_queue(&ctx->irq_work);
} else {
bpf_task_work_destroy(&ctx->irq_work);
}
}
static void bpf_task_work_cancel(struct bpf_task_work_ctx *ctx)
{
/*
* Scheduled task_work callback holds ctx ref, so if we successfully
* cancelled, we put that ref on callback's behalf. If we couldn't
* cancel, callback will inevitably run or has already completed
* running, and it would have taken care of its ctx ref itself.
*/
if (task_work_cancel(ctx->task, &ctx->work))
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_callback(struct callback_head *cb)
{
struct bpf_task_work_ctx *ctx = container_of(cb, struct bpf_task_work_ctx, work);
enum bpf_task_work_state state;
u32 idx;
void *key;
/* Read lock is needed to protect ctx and map key/value access */
guard(rcu_tasks_trace)();
/*
* This callback may start running before bpf_task_work_irq() switched to
* SCHEDULED state, so handle both transition variants SCHEDULING|SCHEDULED -> RUNNING.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_RUNNING);
if (state == BPF_TW_SCHEDULED)
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULED, BPF_TW_RUNNING);
if (state == BPF_TW_FREED) {
bpf_task_work_ctx_put(ctx);
return;
}
key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx);
migrate_disable();
ctx->callback_fn(ctx->map, key, ctx->map_val);
migrate_enable();
bpf_task_work_ctx_reset(ctx);
(void)cmpxchg(&ctx->state, BPF_TW_RUNNING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_irq(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
enum bpf_task_work_state state;
int err;
guard(rcu)();
if (cmpxchg(&ctx->state, BPF_TW_PENDING, BPF_TW_SCHEDULING) != BPF_TW_PENDING) {
bpf_task_work_ctx_put(ctx);
return;
}
err = task_work_add(ctx->task, &ctx->work, ctx->mode);
if (err) {
bpf_task_work_ctx_reset(ctx);
/*
* try to switch back to STANDBY for another task_work reuse, but we might have
* gone to FREED already, which is fine as we already cleaned up after ourselves
*/
(void)cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
return;
}
/*
* It's technically possible for just scheduled task_work callback to
* complete running by now, going SCHEDULING -> RUNNING and then
* dropping its ctx refcount. Instead of capturing an extra ref just
* to protect below ctx->state access, we rely on rcu_read_lock
* above to prevent kfree_rcu from freeing ctx before we return.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_SCHEDULED);
if (state == BPF_TW_FREED)
bpf_task_work_cancel(ctx); /* clean up if we switched into FREED state */
}
static struct bpf_task_work_ctx *bpf_task_work_fetch_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_kern *twk = (void *)tw;
struct bpf_task_work_ctx *ctx, *old_ctx;
ctx = READ_ONCE(twk->ctx);
if (ctx)
return ctx;
ctx = bpf_map_kmalloc_nolock(map, sizeof(*ctx), 0, NUMA_NO_NODE);
if (!ctx)
return ERR_PTR(-ENOMEM);
memset(ctx, 0, sizeof(*ctx));
refcount_set(&ctx->refcnt, 1); /* map's own ref */
ctx->state = BPF_TW_STANDBY;
old_ctx = cmpxchg(&twk->ctx, NULL, ctx);
if (old_ctx) {
/*
* tw->ctx is set by concurrent BPF program, release allocated
* memory and try to reuse already set context.
*/
kfree_nolock(ctx);
return old_ctx;
}
return ctx; /* Success */
}
static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_ctx *ctx;
/*
* Sleepable BPF programs hold rcu_read_lock_trace but not
* regular rcu_read_lock. Since kfree_rcu waits for regular
* RCU GP, the ctx can be freed while we're between reading
* the pointer and incrementing the refcount. Take regular
* rcu_read_lock to prevent kfree_rcu from freeing the ctx
* before we can tryget it.
*/
scoped_guard(rcu) {
ctx = bpf_task_work_fetch_ctx(tw, map);
if (IS_ERR(ctx))
return ctx;
/* try to get ref for task_work callback to hold */
if (!bpf_task_work_ctx_tryget(ctx))
return ERR_PTR(-EBUSY);
}
if (cmpxchg(&ctx->state, BPF_TW_STANDBY, BPF_TW_PENDING) != BPF_TW_STANDBY) {
/* lost acquiring race or map_release_uref() stole it from us, put ref and bail */
bpf_task_work_ctx_put(ctx);
return ERR_PTR(-EBUSY);
}
/*
* If no process or bpffs is holding a reference to the map, no new callbacks should be
* scheduled. This does not address any race or correctness issue, but rather is a policy
* choice: dropping user references should stop everything.
*/
if (!atomic64_read(&map->usercnt)) {
/* drop ref we just got for task_work callback itself */
bpf_task_work_ctx_put(ctx);
/* transfer map's ref into cancel_and_free() */
bpf_task_work_cancel_and_free(tw);
return ERR_PTR(-EBUSY);
}
return ctx;
}
static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw,
struct bpf_map *map, bpf_task_work_callback_t callback_fn,
struct bpf_prog_aux *aux, enum task_work_notify_mode mode)
{
struct bpf_prog *prog;
struct bpf_task_work_ctx *ctx;
int err;
BTF_TYPE_EMIT(struct bpf_task_work);
prog = bpf_prog_inc_not_zero(aux->prog);
if (IS_ERR(prog))
return -EBADF;
task = bpf_task_acquire(task);
if (!task) {
err = -EBADF;
goto release_prog;
}
ctx = bpf_task_work_acquire_ctx(tw, map);
if (IS_ERR(ctx)) {
err = PTR_ERR(ctx);
goto release_all;
}
ctx->task = task;
ctx->callback_fn = callback_fn;
ctx->prog = prog;
ctx->mode = mode;
ctx->map = map;
ctx->map_val = (void *)tw - map->record->task_work_off;
init_task_work(&ctx->work, bpf_task_work_callback);
init_irq_work(&ctx->irq_work, bpf_task_work_irq);
irq_work_queue(&ctx->irq_work);
return 0;
release_all:
bpf_task_release(task);
release_prog:
bpf_prog_put(prog);
return err;
}
/**
* bpf_task_work_schedule_signal - Schedule BPF callback using task_work_add with TWA_SIGNAL
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL);
}
/**
* bpf_task_work_schedule_resume - Schedule BPF callback using task_work_add with TWA_RESUME
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME);
}
static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
struct bpf_dynptr_kern *ptr)
{
struct bpf_dynptr_file_impl *state;
/* flags is currently unsupported */
if (flags) {
bpf_dynptr_set_null(ptr);
return -EINVAL;
}
state = kmalloc_nolock(sizeof(*state), 0, NUMA_NO_NODE);
if (!state) {
bpf_dynptr_set_null(ptr);
return -ENOMEM;
}
state->offset = 0;
state->size = U64_MAX; /* Don't restrict size, as file may change anyways */
freader_init_from_file(&state->freader, NULL, 0, file, may_sleep);
bpf_dynptr_init(ptr, state, BPF_DYNPTR_TYPE_FILE, 0, 0);
bpf_dynptr_set_rdonly(ptr);
return 0;
}
__bpf_kfunc int bpf_dynptr_from_file(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, false, (struct bpf_dynptr_kern *)ptr__uninit);
}
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, true, (struct bpf_dynptr_kern *)ptr__uninit);
}
__bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)dynptr;
struct bpf_dynptr_file_impl *df = ptr->data;
if (!df)
return 0;
freader_cleanup(&df->freader);
kfree_nolock(df);
bpf_dynptr_set_null(ptr);
return 0;
}
/**
* bpf_timer_cancel_async - try to deactivate a timer
* @timer: bpf_timer to stop
*
* Returns:
*
* * 0 when the timer was not active
* * 1 when the timer was active
* * -1 when the timer is currently executing the callback function and
* cannot be stopped
* * -ECANCELED when the timer will be cancelled asynchronously
* * -ENOMEM when out of memory
* * -EINVAL when the timer was not initialized
* * -ENOENT when this kfunc is racing with timer deletion
*/
__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer)
{
struct bpf_async_kern *async = (void *)timer;
struct bpf_async_cb *cb;
int ret;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
/*
* Unlike hrtimer_start() it's ok to synchronously call
* hrtimer_try_to_cancel() when refcnt reached zero, but deferring to
* irq_work is not, since irq callback may execute after RCU GP and
* cb could be freed at that time. Check for refcnt zero for
* consistency.
*/
if (!refcount_inc_not_zero(&cb->refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
ret = hrtimer_try_to_cancel(&t->timer);
bpf_async_refcount_put(cb);
return ret;
} else {
ret = bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
return ret ? ret : -ECANCELED;
}
}
__bpf_kfunc_end_defs();
static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_cancel(ctx); /* this might put task_work callback's ref */
bpf_task_work_ctx_put(ctx); /* and here we put map's own ref that was transferred to us */
}
void bpf_task_work_cancel_and_free(void *val)
{
struct bpf_task_work_kern *twk = val;
struct bpf_task_work_ctx *ctx;
enum bpf_task_work_state state;
ctx = xchg(&twk->ctx, NULL);
if (!ctx)
return;
state = xchg(&ctx->state, BPF_TW_FREED);
if (state == BPF_TW_SCHEDULED) {
/* run in irq_work to avoid locks in NMI */
init_irq_work(&ctx->irq_work, bpf_task_work_cancel_scheduled);
irq_work_queue(&ctx->irq_work);
return;
}
bpf_task_work_ctx_put(ctx); /* put bpf map's ref */
}
BTF_KFUNCS_START(generic_btf_ids)
#ifdef CONFIG_CRASH_DUMP
BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
#endif
BTF_ID_FLAGS(func, bpf_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_percpu_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_refcount_acquire, KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE | KF_RET_NULL | KF_RCU)
BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_front_impl)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_back_impl)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_rbtree_add_impl)
BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_cgroup_ancestor, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU)
BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
#endif
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_send_signal_task)
#endif
#ifdef CONFIG_KEYS
BTF_ID_FLAGS(func, bpf_lookup_user_key, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_lookup_system_key, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_key_put, KF_RELEASE)
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
BTF_ID_FLAGS(func, bpf_verify_pkcs7_signature, KF_SLEEPABLE)
#endif
#endif
#ifdef CONFIG_S390
BTF_ID_FLAGS(func, bpf_get_lowcore)
#endif
BTF_KFUNCS_END(generic_btf_ids)
static const struct btf_kfunc_id_set generic_kfunc_set = {
.owner = THIS_MODULE,
.set = &generic_btf_ids,
};
BTF_ID_LIST(generic_dtor_ids)
BTF_ID(struct, task_struct)
BTF_ID(func, bpf_task_release_dtor)
#ifdef CONFIG_CGROUPS
BTF_ID(struct, cgroup)
BTF_ID(func, bpf_cgroup_release_dtor)
#endif
BTF_KFUNCS_START(common_btf_ids)
BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rcu_read_lock)
BTF_ID_FLAGS(func, bpf_rcu_read_unlock)
BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_css_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_css_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_destroy, KF_ITER_DESTROY)
#endif
BTF_ID_FLAGS(func, bpf_iter_task_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_dynptr_adjust)
BTF_ID_FLAGS(func, bpf_dynptr_is_null)
BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
BTF_ID_FLAGS(func, bpf_dynptr_size)
BTF_ID_FLAGS(func, bpf_dynptr_clone)
BTF_ID_FLAGS(func, bpf_dynptr_copy)
BTF_ID_FLAGS(func, bpf_dynptr_memset)
#ifdef CONFIG_NET
BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
#endif
BTF_ID_FLAGS(func, bpf_wq_init)
BTF_ID_FLAGS(func, bpf_wq_set_callback, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_wq_start)
BTF_ID_FLAGS(func, bpf_preempt_disable)
BTF_ID_FLAGS(func, bpf_preempt_enable)
BTF_ID_FLAGS(func, bpf_iter_bits_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_bits_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_bits_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_copy_from_user_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_get_kmem_cache)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_local_irq_save)
BTF_ID_FLAGS(func, bpf_local_irq_restore)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr)
BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE)
#endif
#ifdef CONFIG_DMA_SHARED_BUFFER
BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
#endif
BTF_ID_FLAGS(func, __bpf_trap)
BTF_ID_FLAGS(func, bpf_strcmp);
BTF_ID_FLAGS(func, bpf_strcasecmp);
BTF_ID_FLAGS(func, bpf_strncasecmp);
BTF_ID_FLAGS(func, bpf_strchr);
BTF_ID_FLAGS(func, bpf_strchrnul);
BTF_ID_FLAGS(func, bpf_strnchr);
BTF_ID_FLAGS(func, bpf_strrchr);
BTF_ID_FLAGS(func, bpf_strlen);
BTF_ID_FLAGS(func, bpf_strnlen);
BTF_ID_FLAGS(func, bpf_strspn);
BTF_ID_FLAGS(func, bpf_strcspn);
BTF_ID_FLAGS(func, bpf_strstr);
BTF_ID_FLAGS(func, bpf_strcasestr);
BTF_ID_FLAGS(func, bpf_strnstr);
BTF_ID_FLAGS(func, bpf_strncasestr);
#if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS)
BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU)
#endif
BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_dynptr_from_file)
BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
BTF_ID_FLAGS(func, bpf_timer_cancel_async)
BTF_KFUNCS_END(common_btf_ids)
static const struct btf_kfunc_id_set common_kfunc_set = {
.owner = THIS_MODULE,
.set = &common_btf_ids,
};
static int __init kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc generic_dtors[] = {
{
.btf_id = generic_dtor_ids[0],
.kfunc_btf_id = generic_dtor_ids[1]
},
#ifdef CONFIG_CGROUPS
{
.btf_id = generic_dtor_ids[2],
.kfunc_btf_id = generic_dtor_ids[3]
},
#endif
};
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &generic_kfunc_set);
ret = ret ?: register_btf_id_dtor_kfuncs(generic_dtors,
ARRAY_SIZE(generic_dtors),
THIS_MODULE);
return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
}
late_initcall(kfunc_init);
/* Get a pointer to dynptr data up to len bytes for read only access. If
* the dynptr doesn't have continuous data up to len bytes, return NULL.
*/
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u64 len)
{
const struct bpf_dynptr *p = (struct bpf_dynptr *)ptr;
return bpf_dynptr_slice(p, 0, NULL, len);
}
/* Get a pointer to dynptr data up to len bytes for read write access. If
* the dynptr doesn't have continuous data up to len bytes, or the dynptr
* is read only, return NULL.
*/
void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u64 len)
{
if (__bpf_dynptr_is_rdonly(ptr))
return NULL;
return (void *)__bpf_dynptr_data(ptr, len);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *val)
{
if (btf_record_has_field(map->record, BPF_TIMER))
bpf_obj_free_timer(map->record, val);
if (btf_record_has_field(map->record, BPF_WORKQUEUE))
bpf_obj_free_workqueue(map->record, val);
if (btf_record_has_field(map->record, BPF_TASK_WORK))
bpf_obj_free_task_work(map->record, val);
}
]
|
| 344/6 |
2026/05/18 19:13 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 67513
output: 41
thoughts: 79
Thoughts: Okay, here's my assessment of the situation:
**Potential Buffer Overflow Vulnerability Review**
Alright, I've just finished going over the `bpf_bprintf_prepare` file. My initial concern, as I suspected, is spot on. Those two `snprintf` calls are screaming potential buffer overflow. The fact that there's no truncation check and the code increments `tmp_buf` based on `err + 1` is a classic recipe for trouble. We're directly writing into the buffer without proper bounds checking – a disaster waiting to happen.
Now, I can't just stop there. I need to be absolutely certain this isn't a systemic issue. One dangerous `snprintf` is bad enough, but a pattern would be a red flag. I’ll run a quick `grep` to verify there aren't any other suspicious calls in the codebase. Let's make sure this is isolated and not a recurring design flaw. I have a strong feeling about this.
|
| 345/6 |
2026/05/18 19:14 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924 SourceSnippet: 923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965 SourceSnippet: 964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
]] TruncatedOutput:false]
|
| 346/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 68090
output: 40
thoughts: 125
Thoughts: Okay, let me think this through.
**Investigating a Potential String Formatting Vulnerability**
Alright, I'm taking a look at this code, and the first thing that jumps out is `bpf_snprintf` on line 1064. Interesting... I'm immediately thinking about format string vulnerabilities. I see the function signature: `BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt, const void *, args, u32, data_len)`. But hold on... that calls `bstr_printf` internally, not `snprintf`. Okay, so `bpf_snprintf` itself might not be the problem directly, but it's a good starting point. This means I need to dig deeper into how the inputs to `bstr_printf` are handled. I need to be sure.
Let me broaden the scope of my search. I'll scan the entire code base for other potentially vulnerable instances. I'll search for uses of both `sprintf` and `snprintf`. I want a comprehensive picture of all the string formatting functions being used, and how they are handled. This will tell me where the critical points are.
|
| 347/6 |
2026/05/18 19:14 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:sprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 348/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 68534
output: 42
thoughts: 13
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Considering String Formatting in this Context**
Right, so I'm thinking about string formatting here. The obvious initial thought is, "No `sprintf`." It's just inherently unsafe, especially with potential user-provided inputs or data from untrusted sources. Buffer overflows are a nightmare. I've seen far too many security vulnerabilities stem from its use.
Then I immediately shift gears and consider `vsnprintf`. That's the go-to alternative, isn't it? A safer approach... I'm thinking about the control it gives me over the output buffer size, that's crucial. I need to ensure I'm handling the size correctly to prevent any memory issues. I can avoid those risks. So, `vsnprintf` is definitely a consideration, assuming I'm handling it with due care. I need to be aware of the edge cases and potential complexities, and handle any return codes.
|
| 349/6 |
2026/05/18 19:14 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:vsnprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 350/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 68815
output: 27
thoughts: 657
Thoughts: **Analyzing Potential Buffer Overflows in `kernel/bpf/helpers.c`**
Okay, so the user specifically mentioned `kernel/bpf/helpers.c` and is asking about the use of `snprintf` and the potential for improper return value handling. The phrasing of the question suggests they might already know about a specific instance of this issue. My initial thought was to quickly check for `vsnprintf` usage, but I didn't see any, so I need to dig deeper.
Let's re-examine the two identified `snprintf` calls. In the first instance, at line 924, `snprintf` is used with a format specifier `%pB`, which seems related to printing a kernel pointer. The critical point is that if `snprintf` truncates the output, the `err` variable will contain the *length* of the string that *would* have been written, and the code increments `tmp_buf` by `err + 1`. If `err` is large, `tmp_buf` could easily exceed the allocated buffer size (`tmp_buf_end - tmp_buf`). This is especially worrisome because in the next iteration of the loop, the code potentially uses this out-of-bounds `tmp_buf` in a call like `bpf_trace_copy_string`. A negative size calculation can become a huge size when cast to `size_t`. This results in `strncpy_from_kernel_nofault` copying a vast amount of data into the already out-of-bounds `tmp_buf`, leading to a buffer overflow. It's a textbook vulnerability!
The second call, at line 965, shares the identical vulnerability because it also increments `tmp_buf` by `err + 1`. It also calls `snprintf` into `tmp_buf` and then increments `tmp_buf` in a similar fashion. The risks are identical.
The question "Is there any **other** place in `kernel/bpf/helpers.c` where `snprintf` is used to format strings into a buffer without checking the return value properly?" is the key here. It suggests that there is a known issue. I'm wondering if this is a known bug or a recent patch, so I need to be thorough.
To avoid missing anything, I need to make sure I don't overlook any other uses of `snprintf`. I'll use `codesearch-file-index` to get a full list of all entities in the file and then search the file content to be sure that I haven't missed any.
|
| 351/6 |
2026/05/18 19:14 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/bpf/helpers.c]
Results: map[Entities:[map[Kind:function Name:____bpf_copy_from_user] map[Kind:function Name:____bpf_copy_from_user_task] map[Kind:function Name:____bpf_current_task_under_cgroup] map[Kind:function Name:____bpf_dynptr_data] map[Kind:function Name:____bpf_dynptr_from_mem] map[Kind:function Name:____bpf_dynptr_read] map[Kind:function Name:____bpf_dynptr_write] map[Kind:function Name:____bpf_event_output_data] map[Kind:function Name:____bpf_get_current_ancestor_cgroup_id] map[Kind:function Name:____bpf_get_current_cgroup_id] map[Kind:function Name:____bpf_get_current_comm] map[Kind:function Name:____bpf_get_current_pid_tgid] map[Kind:function Name:____bpf_get_current_uid_gid] map[Kind:function Name:____bpf_get_ns_current_pid_tgid] map[Kind:function Name:____bpf_get_numa_node_id] map[Kind:function Name:____bpf_get_smp_processor_id] map[Kind:function Name:____bpf_jiffies64] map[Kind:function Name:____bpf_kptr_xchg] map[Kind:function Name:____bpf_ktime_get_boot_ns] map[Kind:function Name:____bpf_ktime_get_coarse_ns] map[Kind:function Name:____bpf_ktime_get_ns] map[Kind:function Name:____bpf_ktime_get_tai_ns] map[Kind:function Name:____bpf_map_delete_elem] map[Kind:function Name:____bpf_map_lookup_elem] map[Kind:function Name:____bpf_map_lookup_percpu_elem] map[Kind:function Name:____bpf_map_peek_elem] map[Kind:function Name:____bpf_map_pop_elem] map[Kind:function Name:____bpf_map_push_elem] map[Kind:function Name:____bpf_map_update_elem] map[Kind:function Name:____bpf_per_cpu_ptr] map[Kind:function Name:____bpf_snprintf] map[Kind:function Name:____bpf_spin_lock] map[Kind:function Name:____bpf_spin_unlock] map[Kind:function Name:____bpf_strncmp] map[Kind:function Name:____bpf_strtol] map[Kind:function Name:____bpf_strtoul] map[Kind:function Name:____bpf_this_cpu_ptr] map[Kind:function Name:____bpf_timer_cancel] map[Kind:function Name:____bpf_timer_init] map[Kind:function Name:____bpf_timer_set_callback] map[Kind:function Name:____bpf_timer_start] map[Kind:function Name:__bpf_async_init] map[Kind:function Name:__bpf_async_set_callback] map[Kind:function Name:__bpf_dynptr_data] map[Kind:function Name:__bpf_dynptr_data_rw] map[Kind:function Name:__bpf_dynptr_is_rdonly] map[Kind:function Name:__bpf_dynptr_read] map[Kind:function Name:__bpf_dynptr_size] map[Kind:function Name:__bpf_dynptr_write] map[Kind:function Name:__bpf_list_add] map[Kind:function Name:__bpf_list_del] map[Kind:function Name:__bpf_obj_drop_impl] map[Kind:function Name:__bpf_rbtree_add] map[Kind:function Name:__bpf_spin_lock] map[Kind:function Name:__bpf_spin_lock_irqsave] map[Kind:function Name:__bpf_spin_unlock] map[Kind:function Name:__bpf_spin_unlock_irqrestore] map[Kind:function Name:__bpf_strncasecmp] map[Kind:function Name:__bpf_strnstr] map[Kind:function Name:__bpf_strtoll] map[Kind:function Name:__bpf_strtoull] map[Kind:function Name:__bpf_trap] map[Kind:function Name:bpf_async_cancel_and_free] map[Kind:function Name:bpf_async_cb_rcu_free] map[Kind:function Name:bpf_async_cb_rcu_tasks_trace_free] map[Kind:function Name:bpf_async_irq_worker] map[Kind:function Name:bpf_async_process_op] map[Kind:function Name:bpf_async_refcount_put] map[Kind:function Name:bpf_async_schedule_op] map[Kind:function Name:bpf_async_update_prog_callback] map[Kind:function Name:bpf_base_func_proto] map[Kind:function Name:bpf_bprintf_cleanup] map[Kind:function Name:bpf_bprintf_prepare] map[Kind:function Name:bpf_cast_to_kern_ctx] map[Kind:function Name:bpf_cgroup_acquire] map[Kind:function Name:bpf_cgroup_ancestor] map[Kind:function Name:bpf_cgroup_from_id] map[Kind:function Name:bpf_cgroup_release] map[Kind:function Name:bpf_cgroup_release_dtor] map[Kind:function Name:bpf_copy_from_user] map[Kind:function Name:bpf_copy_from_user_str] map[Kind:function Name:bpf_copy_from_user_task] map[Kind:function Name:bpf_copy_from_user_task_str] map[Kind:function Name:bpf_current_task_under_cgroup] map[Kind:function Name:bpf_dynptr_adjust] map[Kind:function Name:bpf_dynptr_advance_offset] map[Kind:function Name:bpf_dynptr_check_size] map[Kind:function Name:bpf_dynptr_clone] map[Kind:function Name:bpf_dynptr_copy] map[Kind:function Name:bpf_dynptr_data] map[Kind:function Name:bpf_dynptr_file_discard] map[Kind:function Name:bpf_dynptr_from_file] map[Kind:function Name:bpf_dynptr_from_file_sleepable] map[Kind:function Name:bpf_dynptr_from_mem] map[Kind:function Name:bpf_dynptr_get_type] map[Kind:function Name:bpf_dynptr_init] map[Kind:function Name:bpf_dynptr_is_null] map[Kind:function Name:bpf_dynptr_is_rdonly] map[Kind:function Name:bpf_dynptr_memset] map[Kind:function Name:bpf_dynptr_read] map[Kind:function Name:bpf_dynptr_set_null] map[Kind:function Name:bpf_dynptr_set_rdonly] map[Kind:function Name:bpf_dynptr_set_size] map[Kind:function Name:bpf_dynptr_set_type] map[Kind:function Name:bpf_dynptr_size] map[Kind:function Name:bpf_dynptr_slice] map[Kind:function Name:bpf_dynptr_slice_rdwr] map[Kind:function Name:bpf_dynptr_write] map[Kind:function Name:bpf_event_output_data] map[Kind:function Name:bpf_file_fetch_bytes] map[Kind:function Name:bpf_get_current_ancestor_cgroup_id] map[Kind:function Name:bpf_get_current_cgroup_id] map[Kind:function Name:bpf_get_current_comm] map[Kind:function Name:bpf_get_current_pid_tgid] map[Kind:function Name:bpf_get_current_uid_gid] map[Kind:function Name:bpf_get_ns_current_pid_tgid] map[Kind:function Name:bpf_get_numa_node_id] map[Kind:function Name:bpf_get_smp_processor_id] map[Kind:function Name:bpf_iter_bits_destroy] map[Kind:function Name:bpf_iter_bits_new] map[Kind:function Name:bpf_iter_bits_next] map[Kind:function Name:bpf_jiffies64] map[Kind:function Name:bpf_key_put] map[Kind:function Name:bpf_kptr_xchg] map[Kind:function Name:bpf_ktime_get_boot_ns] map[Kind:function Name:bpf_ktime_get_coarse_ns] map[Kind:function Name:bpf_ktime_get_ns] map[Kind:function Name:bpf_ktime_get_tai_ns] map[Kind:function Name:bpf_list_back] map[Kind:function Name:bpf_list_front] map[Kind:function Name:bpf_list_head_free] map[Kind:function Name:bpf_list_pop_back] map[Kind:function Name:bpf_list_pop_front] map[Kind:function Name:bpf_list_push_back] map[Kind:function Name:bpf_list_push_back_impl] map[Kind:function Name:bpf_list_push_front] map[Kind:function Name:bpf_list_push_front_impl] map[Kind:function Name:bpf_local_irq_restore] map[Kind:function Name:bpf_local_irq_save] map[Kind:function Name:bpf_lookup_system_key] map[Kind:function Name:bpf_lookup_user_key] map[Kind:function Name:bpf_map_delete_elem] map[Kind:function Name:bpf_map_free_internal_structs] map[Kind:function Name:bpf_map_lookup_elem] map[Kind:function Name:bpf_map_lookup_percpu_elem] map[Kind:function Name:bpf_map_peek_elem] map[Kind:function Name:bpf_map_pop_elem] map[Kind:function Name:bpf_map_push_elem] map[Kind:function Name:bpf_map_update_elem] map[Kind:function Name:bpf_obj_drop] map[Kind:function Name:bpf_obj_drop_impl] map[Kind:function Name:bpf_obj_new] map[Kind:function Name:bpf_obj_new_impl] map[Kind:function Name:bpf_per_cpu_ptr] map[Kind:function Name:bpf_percpu_obj_drop] map[Kind:function Name:bpf_percpu_obj_drop_impl] map[Kind:function Name:bpf_percpu_obj_new] map[Kind:function Name:bpf_percpu_obj_new_impl] map[Kind:function Name:bpf_preempt_disable] map[Kind:function Name:bpf_preempt_enable] map[Kind:function Name:bpf_put_buffers] map[Kind:function Name:bpf_rb_root_free] map[Kind:function Name:bpf_rbtree_add] map[Kind:function Name:bpf_rbtree_add_impl] map[Kind:function Name:bpf_rbtree_first] map[Kind:function Name:bpf_rbtree_left] map[Kind:function Name:bpf_rbtree_remove] map[Kind:function Name:bpf_rbtree_right] map[Kind:function Name:bpf_rbtree_root] map[Kind:function Name:bpf_rcu_read_lock] map[Kind:function Name:bpf_rcu_read_unlock] map[Kind:function Name:bpf_rdonly_cast] map[Kind:function Name:bpf_refcount_acquire] map[Kind:function Name:bpf_refcount_acquire_impl] map[Kind:function Name:bpf_snprintf] map[Kind:function Name:bpf_spin_lock] map[Kind:function Name:bpf_spin_unlock] map[Kind:function Name:bpf_stack_walker] map[Kind:function Name:bpf_strcasecmp] map[Kind:function Name:bpf_strcasestr] map[Kind:function Name:bpf_strchr] map[Kind:function Name:bpf_strchrnul] map[Kind:function Name:bpf_strcmp] map[Kind:function Name:bpf_strcspn] map[Kind:function Name:bpf_strlen] map[Kind:function Name:bpf_strncasecmp] map[Kind:function Name:bpf_strncasestr] map[Kind:function Name:bpf_strnchr] map[Kind:function Name:bpf_strncmp] map[Kind:function Name:bpf_strnlen] map[Kind:function Name:bpf_strnstr] map[Kind:function Name:bpf_strrchr] map[Kind:function Name:bpf_strspn] map[Kind:function Name:bpf_strstr] map[Kind:function Name:bpf_strtol] map[Kind:function Name:bpf_strtoul] map[Kind:function Name:bpf_task_acquire] map[Kind:function Name:bpf_task_from_pid] map[Kind:function Name:bpf_task_from_vpid] map[Kind:function Name:bpf_task_get_cgroup1] map[Kind:function Name:bpf_task_release] map[Kind:function Name:bpf_task_release_dtor] map[Kind:function Name:bpf_task_under_cgroup] map[Kind:function Name:bpf_task_work_acquire_ctx] map[Kind:function Name:bpf_task_work_callback] map[Kind:function Name:bpf_task_work_cancel] map[Kind:function Name:bpf_task_work_cancel_and_free] map[Kind:function Name:bpf_task_work_cancel_scheduled] map[Kind:function Name:bpf_task_work_ctx_put] map[Kind:function Name:bpf_task_work_ctx_reset] map[Kind:function Name:bpf_task_work_ctx_tryget] map[Kind:function Name:bpf_task_work_destroy] map[Kind:function Name:bpf_task_work_fetch_ctx] map[Kind:function Name:bpf_task_work_irq] map[Kind:function Name:bpf_task_work_schedule] map[Kind:function Name:bpf_task_work_schedule_resume] map[Kind:function Name:bpf_task_work_schedule_signal] map[Kind:function Name:bpf_this_cpu_ptr] map[Kind:function Name:bpf_throw] map[Kind:function Name:bpf_timer_cancel] map[Kind:function Name:bpf_timer_cancel_and_free] map[Kind:function Name:bpf_timer_cancel_async] map[Kind:function Name:bpf_timer_cb] map[Kind:function Name:bpf_timer_init] map[Kind:function Name:bpf_timer_set_callback] map[Kind:function Name:bpf_timer_start] map[Kind:function Name:bpf_trace_copy_string] map[Kind:function Name:bpf_try_get_buffers] map[Kind:function Name:bpf_verify_pkcs7_signature] map[Kind:function Name:bpf_wq_cancel_and_free] map[Kind:function Name:bpf_wq_init] map[Kind:function Name:bpf_wq_set_callback] map[Kind:function Name:bpf_wq_start] map[Kind:function Name:bpf_wq_work] map[Kind:function Name:copy_map_value_locked] map[Kind:function Name:defer_timer_wq_op] map[Kind:function Name:kfunc_init] map[Kind:function Name:make_file_dynptr] map[Kind:function Name:map_key_from_value] map[Kind:function Name:swap_ulong_in_u64] map[Kind:function Name:worker_for_call_rcu] map[Kind:struct Name:bpf_async_cb] map[Kind:struct Name:bpf_async_cmd] map[Kind:struct Name:bpf_async_kern] map[Kind:struct Name:bpf_dynptr_file_impl] map[Kind:struct Name:bpf_hrtimer] map[Kind:struct Name:bpf_iter_bits] map[Kind:struct Name:bpf_iter_bits_kern] map[Kind:struct Name:bpf_task_work_ctx] map[Kind:struct Name:bpf_task_work_kern] map[Kind:struct Name:bpf_throw_ctx] map[Kind:struct Name:bpf_work] map[Kind:global_variable Name:__UNIQUE_ID_addressable_bpf_base_func_proto_789] map[Kind:global_variable Name:__UNIQUE_ID_addressable_kfunc_init_832] map[Kind:global_variable Name:bpf_copy_from_user_proto] map[Kind:global_variable Name:bpf_copy_from_user_task_proto] map[Kind:global_variable Name:bpf_current_task_under_cgroup_proto] map[Kind:global_variable Name:bpf_dynptr_data_proto] map[Kind:global_variable Name:bpf_dynptr_from_mem_proto] map[Kind:global_variable Name:bpf_dynptr_read_proto] map[Kind:global_variable Name:bpf_dynptr_write_proto] map[Kind:global_variable Name:bpf_event_output_data_proto] map[Kind:global_variable Name:bpf_get_current_ancestor_cgroup_id_proto] map[Kind:global_variable Name:bpf_get_current_cgroup_id_proto] map[Kind:global_variable Name:bpf_get_current_comm_proto] map[Kind:global_variable Name:bpf_get_current_pid_tgid_proto] map[Kind:global_variable Name:bpf_get_current_uid_gid_proto] map[Kind:global_variable Name:bpf_get_ns_current_pid_tgid_proto] map[Kind:global_variable Name:bpf_get_numa_node_id_proto] map[Kind:global_variable Name:bpf_get_prandom_u32_proto] map[Kind:global_variable Name:bpf_get_raw_smp_processor_id_proto] map[Kind:global_variable Name:bpf_get_smp_processor_id_proto] map[Kind:global_variable Name:bpf_jiffies64_proto] map[Kind:global_variable Name:bpf_kptr_xchg_proto] map[Kind:global_variable Name:bpf_ktime_get_boot_ns_proto] map[Kind:global_variable Name:bpf_ktime_get_coarse_ns_proto] map[Kind:global_variable Name:bpf_ktime_get_ns_proto] map[Kind:global_variable Name:bpf_ktime_get_tai_ns_proto] map[Kind:global_variable Name:bpf_map_delete_elem_proto] map[Kind:global_variable Name:bpf_map_lookup_elem_proto] map[Kind:global_variable Name:bpf_map_lookup_percpu_elem_proto] map[Kind:global_variable Name:bpf_map_peek_elem_proto] map[Kind:global_variable Name:bpf_map_pop_elem_proto] map[Kind:global_variable Name:bpf_map_push_elem_proto] map[Kind:global_variable Name:bpf_map_update_elem_proto] map[Kind:global_variable Name:bpf_per_cpu_ptr_proto] map[Kind:global_variable Name:bpf_snprintf_proto] map[Kind:global_variable Name:bpf_spin_lock_proto] map[Kind:global_variable Name:bpf_spin_unlock_proto] map[Kind:global_variable Name:bpf_strncmp_proto] map[Kind:global_variable Name:bpf_strtol_proto] map[Kind:global_variable Name:bpf_strtoul_proto] map[Kind:global_variable Name:bpf_this_cpu_ptr_proto] map[Kind:global_variable Name:bpf_timer_cancel_proto] map[Kind:global_variable Name:bpf_timer_init_proto] map[Kind:global_variable Name:bpf_timer_set_callback_proto] map[Kind:global_variable Name:bpf_timer_start_proto] map[Kind:global_variable Name:common_btf_ids] map[Kind:global_variable Name:common_kfunc_set] map[Kind:global_variable Name:generic_btf_ids] map[Kind:global_variable Name:generic_kfunc_set] map[Kind:enum Name:bpf_async_op] map[Kind:enum Name:bpf_async_type] map[Kind:enum Name:bpf_task_work_state] map[Kind:typedef Name:bpf_task_work_callback_t] map[Kind:typedef Name:btf_bpf_copy_from_user] map[Kind:typedef Name:btf_bpf_copy_from_user_task] map[Kind:typedef Name:btf_bpf_current_task_under_cgroup] map[Kind:typedef Name:btf_bpf_dynptr_data] map[Kind:typedef Name:btf_bpf_dynptr_from_mem] map[Kind:typedef Name:btf_bpf_dynptr_read] map[Kind:typedef Name:btf_bpf_dynptr_write] map[Kind:typedef Name:btf_bpf_event_output_data] map[Kind:typedef Name:btf_bpf_get_current_ancestor_cgroup_id] map[Kind:typedef Name:btf_bpf_get_current_cgroup_id] map[Kind:typedef Name:btf_bpf_get_current_comm] map[Kind:typedef Name:btf_bpf_get_current_pid_tgid] map[Kind:typedef Name:btf_bpf_get_current_uid_gid] map[Kind:typedef Name:btf_bpf_get_ns_current_pid_tgid] map[Kind:typedef Name:btf_bpf_get_numa_node_id] map[Kind:typedef Name:btf_bpf_get_smp_processor_id] map[Kind:typedef Name:btf_bpf_jiffies64] map[Kind:typedef Name:btf_bpf_kptr_xchg] map[Kind:typedef Name:btf_bpf_ktime_get_boot_ns] map[Kind:typedef Name:btf_bpf_ktime_get_coarse_ns] map[Kind:typedef Name:btf_bpf_ktime_get_ns] map[Kind:typedef Name:btf_bpf_ktime_get_tai_ns] map[Kind:typedef Name:btf_bpf_map_delete_elem] map[Kind:typedef Name:btf_bpf_map_lookup_elem] map[Kind:typedef Name:btf_bpf_map_lookup_percpu_elem] map[Kind:typedef Name:btf_bpf_map_peek_elem] map[Kind:typedef Name:btf_bpf_map_pop_elem] map[Kind:typedef Name:btf_bpf_map_push_elem] map[Kind:typedef Name:btf_bpf_map_update_elem] map[Kind:typedef Name:btf_bpf_per_cpu_ptr] map[Kind:typedef Name:btf_bpf_snprintf] map[Kind:typedef Name:btf_bpf_spin_lock] map[Kind:typedef Name:btf_bpf_spin_unlock] map[Kind:typedef Name:btf_bpf_strncmp] map[Kind:typedef Name:btf_bpf_strtol] map[Kind:typedef Name:btf_bpf_strtoul] map[Kind:typedef Name:btf_bpf_this_cpu_ptr] map[Kind:typedef Name:btf_bpf_timer_cancel] map[Kind:typedef Name:btf_bpf_timer_init] map[Kind:typedef Name:btf_bpf_timer_set_callback] map[Kind:typedef Name:btf_bpf_timer_start]]]
|
| 352/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 77081
output: 41
thoughts: 260
Thoughts: **My BPF Snprintf Analysis**
Okay, I've already pinpointed those two `snprintf` calls within `bpf_bprintf_prepare`. Let me just double-check that function again, just to be absolutely certain. Yes, I see them: lines 924-928, the `%pB` handling, and lines 965-968 with the `ip_spec` and `cur_ip` formatting.
But I need to be thorough. Are there *any* other instances of `snprintf` in the `kernel/bpf/helpers.c` file that I might have missed initially? Just because I'm confident doesn't mean I can't be wrong. I'll rerun my `grep` search using `codesearch-find-references`, but I'll pay closer attention this time, looking out for any subtle formatting variations that could have thrown off the initial search. We can't let any potential security issues slip through the cracks.
|
| 353/6 |
2026/05/18 19:14 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924 SourceSnippet: 923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965 SourceSnippet: 964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
]] TruncatedOutput:false]
|
| 354/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 77843
output: 33
thoughts: 322
Thoughts: **Analyzing `snprintf` Usage in `kernel/bpf/helpers.c`**
Alright, just these two instances of `snprintf` to focus on for now. But, before I get too deep, it's a good practice to check for any *other* relevant uses of `snprintf` within the `kernel/bpf/` directory, just in case something else is intertwined, even though the user specifically asked about `helpers.c`. Though, I'll keep the focus on *this* file for now.
Let's revisit that first snippet in `helpers.c`. Ah, right, that's line 922:
```c
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
```
Okay, `%pB` is our key here – that's the symbol lookup with offset specifier. And that has the potential to produce a *long* string. `tmp_buf`, that's part of `buffers->bin_args`. So, the size of `tmp_buf` is limited by `MAX_BPRINTF_BIN_ARGS`. Time to find the value of this. I'd expect it to be defined in either `include/linux/bpf.h` or, perhaps, even `kernel/bpf/helpers.c`. I did not find it in `helpers.c` and haven't looked in `bpf.h` yet, but I'll search the entire tree to be thorough, just to be sure.
|
| 355/6 |
2026/05/18 19:14 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:MAX_BPRINTF_BIN_ARGS]
Results: map[References:<nil> TruncatedOutput:false]
|
| 356/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 78656
output: 48
thoughts: 22
Thoughts: Okay, here's my summarized thought process, presented as if I were thinking it:
**Analyzing the Search Strategy**
Right, let's get down to brass tacks. I'm going to need to locate that `MAX_BPRINTF_BIN_ARGS` string within the codebase. Given the nature of this project, and my experience, `grep` is the obvious tool of choice here. It's fast, efficient, and perfectly suited for this kind of quick text search across multiple files. I can quickly scan the entire project directory with grep for the string I am looking for.
|
| 357/6 |
2026/05/18 19:14 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:MAX_BPRINTF_BIN_ARGS SourceTreePrefix:include/linux/bpf.h]
Results: map[References:<nil> TruncatedOutput:false]
|
| 358/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 78872
output: 45
thoughts: 14
Thoughts: Okay, here's my summary, presented as a direct, first-person narrative, as if I were experiencing these thoughts:
**Navigating the BPF Landscape**
Alright, so I need to find something... and my instinct tells me the best place to start looking is within the `kernel/bpf/` directory. It's the most logical starting point for any BPF-related kernel functionality or code. That's where I'd expect to find the core implementations, definitions, or at least pointers to whatever I'm after. It's essentially the BPF home base, and I'll begin my search there, it's the right level of abstraction.
|
| 359/6 |
2026/05/18 19:14 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:MAX_BPRINTF_BIN_ARGS SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 360/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 79095
output: 48
thoughts: 182
Thoughts: Here's my thought process, summarized:
**Investigating a Suspected Macro Definition**
Okay, I'm pretty sure I remember encountering this `MAX_BPRINTF_BIN_ARGS` in some code recently. I should probably double-check the file content – yep, it's there, specifically line 840 of `kernel/bpf/helpers.c`.
I wonder why my `codesearch-find-references` didn't turn anything up. Could it be a macro? That would explain it. Maybe I should have used `codesearch-definition-source` instead. Wait, no, I already tried that and it didn't work.
Alright, time to go back to the top of `kernel/bpf/helpers.c` and take a closer look. Actually, maybe just searching the file for that line directly is the most efficient approach for now.
|
| 361/6 |
2026/05/18 19:14 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:MAX_BPRINTF_BIN_ARGS SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 362/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 79535
output: 44
thoughts: 129
Thoughts: **Debugging a Peculiar `read-file` Output**
Okay, this is seriously strange. I'm staring at this `read-file` output, and something just doesn't sit right. It's in the C code, specifically this snippet:
```c
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
```
My initial thought is, what's going on with these `bin_args`? It's assigning a buffer pointer and seems like the intention is to use a section of the provided buffer. Let me try to understand this.
I need to quickly dig into the definition of `struct bpf_bprintf_buffers`. My hunch is that the crucial information regarding this structure is probably defined in `include/linux/bpf.h`. I'll open that header and check what's going on there.
|
| 363/6 |
2026/05/18 19:14 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_buffers]
Results: map[SourceCode:3877: struct bpf_bprintf_buffers {
3878: char bin_args[MAX_BPRINTF_BIN_ARGS];
3879: char buf[MAX_BPRINTF_BUF];
3880: };
SourceFile:include/linux/bpf.h]
|
| 364/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 80045
output: 23
thoughts: 37
Thoughts: Here's my thought process, summarized as you requested:
**Navigating the BPF Landscape**
Alright, let's dive in. I need to track down the definition of `MAX_BPRINTF_BIN_ARGS`. Given that it's BPF-related, I'm pretty certain it's hanging out in the kernel's include directory, specifically under `linux/`. That's where all the core BPF structures and definitions reside. So, first stop: `include/linux/bpf.h`. That's the most logical place to begin my search. I'll need to open that header file and locate `MAX_BPRINTF_BIN_ARGS` within its contents. Hopefully it's not too deeply nested or obscured by a ton of preprocessor directives. Let's see what we find.
|
| 365/6 |
2026/05/18 19:14 |
tool |
read-file |
0m
Args: map[File:include/linux/bpf.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0-only */
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*/
#ifndef _LINUX_BPF_H
#define _LINUX_BPF_H 1
#include <uapi/linux/bpf.h>
#include <uapi/linux/filter.h>
#include <crypto/sha2.h>
#include <linux/workqueue.h>
#include <linux/file.h>
#include <linux/percpu.h>
#include <linux/err.h>
#include <linux/rbtree_latch.h>
#include <linux/numa.h>
#include <linux/mm_types.h>
#include <linux/wait.h>
#include <linux/refcount.h>
#include <linux/mutex.h>
#include <linux/module.h>
#include <linux/kallsyms.h>
#include <linux/capability.h>
#include <linux/sched/mm.h>
#include <linux/slab.h>
#include <linux/percpu-refcount.h>
#include <linux/stddef.h>
#include <linux/bpfptr.h>
#include <linux/btf.h>
#include <linux/rcupdate_trace.h>
#include <linux/static_call.h>
#include <linux/memcontrol.h>
#include <linux/cfi.h>
#include <asm/rqspinlock.h>
struct bpf_verifier_env;
struct bpf_verifier_log;
struct perf_event;
struct bpf_prog;
struct bpf_prog_aux;
struct bpf_map;
struct bpf_arena;
struct sock;
struct seq_file;
struct btf;
struct btf_type;
struct exception_table_entry;
struct seq_operations;
struct bpf_iter_aux_info;
struct bpf_local_storage;
struct bpf_local_storage_map;
struct kobject;
struct mem_cgroup;
struct module;
struct bpf_func_state;
struct ftrace_ops;
struct cgroup;
struct bpf_token;
struct user_namespace;
struct super_block;
struct inode;
extern struct idr btf_idr;
extern spinlock_t btf_idr_lock;
extern struct kobject *btf_kobj;
extern struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma;
extern bool bpf_global_ma_set;
typedef u64 (*bpf_callback_t)(u64, u64, u64, u64, u64);
typedef int (*bpf_iter_init_seq_priv_t)(void *private_data,
struct bpf_iter_aux_info *aux);
typedef void (*bpf_iter_fini_seq_priv_t)(void *private_data);
typedef unsigned int (*bpf_func_t)(const void *,
const struct bpf_insn *);
struct bpf_iter_seq_info {
const struct seq_operations *seq_ops;
bpf_iter_init_seq_priv_t init_seq_private;
bpf_iter_fini_seq_priv_t fini_seq_private;
u32 seq_priv_size;
};
/* map is generic key/value storage optionally accessible by eBPF programs */
struct bpf_map_ops {
/* funcs callable from userspace (via syscall) */
int (*map_alloc_check)(union bpf_attr *attr);
struct bpf_map *(*map_alloc)(union bpf_attr *attr);
void (*map_release)(struct bpf_map *map, struct file *map_file);
void (*map_free)(struct bpf_map *map);
int (*map_get_next_key)(struct bpf_map *map, void *key, void *next_key);
void (*map_release_uref)(struct bpf_map *map);
void *(*map_lookup_elem_sys_only)(struct bpf_map *map, void *key);
int (*map_lookup_batch)(struct bpf_map *map, const union bpf_attr *attr,
union bpf_attr __user *uattr);
int (*map_lookup_and_delete_elem)(struct bpf_map *map, void *key,
void *value, u64 flags);
int (*map_lookup_and_delete_batch)(struct bpf_map *map,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
int (*map_update_batch)(struct bpf_map *map, struct file *map_file,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
int (*map_delete_batch)(struct bpf_map *map, const union bpf_attr *attr,
union bpf_attr __user *uattr);
/* funcs callable from userspace and from eBPF programs */
void *(*map_lookup_elem)(struct bpf_map *map, void *key);
long (*map_update_elem)(struct bpf_map *map, void *key, void *value, u64 flags);
long (*map_delete_elem)(struct bpf_map *map, void *key);
long (*map_push_elem)(struct bpf_map *map, void *value, u64 flags);
long (*map_pop_elem)(struct bpf_map *map, void *value);
long (*map_peek_elem)(struct bpf_map *map, void *value);
void *(*map_lookup_percpu_elem)(struct bpf_map *map, void *key, u32 cpu);
int (*map_get_hash)(struct bpf_map *map, u32 hash_buf_size, void *hash_buf);
/* funcs called by prog_array and perf_event_array map */
void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file,
int fd);
/* If need_defer is true, the implementation should guarantee that
* the to-be-put element is still alive before the bpf program, which
* may manipulate it, exists.
*/
void (*map_fd_put_ptr)(struct bpf_map *map, void *ptr, bool need_defer);
int (*map_gen_lookup)(struct bpf_map *map, struct bpf_insn *insn_buf);
u32 (*map_fd_sys_lookup_elem)(void *ptr);
void (*map_seq_show_elem)(struct bpf_map *map, void *key,
struct seq_file *m);
int (*map_check_btf)(struct bpf_map *map,
const struct btf *btf,
const struct btf_type *key_type,
const struct btf_type *value_type);
/* Prog poke tracking helpers. */
int (*map_poke_track)(struct bpf_map *map, struct bpf_prog_aux *aux);
void (*map_poke_untrack)(struct bpf_map *map, struct bpf_prog_aux *aux);
void (*map_poke_run)(struct bpf_map *map, u32 key, struct bpf_prog *old,
struct bpf_prog *new);
/* Direct value access helpers. */
int (*map_direct_value_addr)(const struct bpf_map *map,
u64 *imm, u32 off);
int (*map_direct_value_meta)(const struct bpf_map *map,
u64 imm, u32 *off);
int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
struct poll_table_struct *pts);
unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
unsigned long len, unsigned long pgoff,
unsigned long flags);
/* Functions called by bpf_local_storage maps */
int (*map_local_storage_charge)(struct bpf_local_storage_map *smap,
void *owner, u32 size);
void (*map_local_storage_uncharge)(struct bpf_local_storage_map *smap,
void *owner, u32 size);
struct bpf_local_storage __rcu ** (*map_owner_storage_ptr)(void *owner);
/* Misc helpers.*/
long (*map_redirect)(struct bpf_map *map, u64 key, u64 flags);
/* map_meta_equal must be implemented for maps that can be
* used as an inner map. It is a runtime check to ensure
* an inner map can be inserted to an outer map.
*
* Some properties of the inner map has been used during the
* verification time. When inserting an inner map at the runtime,
* map_meta_equal has to ensure the inserting map has the same
* properties that the verifier has used earlier.
*/
bool (*map_meta_equal)(const struct bpf_map *meta0,
const struct bpf_map *meta1);
int (*map_set_for_each_callback_args)(struct bpf_verifier_env *env,
struct bpf_func_state *caller,
struct bpf_func_state *callee);
long (*map_for_each_callback)(struct bpf_map *map,
bpf_callback_t callback_fn,
void *callback_ctx, u64 flags);
u64 (*map_mem_usage)(const struct bpf_map *map);
/* BTF id of struct allocated by map_alloc */
int *map_btf_id;
/* bpf_iter info used to open a seq_file */
const struct bpf_iter_seq_info *iter_seq_info;
};
enum {
/* Support at most 11 fields in a BTF type */
BTF_FIELDS_MAX = 11,
};
enum btf_field_type {
BPF_SPIN_LOCK = (1 << 0),
BPF_TIMER = (1 << 1),
BPF_KPTR_UNREF = (1 << 2),
BPF_KPTR_REF = (1 << 3),
BPF_KPTR_PERCPU = (1 << 4),
BPF_KPTR = BPF_KPTR_UNREF | BPF_KPTR_REF | BPF_KPTR_PERCPU,
BPF_LIST_HEAD = (1 << 5),
BPF_LIST_NODE = (1 << 6),
BPF_RB_ROOT = (1 << 7),
BPF_RB_NODE = (1 << 8),
BPF_GRAPH_NODE = BPF_RB_NODE | BPF_LIST_NODE,
BPF_GRAPH_ROOT = BPF_RB_ROOT | BPF_LIST_HEAD,
BPF_REFCOUNT = (1 << 9),
BPF_WORKQUEUE = (1 << 10),
BPF_UPTR = (1 << 11),
BPF_RES_SPIN_LOCK = (1 << 12),
BPF_TASK_WORK = (1 << 13),
};
enum bpf_cgroup_storage_type {
BPF_CGROUP_STORAGE_SHARED,
BPF_CGROUP_STORAGE_PERCPU,
__BPF_CGROUP_STORAGE_MAX
#define MAX_BPF_CGROUP_STORAGE_TYPE __BPF_CGROUP_STORAGE_MAX
};
#ifdef CONFIG_CGROUP_BPF
# define for_each_cgroup_storage_type(stype) \
for (stype = 0; stype < MAX_BPF_CGROUP_STORAGE_TYPE; stype++)
#else
# define for_each_cgroup_storage_type(stype) for (; false; )
#endif /* CONFIG_CGROUP_BPF */
typedef void (*btf_dtor_kfunc_t)(void *);
struct btf_field_kptr {
struct btf *btf;
struct module *module;
/* dtor used if btf_is_kernel(btf), otherwise the type is
* program-allocated, dtor is NULL, and __bpf_obj_drop_impl is used
*/
btf_dtor_kfunc_t dtor;
u32 btf_id;
};
struct btf_field_graph_root {
struct btf *btf;
u32 value_btf_id;
u32 node_offset;
struct btf_record *value_rec;
};
struct btf_field {
u32 offset;
u32 size;
enum btf_field_type type;
union {
struct btf_field_kptr kptr;
struct btf_field_graph_root graph_root;
};
};
struct btf_record {
u32 cnt;
u32 field_mask;
int spin_lock_off;
int res_spin_lock_off;
int timer_off;
int wq_off;
int refcount_off;
int task_work_off;
struct btf_field fields[];
};
/* Non-opaque version of bpf_rb_node in uapi/linux/bpf.h */
struct bpf_rb_node_kern {
struct rb_node rb_node;
void *owner;
} __attribute__((aligned(8)));
/* Non-opaque version of bpf_list_node in uapi/linux/bpf.h */
struct bpf_list_node_kern {
struct list_head list_head;
void *owner;
} __attribute__((aligned(8)));
/* 'Ownership' of program-containing map is claimed by the first program
* that is going to use this map or by the first program which FD is
* stored in the map to make sure that all callers and callees have the
* same prog type, JITed flag and xdp_has_frags flag.
*/
struct bpf_map_owner {
enum bpf_prog_type type;
bool jited;
bool xdp_has_frags;
bool sleepable;
u64 storage_cookie[MAX_BPF_CGROUP_STORAGE_TYPE];
const struct btf_type *attach_func_proto;
enum bpf_attach_type expected_attach_type;
};
struct bpf_map {
u8 sha[SHA256_DIGEST_SIZE];
const struct bpf_map_ops *ops;
struct bpf_map *inner_map_meta;
#ifdef CONFIG_SECURITY
void *security;
#endif
enum bpf_map_type map_type;
u32 key_size;
u32 value_size;
u32 max_entries;
u64 map_extra; /* any per-map-type extra fields */
u32 map_flags;
u32 id;
struct btf_record *record;
int numa_node;
u32 btf_key_type_id;
u32 btf_value_type_id;
u32 btf_vmlinux_value_type_id;
struct btf *btf;
#ifdef CONFIG_MEMCG
struct obj_cgroup *objcg;
#endif
char name[BPF_OBJ_NAME_LEN];
struct mutex freeze_mutex;
atomic64_t refcnt;
atomic64_t usercnt;
/* rcu is used before freeing and work is only used during freeing */
union {
struct work_struct work;
struct rcu_head rcu;
};
atomic64_t writecnt;
spinlock_t owner_lock;
struct bpf_map_owner *owner;
bool bypass_spec_v1;
bool frozen; /* write-once; write-protected by freeze_mutex */
bool free_after_mult_rcu_gp;
bool free_after_rcu_gp;
atomic64_t sleepable_refcnt;
s64 __percpu *elem_count;
u64 cookie; /* write-once */
char *excl_prog_sha;
};
static inline const char *btf_field_type_name(enum btf_field_type type)
{
switch (type) {
case BPF_SPIN_LOCK:
return "bpf_spin_lock";
case BPF_RES_SPIN_LOCK:
return "bpf_res_spin_lock";
case BPF_TIMER:
return "bpf_timer";
case BPF_WORKQUEUE:
return "bpf_wq";
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
return "kptr";
case BPF_KPTR_PERCPU:
return "percpu_kptr";
case BPF_UPTR:
return "uptr";
case BPF_LIST_HEAD:
return "bpf_list_head";
case BPF_LIST_NODE:
return "bpf_list_node";
case BPF_RB_ROOT:
return "bpf_rb_root";
case BPF_RB_NODE:
return "bpf_rb_node";
case BPF_REFCOUNT:
return "bpf_refcount";
case BPF_TASK_WORK:
return "bpf_task_work";
default:
WARN_ON_ONCE(1);
return "unknown";
}
}
#if IS_ENABLED(CONFIG_DEBUG_KERNEL)
#define BPF_WARN_ONCE(cond, format...) WARN_ONCE(cond, format)
#else
#define BPF_WARN_ONCE(cond, format...) BUILD_BUG_ON_INVALID(cond)
#endif
static inline u32 btf_field_type_size(enum btf_field_type type)
{
switch (type) {
case BPF_SPIN_LOCK:
return sizeof(struct bpf_spin_lock);
case BPF_RES_SPIN_LOCK:
return sizeof(struct bpf_res_spin_lock);
case BPF_TIMER:
return sizeof(struct bpf_timer);
case BPF_WORKQUEUE:
return sizeof(struct bpf_wq);
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
case BPF_KPTR_PERCPU:
case BPF_UPTR:
return sizeof(u64);
case BPF_LIST_HEAD:
return sizeof(struct bpf_list_head);
case BPF_LIST_NODE:
return sizeof(struct bpf_list_node);
case BPF_RB_ROOT:
return sizeof(struct bpf_rb_root);
case BPF_RB_NODE:
return sizeof(struct bpf_rb_node);
case BPF_REFCOUNT:
return sizeof(struct bpf_refcount);
case BPF_TASK_WORK:
return sizeof(struct bpf_task_work);
default:
WARN_ON_ONCE(1);
return 0;
}
}
static inline u32 btf_field_type_align(enum btf_field_type type)
{
switch (type) {
case BPF_SPIN_LOCK:
return __alignof__(struct bpf_spin_lock);
case BPF_RES_SPIN_LOCK:
return __alignof__(struct bpf_res_spin_lock);
case BPF_TIMER:
return __alignof__(struct bpf_timer);
case BPF_WORKQUEUE:
return __alignof__(struct bpf_wq);
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
case BPF_KPTR_PERCPU:
case BPF_UPTR:
return __alignof__(u64);
case BPF_LIST_HEAD:
return __alignof__(struct bpf_list_head);
case BPF_LIST_NODE:
return __alignof__(struct bpf_list_node);
case BPF_RB_ROOT:
return __alignof__(struct bpf_rb_root);
case BPF_RB_NODE:
return __alignof__(struct bpf_rb_node);
case BPF_REFCOUNT:
return __alignof__(struct bpf_refcount);
case BPF_TASK_WORK:
return __alignof__(struct bpf_task_work);
default:
WARN_ON_ONCE(1);
return 0;
}
}
static inline void bpf_obj_init_field(const struct btf_field *field, void *addr)
{
memset(addr, 0, field->size);
switch (field->type) {
case BPF_REFCOUNT:
refcount_set((refcount_t *)addr, 1);
break;
case BPF_RB_NODE:
RB_CLEAR_NODE((struct rb_node *)addr);
break;
case BPF_LIST_HEAD:
case BPF_LIST_NODE:
INIT_LIST_HEAD((struct list_head *)addr);
break;
case BPF_RB_ROOT:
/* RB_ROOT_CACHED 0-inits, no need to do anything after memset */
case BPF_SPIN_LOCK:
case BPF_RES_SPIN_LOCK:
case BPF_TIMER:
case BPF_WORKQUEUE:
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
case BPF_KPTR_PERCPU:
case BPF_UPTR:
case BPF_TASK_WORK:
break;
default:
WARN_ON_ONCE(1);
return;
}
}
static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_field_type type)
{
if (IS_ERR_OR_NULL(rec))
return false;
return rec->field_mask & type;
}
static inline void bpf_obj_init(const struct btf_record *rec, void *obj)
{
int i;
if (IS_ERR_OR_NULL(rec))
return;
for (i = 0; i < rec->cnt; i++)
bpf_obj_init_field(&rec->fields[i], obj + rec->fields[i].offset);
}
/* 'dst' must be a temporary buffer and should not point to memory that is being
* used in parallel by a bpf program or bpf syscall, otherwise the access from
* the bpf program or bpf syscall may be corrupted by the reinitialization,
* leading to weird problems. Even 'dst' is newly-allocated from bpf memory
* allocator, it is still possible for 'dst' to be used in parallel by a bpf
* program or bpf syscall.
*/
static inline void check_and_init_map_value(struct bpf_map *map, void *dst)
{
bpf_obj_init(map->record, dst);
}
/* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
* forced to use 'long' read/writes to try to atomically copy long counters.
* Best-effort only. No barriers here, since it _will_ race with concurrent
* updates from BPF programs. Called from bpf syscall and mostly used with
* size 8 or 16 bytes, so ask compiler to inline it.
*/
static inline void bpf_long_memcpy(void *dst, const void *src, u32 size)
{
const long *lsrc = src;
long *ldst = dst;
size /= sizeof(long);
while (size--)
data_race(*ldst++ = *lsrc++);
}
/* copy everything but bpf_spin_lock, bpf_timer, and kptrs. There could be one of each. */
static inline void bpf_obj_memcpy(struct btf_record *rec,
void *dst, void *src, u32 size,
bool long_memcpy)
{
u32 curr_off = 0;
int i;
if (IS_ERR_OR_NULL(rec)) {
if (long_memcpy)
bpf_long_memcpy(dst, src, round_up(size, 8));
else
memcpy(dst, src, size);
return;
}
for (i = 0; i < rec->cnt; i++) {
u32 next_off = rec->fields[i].offset;
u32 sz = next_off - curr_off;
memcpy(dst + curr_off, src + curr_off, sz);
curr_off += rec->fields[i].size + sz;
}
memcpy(dst + curr_off, src + curr_off, size - curr_off);
}
static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
{
bpf_obj_memcpy(map->record, dst, src, map->value_size, false);
}
static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src)
{
bpf_obj_memcpy(map->record, dst, src, map->value_size, true);
}
static inline void bpf_obj_swap_uptrs(const struct btf_record *rec, void *dst, void *src)
{
unsigned long *src_uptr, *dst_uptr;
const struct btf_field *field;
int i;
if (!btf_record_has_field(rec, BPF_UPTR))
return;
for (i = 0, field = rec->fields; i < rec->cnt; i++, field++) {
if (field->type != BPF_UPTR)
continue;
src_uptr = src + field->offset;
dst_uptr = dst + field->offset;
swap(*src_uptr, *dst_uptr);
}
}
static inline void bpf_obj_memzero(struct btf_record *rec, void *dst, u32 size)
{
u32 curr_off = 0;
int i;
if (IS_ERR_OR_NULL(rec)) {
memset(dst, 0, size);
return;
}
for (i = 0; i < rec->cnt; i++) {
u32 next_off = rec->fields[i].offset;
u32 sz = next_off - curr_off;
memset(dst + curr_off, 0, sz);
curr_off += rec->fields[i].size + sz;
}
memset(dst + curr_off, 0, size - curr_off);
}
static inline void zero_map_value(struct bpf_map *map, void *dst)
{
bpf_obj_memzero(map->record, dst, map->value_size);
}
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
bool lock_src);
void bpf_timer_cancel_and_free(void *timer);
void bpf_wq_cancel_and_free(void *timer);
void bpf_task_work_cancel_and_free(void *timer);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock);
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock);
u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);
u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);
int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
struct bpf_offload_dev;
struct bpf_offloaded_map;
struct bpf_map_dev_ops {
int (*map_get_next_key)(struct bpf_offloaded_map *map,
void *key, void *next_key);
int (*map_lookup_elem)(struct bpf_offloaded_map *map,
void *key, void *value);
int (*map_update_elem)(struct bpf_offloaded_map *map,
void *key, void *value, u64 flags);
int (*map_delete_elem)(struct bpf_offloaded_map *map, void *key);
};
struct bpf_offloaded_map {
struct bpf_map map;
struct net_device *netdev;
const struct bpf_map_dev_ops *dev_ops;
void *dev_priv;
struct list_head offloads;
};
static inline struct bpf_offloaded_map *map_to_offmap(struct bpf_map *map)
{
return container_of(map, struct bpf_offloaded_map, map);
}
static inline bool bpf_map_offload_neutral(const struct bpf_map *map)
{
return map->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY;
}
static inline bool bpf_map_support_seq_show(const struct bpf_map *map)
{
return (map->btf_value_type_id || map->btf_vmlinux_value_type_id) &&
map->ops->map_seq_show_elem;
}
int map_check_no_btf(struct bpf_map *map,
const struct btf *btf,
const struct btf_type *key_type,
const struct btf_type *value_type);
bool bpf_map_meta_equal(const struct bpf_map *meta0,
const struct bpf_map *meta1);
static inline bool bpf_map_has_internal_structs(struct bpf_map *map)
{
return btf_record_has_field(map->record, BPF_TIMER | BPF_WORKQUEUE | BPF_TASK_WORK);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,
struct bpf_dynptr *ptr__uninit);
#if defined(CONFIG_MMU) && defined(CONFIG_64BIT)
void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
u64 flags);
void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt);
#else
static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
int node_id, u64 flags)
{
return NULL;
}
static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt)
{
}
#endif
extern const struct bpf_map_ops bpf_map_offload_ops;
/* bpf_type_flag contains a set of flags that are applicable to the values of
* arg_type, ret_type and reg_type. For example, a pointer value may be null,
* or a memory is read-only. We classify types into two categories: base types
* and extended types. Extended types are base types combined with a type flag.
*
* Currently there are no more than 32 base types in arg_type, ret_type and
* reg_types.
*/
#define BPF_BASE_TYPE_BITS 8
enum bpf_type_flag {
/* PTR may be NULL. */
PTR_MAYBE_NULL = BIT(0 + BPF_BASE_TYPE_BITS),
/* MEM is read-only. When applied on bpf_arg, it indicates the arg is
* compatible with both mutable and immutable memory.
*/
MEM_RDONLY = BIT(1 + BPF_BASE_TYPE_BITS),
/* MEM points to BPF ring buffer reservation. */
MEM_RINGBUF = BIT(2 + BPF_BASE_TYPE_BITS),
/* MEM is in user address space. */
MEM_USER = BIT(3 + BPF_BASE_TYPE_BITS),
/* MEM is a percpu memory. MEM_PERCPU tags PTR_TO_BTF_ID. When tagged
* with MEM_PERCPU, PTR_TO_BTF_ID _cannot_ be directly accessed. In
* order to drop this tag, it must be passed into bpf_per_cpu_ptr()
* or bpf_this_cpu_ptr(), which will return the pointer corresponding
* to the specified cpu.
*/
MEM_PERCPU = BIT(4 + BPF_BASE_TYPE_BITS),
/* Indicates that the argument will be released. */
OBJ_RELEASE = BIT(5 + BPF_BASE_TYPE_BITS),
/* PTR is not trusted. This is only used with PTR_TO_BTF_ID, to mark
* unreferenced and referenced kptr loaded from map value using a load
* instruction, so that they can only be dereferenced but not escape the
* BPF program into the kernel (i.e. cannot be passed as arguments to
* kfunc or bpf helpers).
*/
PTR_UNTRUSTED = BIT(6 + BPF_BASE_TYPE_BITS),
/* MEM can be uninitialized. */
MEM_UNINIT = BIT(7 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to memory local to the bpf program. */
DYNPTR_TYPE_LOCAL = BIT(8 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to a kernel-produced ringbuf record. */
DYNPTR_TYPE_RINGBUF = BIT(9 + BPF_BASE_TYPE_BITS),
/* Size is known at compile time. */
MEM_FIXED_SIZE = BIT(10 + BPF_BASE_TYPE_BITS),
/* MEM is of an allocated object of type in program BTF. This is used to
* tag PTR_TO_BTF_ID allocated using bpf_obj_new.
*/
MEM_ALLOC = BIT(11 + BPF_BASE_TYPE_BITS),
/* PTR was passed from the kernel in a trusted context, and may be
* passed to kfuncs or BPF helper functions.
* Confusingly, this is _not_ the opposite of PTR_UNTRUSTED above.
* PTR_UNTRUSTED refers to a kptr that was read directly from a map
* without invoking bpf_kptr_xchg(). What we really need to know is
* whether a pointer is safe to pass to a kfunc or BPF helper function.
* While PTR_UNTRUSTED pointers are unsafe to pass to kfuncs and BPF
* helpers, they do not cover all possible instances of unsafe
* pointers. For example, a pointer that was obtained from walking a
* struct will _not_ get the PTR_UNTRUSTED type modifier, despite the
* fact that it may be NULL, invalid, etc. This is due to backwards
* compatibility requirements, as this was the behavior that was first
* introduced when kptrs were added. The behavior is now considered
* deprecated, and PTR_UNTRUSTED will eventually be removed.
*
* PTR_TRUSTED, on the other hand, is a pointer that the kernel
* guarantees to be valid and safe to pass to kfuncs and BPF helpers.
* For example, pointers passed to tracepoint arguments are considered
* PTR_TRUSTED, as are pointers that are passed to struct_ops
* callbacks. As alluded to above, pointers that are obtained from
* walking PTR_TRUSTED pointers are _not_ trusted. For example, if a
* struct task_struct *task is PTR_TRUSTED, then accessing
* task->last_wakee will lose the PTR_TRUSTED modifier when it's stored
* in a BPF register. Similarly, pointers passed to certain programs
* types such as kretprobes are not guaranteed to be valid, as they may
* for example contain an object that was recently freed.
*/
PTR_TRUSTED = BIT(12 + BPF_BASE_TYPE_BITS),
/* MEM is tagged with rcu and memory access needs rcu_read_lock protection. */
MEM_RCU = BIT(13 + BPF_BASE_TYPE_BITS),
/* Used to tag PTR_TO_BTF_ID | MEM_ALLOC references which are non-owning.
* Currently only valid for linked-list and rbtree nodes. If the nodes
* have a bpf_refcount_field, they must be tagged MEM_RCU as well.
*/
NON_OWN_REF = BIT(14 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to sk_buff */
DYNPTR_TYPE_SKB = BIT(15 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to xdp_buff */
DYNPTR_TYPE_XDP = BIT(16 + BPF_BASE_TYPE_BITS),
/* Memory must be aligned on some architectures, used in combination with
* MEM_FIXED_SIZE.
*/
MEM_ALIGNED = BIT(17 + BPF_BASE_TYPE_BITS),
/* MEM is being written to, often combined with MEM_UNINIT. Non-presence
* of MEM_WRITE means that MEM is only being read. MEM_WRITE without the
* MEM_UNINIT means that memory needs to be initialized since it is also
* read.
*/
MEM_WRITE = BIT(18 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to skb_metadata_end()-skb_metadata_len() */
DYNPTR_TYPE_SKB_META = BIT(19 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to file */
DYNPTR_TYPE_FILE = BIT(20 + BPF_BASE_TYPE_BITS),
__BPF_TYPE_FLAG_MAX,
__BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1,
};
#define DYNPTR_TYPE_FLAG_MASK (DYNPTR_TYPE_LOCAL | DYNPTR_TYPE_RINGBUF | DYNPTR_TYPE_SKB \
| DYNPTR_TYPE_XDP | DYNPTR_TYPE_SKB_META | DYNPTR_TYPE_FILE)
/* Max number of base types. */
#define BPF_BASE_TYPE_LIMIT (1UL << BPF_BASE_TYPE_BITS)
/* Max number of all types. */
#define BPF_TYPE_LIMIT (__BPF_TYPE_LAST_FLAG | (__BPF_TYPE_LAST_FLAG - 1))
/* function argument constraints */
enum bpf_arg_type {
ARG_DONTCARE = 0, /* unused argument in helper function */
/* the following constraints used to prototype
* bpf_map_lookup/update/delete_elem() functions
*/
ARG_CONST_MAP_PTR, /* const argument used as pointer to bpf_map */
ARG_PTR_TO_MAP_KEY, /* pointer to stack used as map key */
ARG_PTR_TO_MAP_VALUE, /* pointer to stack used as map value */
/* Used to prototype bpf_memcmp() and other functions that access data
* on eBPF program stack
*/
ARG_PTR_TO_MEM, /* pointer to valid memory (stack, packet, map value) */
ARG_PTR_TO_ARENA,
ARG_CONST_SIZE, /* number of bytes accessed from memory */
ARG_CONST_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */
ARG_PTR_TO_CTX, /* pointer to context */
ARG_ANYTHING, /* any (initialized) argument is ok */
ARG_PTR_TO_SPIN_LOCK, /* pointer to bpf_spin_lock */
ARG_PTR_TO_SOCK_COMMON, /* pointer to sock_common */
ARG_PTR_TO_SOCKET, /* pointer to bpf_sock (fullsock) */
ARG_PTR_TO_BTF_ID, /* pointer to in-kernel struct */
ARG_PTR_TO_RINGBUF_MEM, /* pointer to dynamically reserved ringbuf memory */
ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */
ARG_PTR_TO_BTF_ID_SOCK_COMMON, /* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */
ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */
ARG_PTR_TO_FUNC, /* pointer to a bpf program function */
ARG_PTR_TO_STACK, /* pointer to stack */
ARG_PTR_TO_CONST_STR, /* pointer to a null terminated read-only string */
ARG_PTR_TO_TIMER, /* pointer to bpf_timer */
ARG_KPTR_XCHG_DEST, /* pointer to destination that kptrs are bpf_kptr_xchg'd into */
ARG_PTR_TO_DYNPTR, /* pointer to bpf_dynptr. See bpf_type_flag for dynptr type */
__BPF_ARG_TYPE_MAX,
/* Extended arg_types. */
ARG_PTR_TO_MAP_VALUE_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_MAP_VALUE,
ARG_PTR_TO_MEM_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_MEM,
ARG_PTR_TO_CTX_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_CTX,
ARG_PTR_TO_SOCKET_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_SOCKET,
ARG_PTR_TO_STACK_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_STACK,
ARG_PTR_TO_BTF_ID_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_BTF_ID,
/* Pointer to memory does not need to be initialized, since helper function
* fills all bytes or clears them in error case.
*/
ARG_PTR_TO_UNINIT_MEM = MEM_UNINIT | MEM_WRITE | ARG_PTR_TO_MEM,
/* Pointer to valid memory of size known at compile time. */
ARG_PTR_TO_FIXED_SIZE_MEM = MEM_FIXED_SIZE | ARG_PTR_TO_MEM,
/* This must be the last entry. Its purpose is to ensure the enum is
* wide enough to hold the higher bits reserved for bpf_type_flag.
*/
__BPF_ARG_TYPE_LIMIT = BPF_TYPE_LIMIT,
};
static_assert(__BPF_ARG_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
/* type of values returned from helper functions */
enum bpf_return_type {
RET_INTEGER, /* function returns integer */
RET_VOID, /* function doesn't return anything */
RET_PTR_TO_MAP_VALUE, /* returns a pointer to map elem value */
RET_PTR_TO_SOCKET, /* returns a pointer to a socket */
RET_PTR_TO_TCP_SOCK, /* returns a pointer to a tcp_sock */
RET_PTR_TO_SOCK_COMMON, /* returns a pointer to a sock_common */
RET_PTR_TO_MEM, /* returns a pointer to memory */
RET_PTR_TO_MEM_OR_BTF_ID, /* returns a pointer to a valid memory or a btf_id */
RET_PTR_TO_BTF_ID, /* returns a pointer to a btf_id */
__BPF_RET_TYPE_MAX,
/* Extended ret_types. */
RET_PTR_TO_MAP_VALUE_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_MAP_VALUE,
RET_PTR_TO_SOCKET_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_SOCKET,
RET_PTR_TO_TCP_SOCK_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_TCP_SOCK,
RET_PTR_TO_SOCK_COMMON_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_SOCK_COMMON,
RET_PTR_TO_RINGBUF_MEM_OR_NULL = PTR_MAYBE_NULL | MEM_RINGBUF | RET_PTR_TO_MEM,
RET_PTR_TO_DYNPTR_MEM_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_MEM,
RET_PTR_TO_BTF_ID_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_BTF_ID,
RET_PTR_TO_BTF_ID_TRUSTED = PTR_TRUSTED | RET_PTR_TO_BTF_ID,
/* This must be the last entry. Its purpose is to ensure the enum is
* wide enough to hold the higher bits reserved for bpf_type_flag.
*/
__BPF_RET_TYPE_LIMIT = BPF_TYPE_LIMIT,
};
static_assert(__BPF_RET_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
/* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs
* to in-kernel helper functions and for adjusting imm32 field in BPF_CALL
* instructions after verifying
*/
struct bpf_func_proto {
u64 (*func)(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
bool gpl_only;
bool pkt_access;
bool might_sleep;
/* set to true if helper follows contract for llvm
* attribute bpf_fastcall:
* - void functions do not scratch r0
* - functions taking N arguments scratch only registers r1-rN
*/
bool allow_fastcall;
enum bpf_return_type ret_type;
union {
struct {
enum bpf_arg_type arg1_type;
enum bpf_arg_type arg2_type;
enum bpf_arg_type arg3_type;
enum bpf_arg_type arg4_type;
enum bpf_arg_type arg5_type;
};
enum bpf_arg_type arg_type[5];
};
union {
struct {
u32 *arg1_btf_id;
u32 *arg2_btf_id;
u32 *arg3_btf_id;
u32 *arg4_btf_id;
u32 *arg5_btf_id;
};
u32 *arg_btf_id[5];
struct {
size_t arg1_size;
size_t arg2_size;
size_t arg3_size;
size_t arg4_size;
size_t arg5_size;
};
size_t arg_size[5];
};
int *ret_btf_id; /* return value btf_id */
bool (*allowed)(const struct bpf_prog *prog);
};
/* bpf_context is intentionally undefined structure. Pointer to bpf_context is
* the first argument to eBPF programs.
* For socket filters: 'struct bpf_context *' == 'struct sk_buff *'
*/
struct bpf_context;
enum bpf_access_type {
BPF_READ = 1,
BPF_WRITE = 2
};
/* types of values stored in eBPF registers */
/* Pointer types represent:
* pointer
* pointer + imm
* pointer + (u16) var
* pointer + (u16) var + imm
* if (range > 0) then [ptr, ptr + range - off) is safe to access
* if (id > 0) means that some 'var' was added
* if (off > 0) means that 'imm' was added
*/
enum bpf_reg_type {
NOT_INIT = 0, /* nothing was written into register */
SCALAR_VALUE, /* reg doesn't contain a valid pointer */
PTR_TO_CTX, /* reg points to bpf_context */
CONST_PTR_TO_MAP, /* reg points to struct bpf_map */
PTR_TO_MAP_VALUE, /* reg points to map element value */
PTR_TO_MAP_KEY, /* reg points to a map element key */
PTR_TO_STACK, /* reg == frame_pointer + offset */
PTR_TO_PACKET_META, /* skb->data - meta_len */
PTR_TO_PACKET, /* reg points to skb->data */
PTR_TO_PACKET_END, /* skb->data + headlen */
PTR_TO_FLOW_KEYS, /* reg points to bpf_flow_keys */
PTR_TO_SOCKET, /* reg points to struct bpf_sock */
PTR_TO_SOCK_COMMON, /* reg points to sock_common */
PTR_TO_TCP_SOCK, /* reg points to struct tcp_sock */
PTR_TO_TP_BUFFER, /* reg points to a writable raw tp's buffer */
PTR_TO_XDP_SOCK, /* reg points to struct xdp_sock */
/* PTR_TO_BTF_ID points to a kernel struct that does not need
* to be null checked by the BPF program. This does not imply the
* pointer is _not_ null and in practice this can easily be a null
* pointer when reading pointer chains. The assumption is program
* context will handle null pointer dereference typically via fault
* handling. The verifier must keep this in mind and can make no
* assumptions about null or non-null when doing branch analysis.
* Further, when passed into helpers the helpers can not, without
* additional context, assume the value is non-null.
*/
PTR_TO_BTF_ID,
PTR_TO_MEM, /* reg points to valid memory region */
PTR_TO_ARENA,
PTR_TO_BUF, /* reg points to a read/write buffer */
PTR_TO_FUNC, /* reg points to a bpf program function */
PTR_TO_INSN, /* reg points to a bpf program instruction */
CONST_PTR_TO_DYNPTR, /* reg points to a const struct bpf_dynptr */
__BPF_REG_TYPE_MAX,
/* Extended reg_types. */
PTR_TO_MAP_VALUE_OR_NULL = PTR_MAYBE_NULL | PTR_TO_MAP_VALUE,
PTR_TO_SOCKET_OR_NULL = PTR_MAYBE_NULL | PTR_TO_SOCKET,
PTR_TO_SOCK_COMMON_OR_NULL = PTR_MAYBE_NULL | PTR_TO_SOCK_COMMON,
PTR_TO_TCP_SOCK_OR_NULL = PTR_MAYBE_NULL | PTR_TO_TCP_SOCK,
/* PTR_TO_BTF_ID_OR_NULL points to a kernel struct that has not
* been checked for null. Used primarily to inform the verifier
* an explicit null check is required for this struct.
*/
PTR_TO_BTF_ID_OR_NULL = PTR_MAYBE_NULL | PTR_TO_BTF_ID,
/* This must be the last entry. Its purpose is to ensure the enum is
* wide enough to hold the higher bits reserved for bpf_type_flag.
*/
__BPF_REG_TYPE_LIMIT = BPF_TYPE_LIMIT,
};
static_assert(__BPF_REG_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
/* The information passed from prog-specific *_is_valid_access
* back to the verifier.
*/
struct bpf_insn_access_aux {
enum bpf_reg_type reg_type;
bool is_ldsx;
union {
int ctx_field_size;
struct {
struct btf *btf;
u32 btf_id;
u32 ref_obj_id;
};
};
struct bpf_verifier_log *log; /* for verbose logs */
bool is_retval; /* is accessing function return value ? */
};
static inline void
bpf_ctx_record_field_size(struct bpf_insn_access_aux *aux, u32 size)
{
aux->ctx_field_size = size;
}
static bool bpf_is_ldimm64(const struct bpf_insn *insn)
{
return insn->code == (BPF_LD | BPF_IMM | BPF_DW);
}
static inline bool bpf_pseudo_func(const struct bpf_insn *insn)
{
return bpf_is_ldimm64(insn) && insn->src_reg == BPF_PSEUDO_FUNC;
}
/* Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an
* atomic load or store, and false if it is a read-modify-write instruction.
*/
static inline bool
bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn)
{
switch (atomic_insn->imm) {
case BPF_LOAD_ACQ:
case BPF_STORE_REL:
return true;
default:
return false;
}
}
struct bpf_prog_ops {
int (*test_run)(struct bpf_prog *prog, const union bpf_attr *kattr,
union bpf_attr __user *uattr);
};
struct bpf_reg_state;
struct bpf_verifier_ops {
/* return eBPF function prototype for verification */
const struct bpf_func_proto *
(*get_func_proto)(enum bpf_func_id func_id,
const struct bpf_prog *prog);
/* return true if 'size' wide access at offset 'off' within bpf_context
* with 'type' (read or write) is allowed
*/
bool (*is_valid_access)(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info);
int (*gen_prologue)(struct bpf_insn *insn, bool direct_write,
const struct bpf_prog *prog);
int (*gen_epilogue)(struct bpf_insn *insn, const struct bpf_prog *prog,
s16 ctx_stack_off);
int (*gen_ld_abs)(const struct bpf_insn *orig,
struct bpf_insn *insn_buf);
u32 (*convert_ctx_access)(enum bpf_access_type type,
const struct bpf_insn *src,
struct bpf_insn *dst,
struct bpf_prog *prog, u32 *target_size);
int (*btf_struct_access)(struct bpf_verifier_log *log,
const struct bpf_reg_state *reg,
int off, int size);
};
struct bpf_prog_offload_ops {
/* verifier basic callbacks */
int (*insn_hook)(struct bpf_verifier_env *env,
int insn_idx, int prev_insn_idx);
int (*finalize)(struct bpf_verifier_env *env);
/* verifier optimization callbacks (called after .finalize) */
int (*replace_insn)(struct bpf_verifier_env *env, u32 off,
struct bpf_insn *insn);
int (*remove_insns)(struct bpf_verifier_env *env, u32 off, u32 cnt);
/* program management callbacks */
int (*prepare)(struct bpf_prog *prog);
int (*translate)(struct bpf_prog *prog);
void (*destroy)(struct bpf_prog *prog);
};
struct bpf_prog_offload {
struct bpf_prog *prog;
struct net_device *netdev;
struct bpf_offload_dev *offdev;
void *dev_priv;
struct list_head offloads;
bool dev_state;
bool opt_failed;
void *jited_image;
u32 jited_len;
};
/* The longest tracepoint has 12 args.
* See include/trace/bpf_probe.h
*/
#define MAX_BPF_FUNC_ARGS 12
/* The maximum number of arguments passed through registers
* a single function may have.
*/
#define MAX_BPF_FUNC_REG_ARGS 5
/* The argument is a structure or a union. */
#define BTF_FMODEL_STRUCT_ARG BIT(0)
/* The argument is signed. */
#define BTF_FMODEL_SIGNED_ARG BIT(1)
struct btf_func_model {
u8 ret_size;
u8 ret_flags;
u8 nr_args;
u8 arg_size[MAX_BPF_FUNC_ARGS];
u8 arg_flags[MAX_BPF_FUNC_ARGS];
};
/* Restore arguments before returning from trampoline to let original function
* continue executing. This flag is used for fentry progs when there are no
* fexit progs.
*/
#define BPF_TRAMP_F_RESTORE_REGS BIT(0)
/* Call original function after fentry progs, but before fexit progs.
* Makes sense for fentry/fexit, normal calls and indirect calls.
*/
#define BPF_TRAMP_F_CALL_ORIG BIT(1)
/* Skip current frame and return to parent. Makes sense for fentry/fexit
* programs only. Should not be used with normal calls and indirect calls.
*/
#define BPF_TRAMP_F_SKIP_FRAME BIT(2)
/* Store IP address of the caller on the trampoline stack,
* so it's available for trampoline's programs.
*/
#define BPF_TRAMP_F_IP_ARG BIT(3)
/* Return the return value of fentry prog. Only used by bpf_struct_ops. */
#define BPF_TRAMP_F_RET_FENTRY_RET BIT(4)
/* Get original function from stack instead of from provided direct address.
* Makes sense for trampolines with fexit or fmod_ret programs.
*/
#define BPF_TRAMP_F_ORIG_STACK BIT(5)
/* This trampoline is on a function with another ftrace_ops with IPMODIFY,
* e.g., a live patch. This flag is set and cleared by ftrace call backs,
*/
#define BPF_TRAMP_F_SHARE_IPMODIFY BIT(6)
/* Indicate that current trampoline is in a tail call context. Then, it has to
* cache and restore tail_call_cnt to avoid infinite tail call loop.
*/
#define BPF_TRAMP_F_TAIL_CALL_CTX BIT(7)
/*
* Indicate the trampoline should be suitable to receive indirect calls;
* without this indirectly calling the generated code can result in #UD/#CP,
* depending on the CFI options.
*
* Used by bpf_struct_ops.
*
* Incompatible with FENTRY usage, overloads @func_addr argument.
*/
#define BPF_TRAMP_F_INDIRECT BIT(8)
/* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50
* bytes on x86.
*/
enum {
#if defined(__s390x__)
BPF_MAX_TRAMP_LINKS = 27,
#else
BPF_MAX_TRAMP_LINKS = 38,
#endif
};
#define BPF_TRAMP_COOKIE_INDEX_SHIFT 8
#define BPF_TRAMP_IS_RETURN_SHIFT 63
struct bpf_tramp_links {
struct bpf_tramp_link *links[BPF_MAX_TRAMP_LINKS];
int nr_links;
};
struct bpf_tramp_run_ctx;
/* Different use cases for BPF trampoline:
* 1. replace nop at the function entry (kprobe equivalent)
* flags = BPF_TRAMP_F_RESTORE_REGS
* fentry = a set of programs to run before returning from trampoline
*
* 2. replace nop at the function entry (kprobe + kretprobe equivalent)
* flags = BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_SKIP_FRAME
* orig_call = fentry_ip + MCOUNT_INSN_SIZE
* fentry = a set of program to run before calling original function
* fexit = a set of program to run after original function
*
* 3. replace direct call instruction anywhere in the function body
* or assign a function pointer for indirect call (like tcp_congestion_ops->cong_avoid)
* With flags = 0
* fentry = a set of programs to run before returning from trampoline
* With flags = BPF_TRAMP_F_CALL_ORIG
* orig_call = original callback addr or direct function addr
* fentry = a set of program to run before calling original function
* fexit = a set of program to run after original function
*/
struct bpf_tramp_image;
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
void *func_addr);
void *arch_alloc_bpf_trampoline(unsigned int size);
void arch_free_bpf_trampoline(void *image, unsigned int size);
int __must_check arch_protect_bpf_trampoline(void *image, unsigned int size);
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr);
u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog,
struct bpf_tramp_run_ctx *run_ctx);
void notrace __bpf_prog_exit_sleepable_recur(struct bpf_prog *prog, u64 start,
struct bpf_tramp_run_ctx *run_ctx);
void notrace __bpf_tramp_enter(struct bpf_tramp_image *tr);
void notrace __bpf_tramp_exit(struct bpf_tramp_image *tr);
typedef u64 (*bpf_trampoline_enter_t)(struct bpf_prog *prog,
struct bpf_tramp_run_ctx *run_ctx);
typedef void (*bpf_trampoline_exit_t)(struct bpf_prog *prog, u64 start,
struct bpf_tramp_run_ctx *run_ctx);
bpf_trampoline_enter_t bpf_trampoline_enter(const struct bpf_prog *prog);
bpf_trampoline_exit_t bpf_trampoline_exit(const struct bpf_prog *prog);
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP
static inline bool bpf_trampoline_use_jmp(u64 flags)
{
return flags & BPF_TRAMP_F_CALL_ORIG && !(flags & BPF_TRAMP_F_SKIP_FRAME);
}
#else
static inline bool bpf_trampoline_use_jmp(u64 flags)
{
return false;
}
#endif
struct bpf_ksym {
unsigned long start;
unsigned long end;
char name[KSYM_NAME_LEN];
struct list_head lnode;
struct latch_tree_node tnode;
bool prog;
u32 fp_start;
u32 fp_end;
};
enum bpf_tramp_prog_type {
BPF_TRAMP_FENTRY,
BPF_TRAMP_FEXIT,
BPF_TRAMP_MODIFY_RETURN,
BPF_TRAMP_MAX,
BPF_TRAMP_REPLACE, /* more than MAX */
BPF_TRAMP_FSESSION,
};
struct bpf_tramp_image {
void *image;
int size;
struct bpf_ksym ksym;
struct percpu_ref pcref;
void *ip_after_call;
void *ip_epilogue;
union {
struct rcu_head rcu;
struct work_struct work;
};
};
struct bpf_trampoline {
/* hlist for trampoline_key_table */
struct hlist_node hlist_key;
/* hlist for trampoline_ip_table */
struct hlist_node hlist_ip;
struct ftrace_ops *fops;
/* serializes access to fields of this trampoline */
struct mutex mutex;
refcount_t refcnt;
u32 flags;
u64 key;
unsigned long ip;
struct {
struct btf_func_model model;
void *addr;
bool ftrace_managed;
} func;
/* if !NULL this is BPF_PROG_TYPE_EXT program that extends another BPF
* program by replacing one of its functions. func.addr is the address
* of the function it replaced.
*/
struct bpf_prog *extension_prog;
/* list of BPF programs using this trampoline */
struct hlist_head progs_hlist[BPF_TRAMP_MAX];
/* Number of attached programs. A counter per kind. */
int progs_cnt[BPF_TRAMP_MAX];
/* Executable image of trampoline */
struct bpf_tramp_image *cur_image;
};
struct bpf_attach_target_info {
struct btf_func_model fmodel;
long tgt_addr;
struct module *tgt_mod;
const char *tgt_name;
const struct btf_type *tgt_type;
};
#define BPF_DISPATCHER_MAX 48 /* Fits in 2048B */
struct bpf_dispatcher_prog {
struct bpf_prog *prog;
refcount_t users;
};
struct bpf_dispatcher {
/* dispatcher mutex */
struct mutex mutex;
void *func;
struct bpf_dispatcher_prog progs[BPF_DISPATCHER_MAX];
int num_progs;
void *image;
void *rw_image;
u32 image_off;
struct bpf_ksym ksym;
#ifdef CONFIG_HAVE_STATIC_CALL
struct static_call_key *sc_key;
void *sc_tramp;
#endif
};
#ifndef __bpfcall
#define __bpfcall __nocfi
#endif
static __always_inline __bpfcall unsigned int bpf_dispatcher_nop_func(
const void *ctx,
const struct bpf_insn *insnsi,
bpf_func_t bpf_func)
{
return bpf_func(ctx, insnsi);
}
/* the implementation of the opaque uapi struct bpf_dynptr */
struct bpf_dynptr_kern {
void *data;
/* Size represents the number of usable bytes of dynptr data.
* If for example the offset is at 4 for a local dynptr whose data is
* of type u64, the number of usable bytes is 4.
*
* The upper 8 bits are reserved. It is as follows:
* Bits 0 - 23 = size
* Bits 24 - 30 = dynptr type
* Bit 31 = whether dynptr is read-only
*/
u32 size;
u32 offset;
} __aligned(8);
enum bpf_dynptr_type {
BPF_DYNPTR_TYPE_INVALID,
/* Points to memory that is local to the bpf program */
BPF_DYNPTR_TYPE_LOCAL,
/* Underlying data is a ringbuf record */
BPF_DYNPTR_TYPE_RINGBUF,
/* Underlying data is a sk_buff */
BPF_DYNPTR_TYPE_SKB,
/* Underlying data is a xdp_buff */
BPF_DYNPTR_TYPE_XDP,
/* Points to skb_metadata_end()-skb_metadata_len() */
BPF_DYNPTR_TYPE_SKB_META,
/* Underlying data is a file */
BPF_DYNPTR_TYPE_FILE,
};
int bpf_dynptr_check_size(u64 size);
u64 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr);
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u64 len);
void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u64 len);
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr);
int __bpf_dynptr_write(const struct bpf_dynptr_kern *dst, u64 offset,
void *src, u64 len, u64 flags);
void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk);
static inline int bpf_dynptr_check_off_len(const struct bpf_dynptr_kern *ptr, u64 offset, u64 len)
{
u64 size = __bpf_dynptr_size(ptr);
if (len > size || offset > size - len)
return -E2BIG;
return 0;
}
#ifdef CONFIG_BPF_JIT
int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog);
int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog);
struct bpf_trampoline *bpf_trampoline_get(u64 key,
struct bpf_attach_target_info *tgt_info);
void bpf_trampoline_put(struct bpf_trampoline *tr);
int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
/*
* When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
* indirection with a direct call to the bpf program. If the architecture does
* not have STATIC_CALL, avoid a double-indirection.
*/
#ifdef CONFIG_HAVE_STATIC_CALL
#define __BPF_DISPATCHER_SC_INIT(_name) \
.sc_key = &STATIC_CALL_KEY(_name), \
.sc_tramp = STATIC_CALL_TRAMP_ADDR(_name),
#define __BPF_DISPATCHER_SC(name) \
DEFINE_STATIC_CALL(bpf_dispatcher_##name##_call, bpf_dispatcher_nop_func)
#define __BPF_DISPATCHER_CALL(name) \
static_call(bpf_dispatcher_##name##_call)(ctx, insnsi, bpf_func)
#define __BPF_DISPATCHER_UPDATE(_d, _new) \
__static_call_update((_d)->sc_key, (_d)->sc_tramp, (_new))
#else
#define __BPF_DISPATCHER_SC_INIT(name)
#define __BPF_DISPATCHER_SC(name)
#define __BPF_DISPATCHER_CALL(name) bpf_func(ctx, insnsi)
#define __BPF_DISPATCHER_UPDATE(_d, _new)
#endif
#define BPF_DISPATCHER_INIT(_name) { \
.mutex = __MUTEX_INITIALIZER(_name.mutex), \
.func = &_name##_func, \
.progs = {}, \
.num_progs = 0, \
.image = NULL, \
.image_off = 0, \
.ksym = { \
.name = #_name, \
.lnode = LIST_HEAD_INIT(_name.ksym.lnode), \
}, \
__BPF_DISPATCHER_SC_INIT(_name##_call) \
}
#define DEFINE_BPF_DISPATCHER(name) \
__BPF_DISPATCHER_SC(name); \
noinline __bpfcall unsigned int bpf_dispatcher_##name##_func( \
const void *ctx, \
const struct bpf_insn *insnsi, \
bpf_func_t bpf_func) \
{ \
return __BPF_DISPATCHER_CALL(name); \
} \
EXPORT_SYMBOL(bpf_dispatcher_##name##_func); \
struct bpf_dispatcher bpf_dispatcher_##name = \
BPF_DISPATCHER_INIT(bpf_dispatcher_##name);
#define DECLARE_BPF_DISPATCHER(name) \
unsigned int bpf_dispatcher_##name##_func( \
const void *ctx, \
const struct bpf_insn *insnsi, \
bpf_func_t bpf_func); \
extern struct bpf_dispatcher bpf_dispatcher_##name;
#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_##name##_func
#define BPF_DISPATCHER_PTR(name) (&bpf_dispatcher_##name)
void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
struct bpf_prog *to);
/* Called only from JIT-enabled code, so there's no need for stubs. */
void bpf_image_ksym_init(void *data, unsigned int size, struct bpf_ksym *ksym);
void bpf_image_ksym_add(struct bpf_ksym *ksym);
void bpf_image_ksym_del(struct bpf_ksym *ksym);
void bpf_ksym_add(struct bpf_ksym *ksym);
void bpf_ksym_del(struct bpf_ksym *ksym);
bool bpf_has_frame_pointer(unsigned long ip);
int bpf_jit_charge_modmem(u32 size);
void bpf_jit_uncharge_modmem(u32 size);
bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,
int insn_idx);
#else
static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
return -ENOTSUPP;
}
static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
return -ENOTSUPP;
}
static inline struct bpf_trampoline *bpf_trampoline_get(u64 key,
struct bpf_attach_target_info *tgt_info)
{
return NULL;
}
static inline void bpf_trampoline_put(struct bpf_trampoline *tr) {}
#define DEFINE_BPF_DISPATCHER(name)
#define DECLARE_BPF_DISPATCHER(name)
#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_nop_func
#define BPF_DISPATCHER_PTR(name) NULL
static inline void bpf_dispatcher_change_prog(struct bpf_dispatcher *d,
struct bpf_prog *from,
struct bpf_prog *to) {}
static inline bool is_bpf_image_address(unsigned long address)
{
return false;
}
static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog)
{
return false;
}
#endif
struct bpf_func_info_aux {
u16 linkage;
bool unreliable;
bool called : 1;
bool verified : 1;
};
enum bpf_jit_poke_reason {
BPF_POKE_REASON_TAIL_CALL,
};
/* Descriptor of pokes pointing /into/ the JITed image. */
struct bpf_jit_poke_descriptor {
void *tailcall_target;
void *tailcall_bypass;
void *bypass_addr;
void *aux;
union {
struct {
struct bpf_map *map;
u32 key;
} tail_call;
};
bool tailcall_target_stable;
u8 adj_off;
u16 reason;
u32 insn_idx;
};
/* reg_type info for ctx arguments */
struct bpf_ctx_arg_aux {
u32 offset;
enum bpf_reg_type reg_type;
struct btf *btf;
u32 btf_id;
u32 ref_obj_id;
bool refcounted;
};
struct btf_mod_pair {
struct btf *btf;
struct module *module;
};
struct bpf_kfunc_desc_tab;
enum bpf_stream_id {
BPF_STDOUT = 1,
BPF_STDERR = 2,
};
struct bpf_stream_elem {
struct llist_node node;
int total_len;
int consumed_len;
char str[];
};
enum {
/* 100k bytes */
BPF_STREAM_MAX_CAPACITY = 100000ULL,
};
struct bpf_stream {
atomic_t capacity;
struct llist_head log; /* list of in-flight stream elements in LIFO order */
struct mutex lock; /* lock protecting backlog_{head,tail} */
struct llist_node *backlog_head; /* list of in-flight stream elements in FIFO order */
struct llist_node *backlog_tail; /* tail of the list above */
};
struct bpf_stream_stage {
struct llist_head log;
int len;
};
struct bpf_prog_aux {
atomic64_t refcnt;
u32 used_map_cnt;
u32 used_btf_cnt;
u32 max_ctx_offset;
u32 max_pkt_offset;
u32 max_tp_access;
u32 stack_depth;
u32 id;
u32 func_cnt; /* used by non-func prog as the number of func progs */
u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */
u32 func_idx; /* 0 for non-func prog, the index in func array for func prog */
u32 attach_btf_id; /* in-kernel BTF type id to attach to */
u32 attach_st_ops_member_off;
u32 ctx_arg_info_size;
u32 max_rdonly_access;
u32 max_rdwr_access;
u32 subprog_start;
struct btf *attach_btf;
struct bpf_ctx_arg_aux *ctx_arg_info;
void __percpu *priv_stack_ptr;
struct mutex dst_mutex; /* protects dst_* pointers below, *after* prog becomes visible */
struct bpf_prog *dst_prog;
struct bpf_trampoline *dst_trampoline;
enum bpf_prog_type saved_dst_prog_type;
enum bpf_attach_type saved_dst_attach_type;
bool verifier_zext; /* Zero extensions has been inserted by verifier. */
bool dev_bound; /* Program is bound to the netdev. */
bool offload_requested; /* Program is bound and offloaded to the netdev. */
bool attach_btf_trace; /* true if attaching to BTF-enabled raw tp */
bool attach_tracing_prog; /* true if tracing another tracing program */
bool func_proto_unreliable;
bool tail_call_reachable;
bool xdp_has_frags;
bool exception_cb;
bool exception_boundary;
bool is_extended; /* true if extended by freplace program */
bool jits_use_priv_stack;
bool priv_stack_requested;
bool changes_pkt_data;
bool might_sleep;
bool kprobe_write_ctx;
u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */
struct bpf_arena *arena;
void (*recursion_detected)(struct bpf_prog *prog); /* callback if recursion is detected */
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
/* function name for valid attach_btf_id */
const char *attach_func_name;
struct bpf_prog **func;
struct bpf_prog_aux *main_prog_aux;
void *jit_data; /* JIT specific data. arch dependent */
struct bpf_jit_poke_descriptor *poke_tab;
struct bpf_kfunc_desc_tab *kfunc_tab;
struct bpf_kfunc_btf_tab *kfunc_btf_tab;
u32 size_poke_tab;
#ifdef CONFIG_FINEIBT
struct bpf_ksym ksym_prefix;
#endif
struct bpf_ksym ksym;
const struct bpf_prog_ops *ops;
const struct bpf_struct_ops *st_ops;
struct bpf_map **used_maps;
struct mutex used_maps_mutex; /* mutex for used_maps and used_map_cnt */
struct btf_mod_pair *used_btfs;
struct bpf_prog *prog;
struct user_struct *user;
u64 load_time; /* ns since boottime */
u32 verified_insns;
int cgroup_atype; /* enum cgroup_bpf_attach_type */
struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
char name[BPF_OBJ_NAME_LEN];
u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
#ifdef CONFIG_SECURITY
void *security;
#endif
struct bpf_token *token;
struct bpf_prog_offload *offload;
struct btf *btf;
struct bpf_func_info *func_info;
struct bpf_func_info_aux *func_info_aux;
/* bpf_line_info loaded from userspace. linfo->insn_off
* has the xlated insn offset.
* Both the main and sub prog share the same linfo.
* The subprog can access its first linfo by
* using the linfo_idx.
*/
struct bpf_line_info *linfo;
/* jited_linfo is the jited addr of the linfo. It has a
* one to one mapping to linfo:
* jited_linfo[i] is the jited addr for the linfo[i]->insn_off.
* Both the main and sub prog share the same jited_linfo.
* The subprog can access its first jited_linfo by
* using the linfo_idx.
*/
void **jited_linfo;
u32 func_info_cnt;
u32 nr_linfo;
/* subprog can use linfo_idx to access its first linfo and
* jited_linfo.
* main prog always has linfo_idx == 0
*/
u32 linfo_idx;
struct module *mod;
u32 num_exentries;
struct exception_table_entry *extable;
union {
struct work_struct work;
struct rcu_head rcu;
};
struct bpf_stream stream[2];
struct mutex st_ops_assoc_mutex;
struct bpf_map __rcu *st_ops_assoc;
};
#define BPF_NR_CONTEXTS 4 /* normal, softirq, hardirq, NMI */
struct bpf_prog {
u16 pages; /* Number of allocated pages */
u16 jited:1, /* Is our filter JIT'ed? */
jit_requested:1,/* archs need to JIT the prog */
gpl_compatible:1, /* Is filter GPL compatible? */
cb_access:1, /* Is control block accessed? */
dst_needed:1, /* Do we need dst entry? */
blinding_requested:1, /* needs constant blinding */
blinded:1, /* Was blinded */
is_func:1, /* program is a bpf function */
kprobe_override:1, /* Do we override a kprobe? */
has_callchain_buf:1, /* callchain buffer allocated? */
enforce_expected_attach_type:1, /* Enforce expected_attach_type checking at attach time */
call_get_stack:1, /* Do we call bpf_get_stack() or bpf_get_stackid() */
call_get_func_ip:1, /* Do we call get_func_ip() */
call_session_cookie:1, /* Do we call bpf_session_cookie() */
tstamp_type_access:1, /* Accessed __sk_buff->tstamp_type */
sleepable:1; /* BPF program is sleepable */
enum bpf_prog_type type; /* Type of BPF program */
enum bpf_attach_type expected_attach_type; /* For some prog types */
u32 len; /* Number of filter blocks */
u32 jited_len; /* Size of jited insns in bytes */
union {
u8 digest[SHA256_DIGEST_SIZE];
u8 tag[BPF_TAG_SIZE];
};
struct bpf_prog_stats __percpu *stats;
u8 __percpu *active; /* u8[BPF_NR_CONTEXTS] for recursion protection */
unsigned int (*bpf_func)(const void *ctx,
const struct bpf_insn *insn);
struct bpf_prog_aux *aux; /* Auxiliary fields */
struct sock_fprog_kern *orig_prog; /* Original BPF program */
/* Instructions for interpreter */
union {
DECLARE_FLEX_ARRAY(struct sock_filter, insns);
DECLARE_FLEX_ARRAY(struct bpf_insn, insnsi);
};
};
struct bpf_array_aux {
/* Programs with direct jumps into programs part of this array. */
struct list_head poke_progs;
struct bpf_map *map;
struct mutex poke_mutex;
struct work_struct work;
};
struct bpf_link {
atomic64_t refcnt;
u32 id;
enum bpf_link_type type;
const struct bpf_link_ops *ops;
struct bpf_prog *prog;
u32 flags;
enum bpf_attach_type attach_type;
/* rcu is used before freeing, work can be used to schedule that
* RCU-based freeing before that, so they never overlap
*/
union {
struct rcu_head rcu;
struct work_struct work;
};
/* whether BPF link itself has "sleepable" semantics, which can differ
* from underlying BPF program having a "sleepable" semantics, as BPF
* link's semantics is determined by target attach hook
*/
bool sleepable;
};
struct bpf_link_ops {
void (*release)(struct bpf_link *link);
/* deallocate link resources callback, called without RCU grace period
* waiting
*/
void (*dealloc)(struct bpf_link *link);
/* deallocate link resources callback, called after RCU grace period;
* if either the underlying BPF program is sleepable or BPF link's
* target hook is sleepable, we'll go through tasks trace RCU GP and
* then "classic" RCU GP; this need for chaining tasks trace and
* classic RCU GPs is designated by setting bpf_link->sleepable flag
*
* For non-sleepable tracepoint links we go through SRCU gp instead,
* since RCU is not used in that case. Sleepable tracepoints still
* follow the scheme above.
*/
void (*dealloc_deferred)(struct bpf_link *link);
int (*detach)(struct bpf_link *link);
int (*update_prog)(struct bpf_link *link, struct bpf_prog *new_prog,
struct bpf_prog *old_prog);
void (*show_fdinfo)(const struct bpf_link *link, struct seq_file *seq);
int (*fill_link_info)(const struct bpf_link *link,
struct bpf_link_info *info);
int (*update_map)(struct bpf_link *link, struct bpf_map *new_map,
struct bpf_map *old_map);
__poll_t (*poll)(struct file *file, struct poll_table_struct *pts);
};
struct bpf_tramp_link {
struct bpf_link link;
struct hlist_node tramp_hlist;
u64 cookie;
};
struct bpf_shim_tramp_link {
struct bpf_tramp_link link;
struct bpf_trampoline *trampoline;
};
struct bpf_tracing_link {
struct bpf_tramp_link link;
struct bpf_trampoline *trampoline;
struct bpf_prog *tgt_prog;
};
struct bpf_fsession_link {
struct bpf_tracing_link link;
struct bpf_tramp_link fexit;
};
struct bpf_raw_tp_link {
struct bpf_link link;
struct bpf_raw_event_map *btp;
u64 cookie;
};
struct bpf_link_primer {
struct bpf_link *link;
struct file *file;
int fd;
u32 id;
};
struct bpf_mount_opts {
kuid_t uid;
kgid_t gid;
umode_t mode;
/* BPF token-related delegation options */
u64 delegate_cmds;
u64 delegate_maps;
u64 delegate_progs;
u64 delegate_attachs;
};
struct bpf_token {
struct work_struct work;
atomic64_t refcnt;
struct user_namespace *userns;
u64 allowed_cmds;
u64 allowed_maps;
u64 allowed_progs;
u64 allowed_attachs;
#ifdef CONFIG_SECURITY
void *security;
#endif
};
struct bpf_struct_ops_value;
struct btf_member;
#define BPF_STRUCT_OPS_MAX_NR_MEMBERS 64
/**
* struct bpf_struct_ops - A structure of callbacks allowing a subsystem to
* define a BPF_MAP_TYPE_STRUCT_OPS map type composed
* of BPF_PROG_TYPE_STRUCT_OPS progs.
* @verifier_ops: A structure of callbacks that are invoked by the verifier
* when determining whether the struct_ops progs in the
* struct_ops map are valid.
* @init: A callback that is invoked a single time, and before any other
* callback, to initialize the structure. A nonzero return value means
* the subsystem could not be initialized.
* @check_member: When defined, a callback invoked by the verifier to allow
* the subsystem to determine if an entry in the struct_ops map
* is valid. A nonzero return value means that the map is
* invalid and should be rejected by the verifier.
* @init_member: A callback that is invoked for each member of the struct_ops
* map to allow the subsystem to initialize the member. A nonzero
* value means the member could not be initialized. This callback
* is exclusive with the @type, @type_id, @value_type, and
* @value_id fields.
* @reg: A callback that is invoked when the struct_ops map has been
* initialized and is being attached to. Zero means the struct_ops map
* has been successfully registered and is live. A nonzero return value
* means the struct_ops map could not be registered.
* @unreg: A callback that is invoked when the struct_ops map should be
* unregistered.
* @update: A callback that is invoked when the live struct_ops map is being
* updated to contain new values. This callback is only invoked when
* the struct_ops map is loaded with BPF_F_LINK. If not defined, the
* it is assumed that the struct_ops map cannot be updated.
* @validate: A callback that is invoked after all of the members have been
* initialized. This callback should perform static checks on the
* map, meaning that it should either fail or succeed
* deterministically. A struct_ops map that has been validated may
* not necessarily succeed in being registered if the call to @reg
* fails. For example, a valid struct_ops map may be loaded, but
* then fail to be registered due to there being another active
* struct_ops map on the system in the subsystem already. For this
* reason, if this callback is not defined, the check is skipped as
* the struct_ops map will have final verification performed in
* @reg.
* @cfi_stubs: Pointer to a structure of stub functions for CFI. These stubs
* provide the correct Control Flow Integrity hashes for the
* trampolines generated by BPF struct_ops.
* @owner: The module that owns this struct_ops. Used for module reference
* counting to ensure the module providing the struct_ops cannot be
* unloaded while in use.
* @name: The name of the struct bpf_struct_ops object.
* @func_models: Func models
*/
struct bpf_struct_ops {
const struct bpf_verifier_ops *verifier_ops;
int (*init)(struct btf *btf);
int (*check_member)(const struct btf_type *t,
const struct btf_member *member,
const struct bpf_prog *prog);
int (*init_member)(const struct btf_type *t,
const struct btf_member *member,
void *kdata, const void *udata);
int (*reg)(void *kdata, struct bpf_link *link);
void (*unreg)(void *kdata, struct bpf_link *link);
int (*update)(void *kdata, void *old_kdata, struct bpf_link *link);
int (*validate)(void *kdata);
void *cfi_stubs;
struct module *owner;
const char *name;
struct btf_func_model func_models[BPF_STRUCT_OPS_MAX_NR_MEMBERS];
};
/* Every member of a struct_ops type has an instance even a member is not
* an operator (function pointer). The "info" field will be assigned to
* prog->aux->ctx_arg_info of BPF struct_ops programs to provide the
* argument information required by the verifier to verify the program.
*
* btf_ctx_access() will lookup prog->aux->ctx_arg_info to find the
* corresponding entry for an given argument.
*/
struct bpf_struct_ops_arg_info {
struct bpf_ctx_arg_aux *info;
u32 cnt;
};
struct bpf_struct_ops_desc {
struct bpf_struct_ops *st_ops;
const struct btf_type *type;
const struct btf_type *value_type;
u32 type_id;
u32 value_id;
/* Collection of argument information for each member */
struct bpf_struct_ops_arg_info *arg_info;
};
enum bpf_struct_ops_state {
BPF_STRUCT_OPS_STATE_INIT,
BPF_STRUCT_OPS_STATE_INUSE,
BPF_STRUCT_OPS_STATE_TOBEFREE,
BPF_STRUCT_OPS_STATE_READY,
};
struct bpf_struct_ops_common_value {
refcount_t refcnt;
enum bpf_struct_ops_state state;
};
static inline bool bpf_prog_get_recursion_context(struct bpf_prog *prog)
{
#ifdef CONFIG_ARM64
u8 rctx = interrupt_context_level();
u8 *active = this_cpu_ptr(prog->active);
u32 val;
preempt_disable();
active[rctx]++;
val = le32_to_cpu(*(__le32 *)active);
preempt_enable();
if (val != BIT(rctx * 8))
return false;
return true;
#else
return this_cpu_inc_return(*(int __percpu *)(prog->active)) == 1;
#endif
}
static inline void bpf_prog_put_recursion_context(struct bpf_prog *prog)
{
#ifdef CONFIG_ARM64
u8 rctx = interrupt_context_level();
u8 *active = this_cpu_ptr(prog->active);
preempt_disable();
active[rctx]--;
preempt_enable();
#else
this_cpu_dec(*(int __percpu *)(prog->active));
#endif
}
#if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL)
/* This macro helps developer to register a struct_ops type and generate
* type information correctly. Developers should use this macro to register
* a struct_ops type instead of calling __register_bpf_struct_ops() directly.
*/
#define register_bpf_struct_ops(st_ops, type) \
({ \
struct bpf_struct_ops_##type { \
struct bpf_struct_ops_common_value common; \
struct type data ____cacheline_aligned_in_smp; \
}; \
BTF_TYPE_EMIT(struct bpf_struct_ops_##type); \
__register_bpf_struct_ops(st_ops); \
})
#define BPF_MODULE_OWNER ((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA))
bool bpf_struct_ops_get(const void *kdata);
void bpf_struct_ops_put(const void *kdata);
int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff);
int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key,
void *value);
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
struct bpf_tramp_link *link,
const struct btf_func_model *model,
void *stub_func,
void **image, u32 *image_off,
bool allow_alloc);
void bpf_struct_ops_image_free(void *image);
static inline bool bpf_try_module_get(const void *data, struct module *owner)
{
if (owner == BPF_MODULE_OWNER)
return bpf_struct_ops_get(data);
else
return try_module_get(owner);
}
static inline void bpf_module_put(const void *data, struct module *owner)
{
if (owner == BPF_MODULE_OWNER)
bpf_struct_ops_put(data);
else
module_put(owner);
}
int bpf_struct_ops_link_create(union bpf_attr *attr);
int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map);
void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog);
void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux);
u32 bpf_struct_ops_id(const void *kdata);
#ifdef CONFIG_NET
/* Define it here to avoid the use of forward declaration */
struct bpf_dummy_ops_state {
int val;
};
struct bpf_dummy_ops {
int (*test_1)(struct bpf_dummy_ops_state *cb);
int (*test_2)(struct bpf_dummy_ops_state *cb, int a1, unsigned short a2,
char a3, unsigned long a4);
int (*test_sleepable)(struct bpf_dummy_ops_state *cb);
};
int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
union bpf_attr __user *uattr);
#endif
int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
struct btf *btf,
struct bpf_verifier_log *log);
void bpf_map_struct_ops_info_fill(struct bpf_map_info *info, struct bpf_map *map);
void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_ops_desc);
#else
#define register_bpf_struct_ops(st_ops, type) ({ (void *)(st_ops); 0; })
static inline bool bpf_try_module_get(const void *data, struct module *owner)
{
return try_module_get(owner);
}
static inline void bpf_module_put(const void *data, struct module *owner)
{
module_put(owner);
}
static inline int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff)
{
return -ENOTSUPP;
}
static inline int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map,
void *key,
void *value)
{
return -EINVAL;
}
static inline int bpf_struct_ops_link_create(union bpf_attr *attr)
{
return -EOPNOTSUPP;
}
static inline int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map)
{
return -EOPNOTSUPP;
}
static inline void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog)
{
}
static inline void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux)
{
return NULL;
}
static inline void bpf_map_struct_ops_info_fill(struct bpf_map_info *info, struct bpf_map *map)
{
}
static inline void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_ops_desc)
{
}
#endif
static inline int bpf_fsession_cnt(struct bpf_tramp_links *links)
{
struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];
int cnt = 0;
for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {
if (fentries.links[i]->link.prog->expected_attach_type == BPF_TRACE_FSESSION)
cnt++;
}
return cnt;
}
static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_link *link)
{
return link->link.prog->call_session_cookie;
}
static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_links *links)
{
struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];
int cnt = 0;
for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {
if (bpf_prog_calls_session_cookie(fentries.links[i]))
cnt++;
}
return cnt;
}
int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
const struct bpf_ctx_arg_aux *info, u32 cnt);
#if defined(CONFIG_CGROUP_BPF) && defined(CONFIG_BPF_LSM)
int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
int cgroup_atype,
enum bpf_attach_type attach_type);
void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog);
#else
static inline int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
int cgroup_atype,
enum bpf_attach_type attach_type)
{
return -EOPNOTSUPP;
}
static inline void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog)
{
}
#endif
struct bpf_array {
struct bpf_map map;
u32 elem_size;
u32 index_mask;
struct bpf_array_aux *aux;
union {
DECLARE_FLEX_ARRAY(char, value) __aligned(8);
DECLARE_FLEX_ARRAY(void *, ptrs) __aligned(8);
DECLARE_FLEX_ARRAY(void __percpu *, pptrs) __aligned(8);
};
};
/*
* The bpf_array_get_next_key() function may be used for all array-like
* maps, i.e., maps with u32 keys with range [0 ,..., max_entries)
*/
int bpf_array_get_next_key(struct bpf_map *map, void *key, void *next_key);
#define BPF_COMPLEXITY_LIMIT_INSNS 1000000 /* yes. 1M insns */
#define MAX_TAIL_CALL_CNT 33
/* Maximum number of loops for bpf_loop and bpf_iter_num.
* It's enum to expose it (and thus make it discoverable) through BTF.
*/
enum {
BPF_MAX_LOOPS = 8 * 1024 * 1024,
BPF_MAX_TIMED_LOOPS = 0xffff,
};
#define BPF_F_ACCESS_MASK (BPF_F_RDONLY | \
BPF_F_RDONLY_PROG | \
BPF_F_WRONLY | \
BPF_F_WRONLY_PROG)
#define BPF_MAP_CAN_READ BIT(0)
#define BPF_MAP_CAN_WRITE BIT(1)
/* Maximum number of user-producer ring buffer samples that can be drained in
* a call to bpf_user_ringbuf_drain().
*/
#define BPF_MAX_USER_RINGBUF_SAMPLES (128 * 1024)
static inline u32 bpf_map_flags_to_cap(struct bpf_map *map)
{
u32 access_flags = map->map_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG);
/* Combination of BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG is
* not possible.
*/
if (access_flags & BPF_F_RDONLY_PROG)
return BPF_MAP_CAN_READ;
else if (access_flags & BPF_F_WRONLY_PROG)
return BPF_MAP_CAN_WRITE;
else
return BPF_MAP_CAN_READ | BPF_MAP_CAN_WRITE;
}
static inline bool bpf_map_flags_access_ok(u32 access_flags)
{
return (access_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG)) !=
(BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG);
}
static inline struct bpf_map_owner *bpf_map_owner_alloc(struct bpf_map *map)
{
return kzalloc_obj(*map->owner, GFP_ATOMIC);
}
static inline void bpf_map_owner_free(struct bpf_map *map)
{
kfree(map->owner);
}
struct bpf_event_entry {
struct perf_event *event;
struct file *perf_file;
struct file *map_file;
struct rcu_head rcu;
};
static inline bool map_type_contains_progs(struct bpf_map *map)
{
return map->map_type == BPF_MAP_TYPE_PROG_ARRAY ||
map->map_type == BPF_MAP_TYPE_DEVMAP ||
map->map_type == BPF_MAP_TYPE_CPUMAP;
}
bool bpf_prog_map_compatible(struct bpf_map *map, const struct bpf_prog *fp);
int bpf_prog_calc_tag(struct bpf_prog *fp);
const struct bpf_func_proto *bpf_get_trace_printk_proto(void);
const struct bpf_func_proto *bpf_get_trace_vprintk_proto(void);
const struct bpf_func_proto *bpf_get_perf_event_read_value_proto(void);
typedef unsigned long (*bpf_ctx_copy_t)(void *dst, const void *src,
unsigned long off, unsigned long len);
typedef u32 (*bpf_convert_ctx_access_t)(enum bpf_access_type type,
const struct bpf_insn *src,
struct bpf_insn *dst,
struct bpf_prog *prog,
u32 *target_size);
u64 bpf_event_output(struct bpf_map *map, u64 flags, void *meta, u64 meta_size,
void *ctx, u64 ctx_size, bpf_ctx_copy_t ctx_copy);
/* an array of programs to be executed under rcu_lock.
*
* Typical usage:
* ret = bpf_prog_run_array(rcu_dereference(&bpf_prog_array), ctx, bpf_prog_run);
*
* the structure returned by bpf_prog_array_alloc() should be populated
* with program pointers and the last pointer must be NULL.
* The user has to keep refcnt on the program and make sure the program
* is removed from the array before bpf_prog_put().
* The 'struct bpf_prog_array *' should only be replaced with xchg()
* since other cpus are walking the array of pointers in parallel.
*/
struct bpf_prog_array_item {
struct bpf_prog *prog;
union {
struct bpf_cgroup_storage *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
u64 bpf_cookie;
};
};
struct bpf_prog_array {
struct rcu_head rcu;
struct bpf_prog_array_item items[];
};
/* to avoid allocating empty bpf_prog_array for cgroups that
* don't have bpf program attached use one global 'bpf_empty_prog_array'
* It will not be modified the caller of bpf_prog_array_alloc()
* (since caller requested prog_cnt == 0)
* that pointer should be 'freed' by bpf_prog_array_free()
*/
extern struct bpf_prog_array bpf_empty_prog_array;
struct bpf_prog_array *bpf_prog_array_alloc(u32 prog_cnt, gfp_t flags);
void bpf_prog_array_free(struct bpf_prog_array *progs);
/* Use when traversal over the bpf_prog_array uses tasks_trace rcu */
void bpf_prog_array_free_sleepable(struct bpf_prog_array *progs);
int bpf_prog_array_length(struct bpf_prog_array *progs);
bool bpf_prog_array_is_empty(struct bpf_prog_array *array);
int bpf_prog_array_copy_to_user(struct bpf_prog_array *progs,
__u32 __user *prog_ids, u32 cnt);
void bpf_prog_array_delete_safe(struct bpf_prog_array *progs,
struct bpf_prog *old_prog);
int bpf_prog_array_delete_safe_at(struct bpf_prog_array *array, int index);
int bpf_prog_array_update_at(struct bpf_prog_array *array, int index,
struct bpf_prog *prog);
int bpf_prog_array_copy_info(struct bpf_prog_array *array,
u32 *prog_ids, u32 request_cnt,
u32 *prog_cnt);
int bpf_prog_array_copy(struct bpf_prog_array *old_array,
struct bpf_prog *exclude_prog,
struct bpf_prog *include_prog,
u64 bpf_cookie,
struct bpf_prog_array **new_array);
struct bpf_run_ctx {};
struct bpf_cg_run_ctx {
struct bpf_run_ctx run_ctx;
const struct bpf_prog_array_item *prog_item;
int retval;
};
struct bpf_trace_run_ctx {
struct bpf_run_ctx run_ctx;
u64 bpf_cookie;
bool is_uprobe;
};
struct bpf_tramp_run_ctx {
struct bpf_run_ctx run_ctx;
u64 bpf_cookie;
struct bpf_run_ctx *saved_run_ctx;
};
static inline struct bpf_run_ctx *bpf_set_run_ctx(struct bpf_run_ctx *new_ctx)
{
struct bpf_run_ctx *old_ctx = NULL;
#ifdef CONFIG_BPF_SYSCALL
old_ctx = current->bpf_ctx;
current->bpf_ctx = new_ctx;
#endif
return old_ctx;
}
static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
{
#ifdef CONFIG_BPF_SYSCALL
current->bpf_ctx = old_ctx;
#endif
}
/* BPF program asks to bypass CAP_NET_BIND_SERVICE in bind. */
#define BPF_RET_BIND_NO_CAP_NET_BIND_SERVICE (1 << 0)
/* BPF program asks to set CN on the packet. */
#define BPF_RET_SET_CN (1 << 0)
typedef u32 (*bpf_prog_run_fn)(const struct bpf_prog *prog, const void *ctx);
static __always_inline u32
bpf_prog_run_array(const struct bpf_prog_array *array,
const void *ctx, bpf_prog_run_fn run_prog)
{
const struct bpf_prog_array_item *item;
const struct bpf_prog *prog;
struct bpf_run_ctx *old_run_ctx;
struct bpf_trace_run_ctx run_ctx;
u32 ret = 1;
RCU_LOCKDEP_WARN(!rcu_read_lock_held(), "no rcu lock held");
if (unlikely(!array))
return ret;
run_ctx.is_uprobe = false;
migrate_disable();
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
item = &array->items[0];
while ((prog = READ_ONCE(item->prog))) {
run_ctx.bpf_cookie = item->bpf_cookie;
ret &= run_prog(prog, ctx);
item++;
}
bpf_reset_run_ctx(old_run_ctx);
migrate_enable();
return ret;
}
/* Notes on RCU design for bpf_prog_arrays containing sleepable programs:
*
* We use the tasks_trace rcu flavor read section to protect the bpf_prog_array
* overall. As a result, we must use the bpf_prog_array_free_sleepable
* in order to use the tasks_trace rcu grace period.
*
* When a non-sleepable program is inside the array, we take the rcu read
* section and disable preemption for that program alone, so it can access
* rcu-protected dynamically sized maps.
*/
static __always_inline u32
bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
const void *ctx, bpf_prog_run_fn run_prog)
{
const struct bpf_prog_array_item *item;
const struct bpf_prog *prog;
struct bpf_run_ctx *old_run_ctx;
struct bpf_trace_run_ctx run_ctx;
u32 ret = 1;
might_fault();
RCU_LOCKDEP_WARN(!rcu_read_lock_trace_held(), "no rcu lock held");
if (unlikely(!array))
return ret;
migrate_disable();
run_ctx.is_uprobe = true;
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
item = &array->items[0];
while ((prog = READ_ONCE(item->prog))) {
if (!prog->sleepable)
rcu_read_lock();
run_ctx.bpf_cookie = item->bpf_cookie;
ret &= run_prog(prog, ctx);
item++;
if (!prog->sleepable)
rcu_read_unlock();
}
bpf_reset_run_ctx(old_run_ctx);
migrate_enable();
return ret;
}
bool bpf_jit_bypass_spec_v1(void);
bool bpf_jit_bypass_spec_v4(void);
#define bpf_rcu_lock_held() \
(rcu_read_lock_held() || rcu_read_lock_trace_held() || rcu_read_lock_bh_held())
#ifdef CONFIG_BPF_SYSCALL
DECLARE_PER_CPU(int, bpf_prog_active);
extern struct mutex bpf_stats_enabled_mutex;
/*
* Block execution of BPF programs attached to instrumentation (perf,
* kprobes, tracepoints) to prevent deadlocks on map operations as any of
* these events can happen inside a region which holds a map bucket lock
* and can deadlock on it.
*/
static inline void bpf_disable_instrumentation(void)
{
migrate_disable();
this_cpu_inc(bpf_prog_active);
}
static inline void bpf_enable_instrumentation(void)
{
this_cpu_dec(bpf_prog_active);
migrate_enable();
}
extern const struct super_operations bpf_super_ops;
extern const struct file_operations bpf_map_fops;
extern const struct file_operations bpf_prog_fops;
extern const struct file_operations bpf_iter_fops;
extern const struct file_operations bpf_token_fops;
#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
extern const struct bpf_prog_ops _name ## _prog_ops; \
extern const struct bpf_verifier_ops _name ## _verifier_ops;
#define BPF_MAP_TYPE(_id, _ops) \
extern const struct bpf_map_ops _ops;
#define BPF_LINK_TYPE(_id, _name)
#include <linux/bpf_types.h>
#undef BPF_PROG_TYPE
#undef BPF_MAP_TYPE
#undef BPF_LINK_TYPE
extern const struct bpf_prog_ops bpf_offload_prog_ops;
extern const struct bpf_verifier_ops tc_cls_act_analyzer_ops;
extern const struct bpf_verifier_ops xdp_analyzer_ops;
struct bpf_prog *bpf_prog_get(u32 ufd);
struct bpf_prog *bpf_prog_get_type_dev(u32 ufd, enum bpf_prog_type type,
bool attach_drv);
void bpf_prog_add(struct bpf_prog *prog, int i);
void bpf_prog_sub(struct bpf_prog *prog, int i);
void bpf_prog_inc(struct bpf_prog *prog);
struct bpf_prog * __must_check bpf_prog_inc_not_zero(struct bpf_prog *prog);
void bpf_prog_put(struct bpf_prog *prog);
void bpf_prog_free_id(struct bpf_prog *prog);
void bpf_map_free_id(struct bpf_map *map);
struct btf_field *btf_record_find(const struct btf_record *rec,
u32 offset, u32 field_mask);
void btf_record_free(struct btf_record *rec);
void bpf_map_free_record(struct bpf_map *map);
struct btf_record *btf_record_dup(const struct btf_record *rec);
bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *rec_b);
void bpf_obj_free_timer(const struct btf_record *rec, void *obj);
void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj);
void bpf_obj_free_task_work(const struct btf_record *rec, void *obj);
void bpf_obj_free_fields(const struct btf_record *rec, void *obj);
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu);
struct bpf_map *bpf_map_get(u32 ufd);
struct bpf_map *bpf_map_get_with_uref(u32 ufd);
/*
* The __bpf_map_get() and __btf_get_by_fd() functions parse a file
* descriptor and return a corresponding map or btf object.
* Their names are double underscored to emphasize the fact that they
* do not increase refcnt. To also increase refcnt use corresponding
* bpf_map_get() and btf_get_by_fd() functions.
*/
static inline struct bpf_map *__bpf_map_get(struct fd f)
{
if (fd_empty(f))
return ERR_PTR(-EBADF);
if (unlikely(fd_file(f)->f_op != &bpf_map_fops))
return ERR_PTR(-EINVAL);
return fd_file(f)->private_data;
}
static inline struct btf *__btf_get_by_fd(struct fd f)
{
if (fd_empty(f))
return ERR_PTR(-EBADF);
if (unlikely(fd_file(f)->f_op != &btf_fops))
return ERR_PTR(-EINVAL);
return fd_file(f)->private_data;
}
void bpf_map_inc(struct bpf_map *map);
void bpf_map_inc_with_uref(struct bpf_map *map);
struct bpf_map *__bpf_map_inc_not_zero(struct bpf_map *map, bool uref);
struct bpf_map * __must_check bpf_map_inc_not_zero(struct bpf_map *map);
void bpf_map_put_with_uref(struct bpf_map *map);
void bpf_map_put(struct bpf_map *map);
void *bpf_map_area_alloc(u64 size, int numa_node);
void *bpf_map_area_mmapable_alloc(u64 size, int numa_node);
void bpf_map_area_free(void *base);
bool bpf_map_write_active(const struct bpf_map *map);
void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr);
int generic_map_lookup_batch(struct bpf_map *map,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
int generic_map_update_batch(struct bpf_map *map, struct file *map_file,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
int generic_map_delete_batch(struct bpf_map *map,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
struct bpf_map *bpf_map_get_curr_or_next(u32 *id);
struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id);
int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
unsigned long nr_pages, struct page **page_array);
#ifdef CONFIG_MEMCG
void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg,
struct mem_cgroup **new_memcg);
void bpf_map_memcg_exit(struct mem_cgroup *old_memcg,
struct mem_cgroup *memcg);
void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags,
int node);
void *bpf_map_kmalloc_nolock(const struct bpf_map *map, size_t size, gfp_t flags,
int node);
void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags);
void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size,
gfp_t flags);
void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size,
size_t align, gfp_t flags);
#else
/*
* These specialized allocators have to be macros for their allocations to be
* accounted separately (to have separate alloc_tag).
*/
#define bpf_map_kmalloc_node(_map, _size, _flags, _node) \
kmalloc_node(_size, _flags, _node)
#define bpf_map_kmalloc_nolock(_map, _size, _flags, _node) \
kmalloc_nolock(_size, _flags, _node)
#define bpf_map_kzalloc(_map, _size, _flags) \
kzalloc(_size, _flags)
#define bpf_map_kvcalloc(_map, _n, _size, _flags) \
kvcalloc(_n, _size, _flags)
#define bpf_map_alloc_percpu(_map, _size, _align, _flags) \
__alloc_percpu_gfp(_size, _align, _flags)
static inline void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg,
struct mem_cgroup **new_memcg)
{
*new_memcg = NULL;
*old_memcg = NULL;
}
static inline void bpf_map_memcg_exit(struct mem_cgroup *old_memcg,
struct mem_cgroup *memcg)
{
}
#endif
static inline int
bpf_map_init_elem_count(struct bpf_map *map)
{
size_t size = sizeof(*map->elem_count), align = size;
gfp_t flags = GFP_USER | __GFP_NOWARN;
map->elem_count = bpf_map_alloc_percpu(map, size, align, flags);
if (!map->elem_count)
return -ENOMEM;
return 0;
}
static inline void
bpf_map_free_elem_count(struct bpf_map *map)
{
free_percpu(map->elem_count);
}
static inline void bpf_map_inc_elem_count(struct bpf_map *map)
{
this_cpu_inc(*map->elem_count);
}
static inline void bpf_map_dec_elem_count(struct bpf_map *map)
{
this_cpu_dec(*map->elem_count);
}
extern int sysctl_unprivileged_bpf_disabled;
bool bpf_token_capable(const struct bpf_token *token, int cap);
static inline bool bpf_allow_ptr_leaks(const struct bpf_token *token)
{
return bpf_token_capable(token, CAP_PERFMON);
}
static inline bool bpf_allow_uninit_stack(const struct bpf_token *token)
{
return bpf_token_capable(token, CAP_PERFMON);
}
static inline bool bpf_bypass_spec_v1(const struct bpf_token *token)
{
return bpf_jit_bypass_spec_v1() ||
cpu_mitigations_off() ||
bpf_token_capable(token, CAP_PERFMON);
}
static inline bool bpf_bypass_spec_v4(const struct bpf_token *token)
{
return bpf_jit_bypass_spec_v4() ||
cpu_mitigations_off() ||
bpf_token_capable(token, CAP_PERFMON);
}
int bpf_map_new_fd(struct bpf_map *map, int flags);
int bpf_prog_new_fd(struct bpf_prog *prog);
void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type);
void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, bool sleepable);
int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer);
int bpf_link_settle(struct bpf_link_primer *primer);
void bpf_link_cleanup(struct bpf_link_primer *primer);
void bpf_link_inc(struct bpf_link *link);
struct bpf_link *bpf_link_inc_not_zero(struct bpf_link *link);
void bpf_link_put(struct bpf_link *link);
int bpf_link_new_fd(struct bpf_link *link);
struct bpf_link *bpf_link_get_from_fd(u32 ufd);
struct bpf_link *bpf_link_get_curr_or_next(u32 *id);
void bpf_token_inc(struct bpf_token *token);
void bpf_token_put(struct bpf_token *token);
int bpf_token_create(union bpf_attr *attr);
struct bpf_token *bpf_token_get_from_fd(u32 ufd);
int bpf_token_get_info_by_fd(struct bpf_token *token,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
bool bpf_token_allow_cmd(const struct bpf_token *token, enum bpf_cmd cmd);
bool bpf_token_allow_map_type(const struct bpf_token *token, enum bpf_map_type type);
bool bpf_token_allow_prog_type(const struct bpf_token *token,
enum bpf_prog_type prog_type,
enum bpf_attach_type attach_type);
int bpf_obj_pin_user(u32 ufd, int path_fd, const char __user *pathname);
int bpf_obj_get_user(int path_fd, const char __user *pathname, int flags);
struct inode *bpf_get_inode(struct super_block *sb, const struct inode *dir,
umode_t mode);
#define BPF_ITER_FUNC_PREFIX "bpf_iter_"
#define DEFINE_BPF_ITER_FUNC(target, args...) \
extern int bpf_iter_ ## target(args); \
int __init bpf_iter_ ## target(args) { return 0; }
/*
* The task type of iterators.
*
* For BPF task iterators, they can be parameterized with various
* parameters to visit only some of tasks.
*
* BPF_TASK_ITER_ALL (default)
* Iterate over resources of every task.
*
* BPF_TASK_ITER_TID
* Iterate over resources of a task/tid.
*
* BPF_TASK_ITER_TGID
* Iterate over resources of every task of a process / task group.
*/
enum bpf_iter_task_type {
BPF_TASK_ITER_ALL = 0,
BPF_TASK_ITER_TID,
BPF_TASK_ITER_TGID,
};
struct bpf_iter_aux_info {
/* for map_elem iter */
struct bpf_map *map;
/* for cgroup iter */
struct {
struct cgroup *start; /* starting cgroup */
enum bpf_cgroup_iter_order order;
} cgroup;
struct {
enum bpf_iter_task_type type;
u32 pid;
} task;
};
typedef int (*bpf_iter_attach_target_t)(struct bpf_prog *prog,
union bpf_iter_link_info *linfo,
struct bpf_iter_aux_info *aux);
typedef void (*bpf_iter_detach_target_t)(struct bpf_iter_aux_info *aux);
typedef void (*bpf_iter_show_fdinfo_t) (const struct bpf_iter_aux_info *aux,
struct seq_file *seq);
typedef int (*bpf_iter_fill_link_info_t)(const struct bpf_iter_aux_info *aux,
struct bpf_link_info *info);
typedef const struct bpf_func_proto *
(*bpf_iter_get_func_proto_t)(enum bpf_func_id func_id,
const struct bpf_prog *prog);
enum bpf_iter_feature {
BPF_ITER_RESCHED = BIT(0),
};
#define BPF_ITER_CTX_ARG_MAX 2
struct bpf_iter_reg {
const char *target;
bpf_iter_attach_target_t attach_target;
bpf_iter_detach_target_t detach_target;
bpf_iter_show_fdinfo_t show_fdinfo;
bpf_iter_fill_link_info_t fill_link_info;
bpf_iter_get_func_proto_t get_func_proto;
u32 ctx_arg_info_size;
u32 feature;
struct bpf_ctx_arg_aux ctx_arg_info[BPF_ITER_CTX_ARG_MAX];
const struct bpf_iter_seq_info *seq_info;
};
struct bpf_iter_meta {
__bpf_md_ptr(struct seq_file *, seq);
u64 session_id;
u64 seq_num;
};
struct bpf_iter__bpf_map_elem {
__bpf_md_ptr(struct bpf_iter_meta *, meta);
__bpf_md_ptr(struct bpf_map *, map);
__bpf_md_ptr(void *, key);
__bpf_md_ptr(void *, value);
};
int bpf_iter_reg_target(const struct bpf_iter_reg *reg_info);
void bpf_iter_unreg_target(const struct bpf_iter_reg *reg_info);
int bpf_iter_prog_supported(struct bpf_prog *prog);
const struct bpf_func_proto *
bpf_iter_get_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
int bpf_iter_link_attach(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_prog *prog);
int bpf_iter_new_fd(struct bpf_link *link);
bool bpf_link_is_iter(struct bpf_link *link);
struct bpf_prog *bpf_iter_get_info(struct bpf_iter_meta *meta, bool in_stop);
int bpf_iter_run_prog(struct bpf_prog *prog, void *ctx);
void bpf_iter_map_show_fdinfo(const struct bpf_iter_aux_info *aux,
struct seq_file *seq);
int bpf_iter_map_fill_link_info(const struct bpf_iter_aux_info *aux,
struct bpf_link_info *info);
int map_set_for_each_callback_args(struct bpf_verifier_env *env,
struct bpf_func_state *caller,
struct bpf_func_state *callee);
int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value, u64 flags);
int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value, u64 flags);
int bpf_percpu_hash_update(struct bpf_map *map, void *key, void *value,
u64 flags);
int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
u64 flags);
int bpf_stackmap_extract(struct bpf_map *map, void *key, void *value, bool delete);
int bpf_fd_array_map_update_elem(struct bpf_map *map, struct file *map_file,
void *key, void *value, u64 map_flags);
int bpf_fd_array_map_lookup_elem(struct bpf_map *map, void *key, u32 *value);
int bpf_fd_htab_map_update_elem(struct bpf_map *map, struct file *map_file,
void *key, void *value, u64 map_flags);
int bpf_fd_htab_map_lookup_elem(struct bpf_map *map, void *key, u32 *value);
int bpf_get_file_flag(int flags);
int bpf_check_uarg_tail_zero(bpfptr_t uaddr, size_t expected_size,
size_t actual_size);
/* verify correctness of eBPF program */
int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size);
#ifndef CONFIG_BPF_JIT_ALWAYS_ON
void bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth);
#endif
struct btf *bpf_get_btf_vmlinux(void);
/* Map specifics */
struct xdp_frame;
struct sk_buff;
struct bpf_dtab_netdev;
struct bpf_cpu_map_entry;
void __dev_flush(struct list_head *flush_list);
int dev_xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf,
struct net_device *dev_rx);
int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_frame *xdpf,
struct net_device *dev_rx);
int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx,
struct bpf_map *map, bool exclude_ingress);
int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb,
const struct bpf_prog *xdp_prog);
int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb,
const struct bpf_prog *xdp_prog,
struct bpf_map *map, bool exclude_ingress);
void __cpu_map_flush(struct list_head *flush_list);
int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf,
struct net_device *dev_rx);
int cpu_map_generic_redirect(struct bpf_cpu_map_entry *rcpu,
struct sk_buff *skb);
/* Return map's numa specified by userspace */
static inline int bpf_map_attr_numa_node(const union bpf_attr *attr)
{
return (attr->map_flags & BPF_F_NUMA_NODE) ?
attr->numa_node : NUMA_NO_NODE;
}
struct bpf_prog *bpf_prog_get_type_path(const char *name, enum bpf_prog_type type);
int array_map_alloc_check(union bpf_attr *attr);
int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_tracing(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_raw_tp(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_sk_lookup(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_nf(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
bool btf_ctx_access(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info);
static inline bool bpf_tracing_ctx_access(int off, int size,
enum bpf_access_type type)
{
if (off < 0 || off >= sizeof(__u64) * MAX_BPF_FUNC_ARGS)
return false;
if (type != BPF_READ)
return false;
if (off % size != 0)
return false;
return true;
}
static inline bool bpf_tracing_btf_ctx_access(int off, int size,
enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
if (!bpf_tracing_ctx_access(off, size, type))
return false;
return btf_ctx_access(off, size, type, prog, info);
}
int btf_struct_access(struct bpf_verifier_log *log,
const struct bpf_reg_state *reg,
int off, int size, enum bpf_access_type atype,
u32 *next_btf_id, enum bpf_type_flag *flag, const char **field_name);
bool btf_struct_ids_match(struct bpf_verifier_log *log,
const struct btf *btf, u32 id, int off,
const struct btf *need_btf, u32 need_type_id,
bool strict);
int btf_distill_func_proto(struct bpf_verifier_log *log,
struct btf *btf,
const struct btf_type *func_proto,
const char *func_name,
struct btf_func_model *m);
struct bpf_reg_state;
int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog);
int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog,
struct btf *btf, const struct btf_type *t);
const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt,
int comp_idx, const char *tag_key);
int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *pt,
int comp_idx, const char *tag_key, int last_id);
struct bpf_prog *bpf_prog_by_id(u32 id);
struct bpf_link *bpf_link_by_id(u32 id);
const struct bpf_func_proto *bpf_base_func_proto(enum bpf_func_id func_id,
const struct bpf_prog *prog);
void bpf_task_storage_free(struct task_struct *task);
void bpf_cgrp_storage_free(struct cgroup *cgroup);
bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog);
const struct btf_func_model *
bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
const struct bpf_insn *insn);
int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
u16 btf_fd_idx, u8 **func_addr);
struct bpf_core_ctx {
struct bpf_verifier_log *log;
const struct btf *btf;
};
bool btf_nested_type_is_trusted(struct bpf_verifier_log *log,
const struct bpf_reg_state *reg,
const char *field_name, u32 btf_id, const char *suffix);
bool btf_type_ids_nocast_alias(struct bpf_verifier_log *log,
const struct btf *reg_btf, u32 reg_id,
const struct btf *arg_btf, u32 arg_id);
int bpf_core_apply(struct bpf_core_ctx *ctx, const struct bpf_core_relo *relo,
int relo_idx, void *insn);
static inline bool unprivileged_ebpf_enabled(void)
{
return !sysctl_unprivileged_bpf_disabled;
}
/* Not all bpf prog type has the bpf_ctx.
* For the bpf prog type that has initialized the bpf_ctx,
* this function can be used to decide if a kernel function
* is called by a bpf program.
*/
static inline bool has_current_bpf_ctx(void)
{
return !!current->bpf_ctx;
}
void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size);
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr);
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr);
void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip);
#else /* !CONFIG_BPF_SYSCALL */
static inline struct bpf_prog *bpf_prog_get(u32 ufd)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline struct bpf_prog *bpf_prog_get_type_dev(u32 ufd,
enum bpf_prog_type type,
bool attach_drv)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline void bpf_prog_add(struct bpf_prog *prog, int i)
{
}
static inline void bpf_prog_sub(struct bpf_prog *prog, int i)
{
}
static inline void bpf_prog_put(struct bpf_prog *prog)
{
}
static inline void bpf_prog_inc(struct bpf_prog *prog)
{
}
static inline struct bpf_prog *__must_check
bpf_prog_inc_not_zero(struct bpf_prog *prog)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops,
struct bpf_prog *prog, enum bpf_attach_type attach_type)
{
}
static inline void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, bool sleepable)
{
}
static inline int bpf_link_prime(struct bpf_link *link,
struct bpf_link_primer *primer)
{
return -EOPNOTSUPP;
}
static inline int bpf_link_settle(struct bpf_link_primer *primer)
{
return -EOPNOTSUPP;
}
static inline void bpf_link_cleanup(struct bpf_link_primer *primer)
{
}
static inline void bpf_link_inc(struct bpf_link *link)
{
}
static inline struct bpf_link *bpf_link_inc_not_zero(struct bpf_link *link)
{
return NULL;
}
static inline void bpf_link_put(struct bpf_link *link)
{
}
static inline int bpf_obj_get_user(const char __user *pathname, int flags)
{
return -EOPNOTSUPP;
}
static inline bool bpf_token_capable(const struct bpf_token *token, int cap)
{
return capable(cap) || (cap != CAP_SYS_ADMIN && capable(CAP_SYS_ADMIN));
}
static inline void bpf_token_inc(struct bpf_token *token)
{
}
static inline void bpf_token_put(struct bpf_token *token)
{
}
static inline struct bpf_token *bpf_token_get_from_fd(u32 ufd)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline int bpf_token_get_info_by_fd(struct bpf_token *token,
const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
return -EOPNOTSUPP;
}
static inline void __dev_flush(struct list_head *flush_list)
{
}
struct xdp_frame;
struct bpf_dtab_netdev;
struct bpf_cpu_map_entry;
static inline
int dev_xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf,
struct net_device *dev_rx)
{
return 0;
}
static inline
int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_frame *xdpf,
struct net_device *dev_rx)
{
return 0;
}
static inline
int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx,
struct bpf_map *map, bool exclude_ingress)
{
return 0;
}
struct sk_buff;
static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst,
struct sk_buff *skb,
const struct bpf_prog *xdp_prog)
{
return 0;
}
static inline
int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb,
const struct bpf_prog *xdp_prog,
struct bpf_map *map, bool exclude_ingress)
{
return 0;
}
static inline void __cpu_map_flush(struct list_head *flush_list)
{
}
static inline int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu,
struct xdp_frame *xdpf,
struct net_device *dev_rx)
{
return 0;
}
static inline int cpu_map_generic_redirect(struct bpf_cpu_map_entry *rcpu,
struct sk_buff *skb)
{
return -EOPNOTSUPP;
}
static inline struct bpf_prog *bpf_prog_get_type_path(const char *name,
enum bpf_prog_type type)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline int bpf_prog_test_run_xdp(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
static inline int bpf_prog_test_run_skb(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
static inline int bpf_prog_test_run_tracing(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
static inline int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
static inline int bpf_prog_test_run_sk_lookup(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
static inline void bpf_map_put(struct bpf_map *map)
{
}
static inline struct bpf_prog *bpf_prog_by_id(u32 id)
{
return ERR_PTR(-ENOTSUPP);
}
static inline int btf_struct_access(struct bpf_verifier_log *log,
const struct bpf_reg_state *reg,
int off, int size, enum bpf_access_type atype,
u32 *next_btf_id, enum bpf_type_flag *flag,
const char **field_name)
{
return -EACCES;
}
static inline const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
return NULL;
}
static inline void bpf_task_storage_free(struct task_struct *task)
{
}
static inline bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog)
{
return false;
}
static inline const struct btf_func_model *
bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
const struct bpf_insn *insn)
{
return NULL;
}
static inline int
bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
u16 btf_fd_idx, u8 **func_addr)
{
return -ENOTSUPP;
}
static inline bool unprivileged_ebpf_enabled(void)
{
return false;
}
static inline bool has_current_bpf_ctx(void)
{
return false;
}
static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)
{
}
static inline void bpf_cgrp_storage_free(struct cgroup *cgroup)
{
}
static inline void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size)
{
}
static inline void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)
{
}
static inline void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
{
}
static inline void bpf_prog_report_arena_violation(bool write, unsigned long addr,
unsigned long fault_ip)
{
}
#endif /* CONFIG_BPF_SYSCALL */
static inline bool bpf_net_capable(void)
{
return capable(CAP_NET_ADMIN) || capable(CAP_SYS_ADMIN);
}
static __always_inline int
bpf_probe_read_kernel_common(void *dst, u32 size, const void *unsafe_ptr)
{
int ret = -EFAULT;
if (IS_ENABLED(CONFIG_BPF_EVENTS))
ret = copy_from_kernel_nofault(dst, unsafe_ptr, size);
if (unlikely(ret < 0))
memset(dst, 0, size);
return ret;
}
void __bpf_free_used_btfs(struct btf_mod_pair *used_btfs, u32 len);
static inline struct bpf_prog *bpf_prog_get_type(u32 ufd,
enum bpf_prog_type type)
{
return bpf_prog_get_type_dev(ufd, type, false);
}
void __bpf_free_used_maps(struct bpf_prog_aux *aux,
struct bpf_map **used_maps, u32 len);
bool bpf_prog_get_ok(struct bpf_prog *, enum bpf_prog_type *, bool);
int bpf_prog_offload_compile(struct bpf_prog *prog);
void bpf_prog_dev_bound_destroy(struct bpf_prog *prog);
int bpf_prog_offload_info_fill(struct bpf_prog_info *info,
struct bpf_prog *prog);
int bpf_map_offload_info_fill(struct bpf_map_info *info, struct bpf_map *map);
int bpf_map_offload_lookup_elem(struct bpf_map *map, void *key, void *value);
int bpf_map_offload_update_elem(struct bpf_map *map,
void *key, void *value, u64 flags);
int bpf_map_offload_delete_elem(struct bpf_map *map, void *key);
int bpf_map_offload_get_next_key(struct bpf_map *map,
void *key, void *next_key);
bool bpf_offload_prog_map_match(struct bpf_prog *prog, struct bpf_map *map);
struct bpf_offload_dev *
bpf_offload_dev_create(const struct bpf_prog_offload_ops *ops, void *priv);
void bpf_offload_dev_destroy(struct bpf_offload_dev *offdev);
void *bpf_offload_dev_priv(struct bpf_offload_dev *offdev);
int bpf_offload_dev_netdev_register(struct bpf_offload_dev *offdev,
struct net_device *netdev);
void bpf_offload_dev_netdev_unregister(struct bpf_offload_dev *offdev,
struct net_device *netdev);
bool bpf_offload_dev_match(struct bpf_prog *prog, struct net_device *netdev);
void unpriv_ebpf_notify(int new_state);
#if defined(CONFIG_NET) && defined(CONFIG_BPF_SYSCALL)
int bpf_dev_bound_kfunc_check(struct bpf_verifier_log *log,
struct bpf_prog_aux *prog_aux);
void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id);
int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr);
int bpf_prog_dev_bound_inherit(struct bpf_prog *new_prog, struct bpf_prog *old_prog);
void bpf_dev_bound_netdev_unregister(struct net_device *dev);
static inline bool bpf_prog_is_dev_bound(const struct bpf_prog_aux *aux)
{
return aux->dev_bound;
}
static inline bool bpf_prog_is_offloaded(const struct bpf_prog_aux *aux)
{
return aux->offload_requested;
}
bool bpf_prog_dev_bound_match(const struct bpf_prog *lhs, const struct bpf_prog *rhs);
static inline bool bpf_map_is_offloaded(struct bpf_map *map)
{
return unlikely(map->ops == &bpf_map_offload_ops);
}
struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr);
void bpf_map_offload_map_free(struct bpf_map *map);
u64 bpf_map_offload_map_mem_usage(const struct bpf_map *map);
int bpf_prog_test_run_syscall(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int sock_map_get_from_fd(const union bpf_attr *attr, struct bpf_prog *prog);
int sock_map_prog_detach(const union bpf_attr *attr, enum bpf_prog_type ptype);
int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value, u64 flags);
int sock_map_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr);
int sock_map_link_create(const union bpf_attr *attr, struct bpf_prog *prog);
void sock_map_unhash(struct sock *sk);
void sock_map_destroy(struct sock *sk);
void sock_map_close(struct sock *sk, long timeout);
#else
static inline int bpf_dev_bound_kfunc_check(struct bpf_verifier_log *log,
struct bpf_prog_aux *prog_aux)
{
return -EOPNOTSUPP;
}
static inline void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog,
u32 func_id)
{
return NULL;
}
static inline int bpf_prog_dev_bound_init(struct bpf_prog *prog,
union bpf_attr *attr)
{
return -EOPNOTSUPP;
}
static inline int bpf_prog_dev_bound_inherit(struct bpf_prog *new_prog,
struct bpf_prog *old_prog)
{
return -EOPNOTSUPP;
}
static inline void bpf_dev_bound_netdev_unregister(struct net_device *dev)
{
}
static inline bool bpf_prog_is_dev_bound(const struct bpf_prog_aux *aux)
{
return false;
}
static inline bool bpf_prog_is_offloaded(struct bpf_prog_aux *aux)
{
return false;
}
static inline bool bpf_prog_dev_bound_match(const struct bpf_prog *lhs, const struct bpf_prog *rhs)
{
return false;
}
static inline bool bpf_map_is_offloaded(struct bpf_map *map)
{
return false;
}
static inline struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline void bpf_map_offload_map_free(struct bpf_map *map)
{
}
static inline u64 bpf_map_offload_map_mem_usage(const struct bpf_map *map)
{
return 0;
}
static inline int bpf_prog_test_run_syscall(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
#ifdef CONFIG_BPF_SYSCALL
static inline int sock_map_get_from_fd(const union bpf_attr *attr,
struct bpf_prog *prog)
{
return -EINVAL;
}
static inline int sock_map_prog_detach(const union bpf_attr *attr,
enum bpf_prog_type ptype)
{
return -EOPNOTSUPP;
}
static inline int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value,
u64 flags)
{
return -EOPNOTSUPP;
}
static inline int sock_map_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
return -EINVAL;
}
static inline int sock_map_link_create(const union bpf_attr *attr, struct bpf_prog *prog)
{
return -EOPNOTSUPP;
}
#endif /* CONFIG_BPF_SYSCALL */
#endif /* CONFIG_NET && CONFIG_BPF_SYSCALL */
static __always_inline void
bpf_prog_inc_misses_counters(const struct bpf_prog_array *array)
{
const struct bpf_prog_array_item *item;
struct bpf_prog *prog;
if (unlikely(!array))
return;
item = &array->items[0];
while ((prog = READ_ONCE(item->prog))) {
bpf_prog_inc_misses_counter(prog);
item++;
}
}
#if defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL)
void bpf_sk_reuseport_detach(struct sock *sk);
int bpf_fd_reuseport_array_lookup_elem(struct bpf_map *map, void *key,
void *value);
int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,
void *value, u64 map_flags);
#else
static inline void bpf_sk_reuseport_detach(struct sock *sk)
{
}
#ifdef CONFIG_BPF_SYSCALL
static inline int bpf_fd_reuseport_array_lookup_elem(struct bpf_map *map,
void *key, void *value)
{
return -EOPNOTSUPP;
}
static inline int bpf_fd_reuseport_array_update_elem(struct bpf_map *map,
void *key, void *value,
u64 map_flags)
{
return -EOPNOTSUPP;
}
#endif /* CONFIG_BPF_SYSCALL */
#endif /* defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL) */
#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL)
struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags);
struct bpf_key *bpf_lookup_system_key(u64 id);
void bpf_key_put(struct bpf_key *bkey);
int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring);
#else
static inline struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags)
{
return NULL;
}
static inline struct bpf_key *bpf_lookup_system_key(u64 id)
{
return NULL;
}
static inline void bpf_key_put(struct bpf_key *bkey)
{
}
static inline int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
return -EOPNOTSUPP;
}
#endif /* defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) */
/* verifier prototypes for helper functions called from eBPF programs */
extern const struct bpf_func_proto bpf_map_lookup_elem_proto;
extern const struct bpf_func_proto bpf_map_update_elem_proto;
extern const struct bpf_func_proto bpf_map_delete_elem_proto;
extern const struct bpf_func_proto bpf_map_push_elem_proto;
extern const struct bpf_func_proto bpf_map_pop_elem_proto;
extern const struct bpf_func_proto bpf_map_peek_elem_proto;
extern const struct bpf_func_proto bpf_map_lookup_percpu_elem_proto;
extern const struct bpf_func_proto bpf_get_prandom_u32_proto;
extern const struct bpf_func_proto bpf_get_smp_processor_id_proto;
extern const struct bpf_func_proto bpf_get_numa_node_id_proto;
extern const struct bpf_func_proto bpf_tail_call_proto;
extern const struct bpf_func_proto bpf_ktime_get_ns_proto;
extern const struct bpf_func_proto bpf_ktime_get_boot_ns_proto;
extern const struct bpf_func_proto bpf_ktime_get_tai_ns_proto;
extern const struct bpf_func_proto bpf_get_current_pid_tgid_proto;
extern const struct bpf_func_proto bpf_get_current_uid_gid_proto;
extern const struct bpf_func_proto bpf_get_current_comm_proto;
extern const struct bpf_func_proto bpf_get_stackid_proto;
extern const struct bpf_func_proto bpf_get_stack_proto;
extern const struct bpf_func_proto bpf_get_stack_sleepable_proto;
extern const struct bpf_func_proto bpf_get_task_stack_proto;
extern const struct bpf_func_proto bpf_get_task_stack_sleepable_proto;
extern const struct bpf_func_proto bpf_get_stackid_proto_pe;
extern const struct bpf_func_proto bpf_get_stack_proto_pe;
extern const struct bpf_func_proto bpf_sock_map_update_proto;
extern const struct bpf_func_proto bpf_sock_hash_update_proto;
extern const struct bpf_func_proto bpf_get_current_cgroup_id_proto;
extern const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto;
extern const struct bpf_func_proto bpf_get_cgroup_classid_curr_proto;
extern const struct bpf_func_proto bpf_current_task_under_cgroup_proto;
extern const struct bpf_func_proto bpf_msg_redirect_hash_proto;
extern const struct bpf_func_proto bpf_msg_redirect_map_proto;
extern const struct bpf_func_proto bpf_sk_redirect_hash_proto;
extern const struct bpf_func_proto bpf_sk_redirect_map_proto;
extern const struct bpf_func_proto bpf_spin_lock_proto;
extern const struct bpf_func_proto bpf_spin_unlock_proto;
extern const struct bpf_func_proto bpf_get_local_storage_proto;
extern const struct bpf_func_proto bpf_strtol_proto;
extern const struct bpf_func_proto bpf_strtoul_proto;
extern const struct bpf_func_proto bpf_tcp_sock_proto;
extern const struct bpf_func_proto bpf_jiffies64_proto;
extern const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto;
extern const struct bpf_func_proto bpf_event_output_data_proto;
extern const struct bpf_func_proto bpf_ringbuf_output_proto;
extern const struct bpf_func_proto bpf_ringbuf_reserve_proto;
extern const struct bpf_func_proto bpf_ringbuf_submit_proto;
extern const struct bpf_func_proto bpf_ringbuf_discard_proto;
extern const struct bpf_func_proto bpf_ringbuf_query_proto;
extern const struct bpf_func_proto bpf_ringbuf_reserve_dynptr_proto;
extern const struct bpf_func_proto bpf_ringbuf_submit_dynptr_proto;
extern const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto;
extern const struct bpf_func_proto bpf_skc_to_tcp6_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_tcp_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_tcp_timewait_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_tcp_request_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_udp6_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_unix_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_mptcp_sock_proto;
extern const struct bpf_func_proto bpf_copy_from_user_proto;
extern const struct bpf_func_proto bpf_snprintf_btf_proto;
extern const struct bpf_func_proto bpf_snprintf_proto;
extern const struct bpf_func_proto bpf_per_cpu_ptr_proto;
extern const struct bpf_func_proto bpf_this_cpu_ptr_proto;
extern const struct bpf_func_proto bpf_ktime_get_coarse_ns_proto;
extern const struct bpf_func_proto bpf_sock_from_file_proto;
extern const struct bpf_func_proto bpf_get_socket_ptr_cookie_proto;
extern const struct bpf_func_proto bpf_task_storage_get_recur_proto;
extern const struct bpf_func_proto bpf_task_storage_get_proto;
extern const struct bpf_func_proto bpf_task_storage_delete_recur_proto;
extern const struct bpf_func_proto bpf_task_storage_delete_proto;
extern const struct bpf_func_proto bpf_for_each_map_elem_proto;
extern const struct bpf_func_proto bpf_btf_find_by_name_kind_proto;
extern const struct bpf_func_proto bpf_sk_setsockopt_proto;
extern const struct bpf_func_proto bpf_sk_getsockopt_proto;
extern const struct bpf_func_proto bpf_sk_setsockopt_nodelay_proto;
extern const struct bpf_func_proto bpf_unlocked_sk_setsockopt_proto;
extern const struct bpf_func_proto bpf_unlocked_sk_getsockopt_proto;
extern const struct bpf_func_proto bpf_find_vma_proto;
extern const struct bpf_func_proto bpf_loop_proto;
extern const struct bpf_func_proto bpf_copy_from_user_task_proto;
extern const struct bpf_func_proto bpf_set_retval_proto;
extern const struct bpf_func_proto bpf_get_retval_proto;
extern const struct bpf_func_proto bpf_user_ringbuf_drain_proto;
extern const struct bpf_func_proto bpf_cgrp_storage_get_proto;
extern const struct bpf_func_proto bpf_cgrp_storage_delete_proto;
const struct bpf_func_proto *tracing_prog_func_proto(
enum bpf_func_id func_id, const struct bpf_prog *prog);
/* Shared helpers among cBPF and eBPF. */
void bpf_user_rnd_init_once(void);
u64 bpf_user_rnd_u32(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
#if defined(CONFIG_NET)
bool bpf_sock_common_is_valid_access(int off, int size,
enum bpf_access_type type,
struct bpf_insn_access_aux *info);
bool bpf_sock_is_valid_access(int off, int size, enum bpf_access_type type,
struct bpf_insn_access_aux *info);
u32 bpf_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size);
int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags,
struct bpf_dynptr *ptr);
#else
static inline bool bpf_sock_common_is_valid_access(int off, int size,
enum bpf_access_type type,
struct bpf_insn_access_aux *info)
{
return false;
}
static inline bool bpf_sock_is_valid_access(int off, int size,
enum bpf_access_type type,
struct bpf_insn_access_aux *info)
{
return false;
}
static inline u32 bpf_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size)
{
return 0;
}
static inline int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags,
struct bpf_dynptr *ptr)
{
return -EOPNOTSUPP;
}
#endif
#ifdef CONFIG_INET
struct sk_reuseport_kern {
struct sk_buff *skb;
struct sock *sk;
struct sock *selected_sk;
struct sock *migrating_sk;
void *data_end;
u32 hash;
u32 reuseport_id;
bool bind_inany;
};
bool bpf_tcp_sock_is_valid_access(int off, int size, enum bpf_access_type type,
struct bpf_insn_access_aux *info);
u32 bpf_tcp_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size);
bool bpf_xdp_sock_is_valid_access(int off, int size, enum bpf_access_type type,
struct bpf_insn_access_aux *info);
u32 bpf_xdp_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size);
#else
static inline bool bpf_tcp_sock_is_valid_access(int off, int size,
enum bpf_access_type type,
struct bpf_insn_access_aux *info)
{
return false;
}
static inline u32 bpf_tcp_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size)
{
return 0;
}
static inline bool bpf_xdp_sock_is_valid_access(int off, int size,
enum bpf_access_type type,
struct bpf_insn_access_aux *info)
{
return false;
}
static inline u32 bpf_xdp_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size)
{
return 0;
}
#endif /* CONFIG_INET */
enum bpf_text_poke_type {
BPF_MOD_NOP,
BPF_MOD_CALL,
BPF_MOD_JUMP,
};
int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
enum bpf_text_poke_type new_t, void *old_addr,
void *new_addr);
void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke,
struct bpf_prog *new, struct bpf_prog *old);
void *bpf_arch_text_copy(void *dst, void *src, size_t len);
int bpf_arch_text_invalidate(void *dst, size_t len);
struct btf_id_set;
bool btf_id_set_contains(const struct btf_id_set *set, u32 id);
#define MAX_BPRINTF_VARARGS 12
#define MAX_BPRINTF_BUF 1024
/* Per-cpu temp buffers used by printf-like helpers to store the bprintf binary
* arguments representation.
*/
#define MAX_BPRINTF_BIN_ARGS 512
struct bpf_bprintf_buffers {
char bin_args[MAX_BPRINTF_BIN_ARGS];
char buf[MAX_BPRINTF_BUF];
};
struct bpf_bprintf_data {
u32 *bin_args;
char *buf;
bool get_bin_args;
bool get_buf;
};
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data);
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs);
void bpf_put_buffers(void);
void bpf_prog_stream_init(struct bpf_prog *prog);
void bpf_prog_stream_free(struct bpf_prog *prog);
int bpf_prog_stream_read(struct bpf_prog *prog, enum bpf_stream_id stream_id, void __user *buf, int len);
void bpf_stream_stage_init(struct bpf_stream_stage *ss);
void bpf_stream_stage_free(struct bpf_stream_stage *ss);
__printf(2, 3)
int bpf_stream_stage_printk(struct bpf_stream_stage *ss, const char *fmt, ...);
int bpf_stream_stage_commit(struct bpf_stream_stage *ss, struct bpf_prog *prog,
enum bpf_stream_id stream_id);
int bpf_stream_stage_dump_stack(struct bpf_stream_stage *ss);
#define bpf_stream_printk(ss, ...) bpf_stream_stage_printk(&ss, __VA_ARGS__)
#define bpf_stream_dump_stack(ss) bpf_stream_stage_dump_stack(&ss)
#define bpf_stream_stage(ss, prog, stream_id, expr) \
({ \
bpf_stream_stage_init(&ss); \
(expr); \
bpf_stream_stage_commit(&ss, prog, stream_id); \
bpf_stream_stage_free(&ss); \
})
#ifdef CONFIG_BPF_LSM
void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype);
void bpf_cgroup_atype_put(int cgroup_atype);
#else
static inline void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) {}
static inline void bpf_cgroup_atype_put(int cgroup_atype) {}
#endif /* CONFIG_BPF_LSM */
struct key;
#ifdef CONFIG_KEYS
struct bpf_key {
struct key *key;
bool has_ref;
};
#endif /* CONFIG_KEYS */
static inline bool type_is_alloc(u32 type)
{
return type & MEM_ALLOC;
}
static inline gfp_t bpf_memcg_flags(gfp_t flags)
{
if (memcg_bpf_enabled())
return flags | __GFP_ACCOUNT;
return flags;
}
static inline bool bpf_is_subprog(const struct bpf_prog *prog)
{
return prog->aux->func_idx != 0;
}
const struct bpf_line_info *bpf_find_linfo(const struct bpf_prog *prog, u32 insn_off);
void bpf_get_linfo_file_line(struct btf *btf, const struct bpf_line_info *linfo,
const char **filep, const char **linep, int *nump);
int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char **filep,
const char **linep, int *nump);
struct bpf_prog *bpf_prog_find_from_stack(void);
int bpf_insn_array_init(struct bpf_map *map, const struct bpf_prog *prog);
int bpf_insn_array_ready(struct bpf_map *map);
void bpf_insn_array_release(struct bpf_map *map);
void bpf_insn_array_adjust(struct bpf_map *map, u32 off, u32 len);
void bpf_insn_array_adjust_after_remove(struct bpf_map *map, u32 off, u32 len);
#ifdef CONFIG_BPF_SYSCALL
void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image);
#else
static inline void
bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image)
{
}
#endif
static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type)
{
switch (map_type) {
case BPF_MAP_TYPE_PERCPU_ARRAY:
case BPF_MAP_TYPE_PERCPU_HASH:
case BPF_MAP_TYPE_LRU_PERCPU_HASH:
case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE:
return true;
default:
return false;
}
}
static inline int bpf_map_check_op_flags(struct bpf_map *map, u64 flags, u64 allowed_flags)
{
u32 cpu;
if ((u32)flags & ~allowed_flags)
return -EINVAL;
if ((flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
return -EINVAL;
if (!(flags & BPF_F_CPU) && flags >> 32)
return -EINVAL;
if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) {
if (!bpf_map_supports_cpu_flags(map->map_type))
return -EINVAL;
if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS))
return -EINVAL;
cpu = flags >> 32;
if ((flags & BPF_F_CPU) && cpu >= num_possible_cpus())
return -ERANGE;
}
return 0;
}
#endif /* _LINUX_BPF_H */
]
|
| 366/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 128125
output: 36
thoughts: 647
Thoughts: **Analyzing Potential Overflow Vulnerabilities in `kernel/bpf/helpers.c`**
Okay, so I've been looking into the memory management within the BPF printf-like helpers, and the user's question about unchecked return values from `snprintf` is key. My initial focus was on identifying instances of potential buffer overflows. I've noted the buffer size limitations: `MAX_BPRINTF_VARARGS` (12), `MAX_BPRINTF_BUF` (1024), and, crucially, `MAX_BPRINTF_BIN_ARGS` (512 bytes). It appears the `bin_args` buffer, holding intermediate representations of the arguments, is limited to 512 bytes.
I understand that `snprintf` is employed to format, specifically, `%pB` and `%pI4`/`%pI6` specifiers into this `bin_args` buffer. The problem is that the symbol name within `%pB` can be quite long. Even though there's `KSYM_NAME_LEN` (128 bytes), `%pB` also incorporates the offset and module name. This easily expands the string beyond 128 bytes, and if multiple `%pB` formats are used in a single string, the 512-byte limit on the `bin_args` buffer can easily be exceeded. I have already identified a few of these in `bpf_bprintf_prepare`.
The user's question is to identify *other* locations.
I now need to broaden the search beyond my initial findings. I'll need to check the implementation of `bpf_snprintf` itself, which, as I understand, provides a wrapper around the functionality. Let's examine `____bpf_snprintf`.
Looking at the code, it calls `bpf_bprintf_prepare`, which I've already scrutinized. Then, `bpf_snprintf` uses `bstr_printf` to actually perform the formatted output. `bstr_printf` is, from what I know, implemented in `lib/vsprintf.c`. The return value of `bstr_printf` is the number of characters written, not including the null terminator. `bpf_snprintf` returns this value + 1, which is, generally, how the standard `snprintf` works: it shows how many characters would have been written had there been sufficient space.
Given this, I have to go through the code again for other potential issues related to checking the return value.
|
| 367/6 |
2026/05/18 19:14 |
tool |
codesearch-definition-comment |
0m
Args: map[ContextFile:kernel/bpf/helpers.c Name:bstr_printf]
Results: map[Comment:/**
* bstr_printf - Format a string from binary arguments and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt_str: The format string to use
* @bin_buf: Binary arguments for the format string
*
* This function like C99 vsnprintf, but the difference is that vsnprintf gets
* arguments from stack, and bstr_printf gets arguments from @bin_buf which is
* a binary buffer that generated by vbin_printf.
*
* The format follows C99 vsnprintf, but has some extensions:
* see vsnprintf comment for details.
*
* The return value is the number of characters which would
* be generated for the given input, excluding the trailing
* '\0', as per ISO C99. If you want to have the exact
* number of characters written into @buf as return value
* (not including the trailing '\0'), use vscnprintf(). If the
* return is greater than or equal to @size, the resulting
* string is truncated.
*/
Kind:function]
|
| 368/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 129627
output: 38
thoughts: 301
Thoughts: **Analyzing `snprintf` Behavior in the BPF Kernel Helpers**
Okay, so I'm digging into how `bstr_printf` and `bpf_snprintf` work, and I'm a bit puzzled. It seems like `bstr_printf` is designed to function like `vsnprintf`, which is pretty standard. But the return value of `bpf_snprintf` being `err + 1` is raising a flag. I'm used to `snprintf` returning the number of characters written *without* including the null terminator. So, if `bstr_printf` is also playing by those rules, then `bpf_snprintf` returning `length + 1` implies it's including the null byte in the returned value. That feels slightly non-standard.
I need to confirm if my understanding of the standard `snprintf` return value is correct – it should be the *would-be* length excluding the NUL, right? If so, this offset of `+1` in `bpf_snprintf` does set it apart from standard `snprintf`. I need to remember that difference!
My main goal is understanding the calls to `snprintf` within `kernel/bpf/helpers.c`, as the prompt directed me there. Right now, I've only found two calls to `snprintf`, and both are located within `bpf_bprintf_prepare`. That seems pretty focused.
However, I need to make sure I'm not missing anything. The prompt specifically mentioned `helpers.c`, but could `snprintf` be used in another helper function, or perhaps somewhere else in the BPF kernel code? Just to be thorough, I'm going to quickly search the entire `kernel/bpf/` directory for any other instances of `snprintf`. I'll broaden the scope a bit, just to eliminate the possibility of overlooking something.
|
| 369/6 |
2026/05/18 19:14 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:snprintf SourceTreePrefix:kernel/bpf/]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__func_get_name SourceFile:kernel/bpf/disasm.c SourceLine:36 SourceSnippet: 35: if (insn->src_reg == BPF_PSEUDO_CALL)
36: snprintf(buff, len, "%+d", insn->imm);
37: else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL)
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__func_get_name SourceFile:kernel/bpf/disasm.c SourceLine:38 SourceSnippet: 37: else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL)
38: snprintf(buff, len, "kernel-function");
39:
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__func_imm_name SourceFile:kernel/bpf/disasm.c SourceLine:50 SourceSnippet: 49:
50: snprintf(buff, len, "0x%llx", (unsigned long long)full_imm);
51: return buff;
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924 SourceSnippet: 923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965 SourceSnippet: 964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_fmt_stack_mask SourceFile:kernel/bpf/backtrack.c SourceLine:239 SourceSnippet: 238: for_each_set_bit(i, mask, 64) {
239: n = snprintf(buf, buf_sz, "%s%d", first ? "" : ",", -(i + 1) * 8);
240: first = false;
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_prog_ksym_set_name SourceFile:kernel/bpf/core.c SourceLine:581 SourceSnippet: 580:
581: sym += snprintf(sym, KSYM_NAME_LEN, "bpf_prog_");
582: sym = bin2hex(sym, prog->tag, sizeof(prog->tag));
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_prog_ksym_set_name SourceFile:kernel/bpf/core.c SourceLine:589 SourceSnippet: 588: func_name = btf_name_by_offset(prog->aux->btf, type->name_off);
589: snprintf(sym, (size_t)(end - sym), "_%s", func_name);
590: return;
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_prog_ksym_set_name SourceFile:kernel/bpf/core.c SourceLine:594 SourceSnippet: 593: if (prog->aux->name[0])
594: snprintf(sym, (size_t)(end - sym), "_%s", prog->aux->name);
595: else
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_struct_ops_ksym_init SourceFile:kernel/bpf/bpf_struct_ops.c SourceLine:647 SourceSnippet: 646: {
647: snprintf(ksym->name, KSYM_NAME_LEN, "bpf__%s_%s", tname, mname);
648: INIT_LIST_HEAD_RCU(&ksym->lnode);
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_tramp_image_alloc SourceFile:kernel/bpf/trampoline.c SourceLine:592 SourceSnippet: 591: INIT_LIST_HEAD_RCU(&ksym->lnode);
592: snprintf(ksym->name, KSYM_NAME_LEN, "bpf_trampoline_%llu", key);
593: bpf_image_ksym_init(image, size, ksym);
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_check_iter_kfuncs SourceFile:kernel/bpf/btf.c SourceLine:8662 SourceSnippet:8661:
8662: snprintf(exp_name, sizeof(exp_name), "bpf_iter_%s_%s", iter_name, sfx);
8663: if (strcmp(func_name, exp_name))
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_nested_type_is_trusted SourceFile:kernel/bpf/btf.c SourceLine:9592 SourceSnippet:9591:
9592: ret = snprintf(safe_tname, sizeof(safe_tname), "%s%s", tname, suffix);
9593: if (ret >= sizeof(safe_tname))
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_show_name SourceFile:kernel/bpf/btf.c SourceLine:1215 SourceSnippet:1214: if (show->flags & BTF_SHOW_NONAME)
1215: snprintf(show->state.name, sizeof(show->state.name), "%s",
1216: parens);
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_show_name SourceFile:kernel/bpf/btf.c SourceLine:1218 SourceSnippet:1217: else
1218: snprintf(show->state.name, sizeof(show->state.name),
1219: "%s%s%s(%s%s%s%s%s%s)%s",
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:find_kfunc_impl_proto SourceFile:kernel/bpf/verifier.c SourceLine:2912 SourceSnippet:2911:
2912: len = snprintf(buf, TMP_STR_BUF_LEN, "%s%s", func_name, KF_IMPL_SUFFIX);
2913: if (len < 0 || len >= TMP_STR_BUF_LEN) {
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:fmt_instance SourceFile:kernel/bpf/liveness.c SourceLine:419 SourceSnippet: 418: {
419: snprintf(env->tmp_str_buf, sizeof(env->tmp_str_buf),
420: "(d%d,cs%d)", instance->depth, instance->callsite);
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:fmt_reg_mask SourceFile:kernel/bpf/backtrack.c SourceLine:220 SourceSnippet: 219: for_each_set_bit(i, mask, 32) {
220: n = snprintf(buf, buf_sz, "%sr%d", first ? "" : ",", i);
221: first = false;
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:fmt_subprog SourceFile:kernel/bpf/liveness.c SourceLine:412 SourceSnippet: 411:
412: snprintf(env->tmp_str_buf, sizeof(env->tmp_str_buf),
413: "subprog#%d%s%s", subprog, name ? " " : "", name ? name : "");
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:format_callchain SourceFile:kernel/bpf/states.c SourceLine:144 SourceSnippet: 143:
144: delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "(");
145: for (i = 0; i < ARRAY_SIZE(callchain->callsites); i++) {
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:format_callchain SourceFile:kernel/bpf/states.c SourceLine:148 SourceSnippet: 147: break;
148: delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "%u,",
149: callchain->callsites[i]);
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:format_callchain SourceFile:kernel/bpf/states.c SourceLine:151 SourceSnippet: 150: }
151: delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "%u)", callchain->scc);
152: return env->tmp_str_buf;
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:reg_type_str SourceFile:kernel/bpf/log.c SourceLine:435 SourceSnippet: 434:
435: snprintf(prefix, sizeof(prefix), "%s%s%s%s%s%s%s",
436: type & MEM_RDONLY ? "rdonly_" : "",
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:reg_type_str SourceFile:kernel/bpf/log.c SourceLine:445 SourceSnippet: 444:
445: snprintf(env->tmp_str_buf, TMP_STR_BUF_LEN, "%s%s%s",
446: prefix, str[base_type(type)], postfix);
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:543 SourceSnippet: 542: if (is_unum_decimal(a.value))
543: return snprintf(str, size, "%llu", a.value);
544: if (is_snum_decimal(a.value))
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:545 SourceSnippet: 544: if (is_snum_decimal(a.value))
545: return snprintf(str, size, "%lld", a.value);
546: else
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:547 SourceSnippet: 546: else
547: return snprintf(str, size, "%#llx", a.value);
548: }
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:549 SourceSnippet: 548: }
549: return snprintf(str, size, "(%#llx; %#llx)", a.value, a.mask);
550: }
]] TruncatedOutput:false]
|
| 370/6 |
2026/05/18 19:14 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 133711
output: 34
thoughts: 627
Thoughts: **Analyzing Potential Security Vulnerabilities in `kernel/bpf/helpers.c`**
Okay, so the user specifically honed in on `kernel/bpf/helpers.c`, likely looking for `snprintf` vulnerabilities. My initial assessment flagged only two calls to `snprintf`. But I'm not going to be complacent; I need to be thorough. First, I wanted to double-check my initial `codesearch-find-references` with a `grep` through the file. Though, I know I've read this file, so I already know that I will find nothing new there.
Right, let's go back and carefully re-examine the `bpf_bprintf_prepare` function. Line by line. Specifically, this block:
```c
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
930:
931: i++;
932: num_spec++;
933: continue;
934: }
```
Wait a second... what if `tmp_buf` is NULL? The code handles this with `if (tmp_buf)`. Okay, good. But if `tmp_buf` is *not* NULL, then there's an `snprintf`. And the return value, `err`, *is* the number of characters that would be written (not necessarily the number actually written), and if `err >= (tmp_buf_end - tmp_buf)`, that means truncation occurred. The real problem here is that `tmp_buf` is then incremented by `err + 1`. If `err` is large and truncation occurred, then this looks like a *significant* bug. Potential buffer overflow alert!
Alright, time to expand my search. Is there *anywhere* else `snprintf` might be lurking in `kernel/bpf/helpers.c`? I'll search again, and this time, I'll be more mindful of any variations I might have missed. Ah, I see a reference to `bpf_snprintf_btf`. Interesting.
```c
extern const struct bpf_func_proto bpf_snprintf_btf_proto;
```
I didn't see it when I first went through, so I need to find the definition of this. It's probably in either `kernel/bpf/btf.c` or, I suppose, *potentially* still in `kernel/bpf/helpers.c`, though unlikely since I just went through the file. I know I didn't see it in `helpers.c`, so now I'll look over in `kernel/bpf/btf.c`.
|
| 371/6 |
2026/05/18 19:15 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf_proto]
Results: map[References:<nil> TruncatedOutput:false]
|
| 372/6 |
2026/05/18 19:15 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 135093
output: 44
thoughts: 20
Thoughts: Okay, here's my summary, presented as a stream of thought:
**Diving into the BTF Snprintf Implementation**
Alright, I need to understand how `bpf_snprintf_btf` works. Given the context, I think the best approach is to start with a comprehensive code search. I'm going to leverage my familiarity with the codebase to quickly locate every instance of `bpf_snprintf_btf`. This will give me a solid overview of its usage patterns and where it fits within the broader BTF (BPF Type Format) infrastructure. I expect this initial search will reveal important call sites, dependencies, and potential interactions with other BPF functions. This should allow me to formulate a more targeted strategy for deeper analysis.
|
| 373/6 |
2026/05/18 19:15 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 374/6 |
2026/05/18 19:15 |
llm |
codeexpert |
1m
Model:
Tokens:
input: 135335
output: 46
thoughts: 17
Thoughts: Here's your summary:
**Initial Investigation: The `bpf_snprintf_btf` String**
Okay, so I need to track down where this `bpf_snprintf_btf` string is being used. Given my background, I'm immediately thinking this likely points to some interaction with the BPF (Berkeley Packet Filter) and possibly BTF (BPF Type Format) data. I'll initiate a search for that specific string within the relevant codebase. I need to understand the context of its usage. Is it in kernel code? User space tools? Compiler toolchains? Understanding the code it exists in is essential. The `snprintf` function suggests it's likely formatting something, so I need to figure out what data is being formatted and for what purpose within this BPF/BTF context. This could be crucial for debugging, performance analysis, or understanding the overall system architecture. I'll need to pay close attention to the surrounding code to determine its role.
|
| 375/6 |
2026/05/18 19:16 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf_proto SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 376/6 |
2026/05/18 19:16 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 135626
output: 46
thoughts: 223
Thoughts: **Debugging a `codesearch-find-references` Issue**
Okay, I'm hitting a snag with `codesearch-find-references` – it's not quite giving me what I need. My initial instinct was to fall back on `grep`, utilizing `codesearch-dir-index` to locate the files and then manually `read-file` them. But wait, I should really give `codesearch-find-references` another shot, but with a more generous search scope. I've been down this road before, haven't I?
Ah, right. I remember now. I already did that, and it actually did uncover `bpf_snprintf_btf_proto` within `kernel/bpf/helpers.c`, specifically on line 763, within the context of `bpf_base_func_proto`. Now I remember.
Let's double-check the `read-file` output for `kernel/bpf/helpers.c` to be certain. Yes, there it is again, line 763:
```c
case BPF_FUNC_snprintf_btf:
return &bpf_snprintf_btf_proto;
```
Right, so I know where it's *used*, but not where it's *defined*. It's almost certainly declared as `extern`. So, the next step is obvious: I'll focus my search for a definition of `bpf_snprintf_btf_proto` within the `kernel/bpf/` directory. That's the logical next step.
|
| 377/6 |
2026/05/18 19:16 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf_proto SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 378/6 |
2026/05/18 19:16 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 136259
output: 50
thoughts: 15
Thoughts: Okay, here's my interpretation of those thoughts, crafted with the understanding that I'm speaking to a fellow expert:
**Investigating BPF with Grep**
Alright, let's see... I'm thinking I'll start with `grep`. It's a quick and dirty way to get a handle on things, especially in a vast codebase like the kernel. Specifically, I'm going to aim it at the `kernel/bpf/` directory. That's the obvious starting point when dealing with anything BPF-related. This should quickly give me a list of relevant files and hopefully reveal some interesting leads, maybe function calls or variable names that catch my eye. It's a pragmatic first step to scoping out the landscape before diving into anything more complex or specialized. Let's see what that uncovers.
|
| 379/6 |
2026/05/18 19:16 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf_proto SourceTreePrefix:kernel/bpf/btf.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 380/6 |
2026/05/18 19:16 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 136519
output: 48
thoughts: 22
Thoughts: Okay, here's my attempt at summarizing that thought process, tailored for an expert audience:
**Investigating a BTF-Related BPF Snprintf Function**
Alright, let's dive into this. My initial thought is to zero in on `bpf_snprintf_btf`. Now, before I get too deep into any specifics, I think it's worth starting with a more general search query: I'll search for `bpf_snprintf_btf` – no need to narrow it down initially with the inclusion of the `_proto` extension at this stage. That approach should give me a good baseline view of the current code base. Let me try that... and then I'll see what the results tell me.
|
| 381/6 |
2026/05/18 19:16 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf SourceTreePrefix:kernel/bpf/btf.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 382/6 |
2026/05/18 19:16 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 136767
output: 42
thoughts: 47
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Debugging BTF Snprintf Issues**
Ugh, this is getting tedious. I'm hitting a wall here, and the usual debugging tricks aren't yielding the insights I need. Alright, let's cut the fluff and go straight to the source. Time to dive into `kernel/bpf/btf.c` and get my hands dirty. I'm going to bypass the higher level abstractions for a minute and go right to the code.
Actually, before I waste time with a brute force examination, let's zero in. I'm seeing something specific, right? I'll search for `snprintf` within `kernel/bpf/btf.c`. That's likely the culprit in these formatting issues I'm chasing. Hopefully, this narrows the scope down and I can quickly pinpoint the problem in the implementation or usage of `snprintf` related to BTF. Let's see what that search turns up...
|
| 383/6 |
2026/05/18 19:16 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:snprintf SourceTreePrefix:kernel/bpf/btf.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_check_iter_kfuncs SourceFile:kernel/bpf/btf.c SourceLine:8662 SourceSnippet:8661:
8662: snprintf(exp_name, sizeof(exp_name), "bpf_iter_%s_%s", iter_name, sfx);
8663: if (strcmp(func_name, exp_name))
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_nested_type_is_trusted SourceFile:kernel/bpf/btf.c SourceLine:9592 SourceSnippet:9591:
9592: ret = snprintf(safe_tname, sizeof(safe_tname), "%s%s", tname, suffix);
9593: if (ret >= sizeof(safe_tname))
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_show_name SourceFile:kernel/bpf/btf.c SourceLine:1215 SourceSnippet:1214: if (show->flags & BTF_SHOW_NONAME)
1215: snprintf(show->state.name, sizeof(show->state.name), "%s",
1216: parens);
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_show_name SourceFile:kernel/bpf/btf.c SourceLine:1218 SourceSnippet:1217: else
1218: snprintf(show->state.name, sizeof(show->state.name),
1219: "%s%s%s(%s%s%s%s%s%s)%s",
]] TruncatedOutput:false]
|
| 384/6 |
2026/05/18 19:16 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 137576
output: 41
thoughts: 340
Thoughts: **My Assessment of Potential Security Issues in BPF Helper Functions**
Okay, let's see. My initial thought process focused on `btf_nested_type_is_trusted`. It checks the return value of `snprintf` properly: if it's equal or greater than the buffer size, there's a problem. That's good. But I need to dive deeper.
My immediate question was about `btf_show_name`. The code uses `snprintf`, but there's no return value check. However, the `show->state.name` buffer is fixed-size. So, truncation is probably acceptable here since this is just display and not for security critical processing.
But the real focus here is `kernel/bpf/helpers.c`.
I've already examined `bpf_bprintf_prepare` thoroughly, looking for vulnerabilities related to `snprintf`. I've checked for calls to `snprintf`, `sprintf`, and `vsnprintf`.
Wait, there's a potential blind spot, `bpf_trace_printk`. It's in `kernel/trace/bpf_trace.c`, right? No, I need to stick to `kernel/bpf/helpers.c`. Even though `bpf_snprintf` is defined in `kernel/bpf/helpers.c`.
Okay, I need to be absolutely certain. I'm going to re-examine the `snprintf` calls within `kernel/bpf/helpers.c` again. I'll use `grep -n "snprintf" kernel/bpf/helpers.c` to make sure I don't miss anything.
|
| 385/6 |
2026/05/18 19:16 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924 SourceSnippet: 923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965 SourceSnippet: 964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
]] TruncatedOutput:false]
|
| 386/6 |
2026/05/18 19:16 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 138558
output: 41
thoughts: 960
Thoughts: **Analyzing snprintf Usage in BPF Context**
Alright, I'm initially pretty confident I've pinpointed the relevant instances of `snprintf`. But, as always, it's crucial to double-check. I'm going to revisit my earlier analysis of the `read-file` output, specifically looking for `bpf_bprintf_prepare` and any other potential uses.
Hang on... I've got to consider the `bpf_snprintf` helper, but it appears it doesn't *directly* call `snprintf`, that's a key distinction. And I should also track down the definition of the `bpf_snprintf_btf` function, which is likely in `kernel/bpf/btf.c`.
Now, let's broaden the search a bit. I need to scan `kernel/bpf/helpers.c` for any other string formatting functions. Ah, `bpf_trace_copy_string` uses `strncpy_from_kernel_nofault` which is good to know. But, are there any *other* uses of `snprintf`, perhaps hidden within a macro? Time to go through the `read-file` output again. I'll search for `snprintf` in the `helpers.c` file content. I've got the whole file contents from the earlier `read-file` output.
Okay, I'm seeing it. At line 924 and 965, there are two distinct calls to `snprintf`. There's also `bpf_snprintf` at line 1064, but that's not calling `snprintf` directly.
Wait, I need to understand that `ip_spec` variable. It's using `snprintf` to format an IP address into `tmp_buf`. It is at line 965. This is essential to understand, as it is a format of the argument and how it is formatted.
Okay, are there any *other* calls? Time to use `grep` on my `read-file` output and narrow my search to find all instances of `snprintf`. Okay, grep results are in. I see `snprintf` at:
- 924
- 965
- 1064 (part of `bpf_snprintf` name)
- 1089 (part of `bpf_snprintf_proto` name)
- 763 (part of `BPF_FUNC_snprintf_btf` name)
- 764 (part of `bpf_snprintf_btf_proto` name)
- 765 (part of `BPF_FUNC_snprintf` name)
- 766 (part of `bpf_snprintf_proto` name)
I need to confirm the use of `bpf_snprintf_btf`. Looks like it is not in `kernel/bpf/helpers.c`.
To be absolutely sure, I'll search for `snprintf(` (with the parenthesis) within the `read-file` output. That should filter out any false positives from function names, etc.
|
| 387/6 |
2026/05/18 19:16 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924 SourceSnippet: 923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965 SourceSnippet: 964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
]] TruncatedOutput:false]
|
| 388/6 |
2026/05/18 19:16 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 140492
output: 42
thoughts: 98
Thoughts: Here's my thought process, summarized as requested:
**Navigating the `bpf_bprintf_prepare` Conundrum**
Okay, just these two instances. That's what I initially concluded. But, hang on... the user specifically used the word "**other**" in their question, "Is there any **other** place...". That's a key detail! They might be looking *outside* of `bpf_bprintf_prepare` entirely. My initial focus was too narrow.
Right, so I didn't find anything outside the `bpf_bprintf_prepare` function, but I need to be *absolutely* sure. Just to be completely thorough, I'm going to re-examine `kernel/bpf/helpers.c` and specifically look for `vsnprintf`. I already looked, but it doesn't hurt to double-check. I need to leave no stone unturned in this investigation, especially given the "other" clue. The potential implications are significant, so thoroughness is paramount.
|
| 389/6 |
2026/05/18 19:16 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:vsnprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 390/6 |
2026/05/18 19:16 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 140870
output: 41
thoughts: 7
Thoughts: Okay, here's my attempt at summarizing that thought process, keeping in mind I'm speaking to a fellow expert:
**Regarding `scnprintf` and its Potential**
Hmm, as I'm considering this, the thought that keeps recurring is the potential applicability of `scnprintf`. Could it actually fit into this situation? I'm already envisioning its strengths and possible pitfalls. It offers a certain level of control, but the implications for memory safety and potential buffer overflows are immediately pressing. I need to quickly assess if the benefits, specifically regarding format string parsing and potential control over the output format, outweigh the risks in this specific context. The need for that granular control is the key driver of this consideration. I must consider the ease of implementation with other options and the resulting security posture.
|
| 391/6 |
2026/05/18 19:17 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:scnprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 392/6 |
2026/05/18 19:17 |
llm |
codeexpert |
1m
Model:
Tokens:
input: 141098
output: 42
thoughts: 8
Thoughts: Okay, here's my attempt at summarizing that thought process, keeping in mind I'm speaking to a fellow expert:
**Investigating a Printf-like Function**
Hmm, I'm thinking about string formatting and output, and specifically, what options are available. I need to consider all the possibilities. My initial thought is to explore how to get this done in a memory-efficient way, potentially avoiding unnecessary copies. The core concept here is about efficient variable argument lists. So, what about `vscnprintf`? Does that give me the flexibility and safety I'm looking for in this scenario? I need to weigh the trade-offs: performance, safety, and ease of use. Is `vscnprintf` the correct tool for the job? I need to analyze it further.
|
| 393/6 |
2026/05/18 19:18 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:vscnprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 394/6 |
2026/05/18 19:18 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 141330
output: 40
thoughts: 6
Thoughts: Okay, here's my attempt at summarizing that thought process, keeping in mind I'm speaking to a fellow expert and adopting a first-person perspective:
**Considering `sprintf` for this Task**
Hmm, now that I'm considering the approach for formatting this, the question of using `sprintf` pops into my head. It's a classic, isn't it? I have to consider the potential for its vulnerabilities related to buffer overflows, and the fact that it's generally considered less safe than some of the newer formatting tools available. However, in certain constrained contexts where performance is critical and I have absolute control over the input, it *could* be considered. I'd have to carefully evaluate if the trade-off of potentially easier syntax (depending on the exact formatting required) is worth the additional scrutiny needed to ensure safety. It depends, really, on the specifics of this situation. Is it a quick-and-dirty script, or a core library function? That will heavily influence my decision. I need to weigh the readability and maintainability against the risks.
|
| 395/6 |
2026/05/18 19:18 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:sprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 396/6 |
2026/05/18 19:18 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 141616
output: 41
thoughts: 7
Thoughts: Okay, here's my attempt at summarizing that thought process, keeping in mind I'm speaking to a fellow expert:
**The `vsprintf` Conundrum**
Hmm, so the task at hand… I'm thinking about the low-level formatting aspects here. My initial gut reaction is to immediately consider `vsprintf`. It's a powerful tool, no question. The ability to handle variable argument lists directly is incredibly tempting for the flexibility it offers. I can see the immediate appeal for dynamic string construction and the control it provides over the formatting process. But, that also means I need to be careful with the implementation and know its limitations. Is it actually the right approach? I'll need to carefully consider the potential pitfalls, security implications, and potential performance bottlenecks associated with using that function. I need to balance the convenience with the potential risks.
|
| 397/6 |
2026/05/18 19:18 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:vsprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 398/6 |
2026/05/18 19:18 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 141859
output: 45
thoughts: 536
Thoughts: **My BPF `snprintf` Investigation**
Okay, so the user's initial concern seems focused on `snprintf` calls within `bpf_bprintf_prepare`. Right, that's the starting point. I need to be methodical.
Wait, the user's concern about `snprintf` in the kernel code. Let me quickly scan, is there any other place in `kernel/bpf/helpers.c` that uses `snprintf` directly outside of a function call? No, that wouldn't make sense, so it's not a global declaration or something odd.
Maybe the concern is with the helper function, `bpf_snprintf` itself? But that calls `bstr_printf`.
Okay, let me trace this: does `bstr_printf`'s return value get handled correctly within `bpf_snprintf`? Let me see the code… Yep, line 1083 calls `bstr_printf`, then line 1087 correctly adds 1 to the error. `bstr_printf` returns *would* have been written, and the return from `bpf_snprintf` correctly indicates the required size, including the null terminator. If `err >= str_size`, we know truncation happened. This looks correct based on the documentation of the helper.
Wait, I should confirm I haven't missed any direct uses of `snprintf` elsewhere in `kernel/bpf/helpers.c`. Even though I scanned before, I'll search for `snprintf` in the output again, carefully.
I've already listed them from the search, I've checked.
Wait, I see `bpf_snprintf_btf`! Oh, no, that's just in the `switch` statement in `bpf_base_func_proto`, that's not a direct call, it's just a case statement.
Let's expand the scope: could `snprintf` be used in some other "helper-like" function in another file within `kernel/bpf/`? The user explicitly said `kernel/bpf/helpers.c`, so I should stick to the stated scope.
Okay, I'm going to step through the list of functions in `kernel/bpf/helpers.c` once more. I'll use `codesearch-file-index`, but I've already done that to be efficient.
Wait, I see `bpf_obj_name_cpy`. I'll check that one.
|
| 399/6 |
2026/05/18 19:18 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_obj_name_cpy]
Results: map[SourceCode:1205: /* dst and src must have at least "size" number of bytes.
1206: * Return strlen on success and < 0 on error.
1207: */
1208: int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size)
1209: {
1210: const char *end = src + size;
1211: const char *orig_src = src;
1212:
1213: memset(dst, 0, size);
1214: /* Copy all isalnum(), '_' and '.' chars. */
1215: while (src < end && *src) {
1216: if (!isalnum(*src) &&
1217: *src != '_' && *src != '.')
1218: return -EINVAL;
1219: *dst++ = *src++;
1220: }
1221:
1222: /* No '\0' found in "size" number of bytes */
1223: if (src == end)
1224: return -EINVAL;
1225:
1226: return src - orig_src;
1227: }
SourceFile:kernel/bpf/syscall.c]
|
| 400/6 |
2026/05/18 19:18 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 143338
output: 41
thoughts: 226
Thoughts: **Analyzing String Formatting in `kernel/bpf/helpers.c`**
Okay, first things first. I need to be absolutely precise with this. The user is asking about *other* instances of `snprintf` in `kernel/bpf/helpers.c` that lack proper return value checks. I already identified `bpf_bprintf_prepare`, but the key here is the "other" part. They likely already know about *that* instance, so I need to make sure I haven't missed anything else in that specific file.
I initially spotted `bpf_obj_name_cpy` in `kernel/bpf/syscall.c` and saw some evidence of it in `include/linux/bpf.h`, but that's not the relevant file. The user's question is explicitly focused on `kernel/bpf/helpers.c`, so my focus should remain there.
To be thorough, I will use `grep` again, but this time I'll run a completely unfiltered search for `snprintf` calls within `kernel/bpf/helpers.c`. I need to ensure I haven't missed any instances during my previous quick scan. Once I have the output from the full grep, I will be able to confirm definitively if only the two places in `bpf_bprintf_prepare` utilize `snprintf`. If I confirm there are no others, my answer is "No, other than `bpf_bprintf_prepare`." If there *are* other places, I'll need to analyze them carefully to see if they, too, exhibit the concerning behavior. I must be precise with my answer; if I don't give the complete picture, I may be misleading the user.
|
| 401/6 |
2026/05/18 19:18 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924 SourceSnippet: 923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965 SourceSnippet: 964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
]] TruncatedOutput:false]
|
| 402/6 |
2026/05/18 19:18 |
llm |
codeexpert |
1m
Model:
Tokens:
input: 144219
output: 41
thoughts: 181
Thoughts: **Investigating Potential `snprintf` Usage in BPF Helpers**
Alright, I've run several searches, and it seems I'm only turning up these two instances. Before I dive too deep, let me double-check. I need to be absolutely certain no other `snprintf` calls are hiding within `kernel/bpf/helpers.c`, *outside* of `bpf_bprintf_prepare`. Let's see... my `codesearch-find-references` output *should* be exhaustive... and yes, it confirms that `bpf_bprintf_prepare` is the *only* place. Good.
Now, hold on a second. While I was specifically told `kernel/bpf/helpers.c`, it's always wise to cast a wider net. Is there *anything* else in the `kernel/bpf/` directory that might be a relevant file to consider, just to make sure I'm not missing something subtle? No, the directive was specific.
Okay, here's a thought: Could `snprintf` be lurking within a macro defined in `kernel/bpf/helpers.c`? That's a sneaky possibility. Let's not assume anything. I'll need to meticulously examine the macros in that file. Time to bust out `grep "^#define kernel/bpf/helpers.c"` and see what's what. Gotta be thorough.
|
| 403/6 |
2026/05/18 19:20 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:#define SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 404/6 |
2026/05/18 19:20 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 144762
output: 40
thoughts: 149
Thoughts: **My Initial Investigation into Potential Kernel Vulnerabilities**
Okay, so I have this file open, and the first thing I'll do is just read through it and scan for any macros, especially those that might be related to string handling. I mean, I *already* have the file content in front of me, so that's the starting point. Let me see... Aha, I see a couple of macros in `kernel/bpf/helpers.c`: `BPF_STRTOX_BASE_MASK` on line 424 and `MAX_BPRINTF_NEST_LEVEL` on line 784.
But neither of these looks like they're directly using `snprintf`, which is good. However, that's not enough; I need to be more thorough here. Before I write this off, I really need to broaden my search. I should check for any other string formatting functions, even if `snprintf` is the most obvious.
Right, I've already checked for `sprintf`, `vsnprintf`, `scnprintf`, and `vscnprintf`. Let's not forget the usual suspects, even though they're usually avoided in kernel code. What about `strcat`, `strcpy`, and `strncpy`? They're often red flags, so I'll need to double-check their usage very carefully. Let's see what turns up.
|
| 405/6 |
2026/05/18 19:20 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:strcpy SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 406/6 |
2026/05/18 19:20 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 145263
output: 41
thoughts: 5
Thoughts: Okay, here's my attempt at summarizing the thought process, tailored for an expert audience, and written in the first person:
**`strcat` - A Flash of Familiarity, then a Quick Review**
Ah, `strcat`. The word just popped into my head. I immediately think of string concatenation in C, the standard library function. My initial reaction is a slight eyebrow raise, a flicker of "Really? We're still talking about this?" because, honestly, I try to steer clear of it when possible. It's that old reliable, isn't it? But, along with its reliability, comes its fragility.
My mind immediately starts running through the usual caveats: buffer overflows, the potential for undefined behavior if the destination buffer isn't large enough. I'm reminded of the numerous security vulnerabilities that have arisen from its misuse. I can almost visualize the old coding guidelines, the "always check your bounds!" mantra echoing in my head. I picture the alternatives, the safer options we often lean towards like `strncpy`, but even then, there's the `NULL` termination that needs to be handled carefully. And, of course, the C++ options, the `std::string` class, which offers so much more in terms of safety and ease of use.
So, I'm mentally parsing the memory layout involved, considering the implications of pointer arithmetic and the need for meticulous memory management if I were ever forced to use it. I'm thinking about the performance implications; whether, in this context, the overhead of a safer function or class might be justifiable. The trade-offs are immediately apparent. It's a reminder of the raw, low-level nature of C, the responsibility it places on the programmer. In a quick moment, I'm already mentally comparing it to other string functions, thinking about the situations where `strcat` might still *arguably* be acceptable, but only with extreme caution and rigorous testing, perhaps in a very controlled environment where the size of the destination string is precisely known and immutable. I then begin thinking about better ways to address the problem entirely.
|
| 407/6 |
2026/05/18 19:20 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:strcat SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 408/6 |
2026/05/18 19:20 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 145764
output: 40
thoughts: 4
Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience:
**`strncpy` - The Eternal Question**
Alright, `strncpy`. That function... it's a constant reminder of C's legacy, isn't it? The question always pops up: *do I really need this thing?* The goal seems simple enough: copy a string, but only up to a specified length. Sounds useful. But... the baggage.
The *gotcha* is always in the details. You've got the null termination (or lack thereof) to worry about. You've got the padding to contend with if the source string is shorter than the specified length. The potential for buffer overruns is drastically reduced with `strncpy` compared to `strcpy`, yes, but it doesn't solve it completely and if the `size` argument is wrong you can get unexpected results.
Then there's the performance consideration. In many cases, I'm probably better off calculating the length and using `memcpy` or a loop. It often comes down to this: am I better off managing the null termination and performance implications myself, or am I willing to trust this somewhat quirky function? There are corner cases where it's the right choice, but the default should probably be to avoid `strncpy` unless you are dealing with very strict memory sizes and do not mind the possibility of a non-null terminated string being created.
Ultimately, the thought process boils down to risk assessment. Is the code robust enough to handle the subtleties? Or am I going to introduce a subtle bug that's going to haunt me in production? It's always a judgement call.
|
| 409/6 |
2026/05/18 19:20 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:strncpy SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 410/6 |
2026/05/18 19:20 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 146170
output: 42
thoughts: 6
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:
**Memory Corruption Potential – `strscpy`**
Alright, `strscpy`... just the mention of that function sends a chill down my spine. The very name evokes memories of countless debugging sessions and security audits. My first thought is always, "Potential for buffer overflows, absolutely." I immediately start mentally assessing the contexts where I've seen it used, or where I *expect* to see it used, and I'm practically running a mental static analysis. I'm already envisioning the possible source and destination buffer sizes and how I'd verify that the length of the string to be copied is accounted for, and whether the null terminator is handled safely. Am I looking at a fixed-size buffer, a dynamically allocated buffer, or something in between? What if the destination buffer isn't large enough to accommodate the source string, especially with that critical null terminator? Is there any bounds checking in place? If not, that's almost a guaranteed vulnerability waiting to happen. I'm already formulating questions: Is this code in a sensitive area (kernel, user mode, network stack)? What is the target architecture, and do I have to take into consideration the compiler and linker settings? `strscpy`… it’s a red flag.
|
| 411/6 |
2026/05/18 19:20 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:strscpy SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 412/6 |
2026/05/18 19:20 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 146525
output: 41
thoughts: 291
Thoughts: **BPF String Function Safety Check**
Okay, first things first, I saw `strscpy_pad` used within `bpf_get_current_comm`, specifically in that BPF helper. Given the context, that function should be safe; the verifier provides guarantees on the `size` parameter. That's a good starting point.
Now, I need to expand my search. I should really check for other string functions used in the BPF context. Let's see... looking at the code, I see `strnchr`, `isascii`, `isprint`, `isspace`, and `ispunct` are used in `bpf_bprintf_prepare`.
Alright, so the user might have something more subtle in mind. They specifically mentioned `kernel/bpf/helpers.c`, but they're hinting at the possibility of other functions acting as helpers being outside of that file. Okay, let me think... could they be alluding to something like `bpf_snprintf_btf`? It's used to format strings, but it's not in the main helpers file.
Time to confirm my suspicions and find out where `bpf_snprintf_btf` is implemented. I'll search the code for `BPF_CALL.*snprintf_btf` and see what comes up. That should give me the implementation location to assess its usage and safety.
|
| 413/6 |
2026/05/18 19:21 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:snprintf_btf SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 414/6 |
2026/05/18 19:21 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 147177
output: 44
thoughts: 22
Thoughts: Okay, here's my attempt at summarizing that thought process, tailored for an expert audience:
**Investigating a BPF Snprintf Vulnerability**
Right, let's dive in. My immediate thought is to track down the function definition `BPF_CALL_5(bpf_snprintf_btf)`. This strongly suggests we're dealing with a BPF (Berkeley Packet Filter) context, and the presence of `snprintf` immediately raises a red flag regarding potential format string vulnerabilities or buffer overflows. The "BTF" suffix likely indicates that we're dealing with debugging information.
My approach is going to focus on understanding how this particular BPF helper function is used, what arguments it takes, and how those arguments are validated (or *aren't* validated). I'll want to trace its call sites to determine the context in which `bpf_snprintf_btf` is being invoked – is it in a privileged environment? What kind of data is being passed to it? The goal is to identify a potential path for crafting malicious input that could lead to a crash, information leak, or even code execution within the kernel. I'll need to scrutinize how format specifiers are handled and whether any bounds checks are implemented. I'm going to start with the kernel source code, but I'll also check any relevant documentation or recent security advisories related to BPF and format string bugs. My understanding of the BPF verifier is going to be key here.
|
| 415/6 |
2026/05/18 19:21 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 416/6 |
2026/05/18 19:21 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 147576
output: 48
thoughts: 27
Thoughts: Okay, here's my summary, taking on the persona you requested:
**Investigating `bpf_snprintf_btf`**
Alright, let's dive in. I need to figure out how `bpf_snprintf_btf` works. Given its name, it's obviously related to the BTF (BPF Type Format) data. I'm guessing it's responsible for, likely, string formatting when dealing with BTF information within the kernel. My first instinct, and the most logical starting point given the context, is to jump directly into the source code. I'll search for the function definition within `kernel/bpf/btf.c`. That's where I expect to find the implementation and any related function calls and structures. Hopefully, I can understand the formatting logic and where it's used within the broader BPF ecosystem from there. Let's see what we find.
|
| 417/6 |
2026/05/18 19:21 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf SourceTreePrefix:kernel/bpf/btf.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 418/6 |
2026/05/18 19:21 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 147867
output: 45
thoughts: 24
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience and presented in the first person:
**Navigating the BTF Landscape**
Right, so I need to find the definition of `snprintf_btf`. My initial instinct, given its naming convention, is that it's probably tucked away in the BPF subsystem, specifically within the BTF (BPF Type Format) implementation. Therefore, I’ll start digging in the `kernel/bpf/` directory, and the filename most likely is `btf.c`. That seems like the logical place to start my search. It's a bit of an educated guess, but usually naming conventions are pretty consistent within the kernel. Let's fire up the search tools and see what we can find. I’m hoping it’s clearly defined there, as any interactions with a formatted string would indicate.
|
| 419/6 |
2026/05/18 19:21 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:snprintf_btf SourceTreePrefix:kernel/bpf/btf.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 420/6 |
2026/05/18 19:21 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 148142
output: 29
thoughts: 17
Thoughts: Okay, here's my attempt at summarizing that thought process, tailored for an expert audience:
**Debugging Strategy: `snprintf_btf` Search**
Alright, so I'm diving into this issue, and the first thing that pops into my head is to track down the source of this particular function, `snprintf_btf`. Given the context, I'm going to start by running a comprehensive search across the entire project codebase. I'm going to leverage my existing knowledge of the tooling to expedite this process - likely a global search through my IDE or the command line. This comprehensive search should allow me to identify not only the definition of `snprintf_btf` but also all its potential call sites, which I can then leverage to understand the functionality and usage more broadly. Understanding where it is used is vital, so as to understand the context of any problems encountered. This is always my first step when chasing a function I'm unfamiliar with.
|
| 421/6 |
2026/05/18 19:21 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:snprintf_btf]
Results: map[References:<nil> TruncatedOutput:false]
|
| 422/6 |
2026/05/18 19:21 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 148409
output: 32
thoughts: 20
Thoughts: Okay, here's my summary, presented as a stream of thought:
**Diving into the BTF Snprintf Implementation**
Alright, I need to understand how `bpf_snprintf_btf` works. Given the context, I think the best approach is to start with a comprehensive code search. I'm going to leverage my familiarity with the codebase to quickly locate every instance of `bpf_snprintf_btf`. This will give me a solid overview of its usage patterns and where it fits within the broader BTF (BPF Type Format) infrastructure. I expect this initial search will reveal important call sites, dependencies, and potential interactions with other BPF functions. This should allow me to formulate a more targeted strategy for deeper analysis.
|
| 423/6 |
2026/05/18 19:22 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf]
Results: map[References:<nil> TruncatedOutput:false]
|
| 424/6 |
2026/05/18 19:22 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 148639
output: 44
thoughts: 25
Thoughts: Okay, here's my summary, presented as a stream of thought:
**Debugging the BTF Code**
Alright, I need to get to the bottom of this. Where's the best place to start looking into this BTF issue? Hmm, let's see... *BPF_CALL*... that's a key function call within the BPF framework. That likely points to where the action is happening, right? So, let's zero in on `kernel/bpf/btf.c`. Yep, I'll search for `BPF_CALL` within that specific file and see what kind of relevant information I can pull out. That sounds like a good starting point. Let's see what we can find!
|
| 425/6 |
2026/05/18 19:22 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:BPF_CALL SourceTreePrefix:kernel/bpf/btf.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 426/6 |
2026/05/18 19:22 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 148885
output: 28
thoughts: 36
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Diving into BTF Implementation**
Right, so I'm thinking, I need to get a handle on the BTF implementation. Best place to start is `kernel/bpf/btf.c`. It's likely a sizable file, so before I just start blindly grepping through it, I'll leverage the `codesearch-file-index` to get a quick overview. That will let me zero in on the key functions and data structures without wasting time on the irrelevant bits. Efficiency is key when dealing with kernel code, especially something as fundamental and complex as the BTF infrastructure. That initial indexing pass will save me a lot of head-scratching later.
|
| 427/6 |
2026/05/18 19:22 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/bpf/btf.c]
Results: map[Entities:[map[Kind:function Name:____bpf_btf_find_by_name_kind] map[Kind:function Name:__btf_array_show] map[Kind:function Name:__btf_kfunc_is_allowed] map[Kind:function Name:__btf_name_by_offset] map[Kind:function Name:__btf_name_char_ok] map[Kind:function Name:__btf_new_fd] map[Kind:function Name:__btf_resolve_size] map[Kind:function Name:__btf_show_indent] map[Kind:function Name:__btf_show_obj_safe] map[Kind:function Name:__btf_struct_show] map[Kind:function Name:__btf_type_int_is_regular] map[Kind:function Name:__btf_verifier_log] map[Kind:function Name:__btf_verifier_log_type] map[Kind:function Name:__get_type_fmodel_flags] map[Kind:function Name:__get_type_size] map[Kind:function Name:__print_cand_cache] map[Kind:function Name:__register_bpf_struct_ops] map[Kind:function Name:__register_btf_kfunc_id_set] map[Kind:function Name:bpf_btf_find_by_name_kind] map[Kind:function Name:bpf_btf_show_fdinfo] map[Kind:function Name:bpf_core_add_cands] map[Kind:function Name:bpf_core_apply] map[Kind:function Name:bpf_core_essential_name_len] map[Kind:function Name:bpf_core_find_cands] map[Kind:function Name:bpf_core_is_flavor_sep] map[Kind:function Name:bpf_core_types_are_compat] map[Kind:function Name:bpf_core_types_match] map[Kind:function Name:bpf_find_btf_id] map[Kind:function Name:bpf_free_cands] map[Kind:function Name:bpf_free_cands_from_cache] map[Kind:function Name:bpf_prog_get_target_btf] map[Kind:function Name:bpf_prog_type_to_kfunc_hook] map[Kind:function Name:bpf_struct_ops_find] map[Kind:function Name:bpf_struct_ops_find_value] map[Kind:function Name:btf_add_struct_ops] map[Kind:function Name:btf_add_type] map[Kind:function Name:btf_alloc_id] map[Kind:function Name:btf_array_check_member] map[Kind:function Name:btf_array_check_meta] map[Kind:function Name:btf_array_log] map[Kind:function Name:btf_array_resolve] map[Kind:function Name:btf_array_show] map[Kind:function Name:btf_base_btf] map[Kind:function Name:btf_bitfield_show] map[Kind:function Name:btf_check_all_metas] map[Kind:function Name:btf_check_all_types] map[Kind:function Name:btf_check_and_fixup_fields] map[Kind:function Name:btf_check_dtor_kfuncs] map[Kind:function Name:btf_check_func_type_match] map[Kind:function Name:btf_check_iter_arg] map[Kind:function Name:btf_check_iter_kfuncs] map[Kind:function Name:btf_check_kfunc_protos] map[Kind:function Name:btf_check_meta] map[Kind:function Name:btf_check_sec_info] map[Kind:function Name:btf_check_sorted] map[Kind:function Name:btf_check_type_match] map[Kind:function Name:btf_check_type_tags] map[Kind:function Name:btf_ctx_access] map[Kind:function Name:btf_ctx_arg_idx] map[Kind:function Name:btf_ctx_arg_offset] map[Kind:function Name:btf_datasec_check_meta] map[Kind:function Name:btf_datasec_log] map[Kind:function Name:btf_datasec_resolve] map[Kind:function Name:btf_datasec_show] map[Kind:function Name:btf_decl_tag_check_meta] map[Kind:function Name:btf_decl_tag_log] map[Kind:function Name:btf_decl_tag_resolve] map[Kind:function Name:btf_df_check_kflag_member] map[Kind:function Name:btf_df_check_member] map[Kind:function Name:btf_df_resolve] map[Kind:function Name:btf_df_show] map[Kind:function Name:btf_distill_func_proto] map[Kind:function Name:btf_enum64_check_meta] map[Kind:function Name:btf_enum64_show] map[Kind:function Name:btf_enum_check_kflag_member] map[Kind:function Name:btf_enum_check_member] map[Kind:function Name:btf_enum_check_meta] map[Kind:function Name:btf_enum_log] map[Kind:function Name:btf_enum_show] map[Kind:function Name:btf_field_cmp] map[Kind:function Name:btf_find_by_name_kind] map[Kind:function Name:btf_find_by_name_kind_bsearch] map[Kind:function Name:btf_find_datasec_var] map[Kind:function Name:btf_find_decl_tag_value] map[Kind:function Name:btf_find_dtor_kfunc] map[Kind:function Name:btf_find_field] map[Kind:function Name:btf_find_field_one] map[Kind:function Name:btf_find_graph_root] map[Kind:function Name:btf_find_kptr] map[Kind:function Name:btf_find_nested_struct] map[Kind:function Name:btf_find_next_decl_tag] map[Kind:function Name:btf_find_struct] map[Kind:function Name:btf_find_struct_field] map[Kind:function Name:btf_find_struct_meta] map[Kind:function Name:btf_float_check_member] map[Kind:function Name:btf_float_check_meta] map[Kind:function Name:btf_float_log] map[Kind:function Name:btf_free] map[Kind:function Name:btf_free_dtor_kfunc_tab] map[Kind:function Name:btf_free_id] map[Kind:function Name:btf_free_kfunc_set_tab] map[Kind:function Name:btf_free_rcu] map[Kind:function Name:btf_free_struct_meta_tab] map[Kind:function Name:btf_free_struct_ops_tab] map[Kind:function Name:btf_func_check] map[Kind:function Name:btf_func_check_meta] map[Kind:function Name:btf_func_proto_check] map[Kind:function Name:btf_func_proto_check_meta] map[Kind:function Name:btf_func_proto_log] map[Kind:function Name:btf_func_resolve] map[Kind:function Name:btf_fwd_check_meta] map[Kind:function Name:btf_fwd_type_log] map[Kind:function Name:btf_generic_check_kflag_member] map[Kind:function Name:btf_get] map[Kind:function Name:btf_get_by_fd] map[Kind:function Name:btf_get_fd_by_id] map[Kind:function Name:btf_get_field_type] map[Kind:function Name:btf_get_info_by_fd] map[Kind:function Name:btf_get_module_btf] map[Kind:function Name:btf_get_name] map[Kind:function Name:btf_get_ptr_to_btf_id] map[Kind:function Name:btf_header] map[Kind:function Name:btf_int128_print] map[Kind:function Name:btf_int128_shift] map[Kind:function Name:btf_int_bits_show] map[Kind:function Name:btf_int_check_kflag_member] map[Kind:function Name:btf_int_check_member] map[Kind:function Name:btf_int_check_meta] map[Kind:function Name:btf_int_encoding_str] map[Kind:function Name:btf_int_log] map[Kind:function Name:btf_int_show] map[Kind:function Name:btf_is_dynptr_ptr] map[Kind:function Name:btf_is_kernel] map[Kind:function Name:btf_is_module] map[Kind:function Name:btf_is_prog_ctx_type] map[Kind:function Name:btf_is_projection_of] map[Kind:function Name:btf_is_vmlinux] map[Kind:function Name:btf_kfunc_flags] map[Kind:function Name:btf_kfunc_id_set_contains] map[Kind:function Name:btf_kfunc_is_allowed] map[Kind:function Name:btf_kfunc_is_modify_return] map[Kind:function Name:btf_member_is_reg_int] map[Kind:function Name:btf_modifier_check_kflag_member] map[Kind:function Name:btf_modifier_check_member] map[Kind:function Name:btf_modifier_resolve] map[Kind:function Name:btf_modifier_show] map[Kind:function Name:btf_name_by_offset] map[Kind:function Name:btf_name_offset_valid] map[Kind:function Name:btf_name_valid_identifier] map[Kind:function Name:btf_name_valid_section] map[Kind:function Name:btf_named_start_id] map[Kind:function Name:btf_nested_type_is_trusted] map[Kind:function Name:btf_new_fd] map[Kind:function Name:btf_nr_types] map[Kind:function Name:btf_obj_id] map[Kind:function Name:btf_param_match_suffix] map[Kind:function Name:btf_parse] map[Kind:function Name:btf_parse_base] map[Kind:function Name:btf_parse_fields] map[Kind:function Name:btf_parse_graph_root] map[Kind:function Name:btf_parse_hdr] map[Kind:function Name:btf_parse_kptr] map[Kind:function Name:btf_parse_layout_sec] map[Kind:function Name:btf_parse_list_head] map[Kind:function Name:btf_parse_rb_root] map[Kind:function Name:btf_parse_str_sec] map[Kind:function Name:btf_parse_struct_metas] map[Kind:function Name:btf_parse_type_sec] map[Kind:function Name:btf_parse_vmlinux] map[Kind:function Name:btf_populate_kfunc_set] map[Kind:function Name:btf_prepare_func_args] map[Kind:function Name:btf_ptr_check_member] map[Kind:function Name:btf_ptr_resolve] map[Kind:function Name:btf_ptr_show] map[Kind:function Name:btf_put] map[Kind:function Name:btf_ref_type_check_meta] map[Kind:function Name:btf_ref_type_log] map[Kind:function Name:btf_release] map[Kind:function Name:btf_relocate_id] map[Kind:function Name:btf_repeat_fields] map[Kind:function Name:btf_resolve] map[Kind:function Name:btf_resolve_size] map[Kind:function Name:btf_resolve_valid] map[Kind:function Name:btf_resolved_type_id] map[Kind:function Name:btf_resolved_type_size] map[Kind:function Name:btf_sec_info_cmp] map[Kind:function Name:btf_seq_show] map[Kind:function Name:btf_set_base_btf] map[Kind:function Name:btf_show] map[Kind:function Name:btf_show_delim] map[Kind:function Name:btf_show_end_aggr_type] map[Kind:function Name:btf_show_end_array_member] map[Kind:function Name:btf_show_end_array_type] map[Kind:function Name:btf_show_end_member] map[Kind:function Name:btf_show_end_struct_type] map[Kind:function Name:btf_show_end_type] map[Kind:function Name:btf_show_indent] map[Kind:function Name:btf_show_name] map[Kind:function Name:btf_show_newline] map[Kind:function Name:btf_show_obj_is_safe] map[Kind:function Name:btf_show_obj_safe] map[Kind:function Name:btf_show_obj_size_left] map[Kind:function Name:btf_show_start_aggr_type] map[Kind:function Name:btf_show_start_array_member] map[Kind:function Name:btf_show_start_array_type] map[Kind:function Name:btf_show_start_member] map[Kind:function Name:btf_show_start_struct_type] map[Kind:function Name:btf_show_start_type] map[Kind:function Name:btf_snprintf_show] map[Kind:function Name:btf_start_id] map[Kind:function Name:btf_str_by_offset] map[Kind:function Name:btf_struct_access] map[Kind:function Name:btf_struct_check_member] map[Kind:function Name:btf_struct_check_meta] map[Kind:function Name:btf_struct_ids_match] map[Kind:function Name:btf_struct_log] map[Kind:function Name:btf_struct_metas_free] map[Kind:function Name:btf_struct_resolve] map[Kind:function Name:btf_struct_show] map[Kind:function Name:btf_struct_walk] map[Kind:function Name:btf_translate_to_vmlinux] map[Kind:function Name:btf_try_get_module] map[Kind:function Name:btf_type_array] map[Kind:function Name:btf_type_by_id] map[Kind:function Name:btf_type_decl_tag] map[Kind:function Name:btf_type_enum] map[Kind:function Name:btf_type_enum64] map[Kind:function Name:btf_type_has_size] map[Kind:function Name:btf_type_id_resolve] map[Kind:function Name:btf_type_id_size] map[Kind:function Name:btf_type_ids_nocast_alias] map[Kind:function Name:btf_type_int] map[Kind:function Name:btf_type_int_is_regular] map[Kind:function Name:btf_type_is_datasec] map[Kind:function Name:btf_type_is_decl_tag] map[Kind:function Name:btf_type_is_decl_tag_target] map[Kind:function Name:btf_type_is_i32] map[Kind:function Name:btf_type_is_i64] map[Kind:function Name:btf_type_is_modifier] map[Kind:function Name:btf_type_is_primitive] map[Kind:function Name:btf_type_is_resolve_source_only] map[Kind:function Name:btf_type_is_void] map[Kind:function Name:btf_type_needs_resolve] map[Kind:function Name:btf_type_nosize] map[Kind:function Name:btf_type_nosize_or_null] map[Kind:function Name:btf_type_ops] map[Kind:function Name:btf_type_resolve_func_ptr] map[Kind:function Name:btf_type_resolve_ptr] map[Kind:function Name:btf_type_seq_show] map[Kind:function Name:btf_type_seq_show_flags] map[Kind:function Name:btf_type_show] map[Kind:function Name:btf_type_skip_modifiers] map[Kind:function Name:btf_type_skip_qualifiers] map[Kind:function Name:btf_type_snprintf_show] map[Kind:function Name:btf_type_str] map[Kind:function Name:btf_type_var] map[Kind:function Name:btf_types_are_same] map[Kind:function Name:btf_validate_prog_ctx_type] map[Kind:function Name:btf_var_check_meta] map[Kind:function Name:btf_var_log] map[Kind:function Name:btf_var_resolve] map[Kind:function Name:btf_var_show] map[Kind:function Name:btf_verifier_env_free] map[Kind:function Name:btf_verifier_log] map[Kind:function Name:btf_verifier_log_hdr] map[Kind:function Name:btf_verifier_log_member] map[Kind:function Name:btf_verifier_log_vsi] map[Kind:function Name:check_btf_kconfigs] map[Kind:function Name:check_cand_cache] map[Kind:function Name:env_resolve_init] map[Kind:function Name:env_stack_peak] map[Kind:function Name:env_stack_pop_resolved] map[Kind:function Name:env_stack_push] map[Kind:function Name:env_stack_set_next_member] map[Kind:function Name:env_type_is_resolve_sink] map[Kind:function Name:env_type_is_resolved] map[Kind:function Name:finalize_log] map[Kind:function Name:find_canonical_prog_ctx_type] map[Kind:function Name:find_kern_ctx_type_id] map[Kind:function Name:get_kern_ctx_btf_id] map[Kind:function Name:hash_cands] map[Kind:function Name:populate_cand_cache] map[Kind:function Name:print_cand_cache] map[Kind:function Name:prog_args_trusted] map[Kind:function Name:register_btf_fmodret_id_set] map[Kind:function Name:register_btf_id_dtor_kfuncs] map[Kind:function Name:register_btf_kfunc_id_set] map[Kind:function Name:sizeof_cands] map[Kind:struct Name:bpf_cand_cache] map[Kind:struct Name:bpf_ctx_convert] map[Kind:struct Name:bpf_raw_tp_null_args] map[Kind:struct Name:btf] map[Kind:struct Name:btf_field_info] map[Kind:struct Name:btf_id_dtor_kfunc_tab] map[Kind:struct Name:btf_kfunc_hook_filter] map[Kind:struct Name:btf_kfunc_set_tab] map[Kind:struct Name:btf_kind_operations] map[Kind:struct Name:btf_sec_info] map[Kind:struct Name:btf_show] map[Kind:struct Name:btf_show_snprintf] map[Kind:struct Name:btf_struct_ops_tab] map[Kind:struct Name:btf_verifier_env] map[Kind:struct Name:resolve_vertex] map[Kind:global_variable Name:__UNIQUE_ID_addressable___register_bpf_struct_ops_1014] map[Kind:global_variable Name:__UNIQUE_ID_addressable_bpf_find_btf_id_981] map[Kind:global_variable Name:__UNIQUE_ID_addressable_btf_ctx_access_999] map[Kind:global_variable Name:__UNIQUE_ID_addressable_btf_type_by_id_982] map[Kind:global_variable Name:__UNIQUE_ID_addressable_register_btf_fmodret_id_set_1011] map[Kind:global_variable Name:__UNIQUE_ID_addressable_register_btf_id_dtor_kfuncs_1013] map[Kind:global_variable Name:__UNIQUE_ID_addressable_register_btf_kfunc_id_set_1010] map[Kind:global_variable Name:alloc_obj_fields] map[Kind:global_variable Name:array_ops] map[Kind:global_variable Name:bpf_btf_find_by_name_kind_proto] map[Kind:global_variable Name:bpf_ctx_convert_map] map[Kind:global_variable Name:btf_fops] map[Kind:global_variable Name:btf_idr] map[Kind:global_variable Name:btf_idr_lock] map[Kind:global_variable Name:btf_kind_str] map[Kind:global_variable Name:btf_sec_info_offset] map[Kind:global_variable Name:cand_cache_mutex] map[Kind:global_variable Name:datasec_ops] map[Kind:global_variable Name:decl_tag_ops] map[Kind:global_variable Name:enum64_ops] map[Kind:global_variable Name:enum_ops] map[Kind:global_variable Name:float_ops] map[Kind:global_variable Name:func_ops] map[Kind:global_variable Name:func_proto_ops] map[Kind:global_variable Name:fwd_ops] map[Kind:global_variable Name:int_ops] map[Kind:global_variable Name:kind_ops] map[Kind:global_variable Name:modifier_ops] map[Kind:global_variable Name:ptr_ops] map[Kind:global_variable Name:raw_tp_null_args] map[Kind:global_variable Name:struct_ops] map[Kind:global_variable Name:var_ops] map[Kind:enum Name:bpf_struct_walk_result] map[Kind:enum Name:btf_arg_tag] map[Kind:enum Name:btf_kfunc_hook] map[Kind:enum Name:resolve_mode] map[Kind:enum Name:verifier_phase] map[Kind:enum Name:visit_state] map[Kind:typedef Name:btf_bpf_btf_find_by_name_kind]]]
|
| 428/6 |
2026/05/18 19:22 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 155687
output: 47
thoughts: 39
Thoughts: Okay, here's my interpretation of those thoughts, tailored for an expert audience, presented in the first person:
**Analyzing BTF Type Printing Functions**
Right, I'm looking at `btf_type_snprintf_show` and also `btf_snprintf_show`. It appears these are some kind of printing or display functions related to BTF (BPF Type Format) information. My first instinct is to dive into `btf_type_snprintf_show`. I'm guessing this function is the core logic for formatting and presenting the detailed type information, while `btf_snprintf_show` might be a more general wrapper or dispatcher for different kinds of BTF data. Let's see what `btf_type_snprintf_show` is actually doing. We need to understand how it handles the different BTF type codes and structures. This is where the magic happens, or where we find the bugs.
|
| 429/6 |
2026/05/18 19:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/btf.c IncludeLines:true Name:btf_type_snprintf_show]
Results: map[SourceCode:8151: int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
8152: char *buf, int len, u64 flags)
8153: {
8154: struct btf_show_snprintf ssnprintf;
8155:
8156: ssnprintf.show.target = buf;
8157: ssnprintf.show.flags = flags;
8158: ssnprintf.show.showfn = btf_snprintf_show;
8159: ssnprintf.len_left = len;
8160: ssnprintf.len = 0;
8161:
8162: btf_type_show(btf, type_id, obj, (struct btf_show *)&ssnprintf);
8163:
8164: /* If we encountered an error, return it. */
8165: if (ssnprintf.show.state.status)
8166: return ssnprintf.show.state.status;
8167:
8168: /* Otherwise return length we would have written */
8169: return ssnprintf.len;
8170: }
SourceFile:kernel/bpf/btf.c]
|
| 430/6 |
2026/05/18 19:22 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 156334
output: 48
thoughts: 64
Thoughts: Here's my thought process, as if I were considering this code:
**Analyzing the BTF Type Printing Function**
Okay, so `btf_type_snprintf_show` is used by `bpf_snprintf_btf`. That's the link. Let's see what `bpf_snprintf_btf` *does*. I need to understand its role. I'll quickly track down its definition to get a clearer picture. Alright, let's search for the function signature. I'll look specifically for `BPF_CALL_5(bpf_snprintf_btf`. That should narrow the search and get me the declaration quickly. I expect this to be a helper to print BPF Type Format (BTF) information.
|
| 431/6 |
2026/05/18 19:22 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf SourceTreePrefix:kernel/bpf/btf.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 432/6 |
2026/05/18 19:22 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 156631
output: 44
thoughts: 23
Thoughts: Okay, here's my attempt at summarizing that thought process, tailored for an expert audience, and written in the first person:
**Kernel BPF Snprintf BTF Exploration**
Alright, let's dive into this. My first move is clear: I need to pinpoint the `bpf_snprintf_btf` function within the kernel source. Given its name, it's highly likely residing within the `kernel/bpf/` directory. That's the logical starting point. I'll initiate a search directly within that path. I'm looking for the implementation, potential usage examples, and any relevant interactions with BTF (BPF Type Format) metadata, of course. I'll need to understand how the function interfaces with the BTF information for formatting and how this applies to BPF programs. This search should provide a foundation for further analysis.
|
| 433/6 |
2026/05/18 19:22 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 434/6 |
2026/05/18 19:22 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 156900
output: 32
thoughts: 20
Thoughts: Okay, here's my summary, presented as a stream of thought:
**Initial Investigation into `bpf_snprintf_btf`**
Alright, let's get down to it. My initial approach here will be to perform a comprehensive search across the entire project's codebase. The keyword I'm starting with is `bpf_snprintf_btf`. I'm hoping to locate every instance of this specific function – that should quickly give me an overview of how it's being used and, importantly, where it resides within the broader architecture. From there, I can begin to unpack its role and relationships. This is a good starting point to build context and understand the implementation.
|
| 435/6 |
2026/05/18 19:23 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf]
Results: map[References:<nil> TruncatedOutput:false]
|
| 436/6 |
2026/05/18 19:23 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 157119
output: 29
thoughts: 17
Thoughts: Okay, here's my attempt at summarizing the thought process, tailored for an expert audience, and written in the first person:
**Diving Deep into BTF and `snprintf`**
Right, let's get down to brass tacks. I need to understand how `snprintf_btf` functions within the broader context. My initial instinct, as always, is to go wide first. I'm thinking a comprehensive search across the entire codebase is the best starting point. This should illuminate all potential usages, dependencies, and implementations of `snprintf_btf`. The goal isn't just to *find* it, but to understand its relationship to the surrounding BTF infrastructure. Are we dealing with a specific helper, an internal library function, or something else entirely? A broad search will give me the map I need.
|
| 437/6 |
2026/05/18 19:23 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:snprintf_btf]
Results: map[References:<nil> TruncatedOutput:false]
|
| 438/6 |
2026/05/18 19:23 |
llm |
codeexpert |
1m
Model:
Tokens:
input: 157360
output: 46
thoughts: 27
Thoughts: Here's my thought process, summarized as you requested:
**Navigating the Kernel's BPF Landscape**
Okay, so I need to find where `BPF_CALL_5` is defined, or at least used, within the kernel's BPF infrastructure. My initial instinct is to target the BTF (BPF Type Format) code, as that's often where function signatures and call details are managed. Given the likely importance of function calls within BPF, `kernel/bpf/btf.c` seems like the most logical starting point. I'll begin by searching for that exact string – `BPF_CALL_5` – within that file. That should give me the entry point I require. From there, I'll be able to trace back the definitions and usage.
|
| 439/6 |
2026/05/18 19:24 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:BPF_CALL_5 SourceTreePrefix:kernel/bpf/btf.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 440/6 |
2026/05/18 19:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 157624
output: 42
thoughts: 21
Thoughts: Okay, here's my thought process summarized as requested:
**Investigating BPF Function Call Locations**
Alright, let's get down to brass tacks. I need to figure out where this `BPF_CALL_5` function is used within the kernel's BPF infrastructure. My initial step will be to dive into the `kernel/bpf/` directory. That's the obvious starting point; it's where the core BPF implementation resides, and it's the most likely place to find the definition and any related usage. I'll focus my search within that specific path. My goal is to pinpoint the exact locations where `BPF_CALL_5` is referenced.
|
| 441/6 |
2026/05/18 19:24 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:BPF_CALL_5 SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 442/6 |
2026/05/18 19:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 157856
output: 23
thoughts: 53
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Debugging a BPF Macro Anomaly**
This is perplexing. I'm seeing an issue with `BPF_CALL_5` – a macro that's practically ubiquitous in BPF code. Something's clearly not right. The first thing that comes to mind is a possible hiccup in the tooling. Could it be a macro expansion problem within the analysis engine? Perhaps a preprocessor issue manifesting in the static analysis?
To narrow this down, I need to check the scope. Let's start with a focused search. I'll reach for `codesearch-dir-index` and direct it specifically to `kernel/bpf/`. That should give me a rapid overview of the source files using this macro and allow me to immediately identify areas where its definition might be non-standard, or where the current analysis might be misinterpreting its invocation. This will let me confirm if the issue is confined, or more widespread.
|
| 443/6 |
2026/05/18 19:24 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:[arena.c arraymap.c backtrack.c bloom_filter.c bpf_cgrp_storage.c bpf_inode_storage.c bpf_insn_array.c bpf_iter.c bpf_local_storage.c bpf_lru_list.c bpf_lru_list.h bpf_lsm.c bpf_lsm_proto.c bpf_struct_ops.c bpf_task_storage.c btf.c btf_iter.c btf_relocate.c cfg.c cgroup.c cgroup_iter.c check_btf.c const_fold.c core.c cpumap.c cpumask.c crypto.c devmap.c disasm.c disasm.h dispatcher.c dmabuf_iter.c fixups.c hashtab.c helpers.c inode.c kmem_cache_iter.c link_iter.c liveness.c local_storage.c log.c lpm_trie.c map_in_map.c map_in_map.h map_iter.c memalloc.c mmap_unlock_work.h mprog.c net_namespace.c offload.c percpu_freelist.c percpu_freelist.h prog_iter.c queue_stack_maps.c range_tree.c range_tree.h relo_core.c reuseport_array.c ringbuf.c rqspinlock.c rqspinlock.h stackmap.c states.c stream.c syscall.c sysfs_btf.c task_iter.c tcx.c tnum.c token.c trampoline.c verifier.c] Subdirs:[preload]]
|
| 444/6 |
2026/05/18 19:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 158757
output: 22
thoughts: 25
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Kernel Tracing & BPF Helper Location**
Alright, so I'm thinking about where to find the relevant code for this tracing functionality. I know the obvious places, but I also want to make sure I'm not missing anything. Specifically, I'm going to double check `kernel/trace/bpf_trace.c`. That's where a lot of the core BPF helper functions are defined, especially the ones used for tracing and instrumentation. I'll need to check if what I'm looking for is present there, or if I need to dig deeper into related subsystems. It's the most likely place to be.
|
| 445/6 |
2026/05/18 19:24 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:[blktrace.c bpf_trace.c bpf_trace.h error_report-traces.c fgraph.c fprobe.c ftrace.c ftrace_internal.h kprobe_event_gen_test.c pid_list.c pid_list.h power-traces.c preemptirq_delay_test.c remote_test.c remote_test_events.h rethook.c ring_buffer.c ring_buffer_benchmark.c rpm-traces.c simple_ring_buffer.c synth_event_gen_test.c trace.c trace.h trace_benchmark.c trace_benchmark.h trace_boot.c trace_branch.c trace_btf.c trace_btf.h trace_clock.c trace_dynevent.c trace_dynevent.h trace_entries.h trace_eprobe.c trace_event_perf.c trace_events.c trace_events_filter.c trace_events_filter_test.h trace_events_hist.c trace_events_inject.c trace_events_synth.c trace_events_trigger.c trace_events_user.c trace_export.c trace_fprobe.c trace_functions.c trace_functions_graph.c trace_hwlat.c trace_irqsoff.c trace_kdb.c trace_kprobe.c trace_kprobe_selftest.c trace_kprobe_selftest.h trace_mmiotrace.c trace_nop.c trace_osnoise.c trace_output.c trace_output.h trace_pid.c trace_preemptirq.c trace_printk.c trace_probe.c trace_probe.h trace_probe_kernel.h trace_probe_tmpl.h trace_recursion_record.c trace_remote.c trace_sched_switch.c trace_sched_wakeup.c trace_selftest.c trace_selftest_dynamic.c trace_seq.c trace_snapshot.c trace_stack.c trace_stat.c trace_stat.h trace_synth.h trace_syscalls.c trace_uprobe.c tracing_map.c tracing_map.h undefsyms_base.c] Subdirs:[rv]]
|
| 446/6 |
2026/05/18 19:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 159715
output: 49
thoughts: 27
Thoughts: Okay, here's my take on those thoughts, framed as an expert's internal monologue:
**Investigating BPF Trace Formatting**
Alright, let's get down to brass tacks. I need to understand how the `bpf_snprintf_btf` function is handling the formatting within the BPF tracepoints. My first stop, naturally, is going to be `kernel/trace/bpf_trace.c`. That's where all the core BPF tracing magic, including likely the formatting routines for BTF-aware tracepoints, should reside. I'll need to check the context, arguments, return values, any inline documentation, and the overall structure of the function within that file. I'm expecting to find the key to how BPF tracing interacts with BTF type information there, and then how it's used to construct human-readable output, likely using `snprintf` or a similar variant for buffer-safe string construction in the kernel. This is probably a central point for understanding the limitations and capabilities of BPF tracepoints as well. Let's see what's there... I'm particularly interested in how they're handling potentially complex data structures and the efficient formatting of these in a kernel context.
|
| 447/6 |
2026/05/18 19:24 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:bpf_snprintf_btf SourceTreePrefix:kernel/trace/bpf_trace.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 448/6 |
2026/05/18 19:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 160072
output: 43
thoughts: 22
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Investigating `snprintf` Usage in `kernel/trace/bpf_trace.c`**
Alright, so I need to track down the use of `snprintf` within the BPF tracing code, specifically in `kernel/trace/bpf_trace.c`. My initial thought is to just search the source for the function name. I'm going to do a quick `grep` or maybe fire up my IDE's search function with that specific file targeted. I'm expecting to find it within the formatting and output stages. I'm guessing it's probably used for creating trace messages, constructing BPF program error reports, or potentially formatting data for userspace consumption. I want to check how it's being used with regard to potential buffer overflows or format string vulnerabilities. I need to keep in mind the context of BPF, including how it interacts with the kernel and userspace, to properly evaluate any potential security risks related to its usage. This search will help me understand the potential impact and risk of format string vulnerabilities.
|
| 449/6 |
2026/05/18 19:24 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:1 Name:snprintf SourceTreePrefix:kernel/trace/bpf_trace.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 450/6 |
2026/05/18 19:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 160389
output: 29
thoughts: 65
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Kernel Trace and BTF Formatting - Initial Concern and Refinement**
Hmm, no `snprintf` in `kernel/trace/bpf_trace.c`. That's unusual, and it immediately sets off alarm bells. Given the nature of BPF tracing, string formatting is practically a core requirement for generating readable output and debugging information. I'd expect to see some kind of standard library functionality in use there, likely for building human-readable trace messages from within the BPF context.
Let me rethink this approach, I need to be sure. I'll take a step back and revisit the entire codebase. Rather than assuming, I should be looking for a potential BPF-safe version of `snprintf` or something similar, like a dedicated `bpf_snprintf_btf` function. I will use `grep` through `codesearch-find-references` with a very focused, basic search string. This should allow me to find any instance of that `bpf_snprintf_btf` function, if it exists, and reveal where it is used. This method will give me a more accurate picture of the situation.
|
| 451/6 |
2026/05/18 19:24 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:snprintf_btf]
Results: map[References:<nil> TruncatedOutput:false]
|
| 452/6 |
2026/05/18 19:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 160761
output: 29
thoughts: 25
Thoughts: Okay, here's my take on those thoughts, framed as an expert's internal monologue:
**Investigating BPF Trace Points**
Right, so I'm thinking about digging into the BPF tracing implementation, specifically the `bpf_trace.c` file within the kernel source. My gut feeling is that I need to pinpoint the relevant index, and `codesearch-file-index` seems like the most efficient way to achieve that. Let's see if that tool can quickly identify the key sections or data structures I'm looking for within that specific file. It could save me significant time instead of manually grepping. I want to jump straight to the heart of the matter.
|
| 453/6 |
2026/05/18 19:24 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/trace/bpf_trace.c]
Results:
|
| 454/6 |
2026/05/18 19:24 |
llm |
codeexpert |
1m
Model:
Tokens:
input: 160971
output: 25
thoughts: 48
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience:
**Kernel Tracing Function Deficiency Inquiry**
Empty? Preposterous. The very idea that `kernel/trace/bpf_trace.c` could be *empty* is fundamentally at odds with the architecture. It's a core component, pivotal in the BPF tracing subsystem, and I *know* it houses the necessary functions. There has to be something present.
Alright, time to bypass the mental hand-wringing. I need definitive proof, a quick verification. No time for speculation. Let me pull up the file contents using `read-file` on `kernel/trace/bpf_trace.c`. That should give a quick answer to this.
|
| 455/6 |
2026/05/18 19:26 |
tool |
read-file |
0m
Args: map[File:kernel/trace/bpf_trace.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2011-2015 PLUMgrid, http://plumgrid.com
* Copyright (c) 2016 Facebook
*/
#include <linux/kernel.h>
#include <linux/types.h>
#include <linux/slab.h>
#include <linux/bpf.h>
#include <linux/bpf_verifier.h>
#include <linux/bpf_perf_event.h>
#include <linux/btf.h>
#include <linux/filter.h>
#include <linux/uaccess.h>
#include <linux/ctype.h>
#include <linux/kprobes.h>
#include <linux/spinlock.h>
#include <linux/syscalls.h>
#include <linux/error-injection.h>
#include <linux/btf_ids.h>
#include <linux/bpf_lsm.h>
#include <linux/fprobe.h>
#include <linux/bsearch.h>
#include <linux/sort.h>
#include <linux/key.h>
#include <linux/namei.h>
#include <net/bpf_sk_storage.h>
#include <uapi/linux/bpf.h>
#include <uapi/linux/btf.h>
#include <asm/tlb.h>
#include "trace_probe.h"
#include "trace.h"
#define CREATE_TRACE_POINTS
#include "bpf_trace.h"
#define bpf_event_rcu_dereference(p) \
rcu_dereference_protected(p, lockdep_is_held(&bpf_event_mutex))
#define MAX_UPROBE_MULTI_CNT (1U << 20)
#define MAX_KPROBE_MULTI_CNT (1U << 20)
#ifdef CONFIG_MODULES
struct bpf_trace_module {
struct module *module;
struct list_head list;
};
static LIST_HEAD(bpf_trace_modules);
static DEFINE_MUTEX(bpf_module_mutex);
static struct bpf_raw_event_map *bpf_get_raw_tracepoint_module(const char *name)
{
struct bpf_raw_event_map *btp, *ret = NULL;
struct bpf_trace_module *btm;
unsigned int i;
mutex_lock(&bpf_module_mutex);
list_for_each_entry(btm, &bpf_trace_modules, list) {
for (i = 0; i < btm->module->num_bpf_raw_events; ++i) {
btp = &btm->module->bpf_raw_events[i];
if (!strcmp(btp->tp->name, name)) {
if (try_module_get(btm->module))
ret = btp;
goto out;
}
}
}
out:
mutex_unlock(&bpf_module_mutex);
return ret;
}
#else
static struct bpf_raw_event_map *bpf_get_raw_tracepoint_module(const char *name)
{
return NULL;
}
#endif /* CONFIG_MODULES */
u64 bpf_get_stackid(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
u64 bpf_get_stack(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
u64 flags, const struct btf **btf,
s32 *btf_id);
static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx);
static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx);
static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx);
static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx);
/**
* trace_call_bpf - invoke BPF program
* @call: tracepoint event
* @ctx: opaque context pointer
*
* kprobe handlers execute BPF programs via this helper.
* Can be used from static tracepoints in the future.
*
* Return: BPF programs always return an integer which is interpreted by
* kprobe handler as:
* 0 - return from kprobe (event is filtered out)
* 1 - store kprobe event into ring buffer
* Other values are reserved and currently alias to 1
*/
unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx)
{
unsigned int ret;
cant_sleep();
if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) {
/*
* since some bpf program is already running on this cpu,
* don't call into another bpf program (same or different)
* and don't send kprobe event into ring-buffer,
* so return zero here
*/
rcu_read_lock();
bpf_prog_inc_misses_counters(rcu_dereference(call->prog_array));
rcu_read_unlock();
ret = 0;
goto out;
}
/*
* Instead of moving rcu_read_lock/rcu_dereference/rcu_read_unlock
* to all call sites, we did a bpf_prog_array_valid() there to check
* whether call->prog_array is empty or not, which is
* a heuristic to speed up execution.
*
* If bpf_prog_array_valid() fetched prog_array was
* non-NULL, we go into trace_call_bpf() and do the actual
* proper rcu_dereference() under RCU lock.
* If it turns out that prog_array is NULL then, we bail out.
* For the opposite, if the bpf_prog_array_valid() fetched pointer
* was NULL, you'll skip the prog_array with the risk of missing
* out of events when it was updated in between this and the
* rcu_dereference() which is accepted risk.
*/
rcu_read_lock();
ret = bpf_prog_run_array(rcu_dereference(call->prog_array),
ctx, bpf_prog_run);
rcu_read_unlock();
out:
__this_cpu_dec(bpf_prog_active);
return ret;
}
#ifdef CONFIG_BPF_KPROBE_OVERRIDE
BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc)
{
regs_set_return_value(regs, rc);
override_function_with_return(regs);
return 0;
}
static const struct bpf_func_proto bpf_override_return_proto = {
.func = bpf_override_return,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_ANYTHING,
};
#endif
static __always_inline int
bpf_probe_read_user_common(void *dst, u32 size, const void __user *unsafe_ptr)
{
int ret;
ret = copy_from_user_nofault(dst, unsafe_ptr, size);
if (unlikely(ret < 0))
memset(dst, 0, size);
return ret;
}
BPF_CALL_3(bpf_probe_read_user, void *, dst, u32, size,
const void __user *, unsafe_ptr)
{
return bpf_probe_read_user_common(dst, size, unsafe_ptr);
}
const struct bpf_func_proto bpf_probe_read_user_proto = {
.func = bpf_probe_read_user,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
static __always_inline int
bpf_probe_read_user_str_common(void *dst, u32 size,
const void __user *unsafe_ptr)
{
int ret;
/*
* NB: We rely on strncpy_from_user() not copying junk past the NUL
* terminator into `dst`.
*
* strncpy_from_user() does long-sized strides in the fast path. If the
* strncpy does not mask out the bytes after the NUL in `unsafe_ptr`,
* then there could be junk after the NUL in `dst`. If user takes `dst`
* and keys a hash map with it, then semantically identical strings can
* occupy multiple entries in the map.
*/
ret = strncpy_from_user_nofault(dst, unsafe_ptr, size);
if (unlikely(ret < 0))
memset(dst, 0, size);
return ret;
}
BPF_CALL_3(bpf_probe_read_user_str, void *, dst, u32, size,
const void __user *, unsafe_ptr)
{
return bpf_probe_read_user_str_common(dst, size, unsafe_ptr);
}
const struct bpf_func_proto bpf_probe_read_user_str_proto = {
.func = bpf_probe_read_user_str,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_probe_read_kernel, void *, dst, u32, size,
const void *, unsafe_ptr)
{
return bpf_probe_read_kernel_common(dst, size, unsafe_ptr);
}
const struct bpf_func_proto bpf_probe_read_kernel_proto = {
.func = bpf_probe_read_kernel,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
static __always_inline int
bpf_probe_read_kernel_str_common(void *dst, u32 size, const void *unsafe_ptr)
{
int ret;
/*
* The strncpy_from_kernel_nofault() call will likely not fill the
* entire buffer, but that's okay in this circumstance as we're probing
* arbitrary memory anyway similar to bpf_probe_read_*() and might
* as well probe the stack. Thus, memory is explicitly cleared
* only in error case, so that improper users ignoring return
* code altogether don't copy garbage; otherwise length of string
* is returned that can be used for bpf_perf_event_output() et al.
*/
ret = strncpy_from_kernel_nofault(dst, unsafe_ptr, size);
if (unlikely(ret < 0))
memset(dst, 0, size);
return ret;
}
BPF_CALL_3(bpf_probe_read_kernel_str, void *, dst, u32, size,
const void *, unsafe_ptr)
{
return bpf_probe_read_kernel_str_common(dst, size, unsafe_ptr);
}
const struct bpf_func_proto bpf_probe_read_kernel_str_proto = {
.func = bpf_probe_read_kernel_str,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
BPF_CALL_3(bpf_probe_read_compat, void *, dst, u32, size,
const void *, unsafe_ptr)
{
if ((unsigned long)unsafe_ptr < TASK_SIZE) {
return bpf_probe_read_user_common(dst, size,
(__force void __user *)unsafe_ptr);
}
return bpf_probe_read_kernel_common(dst, size, unsafe_ptr);
}
static const struct bpf_func_proto bpf_probe_read_compat_proto = {
.func = bpf_probe_read_compat,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_probe_read_compat_str, void *, dst, u32, size,
const void *, unsafe_ptr)
{
if ((unsigned long)unsafe_ptr < TASK_SIZE) {
return bpf_probe_read_user_str_common(dst, size,
(__force void __user *)unsafe_ptr);
}
return bpf_probe_read_kernel_str_common(dst, size, unsafe_ptr);
}
static const struct bpf_func_proto bpf_probe_read_compat_str_proto = {
.func = bpf_probe_read_compat_str,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
#endif /* CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE */
BPF_CALL_3(bpf_probe_write_user, void __user *, unsafe_ptr, const void *, src,
u32, size)
{
/*
* Ensure we're in user context which is safe for the helper to
* run. This helper has no business in a kthread.
*
* access_ok() should prevent writing to non-user memory, but in
* some situations (nommu, temporary switch, etc) access_ok() does
* not provide enough validation, hence the check on KERNEL_DS.
*
* nmi_uaccess_okay() ensures the probe is not run in an interim
* state, when the task or mm are switched. This is specifically
* required to prevent the use of temporary mm.
*/
if (unlikely(in_interrupt() ||
current->flags & (PF_KTHREAD | PF_EXITING)))
return -EPERM;
if (unlikely(!nmi_uaccess_okay()))
return -EPERM;
return copy_to_user_nofault(unsafe_ptr, src, size);
}
static const struct bpf_func_proto bpf_probe_write_user_proto = {
.func = bpf_probe_write_user,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
};
#define MAX_TRACE_PRINTK_VARARGS 3
#define BPF_TRACE_PRINTK_SIZE 1024
BPF_CALL_5(bpf_trace_printk, char *, fmt, u32, fmt_size, u64, arg1,
u64, arg2, u64, arg3)
{
u64 args[MAX_TRACE_PRINTK_VARARGS] = { arg1, arg2, arg3 };
struct bpf_bprintf_data data = {
.get_bin_args = true,
.get_buf = true,
};
int ret;
ret = bpf_bprintf_prepare(fmt, fmt_size, args,
MAX_TRACE_PRINTK_VARARGS, &data);
if (ret < 0)
return ret;
ret = bstr_printf(data.buf, MAX_BPRINTF_BUF, fmt, data.bin_args);
trace_bpf_trace_printk(data.buf);
bpf_bprintf_cleanup(&data);
return ret;
}
static const struct bpf_func_proto bpf_trace_printk_proto = {
.func = bpf_trace_printk,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
};
static void __set_printk_clr_event(struct work_struct *work)
{
/*
* This program might be calling bpf_trace_printk,
* so enable the associated bpf_trace/bpf_trace_printk event.
* Repeat this each time as it is possible a user has
* disabled bpf_trace_printk events. By loading a program
* calling bpf_trace_printk() however the user has expressed
* the intent to see such events.
*/
if (trace_set_clr_event("bpf_trace", "bpf_trace_printk", 1))
pr_warn_ratelimited("could not enable bpf_trace_printk events");
}
static DECLARE_WORK(set_printk_work, __set_printk_clr_event);
const struct bpf_func_proto *bpf_get_trace_printk_proto(void)
{
schedule_work(&set_printk_work);
return &bpf_trace_printk_proto;
}
BPF_CALL_4(bpf_trace_vprintk, char *, fmt, u32, fmt_size, const void *, args,
u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
.get_buf = true,
};
int ret, num_args;
if (data_len & 7 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
ret = bpf_bprintf_prepare(fmt, fmt_size, args, num_args, &data);
if (ret < 0)
return ret;
ret = bstr_printf(data.buf, MAX_BPRINTF_BUF, fmt, data.bin_args);
trace_bpf_trace_printk(data.buf);
bpf_bprintf_cleanup(&data);
return ret;
}
static const struct bpf_func_proto bpf_trace_vprintk_proto = {
.func = bpf_trace_vprintk,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
};
const struct bpf_func_proto *bpf_get_trace_vprintk_proto(void)
{
schedule_work(&set_printk_work);
return &bpf_trace_vprintk_proto;
}
BPF_CALL_5(bpf_seq_printf, struct seq_file *, m, char *, fmt, u32, fmt_size,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len & 7 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
err = bpf_bprintf_prepare(fmt, fmt_size, args, num_args, &data);
if (err < 0)
return err;
seq_bprintf(m, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return seq_has_overflowed(m) ? -EOVERFLOW : 0;
}
BTF_ID_LIST_SINGLE(btf_seq_file_ids, struct, seq_file)
static const struct bpf_func_proto bpf_seq_printf_proto = {
.func = bpf_seq_printf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_seq_file_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len)
{
return seq_write(m, data, len) ? -EOVERFLOW : 0;
}
static const struct bpf_func_proto bpf_seq_write_proto = {
.func = bpf_seq_write,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_seq_file_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_4(bpf_seq_printf_btf, struct seq_file *, m, struct btf_ptr *, ptr,
u32, btf_ptr_size, u64, flags)
{
const struct btf *btf;
s32 btf_id;
int ret;
ret = bpf_btf_printf_prepare(ptr, btf_ptr_size, flags, &btf, &btf_id);
if (ret)
return ret;
return btf_type_seq_show_flags(btf, btf_id, ptr->ptr, m, flags);
}
static const struct bpf_func_proto bpf_seq_printf_btf_proto = {
.func = bpf_seq_printf_btf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_seq_file_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
static __always_inline int
get_map_perf_counter(struct bpf_map *map, u64 flags,
u64 *value, u64 *enabled, u64 *running)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
unsigned int cpu = smp_processor_id();
u64 index = flags & BPF_F_INDEX_MASK;
struct bpf_event_entry *ee;
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
if (index == BPF_F_CURRENT_CPU)
index = cpu;
if (unlikely(index >= array->map.max_entries))
return -E2BIG;
ee = READ_ONCE(array->ptrs[index]);
if (!ee)
return -ENOENT;
return perf_event_read_local(ee->event, value, enabled, running);
}
BPF_CALL_2(bpf_perf_event_read, struct bpf_map *, map, u64, flags)
{
u64 value = 0;
int err;
err = get_map_perf_counter(map, flags, &value, NULL, NULL);
/*
* this api is ugly since we miss [-22..-2] range of valid
* counter values, but that's uapi
*/
if (err)
return err;
return value;
}
const struct bpf_func_proto bpf_perf_event_read_proto = {
.func = bpf_perf_event_read,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
};
BPF_CALL_4(bpf_perf_event_read_value, struct bpf_map *, map, u64, flags,
struct bpf_perf_event_value *, buf, u32, size)
{
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_perf_event_value)))
goto clear;
err = get_map_perf_counter(map, flags, &buf->counter, &buf->enabled,
&buf->running);
if (unlikely(err))
goto clear;
return 0;
clear:
memset(buf, 0, size);
return err;
}
static const struct bpf_func_proto bpf_perf_event_read_value_proto = {
.func = bpf_perf_event_read_value,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
};
const struct bpf_func_proto *bpf_get_perf_event_read_value_proto(void)
{
return &bpf_perf_event_read_value_proto;
}
static __always_inline u64
__bpf_perf_event_output(struct pt_regs *regs, struct bpf_map *map,
u64 flags, struct perf_raw_record *raw,
struct perf_sample_data *sd)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
unsigned int cpu = smp_processor_id();
u64 index = flags & BPF_F_INDEX_MASK;
struct bpf_event_entry *ee;
struct perf_event *event;
if (index == BPF_F_CURRENT_CPU)
index = cpu;
if (unlikely(index >= array->map.max_entries))
return -E2BIG;
ee = READ_ONCE(array->ptrs[index]);
if (!ee)
return -ENOENT;
event = ee->event;
if (unlikely(event->attr.type != PERF_TYPE_SOFTWARE ||
event->attr.config != PERF_COUNT_SW_BPF_OUTPUT))
return -EINVAL;
if (unlikely(event->oncpu != cpu))
return -EOPNOTSUPP;
perf_sample_save_raw_data(sd, event, raw);
return perf_event_output(event, sd, regs);
}
/*
* Support executing tracepoints in normal, irq, and nmi context that each call
* bpf_perf_event_output
*/
struct bpf_trace_sample_data {
struct perf_sample_data sds[3];
};
static DEFINE_PER_CPU(struct bpf_trace_sample_data, bpf_trace_sds);
static DEFINE_PER_CPU(int, bpf_trace_nest_level);
BPF_CALL_5(bpf_perf_event_output, struct pt_regs *, regs, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
struct bpf_trace_sample_data *sds;
struct perf_raw_record raw = {
.frag = {
.size = size,
.data = data,
},
};
struct perf_sample_data *sd;
int nest_level, err;
preempt_disable();
sds = this_cpu_ptr(&bpf_trace_sds);
nest_level = this_cpu_inc_return(bpf_trace_nest_level);
if (WARN_ON_ONCE(nest_level > ARRAY_SIZE(sds->sds))) {
err = -EBUSY;
goto out;
}
sd = &sds->sds[nest_level - 1];
if (unlikely(flags & ~(BPF_F_INDEX_MASK))) {
err = -EINVAL;
goto out;
}
perf_sample_data_init(sd, 0, 0);
err = __bpf_perf_event_output(regs, map, flags, &raw, sd);
out:
this_cpu_dec(bpf_trace_nest_level);
preempt_enable();
return err;
}
static const struct bpf_func_proto bpf_perf_event_output_proto = {
.func = bpf_perf_event_output,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
static DEFINE_PER_CPU(int, bpf_event_output_nest_level);
struct bpf_nested_pt_regs {
struct pt_regs regs[3];
};
static DEFINE_PER_CPU(struct bpf_nested_pt_regs, bpf_pt_regs);
static DEFINE_PER_CPU(struct bpf_trace_sample_data, bpf_misc_sds);
u64 bpf_event_output(struct bpf_map *map, u64 flags, void *meta, u64 meta_size,
void *ctx, u64 ctx_size, bpf_ctx_copy_t ctx_copy)
{
struct perf_raw_frag frag = {
.copy = ctx_copy,
.size = ctx_size,
.data = ctx,
};
struct perf_raw_record raw = {
.frag = {
{
.next = ctx_size ? &frag : NULL,
},
.size = meta_size,
.data = meta,
},
};
struct perf_sample_data *sd;
struct pt_regs *regs;
int nest_level;
u64 ret;
preempt_disable();
nest_level = this_cpu_inc_return(bpf_event_output_nest_level);
if (WARN_ON_ONCE(nest_level > ARRAY_SIZE(bpf_misc_sds.sds))) {
ret = -EBUSY;
goto out;
}
sd = this_cpu_ptr(&bpf_misc_sds.sds[nest_level - 1]);
regs = this_cpu_ptr(&bpf_pt_regs.regs[nest_level - 1]);
perf_fetch_caller_regs(regs);
perf_sample_data_init(sd, 0, 0);
ret = __bpf_perf_event_output(regs, map, flags, &raw, sd);
out:
this_cpu_dec(bpf_event_output_nest_level);
preempt_enable();
return ret;
}
BPF_CALL_0(bpf_get_current_task)
{
return (long) current;
}
const struct bpf_func_proto bpf_get_current_task_proto = {
.func = bpf_get_current_task,
.gpl_only = true,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_task_btf)
{
return (unsigned long) current;
}
const struct bpf_func_proto bpf_get_current_task_btf_proto = {
.func = bpf_get_current_task_btf,
.gpl_only = true,
.ret_type = RET_PTR_TO_BTF_ID_TRUSTED,
.ret_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
};
BPF_CALL_1(bpf_task_pt_regs, struct task_struct *, task)
{
return (unsigned long) task_pt_regs(task);
}
BTF_ID_LIST_SINGLE(bpf_task_pt_regs_ids, struct, pt_regs)
const struct bpf_func_proto bpf_task_pt_regs_proto = {
.func = bpf_task_pt_regs,
.gpl_only = true,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.ret_type = RET_PTR_TO_BTF_ID,
.ret_btf_id = &bpf_task_pt_regs_ids[0],
};
struct send_signal_irq_work {
struct irq_work irq_work;
struct task_struct *task;
u32 sig;
enum pid_type type;
bool has_siginfo;
struct kernel_siginfo info;
};
static DEFINE_PER_CPU(struct send_signal_irq_work, send_signal_work);
static void do_bpf_send_signal(struct irq_work *entry)
{
struct send_signal_irq_work *work;
struct kernel_siginfo *siginfo;
work = container_of(entry, struct send_signal_irq_work, irq_work);
siginfo = work->has_siginfo ? &work->info : SEND_SIG_PRIV;
group_send_sig_info(work->sig, siginfo, work->task, work->type);
put_task_struct(work->task);
}
static int bpf_send_signal_common(u32 sig, enum pid_type type, struct task_struct *task, u64 value)
{
struct send_signal_irq_work *work = NULL;
struct kernel_siginfo info;
struct kernel_siginfo *siginfo;
if (!task) {
task = current;
siginfo = SEND_SIG_PRIV;
} else {
clear_siginfo(&info);
info.si_signo = sig;
info.si_errno = 0;
info.si_code = SI_KERNEL;
info.si_pid = 0;
info.si_uid = 0;
info.si_value.sival_ptr = (void __user __force *)(unsigned long)value;
siginfo = &info;
}
/* Similar to bpf_probe_write_user, task needs to be
* in a sound condition and kernel memory access be
* permitted in order to send signal to the current
* task.
*/
if (unlikely(task->flags & (PF_KTHREAD | PF_EXITING)))
return -EPERM;
if (unlikely(!nmi_uaccess_okay()))
return -EPERM;
/* Task should not be pid=1 to avoid kernel panic. */
if (unlikely(is_global_init(task)))
return -EPERM;
if (preempt_count() != 0 || irqs_disabled()) {
/* Do an early check on signal validity. Otherwise,
* the error is lost in deferred irq_work.
*/
if (unlikely(!valid_signal(sig)))
return -EINVAL;
work = this_cpu_ptr(&send_signal_work);
if (irq_work_is_busy(&work->irq_work))
return -EBUSY;
/* Add the current task, which is the target of sending signal,
* to the irq_work. The current task may change when queued
* irq works get executed.
*/
work->task = get_task_struct(task);
work->has_siginfo = siginfo == &info;
if (work->has_siginfo)
copy_siginfo(&work->info, &info);
work->sig = sig;
work->type = type;
irq_work_queue(&work->irq_work);
return 0;
}
return group_send_sig_info(sig, siginfo, task, type);
}
BPF_CALL_1(bpf_send_signal, u32, sig)
{
return bpf_send_signal_common(sig, PIDTYPE_TGID, NULL, 0);
}
const struct bpf_func_proto bpf_send_signal_proto = {
.func = bpf_send_signal,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_send_signal_thread, u32, sig)
{
return bpf_send_signal_common(sig, PIDTYPE_PID, NULL, 0);
}
const struct bpf_func_proto bpf_send_signal_thread_proto = {
.func = bpf_send_signal_thread,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_d_path, const struct path *, path, char *, buf, u32, sz)
{
struct path copy;
long len;
char *p;
if (!sz)
return 0;
/*
* The path pointer is verified as trusted and safe to use,
* but let's double check it's valid anyway to workaround
* potentially broken verifier.
*/
len = copy_from_kernel_nofault(©, path, sizeof(*path));
if (len < 0)
return len;
p = d_path(©, buf, sz);
if (IS_ERR(p)) {
len = PTR_ERR(p);
} else {
len = buf + sz - p;
memmove(buf, p, len);
}
return len;
}
BTF_SET_START(btf_allowlist_d_path)
#ifdef CONFIG_SECURITY
BTF_ID(func, security_file_permission)
BTF_ID(func, security_inode_getattr)
BTF_ID(func, security_file_open)
#endif
#ifdef CONFIG_SECURITY_PATH
BTF_ID(func, security_path_truncate)
#endif
BTF_ID(func, vfs_truncate)
BTF_ID(func, vfs_fallocate)
BTF_ID(func, dentry_open)
BTF_ID(func, vfs_getattr)
BTF_ID(func, filp_close)
BTF_SET_END(btf_allowlist_d_path)
static bool bpf_d_path_allowed(const struct bpf_prog *prog)
{
if (prog->type == BPF_PROG_TYPE_TRACING &&
prog->expected_attach_type == BPF_TRACE_ITER)
return true;
if (prog->type == BPF_PROG_TYPE_LSM)
return bpf_lsm_is_sleepable_hook(prog->aux->attach_btf_id);
return btf_id_set_contains(&btf_allowlist_d_path,
prog->aux->attach_btf_id);
}
BTF_ID_LIST_SINGLE(bpf_d_path_btf_ids, struct, path)
static const struct bpf_func_proto bpf_d_path_proto = {
.func = bpf_d_path,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_BTF_ID,
.arg1_btf_id = &bpf_d_path_btf_ids[0],
.arg2_type = ARG_PTR_TO_MEM | MEM_WRITE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.allowed = bpf_d_path_allowed,
};
#define BTF_F_ALL (BTF_F_COMPACT | BTF_F_NONAME | \
BTF_F_PTR_RAW | BTF_F_ZERO)
static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
u64 flags, const struct btf **btf,
s32 *btf_id)
{
const struct btf_type *t;
if (unlikely(flags & ~(BTF_F_ALL)))
return -EINVAL;
if (btf_ptr_size != sizeof(struct btf_ptr))
return -EINVAL;
*btf = bpf_get_btf_vmlinux();
if (IS_ERR_OR_NULL(*btf))
return IS_ERR(*btf) ? PTR_ERR(*btf) : -EINVAL;
if (ptr->type_id > 0)
*btf_id = ptr->type_id;
else
return -EINVAL;
if (*btf_id > 0)
t = btf_type_by_id(*btf, *btf_id);
if (*btf_id <= 0 || !t)
return -ENOENT;
return 0;
}
BPF_CALL_5(bpf_snprintf_btf, char *, str, u32, str_size, struct btf_ptr *, ptr,
u32, btf_ptr_size, u64, flags)
{
const struct btf *btf;
s32 btf_id;
int ret;
ret = bpf_btf_printf_prepare(ptr, btf_ptr_size, flags, &btf, &btf_id);
if (ret)
return ret;
return btf_type_snprintf_show(btf, btf_id, ptr->ptr, str, str_size,
flags);
}
const struct bpf_func_proto bpf_snprintf_btf_proto = {
.func = bpf_snprintf_btf,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE,
.arg5_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_get_func_ip_tracing, void *, ctx)
{
/* This helper call is inlined by verifier. */
return ((u64 *)ctx)[-2];
}
static const struct bpf_func_proto bpf_get_func_ip_proto_tracing = {
.func = bpf_get_func_ip_tracing,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
static inline unsigned long get_entry_ip(unsigned long fentry_ip)
{
#ifdef CONFIG_X86_KERNEL_IBT
if (is_endbr((void *)(fentry_ip - ENDBR_INSN_SIZE)))
fentry_ip -= ENDBR_INSN_SIZE;
#endif
return fentry_ip;
}
BPF_CALL_1(bpf_get_func_ip_kprobe, struct pt_regs *, regs)
{
struct bpf_trace_run_ctx *run_ctx __maybe_unused;
struct kprobe *kp;
#ifdef CONFIG_UPROBES
run_ctx = container_of(current->bpf_ctx, struct bpf_trace_run_ctx, run_ctx);
if (run_ctx->is_uprobe)
return ((struct uprobe_dispatch_data *)current->utask->vaddr)->bp_addr;
#endif
kp = kprobe_running();
if (!kp || !(kp->flags & KPROBE_FLAG_ON_FUNC_ENTRY))
return 0;
return get_entry_ip((uintptr_t)kp->addr);
}
static const struct bpf_func_proto bpf_get_func_ip_proto_kprobe = {
.func = bpf_get_func_ip_kprobe,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_func_ip_kprobe_multi, struct pt_regs *, regs)
{
return bpf_kprobe_multi_entry_ip(current->bpf_ctx);
}
static const struct bpf_func_proto bpf_get_func_ip_proto_kprobe_multi = {
.func = bpf_get_func_ip_kprobe_multi,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_attach_cookie_kprobe_multi, struct pt_regs *, regs)
{
return bpf_kprobe_multi_cookie(current->bpf_ctx);
}
static const struct bpf_func_proto bpf_get_attach_cookie_proto_kmulti = {
.func = bpf_get_attach_cookie_kprobe_multi,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_func_ip_uprobe_multi, struct pt_regs *, regs)
{
return bpf_uprobe_multi_entry_ip(current->bpf_ctx);
}
static const struct bpf_func_proto bpf_get_func_ip_proto_uprobe_multi = {
.func = bpf_get_func_ip_uprobe_multi,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_attach_cookie_uprobe_multi, struct pt_regs *, regs)
{
return bpf_uprobe_multi_cookie(current->bpf_ctx);
}
static const struct bpf_func_proto bpf_get_attach_cookie_proto_umulti = {
.func = bpf_get_attach_cookie_uprobe_multi,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_attach_cookie_trace, void *, ctx)
{
struct bpf_trace_run_ctx *run_ctx;
run_ctx = container_of(current->bpf_ctx, struct bpf_trace_run_ctx, run_ctx);
return run_ctx->bpf_cookie;
}
static const struct bpf_func_proto bpf_get_attach_cookie_proto_trace = {
.func = bpf_get_attach_cookie_trace,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_attach_cookie_pe, struct bpf_perf_event_data_kern *, ctx)
{
return ctx->event->bpf_cookie;
}
static const struct bpf_func_proto bpf_get_attach_cookie_proto_pe = {
.func = bpf_get_attach_cookie_pe,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_attach_cookie_tracing, void *, ctx)
{
struct bpf_trace_run_ctx *run_ctx;
run_ctx = container_of(current->bpf_ctx, struct bpf_trace_run_ctx, run_ctx);
return run_ctx->bpf_cookie;
}
static const struct bpf_func_proto bpf_get_attach_cookie_proto_tracing = {
.func = bpf_get_attach_cookie_tracing,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_3(bpf_get_branch_snapshot, void *, buf, u32, size, u64, flags)
{
static const u32 br_entry_size = sizeof(struct perf_branch_entry);
u32 entry_cnt = size / br_entry_size;
entry_cnt = static_call(perf_snapshot_branch_stack)(buf, entry_cnt);
if (unlikely(flags))
return -EINVAL;
if (!entry_cnt)
return -ENOENT;
return entry_cnt * br_entry_size;
}
const struct bpf_func_proto bpf_get_branch_snapshot_proto = {
.func = bpf_get_branch_snapshot,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(get_func_arg, void *, ctx, u32, n, u64 *, value)
{
/* This helper call is inlined by verifier. */
u64 nr_args = ((u64 *)ctx)[-1] & 0xFF;
if ((u64) n >= nr_args)
return -EINVAL;
*value = ((u64 *)ctx)[n];
return 0;
}
static const struct bpf_func_proto bpf_get_func_arg_proto = {
.func = get_func_arg,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg3_size = sizeof(u64),
};
BPF_CALL_2(get_func_ret, void *, ctx, u64 *, value)
{
/* This helper call is inlined by verifier. */
u64 nr_args = ((u64 *)ctx)[-1] & 0xFF;
*value = ((u64 *)ctx)[nr_args];
return 0;
}
static const struct bpf_func_proto bpf_get_func_ret_proto = {
.func = get_func_ret,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg2_size = sizeof(u64),
};
BPF_CALL_1(get_func_arg_cnt, void *, ctx)
{
/* This helper call is inlined by verifier. */
return ((u64 *)ctx)[-1] & 0xFF;
}
static const struct bpf_func_proto bpf_get_func_arg_cnt_proto = {
.func = get_func_arg_cnt,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
static const struct bpf_func_proto *
bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
const struct bpf_func_proto *func_proto;
switch (func_id) {
case BPF_FUNC_get_smp_processor_id:
return &bpf_get_smp_processor_id_proto;
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
case BPF_FUNC_probe_read:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_compat_proto;
case BPF_FUNC_probe_read_str:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_compat_str_proto;
#endif
case BPF_FUNC_get_func_ip:
return &bpf_get_func_ip_proto_tracing;
default:
break;
}
func_proto = bpf_base_func_proto(func_id, prog);
if (func_proto)
return func_proto;
if (!bpf_token_capable(prog->aux->token, CAP_SYS_ADMIN))
return NULL;
switch (func_id) {
case BPF_FUNC_probe_write_user:
return security_locked_down(LOCKDOWN_BPF_WRITE_USER) < 0 ?
NULL : &bpf_probe_write_user_proto;
default:
return NULL;
}
}
static bool is_kprobe_multi(const struct bpf_prog *prog)
{
return prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI ||
prog->expected_attach_type == BPF_TRACE_KPROBE_SESSION;
}
static inline bool is_kprobe_session(const struct bpf_prog *prog)
{
return prog->type == BPF_PROG_TYPE_KPROBE &&
prog->expected_attach_type == BPF_TRACE_KPROBE_SESSION;
}
static inline bool is_uprobe_multi(const struct bpf_prog *prog)
{
return prog->expected_attach_type == BPF_TRACE_UPROBE_MULTI ||
prog->expected_attach_type == BPF_TRACE_UPROBE_SESSION;
}
static inline bool is_uprobe_session(const struct bpf_prog *prog)
{
return prog->type == BPF_PROG_TYPE_KPROBE &&
prog->expected_attach_type == BPF_TRACE_UPROBE_SESSION;
}
static inline bool is_trace_fsession(const struct bpf_prog *prog)
{
return prog->type == BPF_PROG_TYPE_TRACING &&
prog->expected_attach_type == BPF_TRACE_FSESSION;
}
static const struct bpf_func_proto *
kprobe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_perf_event_output:
return &bpf_perf_event_output_proto;
case BPF_FUNC_get_stackid:
return &bpf_get_stackid_proto;
case BPF_FUNC_get_stack:
return prog->sleepable ? &bpf_get_stack_sleepable_proto : &bpf_get_stack_proto;
#ifdef CONFIG_BPF_KPROBE_OVERRIDE
case BPF_FUNC_override_return:
return &bpf_override_return_proto;
#endif
case BPF_FUNC_get_func_ip:
if (is_kprobe_multi(prog))
return &bpf_get_func_ip_proto_kprobe_multi;
if (is_uprobe_multi(prog))
return &bpf_get_func_ip_proto_uprobe_multi;
return &bpf_get_func_ip_proto_kprobe;
case BPF_FUNC_get_attach_cookie:
if (is_kprobe_multi(prog))
return &bpf_get_attach_cookie_proto_kmulti;
if (is_uprobe_multi(prog))
return &bpf_get_attach_cookie_proto_umulti;
return &bpf_get_attach_cookie_proto_trace;
default:
return bpf_tracing_func_proto(func_id, prog);
}
}
/* bpf+kprobe programs can access fields of 'struct pt_regs' */
static bool kprobe_prog_is_valid_access(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
if (off < 0 || off >= sizeof(struct pt_regs))
return false;
if (off % size != 0)
return false;
/*
* Assertion for 32 bit to make sure last 8 byte access
* (BPF_DW) to the last 4 byte member is disallowed.
*/
if (off + size > sizeof(struct pt_regs))
return false;
if (type == BPF_WRITE)
prog->aux->kprobe_write_ctx = true;
return true;
}
const struct bpf_verifier_ops kprobe_verifier_ops = {
.get_func_proto = kprobe_prog_func_proto,
.is_valid_access = kprobe_prog_is_valid_access,
};
const struct bpf_prog_ops kprobe_prog_ops = {
};
BPF_CALL_5(bpf_perf_event_output_tp, void *, tp_buff, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
struct pt_regs *regs = *(struct pt_regs **)tp_buff;
/*
* r1 points to perf tracepoint buffer where first 8 bytes are hidden
* from bpf program and contain a pointer to 'struct pt_regs'. Fetch it
* from there and call the same bpf_perf_event_output() helper inline.
*/
return ____bpf_perf_event_output(regs, map, flags, data, size);
}
static const struct bpf_func_proto bpf_perf_event_output_proto_tp = {
.func = bpf_perf_event_output_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_get_stackid_tp, void *, tp_buff, struct bpf_map *, map,
u64, flags)
{
struct pt_regs *regs = *(struct pt_regs **)tp_buff;
/*
* Same comment as in bpf_perf_event_output_tp(), only that this time
* the other helper's function body cannot be inlined due to being
* external, thus we need to call raw helper function.
*/
return bpf_get_stackid((unsigned long) regs, (unsigned long) map,
flags, 0, 0);
}
static const struct bpf_func_proto bpf_get_stackid_proto_tp = {
.func = bpf_get_stackid_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_4(bpf_get_stack_tp, void *, tp_buff, void *, buf, u32, size,
u64, flags)
{
struct pt_regs *regs = *(struct pt_regs **)tp_buff;
return bpf_get_stack((unsigned long) regs, (unsigned long) buf,
(unsigned long) size, flags, 0);
}
static const struct bpf_func_proto bpf_get_stack_proto_tp = {
.func = bpf_get_stack_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
static const struct bpf_func_proto *
tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_perf_event_output:
return &bpf_perf_event_output_proto_tp;
case BPF_FUNC_get_stackid:
return &bpf_get_stackid_proto_tp;
case BPF_FUNC_get_stack:
return &bpf_get_stack_proto_tp;
case BPF_FUNC_get_attach_cookie:
return &bpf_get_attach_cookie_proto_trace;
default:
return bpf_tracing_func_proto(func_id, prog);
}
}
static bool tp_prog_is_valid_access(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
if (off < sizeof(void *) || off >= PERF_MAX_TRACE_SIZE)
return false;
if (type != BPF_READ)
return false;
if (off % size != 0)
return false;
BUILD_BUG_ON(PERF_MAX_TRACE_SIZE % sizeof(__u64));
return true;
}
const struct bpf_verifier_ops tracepoint_verifier_ops = {
.get_func_proto = tp_prog_func_proto,
.is_valid_access = tp_prog_is_valid_access,
};
const struct bpf_prog_ops tracepoint_prog_ops = {
};
BPF_CALL_3(bpf_perf_prog_read_value, struct bpf_perf_event_data_kern *, ctx,
struct bpf_perf_event_value *, buf, u32, size)
{
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_perf_event_value)))
goto clear;
err = perf_event_read_local(ctx->event, &buf->counter, &buf->enabled,
&buf->running);
if (unlikely(err))
goto clear;
return 0;
clear:
memset(buf, 0, size);
return err;
}
static const struct bpf_func_proto bpf_perf_prog_read_value_proto = {
.func = bpf_perf_prog_read_value,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE,
};
BPF_CALL_4(bpf_read_branch_records, struct bpf_perf_event_data_kern *, ctx,
void *, buf, u32, size, u64, flags)
{
static const u32 br_entry_size = sizeof(struct perf_branch_entry);
struct perf_branch_stack *br_stack = ctx->data->br_stack;
u32 to_copy;
if (unlikely(flags & ~BPF_F_GET_BRANCH_RECORDS_SIZE))
return -EINVAL;
if (unlikely(!(ctx->data->sample_flags & PERF_SAMPLE_BRANCH_STACK)))
return -ENOENT;
if (unlikely(!br_stack))
return -ENOENT;
if (flags & BPF_F_GET_BRANCH_RECORDS_SIZE)
return br_stack->nr * br_entry_size;
if (!buf || (size % br_entry_size != 0))
return -EINVAL;
to_copy = min_t(u32, br_stack->nr * br_entry_size, size);
memcpy(buf, br_stack->entries, to_copy);
return to_copy;
}
static const struct bpf_func_proto bpf_read_branch_records_proto = {
.func = bpf_read_branch_records,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
static const struct bpf_func_proto *
pe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_perf_event_output:
return &bpf_perf_event_output_proto_tp;
case BPF_FUNC_get_stackid:
return &bpf_get_stackid_proto_pe;
case BPF_FUNC_get_stack:
return &bpf_get_stack_proto_pe;
case BPF_FUNC_perf_prog_read_value:
return &bpf_perf_prog_read_value_proto;
case BPF_FUNC_read_branch_records:
return &bpf_read_branch_records_proto;
case BPF_FUNC_get_attach_cookie:
return &bpf_get_attach_cookie_proto_pe;
default:
return bpf_tracing_func_proto(func_id, prog);
}
}
/*
* bpf_raw_tp_regs are separate from bpf_pt_regs used from skb/xdp
* to avoid potential recursive reuse issue when/if tracepoints are added
* inside bpf_*_event_output, bpf_get_stackid and/or bpf_get_stack.
*
* Since raw tracepoints run despite bpf_prog_active, support concurrent usage
* in normal, irq, and nmi context.
*/
struct bpf_raw_tp_regs {
struct pt_regs regs[3];
};
static DEFINE_PER_CPU(struct bpf_raw_tp_regs, bpf_raw_tp_regs);
static DEFINE_PER_CPU(int, bpf_raw_tp_nest_level);
static struct pt_regs *get_bpf_raw_tp_regs(void)
{
struct bpf_raw_tp_regs *tp_regs = this_cpu_ptr(&bpf_raw_tp_regs);
int nest_level = this_cpu_inc_return(bpf_raw_tp_nest_level);
if (nest_level > ARRAY_SIZE(tp_regs->regs)) {
this_cpu_dec(bpf_raw_tp_nest_level);
return ERR_PTR(-EBUSY);
}
return &tp_regs->regs[nest_level - 1];
}
static void put_bpf_raw_tp_regs(void)
{
this_cpu_dec(bpf_raw_tp_nest_level);
}
BPF_CALL_5(bpf_perf_event_output_raw_tp, struct bpf_raw_tracepoint_args *, args,
struct bpf_map *, map, u64, flags, void *, data, u64, size)
{
struct pt_regs *regs = get_bpf_raw_tp_regs();
int ret;
if (IS_ERR(regs))
return PTR_ERR(regs);
perf_fetch_caller_regs(regs);
ret = ____bpf_perf_event_output(regs, map, flags, data, size);
put_bpf_raw_tp_regs();
return ret;
}
static const struct bpf_func_proto bpf_perf_event_output_proto_raw_tp = {
.func = bpf_perf_event_output_raw_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
extern const struct bpf_func_proto bpf_skb_output_proto;
extern const struct bpf_func_proto bpf_xdp_output_proto;
extern const struct bpf_func_proto bpf_xdp_get_buff_len_trace_proto;
BPF_CALL_3(bpf_get_stackid_raw_tp, struct bpf_raw_tracepoint_args *, args,
struct bpf_map *, map, u64, flags)
{
struct pt_regs *regs = get_bpf_raw_tp_regs();
int ret;
if (IS_ERR(regs))
return PTR_ERR(regs);
perf_fetch_caller_regs(regs);
/* similar to bpf_perf_event_output_tp, but pt_regs fetched differently */
ret = bpf_get_stackid((unsigned long) regs, (unsigned long) map,
flags, 0, 0);
put_bpf_raw_tp_regs();
return ret;
}
static const struct bpf_func_proto bpf_get_stackid_proto_raw_tp = {
.func = bpf_get_stackid_raw_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args,
void *, buf, u32, size, u64, flags)
{
struct pt_regs *regs = get_bpf_raw_tp_regs();
int ret;
if (IS_ERR(regs))
return PTR_ERR(regs);
perf_fetch_caller_regs(regs);
ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf,
(unsigned long) size, flags, 0);
put_bpf_raw_tp_regs();
return ret;
}
static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
.func = bpf_get_stack_raw_tp,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_UNINIT_MEM,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
};
static const struct bpf_func_proto *
raw_tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_perf_event_output:
return &bpf_perf_event_output_proto_raw_tp;
case BPF_FUNC_get_stackid:
return &bpf_get_stackid_proto_raw_tp;
case BPF_FUNC_get_stack:
return &bpf_get_stack_proto_raw_tp;
case BPF_FUNC_get_attach_cookie:
return &bpf_get_attach_cookie_proto_tracing;
default:
return bpf_tracing_func_proto(func_id, prog);
}
}
const struct bpf_func_proto *
tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
const struct bpf_func_proto *fn;
switch (func_id) {
#ifdef CONFIG_NET
case BPF_FUNC_skb_output:
return &bpf_skb_output_proto;
case BPF_FUNC_xdp_output:
return &bpf_xdp_output_proto;
case BPF_FUNC_skc_to_tcp6_sock:
return &bpf_skc_to_tcp6_sock_proto;
case BPF_FUNC_skc_to_tcp_sock:
return &bpf_skc_to_tcp_sock_proto;
case BPF_FUNC_skc_to_tcp_timewait_sock:
return &bpf_skc_to_tcp_timewait_sock_proto;
case BPF_FUNC_skc_to_tcp_request_sock:
return &bpf_skc_to_tcp_request_sock_proto;
case BPF_FUNC_skc_to_udp6_sock:
return &bpf_skc_to_udp6_sock_proto;
case BPF_FUNC_skc_to_unix_sock:
return &bpf_skc_to_unix_sock_proto;
case BPF_FUNC_skc_to_mptcp_sock:
return &bpf_skc_to_mptcp_sock_proto;
case BPF_FUNC_sk_storage_get:
return &bpf_sk_storage_get_tracing_proto;
case BPF_FUNC_sk_storage_delete:
return &bpf_sk_storage_delete_tracing_proto;
case BPF_FUNC_sock_from_file:
return &bpf_sock_from_file_proto;
case BPF_FUNC_get_socket_cookie:
return &bpf_get_socket_ptr_cookie_proto;
case BPF_FUNC_xdp_get_buff_len:
return &bpf_xdp_get_buff_len_trace_proto;
#endif
case BPF_FUNC_seq_printf:
return prog->expected_attach_type == BPF_TRACE_ITER ?
&bpf_seq_printf_proto :
NULL;
case BPF_FUNC_seq_write:
return prog->expected_attach_type == BPF_TRACE_ITER ?
&bpf_seq_write_proto :
NULL;
case BPF_FUNC_seq_printf_btf:
return prog->expected_attach_type == BPF_TRACE_ITER ?
&bpf_seq_printf_btf_proto :
NULL;
case BPF_FUNC_d_path:
return &bpf_d_path_proto;
case BPF_FUNC_get_func_arg:
if (bpf_prog_has_trampoline(prog) ||
prog->expected_attach_type == BPF_TRACE_RAW_TP)
return &bpf_get_func_arg_proto;
return NULL;
case BPF_FUNC_get_func_ret:
return bpf_prog_has_trampoline(prog) ? &bpf_get_func_ret_proto : NULL;
case BPF_FUNC_get_func_arg_cnt:
if (bpf_prog_has_trampoline(prog) ||
prog->expected_attach_type == BPF_TRACE_RAW_TP)
return &bpf_get_func_arg_cnt_proto;
return NULL;
case BPF_FUNC_get_attach_cookie:
if (prog->type == BPF_PROG_TYPE_TRACING &&
prog->expected_attach_type == BPF_TRACE_RAW_TP)
return &bpf_get_attach_cookie_proto_tracing;
return bpf_prog_has_trampoline(prog) ? &bpf_get_attach_cookie_proto_tracing : NULL;
default:
fn = raw_tp_prog_func_proto(func_id, prog);
if (!fn && prog->expected_attach_type == BPF_TRACE_ITER)
fn = bpf_iter_get_func_proto(func_id, prog);
return fn;
}
}
static bool raw_tp_prog_is_valid_access(int off, int size,
enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
return bpf_tracing_ctx_access(off, size, type);
}
static bool tracing_prog_is_valid_access(int off, int size,
enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
}
int __weak bpf_prog_test_run_tracing(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
const struct bpf_verifier_ops raw_tracepoint_verifier_ops = {
.get_func_proto = raw_tp_prog_func_proto,
.is_valid_access = raw_tp_prog_is_valid_access,
};
const struct bpf_prog_ops raw_tracepoint_prog_ops = {
#ifdef CONFIG_NET
.test_run = bpf_prog_test_run_raw_tp,
#endif
};
const struct bpf_verifier_ops tracing_verifier_ops = {
.get_func_proto = tracing_prog_func_proto,
.is_valid_access = tracing_prog_is_valid_access,
};
const struct bpf_prog_ops tracing_prog_ops = {
.test_run = bpf_prog_test_run_tracing,
};
static bool raw_tp_writable_prog_is_valid_access(int off, int size,
enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
if (off == 0) {
if (size != sizeof(u64) || type != BPF_READ)
return false;
info->reg_type = PTR_TO_TP_BUFFER;
}
return raw_tp_prog_is_valid_access(off, size, type, prog, info);
}
const struct bpf_verifier_ops raw_tracepoint_writable_verifier_ops = {
.get_func_proto = raw_tp_prog_func_proto,
.is_valid_access = raw_tp_writable_prog_is_valid_access,
};
const struct bpf_prog_ops raw_tracepoint_writable_prog_ops = {
};
static bool pe_prog_is_valid_access(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
const int size_u64 = sizeof(u64);
if (off < 0 || off >= sizeof(struct bpf_perf_event_data))
return false;
if (type != BPF_READ)
return false;
if (off % size != 0) {
if (sizeof(unsigned long) != 4)
return false;
if (size != 8)
return false;
if (off % size != 4)
return false;
}
switch (off) {
case bpf_ctx_range(struct bpf_perf_event_data, sample_period):
bpf_ctx_record_field_size(info, size_u64);
if (!bpf_ctx_narrow_access_ok(off, size, size_u64))
return false;
break;
case bpf_ctx_range(struct bpf_perf_event_data, addr):
bpf_ctx_record_field_size(info, size_u64);
if (!bpf_ctx_narrow_access_ok(off, size, size_u64))
return false;
break;
default:
if (size != sizeof(long))
return false;
}
return true;
}
static u32 pe_prog_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog, u32 *target_size)
{
struct bpf_insn *insn = insn_buf;
switch (si->off) {
case offsetof(struct bpf_perf_event_data, sample_period):
*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_perf_event_data_kern,
data), si->dst_reg, si->src_reg,
offsetof(struct bpf_perf_event_data_kern, data));
*insn++ = BPF_LDX_MEM(BPF_DW, si->dst_reg, si->dst_reg,
bpf_target_off(struct perf_sample_data, period, 8,
target_size));
break;
case offsetof(struct bpf_perf_event_data, addr):
*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_perf_event_data_kern,
data), si->dst_reg, si->src_reg,
offsetof(struct bpf_perf_event_data_kern, data));
*insn++ = BPF_LDX_MEM(BPF_DW, si->dst_reg, si->dst_reg,
bpf_target_off(struct perf_sample_data, addr, 8,
target_size));
break;
default:
*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_perf_event_data_kern,
regs), si->dst_reg, si->src_reg,
offsetof(struct bpf_perf_event_data_kern, regs));
*insn++ = BPF_LDX_MEM(BPF_SIZEOF(long), si->dst_reg, si->dst_reg,
si->off);
break;
}
return insn - insn_buf;
}
const struct bpf_verifier_ops perf_event_verifier_ops = {
.get_func_proto = pe_prog_func_proto,
.is_valid_access = pe_prog_is_valid_access,
.convert_ctx_access = pe_prog_convert_ctx_access,
};
const struct bpf_prog_ops perf_event_prog_ops = {
};
static DEFINE_MUTEX(bpf_event_mutex);
#define BPF_TRACE_MAX_PROGS 64
int perf_event_attach_bpf_prog(struct perf_event *event,
struct bpf_prog *prog,
u64 bpf_cookie)
{
struct bpf_prog_array *old_array;
struct bpf_prog_array *new_array;
int ret = -EEXIST;
/*
* Kprobe override only works if they are on the function entry,
* and only if they are on the opt-in list.
*/
if (prog->kprobe_override &&
(!trace_kprobe_on_func_entry(event->tp_event) ||
!trace_kprobe_error_injectable(event->tp_event)))
return -EINVAL;
mutex_lock(&bpf_event_mutex);
if (event->prog)
goto unlock;
old_array = bpf_event_rcu_dereference(event->tp_event->prog_array);
if (old_array &&
bpf_prog_array_length(old_array) >= BPF_TRACE_MAX_PROGS) {
ret = -E2BIG;
goto unlock;
}
ret = bpf_prog_array_copy(old_array, NULL, prog, bpf_cookie, &new_array);
if (ret < 0)
goto unlock;
/* set the new array to event->tp_event and set event->prog */
event->prog = prog;
event->bpf_cookie = bpf_cookie;
rcu_assign_pointer(event->tp_event->prog_array, new_array);
bpf_prog_array_free_sleepable(old_array);
unlock:
mutex_unlock(&bpf_event_mutex);
return ret;
}
void perf_event_detach_bpf_prog(struct perf_event *event)
{
struct bpf_prog_array *old_array;
struct bpf_prog_array *new_array;
struct bpf_prog *prog = NULL;
int ret;
mutex_lock(&bpf_event_mutex);
if (!event->prog)
goto unlock;
old_array = bpf_event_rcu_dereference(event->tp_event->prog_array);
if (!old_array)
goto put;
ret = bpf_prog_array_copy(old_array, event->prog, NULL, 0, &new_array);
if (ret < 0) {
bpf_prog_array_delete_safe(old_array, event->prog);
} else {
rcu_assign_pointer(event->tp_event->prog_array, new_array);
bpf_prog_array_free_sleepable(old_array);
}
put:
prog = event->prog;
event->prog = NULL;
unlock:
mutex_unlock(&bpf_event_mutex);
if (prog) {
/*
* It could be that the bpf_prog is not sleepable (and will be freed
* via normal RCU), but is called from a point that supports sleepable
* programs and uses tasks-trace-RCU.
*/
synchronize_rcu_tasks_trace();
bpf_prog_put(prog);
}
}
int perf_event_query_prog_array(struct perf_event *event, void __user *info)
{
struct perf_event_query_bpf __user *uquery = info;
struct perf_event_query_bpf query = {};
struct bpf_prog_array *progs;
u32 *ids, prog_cnt, ids_len;
int ret;
if (!perfmon_capable())
return -EPERM;
if (event->attr.type != PERF_TYPE_TRACEPOINT)
return -EINVAL;
if (copy_from_user(&query, uquery, sizeof(query)))
return -EFAULT;
ids_len = query.ids_len;
if (ids_len > BPF_TRACE_MAX_PROGS)
return -E2BIG;
ids = kcalloc(ids_len, sizeof(u32), GFP_USER | __GFP_NOWARN);
if (!ids)
return -ENOMEM;
/*
* The above kcalloc returns ZERO_SIZE_PTR when ids_len = 0, which
* is required when user only wants to check for uquery->prog_cnt.
* There is no need to check for it since the case is handled
* gracefully in bpf_prog_array_copy_info.
*/
mutex_lock(&bpf_event_mutex);
progs = bpf_event_rcu_dereference(event->tp_event->prog_array);
ret = bpf_prog_array_copy_info(progs, ids, ids_len, &prog_cnt);
mutex_unlock(&bpf_event_mutex);
if (copy_to_user(&uquery->prog_cnt, &prog_cnt, sizeof(prog_cnt)) ||
copy_to_user(uquery->ids, ids, ids_len * sizeof(u32)))
ret = -EFAULT;
kfree(ids);
return ret;
}
extern struct bpf_raw_event_map __start__bpf_raw_tp[];
extern struct bpf_raw_event_map __stop__bpf_raw_tp[];
struct bpf_raw_event_map *bpf_get_raw_tracepoint(const char *name)
{
struct bpf_raw_event_map *btp = __start__bpf_raw_tp;
for (; btp < __stop__bpf_raw_tp; btp++) {
if (!strcmp(btp->tp->name, name))
return btp;
}
return bpf_get_raw_tracepoint_module(name);
}
void bpf_put_raw_tracepoint(struct bpf_raw_event_map *btp)
{
struct module *mod;
guard(rcu)();
mod = __module_address((unsigned long)btp);
module_put(mod);
}
static __always_inline
void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args)
{
struct bpf_prog *prog = link->link.prog;
struct bpf_run_ctx *old_run_ctx;
struct bpf_trace_run_ctx run_ctx;
rcu_read_lock_dont_migrate();
if (unlikely(!bpf_prog_get_recursion_context(prog))) {
bpf_prog_inc_misses_counter(prog);
goto out;
}
run_ctx.bpf_cookie = link->cookie;
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
(void) bpf_prog_run(prog, args);
bpf_reset_run_ctx(old_run_ctx);
out:
bpf_prog_put_recursion_context(prog);
rcu_read_unlock_migrate();
}
#define UNPACK(...) __VA_ARGS__
#define REPEAT_1(FN, DL, X, ...) FN(X)
#define REPEAT_2(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_1(FN, DL, __VA_ARGS__)
#define REPEAT_3(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_2(FN, DL, __VA_ARGS__)
#define REPEAT_4(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_3(FN, DL, __VA_ARGS__)
#define REPEAT_5(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_4(FN, DL, __VA_ARGS__)
#define REPEAT_6(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_5(FN, DL, __VA_ARGS__)
#define REPEAT_7(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_6(FN, DL, __VA_ARGS__)
#define REPEAT_8(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_7(FN, DL, __VA_ARGS__)
#define REPEAT_9(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_8(FN, DL, __VA_ARGS__)
#define REPEAT_10(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_9(FN, DL, __VA_ARGS__)
#define REPEAT_11(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_10(FN, DL, __VA_ARGS__)
#define REPEAT_12(FN, DL, X, ...) FN(X) UNPACK DL REPEAT_11(FN, DL, __VA_ARGS__)
#define REPEAT(X, FN, DL, ...) REPEAT_##X(FN, DL, __VA_ARGS__)
#define SARG(X) u64 arg##X
#define COPY(X) args[X] = arg##X
#define __DL_COM (,)
#define __DL_SEM (;)
#define __SEQ_0_11 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11
#define BPF_TRACE_DEFN_x(x) \
void bpf_trace_run##x(struct bpf_raw_tp_link *link, \
REPEAT(x, SARG, __DL_COM, __SEQ_0_11)) \
{ \
u64 args[x]; \
REPEAT(x, COPY, __DL_SEM, __SEQ_0_11); \
__bpf_trace_run(link, args); \
} \
EXPORT_SYMBOL_GPL(bpf_trace_run##x)
BPF_TRACE_DEFN_x(1);
BPF_TRACE_DEFN_x(2);
BPF_TRACE_DEFN_x(3);
BPF_TRACE_DEFN_x(4);
BPF_TRACE_DEFN_x(5);
BPF_TRACE_DEFN_x(6);
BPF_TRACE_DEFN_x(7);
BPF_TRACE_DEFN_x(8);
BPF_TRACE_DEFN_x(9);
BPF_TRACE_DEFN_x(10);
BPF_TRACE_DEFN_x(11);
BPF_TRACE_DEFN_x(12);
int bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_raw_tp_link *link)
{
struct tracepoint *tp = btp->tp;
struct bpf_prog *prog = link->link.prog;
/*
* check that program doesn't access arguments beyond what's
* available in this tracepoint
*/
if (prog->aux->max_ctx_offset > btp->num_args * sizeof(u64))
return -EINVAL;
if (prog->aux->max_tp_access > btp->writable_size)
return -EINVAL;
return tracepoint_probe_register_may_exist(tp, (void *)btp->bpf_func, link);
}
int bpf_probe_unregister(struct bpf_raw_event_map *btp, struct bpf_raw_tp_link *link)
{
return tracepoint_probe_unregister(btp->tp, (void *)btp->bpf_func, link);
}
int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id,
u32 *fd_type, const char **buf,
u64 *probe_offset, u64 *probe_addr,
unsigned long *missed)
{
bool is_tracepoint, is_syscall_tp;
struct bpf_prog *prog;
int flags, err = 0;
prog = event->prog;
if (!prog)
return -ENOENT;
/* not supporting BPF_PROG_TYPE_PERF_EVENT yet */
if (prog->type == BPF_PROG_TYPE_PERF_EVENT)
return -EOPNOTSUPP;
*prog_id = prog->aux->id;
flags = event->tp_event->flags;
is_tracepoint = flags & TRACE_EVENT_FL_TRACEPOINT;
is_syscall_tp = is_syscall_trace_event(event->tp_event);
if (is_tracepoint || is_syscall_tp) {
*buf = is_tracepoint ? event->tp_event->tp->name
: event->tp_event->name;
/* We allow NULL pointer for tracepoint */
if (fd_type)
*fd_type = BPF_FD_TYPE_TRACEPOINT;
if (probe_offset)
*probe_offset = 0x0;
if (probe_addr)
*probe_addr = 0x0;
} else {
/* kprobe/uprobe */
err = -EOPNOTSUPP;
#ifdef CONFIG_KPROBE_EVENTS
if (flags & TRACE_EVENT_FL_KPROBE)
err = bpf_get_kprobe_info(event, fd_type, buf,
probe_offset, probe_addr, missed,
event->attr.type == PERF_TYPE_TRACEPOINT);
#endif
#ifdef CONFIG_UPROBE_EVENTS
if (flags & TRACE_EVENT_FL_UPROBE)
err = bpf_get_uprobe_info(event, fd_type, buf,
probe_offset, probe_addr,
event->attr.type == PERF_TYPE_TRACEPOINT);
#endif
}
return err;
}
static int __init send_signal_irq_work_init(void)
{
int cpu;
struct send_signal_irq_work *work;
for_each_possible_cpu(cpu) {
work = per_cpu_ptr(&send_signal_work, cpu);
init_irq_work(&work->irq_work, do_bpf_send_signal);
}
return 0;
}
subsys_initcall(send_signal_irq_work_init);
#ifdef CONFIG_MODULES
static int bpf_event_notify(struct notifier_block *nb, unsigned long op,
void *module)
{
struct bpf_trace_module *btm, *tmp;
struct module *mod = module;
int ret = 0;
if (mod->num_bpf_raw_events == 0 ||
(op != MODULE_STATE_COMING && op != MODULE_STATE_GOING))
goto out;
mutex_lock(&bpf_module_mutex);
switch (op) {
case MODULE_STATE_COMING:
btm = kzalloc_obj(*btm);
if (btm) {
btm->module = module;
list_add(&btm->list, &bpf_trace_modules);
} else {
ret = -ENOMEM;
}
break;
case MODULE_STATE_GOING:
list_for_each_entry_safe(btm, tmp, &bpf_trace_modules, list) {
if (btm->module == module) {
list_del(&btm->list);
kfree(btm);
break;
}
}
break;
}
mutex_unlock(&bpf_module_mutex);
out:
return notifier_from_errno(ret);
}
static struct notifier_block bpf_module_nb = {
.notifier_call = bpf_event_notify,
};
static int __init bpf_event_init(void)
{
register_module_notifier(&bpf_module_nb);
return 0;
}
fs_initcall(bpf_event_init);
#endif /* CONFIG_MODULES */
struct bpf_session_run_ctx {
struct bpf_run_ctx run_ctx;
bool is_return;
void *data;
};
#ifdef CONFIG_FPROBE
struct bpf_kprobe_multi_link {
struct bpf_link link;
struct fprobe fp;
unsigned long *addrs;
u64 *cookies;
u32 cnt;
u32 mods_cnt;
struct module **mods;
};
struct bpf_kprobe_multi_run_ctx {
struct bpf_session_run_ctx session_ctx;
struct bpf_kprobe_multi_link *link;
unsigned long entry_ip;
};
struct user_syms {
const char **syms;
char *buf;
};
#ifndef CONFIG_HAVE_FTRACE_REGS_HAVING_PT_REGS
static DEFINE_PER_CPU(struct pt_regs, bpf_kprobe_multi_pt_regs);
#define bpf_kprobe_multi_pt_regs_ptr() this_cpu_ptr(&bpf_kprobe_multi_pt_regs)
#else
#define bpf_kprobe_multi_pt_regs_ptr() (NULL)
#endif
static unsigned long ftrace_get_entry_ip(unsigned long fentry_ip)
{
unsigned long ip = ftrace_get_symaddr(fentry_ip);
return ip ? : fentry_ip;
}
static int copy_user_syms(struct user_syms *us, unsigned long __user *usyms, u32 cnt)
{
unsigned long __user usymbol;
const char **syms = NULL;
char *buf = NULL, *p;
int err = -ENOMEM;
unsigned int i;
syms = kvmalloc_array(cnt, sizeof(*syms), GFP_KERNEL);
if (!syms)
goto error;
buf = kvmalloc_array(cnt, KSYM_NAME_LEN, GFP_KERNEL);
if (!buf)
goto error;
for (p = buf, i = 0; i < cnt; i++) {
if (__get_user(usymbol, usyms + i)) {
err = -EFAULT;
goto error;
}
err = strncpy_from_user(p, (const char __user *) usymbol, KSYM_NAME_LEN);
if (err == KSYM_NAME_LEN)
err = -E2BIG;
if (err < 0)
goto error;
syms[i] = p;
p += err + 1;
}
us->syms = syms;
us->buf = buf;
return 0;
error:
if (err) {
kvfree(syms);
kvfree(buf);
}
return err;
}
static void kprobe_multi_put_modules(struct module **mods, u32 cnt)
{
u32 i;
for (i = 0; i < cnt; i++)
module_put(mods[i]);
}
static void free_user_syms(struct user_syms *us)
{
kvfree(us->syms);
kvfree(us->buf);
}
static void bpf_kprobe_multi_link_release(struct bpf_link *link)
{
struct bpf_kprobe_multi_link *kmulti_link;
kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link);
/* Don't wait for RCU GP here. */
unregister_fprobe_async(&kmulti_link->fp);
kprobe_multi_put_modules(kmulti_link->mods, kmulti_link->mods_cnt);
}
static void bpf_kprobe_multi_link_dealloc(struct bpf_link *link)
{
struct bpf_kprobe_multi_link *kmulti_link;
kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link);
kvfree(kmulti_link->addrs);
kvfree(kmulti_link->cookies);
kfree(kmulti_link->mods);
kfree(kmulti_link);
}
static int bpf_kprobe_multi_link_fill_link_info(const struct bpf_link *link,
struct bpf_link_info *info)
{
u64 __user *ucookies = u64_to_user_ptr(info->kprobe_multi.cookies);
u64 __user *uaddrs = u64_to_user_ptr(info->kprobe_multi.addrs);
struct bpf_kprobe_multi_link *kmulti_link;
u32 ucount = info->kprobe_multi.count;
int err = 0, i;
if (!uaddrs ^ !ucount)
return -EINVAL;
if (ucookies && !ucount)
return -EINVAL;
kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link);
info->kprobe_multi.count = kmulti_link->cnt;
info->kprobe_multi.flags = kmulti_link->link.flags;
info->kprobe_multi.missed = kmulti_link->fp.nmissed;
if (!uaddrs)
return 0;
if (ucount < kmulti_link->cnt)
err = -ENOSPC;
else
ucount = kmulti_link->cnt;
if (ucookies) {
if (kmulti_link->cookies) {
if (copy_to_user(ucookies, kmulti_link->cookies, ucount * sizeof(u64)))
return -EFAULT;
} else {
for (i = 0; i < ucount; i++) {
if (put_user(0, ucookies + i))
return -EFAULT;
}
}
}
if (kallsyms_show_value(current_cred())) {
if (copy_to_user(uaddrs, kmulti_link->addrs, ucount * sizeof(u64)))
return -EFAULT;
} else {
for (i = 0; i < ucount; i++) {
if (put_user(0, uaddrs + i))
return -EFAULT;
}
}
return err;
}
#ifdef CONFIG_PROC_FS
static void bpf_kprobe_multi_show_fdinfo(const struct bpf_link *link,
struct seq_file *seq)
{
struct bpf_kprobe_multi_link *kmulti_link;
bool has_cookies;
kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link);
has_cookies = !!kmulti_link->cookies;
seq_printf(seq,
"kprobe_cnt:\t%u\n"
"missed:\t%lu\n",
kmulti_link->cnt,
kmulti_link->fp.nmissed);
seq_printf(seq, "%s\t %s\n", "cookie", "func");
for (int i = 0; i < kmulti_link->cnt; i++) {
seq_printf(seq,
"%llu\t %pS\n",
has_cookies ? kmulti_link->cookies[i] : 0,
(void *)kmulti_link->addrs[i]);
}
}
#endif
static const struct bpf_link_ops bpf_kprobe_multi_link_lops = {
.release = bpf_kprobe_multi_link_release,
.dealloc_deferred = bpf_kprobe_multi_link_dealloc,
.fill_link_info = bpf_kprobe_multi_link_fill_link_info,
#ifdef CONFIG_PROC_FS
.show_fdinfo = bpf_kprobe_multi_show_fdinfo,
#endif
};
static void bpf_kprobe_multi_cookie_swap(void *a, void *b, int size, const void *priv)
{
const struct bpf_kprobe_multi_link *link = priv;
unsigned long *addr_a = a, *addr_b = b;
u64 *cookie_a, *cookie_b;
cookie_a = link->cookies + (addr_a - link->addrs);
cookie_b = link->cookies + (addr_b - link->addrs);
/* swap addr_a/addr_b and cookie_a/cookie_b values */
swap(*addr_a, *addr_b);
swap(*cookie_a, *cookie_b);
}
static int bpf_kprobe_multi_addrs_cmp(const void *a, const void *b)
{
const unsigned long *addr_a = a, *addr_b = b;
if (*addr_a == *addr_b)
return 0;
return *addr_a < *addr_b ? -1 : 1;
}
static int bpf_kprobe_multi_cookie_cmp(const void *a, const void *b, const void *priv)
{
return bpf_kprobe_multi_addrs_cmp(a, b);
}
static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx)
{
struct bpf_kprobe_multi_run_ctx *run_ctx;
struct bpf_kprobe_multi_link *link;
u64 *cookie, entry_ip;
unsigned long *addr;
if (WARN_ON_ONCE(!ctx))
return 0;
run_ctx = container_of(current->bpf_ctx, struct bpf_kprobe_multi_run_ctx,
session_ctx.run_ctx);
link = run_ctx->link;
if (!link->cookies)
return 0;
entry_ip = run_ctx->entry_ip;
addr = bsearch(&entry_ip, link->addrs, link->cnt, sizeof(entry_ip),
bpf_kprobe_multi_addrs_cmp);
if (!addr)
return 0;
cookie = link->cookies + (addr - link->addrs);
return *cookie;
}
static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
{
struct bpf_kprobe_multi_run_ctx *run_ctx;
run_ctx = container_of(current->bpf_ctx, struct bpf_kprobe_multi_run_ctx,
session_ctx.run_ctx);
return run_ctx->entry_ip;
}
static __always_inline int
kprobe_multi_link_prog_run(struct bpf_kprobe_multi_link *link,
unsigned long entry_ip, struct ftrace_regs *fregs,
bool is_return, void *data)
{
struct bpf_kprobe_multi_run_ctx run_ctx = {
.session_ctx = {
.is_return = is_return,
.data = data,
},
.link = link,
.entry_ip = entry_ip,
};
struct bpf_run_ctx *old_run_ctx;
struct pt_regs *regs;
int err;
/*
* graph tracer framework ensures we won't migrate, so there is no need
* to use migrate_disable for bpf_prog_run again. The check here just for
* __this_cpu_inc_return.
*/
cant_sleep();
if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1)) {
bpf_prog_inc_misses_counter(link->link.prog);
err = 1;
goto out;
}
rcu_read_lock();
regs = ftrace_partial_regs(fregs, bpf_kprobe_multi_pt_regs_ptr());
old_run_ctx = bpf_set_run_ctx(&run_ctx.session_ctx.run_ctx);
err = bpf_prog_run(link->link.prog, regs);
bpf_reset_run_ctx(old_run_ctx);
ftrace_partial_regs_update(fregs, bpf_kprobe_multi_pt_regs_ptr());
rcu_read_unlock();
out:
__this_cpu_dec(bpf_prog_active);
return err;
}
static int
kprobe_multi_link_handler(struct fprobe *fp, unsigned long fentry_ip,
unsigned long ret_ip, struct ftrace_regs *fregs,
void *data)
{
struct bpf_kprobe_multi_link *link;
int err;
link = container_of(fp, struct bpf_kprobe_multi_link, fp);
err = kprobe_multi_link_prog_run(link, ftrace_get_entry_ip(fentry_ip),
fregs, false, data);
return is_kprobe_session(link->link.prog) ? err : 0;
}
static void
kprobe_multi_link_exit_handler(struct fprobe *fp, unsigned long fentry_ip,
unsigned long ret_ip, struct ftrace_regs *fregs,
void *data)
{
struct bpf_kprobe_multi_link *link;
link = container_of(fp, struct bpf_kprobe_multi_link, fp);
kprobe_multi_link_prog_run(link, ftrace_get_entry_ip(fentry_ip),
fregs, true, data);
}
static int symbols_cmp_r(const void *a, const void *b, const void *priv)
{
const char **str_a = (const char **) a;
const char **str_b = (const char **) b;
return strcmp(*str_a, *str_b);
}
struct multi_symbols_sort {
const char **funcs;
u64 *cookies;
};
static void symbols_swap_r(void *a, void *b, int size, const void *priv)
{
const struct multi_symbols_sort *data = priv;
const char **name_a = a, **name_b = b;
swap(*name_a, *name_b);
/* If defined, swap also related cookies. */
if (data->cookies) {
u64 *cookie_a, *cookie_b;
cookie_a = data->cookies + (name_a - data->funcs);
cookie_b = data->cookies + (name_b - data->funcs);
swap(*cookie_a, *cookie_b);
}
}
struct modules_array {
struct module **mods;
int mods_cnt;
int mods_cap;
};
static int add_module(struct modules_array *arr, struct module *mod)
{
struct module **mods;
if (arr->mods_cnt == arr->mods_cap) {
arr->mods_cap = max(16, arr->mods_cap * 3 / 2);
mods = krealloc_array(arr->mods, arr->mods_cap, sizeof(*mods), GFP_KERNEL);
if (!mods)
return -ENOMEM;
arr->mods = mods;
}
arr->mods[arr->mods_cnt] = mod;
arr->mods_cnt++;
return 0;
}
static bool has_module(struct modules_array *arr, struct module *mod)
{
int i;
for (i = arr->mods_cnt - 1; i >= 0; i--) {
if (arr->mods[i] == mod)
return true;
}
return false;
}
static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u32 addrs_cnt)
{
struct modules_array arr = {};
u32 i, err = 0;
for (i = 0; i < addrs_cnt; i++) {
bool skip_add = false;
struct module *mod;
scoped_guard(rcu) {
mod = __module_address(addrs[i]);
/* Either no module or it's already stored */
if (!mod || has_module(&arr, mod)) {
skip_add = true;
break; /* scoped_guard */
}
if (!try_module_get(mod))
err = -EINVAL;
}
if (skip_add)
continue;
if (err)
break;
err = add_module(&arr, mod);
if (err) {
module_put(mod);
break;
}
}
/* We return either err < 0 in case of error, ... */
if (err) {
kprobe_multi_put_modules(arr.mods, arr.mods_cnt);
kfree(arr.mods);
return err;
}
/* or number of modules found if everything is ok. */
*mods = arr.mods;
return arr.mods_cnt;
}
static int addrs_check_error_injection_list(unsigned long *addrs, u32 cnt)
{
u32 i;
for (i = 0; i < cnt; i++) {
if (!within_error_injection_list(addrs[i]))
return -EINVAL;
}
return 0;
}
int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
struct bpf_kprobe_multi_link *link = NULL;
struct bpf_link_primer link_primer;
void __user *ucookies;
unsigned long *addrs;
u32 flags, cnt, size;
void __user *uaddrs;
u64 *cookies = NULL;
void __user *usyms;
int err;
/* no support for 32bit archs yet */
if (sizeof(u64) != sizeof(void *))
return -EOPNOTSUPP;
if (attr->link_create.flags)
return -EINVAL;
if (!is_kprobe_multi(prog))
return -EINVAL;
/* kprobe_multi is not allowed to be sleepable. */
if (prog->sleepable)
return -EINVAL;
/* Writing to context is not allowed for kprobes. */
if (prog->aux->kprobe_write_ctx)
return -EINVAL;
flags = attr->link_create.kprobe_multi.flags;
if (flags & ~BPF_F_KPROBE_MULTI_RETURN)
return -EINVAL;
uaddrs = u64_to_user_ptr(attr->link_create.kprobe_multi.addrs);
usyms = u64_to_user_ptr(attr->link_create.kprobe_multi.syms);
if (!!uaddrs == !!usyms)
return -EINVAL;
cnt = attr->link_create.kprobe_multi.cnt;
if (!cnt)
return -EINVAL;
if (cnt > MAX_KPROBE_MULTI_CNT)
return -E2BIG;
size = cnt * sizeof(*addrs);
addrs = kvmalloc_array(cnt, sizeof(*addrs), GFP_KERNEL);
if (!addrs)
return -ENOMEM;
ucookies = u64_to_user_ptr(attr->link_create.kprobe_multi.cookies);
if (ucookies) {
cookies = kvmalloc_array(cnt, sizeof(*addrs), GFP_KERNEL);
if (!cookies) {
err = -ENOMEM;
goto error;
}
if (copy_from_user(cookies, ucookies, size)) {
err = -EFAULT;
goto error;
}
}
if (uaddrs) {
if (copy_from_user(addrs, uaddrs, size)) {
err = -EFAULT;
goto error;
}
} else {
struct multi_symbols_sort data = {
.cookies = cookies,
};
struct user_syms us;
err = copy_user_syms(&us, usyms, cnt);
if (err)
goto error;
if (cookies)
data.funcs = us.syms;
sort_r(us.syms, cnt, sizeof(*us.syms), symbols_cmp_r,
symbols_swap_r, &data);
err = ftrace_lookup_symbols(us.syms, cnt, addrs);
free_user_syms(&us);
if (err)
goto error;
}
if (prog->kprobe_override && addrs_check_error_injection_list(addrs, cnt)) {
err = -EINVAL;
goto error;
}
link = kzalloc_obj(*link);
if (!link) {
err = -ENOMEM;
goto error;
}
bpf_link_init(&link->link, BPF_LINK_TYPE_KPROBE_MULTI,
&bpf_kprobe_multi_link_lops, prog, attr->link_create.attach_type);
err = bpf_link_prime(&link->link, &link_primer);
if (err)
goto error;
if (!(flags & BPF_F_KPROBE_MULTI_RETURN))
link->fp.entry_handler = kprobe_multi_link_handler;
if ((flags & BPF_F_KPROBE_MULTI_RETURN) || is_kprobe_session(prog))
link->fp.exit_handler = kprobe_multi_link_exit_handler;
if (is_kprobe_session(prog))
link->fp.entry_data_size = sizeof(u64);
link->addrs = addrs;
link->cookies = cookies;
link->cnt = cnt;
link->link.flags = flags;
if (cookies) {
/*
* Sorting addresses will trigger sorting cookies as well
* (check bpf_kprobe_multi_cookie_swap). This way we can
* find cookie based on the address in bpf_get_attach_cookie
* helper.
*/
sort_r(addrs, cnt, sizeof(*addrs),
bpf_kprobe_multi_cookie_cmp,
bpf_kprobe_multi_cookie_swap,
link);
}
err = get_modules_for_addrs(&link->mods, addrs, cnt);
if (err < 0) {
bpf_link_cleanup(&link_primer);
return err;
}
link->mods_cnt = err;
err = register_fprobe_ips(&link->fp, addrs, cnt);
if (err) {
kprobe_multi_put_modules(link->mods, link->mods_cnt);
bpf_link_cleanup(&link_primer);
return err;
}
return bpf_link_settle(&link_primer);
error:
kfree(link);
kvfree(addrs);
kvfree(cookies);
return err;
}
#else /* !CONFIG_FPROBE */
int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
return -EOPNOTSUPP;
}
static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx)
{
return 0;
}
static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
{
return 0;
}
#endif
#ifdef CONFIG_UPROBES
struct bpf_uprobe_multi_link;
struct bpf_uprobe {
struct bpf_uprobe_multi_link *link;
loff_t offset;
unsigned long ref_ctr_offset;
u64 cookie;
struct uprobe *uprobe;
struct uprobe_consumer consumer;
bool session;
};
struct bpf_uprobe_multi_link {
struct path path;
struct bpf_link link;
u32 cnt;
struct bpf_uprobe *uprobes;
struct task_struct *task;
};
struct bpf_uprobe_multi_run_ctx {
struct bpf_session_run_ctx session_ctx;
unsigned long entry_ip;
struct bpf_uprobe *uprobe;
};
static void bpf_uprobe_unregister(struct bpf_uprobe *uprobes, u32 cnt)
{
u32 i;
for (i = 0; i < cnt; i++)
uprobe_unregister_nosync(uprobes[i].uprobe, &uprobes[i].consumer);
if (cnt)
uprobe_unregister_sync();
}
static void bpf_uprobe_multi_link_release(struct bpf_link *link)
{
struct bpf_uprobe_multi_link *umulti_link;
umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
bpf_uprobe_unregister(umulti_link->uprobes, umulti_link->cnt);
if (umulti_link->task)
put_task_struct(umulti_link->task);
path_put(&umulti_link->path);
}
static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
{
struct bpf_uprobe_multi_link *umulti_link;
umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
kvfree(umulti_link->uprobes);
kfree(umulti_link);
}
static int bpf_uprobe_multi_link_fill_link_info(const struct bpf_link *link,
struct bpf_link_info *info)
{
u64 __user *uref_ctr_offsets = u64_to_user_ptr(info->uprobe_multi.ref_ctr_offsets);
u64 __user *ucookies = u64_to_user_ptr(info->uprobe_multi.cookies);
u64 __user *uoffsets = u64_to_user_ptr(info->uprobe_multi.offsets);
u64 __user *upath = u64_to_user_ptr(info->uprobe_multi.path);
u32 upath_size = info->uprobe_multi.path_size;
struct bpf_uprobe_multi_link *umulti_link;
u32 ucount = info->uprobe_multi.count;
int err = 0, i;
char *p, *buf;
long left = 0;
if (!upath ^ !upath_size)
return -EINVAL;
if ((uoffsets || uref_ctr_offsets || ucookies) && !ucount)
return -EINVAL;
umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
info->uprobe_multi.count = umulti_link->cnt;
info->uprobe_multi.flags = umulti_link->link.flags;
info->uprobe_multi.pid = umulti_link->task ?
task_pid_nr_ns(umulti_link->task, task_active_pid_ns(current)) : 0;
upath_size = upath_size ? min_t(u32, upath_size, PATH_MAX) : PATH_MAX;
buf = kmalloc(upath_size, GFP_KERNEL);
if (!buf)
return -ENOMEM;
p = d_path(&umulti_link->path, buf, upath_size);
if (IS_ERR(p)) {
kfree(buf);
return PTR_ERR(p);
}
upath_size = buf + upath_size - p;
if (upath)
left = copy_to_user(upath, p, upath_size);
kfree(buf);
if (left)
return -EFAULT;
info->uprobe_multi.path_size = upath_size;
if (!uoffsets && !ucookies && !uref_ctr_offsets)
return 0;
if (ucount < umulti_link->cnt)
err = -ENOSPC;
else
ucount = umulti_link->cnt;
for (i = 0; i < ucount; i++) {
if (uoffsets &&
put_user(umulti_link->uprobes[i].offset, uoffsets + i))
return -EFAULT;
if (uref_ctr_offsets &&
put_user(umulti_link->uprobes[i].ref_ctr_offset, uref_ctr_offsets + i))
return -EFAULT;
if (ucookies &&
put_user(umulti_link->uprobes[i].cookie, ucookies + i))
return -EFAULT;
}
return err;
}
#ifdef CONFIG_PROC_FS
static void bpf_uprobe_multi_show_fdinfo(const struct bpf_link *link,
struct seq_file *seq)
{
struct bpf_uprobe_multi_link *umulti_link;
char *p, *buf;
pid_t pid;
umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
buf = kmalloc(PATH_MAX, GFP_KERNEL);
if (!buf)
return;
p = d_path(&umulti_link->path, buf, PATH_MAX);
if (IS_ERR(p)) {
kfree(buf);
return;
}
pid = umulti_link->task ?
task_pid_nr_ns(umulti_link->task, task_active_pid_ns(current)) : 0;
seq_printf(seq,
"uprobe_cnt:\t%u\n"
"pid:\t%u\n"
"path:\t%s\n",
umulti_link->cnt, pid, p);
seq_printf(seq, "%s\t %s\t %s\n", "cookie", "offset", "ref_ctr_offset");
for (int i = 0; i < umulti_link->cnt; i++) {
seq_printf(seq,
"%llu\t %#llx\t %#lx\n",
umulti_link->uprobes[i].cookie,
umulti_link->uprobes[i].offset,
umulti_link->uprobes[i].ref_ctr_offset);
}
kfree(buf);
}
#endif
static const struct bpf_link_ops bpf_uprobe_multi_link_lops = {
.release = bpf_uprobe_multi_link_release,
.dealloc_deferred = bpf_uprobe_multi_link_dealloc,
.fill_link_info = bpf_uprobe_multi_link_fill_link_info,
#ifdef CONFIG_PROC_FS
.show_fdinfo = bpf_uprobe_multi_show_fdinfo,
#endif
};
static int uprobe_prog_run(struct bpf_uprobe *uprobe,
unsigned long entry_ip,
struct pt_regs *regs,
bool is_return, void *data)
{
struct bpf_uprobe_multi_link *link = uprobe->link;
struct bpf_uprobe_multi_run_ctx run_ctx = {
.session_ctx = {
.is_return = is_return,
.data = data,
},
.entry_ip = entry_ip,
.uprobe = uprobe,
};
struct bpf_prog *prog = link->link.prog;
bool sleepable = prog->sleepable;
struct bpf_run_ctx *old_run_ctx;
int err;
if (link->task && !same_thread_group(current, link->task))
return 0;
if (sleepable)
rcu_read_lock_trace();
else
rcu_read_lock();
migrate_disable();
old_run_ctx = bpf_set_run_ctx(&run_ctx.session_ctx.run_ctx);
err = bpf_prog_run(link->link.prog, regs);
bpf_reset_run_ctx(old_run_ctx);
migrate_enable();
if (sleepable)
rcu_read_unlock_trace();
else
rcu_read_unlock();
return err;
}
static bool
uprobe_multi_link_filter(struct uprobe_consumer *con, struct mm_struct *mm)
{
struct bpf_uprobe *uprobe;
uprobe = container_of(con, struct bpf_uprobe, consumer);
return uprobe->link->task->mm == mm;
}
static int
uprobe_multi_link_handler(struct uprobe_consumer *con, struct pt_regs *regs,
__u64 *data)
{
struct bpf_uprobe *uprobe;
int ret;
uprobe = container_of(con, struct bpf_uprobe, consumer);
ret = uprobe_prog_run(uprobe, instruction_pointer(regs), regs, false, data);
if (uprobe->session)
return ret ? UPROBE_HANDLER_IGNORE : 0;
return 0;
}
static int
uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, struct pt_regs *regs,
__u64 *data)
{
struct bpf_uprobe *uprobe;
uprobe = container_of(con, struct bpf_uprobe, consumer);
uprobe_prog_run(uprobe, func, regs, true, data);
return 0;
}
static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
{
struct bpf_uprobe_multi_run_ctx *run_ctx;
run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx,
session_ctx.run_ctx);
return run_ctx->entry_ip;
}
static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
{
struct bpf_uprobe_multi_run_ctx *run_ctx;
run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx,
session_ctx.run_ctx);
return run_ctx->uprobe->cookie;
}
int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
struct bpf_uprobe_multi_link *link = NULL;
unsigned long __user *uref_ctr_offsets;
struct bpf_link_primer link_primer;
struct bpf_uprobe *uprobes = NULL;
struct task_struct *task = NULL;
unsigned long __user *uoffsets;
u64 __user *ucookies;
void __user *upath;
u32 flags, cnt, i;
struct path path;
char *name;
pid_t pid;
int err;
/* no support for 32bit archs yet */
if (sizeof(u64) != sizeof(void *))
return -EOPNOTSUPP;
if (attr->link_create.flags)
return -EINVAL;
if (!is_uprobe_multi(prog))
return -EINVAL;
flags = attr->link_create.uprobe_multi.flags;
if (flags & ~BPF_F_UPROBE_MULTI_RETURN)
return -EINVAL;
/*
* path, offsets and cnt are mandatory,
* ref_ctr_offsets and cookies are optional
*/
upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path);
uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets);
cnt = attr->link_create.uprobe_multi.cnt;
pid = attr->link_create.uprobe_multi.pid;
if (!upath || !uoffsets || !cnt || pid < 0)
return -EINVAL;
if (cnt > MAX_UPROBE_MULTI_CNT)
return -E2BIG;
uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets);
ucookies = u64_to_user_ptr(attr->link_create.uprobe_multi.cookies);
name = strndup_user(upath, PATH_MAX);
if (IS_ERR(name)) {
err = PTR_ERR(name);
return err;
}
err = kern_path(name, LOOKUP_FOLLOW, &path);
kfree(name);
if (err)
return err;
if (!d_is_reg(path.dentry)) {
err = -EBADF;
goto error_path_put;
}
if (pid) {
rcu_read_lock();
task = get_pid_task(find_vpid(pid), PIDTYPE_TGID);
rcu_read_unlock();
if (!task) {
err = -ESRCH;
goto error_path_put;
}
}
err = -ENOMEM;
link = kzalloc_obj(*link);
uprobes = kvzalloc_objs(*uprobes, cnt);
if (!uprobes || !link)
goto error_free;
for (i = 0; i < cnt; i++) {
if (__get_user(uprobes[i].offset, uoffsets + i)) {
err = -EFAULT;
goto error_free;
}
if (uprobes[i].offset < 0) {
err = -EINVAL;
goto error_free;
}
if (uref_ctr_offsets && __get_user(uprobes[i].ref_ctr_offset, uref_ctr_offsets + i)) {
err = -EFAULT;
goto error_free;
}
if (ucookies && __get_user(uprobes[i].cookie, ucookies + i)) {
err = -EFAULT;
goto error_free;
}
uprobes[i].link = link;
if (!(flags & BPF_F_UPROBE_MULTI_RETURN))
uprobes[i].consumer.handler = uprobe_multi_link_handler;
if (flags & BPF_F_UPROBE_MULTI_RETURN || is_uprobe_session(prog))
uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
if (is_uprobe_session(prog))
uprobes[i].session = true;
if (pid)
uprobes[i].consumer.filter = uprobe_multi_link_filter;
}
link->cnt = cnt;
link->uprobes = uprobes;
link->path = path;
link->task = task;
link->link.flags = flags;
bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
&bpf_uprobe_multi_link_lops, prog, attr->link_create.attach_type);
for (i = 0; i < cnt; i++) {
uprobes[i].uprobe = uprobe_register(d_real_inode(link->path.dentry),
uprobes[i].offset,
uprobes[i].ref_ctr_offset,
&uprobes[i].consumer);
if (IS_ERR(uprobes[i].uprobe)) {
err = PTR_ERR(uprobes[i].uprobe);
link->cnt = i;
goto error_unregister;
}
}
err = bpf_link_prime(&link->link, &link_primer);
if (err)
goto error_unregister;
return bpf_link_settle(&link_primer);
error_unregister:
bpf_uprobe_unregister(uprobes, link->cnt);
error_free:
kvfree(uprobes);
kfree(link);
if (task)
put_task_struct(task);
error_path_put:
path_put(&path);
return err;
}
#else /* !CONFIG_UPROBES */
int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
return -EOPNOTSUPP;
}
static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
{
return 0;
}
static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
{
return 0;
}
#endif /* CONFIG_UPROBES */
__bpf_kfunc_start_defs();
__bpf_kfunc bool bpf_session_is_return(void *ctx)
{
struct bpf_session_run_ctx *session_ctx;
session_ctx = container_of(current->bpf_ctx, struct bpf_session_run_ctx, run_ctx);
return session_ctx->is_return;
}
__bpf_kfunc __u64 *bpf_session_cookie(void *ctx)
{
struct bpf_session_run_ctx *session_ctx;
session_ctx = container_of(current->bpf_ctx, struct bpf_session_run_ctx, run_ctx);
return session_ctx->data;
}
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(session_kfunc_set_ids)
BTF_ID_FLAGS(func, bpf_session_is_return)
BTF_ID_FLAGS(func, bpf_session_cookie)
BTF_KFUNCS_END(session_kfunc_set_ids)
static int bpf_session_filter(const struct bpf_prog *prog, u32 kfunc_id)
{
if (!btf_id_set8_contains(&session_kfunc_set_ids, kfunc_id))
return 0;
if (!is_kprobe_session(prog) && !is_uprobe_session(prog) && !is_trace_fsession(prog))
return -EACCES;
return 0;
}
static const struct btf_kfunc_id_set bpf_session_kfunc_set = {
.owner = THIS_MODULE,
.set = &session_kfunc_set_ids,
.filter = bpf_session_filter,
};
static int __init bpf_trace_kfuncs_init(void)
{
int err = 0;
err = err ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_KPROBE, &bpf_session_kfunc_set);
err = err ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_session_kfunc_set);
return err;
}
late_initcall(bpf_trace_kfuncs_init);
typedef int (*copy_fn_t)(void *dst, const void *src, u32 size, struct task_struct *tsk);
/*
* The __always_inline is to make sure the compiler doesn't
* generate indirect calls into callbacks, which is expensive,
* on some kernel configurations. This allows compiler to put
* direct calls into all the specific callback implementations
* (copy_user_data_sleepable, copy_user_data_nofault, and so on)
*/
static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u64 doff, u64 size,
const void *unsafe_src,
copy_fn_t str_copy_fn,
struct task_struct *tsk)
{
struct bpf_dynptr_kern *dst;
u64 chunk_sz, off;
void *dst_slice;
int cnt, err;
char buf[256];
dst_slice = bpf_dynptr_slice_rdwr(dptr, doff, NULL, size);
if (likely(dst_slice))
return str_copy_fn(dst_slice, unsafe_src, size, tsk);
dst = (struct bpf_dynptr_kern *)dptr;
if (bpf_dynptr_check_off_len(dst, doff, size))
return -E2BIG;
for (off = 0; off < size; off += chunk_sz - 1) {
chunk_sz = min_t(u64, sizeof(buf), size - off);
/* Expect str_copy_fn to return count of copied bytes, including
* zero terminator. Next iteration increment off by chunk_sz - 1 to
* overwrite NUL.
*/
cnt = str_copy_fn(buf, unsafe_src + off, chunk_sz, tsk);
if (cnt < 0)
return cnt;
err = __bpf_dynptr_write(dst, doff + off, buf, cnt, 0);
if (err)
return err;
if (cnt < chunk_sz || chunk_sz == 1) /* we are done */
return off + cnt;
}
return off;
}
static __always_inline int __bpf_dynptr_copy(const struct bpf_dynptr *dptr, u64 doff,
u64 size, const void *unsafe_src,
copy_fn_t copy_fn, struct task_struct *tsk)
{
struct bpf_dynptr_kern *dst;
void *dst_slice;
char buf[256];
u64 off, chunk_sz;
int err;
dst_slice = bpf_dynptr_slice_rdwr(dptr, doff, NULL, size);
if (likely(dst_slice))
return copy_fn(dst_slice, unsafe_src, size, tsk);
dst = (struct bpf_dynptr_kern *)dptr;
if (bpf_dynptr_check_off_len(dst, doff, size))
return -E2BIG;
for (off = 0; off < size; off += chunk_sz) {
chunk_sz = min_t(u64, sizeof(buf), size - off);
err = copy_fn(buf, unsafe_src + off, chunk_sz, tsk);
if (err)
return err;
err = __bpf_dynptr_write(dst, doff + off, buf, chunk_sz, 0);
if (err)
return err;
}
return 0;
}
static __always_inline int copy_user_data_nofault(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
return copy_from_user_nofault(dst, (const void __user *)unsafe_src, size);
}
static __always_inline int copy_user_data_sleepable(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
int ret;
if (!tsk) { /* Read from the current task */
ret = copy_from_user(dst, (const void __user *)unsafe_src, size);
if (ret)
return -EFAULT;
return 0;
}
ret = access_process_vm(tsk, (unsigned long)unsafe_src, dst, size, 0);
if (ret != size)
return -EFAULT;
return 0;
}
static __always_inline int copy_kernel_data_nofault(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
return copy_from_kernel_nofault(dst, unsafe_src, size);
}
static __always_inline int copy_user_str_nofault(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
return strncpy_from_user_nofault(dst, (const void __user *)unsafe_src, size);
}
static __always_inline int copy_user_str_sleepable(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
int ret;
if (unlikely(size == 0))
return 0;
if (tsk) {
ret = copy_remote_vm_str(tsk, (unsigned long)unsafe_src, dst, size, 0);
} else {
ret = strncpy_from_user(dst, (const void __user *)unsafe_src, size - 1);
/* strncpy_from_user does not guarantee NUL termination */
if (ret >= 0)
((char *)dst)[ret] = '\0';
}
if (ret < 0)
return ret;
return ret + 1;
}
static __always_inline int copy_kernel_str_nofault(void *dst, const void *unsafe_src,
u32 size, struct task_struct *tsk)
{
return strncpy_from_kernel_nofault(dst, unsafe_src, size);
}
__bpf_kfunc_start_defs();
__bpf_kfunc int bpf_send_signal_task(struct task_struct *task, int sig, enum pid_type type,
u64 value)
{
if (type != PIDTYPE_PID && type != PIDTYPE_TGID)
return -EINVAL;
return bpf_send_signal_common(sig, type, task, value);
}
__bpf_kfunc int bpf_probe_read_user_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_data_nofault, NULL);
}
__bpf_kfunc int bpf_probe_read_kernel_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void *unsafe_ptr__ign)
{
return __bpf_dynptr_copy(dptr, off, size, unsafe_ptr__ign,
copy_kernel_data_nofault, NULL);
}
__bpf_kfunc int bpf_probe_read_user_str_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_str_nofault, NULL);
}
__bpf_kfunc int bpf_probe_read_kernel_str_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void *unsafe_ptr__ign)
{
return __bpf_dynptr_copy_str(dptr, off, size, unsafe_ptr__ign,
copy_kernel_str_nofault, NULL);
}
__bpf_kfunc int bpf_copy_from_user_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_data_sleepable, NULL);
}
__bpf_kfunc int bpf_copy_from_user_str_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign)
{
return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_str_sleepable, NULL);
}
__bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign,
struct task_struct *tsk)
{
return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_data_sleepable, tsk);
}
__bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 off,
u64 size, const void __user *unsafe_ptr__ign,
struct task_struct *tsk)
{
return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign,
copy_user_str_sleepable, tsk);
}
__bpf_kfunc_end_defs();
]
|
| 456/6 |
2026/05/18 19:26 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 200238
output: 42
thoughts: 1738
Thoughts: **Analyzing `snprintf` Usage in `kernel/bpf/helpers.c`**
Okay, let's break this down. My initial scan of `kernel/trace/bpf_trace.c` revealed that `bpf_trace_printk` and `bpf_trace_vprintk` use `bpf_bprintf_prepare`. These calls themselves appear sound, as they appropriately handle the result of `bpf_bprintf_prepare`. I also re-examined `bpf_snprintf_btf`, but the call to `btf_type_snprintf_show` returns a length, consistent with `snprintf`'s typical behavior.
Now, focusing on the user's explicit question regarding `kernel/bpf/helpers.c`, I've re-examined that file meticulously, specifically checking for `snprintf` calls. Initially, I found two, and I confirmed that the primary area of concern is indeed within `bpf_bprintf_prepare`.
The core issue seems to revolve around the handling of `bpf_trace_copy_string`. It uses functions like `strncpy_from_kernel_nofault` and `strncpy_from_user_nofault` that return the length of the string copied, or a negative error value in case of an error. The crucial point here is that if truncation happens, these functions return the buffer size (`bufsz`). In the subsequent line, `tmp_buf` is incremented by `err`, which works as expected.
I still have a lingering concern about the lack of error checking on those `snprintf` calls.
I meticulously re-checked `kernel/bpf/helpers.c` for any missed `snprintf` calls. I re-ran `grep` with greater precision, focusing on the file and even re-checked for `vsnprintf`. I'm confident in my search. My re-checking of `bpf_bprintf_prepare` confirmed the existing `snprintf` use, specifically the `ip_spec[] = "%pXX"` case. I re-checked for any direct `snprintf` calls to make sure, there are only the same two `snprintf` that I have already found.
My conclusion is that I should emphasize that the problem exists within `kernel/bpf/helpers.c`, specifically in `bpf_bprintf_prepare` where `snprintf` calls that have no return value checks are happening, but I can't find others that may have been missed.
|
| 457/6 |
2026/05/18 19:26 |
tool |
codesearch-find-references |
0m
Args: map[IncludeSnippetLines:10 Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924 SourceSnippet: 914: fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
915: fmt[i + 1] == 'S') {
916: if (tmp_buf)
917: cur_arg = raw_args[num_spec];
918: i++;
919: goto nocopy_fmt;
920: }
921:
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
930:
931: i++;
932: num_spec++;
933: continue;
934: }
] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965 SourceSnippet: 955: sizeof_cur_ip);
956: if (err < 0)
957: memset(cur_ip, 0, sizeof_cur_ip);
958:
959: /* hack: bstr_printf expects IP addresses to be
960: * pre-formatted as strings, ironically, the easiest way
961: * to do that is to call snprintf.
962: */
963: ip_spec[2] = fmt[i - 1];
964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
969: num_spec++;
970:
971: continue;
972: } else if (fmt[i] == 's') {
973: fmt_ptype = fmt[i];
974: fmt_str:
975: if (fmt[i + 1] != 0 &&
]] TruncatedOutput:false]
|
| 458/6 |
2026/05/18 19:26 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 203297
output: 43
thoughts: 234
Thoughts: **My Assessment of `snprintf` Usage in `kernel/bpf/helpers.c`**
Okay, so I've been digging into `kernel/bpf/helpers.c` to address the potential `snprintf` vulnerabilities, as requested. I’ve zeroed in on the two instances where the return value of `snprintf` isn't rigorously checked, which could indeed lead to buffer overflows if truncation occurs.
Hold on, I should probably expand the scope a bit. While the immediate concern is `snprintf`, it’s good practice to make sure there aren't any other string formatting functions, like `sprintf` or similar, that might introduce similar risks in this context. Although the focus was `snprintf` this is still a valid consideration.
But wait a minute, the user’s query explicitly focused on `kernel/bpf/helpers.c`, so before I stray too far, I should double-check that I haven’t missed anything within *that* specific file. I know I initially reviewed it, but it's worth a second look just to be absolutely certain.
Actually, I already did that, and I know I didn't miss anything.
Ah, I see `bpf_snprintf` helper. It in turn calls `bstr_printf`. Perhaps it's worth checking to see if there is another helper that uses `snprintf`? I'll check all the `BPF_CALL` functions in `kernel/bpf/helpers.c` to see if there are any other instances.
|
| 459/6 |
2026/05/18 19:26 |
tool |
codesearch-find-references |
0m
Error:
Args: map[IncludeSnippetLines:1 Name:BPF_CALL SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 460/6 |
2026/05/18 19:26 |
llm |
codeexpert-compressor |
0m
Model:
Reply: The investigation into `kernel/bpf/helpers.c` regarding the use of `snprintf` without proper return value checking has reached a definitive conclusion. Below is a summary of the current state of the research.
### 1. Approaches and Results
* **Code Search for `snprintf` and variants:**
* Performed a targeted search for `snprintf`, `sprintf`, `vsnprintf`, `scnprintf`, and `vscnprintf` within `kernel/bpf/helpers.c`.
* **Result:** Only two calls to `snprintf` exist in this file, both located within the function `bpf_bprintf_prepare`. No instances of `sprintf` or other variants were found in this specific file.
* **Deep Dive into `bpf_bprintf_prepare`:**
* Analyzed the logic of `bpf_bprintf_prepare` (lines 804–1062) to see how the `snprintf` return value is utilized.
* **Result:** In both identified locations, the return value `err` is used directly for pointer arithmetic (`tmp_buf += err + 1`) without checking if `err` exceeds the remaining buffer space.
* **Investigation of Related Helpers:**
* Examined `bpf_snprintf` and `bpf_snprintf_btf`.
* **Result:** `bpf_snprintf` uses `bstr_printf`, not `snprintf`. `bpf_snprintf_btf` (found in `kernel/trace/bpf_trace.c`) uses `btf_type_snprintf_show` (found in `kernel/bpf/btf.c`), which internally manages a `btf_show` structure.
* **Cross-file Verification:**
* Checked `kernel/trace/bpf_trace.c` for similar patterns in `bpf_trace_printk` and `bpf_trace_vprintk`.
* **Result:** These functions use `bstr_printf` into a fixed-size buffer (`MAX_BPRINTF_BUF`) and do not exhibit the same pointer arithmetic vulnerability seen in `helpers.c`.
### 2. Current Hypotheses and Findings
The investigation confirms that there are exactly **two** places in `kernel/bpf/helpers.c` where `snprintf` is used improperly. The core issue is that `snprintf` returns the number of characters that *would* have been written, but the code assumes it returns the number of characters *actually* written.
**The Vulnerable Logic:**
In `bpf_bprintf_prepare`, the code iterates through a format string. When it encounters certain specifiers, it uses `snprintf` to format data into `tmp_buf`.
* **Location 1 (Line 924):** Formatting `%pB` (symbolic kernel pointers).
* **Location 2 (Line 965):** Formatting IP addresses (`%pI4`, `%pI6`, etc.).
In both cases, the code performs:
```c
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf, ...);
tmp_buf += err + 1;
```
If `snprintf` truncates the output because the buffer is full, `err` will be greater than or equal to the size provided (`tmp_buf_end - tmp_buf`). Consequently, `tmp_buf` is advanced beyond `tmp_buf_end`, leading to an out-of-bounds pointer that may be used for subsequent writes in the next loop iteration.
### 3. Critical File Paths and Code Snippets
* **File:** `kernel/bpf/helpers.c`
* **Function:** `bpf_bprintf_prepare`
* **Snippet 1 (Line 924):**
```c
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
```
* **Snippet 2 (Line 965):**
```c
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
```
* **Buffer Definition:** The buffer `tmp_buf` points to `buffers->bin_args`, which has a fixed size defined by `MAX_BPRINTF_BIN_ARGS` (512 bytes) in `include/linux/bpf.h`.
### 4. Observations on Reasoning Loops
The investigation was linear and avoided repetitive tool calls. Initial searches for `snprintf_btf` failed because the helper is named `bpf_snprintf_btf` and its implementation is split across `kernel/trace/bpf_trace.c` and `kernel/bpf/btf.c`. Once the file index for `kernel/bpf/` was reviewed, the location of all relevant string-handling logic was clarified. No further tool calls are required to answer the user's request.
Tokens:
input: 189798
output: 1153
thoughts: 1186
|
| 461/6 |
2026/05/18 19:27 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 3453
output: 23
thoughts: 280
Thoughts: **Verifying the String Formatting Vulnerability Analysis**
Okay, so the user is right to push me on this. The previous summary stated, and I quote, "Only two calls to `snprintf` exist in `kernel/bpf/helpers.c`, both in `bpf_bprintf_prepare`." It also mentioned those two calls have the truncation issue, and no `sprintf`, `vsnprintf`, etc. were found.
But I need to be *absolutely* certain. My initial scan might have missed something subtle, or perhaps the code has been updated since that analysis. I need to be rigorous.
My next step is to re-run that search for `snprintf` within `kernel/bpf/helpers.c`. I'll be thorough this time. Then, I need to expand my search. I also need to ensure that no *other* formatting functions slipped under the radar. I'll explicitly look for `scnprintf` and any other similar formatting functions to make sure there are no other points of potential vulnerability within that file. This is about being certain.
The summary specifically states that no other variants were found, but as any expert knows, you have to verify and re-verify! I'm going to run that search again, just to be sure.
|
| 462/6 |
2026/05/18 19:27 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:[arena.c arraymap.c backtrack.c bloom_filter.c bpf_cgrp_storage.c bpf_inode_storage.c bpf_insn_array.c bpf_iter.c bpf_local_storage.c bpf_lru_list.c bpf_lru_list.h bpf_lsm.c bpf_lsm_proto.c bpf_struct_ops.c bpf_task_storage.c btf.c btf_iter.c btf_relocate.c cfg.c cgroup.c cgroup_iter.c check_btf.c const_fold.c core.c cpumap.c cpumask.c crypto.c devmap.c disasm.c disasm.h dispatcher.c dmabuf_iter.c fixups.c hashtab.c helpers.c inode.c kmem_cache_iter.c link_iter.c liveness.c local_storage.c log.c lpm_trie.c map_in_map.c map_in_map.h map_iter.c memalloc.c mmap_unlock_work.h mprog.c net_namespace.c offload.c percpu_freelist.c percpu_freelist.h prog_iter.c queue_stack_maps.c range_tree.c range_tree.h relo_core.c reuseport_array.c ringbuf.c rqspinlock.c rqspinlock.h stackmap.c states.c stream.c syscall.c sysfs_btf.c task_iter.c tcx.c tnum.c token.c trampoline.c verifier.c] Subdirs:[preload]]
|
| 463/6 |
2026/05/18 19:27 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 4639
output: 23
thoughts: 20
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience, presented in the first person:
**Kernel BPF Helper Dive**
Alright, let's get down to it. My initial thought is to dive straight into the heart of the matter. If I'm thinking about how `snprintf` might be handled within the BPF context, my first instinct is to zero in on `kernel/bpf/helpers.c`. That's the most logical place to start digging for any potential helper functions or implementations related to formatted string output within the BPF environment. It's the standard location for core BPF helper definitions, and `snprintf`, while potentially implemented elsewhere in the kernel, would likely have its BPF interface or relevant wrapper defined here if it exists. From there, I'll be able to quickly determine its availability, limitations (if any), and how it plays with other BPF features. This search will be the foundation to better understand the kernel functionality.
|
| 464/6 |
2026/05/18 19:27 |
tool |
read-file |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*/
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/bpf-cgroup.h>
#include <linux/cgroup.h>
#include <linux/rcupdate.h>
#include <linux/random.h>
#include <linux/smp.h>
#include <linux/topology.h>
#include <linux/ktime.h>
#include <linux/sched.h>
#include <linux/uidgid.h>
#include <linux/filter.h>
#include <linux/ctype.h>
#include <linux/jiffies.h>
#include <linux/pid_namespace.h>
#include <linux/poison.h>
#include <linux/proc_ns.h>
#include <linux/sched/task.h>
#include <linux/security.h>
#include <linux/btf_ids.h>
#include <linux/bpf_mem_alloc.h>
#include <linux/kasan.h>
#include <linux/bpf_verifier.h>
#include <linux/uaccess.h>
#include <linux/verification.h>
#include <linux/task_work.h>
#include <linux/irq_work.h>
#include <linux/buildid.h>
#include "../../lib/kstrtox.h"
/* If kernel subsystem is allowing eBPF programs to call this function,
* inside its own verifier_ops->get_func_proto() callback it should return
* bpf_map_lookup_elem_proto, so that verifier can properly check the arguments
*
* Different map implementations will rely on rcu in map methods
* lookup/update/delete, therefore eBPF programs must run under rcu lock
* if program is allowed to access maps, so check rcu_read_lock_held() or
* rcu_read_lock_trace_held() in all three functions.
*/
BPF_CALL_2(bpf_map_lookup_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_elem(map, key);
}
const struct bpf_func_proto bpf_map_lookup_elem_proto = {
.func = bpf_map_lookup_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_4(bpf_map_update_elem, struct bpf_map *, map, void *, key,
void *, value, u64, flags)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_update_elem(map, key, value, flags);
}
const struct bpf_func_proto bpf_map_update_elem_proto = {
.func = bpf_map_update_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_PTR_TO_MAP_VALUE,
.arg4_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_delete_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_delete_elem(map, key);
}
const struct bpf_func_proto bpf_map_delete_elem_proto = {
.func = bpf_map_delete_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_3(bpf_map_push_elem, struct bpf_map *, map, void *, value, u64, flags)
{
return map->ops->map_push_elem(map, value, flags);
}
const struct bpf_func_proto bpf_map_push_elem_proto = {
.func = bpf_map_push_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_pop_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_pop_elem(map, value);
}
const struct bpf_func_proto bpf_map_pop_elem_proto = {
.func = bpf_map_pop_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_2(bpf_map_peek_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_peek_elem(map, value);
}
const struct bpf_func_proto bpf_map_peek_elem_proto = {
.func = bpf_map_peek_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_3(bpf_map_lookup_percpu_elem, struct bpf_map *, map, void *, key, u32, cpu)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_percpu_elem(map, key, cpu);
}
const struct bpf_func_proto bpf_map_lookup_percpu_elem_proto = {
.func = bpf_map_lookup_percpu_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_ANYTHING,
};
const struct bpf_func_proto bpf_get_prandom_u32_proto = {
.func = bpf_user_rnd_u32,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_smp_processor_id)
{
return smp_processor_id();
}
const struct bpf_func_proto bpf_get_smp_processor_id_proto = {
.func = bpf_get_smp_processor_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.allow_fastcall = true,
};
BPF_CALL_0(bpf_get_numa_node_id)
{
return numa_node_id();
}
const struct bpf_func_proto bpf_get_numa_node_id_proto = {
.func = bpf_get_numa_node_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_ns)
{
/* NMI safe access to clock monotonic */
return ktime_get_mono_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_ns_proto = {
.func = bpf_ktime_get_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_boot_ns)
{
/* NMI safe access to clock boottime */
return ktime_get_boot_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_boot_ns_proto = {
.func = bpf_ktime_get_boot_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_coarse_ns)
{
return ktime_get_coarse_ns();
}
const struct bpf_func_proto bpf_ktime_get_coarse_ns_proto = {
.func = bpf_ktime_get_coarse_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_tai_ns)
{
/* NMI safe access to clock tai */
return ktime_get_tai_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_tai_ns_proto = {
.func = bpf_ktime_get_tai_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_pid_tgid)
{
struct task_struct *task = current;
if (unlikely(!task))
return -EINVAL;
return (u64) task->tgid << 32 | task->pid;
}
const struct bpf_func_proto bpf_get_current_pid_tgid_proto = {
.func = bpf_get_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_uid_gid)
{
struct task_struct *task = current;
kuid_t uid;
kgid_t gid;
if (unlikely(!task))
return -EINVAL;
current_uid_gid(&uid, &gid);
return (u64) from_kgid(&init_user_ns, gid) << 32 |
from_kuid(&init_user_ns, uid);
}
const struct bpf_func_proto bpf_get_current_uid_gid_proto = {
.func = bpf_get_current_uid_gid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_2(bpf_get_current_comm, char *, buf, u32, size)
{
struct task_struct *task = current;
if (unlikely(!task))
goto err_clear;
/* Verifier guarantees that size > 0 */
strscpy_pad(buf, task->comm, size);
return 0;
err_clear:
memset(buf, 0, size);
return -EINVAL;
}
const struct bpf_func_proto bpf_get_current_comm_proto = {
.func = bpf_get_current_comm,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE,
};
#if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK)
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
union {
__u32 val;
arch_spinlock_t lock;
} u = { .lock = __ARCH_SPIN_LOCK_UNLOCKED };
compiletime_assert(u.val == 0, "__ARCH_SPIN_LOCK_UNLOCKED not 0");
BUILD_BUG_ON(sizeof(*l) != sizeof(__u32));
BUILD_BUG_ON(sizeof(*lock) != sizeof(__u32));
preempt_disable();
arch_spin_lock(l);
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
arch_spin_unlock(l);
preempt_enable();
}
#else
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
BUILD_BUG_ON(sizeof(*l) != sizeof(*lock));
do {
atomic_cond_read_relaxed(l, !VAL);
} while (atomic_xchg(l, 1));
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
atomic_set_release(l, 0);
}
#endif
static DEFINE_PER_CPU(unsigned long, irqsave_flags);
static inline void __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
{
unsigned long flags;
local_irq_save(flags);
__bpf_spin_lock(lock);
__this_cpu_write(irqsave_flags, flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
{
__bpf_spin_lock_irqsave(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_lock_proto = {
.func = bpf_spin_lock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
{
unsigned long flags;
flags = __this_cpu_read(irqsave_flags);
__bpf_spin_unlock(lock);
local_irq_restore(flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
{
__bpf_spin_unlock_irqrestore(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_unlock_proto = {
.func = bpf_spin_unlock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
bool lock_src)
{
struct bpf_spin_lock *lock;
if (lock_src)
lock = src + map->record->spin_lock_off;
else
lock = dst + map->record->spin_lock_off;
preempt_disable();
__bpf_spin_lock_irqsave(lock);
copy_map_value(map, dst, src);
__bpf_spin_unlock_irqrestore(lock);
preempt_enable();
}
BPF_CALL_0(bpf_jiffies64)
{
return get_jiffies_64();
}
const struct bpf_func_proto bpf_jiffies64_proto = {
.func = bpf_jiffies64,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
#ifdef CONFIG_CGROUPS
BPF_CALL_0(bpf_get_current_cgroup_id)
{
struct cgroup *cgrp;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
cgrp_id = cgroup_id(cgrp);
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_cgroup_id_proto = {
.func = bpf_get_current_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_1(bpf_get_current_ancestor_cgroup_id, int, ancestor_level)
{
struct cgroup *cgrp;
struct cgroup *ancestor;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
ancestor = cgroup_ancestor(cgrp, ancestor_level);
cgrp_id = ancestor ? cgroup_id(ancestor) : 0;
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto = {
.func = bpf_get_current_ancestor_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
#endif /* CONFIG_CGROUPS */
#define BPF_STRTOX_BASE_MASK 0x1F
static int __bpf_strtoull(const char *buf, size_t buf_len, u64 flags,
unsigned long long *res, bool *is_negative)
{
unsigned int base = flags & BPF_STRTOX_BASE_MASK;
const char *cur_buf = buf;
size_t cur_len = buf_len;
unsigned int consumed;
size_t val_len;
char str[64];
if (!buf || !buf_len || !res || !is_negative)
return -EINVAL;
if (base != 0 && base != 8 && base != 10 && base != 16)
return -EINVAL;
if (flags & ~BPF_STRTOX_BASE_MASK)
return -EINVAL;
while (cur_buf < buf + buf_len && isspace(*cur_buf))
++cur_buf;
*is_negative = (cur_buf < buf + buf_len && *cur_buf == '-');
if (*is_negative)
++cur_buf;
consumed = cur_buf - buf;
cur_len -= consumed;
if (!cur_len)
return -EINVAL;
cur_len = min(cur_len, sizeof(str) - 1);
memcpy(str, cur_buf, cur_len);
str[cur_len] = '\0';
cur_buf = str;
cur_buf = _parse_integer_fixup_radix(cur_buf, &base);
val_len = _parse_integer(cur_buf, base, res);
if (val_len & KSTRTOX_OVERFLOW)
return -ERANGE;
if (val_len == 0)
return -EINVAL;
cur_buf += val_len;
consumed += cur_buf - str;
return consumed;
}
static int __bpf_strtoll(const char *buf, size_t buf_len, u64 flags,
long long *res)
{
unsigned long long _res;
bool is_negative;
int err;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative) {
if ((long long)-_res > 0)
return -ERANGE;
*res = -_res;
} else {
if ((long long)_res < 0)
return -ERANGE;
*res = _res;
}
return err;
}
BPF_CALL_4(bpf_strtol, const char *, buf, size_t, buf_len, u64, flags,
s64 *, res)
{
long long _res;
int err;
*res = 0;
err = __bpf_strtoll(buf, buf_len, flags, &_res);
if (err < 0)
return err;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtol_proto = {
.func = bpf_strtol,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(s64),
};
BPF_CALL_4(bpf_strtoul, const char *, buf, size_t, buf_len, u64, flags,
u64 *, res)
{
unsigned long long _res;
bool is_negative;
int err;
*res = 0;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative)
return -EINVAL;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtoul_proto = {
.func = bpf_strtoul,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(u64),
};
BPF_CALL_3(bpf_strncmp, const char *, s1, u32, s1_sz, const char *, s2)
{
return strncmp(s1, s2, s1_sz);
}
static const struct bpf_func_proto bpf_strncmp_proto = {
.func = bpf_strncmp,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_CONST_STR,
};
BPF_CALL_4(bpf_get_ns_current_pid_tgid, u64, dev, u64, ino,
struct bpf_pidns_info *, nsdata, u32, size)
{
struct task_struct *task = current;
struct pid_namespace *pidns;
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_pidns_info)))
goto clear;
if (unlikely((u64)(dev_t)dev != dev))
goto clear;
if (unlikely(!task))
goto clear;
pidns = task_active_pid_ns(task);
if (unlikely(!pidns)) {
err = -ENOENT;
goto clear;
}
if (!ns_match(&pidns->ns, (dev_t)dev, ino))
goto clear;
nsdata->pid = task_pid_nr_ns(task, pidns);
nsdata->tgid = task_tgid_nr_ns(task, pidns);
return 0;
clear:
memset((void *)nsdata, 0, (size_t) size);
return err;
}
const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = {
.func = bpf_get_ns_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
};
static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
.func = bpf_get_raw_cpu_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
}
const struct bpf_func_proto bpf_event_output_data_proto = {
.func = bpf_event_output_data,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
const void __user *, user_ptr)
{
int ret = copy_from_user(dst, user_ptr, size);
if (unlikely(ret)) {
memset(dst, 0, size);
ret = -EFAULT;
}
return ret;
}
const struct bpf_func_proto bpf_copy_from_user_proto = {
.func = bpf_copy_from_user,
.gpl_only = false,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_5(bpf_copy_from_user_task, void *, dst, u32, size,
const void __user *, user_ptr, struct task_struct *, tsk, u64, flags)
{
int ret;
/* flags is not used yet */
if (unlikely(flags))
return -EINVAL;
if (unlikely(!size))
return 0;
ret = access_process_vm(tsk, (unsigned long)user_ptr, dst, size, 0);
if (ret == size)
return 0;
memset(dst, 0, size);
/* Return -EFAULT for partial read */
return ret < 0 ? ret : -EFAULT;
}
const struct bpf_func_proto bpf_copy_from_user_task_proto = {
.func = bpf_copy_from_user_task,
.gpl_only = true,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_BTF_ID,
.arg4_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.arg5_type = ARG_ANYTHING
};
BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
{
if (cpu >= nr_cpu_ids)
return (unsigned long)NULL;
return (unsigned long)per_cpu_ptr((const void __percpu *)(const uintptr_t)ptr, cpu);
}
const struct bpf_func_proto bpf_per_cpu_ptr_proto = {
.func = bpf_per_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | PTR_MAYBE_NULL | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
.arg2_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_this_cpu_ptr, const void *, percpu_ptr)
{
return (unsigned long)this_cpu_ptr((const void __percpu *)(const uintptr_t)percpu_ptr);
}
const struct bpf_func_proto bpf_this_cpu_ptr_proto = {
.func = bpf_this_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
};
static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,
size_t bufsz)
{
void __user *user_ptr = (__force void __user *)unsafe_ptr;
buf[0] = 0;
switch (fmt_ptype) {
case 's':
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
if ((unsigned long)unsafe_ptr < TASK_SIZE)
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
fallthrough;
#endif
case 'k':
return strncpy_from_kernel_nofault(buf, unsafe_ptr, bufsz);
case 'u':
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
}
return -EINVAL;
}
/* Support executing three nested bprintf helper calls on a given CPU */
#define MAX_BPRINTF_NEST_LEVEL 3
static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs);
static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
{
int nest_level;
preempt_disable();
nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
return -EBUSY;
}
*bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
return 0;
}
void bpf_put_buffers(void)
{
if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
return;
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
}
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
{
if (!data->bin_args && !data->buf)
return;
bpf_put_buffers();
}
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)
{
if (map->map_type == BPF_MAP_TYPE_ARRAY) {
struct bpf_array *array = container_of(map, struct bpf_array, map);
*arr_idx = ((char *)value - array->value) / array->elem_size;
return arr_idx;
}
return (void *)value - round_up(map->key_size, 8);
}
enum bpf_async_type {
BPF_ASYNC_TYPE_TIMER = 0,
BPF_ASYNC_TYPE_WQ,
};
enum bpf_async_op {
BPF_ASYNC_START,
BPF_ASYNC_CANCEL
};
struct bpf_async_cmd {
struct llist_node node;
u64 nsec;
u32 mode;
enum bpf_async_op op;
};
struct bpf_async_cb {
struct bpf_map *map;
struct bpf_prog *prog;
void __rcu *callback_fn;
void *value;
struct rcu_head rcu;
u64 flags;
struct irq_work worker;
refcount_t refcnt;
enum bpf_async_type type;
struct llist_head async_cmds;
};
/* BPF map elements can contain 'struct bpf_timer'.
* Such map owns all of its BPF timers.
* 'struct bpf_timer' is allocated as part of map element allocation
* and it's zero initialized.
* That space is used to keep 'struct bpf_async_kern'.
* bpf_timer_init() allocates 'struct bpf_hrtimer', inits hrtimer, and
* remembers 'struct bpf_map *' pointer it's part of.
* bpf_timer_set_callback() increments prog refcnt and assign bpf callback_fn.
* bpf_timer_start() arms the timer.
* If user space reference to a map goes to zero at this point
* ops->map_release_uref callback is responsible for cancelling the timers,
* freeing their memory, and decrementing prog's refcnts.
* bpf_timer_cancel() cancels the timer and decrements prog's refcnt.
* Inner maps can contain bpf timers as well. ops->map_release_uref is
* freeing the timers when inner map is replaced or deleted by user space.
*/
struct bpf_hrtimer {
struct bpf_async_cb cb;
struct hrtimer timer;
atomic_t cancelling;
};
struct bpf_work {
struct bpf_async_cb cb;
struct work_struct work;
};
/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */
struct bpf_async_kern {
union {
struct bpf_async_cb *cb;
struct bpf_hrtimer *timer;
struct bpf_work *work;
};
} __attribute__((aligned(8)));
static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);
static void bpf_async_refcount_put(struct bpf_async_cb *cb);
static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer)
{
struct bpf_hrtimer *t = container_of(hrtimer, struct bpf_hrtimer, timer);
struct bpf_map *map = t->cb.map;
void *value = t->cb.value;
bpf_callback_t callback_fn;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_timer);
callback_fn = rcu_dereference_check(t->cb.callback_fn, rcu_read_lock_bh_held());
if (!callback_fn)
goto out;
/* bpf_timer_cb() runs in hrtimer_run_softirq. It doesn't migrate and
* cannot be preempted by another bpf_timer_cb() on the same cpu.
* Remember the timer this callback is servicing to prevent
* deadlock if callback_fn() calls bpf_timer_cancel() or
* bpf_map_delete_elem() on the same timer.
*/
this_cpu_write(hrtimer_running, t);
key = map_key_from_value(map, value, &idx);
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
/* The verifier checked that return value is zero. */
this_cpu_write(hrtimer_running, NULL);
out:
return HRTIMER_NORESTART;
}
static void bpf_wq_work(struct work_struct *work)
{
struct bpf_work *w = container_of(work, struct bpf_work, work);
struct bpf_async_cb *cb = &w->cb;
struct bpf_map *map = cb->map;
bpf_callback_t callback_fn;
void *value = cb->value;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_wq);
callback_fn = READ_ONCE(cb->callback_fn);
if (!callback_fn)
return;
key = map_key_from_value(map, value, &idx);
rcu_read_lock_trace();
migrate_disable();
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
migrate_enable();
rcu_read_unlock_trace();
}
static void bpf_async_cb_rcu_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
/*
* Drop the last reference to prog only after RCU GP, as set_callback()
* may race with cancel_and_free()
*/
if (cb->prog)
bpf_prog_put(cb->prog);
kfree_nolock(cb);
}
/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */
static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
bool retry = false;
/*
* bpf_async_cancel_and_free() tried to cancel timer/wq, but it
* could have raced with timer/wq_start. Now refcnt is zero and
* srcu/rcu GP completed. Cancel timer/wq again.
*/
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER:
if (hrtimer_try_to_cancel(&t->timer) < 0)
retry = true;
break;
case BPF_ASYNC_TYPE_WQ:
if (!cancel_work(&w->work) && work_busy(&w->work))
retry = true;
break;
}
if (retry) {
/*
* hrtimer or wq callback may still be running. It must be
* in rcu_tasks_trace or rcu CS, so wait for GP again.
* It won't retry forever, since refcnt zero prevents all
* operations on timer/wq.
*/
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
return;
}
/* RCU Tasks Trace grace period implies RCU grace period. */
bpf_async_cb_rcu_free(rcu);
}
static void worker_for_call_rcu(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
static void bpf_async_refcount_put(struct bpf_async_cb *cb)
{
if (!refcount_dec_and_test(&cb->refcnt))
return;
if (irqs_disabled()) {
cb->worker = IRQ_WORK_INIT(worker_for_call_rcu);
irq_work_queue(&cb->worker);
} else {
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async);
static void bpf_async_irq_worker(struct irq_work *work);
static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags,
enum bpf_async_type type)
{
struct bpf_async_cb *cb, *old_cb;
struct bpf_hrtimer *t;
struct bpf_work *w;
clockid_t clockid;
size_t size;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
size = sizeof(struct bpf_hrtimer);
break;
case BPF_ASYNC_TYPE_WQ:
size = sizeof(struct bpf_work);
break;
default:
return -EINVAL;
}
old_cb = READ_ONCE(async->cb);
if (old_cb)
return -EBUSY;
cb = bpf_map_kmalloc_nolock(map, size, 0, map->numa_node);
if (!cb)
return -ENOMEM;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
clockid = flags & (MAX_CLOCKS - 1);
t = (struct bpf_hrtimer *)cb;
atomic_set(&t->cancelling, 0);
hrtimer_setup(&t->timer, bpf_timer_cb, clockid, HRTIMER_MODE_REL_SOFT);
cb->value = (void *)async - map->record->timer_off;
break;
case BPF_ASYNC_TYPE_WQ:
w = (struct bpf_work *)cb;
INIT_WORK(&w->work, bpf_wq_work);
cb->value = (void *)async - map->record->wq_off;
break;
}
cb->map = map;
cb->prog = NULL;
cb->flags = flags;
cb->worker = IRQ_WORK_INIT(bpf_async_irq_worker);
init_llist_head(&cb->async_cmds);
refcount_set(&cb->refcnt, 1); /* map's reference */
cb->type = type;
rcu_assign_pointer(cb->callback_fn, NULL);
old_cb = cmpxchg(&async->cb, NULL, cb);
if (old_cb) {
/* Lost the race to initialize this bpf_async_kern, drop the allocated object */
kfree_nolock(cb);
return -EBUSY;
}
/* Guarantee the order between async->cb and map->usercnt. So
* when there are concurrent uref release and bpf timer init, either
* bpf_timer_cancel_and_free() called by uref release reads a no-NULL
* timer or atomic64_read() below returns a zero usercnt.
*/
smp_mb();
if (!atomic64_read(&map->usercnt)) {
/* maps with timers must be either held by user space
* or pinned in bpffs.
*/
bpf_async_cancel_and_free(async);
return -EPERM;
}
return 0;
}
BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
u64, flags)
{
clock_t clockid = flags & (MAX_CLOCKS - 1);
BUILD_BUG_ON(MAX_CLOCKS != 16);
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer));
if (flags >= MAX_CLOCKS ||
/* similar to timerfd except _ALARM variants are not supported */
(clockid != CLOCK_MONOTONIC &&
clockid != CLOCK_REALTIME &&
clockid != CLOCK_BOOTTIME))
return -EINVAL;
return __bpf_async_init(timer, map, flags, BPF_ASYNC_TYPE_TIMER);
}
static const struct bpf_func_proto bpf_timer_init_proto = {
.func = bpf_timer_init,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
static int bpf_async_update_prog_callback(struct bpf_async_cb *cb,
struct bpf_prog *prog,
void *callback_fn)
{
struct bpf_prog *prev;
/* Acquire a guard reference on prog to prevent it from being freed during the loop */
if (prog) {
prog = bpf_prog_inc_not_zero(prog);
if (IS_ERR(prog))
return PTR_ERR(prog);
}
do {
if (prog)
prog = bpf_prog_inc_not_zero(prog);
prev = xchg(&cb->prog, prog);
rcu_assign_pointer(cb->callback_fn, callback_fn);
/*
* Release previous prog, make sure that if other CPU is contending,
* to set bpf_prog, references are not leaked as each iteration acquires and
* releases one reference.
*/
if (prev)
bpf_prog_put(prev);
} while (READ_ONCE(cb->prog) != prog ||
(void __force *)READ_ONCE(cb->callback_fn) != callback_fn);
if (prog)
bpf_prog_put(prog);
return 0;
}
static DEFINE_PER_CPU(struct bpf_async_cb *, async_cb_running);
static int bpf_async_schedule_op(struct bpf_async_cb *cb, enum bpf_async_op op,
u64 nsec, u32 timer_mode)
{
/*
* Do not schedule another operation on this cpu if it's in irq_work
* callback that is processing async_cmds queue. Otherwise the following
* loop is possible:
* bpf_timer_start() -> bpf_async_schedule_op() -> irq_work_queue().
* irqrestore -> bpf_async_irq_worker() -> tracepoint -> bpf_timer_start().
*/
if (this_cpu_read(async_cb_running) == cb) {
bpf_async_refcount_put(cb);
return -EDEADLK;
}
struct bpf_async_cmd *cmd = kmalloc_nolock(sizeof(*cmd), 0, NUMA_NO_NODE);
if (!cmd) {
bpf_async_refcount_put(cb);
return -ENOMEM;
}
init_llist_node(&cmd->node);
cmd->nsec = nsec;
cmd->mode = timer_mode;
cmd->op = op;
if (llist_add(&cmd->node, &cb->async_cmds))
irq_work_queue(&cb->worker);
return 0;
}
static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback_fn,
struct bpf_prog *prog)
{
struct bpf_async_cb *cb;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
return bpf_async_update_prog_callback(cb, prog, callback_fn);
}
BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
struct bpf_prog_aux *, aux)
{
return __bpf_async_set_callback(timer, callback_fn, aux->prog);
}
static const struct bpf_func_proto bpf_timer_set_callback_proto = {
.func = bpf_timer_set_callback,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_PTR_TO_FUNC,
};
static bool defer_timer_wq_op(void)
{
return in_hardirq() || irqs_disabled();
}
BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags)
{
struct bpf_hrtimer *t;
u32 mode;
if (flags & ~(BPF_F_TIMER_ABS | BPF_F_TIMER_CPU_PIN))
return -EINVAL;
t = READ_ONCE(async->timer);
if (!t || !READ_ONCE(t->cb.prog))
return -EINVAL;
if (flags & BPF_F_TIMER_ABS)
mode = HRTIMER_MODE_ABS_SOFT;
else
mode = HRTIMER_MODE_REL_SOFT;
if (flags & BPF_F_TIMER_CPU_PIN)
mode |= HRTIMER_MODE_PINNED;
/*
* bpf_async_cancel_and_free() could have dropped refcnt to zero. In
* such case BPF progs are not allowed to arm the timer to prevent UAF.
*/
if (!refcount_inc_not_zero(&t->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
hrtimer_start(&t->timer, ns_to_ktime(nsecs), mode);
bpf_async_refcount_put(&t->cb);
return 0;
} else {
return bpf_async_schedule_op(&t->cb, BPF_ASYNC_START, nsecs, mode);
}
}
static const struct bpf_func_proto bpf_timer_start_proto = {
.func = bpf_timer_start,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async)
{
struct bpf_hrtimer *t, *cur_t;
bool inc = false;
int ret = 0;
if (defer_timer_wq_op())
return -EOPNOTSUPP;
t = READ_ONCE(async->timer);
if (!t)
return -EINVAL;
cur_t = this_cpu_read(hrtimer_running);
if (cur_t == t) {
/* If bpf callback_fn is trying to bpf_timer_cancel()
* its own timer the hrtimer_cancel() will deadlock
* since it waits for callback_fn to finish.
*/
return -EDEADLK;
}
/* Only account in-flight cancellations when invoked from a timer
* callback, since we want to avoid waiting only if other _callbacks_
* are waiting on us, to avoid introducing lockups. Non-callback paths
* are ok, since nobody would synchronously wait for their completion.
*/
if (!cur_t)
goto drop;
atomic_inc(&t->cancelling);
/* Need full barrier after relaxed atomic_inc */
smp_mb__after_atomic();
inc = true;
if (atomic_read(&cur_t->cancelling)) {
/* We're cancelling timer t, while some other timer callback is
* attempting to cancel us. In such a case, it might be possible
* that timer t belongs to the other callback, or some other
* callback waiting upon it (creating transitive dependencies
* upon us), and we will enter a deadlock if we continue
* cancelling and waiting for it synchronously, since it might
* do the same. Bail!
*/
atomic_dec(&t->cancelling);
return -EDEADLK;
}
drop:
bpf_async_update_prog_callback(&t->cb, NULL, NULL);
/* Cancel the timer and wait for associated callback to finish
* if it was running.
*/
ret = hrtimer_cancel(&t->timer);
if (inc)
atomic_dec(&t->cancelling);
return ret;
}
static const struct bpf_func_proto bpf_timer_cancel_proto = {
.func = bpf_timer_cancel,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
};
static void bpf_async_process_op(struct bpf_async_cb *cb, u32 op,
u64 timer_nsec, u32 timer_mode)
{
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER: {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
switch (op) {
case BPF_ASYNC_START:
hrtimer_start(&t->timer, ns_to_ktime(timer_nsec), timer_mode);
break;
case BPF_ASYNC_CANCEL:
hrtimer_try_to_cancel(&t->timer);
break;
}
break;
}
case BPF_ASYNC_TYPE_WQ: {
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
switch (op) {
case BPF_ASYNC_START:
schedule_work(&w->work);
break;
case BPF_ASYNC_CANCEL:
cancel_work(&w->work);
break;
}
break;
}
}
bpf_async_refcount_put(cb);
}
static void bpf_async_irq_worker(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
struct llist_node *pos, *n, *list;
list = llist_del_all(&cb->async_cmds);
if (!list)
return;
list = llist_reverse_order(list);
this_cpu_write(async_cb_running, cb);
llist_for_each_safe(pos, n, list) {
struct bpf_async_cmd *cmd;
cmd = container_of(pos, struct bpf_async_cmd, node);
bpf_async_process_op(cb, cmd->op, cmd->nsec, cmd->mode);
kfree_nolock(cmd);
}
this_cpu_write(async_cb_running, NULL);
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async)
{
struct bpf_async_cb *cb;
if (!READ_ONCE(async->cb))
return;
cb = xchg(&async->cb, NULL);
if (!cb)
return;
bpf_async_update_prog_callback(cb, NULL, NULL);
/*
* No refcount_inc_not_zero(&cb->refcnt) here. Dropping the last
* refcnt. Either synchronously or asynchronously in irq_work.
*/
if (!defer_timer_wq_op()) {
bpf_async_process_op(cb, BPF_ASYNC_CANCEL, 0, 0);
} else {
(void)bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
/*
* bpf_async_schedule_op() either enqueues allocated cmd into llist
* or fails with ENOMEM and drop the last refcnt.
* This is unlikely, but safe, since bpf_async_cb_rcu_tasks_trace_free()
* callback will do additional timer/wq_cancel due to races anyway.
*/
}
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_timer_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_wq_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
{
unsigned long *kptr = dst;
/* This helper may be inlined by verifier. */
return xchg(kptr, (unsigned long)ptr);
}
/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg()
* helper is determined dynamically by the verifier. Use BPF_PTR_POISON to
* denote type that verifier will determine.
*/
static const struct bpf_func_proto bpf_kptr_xchg_proto = {
.func = bpf_kptr_xchg,
.gpl_only = false,
.ret_type = RET_PTR_TO_BTF_ID_OR_NULL,
.ret_btf_id = BPF_PTR_POISON,
.arg1_type = ARG_KPTR_XCHG_DEST,
.arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL | OBJ_RELEASE,
.arg2_btf_id = BPF_PTR_POISON,
};
struct bpf_dynptr_file_impl {
struct freader freader;
/* 64 bit offset and size overriding 32 bit ones in bpf_dynptr_kern */
u64 offset;
u64 size;
};
/* Since the upper 8 bits of dynptr->size is reserved, the
* maximum supported size is 2^24 - 1.
*/
#define DYNPTR_MAX_SIZE ((1UL << 24) - 1)
#define DYNPTR_TYPE_SHIFT 28
#define DYNPTR_SIZE_MASK 0xFFFFFF
#define DYNPTR_RDONLY_BIT BIT(31)
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr)
{
return ptr->size & DYNPTR_RDONLY_BIT;
}
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
{
ptr->size |= DYNPTR_RDONLY_BIT;
}
static void bpf_dynptr_set_type(struct bpf_dynptr_kern *ptr, enum bpf_dynptr_type type)
{
ptr->size |= type << DYNPTR_TYPE_SHIFT;
}
static enum bpf_dynptr_type bpf_dynptr_get_type(const struct bpf_dynptr_kern *ptr)
{
return (ptr->size & ~(DYNPTR_RDONLY_BIT)) >> DYNPTR_TYPE_SHIFT;
}
u64 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
return df->size;
}
return ptr->size & DYNPTR_SIZE_MASK;
}
static void bpf_dynptr_advance_offset(struct bpf_dynptr_kern *ptr, u64 off)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->offset += off;
return;
}
ptr->offset += off;
}
static void bpf_dynptr_set_size(struct bpf_dynptr_kern *ptr, u64 new_size)
{
u32 metadata = ptr->size & ~DYNPTR_SIZE_MASK;
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->size = new_size;
return;
}
ptr->size = (u32)new_size | metadata;
}
int bpf_dynptr_check_size(u64 size)
{
return size > DYNPTR_MAX_SIZE ? -E2BIG : 0;
}
static int bpf_file_fetch_bytes(struct bpf_dynptr_file_impl *df, u64 offset, void *buf, u64 len)
{
const void *ptr;
if (!buf)
return -EINVAL;
df->freader.buf = buf;
df->freader.buf_sz = len;
ptr = freader_fetch(&df->freader, offset + df->offset, len);
if (!ptr)
return df->freader.err;
if (ptr != buf) /* Force copying into the buffer */
memcpy(buf, ptr, len);
return 0;
}
void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size)
{
ptr->data = data;
ptr->offset = offset;
ptr->size = size;
bpf_dynptr_set_type(ptr, type);
}
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)
{
memset(ptr, 0, sizeof(*ptr));
}
BPF_CALL_4(bpf_dynptr_from_mem, void *, data, u64, size, u64, flags, struct bpf_dynptr_kern *, ptr)
{
int err;
BTF_TYPE_EMIT(struct bpf_dynptr);
err = bpf_dynptr_check_size(size);
if (err)
goto error;
/* flags is currently unsupported */
if (flags) {
err = -EINVAL;
goto error;
}
bpf_dynptr_init(ptr, data, BPF_DYNPTR_TYPE_LOCAL, 0, size);
return 0;
error:
bpf_dynptr_set_null(ptr);
return err;
}
static const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
.func = bpf_dynptr_from_mem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL | MEM_UNINIT | MEM_WRITE,
};
static int __bpf_dynptr_read(void *dst, u64 len, const struct bpf_dynptr_kern *src,
u64 offset, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!src->data || flags)
return -EINVAL;
err = bpf_dynptr_check_off_len(src, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(src);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst, src->data + src->offset + offset, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_XDP:
return __bpf_xdp_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_SKB_META:
memmove(dst, bpf_skb_meta_pointer(src->data, src->offset + offset), len);
return 0;
case BPF_DYNPTR_TYPE_FILE:
return bpf_file_fetch_bytes(src->data, offset, dst, len);
default:
WARN_ONCE(true, "bpf_dynptr_read: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_read, void *, dst, u64, len, const struct bpf_dynptr_kern *, src,
u64, offset, u64, flags)
{
return __bpf_dynptr_read(dst, len, src, offset, flags);
}
static const struct bpf_func_proto bpf_dynptr_read_proto = {
.func = bpf_dynptr_read,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
int __bpf_dynptr_write(const struct bpf_dynptr_kern *dst, u64 offset, void *src,
u64 len, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!dst->data || __bpf_dynptr_is_rdonly(dst))
return -EINVAL;
err = bpf_dynptr_check_off_len(dst, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(dst);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
if (flags)
return -EINVAL;
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst->data + dst->offset + offset, src, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_store_bytes(dst->data, dst->offset + offset, src, len,
flags);
case BPF_DYNPTR_TYPE_XDP:
if (flags)
return -EINVAL;
return __bpf_xdp_store_bytes(dst->data, dst->offset + offset, src, len);
case BPF_DYNPTR_TYPE_SKB_META:
return __bpf_skb_meta_store_bytes(dst->data, dst->offset + offset, src,
len, flags);
default:
WARN_ONCE(true, "bpf_dynptr_write: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_write, const struct bpf_dynptr_kern *, dst, u64, offset, void *, src,
u64, len, u64, flags)
{
return __bpf_dynptr_write(dst, offset, src, len, flags);
}
static const struct bpf_func_proto bpf_dynptr_write_proto = {
.func = bpf_dynptr_write,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_dynptr_data, const struct bpf_dynptr_kern *, ptr, u64, offset, u64, len)
{
enum bpf_dynptr_type type;
int err;
if (!ptr->data)
return 0;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return 0;
if (__bpf_dynptr_is_rdonly(ptr))
return 0;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return (unsigned long)(ptr->data + ptr->offset + offset);
case BPF_DYNPTR_TYPE_SKB:
case BPF_DYNPTR_TYPE_XDP:
case BPF_DYNPTR_TYPE_SKB_META:
/* skb and xdp dynptrs should use bpf_dynptr_slice / bpf_dynptr_slice_rdwr */
return 0;
default:
WARN_ONCE(true, "bpf_dynptr_data: unknown dynptr type %d\n", type);
return 0;
}
}
static const struct bpf_func_proto bpf_dynptr_data_proto = {
.func = bpf_dynptr_data,
.gpl_only = false,
.ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
};
const struct bpf_func_proto bpf_get_current_task_proto __weak;
const struct bpf_func_proto bpf_get_current_task_btf_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_str_proto __weak;
const struct bpf_func_proto bpf_task_pt_regs_proto __weak;
const struct bpf_func_proto bpf_perf_event_read_proto __weak;
const struct bpf_func_proto bpf_send_signal_proto __weak;
const struct bpf_func_proto bpf_send_signal_thread_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_sleepable_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_proto __weak;
const struct bpf_func_proto bpf_get_branch_snapshot_proto __weak;
const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_map_lookup_elem:
return &bpf_map_lookup_elem_proto;
case BPF_FUNC_map_update_elem:
return &bpf_map_update_elem_proto;
case BPF_FUNC_map_delete_elem:
return &bpf_map_delete_elem_proto;
case BPF_FUNC_map_push_elem:
return &bpf_map_push_elem_proto;
case BPF_FUNC_map_pop_elem:
return &bpf_map_pop_elem_proto;
case BPF_FUNC_map_peek_elem:
return &bpf_map_peek_elem_proto;
case BPF_FUNC_map_lookup_percpu_elem:
return &bpf_map_lookup_percpu_elem_proto;
case BPF_FUNC_get_prandom_u32:
return &bpf_get_prandom_u32_proto;
case BPF_FUNC_get_smp_processor_id:
return &bpf_get_raw_smp_processor_id_proto;
case BPF_FUNC_get_numa_node_id:
return &bpf_get_numa_node_id_proto;
case BPF_FUNC_tail_call:
return &bpf_tail_call_proto;
case BPF_FUNC_ktime_get_ns:
return &bpf_ktime_get_ns_proto;
case BPF_FUNC_ktime_get_boot_ns:
return &bpf_ktime_get_boot_ns_proto;
case BPF_FUNC_ktime_get_tai_ns:
return &bpf_ktime_get_tai_ns_proto;
case BPF_FUNC_ringbuf_output:
return &bpf_ringbuf_output_proto;
case BPF_FUNC_ringbuf_reserve:
return &bpf_ringbuf_reserve_proto;
case BPF_FUNC_ringbuf_submit:
return &bpf_ringbuf_submit_proto;
case BPF_FUNC_ringbuf_discard:
return &bpf_ringbuf_discard_proto;
case BPF_FUNC_ringbuf_query:
return &bpf_ringbuf_query_proto;
case BPF_FUNC_strncmp:
return &bpf_strncmp_proto;
case BPF_FUNC_strtol:
return &bpf_strtol_proto;
case BPF_FUNC_strtoul:
return &bpf_strtoul_proto;
case BPF_FUNC_get_current_pid_tgid:
return &bpf_get_current_pid_tgid_proto;
case BPF_FUNC_get_ns_current_pid_tgid:
return &bpf_get_ns_current_pid_tgid_proto;
case BPF_FUNC_get_current_uid_gid:
return &bpf_get_current_uid_gid_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_BPF))
return NULL;
switch (func_id) {
case BPF_FUNC_spin_lock:
return &bpf_spin_lock_proto;
case BPF_FUNC_spin_unlock:
return &bpf_spin_unlock_proto;
case BPF_FUNC_jiffies64:
return &bpf_jiffies64_proto;
case BPF_FUNC_per_cpu_ptr:
return &bpf_per_cpu_ptr_proto;
case BPF_FUNC_this_cpu_ptr:
return &bpf_this_cpu_ptr_proto;
case BPF_FUNC_timer_init:
return &bpf_timer_init_proto;
case BPF_FUNC_timer_set_callback:
return &bpf_timer_set_callback_proto;
case BPF_FUNC_timer_start:
return &bpf_timer_start_proto;
case BPF_FUNC_timer_cancel:
return &bpf_timer_cancel_proto;
case BPF_FUNC_kptr_xchg:
return &bpf_kptr_xchg_proto;
case BPF_FUNC_for_each_map_elem:
return &bpf_for_each_map_elem_proto;
case BPF_FUNC_loop:
return &bpf_loop_proto;
case BPF_FUNC_user_ringbuf_drain:
return &bpf_user_ringbuf_drain_proto;
case BPF_FUNC_ringbuf_reserve_dynptr:
return &bpf_ringbuf_reserve_dynptr_proto;
case BPF_FUNC_ringbuf_submit_dynptr:
return &bpf_ringbuf_submit_dynptr_proto;
case BPF_FUNC_ringbuf_discard_dynptr:
return &bpf_ringbuf_discard_dynptr_proto;
case BPF_FUNC_dynptr_from_mem:
return &bpf_dynptr_from_mem_proto;
case BPF_FUNC_dynptr_read:
return &bpf_dynptr_read_proto;
case BPF_FUNC_dynptr_write:
return &bpf_dynptr_write_proto;
case BPF_FUNC_dynptr_data:
return &bpf_dynptr_data_proto;
#ifdef CONFIG_CGROUPS
case BPF_FUNC_cgrp_storage_get:
return &bpf_cgrp_storage_get_proto;
case BPF_FUNC_cgrp_storage_delete:
return &bpf_cgrp_storage_delete_proto;
case BPF_FUNC_get_current_cgroup_id:
return &bpf_get_current_cgroup_id_proto;
case BPF_FUNC_get_current_ancestor_cgroup_id:
return &bpf_get_current_ancestor_cgroup_id_proto;
case BPF_FUNC_current_task_under_cgroup:
return &bpf_current_task_under_cgroup_proto;
#endif
#ifdef CONFIG_CGROUP_NET_CLASSID
case BPF_FUNC_get_cgroup_classid:
return &bpf_get_cgroup_classid_curr_proto;
#endif
case BPF_FUNC_task_storage_get:
return &bpf_task_storage_get_proto;
case BPF_FUNC_task_storage_delete:
return &bpf_task_storage_delete_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_PERFMON))
return NULL;
switch (func_id) {
case BPF_FUNC_trace_printk:
return bpf_get_trace_printk_proto();
case BPF_FUNC_get_current_task:
return &bpf_get_current_task_proto;
case BPF_FUNC_get_current_task_btf:
return &bpf_get_current_task_btf_proto;
case BPF_FUNC_get_current_comm:
return &bpf_get_current_comm_proto;
case BPF_FUNC_probe_read_user:
return &bpf_probe_read_user_proto;
case BPF_FUNC_probe_read_kernel:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_proto;
case BPF_FUNC_probe_read_user_str:
return &bpf_probe_read_user_str_proto;
case BPF_FUNC_probe_read_kernel_str:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_str_proto;
case BPF_FUNC_copy_from_user:
return &bpf_copy_from_user_proto;
case BPF_FUNC_copy_from_user_task:
return &bpf_copy_from_user_task_proto;
case BPF_FUNC_snprintf_btf:
return &bpf_snprintf_btf_proto;
case BPF_FUNC_snprintf:
return &bpf_snprintf_proto;
case BPF_FUNC_task_pt_regs:
return &bpf_task_pt_regs_proto;
case BPF_FUNC_trace_vprintk:
return bpf_get_trace_vprintk_proto();
case BPF_FUNC_perf_event_read_value:
return bpf_get_perf_event_read_value_proto();
case BPF_FUNC_perf_event_read:
return &bpf_perf_event_read_proto;
case BPF_FUNC_send_signal:
return &bpf_send_signal_proto;
case BPF_FUNC_send_signal_thread:
return &bpf_send_signal_thread_proto;
case BPF_FUNC_get_task_stack:
return prog->sleepable ? &bpf_get_task_stack_sleepable_proto
: &bpf_get_task_stack_proto;
case BPF_FUNC_get_branch_snapshot:
return &bpf_get_branch_snapshot_proto;
case BPF_FUNC_find_vma:
return &bpf_find_vma_proto;
default:
return NULL;
}
}
EXPORT_SYMBOL_GPL(bpf_base_func_proto);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock)
{
struct list_head *head = list_head, *orig_head = list_head;
BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
/* Do the actual list draining outside the lock to not hold the lock for
* too long, and also prevent deadlocks if tracing programs end up
* executing on entry/exit of functions called inside the critical
* section, and end up doing map ops that call bpf_list_head_free for
* the same map value again.
*/
__bpf_spin_lock_irqsave(spin_lock);
if (!head->next || list_empty(head))
goto unlock;
head = head->next;
unlock:
INIT_LIST_HEAD(orig_head);
__bpf_spin_unlock_irqrestore(spin_lock);
while (head != orig_head) {
void *obj = head;
obj -= field->graph_root.node_offset;
head = head->next;
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
/* Like rbtree_postorder_for_each_entry_safe, but 'pos' and 'n' are
* 'rb_node *', so field name of rb_node within containing struct is not
* needed.
*
* Since bpf_rb_tree's node type has a corresponding struct btf_field with
* graph_root.node_offset, it's not necessary to know field name
* or type of node struct
*/
#define bpf_rbtree_postorder_for_each_entry_safe(pos, n, root) \
for (pos = rb_first_postorder(root); \
pos && ({ n = rb_next_postorder(pos); 1; }); \
pos = n)
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock)
{
struct rb_root_cached orig_root, *root = rb_root;
struct rb_node *pos, *n;
void *obj;
BUILD_BUG_ON(sizeof(struct rb_root_cached) > sizeof(struct bpf_rb_root));
BUILD_BUG_ON(__alignof__(struct rb_root_cached) > __alignof__(struct bpf_rb_root));
__bpf_spin_lock_irqsave(spin_lock);
orig_root = *root;
*root = RB_ROOT_CACHED;
__bpf_spin_unlock_irqrestore(spin_lock);
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
obj = pos;
obj -= field->graph_root.node_offset;
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
__bpf_kfunc_start_defs();
/**
* bpf_obj_new() - allocate an object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate an object of the type identified by @local_type_id__k and
* initialize its special fields. BPF programs can use
* bpf_core_type_id_local() to provide @local_type_id__k. The verifier
* rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
void *p;
p = bpf_mem_alloc(&bpf_global_ma, size);
if (!p)
return NULL;
if (meta)
bpf_obj_init(meta->record, p);
return p;
}
__bpf_kfunc void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_obj_new(local_type_id__k, meta__ign);
}
/**
* bpf_percpu_obj_new() - allocate a percpu object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate a percpu object of the type identified by @local_type_id__k. BPF
* programs can use bpf_core_type_id_local() to provide @local_type_id__k.
* The verifier rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated percpu object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_percpu_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
/* The verifier has ensured that meta must be NULL */
return bpf_mem_alloc(&bpf_global_percpu_ma, size);
}
__bpf_kfunc void *bpf_percpu_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_percpu_obj_new(local_type_id__k, meta__ign);
}
/* Must be called under migrate_disable(), as required by bpf_mem_free */
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu)
{
struct bpf_mem_alloc *ma;
if (rec && rec->refcount_off >= 0 &&
!refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) {
/* Object is refcounted and refcount_dec didn't result in 0
* refcount. Return without freeing the object
*/
return;
}
if (rec)
bpf_obj_free_fields(rec, p);
if (percpu)
ma = &bpf_global_percpu_ma;
else
ma = &bpf_global_ma;
bpf_mem_free_rcu(ma, p);
}
/**
* bpf_obj_drop() - drop a previously allocated object
* @p__alloc: object to free
* @meta: verifier-supplied struct metadata
*
* Destroy special fields in @p__alloc as needed and free the object. The
* verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
void *p = p__alloc;
__bpf_obj_drop_impl(p, meta ? meta->record : NULL, false);
}
__bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
{
return bpf_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_percpu_obj_drop() - drop a previously allocated percpu object
* @p__alloc: percpu object to free
* @meta: verifier-supplied struct metadata
*
* Free @p__alloc. The verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_percpu_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
/* The verifier has ensured that meta must be NULL */
bpf_mem_free_rcu(&bpf_global_percpu_ma, p__alloc);
}
__bpf_kfunc void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign)
{
bpf_percpu_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_refcount_acquire() - turn a local kptr into an owning reference
* @p__refcounted_kptr: non-owning local kptr
* @meta: verifier-supplied struct metadata
*
* Increment the refcount for @p__refcounted_kptr. The verifier rewrites
* @meta; BPF programs do not set it.
*
* Return: Owning reference to @p__refcounted_kptr, or %NULL on failure.
*/
__bpf_kfunc void *bpf_refcount_acquire(void *p__refcounted_kptr, struct btf_struct_meta *meta)
{
struct bpf_refcount *ref;
/* Could just cast directly to refcount_t *, but need some code using
* bpf_refcount type so that it is emitted in vmlinux BTF
*/
ref = (struct bpf_refcount *)(p__refcounted_kptr + meta->record->refcount_off);
if (!refcount_inc_not_zero((refcount_t *)ref))
return NULL;
/* Verifier strips KF_RET_NULL if input is owned ref, see is_kfunc_ret_null
* in verifier.c
*/
return (void *)p__refcounted_kptr;
}
__bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta__ign)
{
return bpf_refcount_acquire(p__refcounted_kptr, meta__ign);
}
static int __bpf_list_add(struct bpf_list_node_kern *node,
struct bpf_list_head *head,
bool tail, struct btf_record *rec, u64 off)
{
struct list_head *n = &node->list_head, *h = (void *)head;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
/* node->owner != NULL implies !list_empty(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
tail ? list_add_tail(n, h) : list_add(n, h);
WRITE_ONCE(node->owner, head);
return 0;
}
/**
* bpf_list_push_front() - add a node to the front of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the front of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_front(head, node, meta__ign, off);
}
/**
* bpf_list_push_back() - add a node to the back of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the back of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_back(head, node, meta__ign, off);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
{
struct list_head *n, *h = (void *)head;
struct bpf_list_node_kern *node;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
if (list_empty(h))
return NULL;
n = tail ? h->prev : h->next;
node = container_of(n, struct bpf_list_node_kern, list_head);
if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))
return NULL;
list_del_init(n);
WRITE_ONCE(node->owner, NULL);
return (struct bpf_list_node *)n;
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
{
return __bpf_list_del(head, false);
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
{
return __bpf_list_del(head, true);
}
__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->next;
}
__bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->prev;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
struct rb_root_cached *r = (struct rb_root_cached *)root;
struct rb_node *n = &node_internal->rb_node;
/* node_internal->owner != root implies either RB_EMPTY_NODE(n) or
* n is owned by some other tree. No need to check RB_EMPTY_NODE(n)
*/
if (READ_ONCE(node_internal->owner) != root)
return NULL;
rb_erase_cached(n, r);
RB_CLEAR_NODE(n);
WRITE_ONCE(node_internal->owner, NULL);
return (struct bpf_rb_node *)n;
}
/* Need to copy rbtree_add_cached's logic here because our 'less' is a BPF
* program
*/
static int __bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node_kern *node,
void *less, struct btf_record *rec, u64 off)
{
struct rb_node **link = &((struct rb_root_cached *)root)->rb_root.rb_node;
struct rb_node *parent = NULL, *n = &node->rb_node;
bpf_callback_t cb = (bpf_callback_t)less;
bool leftmost = true;
/* node->owner != NULL implies !RB_EMPTY_NODE(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
while (*link) {
parent = *link;
if (cb((uintptr_t)node, (uintptr_t)parent, 0, 0, 0)) {
link = &parent->rb_left;
} else {
link = &parent->rb_right;
leftmost = false;
}
}
rb_link_node(n, parent, link);
rb_insert_color_cached(n, (struct rb_root_cached *)root, leftmost);
WRITE_ONCE(node->owner, root);
return 0;
}
/**
* bpf_rbtree_add() - add a node to a BPF rbtree
* @root: tree root
* @node: node to insert
* @less: comparator used to order nodes
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node into @root using @less. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked in a tree.
*/
__bpf_kfunc int bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_rb_node_kern *n = (void *)node;
return __bpf_rbtree_add(root, n, (void *)less, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
void *meta__ign, u64 off)
{
return bpf_rbtree_add(root, node, less, meta__ign, off);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)rb_first_cached(r);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_root(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)r->rb_root.rb_node;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_left(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_left;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_right(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_right;
}
/**
* bpf_task_acquire - Acquire a reference to a task. A task acquired by this
* kfunc which is not stored in a map as a kptr, must be released by calling
* bpf_task_release().
* @p: The task on which a reference is being acquired.
*/
__bpf_kfunc struct task_struct *bpf_task_acquire(struct task_struct *p)
{
if (refcount_inc_not_zero(&p->rcu_users))
return p;
return NULL;
}
/**
* bpf_task_release - Release the reference acquired on a task.
* @p: The task on which a reference is being released.
*/
__bpf_kfunc void bpf_task_release(struct task_struct *p)
{
put_task_struct_rcu_user(p);
}
__bpf_kfunc void bpf_task_release_dtor(void *p)
{
put_task_struct_rcu_user(p);
}
CFI_NOSEAL(bpf_task_release_dtor);
#ifdef CONFIG_CGROUPS
/**
* bpf_cgroup_acquire - Acquire a reference to a cgroup. A cgroup acquired by
* this kfunc which is not stored in a map as a kptr, must be released by
* calling bpf_cgroup_release().
* @cgrp: The cgroup on which a reference is being acquired.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)
{
return cgroup_tryget(cgrp) ? cgrp : NULL;
}
/**
* bpf_cgroup_release - Release the reference acquired on a cgroup.
* If this kfunc is invoked in an RCU read region, the cgroup is guaranteed to
* not be freed until the current grace period has ended, even if its refcount
* drops to 0.
* @cgrp: The cgroup on which a reference is being released.
*/
__bpf_kfunc void bpf_cgroup_release(struct cgroup *cgrp)
{
cgroup_put(cgrp);
}
__bpf_kfunc void bpf_cgroup_release_dtor(void *cgrp)
{
cgroup_put(cgrp);
}
CFI_NOSEAL(bpf_cgroup_release_dtor);
/**
* bpf_cgroup_ancestor - Perform a lookup on an entry in a cgroup's ancestor
* array. A cgroup returned by this kfunc which is not subsequently stored in a
* map, must be released by calling bpf_cgroup_release().
* @cgrp: The cgroup for which we're performing a lookup.
* @level: The level of ancestor to look up.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)
{
struct cgroup *ancestor;
if (level > cgrp->level || level < 0)
return NULL;
/* cgrp's refcnt could be 0 here, but ancestors can still be accessed */
ancestor = cgrp->ancestors[level];
if (!cgroup_tryget(ancestor))
return NULL;
return ancestor;
}
/**
* bpf_cgroup_from_id - Find a cgroup from its ID. A cgroup returned by this
* kfunc which is not subsequently stored in a map, must be released by calling
* bpf_cgroup_release().
* @cgid: cgroup id.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_from_id(u64 cgid)
{
struct cgroup *cgrp;
cgrp = __cgroup_get_from_id(cgid);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
/**
* bpf_task_under_cgroup - wrap task_under_cgroup_hierarchy() as a kfunc, test
* task's membership of cgroup ancestry.
* @task: the task to be tested
* @ancestor: possible ancestor of @task's cgroup
*
* Tests whether @task's default cgroup hierarchy is a descendant of @ancestor.
* It follows all the same rules as cgroup_is_descendant, and only applies
* to the default hierarchy.
*/
__bpf_kfunc long bpf_task_under_cgroup(struct task_struct *task,
struct cgroup *ancestor)
{
long ret;
rcu_read_lock();
ret = task_under_cgroup_hierarchy(task, ancestor);
rcu_read_unlock();
return ret;
}
BPF_CALL_2(bpf_current_task_under_cgroup, struct bpf_map *, map, u32, idx)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
struct cgroup *cgrp;
if (unlikely(idx >= array->map.max_entries))
return -E2BIG;
cgrp = READ_ONCE(array->ptrs[idx]);
if (unlikely(!cgrp))
return -EAGAIN;
return task_under_cgroup_hierarchy(current, cgrp);
}
const struct bpf_func_proto bpf_current_task_under_cgroup_proto = {
.func = bpf_current_task_under_cgroup,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
};
/**
* bpf_task_get_cgroup1 - Acquires the associated cgroup of a task within a
* specific cgroup1 hierarchy. The cgroup1 hierarchy is identified by its
* hierarchy ID.
* @task: The target task
* @hierarchy_id: The ID of a cgroup1 hierarchy
*
* On success, the cgroup is returen. On failure, NULL is returned.
*/
__bpf_kfunc struct cgroup *
bpf_task_get_cgroup1(struct task_struct *task, int hierarchy_id)
{
struct cgroup *cgrp = task_get_cgroup1(task, hierarchy_id);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
#endif /* CONFIG_CGROUPS */
/**
* bpf_task_from_pid - Find a struct task_struct from its pid by looking it up
* in the root pid namespace idr. If a task is returned, it must either be
* stored in a map, or released with bpf_task_release().
* @pid: The pid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_pid_ns(pid, &init_pid_ns);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_task_from_vpid - Find a struct task_struct from its vpid by looking it up
* in the pid namespace of the current task. If a task is returned, it must
* either be stored in a map, or released with bpf_task_release().
* @vpid: The vpid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_vpid(vpid);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_dynptr_slice() - Obtain a read-only pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* If the intention is to write to the data slice, please use
* bpf_dynptr_slice_rdwr.
*
* The user must check that the returned pointer is not null before using it.
*
* Please note that in the case of skb and xdp dynptrs, bpf_dynptr_slice
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a read-only
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
enum bpf_dynptr_type type;
u64 len = buffer__szk;
int err;
if (!ptr->data)
return NULL;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return NULL;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return ptr->data + ptr->offset + offset;
case BPF_DYNPTR_TYPE_SKB:
if (buffer__nullable)
return skb_header_pointer(ptr->data, ptr->offset + offset, len, buffer__nullable);
else
return skb_pointer_if_linear(ptr->data, ptr->offset + offset, len);
case BPF_DYNPTR_TYPE_XDP:
{
void *xdp_ptr = bpf_xdp_pointer(ptr->data, ptr->offset + offset, len);
if (!IS_ERR_OR_NULL(xdp_ptr))
return xdp_ptr;
if (!buffer__nullable)
return NULL;
bpf_xdp_copy_buf(ptr->data, ptr->offset + offset, buffer__nullable, len, false);
return buffer__nullable;
}
case BPF_DYNPTR_TYPE_SKB_META:
return bpf_skb_meta_pointer(ptr->data, ptr->offset + offset);
case BPF_DYNPTR_TYPE_FILE:
err = bpf_file_fetch_bytes(ptr->data, offset, buffer__nullable, buffer__szk);
return err ? NULL : buffer__nullable;
default:
WARN_ONCE(true, "unknown dynptr type %d\n", type);
return NULL;
}
}
/**
* bpf_dynptr_slice_rdwr() - Obtain a writable pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* The returned pointer is writable and may point to either directly the dynptr
* data at the requested offset or to the buffer if unable to obtain a direct
* data pointer to (example: the requested slice is to the paged area of an skb
* packet). In the case where the returned pointer is to the buffer, the user
* is responsible for persisting writes through calling bpf_dynptr_write(). This
* usually looks something like this pattern:
*
* struct eth_hdr *eth = bpf_dynptr_slice_rdwr(&dynptr, 0, buffer, sizeof(buffer));
* if (!eth)
* return TC_ACT_SHOT;
*
* // mutate eth header //
*
* if (eth == buffer)
* bpf_dynptr_write(&ptr, 0, buffer, sizeof(buffer), 0);
*
* Please note that, as in the example above, the user must check that the
* returned pointer is not null before using it.
*
* Please also note that in the case of skb and xdp dynptrs, bpf_dynptr_slice_rdwr
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice_rdwr will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data || __bpf_dynptr_is_rdonly(ptr))
return NULL;
/* bpf_dynptr_slice_rdwr is the same logic as bpf_dynptr_slice.
*
* For skb-type dynptrs, it is safe to write into the returned pointer
* if the bpf program allows skb data writes. There are two possibilities
* that may occur when calling bpf_dynptr_slice_rdwr:
*
* 1) The requested slice is in the head of the skb. In this case, the
* returned pointer is directly to skb data, and if the skb is cloned, the
* verifier will have uncloned it (see bpf_unclone_prologue()) already.
* The pointer can be directly written into.
*
* 2) Some portion of the requested slice is in the paged buffer area.
* In this case, the requested data will be copied out into the buffer
* and the returned pointer will be a pointer to the buffer. The skb
* will not be pulled. To persist the write, the user will need to call
* bpf_dynptr_write(), which will pull the skb and commit the write.
*
* Similarly for xdp programs, if the requested slice is not across xdp
* fragments, then a direct pointer will be returned, otherwise the data
* will be copied out into the buffer and the user will need to call
* bpf_dynptr_write() to commit changes.
*/
return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
}
__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 size;
if (!ptr->data || start > end)
return -EINVAL;
size = __bpf_dynptr_size(ptr);
if (start > size || end > size)
return -ERANGE;
bpf_dynptr_advance_offset(ptr, start);
bpf_dynptr_set_size(ptr, end - start);
return 0;
}
__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
return !ptr->data;
}
__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return false;
return __bpf_dynptr_is_rdonly(ptr);
}
__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return -EINVAL;
return __bpf_dynptr_size(ptr);
}
__bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
struct bpf_dynptr *clone__uninit)
{
struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data) {
bpf_dynptr_set_null(clone);
return -EINVAL;
}
*clone = *ptr;
return 0;
}
/**
* bpf_dynptr_copy() - Copy data from one dynptr to another.
* @dst_ptr: Destination dynptr - where data should be copied to
* @dst_off: Offset into the destination dynptr
* @src_ptr: Source dynptr - where data should be copied from
* @src_off: Offset into the source dynptr
* @size: Length of the data to copy from source to destination
*
* Copies data from source dynptr to destination dynptr.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
{
struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
void *src_slice, *dst_slice;
char buf[256];
u64 off;
src_slice = bpf_dynptr_slice(src_ptr, src_off, NULL, size);
dst_slice = bpf_dynptr_slice_rdwr(dst_ptr, dst_off, NULL, size);
if (src_slice && dst_slice) {
memmove(dst_slice, src_slice, size);
return 0;
}
if (src_slice)
return __bpf_dynptr_write(dst, dst_off, src_slice, size, 0);
if (dst_slice)
return __bpf_dynptr_read(dst_slice, size, src, src_off, 0);
if (bpf_dynptr_check_off_len(dst, dst_off, size) ||
bpf_dynptr_check_off_len(src, src_off, size))
return -E2BIG;
off = 0;
while (off < size) {
u64 chunk_sz = min_t(u64, sizeof(buf), size - off);
int err;
err = __bpf_dynptr_read(buf, chunk_sz, src, src_off + off, 0);
if (err)
return err;
err = __bpf_dynptr_write(dst, dst_off + off, buf, chunk_sz, 0);
if (err)
return err;
off += chunk_sz;
}
return 0;
}
/**
* bpf_dynptr_memset() - Fill dynptr memory with a constant byte.
* @p: Destination dynptr - where data will be filled
* @offset: Offset into the dynptr to start filling from
* @size: Number of bytes to fill
* @val: Constant byte to fill the memory with
*
* Fills the @size bytes of the memory area pointed to by @p
* at @offset with the constant byte @val.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 chunk_sz, write_off;
char buf[256];
void* slice;
int err;
slice = bpf_dynptr_slice_rdwr(p, offset, NULL, size);
if (likely(slice)) {
memset(slice, val, size);
return 0;
}
if (__bpf_dynptr_is_rdonly(ptr))
return -EINVAL;
err = bpf_dynptr_check_off_len(ptr, offset, size);
if (err)
return err;
/* Non-linear data under the dynptr, write from a local buffer */
chunk_sz = min_t(u64, sizeof(buf), size);
memset(buf, val, chunk_sz);
for (write_off = 0; write_off < size; write_off += chunk_sz) {
chunk_sz = min_t(u64, sizeof(buf), size - write_off);
err = __bpf_dynptr_write(ptr, offset + write_off, buf, chunk_sz, 0);
if (err)
return err;
}
return 0;
}
__bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
{
return obj;
}
__bpf_kfunc void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k)
{
return (void *)obj__ign;
}
__bpf_kfunc void bpf_rcu_read_lock(void)
{
rcu_read_lock();
}
__bpf_kfunc void bpf_rcu_read_unlock(void)
{
rcu_read_unlock();
}
struct bpf_throw_ctx {
struct bpf_prog_aux *aux;
u64 sp;
u64 bp;
int cnt;
};
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
{
struct bpf_throw_ctx *ctx = cookie;
struct bpf_prog *prog;
/*
* The RCU read lock is held to safely traverse the latch tree, but we
* don't need its protection when accessing the prog, since it has an
* active stack frame on the current stack trace, and won't disappear.
*/
rcu_read_lock();
prog = bpf_prog_ksym_find(ip);
rcu_read_unlock();
if (!prog)
return !ctx->cnt;
ctx->cnt++;
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
ctx->sp = sp;
ctx->bp = bp;
return false;
}
__bpf_kfunc void bpf_throw(u64 cookie)
{
struct bpf_throw_ctx ctx = {};
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
WARN_ON_ONCE(!ctx.aux->exception_boundary);
WARN_ON_ONCE(!ctx.bp);
WARN_ON_ONCE(!ctx.cnt);
/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
* deeper stack depths than ctx.sp as we do not return from bpf_throw,
* which skips compiler generated instrumentation to do the same.
*/
kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_map *map = p__map;
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq));
if (flags)
return -EINVAL;
return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ);
}
__bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_work *w;
if (flags)
return -EINVAL;
w = READ_ONCE(async->work);
if (!w || !READ_ONCE(w->cb.prog))
return -EINVAL;
if (!refcount_inc_not_zero(&w->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
schedule_work(&w->work);
bpf_async_refcount_put(&w->cb);
return 0;
} else {
return bpf_async_schedule_op(&w->cb, BPF_ASYNC_START, 0, 0);
}
}
__bpf_kfunc int bpf_wq_set_callback(struct bpf_wq *wq,
int (callback_fn)(void *map, int *key, void *value),
unsigned int flags,
struct bpf_prog_aux *aux)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
if (flags)
return -EINVAL;
return __bpf_async_set_callback(async, callback_fn, aux->prog);
}
__bpf_kfunc void bpf_preempt_disable(void)
{
preempt_disable();
}
__bpf_kfunc void bpf_preempt_enable(void)
{
preempt_enable();
}
struct bpf_iter_bits {
__u64 __opaque[2];
} __aligned(8);
#define BITS_ITER_NR_WORDS_MAX 511
struct bpf_iter_bits_kern {
union {
__u64 *bits;
__u64 bits_copy;
};
int nr_bits;
int bit;
} __aligned(8);
/* On 64-bit hosts, unsigned long and u64 have the same size, so passing
* a u64 pointer and an unsigned long pointer to find_next_bit() will
* return the same result, as both point to the same 8-byte area.
*
* For 32-bit little-endian hosts, using a u64 pointer or unsigned long
* pointer also makes no difference. This is because the first iterated
* unsigned long is composed of bits 0-31 of the u64 and the second unsigned
* long is composed of bits 32-63 of the u64.
*
* However, for 32-bit big-endian hosts, this is not the case. The first
* iterated unsigned long will be bits 32-63 of the u64, so swap these two
* ulong values within the u64.
*/
static void swap_ulong_in_u64(u64 *bits, unsigned int nr)
{
#if (BITS_PER_LONG == 32) && defined(__BIG_ENDIAN)
unsigned int i;
for (i = 0; i < nr; i++)
bits[i] = (bits[i] >> 32) | ((u64)(u32)bits[i] << 32);
#endif
}
/**
* bpf_iter_bits_new() - Initialize a new bits iterator for a given memory area
* @it: The new bpf_iter_bits to be created
* @unsafe_ptr__ign: A pointer pointing to a memory area to be iterated over
* @nr_words: The size of the specified memory area, measured in 8-byte units.
* The maximum value of @nr_words is @BITS_ITER_NR_WORDS_MAX. This limit may be
* further reduced by the BPF memory allocator implementation.
*
* This function initializes a new bpf_iter_bits structure for iterating over
* a memory area which is specified by the @unsafe_ptr__ign and @nr_words. It
* copies the data of the memory area to the newly created bpf_iter_bits @it for
* subsequent iteration operations.
*
* On success, 0 is returned. On failure, ERR is returned.
*/
__bpf_kfunc int
bpf_iter_bits_new(struct bpf_iter_bits *it, const u64 *unsafe_ptr__ign, u32 nr_words)
{
struct bpf_iter_bits_kern *kit = (void *)it;
u32 nr_bytes = nr_words * sizeof(u64);
u32 nr_bits = BYTES_TO_BITS(nr_bytes);
int err;
BUILD_BUG_ON(sizeof(struct bpf_iter_bits_kern) != sizeof(struct bpf_iter_bits));
BUILD_BUG_ON(__alignof__(struct bpf_iter_bits_kern) !=
__alignof__(struct bpf_iter_bits));
kit->nr_bits = 0;
kit->bits_copy = 0;
kit->bit = -1;
if (!unsafe_ptr__ign || !nr_words)
return -EINVAL;
if (nr_words > BITS_ITER_NR_WORDS_MAX)
return -E2BIG;
/* Optimization for u64 mask */
if (nr_bits == 64) {
err = bpf_probe_read_kernel_common(&kit->bits_copy, nr_bytes, unsafe_ptr__ign);
if (err)
return -EFAULT;
swap_ulong_in_u64(&kit->bits_copy, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
if (bpf_mem_alloc_check_size(false, nr_bytes))
return -E2BIG;
/* Fallback to memalloc */
kit->bits = bpf_mem_alloc(&bpf_global_ma, nr_bytes);
if (!kit->bits)
return -ENOMEM;
err = bpf_probe_read_kernel_common(kit->bits, nr_bytes, unsafe_ptr__ign);
if (err) {
bpf_mem_free(&bpf_global_ma, kit->bits);
return err;
}
swap_ulong_in_u64(kit->bits, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
/**
* bpf_iter_bits_next() - Get the next bit in a bpf_iter_bits
* @it: The bpf_iter_bits to be checked
*
* This function returns a pointer to a number representing the value of the
* next bit in the bits.
*
* If there are no further bits available, it returns NULL.
*/
__bpf_kfunc int *bpf_iter_bits_next(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
int bit = kit->bit, nr_bits = kit->nr_bits;
const void *bits;
if (!nr_bits || bit >= nr_bits)
return NULL;
bits = nr_bits == 64 ? &kit->bits_copy : kit->bits;
bit = find_next_bit(bits, nr_bits, bit + 1);
if (bit >= nr_bits) {
kit->bit = bit;
return NULL;
}
kit->bit = bit;
return &kit->bit;
}
/**
* bpf_iter_bits_destroy() - Destroy a bpf_iter_bits
* @it: The bpf_iter_bits to be destroyed
*
* Destroy the resource associated with the bpf_iter_bits.
*/
__bpf_kfunc void bpf_iter_bits_destroy(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
if (kit->nr_bits <= 64)
return;
bpf_mem_free(&bpf_global_ma, kit->bits);
}
/**
* bpf_copy_from_user_str() - Copy a string from an unsafe user address
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address, in user space.
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL-terminated string from userspace to BPF space. If user string is
* too long this will still ensure zero termination in the dst buffer unless
* buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst to 0 on success and
* memset all of @dst on failure.
*/
__bpf_kfunc int bpf_copy_from_user_str(void *dst, u32 dst__sz, const void __user *unsafe_ptr__ign, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(!dst__sz))
return 0;
ret = strncpy_from_user(dst, unsafe_ptr__ign, dst__sz - 1);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst + ret, 0, dst__sz - ret);
else
((char *)dst)[ret] = '\0';
return ret + 1;
}
/**
* bpf_copy_from_user_task_str() - Copy a string from an task's address space
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address in the task's address space.
* @tsk: The task whose address space will be used
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL terminated string from a task's address space to @dst__sz
* buffer. If user string is too long this will still ensure zero termination
* in the @dst__sz buffer unless buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst__sz to 0 on success
* and memset all of @dst__sz on failure.
*
* Return: The number of copied bytes on success including the NUL terminator.
* A negative error code on failure.
*/
__bpf_kfunc int bpf_copy_from_user_task_str(void *dst, u32 dst__sz,
const void __user *unsafe_ptr__ign,
struct task_struct *tsk, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(dst__sz == 0))
return 0;
ret = copy_remote_vm_str(tsk, (unsigned long)unsafe_ptr__ign, dst, dst__sz, 0);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset(dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset(dst + ret, 0, dst__sz - ret);
return ret + 1;
}
/* Keep unsinged long in prototype so that kfunc is usable when emitted to
* vmlinux.h in BPF programs directly, but note that while in BPF prog, the
* unsigned long always points to 8-byte region on stack, the kernel may only
* read and write the 4-bytes on 32-bit.
*/
__bpf_kfunc void bpf_local_irq_save(unsigned long *flags__irq_flag)
{
local_irq_save(*flags__irq_flag);
}
__bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)
{
local_irq_restore(*flags__irq_flag);
}
__bpf_kfunc void __bpf_trap(void)
{
}
/*
* Kfuncs for string operations.
*
* Since strings are not necessarily %NUL-terminated, we cannot directly call
* in-kernel implementations. Instead, we open-code the implementations using
* __get_kernel_nofault instead of plain dereference to make them safe.
*/
static int __bpf_strncasecmp(const char *s1, const char *s2, bool ignore_case, size_t len)
{
char c1, c2;
int i;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < len && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c1, s1, char, err_out);
__get_kernel_nofault(&c2, s2, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 != c2)
return c1 < c2 ? -1 : 1;
if (c1 == '\0')
return 0;
s1++;
s2++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : 0;
err_out:
return -EFAULT;
}
/**
* bpf_strcmp - Compare two strings
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, false, XATTR_SIZE_MAX);
}
/**
* bpf_strcasecmp - Compare two strings, ignoring the case of the characters
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasecmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, XATTR_SIZE_MAX);
}
/*
* bpf_strncasecmp - Compare two length-limited strings, ignoring case
* @s1__ign: One string
* @s2__ign: Another string
* @len: The maximum number of characters to compare
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasecmp(const char *s1__ign, const char *s2__ign, size_t len)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, len);
}
/**
* bpf_strnchr - Find a character in a length limited string
* @s__ign: The string to be searched
* @count: The number of characters to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in the first @count characters of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnchr(const char *s__ign, size_t count, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
return i;
if (sc == '\0')
return -ENOENT;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : -ENOENT;
err_out:
return -EFAULT;
}
/**
* bpf_strchr - Find the first occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - The index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchr(const char *s__ign, char c)
{
return bpf_strnchr(s__ign, XATTR_SIZE_MAX, c);
}
/**
* bpf_strchrnul - Find and return a character in a string, or end of string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign or index of
* the null byte at the end of @s__ign when @c is not found
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchrnul(const char *s__ign, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == '\0' || sc == c)
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strrchr - Find the last occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the last occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strrchr(const char *s__ign, int c)
{
char sc;
int i, last = -ENOENT;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
last = i;
if (sc == '\0')
return last;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strnlen - Calculate the length of a length-limited string
* @s__ign: The string
* @count: The maximum number of characters to count
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnlen(const char *s__ign, size_t count)
{
char c;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c, s__ign, char, err_out);
if (c == '\0')
return i;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : i;
err_out:
return -EFAULT;
}
/**
* bpf_strlen - Calculate the length of a string
* @s__ign: The string
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strlen(const char *s__ign)
{
return bpf_strnlen(s__ign, XATTR_SIZE_MAX);
}
/**
* bpf_strspn - Calculate the length of the initial substring of @s__ign which
* only contains letters in @accept__ign
* @s__ign: The string to be searched
* @accept__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which only
* contains letters from @accept__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strspn(const char *s__ign, const char *accept__ign)
{
char cs, ca;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(accept__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&ca, accept__ign + j, char, err_out);
if (cs == ca || ca == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (ca == '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strcspn - Calculate the length of the initial substring of @s__ign which
* does not contain letters in @reject__ign
* @s__ign: The string to be searched
* @reject__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which does not
* contain letters from @reject__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcspn(const char *s__ign, const char *reject__ign)
{
char cs, cr;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(reject__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&cr, reject__ign + j, char, err_out);
if (cs == cr || cr == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (cr != '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
static int __bpf_strnstr(const char *s1, const char *s2, size_t len,
bool ignore_case)
{
char c1, c2;
int i, j;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
for (j = 0; i + j <= len && j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&c2, s2 + j, char, err_out);
if (c2 == '\0')
return i;
/*
* We allow reading an extra byte from s2 (note the
* `i + j <= len` above) to cover the case when s2 is
* a suffix of the first len chars of s1.
*/
if (i + j == len)
break;
__get_kernel_nofault(&c1, s1 + j, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 == '\0')
return -ENOENT;
if (c1 != c2)
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (i + j == len)
return -ENOENT;
s1++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strstr - Find the first substring in a string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strstr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, false);
}
/**
* bpf_strcasestr - Find the first substring in a string, ignoring the case of
* the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasestr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, true);
}
/**
* bpf_strnstr - Find the first substring in a length-limited string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strnstr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, false);
}
/**
* bpf_strncasestr - Find the first substring in a length-limited string,
* ignoring the case of the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasestr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, true);
}
#ifdef CONFIG_KEYS
/**
* bpf_lookup_user_key - lookup a key by its serial
* @serial: key handle serial number
* @flags: lookup-specific flags
*
* Search a key with a given *serial* and the provided *flags*.
* If found, increment the reference count of the key by one, and
* return it in the bpf_key structure.
*
* The bpf_key structure must be passed to bpf_key_put() when done
* with it, so that the key reference count is decremented and the
* bpf_key structure is freed.
*
* Permission checks are deferred to the time the key is used by
* one of the available key-specific kfuncs.
*
* Set *flags* with KEY_LOOKUP_CREATE, to attempt creating a requested
* special keyring (e.g. session keyring), if it doesn't yet exist.
* Set *flags* with KEY_LOOKUP_PARTIAL, to lookup a key without waiting
* for the key construction, and to retrieve uninstantiated keys (keys
* without data attached to them).
*
* Return: a bpf_key pointer with a valid key pointer if the key is found, a
* NULL pointer otherwise.
*/
__bpf_kfunc struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags)
{
key_ref_t key_ref;
struct bpf_key *bkey;
if (flags & ~KEY_LOOKUP_ALL)
return NULL;
/*
* Permission check is deferred until the key is used, as the
* intent of the caller is unknown here.
*/
key_ref = lookup_user_key(serial, flags, KEY_DEFER_PERM_CHECK);
if (IS_ERR(key_ref))
return NULL;
bkey = kmalloc_obj(*bkey);
if (!bkey) {
key_put(key_ref_to_ptr(key_ref));
return NULL;
}
bkey->key = key_ref_to_ptr(key_ref);
bkey->has_ref = true;
return bkey;
}
/**
* bpf_lookup_system_key - lookup a key by a system-defined ID
* @id: key ID
*
* Obtain a bpf_key structure with a key pointer set to the passed key ID.
* The key pointer is marked as invalid, to prevent bpf_key_put() from
* attempting to decrement the key reference count on that pointer. The key
* pointer set in such way is currently understood only by
* verify_pkcs7_signature().
*
* Set *id* to one of the values defined in include/linux/verification.h:
* 0 for the primary keyring (immutable keyring of system keys);
* VERIFY_USE_SECONDARY_KEYRING for both the primary and secondary keyring
* (where keys can be added only if they are vouched for by existing keys
* in those keyrings); VERIFY_USE_PLATFORM_KEYRING for the platform
* keyring (primarily used by the integrity subsystem to verify a kexec'ed
* kerned image and, possibly, the initramfs signature).
*
* Return: a bpf_key pointer with an invalid key pointer set from the
* pre-determined ID on success, a NULL pointer otherwise
*/
__bpf_kfunc struct bpf_key *bpf_lookup_system_key(u64 id)
{
struct bpf_key *bkey;
if (system_keyring_id_check(id) < 0)
return NULL;
bkey = kmalloc_obj(*bkey, GFP_ATOMIC);
if (!bkey)
return NULL;
bkey->key = (struct key *)(unsigned long)id;
bkey->has_ref = false;
return bkey;
}
/**
* bpf_key_put - decrement key reference count if key is valid and free bpf_key
* @bkey: bpf_key structure
*
* Decrement the reference count of the key inside *bkey*, if the pointer
* is valid, and free *bkey*.
*/
__bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
{
if (bkey->has_ref)
key_put(bkey->key);
kfree(bkey);
}
/**
* bpf_verify_pkcs7_signature - verify a PKCS#7 signature
* @data_p: data to verify
* @sig_p: signature of the data
* @trusted_keyring: keyring with keys trusted for signature verification
*
* Verify the PKCS#7 signature *sig_ptr* against the supplied *data_ptr*
* with keys in a keyring referenced by *trusted_keyring*.
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const void *data, *sig;
u32 data_len, sig_len;
int ret;
if (trusted_keyring->has_ref) {
/*
* Do the permission check deferred in bpf_lookup_user_key().
* See bpf_lookup_user_key() for more details.
*
* A call to key_task_permission() here would be redundant, as
* it is already done by keyring_search() called by
* find_asymmetric_key().
*/
ret = key_validate(trusted_keyring->key);
if (ret < 0)
return ret;
}
data_len = __bpf_dynptr_size(data_ptr);
data = __bpf_dynptr_data(data_ptr, data_len);
sig_len = __bpf_dynptr_size(sig_ptr);
sig = __bpf_dynptr_data(sig_ptr, sig_len);
return verify_pkcs7_signature(data, data_len, sig, sig_len,
trusted_keyring->key,
VERIFYING_BPF_SIGNATURE, NULL,
NULL);
#else
return -EOPNOTSUPP;
#endif /* CONFIG_SYSTEM_DATA_VERIFICATION */
}
#endif /* CONFIG_KEYS */
typedef int (*bpf_task_work_callback_t)(struct bpf_map *map, void *key, void *value);
enum bpf_task_work_state {
/* bpf_task_work is ready to be used */
BPF_TW_STANDBY = 0,
/* irq work scheduling in progress */
BPF_TW_PENDING,
/* task work scheduling in progress */
BPF_TW_SCHEDULING,
/* task work is scheduled successfully */
BPF_TW_SCHEDULED,
/* callback is running */
BPF_TW_RUNNING,
/* associated BPF map value is deleted */
BPF_TW_FREED,
};
struct bpf_task_work_ctx {
enum bpf_task_work_state state;
refcount_t refcnt;
struct callback_head work;
struct irq_work irq_work;
/* bpf_prog that schedules task work */
struct bpf_prog *prog;
/* task for which callback is scheduled */
struct task_struct *task;
/* the map and map value associated with this context */
struct bpf_map *map;
void *map_val;
enum task_work_notify_mode mode;
bpf_task_work_callback_t callback_fn;
struct rcu_head rcu;
} __aligned(8);
/* Actual type for struct bpf_task_work */
struct bpf_task_work_kern {
struct bpf_task_work_ctx *ctx;
};
static void bpf_task_work_ctx_reset(struct bpf_task_work_ctx *ctx)
{
if (ctx->prog) {
bpf_prog_put(ctx->prog);
ctx->prog = NULL;
}
if (ctx->task) {
bpf_task_release(ctx->task);
ctx->task = NULL;
}
}
static bool bpf_task_work_ctx_tryget(struct bpf_task_work_ctx *ctx)
{
return refcount_inc_not_zero(&ctx->refcnt);
}
static void bpf_task_work_destroy(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_ctx_reset(ctx);
kfree_rcu(ctx, rcu);
}
static void bpf_task_work_ctx_put(struct bpf_task_work_ctx *ctx)
{
if (!refcount_dec_and_test(&ctx->refcnt))
return;
if (irqs_disabled()) {
ctx->irq_work = IRQ_WORK_INIT(bpf_task_work_destroy);
irq_work_queue(&ctx->irq_work);
} else {
bpf_task_work_destroy(&ctx->irq_work);
}
}
static void bpf_task_work_cancel(struct bpf_task_work_ctx *ctx)
{
/*
* Scheduled task_work callback holds ctx ref, so if we successfully
* cancelled, we put that ref on callback's behalf. If we couldn't
* cancel, callback will inevitably run or has already completed
* running, and it would have taken care of its ctx ref itself.
*/
if (task_work_cancel(ctx->task, &ctx->work))
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_callback(struct callback_head *cb)
{
struct bpf_task_work_ctx *ctx = container_of(cb, struct bpf_task_work_ctx, work);
enum bpf_task_work_state state;
u32 idx;
void *key;
/* Read lock is needed to protect ctx and map key/value access */
guard(rcu_tasks_trace)();
/*
* This callback may start running before bpf_task_work_irq() switched to
* SCHEDULED state, so handle both transition variants SCHEDULING|SCHEDULED -> RUNNING.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_RUNNING);
if (state == BPF_TW_SCHEDULED)
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULED, BPF_TW_RUNNING);
if (state == BPF_TW_FREED) {
bpf_task_work_ctx_put(ctx);
return;
}
key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx);
migrate_disable();
ctx->callback_fn(ctx->map, key, ctx->map_val);
migrate_enable();
bpf_task_work_ctx_reset(ctx);
(void)cmpxchg(&ctx->state, BPF_TW_RUNNING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_irq(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
enum bpf_task_work_state state;
int err;
guard(rcu)();
if (cmpxchg(&ctx->state, BPF_TW_PENDING, BPF_TW_SCHEDULING) != BPF_TW_PENDING) {
bpf_task_work_ctx_put(ctx);
return;
}
err = task_work_add(ctx->task, &ctx->work, ctx->mode);
if (err) {
bpf_task_work_ctx_reset(ctx);
/*
* try to switch back to STANDBY for another task_work reuse, but we might have
* gone to FREED already, which is fine as we already cleaned up after ourselves
*/
(void)cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
return;
}
/*
* It's technically possible for just scheduled task_work callback to
* complete running by now, going SCHEDULING -> RUNNING and then
* dropping its ctx refcount. Instead of capturing an extra ref just
* to protect below ctx->state access, we rely on rcu_read_lock
* above to prevent kfree_rcu from freeing ctx before we return.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_SCHEDULED);
if (state == BPF_TW_FREED)
bpf_task_work_cancel(ctx); /* clean up if we switched into FREED state */
}
static struct bpf_task_work_ctx *bpf_task_work_fetch_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_kern *twk = (void *)tw;
struct bpf_task_work_ctx *ctx, *old_ctx;
ctx = READ_ONCE(twk->ctx);
if (ctx)
return ctx;
ctx = bpf_map_kmalloc_nolock(map, sizeof(*ctx), 0, NUMA_NO_NODE);
if (!ctx)
return ERR_PTR(-ENOMEM);
memset(ctx, 0, sizeof(*ctx));
refcount_set(&ctx->refcnt, 1); /* map's own ref */
ctx->state = BPF_TW_STANDBY;
old_ctx = cmpxchg(&twk->ctx, NULL, ctx);
if (old_ctx) {
/*
* tw->ctx is set by concurrent BPF program, release allocated
* memory and try to reuse already set context.
*/
kfree_nolock(ctx);
return old_ctx;
}
return ctx; /* Success */
}
static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_ctx *ctx;
/*
* Sleepable BPF programs hold rcu_read_lock_trace but not
* regular rcu_read_lock. Since kfree_rcu waits for regular
* RCU GP, the ctx can be freed while we're between reading
* the pointer and incrementing the refcount. Take regular
* rcu_read_lock to prevent kfree_rcu from freeing the ctx
* before we can tryget it.
*/
scoped_guard(rcu) {
ctx = bpf_task_work_fetch_ctx(tw, map);
if (IS_ERR(ctx))
return ctx;
/* try to get ref for task_work callback to hold */
if (!bpf_task_work_ctx_tryget(ctx))
return ERR_PTR(-EBUSY);
}
if (cmpxchg(&ctx->state, BPF_TW_STANDBY, BPF_TW_PENDING) != BPF_TW_STANDBY) {
/* lost acquiring race or map_release_uref() stole it from us, put ref and bail */
bpf_task_work_ctx_put(ctx);
return ERR_PTR(-EBUSY);
}
/*
* If no process or bpffs is holding a reference to the map, no new callbacks should be
* scheduled. This does not address any race or correctness issue, but rather is a policy
* choice: dropping user references should stop everything.
*/
if (!atomic64_read(&map->usercnt)) {
/* drop ref we just got for task_work callback itself */
bpf_task_work_ctx_put(ctx);
/* transfer map's ref into cancel_and_free() */
bpf_task_work_cancel_and_free(tw);
return ERR_PTR(-EBUSY);
}
return ctx;
}
static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw,
struct bpf_map *map, bpf_task_work_callback_t callback_fn,
struct bpf_prog_aux *aux, enum task_work_notify_mode mode)
{
struct bpf_prog *prog;
struct bpf_task_work_ctx *ctx;
int err;
BTF_TYPE_EMIT(struct bpf_task_work);
prog = bpf_prog_inc_not_zero(aux->prog);
if (IS_ERR(prog))
return -EBADF;
task = bpf_task_acquire(task);
if (!task) {
err = -EBADF;
goto release_prog;
}
ctx = bpf_task_work_acquire_ctx(tw, map);
if (IS_ERR(ctx)) {
err = PTR_ERR(ctx);
goto release_all;
}
ctx->task = task;
ctx->callback_fn = callback_fn;
ctx->prog = prog;
ctx->mode = mode;
ctx->map = map;
ctx->map_val = (void *)tw - map->record->task_work_off;
init_task_work(&ctx->work, bpf_task_work_callback);
init_irq_work(&ctx->irq_work, bpf_task_work_irq);
irq_work_queue(&ctx->irq_work);
return 0;
release_all:
bpf_task_release(task);
release_prog:
bpf_prog_put(prog);
return err;
}
/**
* bpf_task_work_schedule_signal - Schedule BPF callback using task_work_add with TWA_SIGNAL
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL);
}
/**
* bpf_task_work_schedule_resume - Schedule BPF callback using task_work_add with TWA_RESUME
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME);
}
static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
struct bpf_dynptr_kern *ptr)
{
struct bpf_dynptr_file_impl *state;
/* flags is currently unsupported */
if (flags) {
bpf_dynptr_set_null(ptr);
return -EINVAL;
}
state = kmalloc_nolock(sizeof(*state), 0, NUMA_NO_NODE);
if (!state) {
bpf_dynptr_set_null(ptr);
return -ENOMEM;
}
state->offset = 0;
state->size = U64_MAX; /* Don't restrict size, as file may change anyways */
freader_init_from_file(&state->freader, NULL, 0, file, may_sleep);
bpf_dynptr_init(ptr, state, BPF_DYNPTR_TYPE_FILE, 0, 0);
bpf_dynptr_set_rdonly(ptr);
return 0;
}
__bpf_kfunc int bpf_dynptr_from_file(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, false, (struct bpf_dynptr_kern *)ptr__uninit);
}
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, true, (struct bpf_dynptr_kern *)ptr__uninit);
}
__bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)dynptr;
struct bpf_dynptr_file_impl *df = ptr->data;
if (!df)
return 0;
freader_cleanup(&df->freader);
kfree_nolock(df);
bpf_dynptr_set_null(ptr);
return 0;
}
/**
* bpf_timer_cancel_async - try to deactivate a timer
* @timer: bpf_timer to stop
*
* Returns:
*
* * 0 when the timer was not active
* * 1 when the timer was active
* * -1 when the timer is currently executing the callback function and
* cannot be stopped
* * -ECANCELED when the timer will be cancelled asynchronously
* * -ENOMEM when out of memory
* * -EINVAL when the timer was not initialized
* * -ENOENT when this kfunc is racing with timer deletion
*/
__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer)
{
struct bpf_async_kern *async = (void *)timer;
struct bpf_async_cb *cb;
int ret;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
/*
* Unlike hrtimer_start() it's ok to synchronously call
* hrtimer_try_to_cancel() when refcnt reached zero, but deferring to
* irq_work is not, since irq callback may execute after RCU GP and
* cb could be freed at that time. Check for refcnt zero for
* consistency.
*/
if (!refcount_inc_not_zero(&cb->refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
ret = hrtimer_try_to_cancel(&t->timer);
bpf_async_refcount_put(cb);
return ret;
} else {
ret = bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
return ret ? ret : -ECANCELED;
}
}
__bpf_kfunc_end_defs();
static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_cancel(ctx); /* this might put task_work callback's ref */
bpf_task_work_ctx_put(ctx); /* and here we put map's own ref that was transferred to us */
}
void bpf_task_work_cancel_and_free(void *val)
{
struct bpf_task_work_kern *twk = val;
struct bpf_task_work_ctx *ctx;
enum bpf_task_work_state state;
ctx = xchg(&twk->ctx, NULL);
if (!ctx)
return;
state = xchg(&ctx->state, BPF_TW_FREED);
if (state == BPF_TW_SCHEDULED) {
/* run in irq_work to avoid locks in NMI */
init_irq_work(&ctx->irq_work, bpf_task_work_cancel_scheduled);
irq_work_queue(&ctx->irq_work);
return;
}
bpf_task_work_ctx_put(ctx); /* put bpf map's ref */
}
BTF_KFUNCS_START(generic_btf_ids)
#ifdef CONFIG_CRASH_DUMP
BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
#endif
BTF_ID_FLAGS(func, bpf_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_percpu_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_refcount_acquire, KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE | KF_RET_NULL | KF_RCU)
BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_front_impl)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_back_impl)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_rbtree_add_impl)
BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_cgroup_ancestor, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU)
BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
#endif
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_send_signal_task)
#endif
#ifdef CONFIG_KEYS
BTF_ID_FLAGS(func, bpf_lookup_user_key, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_lookup_system_key, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_key_put, KF_RELEASE)
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
BTF_ID_FLAGS(func, bpf_verify_pkcs7_signature, KF_SLEEPABLE)
#endif
#endif
#ifdef CONFIG_S390
BTF_ID_FLAGS(func, bpf_get_lowcore)
#endif
BTF_KFUNCS_END(generic_btf_ids)
static const struct btf_kfunc_id_set generic_kfunc_set = {
.owner = THIS_MODULE,
.set = &generic_btf_ids,
};
BTF_ID_LIST(generic_dtor_ids)
BTF_ID(struct, task_struct)
BTF_ID(func, bpf_task_release_dtor)
#ifdef CONFIG_CGROUPS
BTF_ID(struct, cgroup)
BTF_ID(func, bpf_cgroup_release_dtor)
#endif
BTF_KFUNCS_START(common_btf_ids)
BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rcu_read_lock)
BTF_ID_FLAGS(func, bpf_rcu_read_unlock)
BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_css_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_css_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_destroy, KF_ITER_DESTROY)
#endif
BTF_ID_FLAGS(func, bpf_iter_task_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_dynptr_adjust)
BTF_ID_FLAGS(func, bpf_dynptr_is_null)
BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
BTF_ID_FLAGS(func, bpf_dynptr_size)
BTF_ID_FLAGS(func, bpf_dynptr_clone)
BTF_ID_FLAGS(func, bpf_dynptr_copy)
BTF_ID_FLAGS(func, bpf_dynptr_memset)
#ifdef CONFIG_NET
BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
#endif
BTF_ID_FLAGS(func, bpf_wq_init)
BTF_ID_FLAGS(func, bpf_wq_set_callback, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_wq_start)
BTF_ID_FLAGS(func, bpf_preempt_disable)
BTF_ID_FLAGS(func, bpf_preempt_enable)
BTF_ID_FLAGS(func, bpf_iter_bits_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_bits_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_bits_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_copy_from_user_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_get_kmem_cache)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_local_irq_save)
BTF_ID_FLAGS(func, bpf_local_irq_restore)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr)
BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE)
#endif
#ifdef CONFIG_DMA_SHARED_BUFFER
BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
#endif
BTF_ID_FLAGS(func, __bpf_trap)
BTF_ID_FLAGS(func, bpf_strcmp);
BTF_ID_FLAGS(func, bpf_strcasecmp);
BTF_ID_FLAGS(func, bpf_strncasecmp);
BTF_ID_FLAGS(func, bpf_strchr);
BTF_ID_FLAGS(func, bpf_strchrnul);
BTF_ID_FLAGS(func, bpf_strnchr);
BTF_ID_FLAGS(func, bpf_strrchr);
BTF_ID_FLAGS(func, bpf_strlen);
BTF_ID_FLAGS(func, bpf_strnlen);
BTF_ID_FLAGS(func, bpf_strspn);
BTF_ID_FLAGS(func, bpf_strcspn);
BTF_ID_FLAGS(func, bpf_strstr);
BTF_ID_FLAGS(func, bpf_strcasestr);
BTF_ID_FLAGS(func, bpf_strnstr);
BTF_ID_FLAGS(func, bpf_strncasestr);
#if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS)
BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU)
#endif
BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_dynptr_from_file)
BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
BTF_ID_FLAGS(func, bpf_timer_cancel_async)
BTF_KFUNCS_END(common_btf_ids)
static const struct btf_kfunc_id_set common_kfunc_set = {
.owner = THIS_MODULE,
.set = &common_btf_ids,
};
static int __init kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc generic_dtors[] = {
{
.btf_id = generic_dtor_ids[0],
.kfunc_btf_id = generic_dtor_ids[1]
},
#ifdef CONFIG_CGROUPS
{
.btf_id = generic_dtor_ids[2],
.kfunc_btf_id = generic_dtor_ids[3]
},
#endif
};
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &generic_kfunc_set);
ret = ret ?: register_btf_id_dtor_kfuncs(generic_dtors,
ARRAY_SIZE(generic_dtors),
THIS_MODULE);
return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
}
late_initcall(kfunc_init);
/* Get a pointer to dynptr data up to len bytes for read only access. If
* the dynptr doesn't have continuous data up to len bytes, return NULL.
*/
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u64 len)
{
const struct bpf_dynptr *p = (struct bpf_dynptr *)ptr;
return bpf_dynptr_slice(p, 0, NULL, len);
}
/* Get a pointer to dynptr data up to len bytes for read write access. If
* the dynptr doesn't have continuous data up to len bytes, or the dynptr
* is read only, return NULL.
*/
void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u64 len)
{
if (__bpf_dynptr_is_rdonly(ptr))
return NULL;
return (void *)__bpf_dynptr_data(ptr, len);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *val)
{
if (btf_record_has_field(map->record, BPF_TIMER))
bpf_obj_free_timer(map->record, val);
if (btf_record_has_field(map->record, BPF_WORKQUEUE))
bpf_obj_free_workqueue(map->record, val);
if (btf_record_has_field(map->record, BPF_TASK_WORK))
bpf_obj_free_task_work(map->record, val);
}
]
|
| 465/6 |
2026/05/18 19:27 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 59448
output: 35
thoughts: 557
Thoughts: **My Analysis of the Code's String Formatting**
Alright, I've got the file loaded. Time to dig into these string formatting routines. First, I spotted two instances of `snprintf` in the `bpf_bprintf_prepare` function: Line 924 and Line 965. Then, of course, there's `bpf_snprintf` itself on Line 1073. No, wait, that's just the function declaration. I should look at the implementation.
The `bpf_snprintf` function's implementation reveals it's a wrapper, calling `bstr_printf` internally. Good, it's not a direct use of standard `snprintf`. I always pay close attention to string formatting, because these are classic attack vectors.
Next step: I did a quick check for other `sprintf`, `vsnprintf`, `scnprintf`, and `vscnprintf` calls within the file to make sure no other dangerous format strings are being used. No luck. I didn't see any of those. The only other formatting function I found, apart from the `snprintf` calls I already saw, is the `bstr_printf` function, which is used inside of `bpf_snprintf`.
Just to be absolutely certain, I'll run `grep` for `snprintf` again, just to double-check I didn't miss anything in my initial scan. Better safe than sorry when dealing with format strings.
|
| 466/6 |
2026/05/18 19:27 |
tool |
codesearch-find-references |
0m
Args: map[Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965]] TruncatedOutput:false]
|
| 467/6 |
2026/05/18 19:27 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 60481
output: 34
thoughts: 299
Thoughts: **My Analysis of `snprintf` Usage in `kernel/bpf/helpers.c`**
Okay, so the user specifically asked about other potentially unsafe uses of `snprintf` in `kernel/bpf/helpers.c`. My initial investigation using `codesearch-find-references` confirms there are *only* two instances of `snprintf` within that file, and they are both contained within the `bpf_bprintf_prepare` function. Therefore, the simple answer to their question, as stated, is "No," there are no *other* places within this specific file that directly use `snprintf` to format strings without proper return value checking.
However, I'm an expert, and I know that simply saying "No" isn't sufficient. The real question is about similar potential vulnerabilities, regardless of the function name. I must also consider if there are other functions that behave *similarly* to `snprintf` – functions that format strings into buffers and return the length (or potential length) of the output. I need to ensure there aren't any sneaky equivalent functions, perhaps ones called from `kernel/bpf/helpers.c`.
The initial search mentioned `bstr_printf`. I'll delve into how `bstr_printf` is used. Specifically, I'm looking at how it's called within `bpf_snprintf`:
```c
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
```
It looks like the return value of `bstr_printf` is used here, so that's good. But I need to check the definition of `bstr_printf` itself to confirm if it has the same return value contract as `snprintf` and if its return value handling is safe, and I'll see where it's defined and how it handles potential overflows. I'll need to check the code in `lib/vsprintf.c` and analyze the function's return value.
|
| 468/6 |
2026/05/18 19:27 |
tool |
codesearch-definition-comment |
0m
Args: map[ContextFile:lib/vsprintf.c Name:bstr_printf]
Results: map[Comment:/**
* bstr_printf - Format a string from binary arguments and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt_str: The format string to use
* @bin_buf: Binary arguments for the format string
*
* This function like C99 vsnprintf, but the difference is that vsnprintf gets
* arguments from stack, and bstr_printf gets arguments from @bin_buf which is
* a binary buffer that generated by vbin_printf.
*
* The format follows C99 vsnprintf, but has some extensions:
* see vsnprintf comment for details.
*
* The return value is the number of characters which would
* be generated for the given input, excluding the trailing
* '\0', as per ISO C99. If you want to have the exact
* number of characters written into @buf as return value
* (not including the trailing '\0'), use vscnprintf(). If the
* return is greater than or equal to @size, the resulting
* string is truncated.
*/
Kind:function]
|
| 469/6 |
2026/05/18 19:27 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 61538
output: 26
thoughts: 157
Thoughts: **Analyzing the bpf_snprintf Implementation**
Okay, so `bstr_printf` conveniently provides the count of characters that would be output. That's good to know. Now, looking at `bpf_snprintf`, I see the return value logic: `err = bstr_printf(...)`, then `return err + 1`. The +1 is potentially interesting. It looks like if `err` is greater than or equal to `str_size`, truncation happened. But, there is an *int* returned from `bpf_snprintf`, which is the BPF helper. I need to be sure that this is working the way I expect. Is this a potential issue? The documentation should have the answer. I will need to consult `include/uapi/linux/bpf.h` to see how that `bpf_snprintf` return value is *really* documented. I need to be precise here.
|
| 470/6 |
2026/05/18 19:27 |
tool |
read-file |
0m
Args: map[File:include/uapi/linux/bpf.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of version 2 of the GNU General Public
* License as published by the Free Software Foundation.
*/
#ifndef _UAPI__LINUX_BPF_H__
#define _UAPI__LINUX_BPF_H__
#include <linux/types.h>
#include <linux/bpf_common.h>
/* Extended instruction set based on top of classic BPF */
/* instruction classes */
#define BPF_JMP32 0x06 /* jmp mode in word width */
#define BPF_ALU64 0x07 /* alu mode in double word width */
/* ld/ldx fields */
#define BPF_DW 0x18 /* double word (64-bit) */
#define BPF_MEMSX 0x80 /* load with sign extension */
#define BPF_ATOMIC 0xc0 /* atomic memory ops - op type in immediate */
#define BPF_XADD 0xc0 /* exclusive add - legacy name */
/* alu/jmp fields */
#define BPF_MOV 0xb0 /* mov reg to reg */
#define BPF_ARSH 0xc0 /* sign extending arithmetic shift right */
/* change endianness of a register */
#define BPF_END 0xd0 /* flags for endianness conversion: */
#define BPF_TO_LE 0x00 /* convert to little-endian */
#define BPF_TO_BE 0x08 /* convert to big-endian */
#define BPF_FROM_LE BPF_TO_LE
#define BPF_FROM_BE BPF_TO_BE
/* jmp encodings */
#define BPF_JNE 0x50 /* jump != */
#define BPF_JLT 0xa0 /* LT is unsigned, '<' */
#define BPF_JLE 0xb0 /* LE is unsigned, '<=' */
#define BPF_JSGT 0x60 /* SGT is signed '>', GT in x86 */
#define BPF_JSGE 0x70 /* SGE is signed '>=', GE in x86 */
#define BPF_JSLT 0xc0 /* SLT is signed, '<' */
#define BPF_JSLE 0xd0 /* SLE is signed, '<=' */
#define BPF_JCOND 0xe0 /* conditional pseudo jumps: may_goto, goto_or_nop */
#define BPF_CALL 0x80 /* function call */
#define BPF_EXIT 0x90 /* function return */
/* atomic op type fields (stored in immediate) */
#define BPF_FETCH 0x01 /* not an opcode on its own, used to build others */
#define BPF_XCHG (0xe0 | BPF_FETCH) /* atomic exchange */
#define BPF_CMPXCHG (0xf0 | BPF_FETCH) /* atomic compare-and-write */
#define BPF_LOAD_ACQ 0x100 /* load-acquire */
#define BPF_STORE_REL 0x110 /* store-release */
enum bpf_cond_pseudo_jmp {
BPF_MAY_GOTO = 0,
};
/* Register numbers */
enum {
BPF_REG_0 = 0,
BPF_REG_1,
BPF_REG_2,
BPF_REG_3,
BPF_REG_4,
BPF_REG_5,
BPF_REG_6,
BPF_REG_7,
BPF_REG_8,
BPF_REG_9,
BPF_REG_10,
__MAX_BPF_REG,
};
/* BPF has 10 general purpose 64-bit registers and stack frame. */
#define MAX_BPF_REG __MAX_BPF_REG
struct bpf_insn {
__u8 code; /* opcode */
__u8 dst_reg:4; /* dest register */
__u8 src_reg:4; /* source register */
__s16 off; /* signed offset */
__s32 imm; /* signed immediate constant */
};
/* Deprecated: use struct bpf_lpm_trie_key_u8 (when the "data" member is needed for
* byte access) or struct bpf_lpm_trie_key_hdr (when using an alternative type for
* the trailing flexible array member) instead.
*/
struct bpf_lpm_trie_key {
__u32 prefixlen; /* up to 32 for AF_INET, 128 for AF_INET6 */
__u8 data[0]; /* Arbitrary size */
};
/* Header for bpf_lpm_trie_key structs */
struct bpf_lpm_trie_key_hdr {
__u32 prefixlen;
};
/* Key of an a BPF_MAP_TYPE_LPM_TRIE entry, with trailing byte array. */
struct bpf_lpm_trie_key_u8 {
union {
struct bpf_lpm_trie_key_hdr hdr;
__u32 prefixlen;
};
__u8 data[]; /* Arbitrary size */
};
struct bpf_cgroup_storage_key {
__u64 cgroup_inode_id; /* cgroup inode id */
__u32 attach_type; /* program attach type (enum bpf_attach_type) */
};
enum bpf_cgroup_iter_order {
BPF_CGROUP_ITER_ORDER_UNSPEC = 0,
BPF_CGROUP_ITER_SELF_ONLY, /* process only a single object. */
BPF_CGROUP_ITER_DESCENDANTS_PRE, /* walk descendants in pre-order. */
BPF_CGROUP_ITER_DESCENDANTS_POST, /* walk descendants in post-order. */
BPF_CGROUP_ITER_ANCESTORS_UP, /* walk ancestors upward. */
/*
* Walks the immediate children of the specified parent
* cgroup_subsys_state. Unlike BPF_CGROUP_ITER_DESCENDANTS_PRE,
* BPF_CGROUP_ITER_DESCENDANTS_POST, and BPF_CGROUP_ITER_ANCESTORS_UP
* the iterator does not include the specified parent as one of the
* returned iterator elements.
*/
BPF_CGROUP_ITER_CHILDREN,
};
union bpf_iter_link_info {
struct {
__u32 map_fd;
} map;
struct {
enum bpf_cgroup_iter_order order;
/* At most one of cgroup_fd and cgroup_id can be non-zero. If
* both are zero, the walk starts from the default cgroup v2
* root. For walking v1 hierarchy, one should always explicitly
* specify cgroup_fd.
*/
__u32 cgroup_fd;
__u64 cgroup_id;
} cgroup;
/* Parameters of task iterators. */
struct {
__u32 tid;
__u32 pid;
__u32 pid_fd;
} task;
};
/* BPF syscall commands, see bpf(2) man-page for more details. */
/**
* DOC: eBPF Syscall Preamble
*
* The operation to be performed by the **bpf**\ () system call is determined
* by the *cmd* argument. Each operation takes an accompanying argument,
* provided via *attr*, which is a pointer to a union of type *bpf_attr* (see
* below). The size argument is the size of the union pointed to by *attr*.
*/
/**
* DOC: eBPF Syscall Commands
*
* BPF_MAP_CREATE
* Description
* Create a map and return a file descriptor that refers to the
* map. The close-on-exec file descriptor flag (see **fcntl**\ (2))
* is automatically enabled for the new file descriptor.
*
* Applying **close**\ (2) to the file descriptor returned by
* **BPF_MAP_CREATE** will delete the map (but see NOTES).
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_MAP_LOOKUP_ELEM
* Description
* Look up an element with a given *key* in the map referred to
* by the file descriptor *map_fd*.
*
* The *flags* argument may be specified as one of the
* following:
*
* **BPF_F_LOCK**
* Look up the value of a spin-locked map without
* returning the lock. This must be specified if the
* elements contain a spinlock.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_MAP_UPDATE_ELEM
* Description
* Create or update an element (key/value pair) in a specified map.
*
* The *flags* argument should be specified as one of the
* following:
*
* **BPF_ANY**
* Create a new element or update an existing element.
* **BPF_NOEXIST**
* Create a new element only if it did not exist.
* **BPF_EXIST**
* Update an existing element.
* **BPF_F_LOCK**
* Update a spin_lock-ed map element.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* May set *errno* to **EINVAL**, **EPERM**, **ENOMEM**,
* **E2BIG**, **EEXIST**, or **ENOENT**.
*
* **E2BIG**
* The number of elements in the map reached the
* *max_entries* limit specified at map creation time.
* **EEXIST**
* If *flags* specifies **BPF_NOEXIST** and the element
* with *key* already exists in the map.
* **ENOENT**
* If *flags* specifies **BPF_EXIST** and the element with
* *key* does not exist in the map.
*
* BPF_MAP_DELETE_ELEM
* Description
* Look up and delete an element by key in a specified map.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_MAP_GET_NEXT_KEY
* Description
* Look up an element by key in a specified map and return the key
* of the next element. Can be used to iterate over all elements
* in the map.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* The following cases can be used to iterate over all elements of
* the map:
*
* * If *key* is not found, the operation returns zero and sets
* the *next_key* pointer to the key of the first element.
* * If *key* is found, the operation returns zero and sets the
* *next_key* pointer to the key of the next element.
* * If *key* is the last element, returns -1 and *errno* is set
* to **ENOENT**.
*
* May set *errno* to **ENOMEM**, **EFAULT**, **EPERM**, or
* **EINVAL** on error.
*
* BPF_PROG_LOAD
* Description
* Verify and load an eBPF program, returning a new file
* descriptor associated with the program.
*
* Applying **close**\ (2) to the file descriptor returned by
* **BPF_PROG_LOAD** will unload the eBPF program (but see NOTES).
*
* The close-on-exec file descriptor flag (see **fcntl**\ (2)) is
* automatically enabled for the new file descriptor.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_OBJ_PIN
* Description
* Pin an eBPF program or map referred by the specified *bpf_fd*
* to the provided *pathname* on the filesystem.
*
* The *pathname* argument must not contain a dot (".").
*
* On success, *pathname* retains a reference to the eBPF object,
* preventing deallocation of the object when the original
* *bpf_fd* is closed. This allow the eBPF object to live beyond
* **close**\ (\ *bpf_fd*\ ), and hence the lifetime of the parent
* process.
*
* Applying **unlink**\ (2) or similar calls to the *pathname*
* unpins the object from the filesystem, removing the reference.
* If no other file descriptors or filesystem nodes refer to the
* same object, it will be deallocated (see NOTES).
*
* The filesystem type for the parent directory of *pathname* must
* be **BPF_FS_MAGIC**.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_OBJ_GET
* Description
* Open a file descriptor for the eBPF object pinned to the
* specified *pathname*.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_PROG_ATTACH
* Description
* Attach an eBPF program to a *target_fd* at the specified
* *attach_type* hook.
*
* The *attach_type* specifies the eBPF attachment point to
* attach the program to, and must be one of *bpf_attach_type*
* (see below).
*
* The *attach_bpf_fd* must be a valid file descriptor for a
* loaded eBPF program of a cgroup, flow dissector, LIRC, sockmap
* or sock_ops type corresponding to the specified *attach_type*.
*
* The *target_fd* must be a valid file descriptor for a kernel
* object which depends on the attach type of *attach_bpf_fd*:
*
* **BPF_PROG_TYPE_CGROUP_DEVICE**,
* **BPF_PROG_TYPE_CGROUP_SKB**,
* **BPF_PROG_TYPE_CGROUP_SOCK**,
* **BPF_PROG_TYPE_CGROUP_SOCK_ADDR**,
* **BPF_PROG_TYPE_CGROUP_SOCKOPT**,
* **BPF_PROG_TYPE_CGROUP_SYSCTL**,
* **BPF_PROG_TYPE_SOCK_OPS**
*
* Control Group v2 hierarchy with the eBPF controller
* enabled. Requires the kernel to be compiled with
* **CONFIG_CGROUP_BPF**.
*
* **BPF_PROG_TYPE_FLOW_DISSECTOR**
*
* Network namespace (eg /proc/self/ns/net).
*
* **BPF_PROG_TYPE_LIRC_MODE2**
*
* LIRC device path (eg /dev/lircN). Requires the kernel
* to be compiled with **CONFIG_BPF_LIRC_MODE2**.
*
* **BPF_PROG_TYPE_SK_SKB**,
* **BPF_PROG_TYPE_SK_MSG**
*
* eBPF map of socket type (eg **BPF_MAP_TYPE_SOCKHASH**).
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_PROG_DETACH
* Description
* Detach the eBPF program associated with the *target_fd* at the
* hook specified by *attach_type*. The program must have been
* previously attached using **BPF_PROG_ATTACH**.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_PROG_TEST_RUN
* Description
* Run the eBPF program associated with the *prog_fd* a *repeat*
* number of times against a provided program context *ctx_in* and
* data *data_in*, and return the modified program context
* *ctx_out*, *data_out* (for example, packet data), result of the
* execution *retval*, and *duration* of the test run.
*
* The sizes of the buffers provided as input and output
* parameters *ctx_in*, *ctx_out*, *data_in*, and *data_out* must
* be provided in the corresponding variables *ctx_size_in*,
* *ctx_size_out*, *data_size_in*, and/or *data_size_out*. If any
* of these parameters are not provided (ie set to NULL), the
* corresponding size field must be zero.
*
* Some program types have particular requirements:
*
* **BPF_PROG_TYPE_SK_LOOKUP**
* *data_in* and *data_out* must be NULL.
*
* **BPF_PROG_TYPE_RAW_TRACEPOINT**,
* **BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE**
*
* *ctx_out*, *data_in* and *data_out* must be NULL.
* *repeat* must be zero.
*
* BPF_PROG_RUN is an alias for BPF_PROG_TEST_RUN.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* **ENOSPC**
* Either *data_size_out* or *ctx_size_out* is too small.
* **ENOTSUPP**
* This command is not supported by the program type of
* the program referred to by *prog_fd*.
*
* BPF_PROG_GET_NEXT_ID
* Description
* Fetch the next eBPF program currently loaded into the kernel.
*
* Looks for the eBPF program with an id greater than *start_id*
* and updates *next_id* on success. If no other eBPF programs
* remain with ids higher than *start_id*, returns -1 and sets
* *errno* to **ENOENT**.
*
* Return
* Returns zero on success. On error, or when no id remains, -1
* is returned and *errno* is set appropriately.
*
* BPF_MAP_GET_NEXT_ID
* Description
* Fetch the next eBPF map currently loaded into the kernel.
*
* Looks for the eBPF map with an id greater than *start_id*
* and updates *next_id* on success. If no other eBPF maps
* remain with ids higher than *start_id*, returns -1 and sets
* *errno* to **ENOENT**.
*
* Return
* Returns zero on success. On error, or when no id remains, -1
* is returned and *errno* is set appropriately.
*
* BPF_PROG_GET_FD_BY_ID
* Description
* Open a file descriptor for the eBPF program corresponding to
* *prog_id*.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_MAP_GET_FD_BY_ID
* Description
* Open a file descriptor for the eBPF map corresponding to
* *map_id*.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_OBJ_GET_INFO_BY_FD
* Description
* Obtain information about the eBPF object corresponding to
* *bpf_fd*.
*
* Populates up to *info_len* bytes of *info*, which will be in
* one of the following formats depending on the eBPF object type
* of *bpf_fd*:
*
* * **struct bpf_prog_info**
* * **struct bpf_map_info**
* * **struct bpf_btf_info**
* * **struct bpf_link_info**
* * **struct bpf_token_info**
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_PROG_QUERY
* Description
* Obtain information about eBPF programs associated with the
* specified *attach_type* hook.
*
* The *target_fd* must be a valid file descriptor for a kernel
* object which depends on the attach type of *attach_bpf_fd*:
*
* **BPF_PROG_TYPE_CGROUP_DEVICE**,
* **BPF_PROG_TYPE_CGROUP_SKB**,
* **BPF_PROG_TYPE_CGROUP_SOCK**,
* **BPF_PROG_TYPE_CGROUP_SOCK_ADDR**,
* **BPF_PROG_TYPE_CGROUP_SOCKOPT**,
* **BPF_PROG_TYPE_CGROUP_SYSCTL**,
* **BPF_PROG_TYPE_SOCK_OPS**
*
* Control Group v2 hierarchy with the eBPF controller
* enabled. Requires the kernel to be compiled with
* **CONFIG_CGROUP_BPF**.
*
* **BPF_PROG_TYPE_FLOW_DISSECTOR**
*
* Network namespace (eg /proc/self/ns/net).
*
* **BPF_PROG_TYPE_LIRC_MODE2**
*
* LIRC device path (eg /dev/lircN). Requires the kernel
* to be compiled with **CONFIG_BPF_LIRC_MODE2**.
*
* **BPF_PROG_QUERY** always fetches the number of programs
* attached and the *attach_flags* which were used to attach those
* programs. Additionally, if *prog_ids* is nonzero and the number
* of attached programs is less than *prog_cnt*, populates
* *prog_ids* with the eBPF program ids of the programs attached
* at *target_fd*.
*
* The following flags may alter the result:
*
* **BPF_F_QUERY_EFFECTIVE**
* Only return information regarding programs which are
* currently effective at the specified *target_fd*.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_RAW_TRACEPOINT_OPEN
* Description
* Attach an eBPF program to a tracepoint *name* to access kernel
* internal arguments of the tracepoint in their raw form.
*
* The *prog_fd* must be a valid file descriptor associated with
* a loaded eBPF program of type **BPF_PROG_TYPE_RAW_TRACEPOINT**.
*
* No ABI guarantees are made about the content of tracepoint
* arguments exposed to the corresponding eBPF program.
*
* Applying **close**\ (2) to the file descriptor returned by
* **BPF_RAW_TRACEPOINT_OPEN** will delete the map (but see NOTES).
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_BTF_LOAD
* Description
* Verify and load BPF Type Format (BTF) metadata into the kernel,
* returning a new file descriptor associated with the metadata.
* BTF is described in more detail at
* https://www.kernel.org/doc/html/latest/bpf/btf.html.
*
* The *btf* parameter must point to valid memory providing
* *btf_size* bytes of BTF binary metadata.
*
* The returned file descriptor can be passed to other **bpf**\ ()
* subcommands such as **BPF_PROG_LOAD** or **BPF_MAP_CREATE** to
* associate the BTF with those objects.
*
* Similar to **BPF_PROG_LOAD**, **BPF_BTF_LOAD** has optional
* parameters to specify a *btf_log_buf*, *btf_log_size* and
* *btf_log_level* which allow the kernel to return freeform log
* output regarding the BTF verification process.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_BTF_GET_FD_BY_ID
* Description
* Open a file descriptor for the BPF Type Format (BTF)
* corresponding to *btf_id*.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_TASK_FD_QUERY
* Description
* Obtain information about eBPF programs associated with the
* target process identified by *pid* and *fd*.
*
* If the *pid* and *fd* are associated with a tracepoint, kprobe
* or uprobe perf event, then the *prog_id* and *fd_type* will
* be populated with the eBPF program id and file descriptor type
* of type **bpf_task_fd_type**. If associated with a kprobe or
* uprobe, the *probe_offset* and *probe_addr* will also be
* populated. Optionally, if *buf* is provided, then up to
* *buf_len* bytes of *buf* will be populated with the name of
* the tracepoint, kprobe or uprobe.
*
* The resulting *prog_id* may be introspected in deeper detail
* using **BPF_PROG_GET_FD_BY_ID** and **BPF_OBJ_GET_INFO_BY_FD**.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_MAP_LOOKUP_AND_DELETE_ELEM
* Description
* Look up an element with the given *key* in the map referred to
* by the file descriptor *fd*, and if found, delete the element.
*
* For **BPF_MAP_TYPE_QUEUE** and **BPF_MAP_TYPE_STACK** map
* types, the *flags* argument needs to be set to 0, but for other
* map types, it may be specified as:
*
* **BPF_F_LOCK**
* Look up and delete the value of a spin-locked map
* without returning the lock. This must be specified if
* the elements contain a spinlock.
*
* The **BPF_MAP_TYPE_QUEUE** and **BPF_MAP_TYPE_STACK** map types
* implement this command as a "pop" operation, deleting the top
* element rather than one corresponding to *key*.
* The *key* and *key_len* parameters should be zeroed when
* issuing this operation for these map types.
*
* This command is only valid for the following map types:
* * **BPF_MAP_TYPE_QUEUE**
* * **BPF_MAP_TYPE_STACK**
* * **BPF_MAP_TYPE_HASH**
* * **BPF_MAP_TYPE_PERCPU_HASH**
* * **BPF_MAP_TYPE_LRU_HASH**
* * **BPF_MAP_TYPE_LRU_PERCPU_HASH**
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_MAP_FREEZE
* Description
* Freeze the permissions of the specified map.
*
* Write permissions may be frozen by passing zero *flags*.
* Upon success, no future syscall invocations may alter the
* map state of *map_fd*. Write operations from eBPF programs
* are still possible for a frozen map.
*
* Not supported for maps of type **BPF_MAP_TYPE_STRUCT_OPS**.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_BTF_GET_NEXT_ID
* Description
* Fetch the next BPF Type Format (BTF) object currently loaded
* into the kernel.
*
* Looks for the BTF object with an id greater than *start_id*
* and updates *next_id* on success. If no other BTF objects
* remain with ids higher than *start_id*, returns -1 and sets
* *errno* to **ENOENT**.
*
* Return
* Returns zero on success. On error, or when no id remains, -1
* is returned and *errno* is set appropriately.
*
* BPF_MAP_LOOKUP_BATCH
* Description
* Iterate and fetch multiple elements in a map.
*
* Two opaque values are used to manage batch operations,
* *in_batch* and *out_batch*. Initially, *in_batch* must be set
* to NULL to begin the batched operation. After each subsequent
* **BPF_MAP_LOOKUP_BATCH**, the caller should pass the resultant
* *out_batch* as the *in_batch* for the next operation to
* continue iteration from the current point. Both *in_batch* and
* *out_batch* must point to memory large enough to hold a key,
* except for maps of type **BPF_MAP_TYPE_{HASH, PERCPU_HASH,
* LRU_HASH, LRU_PERCPU_HASH}**, for which batch parameters
* must be at least 4 bytes wide regardless of key size.
*
* The *keys* and *values* are output parameters which must point
* to memory large enough to hold *count* items based on the key
* and value size of the map *map_fd*. The *keys* buffer must be
* of *key_size* * *count*. The *values* buffer must be of
* *value_size* * *count*.
*
* The *elem_flags* argument may be specified as one of the
* following:
*
* **BPF_F_LOCK**
* Look up the value of a spin-locked map without
* returning the lock. This must be specified if the
* elements contain a spinlock.
*
* On success, *count* elements from the map are copied into the
* user buffer, with the keys copied into *keys* and the values
* copied into the corresponding indices in *values*.
*
* If an error is returned and *errno* is not **EFAULT**, *count*
* is set to the number of successfully processed elements.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* May set *errno* to **ENOSPC** to indicate that *keys* or
* *values* is too small to dump an entire bucket during
* iteration of a hash-based map type.
*
* BPF_MAP_LOOKUP_AND_DELETE_BATCH
* Description
* Iterate and delete all elements in a map.
*
* This operation has the same behavior as
* **BPF_MAP_LOOKUP_BATCH** with two exceptions:
*
* * Every element that is successfully returned is also deleted
* from the map. This is at least *count* elements. Note that
* *count* is both an input and an output parameter.
* * Upon returning with *errno* set to **EFAULT**, up to
* *count* elements may be deleted without returning the keys
* and values of the deleted elements.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_MAP_UPDATE_BATCH
* Description
* Update multiple elements in a map by *key*.
*
* The *keys* and *values* are input parameters which must point
* to memory large enough to hold *count* items based on the key
* and value size of the map *map_fd*. The *keys* buffer must be
* of *key_size* * *count*. The *values* buffer must be of
* *value_size* * *count*.
*
* Each element specified in *keys* is sequentially updated to the
* value in the corresponding index in *values*. The *in_batch*
* and *out_batch* parameters are ignored and should be zeroed.
*
* The *elem_flags* argument should be specified as one of the
* following:
*
* **BPF_ANY**
* Create new elements or update a existing elements.
* **BPF_NOEXIST**
* Create new elements only if they do not exist.
* **BPF_EXIST**
* Update existing elements.
* **BPF_F_LOCK**
* Update spin_lock-ed map elements. This must be
* specified if the map value contains a spinlock.
*
* On success, *count* elements from the map are updated.
*
* If an error is returned and *errno* is not **EFAULT**, *count*
* is set to the number of successfully processed elements.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* May set *errno* to **EINVAL**, **EPERM**, **ENOMEM**, or
* **E2BIG**. **E2BIG** indicates that the number of elements in
* the map reached the *max_entries* limit specified at map
* creation time.
*
* May set *errno* to one of the following error codes under
* specific circumstances:
*
* **EEXIST**
* If *flags* specifies **BPF_NOEXIST** and the element
* with *key* already exists in the map.
* **ENOENT**
* If *flags* specifies **BPF_EXIST** and the element with
* *key* does not exist in the map.
*
* BPF_MAP_DELETE_BATCH
* Description
* Delete multiple elements in a map by *key*.
*
* The *keys* parameter is an input parameter which must point
* to memory large enough to hold *count* items based on the key
* size of the map *map_fd*, that is, *key_size* * *count*.
*
* Each element specified in *keys* is sequentially deleted. The
* *in_batch*, *out_batch*, and *values* parameters are ignored
* and should be zeroed.
*
* The *elem_flags* argument may be specified as one of the
* following:
*
* **BPF_F_LOCK**
* Look up the value of a spin-locked map without
* returning the lock. This must be specified if the
* elements contain a spinlock.
*
* On success, *count* elements from the map are updated.
*
* If an error is returned and *errno* is not **EFAULT**, *count*
* is set to the number of successfully processed elements. If
* *errno* is **EFAULT**, up to *count* elements may be been
* deleted.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_LINK_CREATE
* Description
* Attach an eBPF program to a *target_fd* at the specified
* *attach_type* hook and return a file descriptor handle for
* managing the link.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_LINK_UPDATE
* Description
* Update the eBPF program in the specified *link_fd* to
* *new_prog_fd*.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_LINK_GET_FD_BY_ID
* Description
* Open a file descriptor for the eBPF Link corresponding to
* *link_id*.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_LINK_GET_NEXT_ID
* Description
* Fetch the next eBPF link currently loaded into the kernel.
*
* Looks for the eBPF link with an id greater than *start_id*
* and updates *next_id* on success. If no other eBPF links
* remain with ids higher than *start_id*, returns -1 and sets
* *errno* to **ENOENT**.
*
* Return
* Returns zero on success. On error, or when no id remains, -1
* is returned and *errno* is set appropriately.
*
* BPF_ENABLE_STATS
* Description
* Enable eBPF runtime statistics gathering.
*
* Runtime statistics gathering for the eBPF runtime is disabled
* by default to minimize the corresponding performance overhead.
* This command enables statistics globally.
*
* Multiple programs may independently enable statistics.
* After gathering the desired statistics, eBPF runtime statistics
* may be disabled again by calling **close**\ (2) for the file
* descriptor returned by this function. Statistics will only be
* disabled system-wide when all outstanding file descriptors
* returned by prior calls for this subcommand are closed.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_ITER_CREATE
* Description
* Create an iterator on top of the specified *link_fd* (as
* previously created using **BPF_LINK_CREATE**) and return a
* file descriptor that can be used to trigger the iteration.
*
* If the resulting file descriptor is pinned to the filesystem
* using **BPF_OBJ_PIN**, then subsequent **read**\ (2) syscalls
* for that path will trigger the iterator to read kernel state
* using the eBPF program attached to *link_fd*.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_LINK_DETACH
* Description
* Forcefully detach the specified *link_fd* from its
* corresponding attachment point.
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_PROG_BIND_MAP
* Description
* Bind a map to the lifetime of an eBPF program.
*
* The map identified by *map_fd* is bound to the program
* identified by *prog_fd* and only released when *prog_fd* is
* released. This may be used in cases where metadata should be
* associated with a program which otherwise does not contain any
* references to the map (for example, embedded in the eBPF
* program instructions).
*
* Return
* Returns zero on success. On error, -1 is returned and *errno*
* is set appropriately.
*
* BPF_TOKEN_CREATE
* Description
* Create BPF token with embedded information about what
* BPF-related functionality it allows:
* - a set of allowed bpf() syscall commands;
* - a set of allowed BPF map types to be created with
* BPF_MAP_CREATE command, if BPF_MAP_CREATE itself is allowed;
* - a set of allowed BPF program types and BPF program attach
* types to be loaded with BPF_PROG_LOAD command, if
* BPF_PROG_LOAD itself is allowed.
*
* BPF token is created (derived) from an instance of BPF FS,
* assuming it has necessary delegation mount options specified.
* This BPF token can be passed as an extra parameter to various
* bpf() syscall commands to grant BPF subsystem functionality to
* unprivileged processes.
*
* When created, BPF token is "associated" with the owning
* user namespace of BPF FS instance (super block) that it was
* derived from, and subsequent BPF operations performed with
* BPF token would be performing capabilities checks (i.e.,
* CAP_BPF, CAP_PERFMON, CAP_NET_ADMIN, CAP_SYS_ADMIN) within
* that user namespace. Without BPF token, such capabilities
* have to be granted in init user namespace, making bpf()
* syscall incompatible with user namespace, for the most part.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_PROG_STREAM_READ_BY_FD
* Description
* Read data of a program's BPF stream. The program is identified
* by *prog_fd*, and the stream is identified by the *stream_id*.
* The data is copied to a buffer pointed to by *stream_buf*, and
* filled less than or equal to *stream_buf_len* bytes.
*
* Return
* Number of bytes read from the stream on success, or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* BPF_PROG_ASSOC_STRUCT_OPS
* Description
* Associate a BPF program with a struct_ops map. The struct_ops
* map is identified by *map_fd* and the BPF program is
* identified by *prog_fd*.
*
* Return
* 0 on success or -1 if an error occurred (in which case,
* *errno* is set appropriately).
*
* NOTES
* eBPF objects (maps and programs) can be shared between processes.
*
* * After **fork**\ (2), the child inherits file descriptors
* referring to the same eBPF objects.
* * File descriptors referring to eBPF objects can be transferred over
* **unix**\ (7) domain sockets.
* * File descriptors referring to eBPF objects can be duplicated in the
* usual way, using **dup**\ (2) and similar calls.
* * File descriptors referring to eBPF objects can be pinned to the
* filesystem using the **BPF_OBJ_PIN** command of **bpf**\ (2).
*
* An eBPF object is deallocated only after all file descriptors referring
* to the object have been closed and no references remain pinned to the
* filesystem or attached (for example, bound to a program or device).
*/
enum bpf_cmd {
BPF_MAP_CREATE,
BPF_MAP_LOOKUP_ELEM,
BPF_MAP_UPDATE_ELEM,
BPF_MAP_DELETE_ELEM,
BPF_MAP_GET_NEXT_KEY,
BPF_PROG_LOAD,
BPF_OBJ_PIN,
BPF_OBJ_GET,
BPF_PROG_ATTACH,
BPF_PROG_DETACH,
BPF_PROG_TEST_RUN,
BPF_PROG_RUN = BPF_PROG_TEST_RUN,
BPF_PROG_GET_NEXT_ID,
BPF_MAP_GET_NEXT_ID,
BPF_PROG_GET_FD_BY_ID,
BPF_MAP_GET_FD_BY_ID,
BPF_OBJ_GET_INFO_BY_FD,
BPF_PROG_QUERY,
BPF_RAW_TRACEPOINT_OPEN,
BPF_BTF_LOAD,
BPF_BTF_GET_FD_BY_ID,
BPF_TASK_FD_QUERY,
BPF_MAP_LOOKUP_AND_DELETE_ELEM,
BPF_MAP_FREEZE,
BPF_BTF_GET_NEXT_ID,
BPF_MAP_LOOKUP_BATCH,
BPF_MAP_LOOKUP_AND_DELETE_BATCH,
BPF_MAP_UPDATE_BATCH,
BPF_MAP_DELETE_BATCH,
BPF_LINK_CREATE,
BPF_LINK_UPDATE,
BPF_LINK_GET_FD_BY_ID,
BPF_LINK_GET_NEXT_ID,
BPF_ENABLE_STATS,
BPF_ITER_CREATE,
BPF_LINK_DETACH,
BPF_PROG_BIND_MAP,
BPF_TOKEN_CREATE,
BPF_PROG_STREAM_READ_BY_FD,
BPF_PROG_ASSOC_STRUCT_OPS,
__MAX_BPF_CMD,
};
enum bpf_map_type {
BPF_MAP_TYPE_UNSPEC,
BPF_MAP_TYPE_HASH,
BPF_MAP_TYPE_ARRAY,
BPF_MAP_TYPE_PROG_ARRAY,
BPF_MAP_TYPE_PERF_EVENT_ARRAY,
BPF_MAP_TYPE_PERCPU_HASH,
BPF_MAP_TYPE_PERCPU_ARRAY,
BPF_MAP_TYPE_STACK_TRACE,
BPF_MAP_TYPE_CGROUP_ARRAY,
BPF_MAP_TYPE_LRU_HASH,
BPF_MAP_TYPE_LRU_PERCPU_HASH,
BPF_MAP_TYPE_LPM_TRIE,
BPF_MAP_TYPE_ARRAY_OF_MAPS,
BPF_MAP_TYPE_HASH_OF_MAPS,
BPF_MAP_TYPE_DEVMAP,
BPF_MAP_TYPE_SOCKMAP,
BPF_MAP_TYPE_CPUMAP,
BPF_MAP_TYPE_XSKMAP,
BPF_MAP_TYPE_SOCKHASH,
BPF_MAP_TYPE_CGROUP_STORAGE_DEPRECATED,
/* BPF_MAP_TYPE_CGROUP_STORAGE is available to bpf programs attaching
* to a cgroup. The newer BPF_MAP_TYPE_CGRP_STORAGE is available to
* both cgroup-attached and other progs and supports all functionality
* provided by BPF_MAP_TYPE_CGROUP_STORAGE. So mark
* BPF_MAP_TYPE_CGROUP_STORAGE deprecated.
*/
BPF_MAP_TYPE_CGROUP_STORAGE = BPF_MAP_TYPE_CGROUP_STORAGE_DEPRECATED,
BPF_MAP_TYPE_REUSEPORT_SOCKARRAY,
BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE_DEPRECATED,
/* BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE is available to bpf programs
* attaching to a cgroup. The new mechanism (BPF_MAP_TYPE_CGRP_STORAGE +
* local percpu kptr) supports all BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE
* functionality and more. So mark * BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE
* deprecated.
*/
BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE = BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE_DEPRECATED,
BPF_MAP_TYPE_QUEUE,
BPF_MAP_TYPE_STACK,
BPF_MAP_TYPE_SK_STORAGE,
BPF_MAP_TYPE_DEVMAP_HASH,
BPF_MAP_TYPE_STRUCT_OPS,
BPF_MAP_TYPE_RINGBUF,
BPF_MAP_TYPE_INODE_STORAGE,
BPF_MAP_TYPE_TASK_STORAGE,
BPF_MAP_TYPE_BLOOM_FILTER,
BPF_MAP_TYPE_USER_RINGBUF,
BPF_MAP_TYPE_CGRP_STORAGE,
BPF_MAP_TYPE_ARENA,
BPF_MAP_TYPE_INSN_ARRAY,
__MAX_BPF_MAP_TYPE
};
/* Note that tracing related programs such as
* BPF_PROG_TYPE_{KPROBE,TRACEPOINT,PERF_EVENT,RAW_TRACEPOINT}
* are not subject to a stable API since kernel internal data
* structures can change from release to release and may
* therefore break existing tracing BPF programs. Tracing BPF
* programs correspond to /a/ specific kernel which is to be
* analyzed, and not /a/ specific kernel /and/ all future ones.
*/
enum bpf_prog_type {
BPF_PROG_TYPE_UNSPEC,
BPF_PROG_TYPE_SOCKET_FILTER,
BPF_PROG_TYPE_KPROBE,
BPF_PROG_TYPE_SCHED_CLS,
BPF_PROG_TYPE_SCHED_ACT,
BPF_PROG_TYPE_TRACEPOINT,
BPF_PROG_TYPE_XDP,
BPF_PROG_TYPE_PERF_EVENT,
BPF_PROG_TYPE_CGROUP_SKB,
BPF_PROG_TYPE_CGROUP_SOCK,
BPF_PROG_TYPE_LWT_IN,
BPF_PROG_TYPE_LWT_OUT,
BPF_PROG_TYPE_LWT_XMIT,
BPF_PROG_TYPE_SOCK_OPS,
BPF_PROG_TYPE_SK_SKB,
BPF_PROG_TYPE_CGROUP_DEVICE,
BPF_PROG_TYPE_SK_MSG,
BPF_PROG_TYPE_RAW_TRACEPOINT,
BPF_PROG_TYPE_CGROUP_SOCK_ADDR,
BPF_PROG_TYPE_LWT_SEG6LOCAL,
BPF_PROG_TYPE_LIRC_MODE2,
BPF_PROG_TYPE_SK_REUSEPORT,
BPF_PROG_TYPE_FLOW_DISSECTOR,
BPF_PROG_TYPE_CGROUP_SYSCTL,
BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE,
BPF_PROG_TYPE_CGROUP_SOCKOPT,
BPF_PROG_TYPE_TRACING,
BPF_PROG_TYPE_STRUCT_OPS,
BPF_PROG_TYPE_EXT,
BPF_PROG_TYPE_LSM,
BPF_PROG_TYPE_SK_LOOKUP,
BPF_PROG_TYPE_SYSCALL, /* a program that can execute syscalls */
BPF_PROG_TYPE_NETFILTER,
__MAX_BPF_PROG_TYPE
};
enum bpf_attach_type {
BPF_CGROUP_INET_INGRESS,
BPF_CGROUP_INET_EGRESS,
BPF_CGROUP_INET_SOCK_CREATE,
BPF_CGROUP_SOCK_OPS,
BPF_SK_SKB_STREAM_PARSER,
BPF_SK_SKB_STREAM_VERDICT,
BPF_CGROUP_DEVICE,
BPF_SK_MSG_VERDICT,
BPF_CGROUP_INET4_BIND,
BPF_CGROUP_INET6_BIND,
BPF_CGROUP_INET4_CONNECT,
BPF_CGROUP_INET6_CONNECT,
BPF_CGROUP_INET4_POST_BIND,
BPF_CGROUP_INET6_POST_BIND,
BPF_CGROUP_UDP4_SENDMSG,
BPF_CGROUP_UDP6_SENDMSG,
BPF_LIRC_MODE2,
BPF_FLOW_DISSECTOR,
BPF_CGROUP_SYSCTL,
BPF_CGROUP_UDP4_RECVMSG,
BPF_CGROUP_UDP6_RECVMSG,
BPF_CGROUP_GETSOCKOPT,
BPF_CGROUP_SETSOCKOPT,
BPF_TRACE_RAW_TP,
BPF_TRACE_FENTRY,
BPF_TRACE_FEXIT,
BPF_MODIFY_RETURN,
BPF_LSM_MAC,
BPF_TRACE_ITER,
BPF_CGROUP_INET4_GETPEERNAME,
BPF_CGROUP_INET6_GETPEERNAME,
BPF_CGROUP_INET4_GETSOCKNAME,
BPF_CGROUP_INET6_GETSOCKNAME,
BPF_XDP_DEVMAP,
BPF_CGROUP_INET_SOCK_RELEASE,
BPF_XDP_CPUMAP,
BPF_SK_LOOKUP,
BPF_XDP,
BPF_SK_SKB_VERDICT,
BPF_SK_REUSEPORT_SELECT,
BPF_SK_REUSEPORT_SELECT_OR_MIGRATE,
BPF_PERF_EVENT,
BPF_TRACE_KPROBE_MULTI,
BPF_LSM_CGROUP,
BPF_STRUCT_OPS,
BPF_NETFILTER,
BPF_TCX_INGRESS,
BPF_TCX_EGRESS,
BPF_TRACE_UPROBE_MULTI,
BPF_CGROUP_UNIX_CONNECT,
BPF_CGROUP_UNIX_SENDMSG,
BPF_CGROUP_UNIX_RECVMSG,
BPF_CGROUP_UNIX_GETPEERNAME,
BPF_CGROUP_UNIX_GETSOCKNAME,
BPF_NETKIT_PRIMARY,
BPF_NETKIT_PEER,
BPF_TRACE_KPROBE_SESSION,
BPF_TRACE_UPROBE_SESSION,
BPF_TRACE_FSESSION,
__MAX_BPF_ATTACH_TYPE
};
#define MAX_BPF_ATTACH_TYPE __MAX_BPF_ATTACH_TYPE
/* Add BPF_LINK_TYPE(type, name) in bpf_types.h to keep bpf_link_type_strs[]
* in sync with the definitions below.
*/
enum bpf_link_type {
BPF_LINK_TYPE_UNSPEC = 0,
BPF_LINK_TYPE_RAW_TRACEPOINT = 1,
BPF_LINK_TYPE_TRACING = 2,
BPF_LINK_TYPE_CGROUP = 3,
BPF_LINK_TYPE_ITER = 4,
BPF_LINK_TYPE_NETNS = 5,
BPF_LINK_TYPE_XDP = 6,
BPF_LINK_TYPE_PERF_EVENT = 7,
BPF_LINK_TYPE_KPROBE_MULTI = 8,
BPF_LINK_TYPE_STRUCT_OPS = 9,
BPF_LINK_TYPE_NETFILTER = 10,
BPF_LINK_TYPE_TCX = 11,
BPF_LINK_TYPE_UPROBE_MULTI = 12,
BPF_LINK_TYPE_NETKIT = 13,
BPF_LINK_TYPE_SOCKMAP = 14,
__MAX_BPF_LINK_TYPE,
};
#define MAX_BPF_LINK_TYPE __MAX_BPF_LINK_TYPE
enum bpf_perf_event_type {
BPF_PERF_EVENT_UNSPEC = 0,
BPF_PERF_EVENT_UPROBE = 1,
BPF_PERF_EVENT_URETPROBE = 2,
BPF_PERF_EVENT_KPROBE = 3,
BPF_PERF_EVENT_KRETPROBE = 4,
BPF_PERF_EVENT_TRACEPOINT = 5,
BPF_PERF_EVENT_EVENT = 6,
};
/* cgroup-bpf attach flags used in BPF_PROG_ATTACH command
*
* NONE(default): No further bpf programs allowed in the subtree.
*
* BPF_F_ALLOW_OVERRIDE: If a sub-cgroup installs some bpf program,
* the program in this cgroup yields to sub-cgroup program.
*
* BPF_F_ALLOW_MULTI: If a sub-cgroup installs some bpf program,
* that cgroup program gets run in addition to the program in this cgroup.
*
* Only one program is allowed to be attached to a cgroup with
* NONE or BPF_F_ALLOW_OVERRIDE flag.
* Attaching another program on top of NONE or BPF_F_ALLOW_OVERRIDE will
* release old program and attach the new one. Attach flags has to match.
*
* Multiple programs are allowed to be attached to a cgroup with
* BPF_F_ALLOW_MULTI flag. They are executed in FIFO order
* (those that were attached first, run first)
* The programs of sub-cgroup are executed first, then programs of
* this cgroup and then programs of parent cgroup.
* When children program makes decision (like picking TCP CA or sock bind)
* parent program has a chance to override it.
*
* With BPF_F_ALLOW_MULTI a new program is added to the end of the list of
* programs for a cgroup. Though it's possible to replace an old program at
* any position by also specifying BPF_F_REPLACE flag and position itself in
* replace_bpf_fd attribute. Old program at this position will be released.
*
* A cgroup with MULTI or OVERRIDE flag allows any attach flags in sub-cgroups.
* A cgroup with NONE doesn't allow any programs in sub-cgroups.
* Ex1:
* cgrp1 (MULTI progs A, B) ->
* cgrp2 (OVERRIDE prog C) ->
* cgrp3 (MULTI prog D) ->
* cgrp4 (OVERRIDE prog E) ->
* cgrp5 (NONE prog F)
* the event in cgrp5 triggers execution of F,D,A,B in that order.
* if prog F is detached, the execution is E,D,A,B
* if prog F and D are detached, the execution is E,A,B
* if prog F, E and D are detached, the execution is C,A,B
*
* All eligible programs are executed regardless of return code from
* earlier programs.
*/
#define BPF_F_ALLOW_OVERRIDE (1U << 0)
#define BPF_F_ALLOW_MULTI (1U << 1)
/* Generic attachment flags. */
#define BPF_F_REPLACE (1U << 2)
#define BPF_F_BEFORE (1U << 3)
#define BPF_F_AFTER (1U << 4)
#define BPF_F_ID (1U << 5)
#define BPF_F_PREORDER (1U << 6)
#define BPF_F_LINK BPF_F_LINK /* 1 << 13 */
/* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
* verifier will perform strict alignment checking as if the kernel
* has been built with CONFIG_EFFICIENT_UNALIGNED_ACCESS not set,
* and NET_IP_ALIGN defined to 2.
*/
#define BPF_F_STRICT_ALIGNMENT (1U << 0)
/* If BPF_F_ANY_ALIGNMENT is used in BPF_PROG_LOAD command, the
* verifier will allow any alignment whatsoever. On platforms
* with strict alignment requirements for loads ands stores (such
* as sparc and mips) the verifier validates that all loads and
* stores provably follow this requirement. This flag turns that
* checking and enforcement off.
*
* It is mostly used for testing when we want to validate the
* context and memory access aspects of the verifier, but because
* of an unaligned access the alignment check would trigger before
* the one we are interested in.
*/
#define BPF_F_ANY_ALIGNMENT (1U << 1)
/* BPF_F_TEST_RND_HI32 is used in BPF_PROG_LOAD command for testing purpose.
* Verifier does sub-register def/use analysis and identifies instructions whose
* def only matters for low 32-bit, high 32-bit is never referenced later
* through implicit zero extension. Therefore verifier notifies JIT back-ends
* that it is safe to ignore clearing high 32-bit for these instructions. This
* saves some back-ends a lot of code-gen. However such optimization is not
* necessary on some arches, for example x86_64, arm64 etc, whose JIT back-ends
* hence hasn't used verifier's analysis result. But, we really want to have a
* way to be able to verify the correctness of the described optimization on
* x86_64 on which testsuites are frequently exercised.
*
* So, this flag is introduced. Once it is set, verifier will randomize high
* 32-bit for those instructions who has been identified as safe to ignore them.
* Then, if verifier is not doing correct analysis, such randomization will
* regress tests to expose bugs.
*/
#define BPF_F_TEST_RND_HI32 (1U << 2)
/* The verifier internal test flag. Behavior is undefined */
#define BPF_F_TEST_STATE_FREQ (1U << 3)
/* If BPF_F_SLEEPABLE is used in BPF_PROG_LOAD command, the verifier will
* restrict map and helper usage for such programs. Sleepable BPF programs can
* only be attached to hooks where kernel execution context allows sleeping.
* Such programs are allowed to use helpers that may sleep like
* bpf_copy_from_user().
*/
#define BPF_F_SLEEPABLE (1U << 4)
/* If BPF_F_XDP_HAS_FRAGS is used in BPF_PROG_LOAD command, the loaded program
* fully support xdp frags.
*/
#define BPF_F_XDP_HAS_FRAGS (1U << 5)
/* If BPF_F_XDP_DEV_BOUND_ONLY is used in BPF_PROG_LOAD command, the loaded
* program becomes device-bound but can access XDP metadata.
*/
#define BPF_F_XDP_DEV_BOUND_ONLY (1U << 6)
/* The verifier internal test flag. Behavior is undefined */
#define BPF_F_TEST_REG_INVARIANTS (1U << 7)
/* link_create.kprobe_multi.flags used in LINK_CREATE command for
* BPF_TRACE_KPROBE_MULTI attach type to create return probe.
*/
enum {
BPF_F_KPROBE_MULTI_RETURN = (1U << 0)
};
/* link_create.uprobe_multi.flags used in LINK_CREATE command for
* BPF_TRACE_UPROBE_MULTI attach type to create return probe.
*/
enum {
BPF_F_UPROBE_MULTI_RETURN = (1U << 0)
};
/* link_create.netfilter.flags used in LINK_CREATE command for
* BPF_PROG_TYPE_NETFILTER to enable IP packet defragmentation.
*/
#define BPF_F_NETFILTER_IP_DEFRAG (1U << 0)
/* When BPF ldimm64's insn[0].src_reg != 0 then this can have
* the following extensions:
*
* insn[0].src_reg: BPF_PSEUDO_MAP_[FD|IDX]
* insn[0].imm: map fd or fd_idx
* insn[1].imm: 0
* insn[0].off: 0
* insn[1].off: 0
* ldimm64 rewrite: address of map
* verifier type: CONST_PTR_TO_MAP
*/
#define BPF_PSEUDO_MAP_FD 1
#define BPF_PSEUDO_MAP_IDX 5
/* insn[0].src_reg: BPF_PSEUDO_MAP_[IDX_]VALUE
* insn[0].imm: map fd or fd_idx
* insn[1].imm: offset into value
* insn[0].off: 0
* insn[1].off: 0
* ldimm64 rewrite: address of map[0]+offset
* verifier type: PTR_TO_MAP_VALUE
*/
#define BPF_PSEUDO_MAP_VALUE 2
#define BPF_PSEUDO_MAP_IDX_VALUE 6
/* insn[0].src_reg: BPF_PSEUDO_BTF_ID
* insn[0].imm: kernel btd id of VAR
* insn[1].imm: 0
* insn[0].off: 0
* insn[1].off: 0
* ldimm64 rewrite: address of the kernel variable
* verifier type: PTR_TO_BTF_ID or PTR_TO_MEM, depending on whether the var
* is struct/union.
*/
#define BPF_PSEUDO_BTF_ID 3
/* insn[0].src_reg: BPF_PSEUDO_FUNC
* insn[0].imm: insn offset to the func
* insn[1].imm: 0
* insn[0].off: 0
* insn[1].off: 0
* ldimm64 rewrite: address of the function
* verifier type: PTR_TO_FUNC.
*/
#define BPF_PSEUDO_FUNC 4
/* when bpf_call->src_reg == BPF_PSEUDO_CALL, bpf_call->imm == pc-relative
* offset to another bpf function
*/
#define BPF_PSEUDO_CALL 1
/* when bpf_call->src_reg == BPF_PSEUDO_KFUNC_CALL,
* bpf_call->imm == btf_id of a BTF_KIND_FUNC in the running kernel
*/
#define BPF_PSEUDO_KFUNC_CALL 2
enum bpf_addr_space_cast {
BPF_ADDR_SPACE_CAST = 1,
};
/* flags for BPF_MAP_UPDATE_ELEM command */
enum {
BPF_ANY = 0, /* create new element or update existing */
BPF_NOEXIST = 1, /* create new element if it didn't exist */
BPF_EXIST = 2, /* update existing element */
BPF_F_LOCK = 4, /* spin_lock-ed map_lookup/map_update */
BPF_F_CPU = 8, /* cpu flag for percpu maps, upper 32-bit of flags is a cpu number */
BPF_F_ALL_CPUS = 16, /* update value across all CPUs for percpu maps */
};
/* flags for BPF_MAP_CREATE command */
enum {
BPF_F_NO_PREALLOC = (1U << 0),
/* Instead of having one common LRU list in the
* BPF_MAP_TYPE_LRU_[PERCPU_]HASH map, use a percpu LRU list
* which can scale and perform better.
* Note, the LRU nodes (including free nodes) cannot be moved
* across different LRU lists.
*/
BPF_F_NO_COMMON_LRU = (1U << 1),
/* Specify numa node during map creation */
BPF_F_NUMA_NODE = (1U << 2),
/* Flags for accessing BPF object from syscall side. */
BPF_F_RDONLY = (1U << 3),
BPF_F_WRONLY = (1U << 4),
/* Flag for stack_map, store build_id+offset instead of pointer */
BPF_F_STACK_BUILD_ID = (1U << 5),
/* Zero-initialize hash function seed. This should only be used for testing. */
BPF_F_ZERO_SEED = (1U << 6),
/* Flags for accessing BPF object from program side. */
BPF_F_RDONLY_PROG = (1U << 7),
BPF_F_WRONLY_PROG = (1U << 8),
/* Clone map from listener for newly accepted socket */
BPF_F_CLONE = (1U << 9),
/* Enable memory-mapping BPF map */
BPF_F_MMAPABLE = (1U << 10),
/* Share perf_event among processes */
BPF_F_PRESERVE_ELEMS = (1U << 11),
/* Create a map that is suitable to be an inner map with dynamic max entries */
BPF_F_INNER_MAP = (1U << 12),
/* Create a map that will be registered/unregesitered by the backed bpf_link */
BPF_F_LINK = (1U << 13),
/* Get path from provided FD in BPF_OBJ_PIN/BPF_OBJ_GET commands */
BPF_F_PATH_FD = (1U << 14),
/* Flag for value_type_btf_obj_fd, the fd is available */
BPF_F_VTYPE_BTF_OBJ_FD = (1U << 15),
/* BPF token FD is passed in a corresponding command's token_fd field */
BPF_F_TOKEN_FD = (1U << 16),
/* When user space page faults in bpf_arena send SIGSEGV instead of inserting new page */
BPF_F_SEGV_ON_FAULT = (1U << 17),
/* Do not translate kernel bpf_arena pointers to user pointers */
BPF_F_NO_USER_CONV = (1U << 18),
/* Enable BPF ringbuf overwrite mode */
BPF_F_RB_OVERWRITE = (1U << 19),
};
/* Flags for BPF_PROG_QUERY. */
/* Query effective (directly attached + inherited from ancestor cgroups)
* programs that will be executed for events within a cgroup.
* attach_flags with this flag are always returned 0.
*/
#define BPF_F_QUERY_EFFECTIVE (1U << 0)
/* Flags for BPF_PROG_TEST_RUN */
/* If set, run the test on the cpu specified by bpf_attr.test.cpu */
#define BPF_F_TEST_RUN_ON_CPU (1U << 0)
/* If set, XDP frames will be transmitted after processing */
#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1)
/* If set, apply CHECKSUM_COMPLETE to skb and validate the checksum */
#define BPF_F_TEST_SKB_CHECKSUM_COMPLETE (1U << 2)
/* type for BPF_ENABLE_STATS */
enum bpf_stats_type {
/* enabled run_time_ns and run_cnt */
BPF_STATS_RUN_TIME = 0,
};
enum bpf_stack_build_id_status {
/* user space need an empty entry to identify end of a trace */
BPF_STACK_BUILD_ID_EMPTY = 0,
/* with valid build_id and offset */
BPF_STACK_BUILD_ID_VALID = 1,
/* couldn't get build_id, fallback to ip */
BPF_STACK_BUILD_ID_IP = 2,
};
#define BPF_BUILD_ID_SIZE 20
struct bpf_stack_build_id {
__s32 status;
unsigned char build_id[BPF_BUILD_ID_SIZE];
union {
__u64 offset;
__u64 ip;
};
};
#define BPF_OBJ_NAME_LEN 16U
enum {
BPF_STREAM_STDOUT = 1,
BPF_STREAM_STDERR = 2,
};
union bpf_attr {
struct { /* anonymous struct used by BPF_MAP_CREATE command */
__u32 map_type; /* one of enum bpf_map_type */
__u32 key_size; /* size of key in bytes */
__u32 value_size; /* size of value in bytes */
__u32 max_entries; /* max number of entries in a map */
__u32 map_flags; /* BPF_MAP_CREATE related
* flags defined above.
*/
__u32 inner_map_fd; /* fd pointing to the inner map */
__u32 numa_node; /* numa node (effective only if
* BPF_F_NUMA_NODE is set).
*/
char map_name[BPF_OBJ_NAME_LEN];
__u32 map_ifindex; /* ifindex of netdev to create on */
__u32 btf_fd; /* fd pointing to a BTF type data */
__u32 btf_key_type_id; /* BTF type_id of the key */
__u32 btf_value_type_id; /* BTF type_id of the value */
__u32 btf_vmlinux_value_type_id;/* BTF type_id of a kernel-
* struct stored as the
* map value
*/
/* Any per-map-type extra fields
*
* BPF_MAP_TYPE_BLOOM_FILTER - the lowest 4 bits indicate the
* number of hash functions (if 0, the bloom filter will default
* to using 5 hash functions).
*
* BPF_MAP_TYPE_ARENA - contains the address where user space
* is going to mmap() the arena. It has to be page aligned.
*/
__u64 map_extra;
__s32 value_type_btf_obj_fd; /* fd pointing to a BTF
* type data for
* btf_vmlinux_value_type_id.
*/
/* BPF token FD to use with BPF_MAP_CREATE operation.
* If provided, map_flags should have BPF_F_TOKEN_FD flag set.
*/
__s32 map_token_fd;
/* Hash of the program that has exclusive access to the map.
*/
__aligned_u64 excl_prog_hash;
/* Size of the passed excl_prog_hash. */
__u32 excl_prog_hash_size;
};
struct { /* anonymous struct used by BPF_MAP_*_ELEM and BPF_MAP_FREEZE commands */
__u32 map_fd;
__aligned_u64 key;
union {
__aligned_u64 value;
__aligned_u64 next_key;
};
__u64 flags;
};
struct { /* struct used by BPF_MAP_*_BATCH commands */
__aligned_u64 in_batch; /* start batch,
* NULL to start from beginning
*/
__aligned_u64 out_batch; /* output: next start batch */
__aligned_u64 keys;
__aligned_u64 values;
__u32 count; /* input/output:
* input: # of key/value
* elements
* output: # of filled elements
*/
__u32 map_fd;
__u64 elem_flags;
__u64 flags;
} batch;
struct { /* anonymous struct used by BPF_PROG_LOAD command */
__u32 prog_type; /* one of enum bpf_prog_type */
__u32 insn_cnt;
__aligned_u64 insns;
__aligned_u64 license;
__u32 log_level; /* verbosity level of verifier */
__u32 log_size; /* size of user buffer */
__aligned_u64 log_buf; /* user supplied buffer */
__u32 kern_version; /* not used */
__u32 prog_flags;
char prog_name[BPF_OBJ_NAME_LEN];
__u32 prog_ifindex; /* ifindex of netdev to prep for */
/* For some prog types expected attach type must be known at
* load time to verify attach type specific parts of prog
* (context accesses, allowed helpers, etc).
*/
__u32 expected_attach_type;
__u32 prog_btf_fd; /* fd pointing to BTF type data */
__u32 func_info_rec_size; /* userspace bpf_func_info size */
__aligned_u64 func_info; /* func info */
__u32 func_info_cnt; /* number of bpf_func_info records */
__u32 line_info_rec_size; /* userspace bpf_line_info size */
__aligned_u64 line_info; /* line info */
__u32 line_info_cnt; /* number of bpf_line_info records */
__u32 attach_btf_id; /* in-kernel BTF type id to attach to */
union {
/* valid prog_fd to attach to bpf prog */
__u32 attach_prog_fd;
/* or valid module BTF object fd or 0 to attach to vmlinux */
__u32 attach_btf_obj_fd;
};
__u32 core_relo_cnt; /* number of bpf_core_relo */
__aligned_u64 fd_array; /* array of FDs */
__aligned_u64 core_relos;
__u32 core_relo_rec_size; /* sizeof(struct bpf_core_relo) */
/* output: actual total log contents size (including termintaing zero).
* It could be both larger than original log_size (if log was
* truncated), or smaller (if log buffer wasn't filled completely).
*/
__u32 log_true_size;
/* BPF token FD to use with BPF_PROG_LOAD operation.
* If provided, prog_flags should have BPF_F_TOKEN_FD flag set.
*/
__s32 prog_token_fd;
/* The fd_array_cnt can be used to pass the length of the
* fd_array array. In this case all the [map] file descriptors
* passed in this array will be bound to the program, even if
* the maps are not referenced directly. The functionality is
* similar to the BPF_PROG_BIND_MAP syscall, but maps can be
* used by the verifier during the program load. If provided,
* then the fd_array[0,...,fd_array_cnt-1] is expected to be
* continuous.
*/
__u32 fd_array_cnt;
/* Pointer to a buffer containing the signature of the BPF
* program.
*/
__aligned_u64 signature;
/* Size of the signature buffer in bytes. */
__u32 signature_size;
/* ID of the kernel keyring to be used for signature
* verification.
*/
__s32 keyring_id;
};
struct { /* anonymous struct used by BPF_OBJ_* commands */
__aligned_u64 pathname;
__u32 bpf_fd;
__u32 file_flags;
/* Same as dirfd in openat() syscall; see openat(2)
* manpage for details of path FD and pathname semantics;
* path_fd should accompanied by BPF_F_PATH_FD flag set in
* file_flags field, otherwise it should be set to zero;
* if BPF_F_PATH_FD flag is not set, AT_FDCWD is assumed.
*/
__s32 path_fd;
};
struct { /* anonymous struct used by BPF_PROG_ATTACH/DETACH commands */
union {
__u32 target_fd; /* target object to attach to or ... */
__u32 target_ifindex; /* target ifindex */
};
__u32 attach_bpf_fd;
__u32 attach_type;
__u32 attach_flags;
__u32 replace_bpf_fd;
union {
__u32 relative_fd;
__u32 relative_id;
};
__u64 expected_revision;
};
struct { /* anonymous struct used by BPF_PROG_TEST_RUN command */
__u32 prog_fd;
__u32 retval;
__u32 data_size_in; /* input: len of data_in */
__u32 data_size_out; /* input/output: len of data_out
* returns ENOSPC if data_out
* is too small.
*/
__aligned_u64 data_in;
__aligned_u64 data_out;
__u32 repeat;
__u32 duration;
__u32 ctx_size_in; /* input: len of ctx_in */
__u32 ctx_size_out; /* input/output: len of ctx_out
* returns ENOSPC if ctx_out
* is too small.
*/
__aligned_u64 ctx_in;
__aligned_u64 ctx_out;
__u32 flags;
__u32 cpu;
__u32 batch_size;
} test;
struct { /* anonymous struct used by BPF_*_GET_*_ID */
union {
__u32 start_id;
__u32 prog_id;
__u32 map_id;
__u32 btf_id;
__u32 link_id;
};
__u32 next_id;
__u32 open_flags;
__s32 fd_by_id_token_fd;
};
struct { /* anonymous struct used by BPF_OBJ_GET_INFO_BY_FD */
__u32 bpf_fd;
__u32 info_len;
__aligned_u64 info;
} info;
struct { /* anonymous struct used by BPF_PROG_QUERY command */
union {
__u32 target_fd; /* target object to query or ... */
__u32 target_ifindex; /* target ifindex */
};
__u32 attach_type;
__u32 query_flags;
__u32 attach_flags;
__aligned_u64 prog_ids;
union {
__u32 prog_cnt;
__u32 count;
};
__u32 :32;
/* output: per-program attach_flags.
* not allowed to be set during effective query.
*/
__aligned_u64 prog_attach_flags;
__aligned_u64 link_ids;
__aligned_u64 link_attach_flags;
__u64 revision;
} query;
struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
__u64 name;
__u32 prog_fd;
__u32 :32;
__aligned_u64 cookie;
} raw_tracepoint;
struct { /* anonymous struct for BPF_BTF_LOAD */
__aligned_u64 btf;
__aligned_u64 btf_log_buf;
__u32 btf_size;
__u32 btf_log_size;
__u32 btf_log_level;
/* output: actual total log contents size (including termintaing zero).
* It could be both larger than original log_size (if log was
* truncated), or smaller (if log buffer wasn't filled completely).
*/
__u32 btf_log_true_size;
__u32 btf_flags;
/* BPF token FD to use with BPF_BTF_LOAD operation.
* If provided, btf_flags should have BPF_F_TOKEN_FD flag set.
*/
__s32 btf_token_fd;
};
struct {
__u32 pid; /* input: pid */
__u32 fd; /* input: fd */
__u32 flags; /* input: flags */
__u32 buf_len; /* input/output: buf len */
__aligned_u64 buf; /* input/output:
* tp_name for tracepoint
* symbol for kprobe
* filename for uprobe
*/
__u32 prog_id; /* output: prod_id */
__u32 fd_type; /* output: BPF_FD_TYPE_* */
__u64 probe_offset; /* output: probe_offset */
__u64 probe_addr; /* output: probe_addr */
} task_fd_query;
struct { /* struct used by BPF_LINK_CREATE command */
union {
__u32 prog_fd; /* eBPF program to attach */
__u32 map_fd; /* struct_ops to attach */
};
union {
__u32 target_fd; /* target object to attach to or ... */
__u32 target_ifindex; /* target ifindex */
};
__u32 attach_type; /* attach type */
__u32 flags; /* extra flags */
union {
__u32 target_btf_id; /* btf_id of target to attach to */
struct {
__aligned_u64 iter_info; /* extra bpf_iter_link_info */
__u32 iter_info_len; /* iter_info length */
};
struct {
/* black box user-provided value passed through
* to BPF program at the execution time and
* accessible through bpf_get_attach_cookie() BPF helper
*/
__u64 bpf_cookie;
} perf_event;
struct {
__u32 flags;
__u32 cnt;
__aligned_u64 syms;
__aligned_u64 addrs;
__aligned_u64 cookies;
} kprobe_multi;
struct {
/* this is overlaid with the target_btf_id above. */
__u32 target_btf_id;
/* black box user-provided value passed through
* to BPF program at the execution time and
* accessible through bpf_get_attach_cookie() BPF helper
*/
__u64 cookie;
} tracing;
struct {
__u32 pf;
__u32 hooknum;
__s32 priority;
__u32 flags;
} netfilter;
struct {
union {
__u32 relative_fd;
__u32 relative_id;
};
__u64 expected_revision;
} tcx;
struct {
__aligned_u64 path;
__aligned_u64 offsets;
__aligned_u64 ref_ctr_offsets;
__aligned_u64 cookies;
__u32 cnt;
__u32 flags;
__u32 pid;
} uprobe_multi;
struct {
union {
__u32 relative_fd;
__u32 relative_id;
};
__u64 expected_revision;
} netkit;
struct {
union {
__u32 relative_fd;
__u32 relative_id;
};
__u64 expected_revision;
} cgroup;
};
} link_create;
struct { /* struct used by BPF_LINK_UPDATE command */
__u32 link_fd; /* link fd */
union {
/* new program fd to update link with */
__u32 new_prog_fd;
/* new struct_ops map fd to update link with */
__u32 new_map_fd;
};
__u32 flags; /* extra flags */
union {
/* expected link's program fd; is specified only if
* BPF_F_REPLACE flag is set in flags.
*/
__u32 old_prog_fd;
/* expected link's map fd; is specified only
* if BPF_F_REPLACE flag is set.
*/
__u32 old_map_fd;
};
} link_update;
struct {
__u32 link_fd;
} link_detach;
struct { /* struct used by BPF_ENABLE_STATS command */
__u32 type;
} enable_stats;
struct { /* struct used by BPF_ITER_CREATE command */
__u32 link_fd;
__u32 flags;
} iter_create;
struct { /* struct used by BPF_PROG_BIND_MAP command */
__u32 prog_fd;
__u32 map_fd;
__u32 flags; /* extra flags */
} prog_bind_map;
struct { /* struct used by BPF_TOKEN_CREATE command */
__u32 flags;
__u32 bpffs_fd;
} token_create;
struct {
__aligned_u64 stream_buf;
__u32 stream_buf_len;
__u32 stream_id;
__u32 prog_fd;
} prog_stream_read;
struct {
__u32 map_fd;
__u32 prog_fd;
__u32 flags;
} prog_assoc_struct_ops;
} __attribute__((aligned(8)));
/* The description below is an attempt at providing documentation to eBPF
* developers about the multiple available eBPF helper functions. It can be
* parsed and used to produce a manual page. The workflow is the following,
* and requires the rst2man utility:
*
* $ ./scripts/bpf_doc.py \
* --filename include/uapi/linux/bpf.h > /tmp/bpf-helpers.rst
* $ rst2man /tmp/bpf-helpers.rst > /tmp/bpf-helpers.7
* $ man /tmp/bpf-helpers.7
*
* Note that in order to produce this external documentation, some RST
* formatting is used in the descriptions to get "bold" and "italics" in
* manual pages. Also note that the few trailing white spaces are
* intentional, removing them would break paragraphs for rst2man.
*
* Start of BPF helper function descriptions:
*
* void *bpf_map_lookup_elem(struct bpf_map *map, const void *key)
* Description
* Perform a lookup in *map* for an entry associated to *key*.
* Return
* Map value associated to *key*, or **NULL** if no entry was
* found.
*
* long bpf_map_update_elem(struct bpf_map *map, const void *key, const void *value, u64 flags)
* Description
* Add or update the value of the entry associated to *key* in
* *map* with *value*. *flags* is one of:
*
* **BPF_NOEXIST**
* The entry for *key* must not exist in the map.
* **BPF_EXIST**
* The entry for *key* must already exist in the map.
* **BPF_ANY**
* No condition on the existence of the entry for *key*.
*
* Flag value **BPF_NOEXIST** cannot be used for maps of types
* **BPF_MAP_TYPE_ARRAY** or **BPF_MAP_TYPE_PERCPU_ARRAY** (all
* elements always exist), the helper would return an error.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_map_delete_elem(struct bpf_map *map, const void *key)
* Description
* Delete entry with *key* from *map*.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_probe_read(void *dst, u32 size, const void *unsafe_ptr)
* Description
* For tracing programs, safely attempt to read *size* bytes from
* kernel space address *unsafe_ptr* and store the data in *dst*.
*
* Generally, use **bpf_probe_read_user**\ () or
* **bpf_probe_read_kernel**\ () instead.
* Return
* 0 on success, or a negative error in case of failure.
*
* u64 bpf_ktime_get_ns(void)
* Description
* Return the time elapsed since system boot, in nanoseconds.
* Does not include time the system was suspended.
* See: **clock_gettime**\ (**CLOCK_MONOTONIC**)
* Return
* Current *ktime*.
*
* long bpf_trace_printk(const char *fmt, u32 fmt_size, ...)
* Description
* This helper is a "printk()-like" facility for debugging. It
* prints a message defined by format *fmt* (of size *fmt_size*)
* to file *\/sys/kernel/tracing/trace* from TraceFS, if
* available. It can take up to three additional **u64**
* arguments (as an eBPF helpers, the total number of arguments is
* limited to five).
*
* Each time the helper is called, it appends a line to the trace.
* Lines are discarded while *\/sys/kernel/tracing/trace* is
* open, use *\/sys/kernel/tracing/trace_pipe* to avoid this.
* The format of the trace is customizable, and the exact output
* one will get depends on the options set in
* *\/sys/kernel/tracing/trace_options* (see also the
* *README* file under the same directory). However, it usually
* defaults to something like:
*
* ::
*
* telnet-470 [001] .N.. 419421.045894: 0x00000001: <formatted msg>
*
* In the above:
*
* * ``telnet`` is the name of the current task.
* * ``470`` is the PID of the current task.
* * ``001`` is the CPU number on which the task is
* running.
* * In ``.N..``, each character refers to a set of
* options (whether irqs are enabled, scheduling
* options, whether hard/softirqs are running, level of
* preempt_disabled respectively). **N** means that
* **TIF_NEED_RESCHED** and **PREEMPT_NEED_RESCHED**
* are set.
* * ``419421.045894`` is a timestamp.
* * ``0x00000001`` is a fake value used by BPF for the
* instruction pointer register.
* * ``<formatted msg>`` is the message formatted with
* *fmt*.
*
* The conversion specifiers supported by *fmt* are similar, but
* more limited than for printk(). They are **%d**, **%i**,
* **%u**, **%x**, **%ld**, **%li**, **%lu**, **%lx**, **%lld**,
* **%lli**, **%llu**, **%llx**, **%p**, **%s**. No modifier (size
* of field, padding with zeroes, etc.) is available, and the
* helper will return **-EINVAL** (but print nothing) if it
* encounters an unknown specifier.
*
* Also, note that **bpf_trace_printk**\ () is slow, and should
* only be used for debugging purposes. For this reason, a notice
* block (spanning several lines) is printed to kernel logs and
* states that the helper should not be used "for production use"
* the first time this helper is used (or more precisely, when
* **trace_printk**\ () buffers are allocated). For passing values
* to user space, perf events should be preferred.
* Return
* The number of bytes written to the buffer, or a negative error
* in case of failure.
*
* u32 bpf_get_prandom_u32(void)
* Description
* Get a pseudo-random number.
*
* From a security point of view, this helper uses its own
* pseudo-random internal state, and cannot be used to infer the
* seed of other random functions in the kernel. However, it is
* essential to note that the generator used by the helper is not
* cryptographically secure.
* Return
* A random 32-bit unsigned value.
*
* u32 bpf_get_smp_processor_id(void)
* Description
* Get the SMP (symmetric multiprocessing) processor id. Note that
* all programs run with migration disabled, which means that the
* SMP processor id is stable during all the execution of the
* program.
* Return
* The SMP id of the processor running the program.
* Attributes
* __bpf_fastcall
*
* long bpf_skb_store_bytes(struct sk_buff *skb, u32 offset, const void *from, u32 len, u64 flags)
* Description
* Store *len* bytes from address *from* into the packet
* associated to *skb*, at *offset*. The *flags* are a combination
* of the following values:
*
* **BPF_F_RECOMPUTE_CSUM**
* Automatically update *skb*\ **->csum** after storing the
* bytes.
* **BPF_F_INVALIDATE_HASH**
* Set *skb*\ **->hash**, *skb*\ **->swhash** and *skb*\
* **->l4hash** to 0.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_l3_csum_replace(struct sk_buff *skb, u32 offset, u64 from, u64 to, u64 size)
* Description
* Recompute the layer 3 (e.g. IP) checksum for the packet
* associated to *skb*. Computation is incremental, so the helper
* must know the former value of the header field that was
* modified (*from*), the new value of this field (*to*), and the
* number of bytes (2 or 4) for this field, stored in *size*.
* Alternatively, it is possible to store the difference between
* the previous and the new values of the header field in *to*, by
* setting *from* and *size* to 0. For both methods, *offset*
* indicates the location of the IP checksum within the packet.
*
* This helper works in combination with **bpf_csum_diff**\ (),
* which does not update the checksum in-place, but offers more
* flexibility and can handle sizes larger than 2 or 4 for the
* checksum to update.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_l4_csum_replace(struct sk_buff *skb, u32 offset, u64 from, u64 to, u64 flags)
* Description
* Recompute the layer 4 (e.g. TCP, UDP or ICMP) checksum for the
* packet associated to *skb*. Computation is incremental, so the
* helper must know the former value of the header field that was
* modified (*from*), the new value of this field (*to*), and the
* number of bytes (2 or 4) for this field, stored on the lowest
* four bits of *flags*. Alternatively, it is possible to store
* the difference between the previous and the new values of the
* header field in *to*, by setting *from* and the four lowest
* bits of *flags* to 0. For both methods, *offset* indicates the
* location of the IP checksum within the packet. In addition to
* the size of the field, *flags* can be added (bitwise OR) actual
* flags. With **BPF_F_MARK_MANGLED_0**, a null checksum is left
* untouched (unless **BPF_F_MARK_ENFORCE** is added as well), and
* for updates resulting in a null checksum the value is set to
* **CSUM_MANGLED_0** instead. Flag **BPF_F_PSEUDO_HDR** indicates
* that the modified header field is part of the pseudo-header.
* Flag **BPF_F_IPV6** should be set for IPv6 packets.
*
* This helper works in combination with **bpf_csum_diff**\ (),
* which does not update the checksum in-place, but offers more
* flexibility and can handle sizes larger than 2 or 4 for the
* checksum to update.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_tail_call(void *ctx, struct bpf_map *prog_array_map, u32 index)
* Description
* This special helper is used to trigger a "tail call", or in
* other words, to jump into another eBPF program. The same stack
* frame is used (but values on stack and in registers for the
* caller are not accessible to the callee). This mechanism allows
* for program chaining, either for raising the maximum number of
* available eBPF instructions, or to execute given programs in
* conditional blocks. For security reasons, there is an upper
* limit to the number of successive tail calls that can be
* performed.
*
* Upon call of this helper, the program attempts to jump into a
* program referenced at index *index* in *prog_array_map*, a
* special map of type **BPF_MAP_TYPE_PROG_ARRAY**, and passes
* *ctx*, a pointer to the context.
*
* If the call succeeds, the kernel immediately runs the first
* instruction of the new program. This is not a function call,
* and it never returns to the previous program. If the call
* fails, then the helper has no effect, and the caller continues
* to run its subsequent instructions. A call can fail if the
* destination program for the jump does not exist (i.e. *index*
* is superior to the number of entries in *prog_array_map*), or
* if the maximum number of tail calls has been reached for this
* chain of programs. This limit is defined in the kernel by the
* macro **MAX_TAIL_CALL_CNT** (not accessible to user space),
* which is currently set to 33.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_clone_redirect(struct sk_buff *skb, u32 ifindex, u64 flags)
* Description
* Clone and redirect the packet associated to *skb* to another
* net device of index *ifindex*. Both ingress and egress
* interfaces can be used for redirection. The **BPF_F_INGRESS**
* value in *flags* is used to make the distinction (ingress path
* is selected if the flag is present, egress path otherwise).
* This is the only flag supported for now.
*
* In comparison with **bpf_redirect**\ () helper,
* **bpf_clone_redirect**\ () has the associated cost of
* duplicating the packet buffer, but this can be executed out of
* the eBPF program. Conversely, **bpf_redirect**\ () is more
* efficient, but it is handled through an action code where the
* redirection happens only after the eBPF program has returned.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure. Positive
* error indicates a potential drop or congestion in the target
* device. The particular positive error codes are not defined.
*
* u64 bpf_get_current_pid_tgid(void)
* Description
* Get the current pid and tgid.
* Return
* A 64-bit integer containing the current tgid and pid, and
* created as such:
* *current_task*\ **->tgid << 32 \|**
* *current_task*\ **->pid**.
*
* u64 bpf_get_current_uid_gid(void)
* Description
* Get the current uid and gid.
* Return
* A 64-bit integer containing the current GID and UID, and
* created as such: *current_gid* **<< 32 \|** *current_uid*.
*
* long bpf_get_current_comm(void *buf, u32 size_of_buf)
* Description
* Copy the **comm** attribute of the current task into *buf* of
* *size_of_buf*. The **comm** attribute contains the name of
* the executable (excluding the path) for the current task. The
* *size_of_buf* must be strictly positive. On success, the
* helper makes sure that the *buf* is NUL-terminated. On failure,
* it is filled with zeroes.
* Return
* 0 on success, or a negative error in case of failure.
*
* u32 bpf_get_cgroup_classid(struct sk_buff *skb)
* Description
* Retrieve the classid for the current task, i.e. for the net_cls
* cgroup to which *skb* belongs.
*
* This helper can be used on TC egress path, but not on ingress.
*
* The net_cls cgroup provides an interface to tag network packets
* based on a user-provided identifier for all traffic coming from
* the tasks belonging to the related cgroup. See also the related
* kernel documentation, available from the Linux sources in file
* *Documentation/admin-guide/cgroup-v1/net_cls.rst*.
*
* The Linux kernel has two versions for cgroups: there are
* cgroups v1 and cgroups v2. Both are available to users, who can
* use a mixture of them, but note that the net_cls cgroup is for
* cgroup v1 only. This makes it incompatible with BPF programs
* run on cgroups, which is a cgroup-v2-only feature (a socket can
* only hold data for one version of cgroups at a time).
*
* This helper is only available is the kernel was compiled with
* the **CONFIG_CGROUP_NET_CLASSID** configuration option set to
* "**y**" or to "**m**".
* Return
* The classid, or 0 for the default unconfigured classid.
*
* long bpf_skb_vlan_push(struct sk_buff *skb, __be16 vlan_proto, u16 vlan_tci)
* Description
* Push a *vlan_tci* (VLAN tag control information) of protocol
* *vlan_proto* to the packet associated to *skb*, then update
* the checksum. Note that if *vlan_proto* is different from
* **ETH_P_8021Q** and **ETH_P_8021AD**, it is considered to
* be **ETH_P_8021Q**.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_skb_vlan_pop(struct sk_buff *skb)
* Description
* Pop a VLAN header from the packet associated to *skb*.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_skb_get_tunnel_key(struct sk_buff *skb, struct bpf_tunnel_key *key, u32 size, u64 flags)
* Description
* Get tunnel metadata. This helper takes a pointer *key* to an
* empty **struct bpf_tunnel_key** of **size**, that will be
* filled with tunnel metadata for the packet associated to *skb*.
* The *flags* can be set to **BPF_F_TUNINFO_IPV6**, which
* indicates that the tunnel is based on IPv6 protocol instead of
* IPv4.
*
* The **struct bpf_tunnel_key** is an object that generalizes the
* principal parameters used by various tunneling protocols into a
* single struct. This way, it can be used to easily make a
* decision based on the contents of the encapsulation header,
* "summarized" in this struct. In particular, it holds the IP
* address of the remote end (IPv4 or IPv6, depending on the case)
* in *key*\ **->remote_ipv4** or *key*\ **->remote_ipv6**. Also,
* this struct exposes the *key*\ **->tunnel_id**, which is
* generally mapped to a VNI (Virtual Network Identifier), making
* it programmable together with the **bpf_skb_set_tunnel_key**\
* () helper.
*
* Let's imagine that the following code is part of a program
* attached to the TC ingress interface, on one end of a GRE
* tunnel, and is supposed to filter out all messages coming from
* remote ends with IPv4 address other than 10.0.0.1:
*
* ::
*
* int ret;
* struct bpf_tunnel_key key = {};
*
* ret = bpf_skb_get_tunnel_key(skb, &key, sizeof(key), 0);
* if (ret < 0)
* return TC_ACT_SHOT; // drop packet
*
* if (key.remote_ipv4 != 0x0a000001)
* return TC_ACT_SHOT; // drop packet
*
* return TC_ACT_OK; // accept packet
*
* This interface can also be used with all encapsulation devices
* that can operate in "collect metadata" mode: instead of having
* one network device per specific configuration, the "collect
* metadata" mode only requires a single device where the
* configuration can be extracted from this helper.
*
* This can be used together with various tunnels such as VXLan,
* Geneve, GRE or IP in IP (IPIP).
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_skb_set_tunnel_key(struct sk_buff *skb, struct bpf_tunnel_key *key, u32 size, u64 flags)
* Description
* Populate tunnel metadata for packet associated to *skb.* The
* tunnel metadata is set to the contents of *key*, of *size*. The
* *flags* can be set to a combination of the following values:
*
* **BPF_F_TUNINFO_IPV6**
* Indicate that the tunnel is based on IPv6 protocol
* instead of IPv4.
* **BPF_F_ZERO_CSUM_TX**
* For IPv4 packets, add a flag to tunnel metadata
* indicating that checksum computation should be skipped
* and checksum set to zeroes.
* **BPF_F_DONT_FRAGMENT**
* Add a flag to tunnel metadata indicating that the
* packet should not be fragmented.
* **BPF_F_SEQ_NUMBER**
* Add a flag to tunnel metadata indicating that a
* sequence number should be added to tunnel header before
* sending the packet. This flag was added for GRE
* encapsulation, but might be used with other protocols
* as well in the future.
* **BPF_F_NO_TUNNEL_KEY**
* Add a flag to tunnel metadata indicating that no tunnel
* key should be set in the resulting tunnel header.
*
* Here is a typical usage on the transmit path:
*
* ::
*
* struct bpf_tunnel_key key;
* populate key ...
* bpf_skb_set_tunnel_key(skb, &key, sizeof(key), 0);
* bpf_clone_redirect(skb, vxlan_dev_ifindex, 0);
*
* See also the description of the **bpf_skb_get_tunnel_key**\ ()
* helper for additional information.
* Return
* 0 on success, or a negative error in case of failure.
*
* u64 bpf_perf_event_read(struct bpf_map *map, u64 flags)
* Description
* Read the value of a perf event counter. This helper relies on a
* *map* of type **BPF_MAP_TYPE_PERF_EVENT_ARRAY**. The nature of
* the perf event counter is selected when *map* is updated with
* perf event file descriptors. The *map* is an array whose size
* is the number of available CPUs, and each cell contains a value
* relative to one CPU. The value to retrieve is indicated by
* *flags*, that contains the index of the CPU to look up, masked
* with **BPF_F_INDEX_MASK**. Alternatively, *flags* can be set to
* **BPF_F_CURRENT_CPU** to indicate that the value for the
* current CPU should be retrieved.
*
* Note that before Linux 4.13, only hardware perf event can be
* retrieved.
*
* Also, be aware that the newer helper
* **bpf_perf_event_read_value**\ () is recommended over
* **bpf_perf_event_read**\ () in general. The latter has some ABI
* quirks where error and counter value are used as a return code
* (which is wrong to do since ranges may overlap). This issue is
* fixed with **bpf_perf_event_read_value**\ (), which at the same
* time provides more features over the **bpf_perf_event_read**\
* () interface. Please refer to the description of
* **bpf_perf_event_read_value**\ () for details.
* Return
* The value of the perf event counter read from the map, or a
* negative error code in case of failure.
*
* long bpf_redirect(u32 ifindex, u64 flags)
* Description
* Redirect the packet to another net device of index *ifindex*.
* This helper is somewhat similar to **bpf_clone_redirect**\
* (), except that the packet is not cloned, which provides
* increased performance.
*
* Except for XDP, both ingress and egress interfaces can be used
* for redirection. The **BPF_F_INGRESS** value in *flags* is used
* to make the distinction (ingress path is selected if the flag
* is present, egress path otherwise). Currently, XDP only
* supports redirection to the egress interface, and accepts no
* flag at all.
*
* The same effect can also be attained with the more generic
* **bpf_redirect_map**\ (), which uses a BPF map to store the
* redirect target instead of providing it directly to the helper.
* Return
* For XDP, the helper returns **XDP_REDIRECT** on success or
* **XDP_ABORTED** on error. For other program types, the values
* are **TC_ACT_REDIRECT** on success or **TC_ACT_SHOT** on
* error.
*
* u32 bpf_get_route_realm(struct sk_buff *skb)
* Description
* Retrieve the realm or the route, that is to say the
* **tclassid** field of the destination for the *skb*. The
* identifier retrieved is a user-provided tag, similar to the
* one used with the net_cls cgroup (see description for
* **bpf_get_cgroup_classid**\ () helper), but here this tag is
* held by a route (a destination entry), not by a task.
*
* Retrieving this identifier works with the clsact TC egress hook
* (see also **tc-bpf(8)**), or alternatively on conventional
* classful egress qdiscs, but not on TC ingress path. In case of
* clsact TC egress hook, this has the advantage that, internally,
* the destination entry has not been dropped yet in the transmit
* path. Therefore, the destination entry does not need to be
* artificially held via **netif_keep_dst**\ () for a classful
* qdisc until the *skb* is freed.
*
* This helper is available only if the kernel was compiled with
* **CONFIG_IP_ROUTE_CLASSID** configuration option.
* Return
* The realm of the route for the packet associated to *skb*, or 0
* if none was found.
*
* long bpf_perf_event_output(void *ctx, struct bpf_map *map, u64 flags, void *data, u64 size)
* Description
* Write raw *data* blob into a special BPF perf event held by
* *map* of type **BPF_MAP_TYPE_PERF_EVENT_ARRAY**. This perf
* event must have the following attributes: **PERF_SAMPLE_RAW**
* as **sample_type**, **PERF_TYPE_SOFTWARE** as **type**, and
* **PERF_COUNT_SW_BPF_OUTPUT** as **config**.
*
* The *flags* are used to indicate the index in *map* for which
* the value must be put, masked with **BPF_F_INDEX_MASK**.
* Alternatively, *flags* can be set to **BPF_F_CURRENT_CPU**
* to indicate that the index of the current CPU core should be
* used.
*
* The value to write, of *size*, is passed through eBPF stack and
* pointed by *data*.
*
* The context of the program *ctx* needs also be passed to the
* helper.
*
* On user space, a program willing to read the values needs to
* call **perf_event_open**\ () on the perf event (either for
* one or for all CPUs) and to store the file descriptor into the
* *map*. This must be done before the eBPF program can send data
* into it. An example is available in file
* *samples/bpf/trace_output_user.c* in the Linux kernel source
* tree (the eBPF program counterpart is in
* *samples/bpf/trace_output.bpf.c*).
*
* **bpf_perf_event_output**\ () achieves better performance
* than **bpf_trace_printk**\ () for sharing data with user
* space, and is much better suitable for streaming data from eBPF
* programs.
*
* Note that this helper is not restricted to tracing use cases
* and can be used with programs attached to TC or XDP as well,
* where it allows for passing data to user space listeners. Data
* can be:
*
* * Only custom structs,
* * Only the packet payload, or
* * A combination of both.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_skb_load_bytes(const void *skb, u32 offset, void *to, u32 len)
* Description
* This helper was provided as an easy way to load data from a
* packet. It can be used to load *len* bytes from *offset* from
* the packet associated to *skb*, into the buffer pointed by
* *to*.
*
* Since Linux 4.7, usage of this helper has mostly been replaced
* by "direct packet access", enabling packet data to be
* manipulated with *skb*\ **->data** and *skb*\ **->data_end**
* pointing respectively to the first byte of packet data and to
* the byte after the last byte of packet data. However, it
* remains useful if one wishes to read large quantities of data
* at once from a packet into the eBPF stack.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_get_stackid(void *ctx, struct bpf_map *map, u64 flags)
* Description
* Walk a user or a kernel stack and return its id. To achieve
* this, the helper needs *ctx*, which is a pointer to the context
* on which the tracing program is executed, and a pointer to a
* *map* of type **BPF_MAP_TYPE_STACK_TRACE**.
*
* The last argument, *flags*, holds the number of stack frames to
* skip (from 0 to 255), masked with
* **BPF_F_SKIP_FIELD_MASK**. The next bits can be used to set
* a combination of the following flags:
*
* **BPF_F_USER_STACK**
* Collect a user space stack instead of a kernel stack.
* **BPF_F_FAST_STACK_CMP**
* Compare stacks by hash only.
* **BPF_F_REUSE_STACKID**
* If two different stacks hash into the same *stackid*,
* discard the old one.
*
* The stack id retrieved is a 32 bit long integer handle which
* can be further combined with other data (including other stack
* ids) and used as a key into maps. This can be useful for
* generating a variety of graphs (such as flame graphs or off-cpu
* graphs).
*
* For walking a stack, this helper is an improvement over
* **bpf_probe_read**\ (), which can be used with unrolled loops
* but is not efficient and consumes a lot of eBPF instructions.
* Instead, **bpf_get_stackid**\ () can collect up to
* **PERF_MAX_STACK_DEPTH** both kernel and user frames. Note that
* this limit can be controlled with the **sysctl** program, and
* that it should be manually increased in order to profile long
* user stacks (such as stacks for Java programs). To do so, use:
*
* ::
*
* # sysctl kernel.perf_event_max_stack=<new value>
* Return
* The positive or null stack id on success, or a negative error
* in case of failure.
*
* s64 bpf_csum_diff(__be32 *from, u32 from_size, __be32 *to, u32 to_size, __wsum seed)
* Description
* Compute a checksum difference, from the raw buffer pointed by
* *from*, of length *from_size* (that must be a multiple of 4),
* towards the raw buffer pointed by *to*, of size *to_size*
* (same remark). An optional *seed* can be added to the value
* (this can be cascaded, the seed may come from a previous call
* to the helper).
*
* This is flexible enough to be used in several ways:
*
* * With *from_size* == 0, *to_size* > 0 and *seed* set to
* checksum, it can be used when pushing new data.
* * With *from_size* > 0, *to_size* == 0 and *seed* set to
* checksum, it can be used when removing data from a packet.
* * With *from_size* > 0, *to_size* > 0 and *seed* set to 0, it
* can be used to compute a diff. Note that *from_size* and
* *to_size* do not need to be equal.
*
* This helper can be used in combination with
* **bpf_l3_csum_replace**\ () and **bpf_l4_csum_replace**\ (), to
* which one can feed in the difference computed with
* **bpf_csum_diff**\ ().
* Return
* The checksum result, or a negative error code in case of
* failure.
*
* long bpf_skb_get_tunnel_opt(struct sk_buff *skb, void *opt, u32 size)
* Description
* Retrieve tunnel options metadata for the packet associated to
* *skb*, and store the raw tunnel option data to the buffer *opt*
* of *size*.
*
* This helper can be used with encapsulation devices that can
* operate in "collect metadata" mode (please refer to the related
* note in the description of **bpf_skb_get_tunnel_key**\ () for
* more details). A particular example where this can be used is
* in combination with the Geneve encapsulation protocol, where it
* allows for pushing (with **bpf_skb_get_tunnel_opt**\ () helper)
* and retrieving arbitrary TLVs (Type-Length-Value headers) from
* the eBPF program. This allows for full customization of these
* headers.
* Return
* The size of the option data retrieved.
*
* long bpf_skb_set_tunnel_opt(struct sk_buff *skb, void *opt, u32 size)
* Description
* Set tunnel options metadata for the packet associated to *skb*
* to the option data contained in the raw buffer *opt* of *size*.
*
* See also the description of the **bpf_skb_get_tunnel_opt**\ ()
* helper for additional information.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_skb_change_proto(struct sk_buff *skb, __be16 proto, u64 flags)
* Description
* Change the protocol of the *skb* to *proto*. Currently
* supported are transition from IPv4 to IPv6, and from IPv6 to
* IPv4. The helper takes care of the groundwork for the
* transition, including resizing the socket buffer. The eBPF
* program is expected to fill the new headers, if any, via
* **skb_store_bytes**\ () and to recompute the checksums with
* **bpf_l3_csum_replace**\ () and **bpf_l4_csum_replace**\
* (). The main case for this helper is to perform NAT64
* operations out of an eBPF program.
*
* Internally, the GSO type is marked as dodgy so that headers are
* checked and segments are recalculated by the GSO/GRO engine.
* The size for GSO target is adapted as well.
*
* All values for *flags* are reserved for future usage, and must
* be left at zero.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_skb_change_type(struct sk_buff *skb, u32 type)
* Description
* Change the packet type for the packet associated to *skb*. This
* comes down to setting *skb*\ **->pkt_type** to *type*, except
* the eBPF program does not have a write access to *skb*\
* **->pkt_type** beside this helper. Using a helper here allows
* for graceful handling of errors.
*
* The major use case is to change incoming *skb*s to
* **PACKET_HOST** in a programmatic way instead of having to
* recirculate via **redirect**\ (..., **BPF_F_INGRESS**), for
* example.
*
* Note that *type* only allows certain values. At this time, they
* are:
*
* **PACKET_HOST**
* Packet is for us.
* **PACKET_BROADCAST**
* Send packet to all.
* **PACKET_MULTICAST**
* Send packet to group.
* **PACKET_OTHERHOST**
* Send packet to someone else.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_skb_under_cgroup(struct sk_buff *skb, struct bpf_map *map, u32 index)
* Description
* Check whether *skb* is a descendant of the cgroup2 held by
* *map* of type **BPF_MAP_TYPE_CGROUP_ARRAY**, at *index*.
* Return
* The return value depends on the result of the test, and can be:
*
* * 0, if the *skb* failed the cgroup2 descendant test.
* * 1, if the *skb* succeeded the cgroup2 descendant test.
* * A negative error code, if an error occurred.
*
* u32 bpf_get_hash_recalc(struct sk_buff *skb)
* Description
* Retrieve the hash of the packet, *skb*\ **->hash**. If it is
* not set, in particular if the hash was cleared due to mangling,
* recompute this hash. Later accesses to the hash can be done
* directly with *skb*\ **->hash**.
*
* Calling **bpf_set_hash_invalid**\ (), changing a packet
* prototype with **bpf_skb_change_proto**\ (), or calling
* **bpf_skb_store_bytes**\ () with the
* **BPF_F_INVALIDATE_HASH** are actions susceptible to clear
* the hash and to trigger a new computation for the next call to
* **bpf_get_hash_recalc**\ ().
* Return
* The 32-bit hash.
*
* u64 bpf_get_current_task(void)
* Description
* Get the current task.
* Return
* A pointer to the current task struct.
*
* long bpf_probe_write_user(void *dst, const void *src, u32 len)
* Description
* Attempt in a safe way to write *len* bytes from the buffer
* *src* to *dst* in memory. It only works for threads that are in
* user context, and *dst* must be a valid user space address.
*
* This helper should not be used to implement any kind of
* security mechanism because of TOC-TOU attacks, but rather to
* debug, divert, and manipulate execution of semi-cooperative
* processes.
*
* Keep in mind that this feature is meant for experiments, and it
* has a risk of crashing the system and running programs.
* Therefore, when an eBPF program using this helper is attached,
* a warning including PID and process name is printed to kernel
* logs.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_current_task_under_cgroup(struct bpf_map *map, u32 index)
* Description
* Check whether the probe is being run is the context of a given
* subset of the cgroup2 hierarchy. The cgroup2 to test is held by
* *map* of type **BPF_MAP_TYPE_CGROUP_ARRAY**, at *index*.
* Return
* The return value depends on the result of the test, and can be:
*
* * 1, if current task belongs to the cgroup2.
* * 0, if current task does not belong to the cgroup2.
* * A negative error code, if an error occurred.
*
* long bpf_skb_change_tail(struct sk_buff *skb, u32 len, u64 flags)
* Description
* Resize (trim or grow) the packet associated to *skb* to the
* new *len*. The *flags* are reserved for future usage, and must
* be left at zero.
*
* The basic idea is that the helper performs the needed work to
* change the size of the packet, then the eBPF program rewrites
* the rest via helpers like **bpf_skb_store_bytes**\ (),
* **bpf_l3_csum_replace**\ (), **bpf_l3_csum_replace**\ ()
* and others. This helper is a slow path utility intended for
* replies with control messages. And because it is targeted for
* slow path, the helper itself can afford to be slow: it
* implicitly linearizes, unclones and drops offloads from the
* *skb*.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_skb_pull_data(struct sk_buff *skb, u32 len)
* Description
* Pull in non-linear data in case the *skb* is non-linear and not
* all of *len* are part of the linear section. Make *len* bytes
* from *skb* readable and writable. If a zero value is passed for
* *len*, then all bytes in the linear part of *skb* will be made
* readable and writable.
*
* This helper is only needed for reading and writing with direct
* packet access.
*
* For direct packet access, testing that offsets to access
* are within packet boundaries (test on *skb*\ **->data_end**) is
* susceptible to fail if offsets are invalid, or if the requested
* data is in non-linear parts of the *skb*. On failure the
* program can just bail out, or in the case of a non-linear
* buffer, use a helper to make the data available. The
* **bpf_skb_load_bytes**\ () helper is a first solution to access
* the data. Another one consists in using **bpf_skb_pull_data**
* to pull in once the non-linear parts, then retesting and
* eventually access the data.
*
* At the same time, this also makes sure the *skb* is uncloned,
* which is a necessary condition for direct write. As this needs
* to be an invariant for the write part only, the verifier
* detects writes and adds a prologue that is calling
* **bpf_skb_pull_data()** to effectively unclone the *skb* from
* the very beginning in case it is indeed cloned.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* s64 bpf_csum_update(struct sk_buff *skb, __wsum csum)
* Description
* Add the checksum *csum* into *skb*\ **->csum** in case the
* driver has supplied a checksum for the entire packet into that
* field. Return an error otherwise. This helper is intended to be
* used in combination with **bpf_csum_diff**\ (), in particular
* when the checksum needs to be updated after data has been
* written into the packet through direct packet access.
* Return
* The checksum on success, or a negative error code in case of
* failure.
*
* void bpf_set_hash_invalid(struct sk_buff *skb)
* Description
* Invalidate the current *skb*\ **->hash**. It can be used after
* mangling on headers through direct packet access, in order to
* indicate that the hash is outdated and to trigger a
* recalculation the next time the kernel tries to access this
* hash or when the **bpf_get_hash_recalc**\ () helper is called.
* Return
* void.
*
* long bpf_get_numa_node_id(void)
* Description
* Return the id of the current NUMA node. The primary use case
* for this helper is the selection of sockets for the local NUMA
* node, when the program is attached to sockets using the
* **SO_ATTACH_REUSEPORT_EBPF** option (see also **socket(7)**),
* but the helper is also available to other eBPF program types,
* similarly to **bpf_get_smp_processor_id**\ ().
* Return
* The id of current NUMA node.
*
* long bpf_skb_change_head(struct sk_buff *skb, u32 len, u64 flags)
* Description
* Grows headroom of packet associated to *skb* and adjusts the
* offset of the MAC header accordingly, adding *len* bytes of
* space. It automatically extends and reallocates memory as
* required.
*
* This helper can be used on a layer 3 *skb* to push a MAC header
* for redirection into a layer 2 device.
*
* All values for *flags* are reserved for future usage, and must
* be left at zero.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_xdp_adjust_head(struct xdp_buff *xdp_md, int delta)
* Description
* Adjust (move) *xdp_md*\ **->data** by *delta* bytes. Note that
* it is possible to use a negative value for *delta*. This helper
* can be used to prepare the packet for pushing or popping
* headers.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_probe_read_str(void *dst, u32 size, const void *unsafe_ptr)
* Description
* Copy a NUL terminated string from an unsafe kernel address
* *unsafe_ptr* to *dst*. See **bpf_probe_read_kernel_str**\ () for
* more details.
*
* Generally, use **bpf_probe_read_user_str**\ () or
* **bpf_probe_read_kernel_str**\ () instead.
* Return
* On success, the strictly positive length of the string,
* including the trailing NUL character. On error, a negative
* value.
*
* u64 bpf_get_socket_cookie(struct sk_buff *skb)
* Description
* If the **struct sk_buff** pointed by *skb* has a known socket,
* retrieve the cookie (generated by the kernel) of this socket.
* If no cookie has been set yet, generate a new cookie. Once
* generated, the socket cookie remains stable for the life of the
* socket. This helper can be useful for monitoring per socket
* networking traffic statistics as it provides a global socket
* identifier that can be assumed unique.
* Return
* A 8-byte long unique number on success, or 0 if the socket
* field is missing inside *skb*.
*
* u64 bpf_get_socket_cookie(struct bpf_sock_addr *ctx)
* Description
* Equivalent to bpf_get_socket_cookie() helper that accepts
* *skb*, but gets socket from **struct bpf_sock_addr** context.
* Return
* A 8-byte long unique number.
*
* u64 bpf_get_socket_cookie(struct bpf_sock_ops *ctx)
* Description
* Equivalent to **bpf_get_socket_cookie**\ () helper that accepts
* *skb*, but gets socket from **struct bpf_sock_ops** context.
* Return
* A 8-byte long unique number.
*
* u64 bpf_get_socket_cookie(struct sock *sk)
* Description
* Equivalent to **bpf_get_socket_cookie**\ () helper that accepts
* *sk*, but gets socket from a BTF **struct sock**. This helper
* also works for sleepable programs.
* Return
* A 8-byte long unique number or 0 if *sk* is NULL.
*
* u32 bpf_get_socket_uid(struct sk_buff *skb)
* Description
* Get the owner UID of the socked associated to *skb*.
* Return
* The owner UID of the socket associated to *skb*. If the socket
* is **NULL**, or if it is not a full socket (i.e. if it is a
* time-wait or a request socket instead), **overflowuid** value
* is returned (note that **overflowuid** might also be the actual
* UID value for the socket).
*
* long bpf_set_hash(struct sk_buff *skb, u32 hash)
* Description
* Set the full hash for *skb* (set the field *skb*\ **->hash**)
* to value *hash*.
* Return
* 0
*
* long bpf_setsockopt(void *bpf_socket, int level, int optname, void *optval, int optlen)
* Description
* Emulate a call to **setsockopt()** on the socket associated to
* *bpf_socket*, which must be a full socket. The *level* at
* which the option resides and the name *optname* of the option
* must be specified, see **setsockopt(2)** for more information.
* The option value of length *optlen* is pointed by *optval*.
*
* *bpf_socket* should be one of the following:
*
* * **struct bpf_sock_ops** for **BPF_PROG_TYPE_SOCK_OPS**.
* * **struct bpf_sock_addr** for **BPF_CGROUP_INET4_CONNECT**,
* **BPF_CGROUP_INET6_CONNECT** and **BPF_CGROUP_UNIX_CONNECT**.
*
* This helper actually implements a subset of **setsockopt()**.
* It supports the following *level*\ s:
*
* * **SOL_SOCKET**, which supports the following *optname*\ s:
* **SO_RCVBUF**, **SO_SNDBUF**, **SO_MAX_PACING_RATE**,
* **SO_PRIORITY**, **SO_RCVLOWAT**, **SO_MARK**,
* **SO_BINDTODEVICE**, **SO_KEEPALIVE**, **SO_REUSEADDR**,
* **SO_REUSEPORT**, **SO_BINDTOIFINDEX**, **SO_TXREHASH**.
* * **IPPROTO_TCP**, which supports the following *optname*\ s:
* **TCP_CONGESTION**, **TCP_BPF_IW**,
* **TCP_BPF_SNDCWND_CLAMP**, **TCP_SAVE_SYN**,
* **TCP_KEEPIDLE**, **TCP_KEEPINTVL**, **TCP_KEEPCNT**,
* **TCP_SYNCNT**, **TCP_USER_TIMEOUT**, **TCP_NOTSENT_LOWAT**,
* **TCP_NODELAY**, **TCP_MAXSEG**, **TCP_WINDOW_CLAMP**,
* **TCP_THIN_LINEAR_TIMEOUTS**, **TCP_BPF_DELACK_MAX**,
* **TCP_BPF_RTO_MIN**, **TCP_BPF_SOCK_OPS_CB_FLAGS**.
* * **IPPROTO_IP**, which supports *optname* **IP_TOS**.
* * **IPPROTO_IPV6**, which supports the following *optname*\ s:
* **IPV6_TCLASS**, **IPV6_AUTOFLOWLABEL**.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_skb_adjust_room(struct sk_buff *skb, s32 len_diff, u32 mode, u64 flags)
* Description
* Grow or shrink the room for data in the packet associated to
* *skb* by *len_diff*, and according to the selected *mode*.
*
* By default, the helper will reset any offloaded checksum
* indicator of the skb to CHECKSUM_NONE. This can be avoided
* by the following flag:
*
* * **BPF_F_ADJ_ROOM_NO_CSUM_RESET**: Do not reset offloaded
* checksum data of the skb to CHECKSUM_NONE.
*
* There are two supported modes at this time:
*
* * **BPF_ADJ_ROOM_MAC**: Adjust room at the mac layer
* (room space is added or removed between the layer 2 and
* layer 3 headers).
*
* * **BPF_ADJ_ROOM_NET**: Adjust room at the network layer
* (room space is added or removed between the layer 3 and
* layer 4 headers).
*
* The following flags are supported at this time:
*
* * **BPF_F_ADJ_ROOM_FIXED_GSO**: Do not adjust gso_size.
* Adjusting mss in this way is not allowed for datagrams.
*
* * **BPF_F_ADJ_ROOM_ENCAP_L3_IPV4**,
* **BPF_F_ADJ_ROOM_ENCAP_L3_IPV6**:
* Any new space is reserved to hold a tunnel header.
* Configure skb offsets and other fields accordingly.
*
* * **BPF_F_ADJ_ROOM_ENCAP_L4_GRE**,
* **BPF_F_ADJ_ROOM_ENCAP_L4_UDP**:
* Use with ENCAP_L3 flags to further specify the tunnel type.
*
* * **BPF_F_ADJ_ROOM_ENCAP_L2**\ (*len*):
* Use with ENCAP_L3/L4 flags to further specify the tunnel
* type; *len* is the length of the inner MAC header.
*
* * **BPF_F_ADJ_ROOM_ENCAP_L2_ETH**:
* Use with BPF_F_ADJ_ROOM_ENCAP_L2 flag to further specify the
* L2 type as Ethernet.
*
* * **BPF_F_ADJ_ROOM_DECAP_L3_IPV4**,
* **BPF_F_ADJ_ROOM_DECAP_L3_IPV6**:
* Indicate the new IP header version after decapsulating the outer
* IP header. Used when the inner and outer IP versions are different.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_redirect_map(struct bpf_map *map, u64 key, u64 flags)
* Description
* Redirect the packet to the endpoint referenced by *map* at
* index *key*. Depending on its type, this *map* can contain
* references to net devices (for forwarding packets through other
* ports), or to CPUs (for redirecting XDP frames to another CPU;
* but this is only implemented for native XDP (with driver
* support) as of this writing).
*
* The lower two bits of *flags* are used as the return code if
* the map lookup fails. This is so that the return value can be
* one of the XDP program return codes up to **XDP_TX**, as chosen
* by the caller. The higher bits of *flags* can be set to
* BPF_F_BROADCAST or BPF_F_EXCLUDE_INGRESS as defined below.
*
* With BPF_F_BROADCAST the packet will be broadcasted to all the
* interfaces in the map, with BPF_F_EXCLUDE_INGRESS the ingress
* interface will be excluded when do broadcasting.
*
* See also **bpf_redirect**\ (), which only supports redirecting
* to an ifindex, but doesn't require a map to do so.
* Return
* **XDP_REDIRECT** on success, or the value of the two lower bits
* of the *flags* argument on error.
*
* long bpf_sk_redirect_map(struct sk_buff *skb, struct bpf_map *map, u32 key, u64 flags)
* Description
* Redirect the packet to the socket referenced by *map* (of type
* **BPF_MAP_TYPE_SOCKMAP**) at index *key*. Both ingress and
* egress interfaces can be used for redirection. The
* **BPF_F_INGRESS** value in *flags* is used to make the
* distinction (ingress path is selected if the flag is present,
* egress path otherwise). This is the only flag supported for now.
* Return
* **SK_PASS** on success, or **SK_DROP** on error.
*
* long bpf_sock_map_update(struct bpf_sock_ops *skops, struct bpf_map *map, void *key, u64 flags)
* Description
* Add an entry to, or update a *map* referencing sockets. The
* *skops* is used as a new value for the entry associated to
* *key*. *flags* is one of:
*
* **BPF_NOEXIST**
* The entry for *key* must not exist in the map.
* **BPF_EXIST**
* The entry for *key* must already exist in the map.
* **BPF_ANY**
* No condition on the existence of the entry for *key*.
*
* If the *map* has eBPF programs (parser and verdict), those will
* be inherited by the socket being added. If the socket is
* already attached to eBPF programs, this results in an error.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_xdp_adjust_meta(struct xdp_buff *xdp_md, int delta)
* Description
* Adjust the address pointed by *xdp_md*\ **->data_meta** by
* *delta* (which can be positive or negative). Note that this
* operation modifies the address stored in *xdp_md*\ **->data**,
* so the latter must be loaded only after the helper has been
* called.
*
* The use of *xdp_md*\ **->data_meta** is optional and programs
* are not required to use it. The rationale is that when the
* packet is processed with XDP (e.g. as DoS filter), it is
* possible to push further meta data along with it before passing
* to the stack, and to give the guarantee that an ingress eBPF
* program attached as a TC classifier on the same device can pick
* this up for further post-processing. Since TC works with socket
* buffers, it remains possible to set from XDP the **mark** or
* **priority** pointers, or other pointers for the socket buffer.
* Having this scratch space generic and programmable allows for
* more flexibility as the user is free to store whatever meta
* data they need.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_perf_event_read_value(struct bpf_map *map, u64 flags, struct bpf_perf_event_value *buf, u32 buf_size)
* Description
* Read the value of a perf event counter, and store it into *buf*
* of size *buf_size*. This helper relies on a *map* of type
* **BPF_MAP_TYPE_PERF_EVENT_ARRAY**. The nature of the perf event
* counter is selected when *map* is updated with perf event file
* descriptors. The *map* is an array whose size is the number of
* available CPUs, and each cell contains a value relative to one
* CPU. The value to retrieve is indicated by *flags*, that
* contains the index of the CPU to look up, masked with
* **BPF_F_INDEX_MASK**. Alternatively, *flags* can be set to
* **BPF_F_CURRENT_CPU** to indicate that the value for the
* current CPU should be retrieved.
*
* This helper behaves in a way close to
* **bpf_perf_event_read**\ () helper, save that instead of
* just returning the value observed, it fills the *buf*
* structure. This allows for additional data to be retrieved: in
* particular, the enabled and running times (in *buf*\
* **->enabled** and *buf*\ **->running**, respectively) are
* copied. In general, **bpf_perf_event_read_value**\ () is
* recommended over **bpf_perf_event_read**\ (), which has some
* ABI issues and provides fewer functionalities.
*
* These values are interesting, because hardware PMU (Performance
* Monitoring Unit) counters are limited resources. When there are
* more PMU based perf events opened than available counters,
* kernel will multiplex these events so each event gets certain
* percentage (but not all) of the PMU time. In case that
* multiplexing happens, the number of samples or counter value
* will not reflect the case compared to when no multiplexing
* occurs. This makes comparison between different runs difficult.
* Typically, the counter value should be normalized before
* comparing to other experiments. The usual normalization is done
* as follows.
*
* ::
*
* normalized_counter = counter * t_enabled / t_running
*
* Where t_enabled is the time enabled for event and t_running is
* the time running for event since last normalization. The
* enabled and running times are accumulated since the perf event
* open. To achieve scaling factor between two invocations of an
* eBPF program, users can use CPU id as the key (which is
* typical for perf array usage model) to remember the previous
* value and do the calculation inside the eBPF program.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_perf_prog_read_value(struct bpf_perf_event_data *ctx, struct bpf_perf_event_value *buf, u32 buf_size)
* Description
* For an eBPF program attached to a perf event, retrieve the
* value of the event counter associated to *ctx* and store it in
* the structure pointed by *buf* and of size *buf_size*. Enabled
* and running times are also stored in the structure (see
* description of helper **bpf_perf_event_read_value**\ () for
* more details).
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_getsockopt(void *bpf_socket, int level, int optname, void *optval, int optlen)
* Description
* Emulate a call to **getsockopt()** on the socket associated to
* *bpf_socket*, which must be a full socket. The *level* at
* which the option resides and the name *optname* of the option
* must be specified, see **getsockopt(2)** for more information.
* The retrieved value is stored in the structure pointed by
* *opval* and of length *optlen*.
*
* *bpf_socket* should be one of the following:
*
* * **struct bpf_sock_ops** for **BPF_PROG_TYPE_SOCK_OPS**.
* * **struct bpf_sock_addr** for **BPF_CGROUP_INET4_CONNECT**,
* **BPF_CGROUP_INET6_CONNECT** and **BPF_CGROUP_UNIX_CONNECT**.
*
* This helper actually implements a subset of **getsockopt()**.
* It supports the same set of *optname*\ s that is supported by
* the **bpf_setsockopt**\ () helper. The exceptions are
* **TCP_BPF_*** is **bpf_setsockopt**\ () only and
* **TCP_SAVED_SYN** is **bpf_getsockopt**\ () only.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_override_return(struct pt_regs *regs, u64 rc)
* Description
* Used for error injection, this helper uses kprobes to override
* the return value of the probed function, and to set it to *rc*.
* The first argument is the context *regs* on which the kprobe
* works.
*
* This helper works by setting the PC (program counter)
* to an override function which is run in place of the original
* probed function. This means the probed function is not run at
* all. The replacement function just returns with the required
* value.
*
* This helper has security implications, and thus is subject to
* restrictions. It is only available if the kernel was compiled
* with the **CONFIG_BPF_KPROBE_OVERRIDE** configuration
* option, and in this case it only works on functions tagged with
* **ALLOW_ERROR_INJECTION** in the kernel code.
* Return
* 0
*
* long bpf_sock_ops_cb_flags_set(struct bpf_sock_ops *bpf_sock, int argval)
* Description
* Attempt to set the value of the **bpf_sock_ops_cb_flags** field
* for the full TCP socket associated to *bpf_sock_ops* to
* *argval*.
*
* The primary use of this field is to determine if there should
* be calls to eBPF programs of type
* **BPF_PROG_TYPE_SOCK_OPS** at various points in the TCP
* code. A program of the same type can change its value, per
* connection and as necessary, when the connection is
* established. This field is directly accessible for reading, but
* this helper must be used for updates in order to return an
* error if an eBPF program tries to set a callback that is not
* supported in the current kernel.
*
* *argval* is a flag array which can combine these flags:
*
* * **BPF_SOCK_OPS_RTO_CB_FLAG** (retransmission time out)
* * **BPF_SOCK_OPS_RETRANS_CB_FLAG** (retransmission)
* * **BPF_SOCK_OPS_STATE_CB_FLAG** (TCP state change)
* * **BPF_SOCK_OPS_RTT_CB_FLAG** (every RTT)
*
* Therefore, this function can be used to clear a callback flag by
* setting the appropriate bit to zero. e.g. to disable the RTO
* callback:
*
* **bpf_sock_ops_cb_flags_set(bpf_sock,**
* **bpf_sock->bpf_sock_ops_cb_flags & ~BPF_SOCK_OPS_RTO_CB_FLAG)**
*
* Here are some examples of where one could call such eBPF
* program:
*
* * When RTO fires.
* * When a packet is retransmitted.
* * When the connection terminates.
* * When a packet is sent.
* * When a packet is received.
* Return
* Code **-EINVAL** if the socket is not a full TCP socket;
* otherwise, a positive number containing the bits that could not
* be set is returned (which comes down to 0 if all bits were set
* as required).
*
* long bpf_msg_redirect_map(struct sk_msg_buff *msg, struct bpf_map *map, u32 key, u64 flags)
* Description
* This helper is used in programs implementing policies at the
* socket level. If the message *msg* is allowed to pass (i.e. if
* the verdict eBPF program returns **SK_PASS**), redirect it to
* the socket referenced by *map* (of type
* **BPF_MAP_TYPE_SOCKMAP**) at index *key*. Both ingress and
* egress interfaces can be used for redirection. The
* **BPF_F_INGRESS** value in *flags* is used to make the
* distinction (ingress path is selected if the flag is present,
* egress path otherwise). This is the only flag supported for now.
* Return
* **SK_PASS** on success, or **SK_DROP** on error.
*
* long bpf_msg_apply_bytes(struct sk_msg_buff *msg, u32 bytes)
* Description
* For socket policies, apply the verdict of the eBPF program to
* the next *bytes* (number of bytes) of message *msg*.
*
* For example, this helper can be used in the following cases:
*
* * A single **sendmsg**\ () or **sendfile**\ () system call
* contains multiple logical messages that the eBPF program is
* supposed to read and for which it should apply a verdict.
* * An eBPF program only cares to read the first *bytes* of a
* *msg*. If the message has a large payload, then setting up
* and calling the eBPF program repeatedly for all bytes, even
* though the verdict is already known, would create unnecessary
* overhead.
*
* When called from within an eBPF program, the helper sets a
* counter internal to the BPF infrastructure, that is used to
* apply the last verdict to the next *bytes*. If *bytes* is
* smaller than the current data being processed from a
* **sendmsg**\ () or **sendfile**\ () system call, the first
* *bytes* will be sent and the eBPF program will be re-run with
* the pointer for start of data pointing to byte number *bytes*
* **+ 1**. If *bytes* is larger than the current data being
* processed, then the eBPF verdict will be applied to multiple
* **sendmsg**\ () or **sendfile**\ () calls until *bytes* are
* consumed.
*
* Note that if a socket closes with the internal counter holding
* a non-zero value, this is not a problem because data is not
* being buffered for *bytes* and is sent as it is received.
* Return
* 0
*
* long bpf_msg_cork_bytes(struct sk_msg_buff *msg, u32 bytes)
* Description
* For socket policies, prevent the execution of the verdict eBPF
* program for message *msg* until *bytes* (byte number) have been
* accumulated.
*
* This can be used when one needs a specific number of bytes
* before a verdict can be assigned, even if the data spans
* multiple **sendmsg**\ () or **sendfile**\ () calls. The extreme
* case would be a user calling **sendmsg**\ () repeatedly with
* 1-byte long message segments. Obviously, this is bad for
* performance, but it is still valid. If the eBPF program needs
* *bytes* bytes to validate a header, this helper can be used to
* prevent the eBPF program to be called again until *bytes* have
* been accumulated.
* Return
* 0
*
* long bpf_msg_pull_data(struct sk_msg_buff *msg, u32 start, u32 end, u64 flags)
* Description
* For socket policies, pull in non-linear data from user space
* for *msg* and set pointers *msg*\ **->data** and *msg*\
* **->data_end** to *start* and *end* bytes offsets into *msg*,
* respectively.
*
* If a program of type **BPF_PROG_TYPE_SK_MSG** is run on a
* *msg* it can only parse data that the (**data**, **data_end**)
* pointers have already consumed. For **sendmsg**\ () hooks this
* is likely the first scatterlist element. But for calls relying
* on the **sendpage** handler (e.g. **sendfile**\ ()) this will
* be the range (**0**, **0**) because the data is shared with
* user space and by default the objective is to avoid allowing
* user space to modify data while (or after) eBPF verdict is
* being decided. This helper can be used to pull in data and to
* set the start and end pointer to given values. Data will be
* copied if necessary (i.e. if data was not linear and if start
* and end pointers do not point to the same chunk).
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
*
* All values for *flags* are reserved for future usage, and must
* be left at zero.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_bind(struct bpf_sock_addr *ctx, struct sockaddr *addr, int addr_len)
* Description
* Bind the socket associated to *ctx* to the address pointed by
* *addr*, of length *addr_len*. This allows for making outgoing
* connection from the desired IP address, which can be useful for
* example when all processes inside a cgroup should use one
* single IP address on a host that has multiple IP configured.
*
* This helper works for IPv4 and IPv6, TCP and UDP sockets. The
* domain (*addr*\ **->sa_family**) must be **AF_INET** (or
* **AF_INET6**). It's advised to pass zero port (**sin_port**
* or **sin6_port**) which triggers IP_BIND_ADDRESS_NO_PORT-like
* behavior and lets the kernel efficiently pick up an unused
* port as long as 4-tuple is unique. Passing non-zero port might
* lead to degraded performance.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_xdp_adjust_tail(struct xdp_buff *xdp_md, int delta)
* Description
* Adjust (move) *xdp_md*\ **->data_end** by *delta* bytes. It is
* possible to both shrink and grow the packet tail.
* Shrink done via *delta* being a negative integer.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_skb_get_xfrm_state(struct sk_buff *skb, u32 index, struct bpf_xfrm_state *xfrm_state, u32 size, u64 flags)
* Description
* Retrieve the XFRM state (IP transform framework, see also
* **ip-xfrm(8)**) at *index* in XFRM "security path" for *skb*.
*
* The retrieved value is stored in the **struct bpf_xfrm_state**
* pointed by *xfrm_state* and of length *size*.
*
* All values for *flags* are reserved for future usage, and must
* be left at zero.
*
* This helper is available only if the kernel was compiled with
* **CONFIG_XFRM** configuration option.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_get_stack(void *ctx, void *buf, u32 size, u64 flags)
* Description
* Return a user or a kernel stack in bpf program provided buffer.
* To achieve this, the helper needs *ctx*, which is a pointer
* to the context on which the tracing program is executed.
* To store the stacktrace, the bpf program provides *buf* with
* a nonnegative *size*.
*
* The last argument, *flags*, holds the number of stack frames to
* skip (from 0 to 255), masked with
* **BPF_F_SKIP_FIELD_MASK**. The next bits can be used to set
* the following flags:
*
* **BPF_F_USER_STACK**
* Collect a user space stack instead of a kernel stack.
* **BPF_F_USER_BUILD_ID**
* Collect (build_id, file_offset) instead of ips for user
* stack, only valid if **BPF_F_USER_STACK** is also
* specified.
*
* *file_offset* is an offset relative to the beginning
* of the executable or shared object file backing the vma
* which the *ip* falls in. It is *not* an offset relative
* to that object's base address. Accordingly, it must be
* adjusted by adding (sh_addr - sh_offset), where
* sh_{addr,offset} correspond to the executable section
* containing *file_offset* in the object, for comparisons
* to symbols' st_value to be valid.
*
* **bpf_get_stack**\ () can collect up to
* **PERF_MAX_STACK_DEPTH** both kernel and user frames, subject
* to sufficient large buffer size. Note that
* this limit can be controlled with the **sysctl** program, and
* that it should be manually increased in order to profile long
* user stacks (such as stacks for Java programs). To do so, use:
*
* ::
*
* # sysctl kernel.perf_event_max_stack=<new value>
* Return
* The non-negative copied *buf* length equal to or less than
* *size* on success, or a negative error in case of failure.
*
* long bpf_skb_load_bytes_relative(const void *skb, u32 offset, void *to, u32 len, u32 start_header)
* Description
* This helper is similar to **bpf_skb_load_bytes**\ () in that
* it provides an easy way to load *len* bytes from *offset*
* from the packet associated to *skb*, into the buffer pointed
* by *to*. The difference to **bpf_skb_load_bytes**\ () is that
* a fifth argument *start_header* exists in order to select a
* base offset to start from. *start_header* can be one of:
*
* **BPF_HDR_START_MAC**
* Base offset to load data from is *skb*'s mac header.
* **BPF_HDR_START_NET**
* Base offset to load data from is *skb*'s network header.
*
* In general, "direct packet access" is the preferred method to
* access packet data, however, this helper is in particular useful
* in socket filters where *skb*\ **->data** does not always point
* to the start of the mac header and where "direct packet access"
* is not available.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_fib_lookup(void *ctx, struct bpf_fib_lookup *params, int plen, u32 flags)
* Description
* Do FIB lookup in kernel tables using parameters in *params*.
* If lookup is successful and result shows packet is to be
* forwarded, the neighbor tables are searched for the nexthop.
* If successful (ie., FIB lookup shows forwarding and nexthop
* is resolved), the nexthop address is returned in ipv4_dst
* or ipv6_dst based on family, smac is set to mac address of
* egress device, dmac is set to nexthop mac address, rt_metric
* is set to metric from route (IPv4/IPv6 only), and ifindex
* is set to the device index of the nexthop from the FIB lookup.
*
* *plen* argument is the size of the passed in struct.
* *flags* argument can be a combination of one or more of the
* following values:
*
* **BPF_FIB_LOOKUP_DIRECT**
* Do a direct table lookup vs full lookup using FIB
* rules.
* **BPF_FIB_LOOKUP_TBID**
* Used with BPF_FIB_LOOKUP_DIRECT.
* Use the routing table ID present in *params*->tbid
* for the fib lookup.
* **BPF_FIB_LOOKUP_OUTPUT**
* Perform lookup from an egress perspective (default is
* ingress).
* **BPF_FIB_LOOKUP_SKIP_NEIGH**
* Skip the neighbour table lookup. *params*->dmac
* and *params*->smac will not be set as output. A common
* use case is to call **bpf_redirect_neigh**\ () after
* doing **bpf_fib_lookup**\ ().
* **BPF_FIB_LOOKUP_SRC**
* Derive and set source IP addr in *params*->ipv{4,6}_src
* for the nexthop. If the src addr cannot be derived,
* **BPF_FIB_LKUP_RET_NO_SRC_ADDR** is returned. In this
* case, *params*->dmac and *params*->smac are not set either.
* **BPF_FIB_LOOKUP_MARK**
* Use the mark present in *params*->mark for the fib lookup.
* This option should not be used with BPF_FIB_LOOKUP_DIRECT,
* as it only has meaning for full lookups.
*
* *ctx* is either **struct xdp_md** for XDP programs or
* **struct sk_buff** tc cls_act programs.
* Return
* * < 0 if any input argument is invalid
* * 0 on success (packet is forwarded, nexthop neighbor exists)
* * > 0 one of **BPF_FIB_LKUP_RET_** codes explaining why the
* packet is not forwarded or needs assist from full stack
*
* If lookup fails with BPF_FIB_LKUP_RET_FRAG_NEEDED, then the MTU
* was exceeded and output params->mtu_result contains the MTU.
*
* long bpf_sock_hash_update(struct bpf_sock_ops *skops, struct bpf_map *map, void *key, u64 flags)
* Description
* Add an entry to, or update a sockhash *map* referencing sockets.
* The *skops* is used as a new value for the entry associated to
* *key*. *flags* is one of:
*
* **BPF_NOEXIST**
* The entry for *key* must not exist in the map.
* **BPF_EXIST**
* The entry for *key* must already exist in the map.
* **BPF_ANY**
* No condition on the existence of the entry for *key*.
*
* If the *map* has eBPF programs (parser and verdict), those will
* be inherited by the socket being added. If the socket is
* already attached to eBPF programs, this results in an error.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_msg_redirect_hash(struct sk_msg_buff *msg, struct bpf_map *map, void *key, u64 flags)
* Description
* This helper is used in programs implementing policies at the
* socket level. If the message *msg* is allowed to pass (i.e. if
* the verdict eBPF program returns **SK_PASS**), redirect it to
* the socket referenced by *map* (of type
* **BPF_MAP_TYPE_SOCKHASH**) using hash *key*. Both ingress and
* egress interfaces can be used for redirection. The
* **BPF_F_INGRESS** value in *flags* is used to make the
* distinction (ingress path is selected if the flag is present,
* egress path otherwise). This is the only flag supported for now.
* Return
* **SK_PASS** on success, or **SK_DROP** on error.
*
* long bpf_sk_redirect_hash(struct sk_buff *skb, struct bpf_map *map, void *key, u64 flags)
* Description
* This helper is used in programs implementing policies at the
* skb socket level. If the sk_buff *skb* is allowed to pass (i.e.
* if the verdict eBPF program returns **SK_PASS**), redirect it
* to the socket referenced by *map* (of type
* **BPF_MAP_TYPE_SOCKHASH**) using hash *key*. Both ingress and
* egress interfaces can be used for redirection. The
* **BPF_F_INGRESS** value in *flags* is used to make the
* distinction (ingress path is selected if the flag is present,
* egress otherwise). This is the only flag supported for now.
* Return
* **SK_PASS** on success, or **SK_DROP** on error.
*
* long bpf_lwt_push_encap(struct sk_buff *skb, u32 type, void *hdr, u32 len)
* Description
* Encapsulate the packet associated to *skb* within a Layer 3
* protocol header. This header is provided in the buffer at
* address *hdr*, with *len* its size in bytes. *type* indicates
* the protocol of the header and can be one of:
*
* **BPF_LWT_ENCAP_SEG6**
* IPv6 encapsulation with Segment Routing Header
* (**struct ipv6_sr_hdr**). *hdr* only contains the SRH,
* the IPv6 header is computed by the kernel.
* **BPF_LWT_ENCAP_SEG6_INLINE**
* Only works if *skb* contains an IPv6 packet. Insert a
* Segment Routing Header (**struct ipv6_sr_hdr**) inside
* the IPv6 header.
* **BPF_LWT_ENCAP_IP**
* IP encapsulation (GRE/GUE/IPIP/etc). The outer header
* must be IPv4 or IPv6, followed by zero or more
* additional headers, up to **LWT_BPF_MAX_HEADROOM**
* total bytes in all prepended headers. Please note that
* if **skb_is_gso**\ (*skb*) is true, no more than two
* headers can be prepended, and the inner header, if
* present, should be either GRE or UDP/GUE.
*
* **BPF_LWT_ENCAP_SEG6**\ \* types can be called by BPF programs
* of type **BPF_PROG_TYPE_LWT_IN**; **BPF_LWT_ENCAP_IP** type can
* be called by bpf programs of types **BPF_PROG_TYPE_LWT_IN** and
* **BPF_PROG_TYPE_LWT_XMIT**.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_lwt_seg6_store_bytes(struct sk_buff *skb, u32 offset, const void *from, u32 len)
* Description
* Store *len* bytes from address *from* into the packet
* associated to *skb*, at *offset*. Only the flags, tag and TLVs
* inside the outermost IPv6 Segment Routing Header can be
* modified through this helper.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_lwt_seg6_adjust_srh(struct sk_buff *skb, u32 offset, s32 delta)
* Description
* Adjust the size allocated to TLVs in the outermost IPv6
* Segment Routing Header contained in the packet associated to
* *skb*, at position *offset* by *delta* bytes. Only offsets
* after the segments are accepted. *delta* can be as well
* positive (growing) as negative (shrinking).
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_lwt_seg6_action(struct sk_buff *skb, u32 action, void *param, u32 param_len)
* Description
* Apply an IPv6 Segment Routing action of type *action* to the
* packet associated to *skb*. Each action takes a parameter
* contained at address *param*, and of length *param_len* bytes.
* *action* can be one of:
*
* **SEG6_LOCAL_ACTION_END_X**
* End.X action: Endpoint with Layer-3 cross-connect.
* Type of *param*: **struct in6_addr**.
* **SEG6_LOCAL_ACTION_END_T**
* End.T action: Endpoint with specific IPv6 table lookup.
* Type of *param*: **int**.
* **SEG6_LOCAL_ACTION_END_B6**
* End.B6 action: Endpoint bound to an SRv6 policy.
* Type of *param*: **struct ipv6_sr_hdr**.
* **SEG6_LOCAL_ACTION_END_B6_ENCAP**
* End.B6.Encap action: Endpoint bound to an SRv6
* encapsulation policy.
* Type of *param*: **struct ipv6_sr_hdr**.
*
* A call to this helper is susceptible to change the underlying
* packet buffer. Therefore, at load time, all checks on pointers
* previously done by the verifier are invalidated and must be
* performed again, if the helper is used in combination with
* direct packet access.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_rc_repeat(void *ctx)
* Description
* This helper is used in programs implementing IR decoding, to
* report a successfully decoded repeat key message. This delays
* the generation of a key up event for previously generated
* key down event.
*
* Some IR protocols like NEC have a special IR message for
* repeating last button, for when a button is held down.
*
* The *ctx* should point to the lirc sample as passed into
* the program.
*
* This helper is only available is the kernel was compiled with
* the **CONFIG_BPF_LIRC_MODE2** configuration option set to
* "**y**".
* Return
* 0
*
* long bpf_rc_keydown(void *ctx, u32 protocol, u64 scancode, u32 toggle)
* Description
* This helper is used in programs implementing IR decoding, to
* report a successfully decoded key press with *scancode*,
* *toggle* value in the given *protocol*. The scancode will be
* translated to a keycode using the rc keymap, and reported as
* an input key down event. After a period a key up event is
* generated. This period can be extended by calling either
* **bpf_rc_keydown**\ () again with the same values, or calling
* **bpf_rc_repeat**\ ().
*
* Some protocols include a toggle bit, in case the button was
* released and pressed again between consecutive scancodes.
*
* The *ctx* should point to the lirc sample as passed into
* the program.
*
* The *protocol* is the decoded protocol number (see
* **enum rc_proto** for some predefined values).
*
* This helper is only available is the kernel was compiled with
* the **CONFIG_BPF_LIRC_MODE2** configuration option set to
* "**y**".
* Return
* 0
*
* u64 bpf_skb_cgroup_id(struct sk_buff *skb)
* Description
* Return the cgroup v2 id of the socket associated with the *skb*.
* This is roughly similar to the **bpf_get_cgroup_classid**\ ()
* helper for cgroup v1 by providing a tag resp. identifier that
* can be matched on or used for map lookups e.g. to implement
* policy. The cgroup v2 id of a given path in the hierarchy is
* exposed in user space through the f_handle API in order to get
* to the same 64-bit id.
*
* This helper can be used on TC egress path, but not on ingress,
* and is available only if the kernel was compiled with the
* **CONFIG_SOCK_CGROUP_DATA** configuration option.
* Return
* The id is returned or 0 in case the id could not be retrieved.
*
* u64 bpf_get_current_cgroup_id(void)
* Description
* Get the current cgroup id based on the cgroup within which
* the current task is running.
* Return
* A 64-bit integer containing the current cgroup id based
* on the cgroup within which the current task is running.
*
* void *bpf_get_local_storage(void *map, u64 flags)
* Description
* Get the pointer to the local storage area.
* The type and the size of the local storage is defined
* by the *map* argument.
* The *flags* meaning is specific for each map type,
* and has to be 0 for cgroup local storage.
*
* Depending on the BPF program type, a local storage area
* can be shared between multiple instances of the BPF program,
* running simultaneously.
*
* A user should care about the synchronization by himself.
* For example, by using the **BPF_ATOMIC** instructions to alter
* the shared data.
* Return
* A pointer to the local storage area.
*
* long bpf_sk_select_reuseport(struct sk_reuseport_md *reuse, struct bpf_map *map, void *key, u64 flags)
* Description
* Select a **SO_REUSEPORT** socket from a
* **BPF_MAP_TYPE_REUSEPORT_SOCKARRAY** *map*.
* It checks the selected socket is matching the incoming
* request in the socket buffer.
* Return
* 0 on success, or a negative error in case of failure.
*
* u64 bpf_skb_ancestor_cgroup_id(struct sk_buff *skb, int ancestor_level)
* Description
* Return id of cgroup v2 that is ancestor of cgroup associated
* with the *skb* at the *ancestor_level*. The root cgroup is at
* *ancestor_level* zero and each step down the hierarchy
* increments the level. If *ancestor_level* == level of cgroup
* associated with *skb*, then return value will be same as that
* of **bpf_skb_cgroup_id**\ ().
*
* The helper is useful to implement policies based on cgroups
* that are upper in hierarchy than immediate cgroup associated
* with *skb*.
*
* The format of returned id and helper limitations are same as in
* **bpf_skb_cgroup_id**\ ().
* Return
* The id is returned or 0 in case the id could not be retrieved.
*
* struct bpf_sock *bpf_sk_lookup_tcp(void *ctx, struct bpf_sock_tuple *tuple, u32 tuple_size, u64 netns, u64 flags)
* Description
* Look for TCP socket matching *tuple*, optionally in a child
* network namespace *netns*. The return value must be checked,
* and if non-**NULL**, released via **bpf_sk_release**\ ().
*
* The *ctx* should point to the context of the program, such as
* the skb or socket (depending on the hook in use). This is used
* to determine the base network namespace for the lookup.
*
* *tuple_size* must be one of:
*
* **sizeof**\ (*tuple*\ **->ipv4**)
* Look for an IPv4 socket.
* **sizeof**\ (*tuple*\ **->ipv6**)
* Look for an IPv6 socket.
*
* If the *netns* is a negative signed 32-bit integer, then the
* socket lookup table in the netns associated with the *ctx*
* will be used. For the TC hooks, this is the netns of the device
* in the skb. For socket hooks, this is the netns of the socket.
* If *netns* is any other signed 32-bit value greater than or
* equal to zero then it specifies the ID of the netns relative to
* the netns associated with the *ctx*. *netns* values beyond the
* range of 32-bit integers are reserved for future use.
*
* All values for *flags* are reserved for future usage, and must
* be left at zero.
*
* This helper is available only if the kernel was compiled with
* **CONFIG_NET** configuration option.
* Return
* Pointer to **struct bpf_sock**, or **NULL** in case of failure.
* For sockets with reuseport option, the **struct bpf_sock**
* result is from *reuse*\ **->socks**\ [] using the hash of the
* tuple.
*
* struct bpf_sock *bpf_sk_lookup_udp(void *ctx, struct bpf_sock_tuple *tuple, u32 tuple_size, u64 netns, u64 flags)
* Description
* Look for UDP socket matching *tuple*, optionally in a child
* network namespace *netns*. The return value must be checked,
* and if non-**NULL**, released via **bpf_sk_release**\ ().
*
* The *ctx* should point to the context of the program, such as
* the skb or socket (depending on the hook in use). This is used
* to determine the base network namespace for the lookup.
*
* *tuple_size* must be one of:
*
* **sizeof**\ (*tuple*\ **->ipv4**)
* Look for an IPv4 socket.
* **sizeof**\ (*tuple*\ **->ipv6**)
* Look for an IPv6 socket.
*
* If the *netns* is a negative signed 32-bit integer, then the
* socket lookup table in the netns associated with the *ctx*
* will be used. For the TC hooks, this is the netns of the device
* in the skb. For socket hooks, this is the netns of the socket.
* If *netns* is any other signed 32-bit value greater than or
* equal to zero then it specifies the ID of the netns relative to
* the netns associated with the *ctx*. *netns* values beyond the
* range of 32-bit integers are reserved for future use.
*
* All values for *flags* are reserved for future usage, and must
* be left at zero.
*
* This helper is available only if the kernel was compiled with
* **CONFIG_NET** configuration option.
* Return
* Pointer to **struct bpf_sock**, or **NULL** in case of failure.
* For sockets with reuseport option, the **struct bpf_sock**
* result is from *reuse*\ **->socks**\ [] using the hash of the
* tuple.
*
* long bpf_sk_release(void *sock)
* Description
* Release the reference held by *sock*. *sock* must be a
* non-**NULL** pointer that was returned from
* **bpf_sk_lookup_xxx**\ ().
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_map_push_elem(struct bpf_map *map, const void *value, u64 flags)
* Description
* Push an element *value* in *map*. *flags* is one of:
*
* **BPF_EXIST**
* If the queue/stack is full, the oldest element is
* removed to make room for this.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_map_pop_elem(struct bpf_map *map, void *value)
* Description
* Pop an element from *map*.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_map_peek_elem(struct bpf_map *map, void *value)
* Description
* Get an element from *map* without removing it.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_msg_push_data(struct sk_msg_buff *msg, u32 start, u32 len, u64 flags)
* Description
* For socket policies, insert *len* bytes into *msg* at offset
* *start*.
*
* If a program of type **BPF_PROG_TYPE_SK_MSG** is run on a
* *msg* it may want to insert metadata or options into the *msg*.
* This can later be read and used by any of the lower layer BPF
* hooks.
*
* This helper may fail if under memory pressure (a malloc
* fails) in these cases BPF programs will get an appropriate
* error and BPF programs will need to handle them.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_msg_pop_data(struct sk_msg_buff *msg, u32 start, u32 len, u64 flags)
* Description
* Will remove *len* bytes from a *msg* starting at byte *start*.
* This may result in **ENOMEM** errors under certain situations if
* an allocation and copy are required due to a full ring buffer.
* However, the helper will try to avoid doing the allocation
* if possible. Other errors can occur if input parameters are
* invalid either due to *start* byte not being valid part of *msg*
* payload and/or *pop* value being to large.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_rc_pointer_rel(void *ctx, s32 rel_x, s32 rel_y)
* Description
* This helper is used in programs implementing IR decoding, to
* report a successfully decoded pointer movement.
*
* The *ctx* should point to the lirc sample as passed into
* the program.
*
* This helper is only available is the kernel was compiled with
* the **CONFIG_BPF_LIRC_MODE2** configuration option set to
* "**y**".
* Return
* 0
*
* long bpf_spin_lock(struct bpf_spin_lock *lock)
* Description
* Acquire a spinlock represented by the pointer *lock*, which is
* stored as part of a value of a map. Taking the lock allows to
* safely update the rest of the fields in that value. The
* spinlock can (and must) later be released with a call to
* **bpf_spin_unlock**\ (\ *lock*\ ).
*
* Spinlocks in BPF programs come with a number of restrictions
* and constraints:
*
* * **bpf_spin_lock** objects are only allowed inside maps of
* types **BPF_MAP_TYPE_HASH** and **BPF_MAP_TYPE_ARRAY** (this
* list could be extended in the future).
* * BTF description of the map is mandatory.
* * The BPF program can take ONE lock at a time, since taking two
* or more could cause dead locks.
* * Only one **struct bpf_spin_lock** is allowed per map element.
* * When the lock is taken, calls (either BPF to BPF or helpers)
* are not allowed.
* * The **BPF_LD_ABS** and **BPF_LD_IND** instructions are not
* allowed inside a spinlock-ed region.
* * The BPF program MUST call **bpf_spin_unlock**\ () to release
* the lock, on all execution paths, before it returns.
* * The BPF program can access **struct bpf_spin_lock** only via
* the **bpf_spin_lock**\ () and **bpf_spin_unlock**\ ()
* helpers. Loading or storing data into the **struct
* bpf_spin_lock** *lock*\ **;** field of a map is not allowed.
* * To use the **bpf_spin_lock**\ () helper, the BTF description
* of the map value must be a struct and have **struct
* bpf_spin_lock** *anyname*\ **;** field at the top level.
* Nested lock inside another struct is not allowed.
* * The **struct bpf_spin_lock** *lock* field in a map value must
* be aligned on a multiple of 4 bytes in that value.
* * Syscall with command **BPF_MAP_LOOKUP_ELEM** does not copy
* the **bpf_spin_lock** field to user space.
* * Syscall with command **BPF_MAP_UPDATE_ELEM**, or update from
* a BPF program, do not update the **bpf_spin_lock** field.
* * **bpf_spin_lock** cannot be on the stack or inside a
* networking packet (it can only be inside of a map values).
* * **bpf_spin_lock** is available to root only.
* * Tracing programs and socket filter programs cannot use
* **bpf_spin_lock**\ () due to insufficient preemption checks
* (but this may change in the future).
* * **bpf_spin_lock** is not allowed in inner maps of map-in-map.
* Return
* 0
*
* long bpf_spin_unlock(struct bpf_spin_lock *lock)
* Description
* Release the *lock* previously locked by a call to
* **bpf_spin_lock**\ (\ *lock*\ ).
* Return
* 0
*
* struct bpf_sock *bpf_sk_fullsock(struct bpf_sock *sk)
* Description
* This helper gets a **struct bpf_sock** pointer such
* that all the fields in this **bpf_sock** can be accessed.
* Return
* A **struct bpf_sock** pointer on success, or **NULL** in
* case of failure.
*
* struct bpf_tcp_sock *bpf_tcp_sock(struct bpf_sock *sk)
* Description
* This helper gets a **struct bpf_tcp_sock** pointer from a
* **struct bpf_sock** pointer.
* Return
* A **struct bpf_tcp_sock** pointer on success, or **NULL** in
* case of failure.
*
* long bpf_skb_ecn_set_ce(struct sk_buff *skb)
* Description
* Set ECN (Explicit Congestion Notification) field of IP header
* to **CE** (Congestion Encountered) if current value is **ECT**
* (ECN Capable Transport). Otherwise, do nothing. Works with IPv6
* and IPv4.
* Return
* 1 if the **CE** flag is set (either by the current helper call
* or because it was already present), 0 if it is not set.
*
* struct bpf_sock *bpf_get_listener_sock(struct bpf_sock *sk)
* Description
* Return a **struct bpf_sock** pointer in **TCP_LISTEN** state.
* **bpf_sk_release**\ () is unnecessary and not allowed.
* Return
* A **struct bpf_sock** pointer on success, or **NULL** in
* case of failure.
*
* struct bpf_sock *bpf_skc_lookup_tcp(void *ctx, struct bpf_sock_tuple *tuple, u32 tuple_size, u64 netns, u64 flags)
* Description
* Look for TCP socket matching *tuple*, optionally in a child
* network namespace *netns*. The return value must be checked,
* and if non-**NULL**, released via **bpf_sk_release**\ ().
*
* This function is identical to **bpf_sk_lookup_tcp**\ (), except
* that it also returns timewait or request sockets. Use
* **bpf_sk_fullsock**\ () or **bpf_tcp_sock**\ () to access the
* full structure.
*
* This helper is available only if the kernel was compiled with
* **CONFIG_NET** configuration option.
* Return
* Pointer to **struct bpf_sock**, or **NULL** in case of failure.
* For sockets with reuseport option, the **struct bpf_sock**
* result is from *reuse*\ **->socks**\ [] using the hash of the
* tuple.
*
* long bpf_tcp_check_syncookie(void *sk, void *iph, u32 iph_len, struct tcphdr *th, u32 th_len)
* Description
* Check whether *iph* and *th* contain a valid SYN cookie ACK for
* the listening socket in *sk*.
*
* *iph* points to the start of the IPv4 or IPv6 header, while
* *iph_len* contains **sizeof**\ (**struct iphdr**) or
* **sizeof**\ (**struct ipv6hdr**).
*
* *th* points to the start of the TCP header, while *th_len*
* contains the length of the TCP header (at least
* **sizeof**\ (**struct tcphdr**)).
* Return
* 0 if *iph* and *th* are a valid SYN cookie ACK, or a negative
* error otherwise.
*
* long bpf_sysctl_get_name(struct bpf_sysctl *ctx, char *buf, size_t buf_len, u64 flags)
* Description
* Get name of sysctl in /proc/sys/ and copy it into provided by
* program buffer *buf* of size *buf_len*.
*
* The buffer is always NUL terminated, unless it's zero-sized.
*
* If *flags* is zero, full name (e.g. "net/ipv4/tcp_mem") is
* copied. Use **BPF_F_SYSCTL_BASE_NAME** flag to copy base name
* only (e.g. "tcp_mem").
* Return
* Number of character copied (not including the trailing NUL).
*
* **-E2BIG** if the buffer wasn't big enough (*buf* will contain
* truncated name in this case).
*
* long bpf_sysctl_get_current_value(struct bpf_sysctl *ctx, char *buf, size_t buf_len)
* Description
* Get current value of sysctl as it is presented in /proc/sys
* (incl. newline, etc), and copy it as a string into provided
* by program buffer *buf* of size *buf_len*.
*
* The whole value is copied, no matter what file position user
* space issued e.g. sys_read at.
*
* The buffer is always NUL terminated, unless it's zero-sized.
* Return
* Number of character copied (not including the trailing NUL).
*
* **-E2BIG** if the buffer wasn't big enough (*buf* will contain
* truncated name in this case).
*
* **-EINVAL** if current value was unavailable, e.g. because
* sysctl is uninitialized and read returns -EIO for it.
*
* long bpf_sysctl_get_new_value(struct bpf_sysctl *ctx, char *buf, size_t buf_len)
* Description
* Get new value being written by user space to sysctl (before
* the actual write happens) and copy it as a string into
* provided by program buffer *buf* of size *buf_len*.
*
* User space may write new value at file position > 0.
*
* The buffer is always NUL terminated, unless it's zero-sized.
* Return
* Number of character copied (not including the trailing NUL).
*
* **-E2BIG** if the buffer wasn't big enough (*buf* will contain
* truncated name in this case).
*
* **-EINVAL** if sysctl is being read.
*
* long bpf_sysctl_set_new_value(struct bpf_sysctl *ctx, const char *buf, size_t buf_len)
* Description
* Override new value being written by user space to sysctl with
* value provided by program in buffer *buf* of size *buf_len*.
*
* *buf* should contain a string in same form as provided by user
* space on sysctl write.
*
* User space may write new value at file position > 0. To override
* the whole sysctl value file position should be set to zero.
* Return
* 0 on success.
*
* **-E2BIG** if the *buf_len* is too big.
*
* **-EINVAL** if sysctl is being read.
*
* long bpf_strtol(const char *buf, size_t buf_len, u64 flags, long *res)
* Description
* Convert the initial part of the string from buffer *buf* of
* size *buf_len* to a long integer according to the given base
* and save the result in *res*.
*
* The string may begin with an arbitrary amount of white space
* (as determined by **isspace**\ (3)) followed by a single
* optional '**-**' sign.
*
* Five least significant bits of *flags* encode base, other bits
* are currently unused.
*
* Base must be either 8, 10, 16 or 0 to detect it automatically
* similar to user space **strtol**\ (3).
* Return
* Number of characters consumed on success. Must be positive but
* no more than *buf_len*.
*
* **-EINVAL** if no valid digits were found or unsupported base
* was provided.
*
* **-ERANGE** if resulting value was out of range.
*
* long bpf_strtoul(const char *buf, size_t buf_len, u64 flags, unsigned long *res)
* Description
* Convert the initial part of the string from buffer *buf* of
* size *buf_len* to an unsigned long integer according to the
* given base and save the result in *res*.
*
* The string may begin with an arbitrary amount of white space
* (as determined by **isspace**\ (3)).
*
* Five least significant bits of *flags* encode base, other bits
* are currently unused.
*
* Base must be either 8, 10, 16 or 0 to detect it automatically
* similar to user space **strtoul**\ (3).
* Return
* Number of characters consumed on success. Must be positive but
* no more than *buf_len*.
*
* **-EINVAL** if no valid digits were found or unsupported base
* was provided.
*
* **-ERANGE** if resulting value was out of range.
*
* void *bpf_sk_storage_get(struct bpf_map *map, void *sk, void *value, u64 flags)
* Description
* Get a bpf-local-storage from a *sk*.
*
* Logically, it could be thought of getting the value from
* a *map* with *sk* as the **key**. From this
* perspective, the usage is not much different from
* **bpf_map_lookup_elem**\ (*map*, **&**\ *sk*) except this
* helper enforces the key must be a full socket and the map must
* be a **BPF_MAP_TYPE_SK_STORAGE** also.
*
* Underneath, the value is stored locally at *sk* instead of
* the *map*. The *map* is used as the bpf-local-storage
* "type". The bpf-local-storage "type" (i.e. the *map*) is
* searched against all bpf-local-storages residing at *sk*.
*
* *sk* is a kernel **struct sock** pointer for LSM program.
* *sk* is a **struct bpf_sock** pointer for other program types.
*
* An optional *flags* (**BPF_SK_STORAGE_GET_F_CREATE**) can be
* used such that a new bpf-local-storage will be
* created if one does not exist. *value* can be used
* together with **BPF_SK_STORAGE_GET_F_CREATE** to specify
* the initial value of a bpf-local-storage. If *value* is
* **NULL**, the new bpf-local-storage will be zero initialized.
* Return
* A bpf-local-storage pointer is returned on success.
*
* **NULL** if not found or there was an error in adding
* a new bpf-local-storage.
*
* long bpf_sk_storage_delete(struct bpf_map *map, void *sk)
* Description
* Delete a bpf-local-storage from a *sk*.
* Return
* 0 on success.
*
* **-ENOENT** if the bpf-local-storage cannot be found.
* **-EINVAL** if sk is not a fullsock (e.g. a request_sock).
*
* long bpf_send_signal(u32 sig)
* Description
* Send signal *sig* to the process of the current task.
* The signal may be delivered to any of this process's threads.
* Return
* 0 on success or successfully queued.
*
* **-EBUSY** if work queue under nmi is full.
*
* **-EINVAL** if *sig* is invalid.
*
* **-EPERM** if no permission to send the *sig*.
*
* **-EAGAIN** if bpf program can try again.
*
* s64 bpf_tcp_gen_syncookie(void *sk, void *iph, u32 iph_len, struct tcphdr *th, u32 th_len)
* Description
* Try to issue a SYN cookie for the packet with corresponding
* IP/TCP headers, *iph* and *th*, on the listening socket in *sk*.
*
* *iph* points to the start of the IPv4 or IPv6 header, while
* *iph_len* contains **sizeof**\ (**struct iphdr**) or
* **sizeof**\ (**struct ipv6hdr**).
*
* *th* points to the start of the TCP header, while *th_len*
* contains the length of the TCP header with options (at least
* **sizeof**\ (**struct tcphdr**)).
* Return
* On success, lower 32 bits hold the generated SYN cookie in
* followed by 16 bits which hold the MSS value for that cookie,
* and the top 16 bits are unused.
*
* On failure, the returned value is one of the following:
*
* **-EINVAL** SYN cookie cannot be issued due to error
*
* **-ENOENT** SYN cookie should not be issued (no SYN flood)
*
* **-EOPNOTSUPP** kernel configuration does not enable SYN cookies
*
* **-EPROTONOSUPPORT** IP packet version is not 4 or 6
*
* long bpf_skb_output(void *ctx, struct bpf_map *map, u64 flags, void *data, u64 size)
* Description
* Write raw *data* blob into a special BPF perf event held by
* *map* of type **BPF_MAP_TYPE_PERF_EVENT_ARRAY**. This perf
* event must have the following attributes: **PERF_SAMPLE_RAW**
* as **sample_type**, **PERF_TYPE_SOFTWARE** as **type**, and
* **PERF_COUNT_SW_BPF_OUTPUT** as **config**.
*
* The *flags* are used to indicate the index in *map* for which
* the value must be put, masked with **BPF_F_INDEX_MASK**.
* Alternatively, *flags* can be set to **BPF_F_CURRENT_CPU**
* to indicate that the index of the current CPU core should be
* used.
*
* The value to write, of *size*, is passed through eBPF stack and
* pointed by *data*.
*
* *ctx* is a pointer to in-kernel struct sk_buff.
*
* This helper is similar to **bpf_perf_event_output**\ () but
* restricted to raw_tracepoint bpf programs.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_probe_read_user(void *dst, u32 size, const void *unsafe_ptr)
* Description
* Safely attempt to read *size* bytes from user space address
* *unsafe_ptr* and store the data in *dst*.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_probe_read_kernel(void *dst, u32 size, const void *unsafe_ptr)
* Description
* Safely attempt to read *size* bytes from kernel space address
* *unsafe_ptr* and store the data in *dst*.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_probe_read_user_str(void *dst, u32 size, const void *unsafe_ptr)
* Description
* Copy a NUL terminated string from an unsafe user address
* *unsafe_ptr* to *dst*. The *size* should include the
* terminating NUL byte. In case the string length is smaller than
* *size*, the target is not padded with further NUL bytes. If the
* string length is larger than *size*, just *size*-1 bytes are
* copied and the last byte is set to NUL.
*
* On success, returns the number of bytes that were written,
* including the terminal NUL. This makes this helper useful in
* tracing programs for reading strings, and more importantly to
* get its length at runtime. See the following snippet:
*
* ::
*
* SEC("kprobe/sys_open")
* void bpf_sys_open(struct pt_regs *ctx)
* {
* char buf[PATHLEN]; // PATHLEN is defined to 256
* int res = bpf_probe_read_user_str(buf, sizeof(buf),
* ctx->di);
*
* // Consume buf, for example push it to
* // userspace via bpf_perf_event_output(); we
* // can use res (the string length) as event
* // size, after checking its boundaries.
* }
*
* In comparison, using **bpf_probe_read_user**\ () helper here
* instead to read the string would require to estimate the length
* at compile time, and would often result in copying more memory
* than necessary.
*
* Another useful use case is when parsing individual process
* arguments or individual environment variables navigating
* *current*\ **->mm->arg_start** and *current*\
* **->mm->env_start**: using this helper and the return value,
* one can quickly iterate at the right offset of the memory area.
* Return
* On success, the strictly positive length of the output string,
* including the trailing NUL character. On error, a negative
* value.
*
* long bpf_probe_read_kernel_str(void *dst, u32 size, const void *unsafe_ptr)
* Description
* Copy a NUL terminated string from an unsafe kernel address *unsafe_ptr*
* to *dst*. Same semantics as with **bpf_probe_read_user_str**\ () apply.
* Return
* On success, the strictly positive length of the string, including
* the trailing NUL character. On error, a negative value.
*
* long bpf_tcp_send_ack(void *tp, u32 rcv_nxt)
* Description
* Send out a tcp-ack. *tp* is the in-kernel struct **tcp_sock**.
* *rcv_nxt* is the ack_seq to be sent out.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_send_signal_thread(u32 sig)
* Description
* Send signal *sig* to the thread corresponding to the current task.
* Return
* 0 on success or successfully queued.
*
* **-EBUSY** if work queue under nmi is full.
*
* **-EINVAL** if *sig* is invalid.
*
* **-EPERM** if no permission to send the *sig*.
*
* **-EAGAIN** if bpf program can try again.
*
* u64 bpf_jiffies64(void)
* Description
* Obtain the 64bit jiffies
* Return
* The 64 bit jiffies
*
* long bpf_read_branch_records(struct bpf_perf_event_data *ctx, void *buf, u32 size, u64 flags)
* Description
* For an eBPF program attached to a perf event, retrieve the
* branch records (**struct perf_branch_entry**) associated to *ctx*
* and store it in the buffer pointed by *buf* up to size
* *size* bytes.
* Return
* On success, number of bytes written to *buf*. On error, a
* negative value.
*
* The *flags* can be set to **BPF_F_GET_BRANCH_RECORDS_SIZE** to
* instead return the number of bytes required to store all the
* branch entries. If this flag is set, *buf* may be NULL.
*
* **-EINVAL** if arguments invalid or **size** not a multiple
* of **sizeof**\ (**struct perf_branch_entry**\ ).
*
* **-ENOENT** if architecture does not support branch records.
*
* long bpf_get_ns_current_pid_tgid(u64 dev, u64 ino, struct bpf_pidns_info *nsdata, u32 size)
* Description
* Returns 0 on success, values for *pid* and *tgid* as seen from the current
* *namespace* will be returned in *nsdata*.
* Return
* 0 on success, or one of the following in case of failure:
*
* **-EINVAL** if dev and inum supplied don't match dev_t and inode number
* with nsfs of current task, or if dev conversion to dev_t lost high bits.
*
* **-ENOENT** if pidns does not exists for the current task.
*
* long bpf_xdp_output(void *ctx, struct bpf_map *map, u64 flags, void *data, u64 size)
* Description
* Write raw *data* blob into a special BPF perf event held by
* *map* of type **BPF_MAP_TYPE_PERF_EVENT_ARRAY**. This perf
* event must have the following attributes: **PERF_SAMPLE_RAW**
* as **sample_type**, **PERF_TYPE_SOFTWARE** as **type**, and
* **PERF_COUNT_SW_BPF_OUTPUT** as **config**.
*
* The *flags* are used to indicate the index in *map* for which
* the value must be put, masked with **BPF_F_INDEX_MASK**.
* Alternatively, *flags* can be set to **BPF_F_CURRENT_CPU**
* to indicate that the index of the current CPU core should be
* used.
*
* The value to write, of *size*, is passed through eBPF stack and
* pointed by *data*.
*
* *ctx* is a pointer to in-kernel struct xdp_buff.
*
* This helper is similar to **bpf_perf_eventoutput**\ () but
* restricted to raw_tracepoint bpf programs.
* Return
* 0 on success, or a negative error in case of failure.
*
* u64 bpf_get_netns_cookie(void *ctx)
* Description
* Retrieve the cookie (generated by the kernel) of the network
* namespace the input *ctx* is associated with. The network
* namespace cookie remains stable for its lifetime and provides
* a global identifier that can be assumed unique. If *ctx* is
* NULL, then the helper returns the cookie for the initial
* network namespace. The cookie itself is very similar to that
* of **bpf_get_socket_cookie**\ () helper, but for network
* namespaces instead of sockets.
* Return
* A 8-byte long opaque number.
*
* u64 bpf_get_current_ancestor_cgroup_id(int ancestor_level)
* Description
* Return id of cgroup v2 that is ancestor of the cgroup associated
* with the current task at the *ancestor_level*. The root cgroup
* is at *ancestor_level* zero and each step down the hierarchy
* increments the level. If *ancestor_level* == level of cgroup
* associated with the current task, then return value will be the
* same as that of **bpf_get_current_cgroup_id**\ ().
*
* The helper is useful to implement policies based on cgroups
* that are upper in hierarchy than immediate cgroup associated
* with the current task.
*
* The format of returned id and helper limitations are same as in
* **bpf_get_current_cgroup_id**\ ().
* Return
* The id is returned or 0 in case the id could not be retrieved.
*
* long bpf_sk_assign(struct sk_buff *skb, void *sk, u64 flags)
* Description
* Helper is overloaded depending on BPF program type. This
* description applies to **BPF_PROG_TYPE_SCHED_CLS** and
* **BPF_PROG_TYPE_SCHED_ACT** programs.
*
* Assign the *sk* to the *skb*. When combined with appropriate
* routing configuration to receive the packet towards the socket,
* will cause *skb* to be delivered to the specified socket.
* Subsequent redirection of *skb* via **bpf_redirect**\ (),
* **bpf_clone_redirect**\ () or other methods outside of BPF may
* interfere with successful delivery to the socket.
*
* This operation is only valid from TC ingress path.
*
* The *flags* argument must be zero.
* Return
* 0 on success, or a negative error in case of failure:
*
* **-EINVAL** if specified *flags* are not supported.
*
* **-ENOENT** if the socket is unavailable for assignment.
*
* **-ENETUNREACH** if the socket is unreachable (wrong netns).
*
* **-EOPNOTSUPP** if the operation is not supported, for example
* a call from outside of TC ingress.
*
* long bpf_sk_assign(struct bpf_sk_lookup *ctx, struct bpf_sock *sk, u64 flags)
* Description
* Helper is overloaded depending on BPF program type. This
* description applies to **BPF_PROG_TYPE_SK_LOOKUP** programs.
*
* Select the *sk* as a result of a socket lookup.
*
* For the operation to succeed passed socket must be compatible
* with the packet description provided by the *ctx* object.
*
* L4 protocol (**IPPROTO_TCP** or **IPPROTO_UDP**) must
* be an exact match. While IP family (**AF_INET** or
* **AF_INET6**) must be compatible, that is IPv6 sockets
* that are not v6-only can be selected for IPv4 packets.
*
* Only TCP listeners and UDP unconnected sockets can be
* selected. *sk* can also be NULL to reset any previous
* selection.
*
* *flags* argument can combination of following values:
*
* * **BPF_SK_LOOKUP_F_REPLACE** to override the previous
* socket selection, potentially done by a BPF program
* that ran before us.
*
* * **BPF_SK_LOOKUP_F_NO_REUSEPORT** to skip
* load-balancing within reuseport group for the socket
* being selected.
*
* On success *ctx->sk* will point to the selected socket.
*
* Return
* 0 on success, or a negative errno in case of failure.
*
* * **-EAFNOSUPPORT** if socket family (*sk->family*) is
* not compatible with packet family (*ctx->family*).
*
* * **-EEXIST** if socket has been already selected,
* potentially by another program, and
* **BPF_SK_LOOKUP_F_REPLACE** flag was not specified.
*
* * **-EINVAL** if unsupported flags were specified.
*
* * **-EPROTOTYPE** if socket L4 protocol
* (*sk->protocol*) doesn't match packet protocol
* (*ctx->protocol*).
*
* * **-ESOCKTNOSUPPORT** if socket is not in allowed
* state (TCP listening or UDP unconnected).
*
* u64 bpf_ktime_get_boot_ns(void)
* Description
* Return the time elapsed since system boot, in nanoseconds.
* Does include the time the system was suspended.
* See: **clock_gettime**\ (**CLOCK_BOOTTIME**)
* Return
* Current *ktime*.
*
* long bpf_seq_printf(struct seq_file *m, const char *fmt, u32 fmt_size, const void *data, u32 data_len)
* Description
* **bpf_seq_printf**\ () uses seq_file **seq_printf**\ () to print
* out the format string.
* The *m* represents the seq_file. The *fmt* and *fmt_size* are for
* the format string itself. The *data* and *data_len* are format string
* arguments. The *data* are a **u64** array and corresponding format string
* values are stored in the array. For strings and pointers where pointees
* are accessed, only the pointer values are stored in the *data* array.
* The *data_len* is the size of *data* in bytes - must be a multiple of 8.
*
* Formats **%s**, **%p{i,I}{4,6}** requires to read kernel memory.
* Reading kernel memory may fail due to either invalid address or
* valid address but requiring a major memory fault. If reading kernel memory
* fails, the string for **%s** will be an empty string, and the ip
* address for **%p{i,I}{4,6}** will be 0. Not returning error to
* bpf program is consistent with what **bpf_trace_printk**\ () does for now.
* Return
* 0 on success, or a negative error in case of failure:
*
* **-EBUSY** if per-CPU memory copy buffer is busy, can try again
* by returning 1 from bpf program.
*
* **-EINVAL** if arguments are invalid, or if *fmt* is invalid/unsupported.
*
* **-E2BIG** if *fmt* contains too many format specifiers.
*
* **-EOVERFLOW** if an overflow happened: The same object will be tried again.
*
* long bpf_seq_write(struct seq_file *m, const void *data, u32 len)
* Description
* **bpf_seq_write**\ () uses seq_file **seq_write**\ () to write the data.
* The *m* represents the seq_file. The *data* and *len* represent the
* data to write in bytes.
* Return
* 0 on success, or a negative error in case of failure:
*
* **-EOVERFLOW** if an overflow happened: The same object will be tried again.
*
* u64 bpf_sk_cgroup_id(void *sk)
* Description
* Return the cgroup v2 id of the socket *sk*.
*
* *sk* must be a non-**NULL** pointer to a socket, e.g. one
* returned from **bpf_sk_lookup_xxx**\ (),
* **bpf_sk_fullsock**\ (), etc. The format of returned id is
* same as in **bpf_skb_cgroup_id**\ ().
*
* This helper is available only if the kernel was compiled with
* the **CONFIG_SOCK_CGROUP_DATA** configuration option.
* Return
* The id is returned or 0 in case the id could not be retrieved.
*
* u64 bpf_sk_ancestor_cgroup_id(void *sk, int ancestor_level)
* Description
* Return id of cgroup v2 that is ancestor of cgroup associated
* with the *sk* at the *ancestor_level*. The root cgroup is at
* *ancestor_level* zero and each step down the hierarchy
* increments the level. If *ancestor_level* == level of cgroup
* associated with *sk*, then return value will be same as that
* of **bpf_sk_cgroup_id**\ ().
*
* The helper is useful to implement policies based on cgroups
* that are upper in hierarchy than immediate cgroup associated
* with *sk*.
*
* The format of returned id and helper limitations are same as in
* **bpf_sk_cgroup_id**\ ().
* Return
* The id is returned or 0 in case the id could not be retrieved.
*
* long bpf_ringbuf_output(void *ringbuf, void *data, u64 size, u64 flags)
* Description
* Copy *size* bytes from *data* into a ring buffer *ringbuf*.
* If **BPF_RB_NO_WAKEUP** is specified in *flags*, no notification
* of new data availability is sent.
* If **BPF_RB_FORCE_WAKEUP** is specified in *flags*, notification
* of new data availability is sent unconditionally.
* If **0** is specified in *flags*, an adaptive notification
* of new data availability is sent.
*
* An adaptive notification is a notification sent whenever the user-space
* process has caught up and consumed all available payloads. In case the user-space
* process is still processing a previous payload, then no notification is needed
* as it will process the newly added payload automatically.
* Return
* 0 on success, or a negative error in case of failure.
*
* void *bpf_ringbuf_reserve(void *ringbuf, u64 size, u64 flags)
* Description
* Reserve *size* bytes of payload in a ring buffer *ringbuf*.
* *flags* must be 0.
* Return
* Valid pointer with *size* bytes of memory available; NULL,
* otherwise.
*
* void bpf_ringbuf_submit(void *data, u64 flags)
* Description
* Submit reserved ring buffer sample, pointed to by *data*.
* If **BPF_RB_NO_WAKEUP** is specified in *flags*, no notification
* of new data availability is sent.
* If **BPF_RB_FORCE_WAKEUP** is specified in *flags*, notification
* of new data availability is sent unconditionally.
* If **0** is specified in *flags*, an adaptive notification
* of new data availability is sent.
*
* See 'bpf_ringbuf_output()' for the definition of adaptive notification.
* Return
* Nothing. Always succeeds.
*
* void bpf_ringbuf_discard(void *data, u64 flags)
* Description
* Discard reserved ring buffer sample, pointed to by *data*.
* If **BPF_RB_NO_WAKEUP** is specified in *flags*, no notification
* of new data availability is sent. Discarded records remain in
* the ring buffer until consumed by user space, so a later submit
* using adaptive wakeup might not wake up the consumer.
* If **BPF_RB_FORCE_WAKEUP** is specified in *flags*, notification
* of new data availability is sent unconditionally.
* If **0** is specified in *flags*, an adaptive notification
* of new data availability is sent.
*
* See 'bpf_ringbuf_output()' for the definition of adaptive notification.
* Return
* Nothing. Always succeeds.
*
* u64 bpf_ringbuf_query(void *ringbuf, u64 flags)
* Description
* Query various characteristics of provided ring buffer. What
* exactly is queries is determined by *flags*:
*
* * **BPF_RB_AVAIL_DATA**: Amount of data not yet consumed.
* * **BPF_RB_RING_SIZE**: The size of ring buffer.
* * **BPF_RB_CONS_POS**: Consumer position (can wrap around).
* * **BPF_RB_PROD_POS**: Producer(s) position (can wrap around).
*
* Data returned is just a momentary snapshot of actual values
* and could be inaccurate, so this facility should be used to
* power heuristics and for reporting, not to make 100% correct
* calculation.
* Return
* Requested value, or 0, if *flags* are not recognized.
*
* long bpf_csum_level(struct sk_buff *skb, u64 level)
* Description
* Change the skbs checksum level by one layer up or down, or
* reset it entirely to none in order to have the stack perform
* checksum validation. The level is applicable to the following
* protocols: TCP, UDP, GRE, SCTP, FCOE. For example, a decap of
* | ETH | IP | UDP | GUE | IP | TCP | into | ETH | IP | TCP |
* through **bpf_skb_adjust_room**\ () helper with passing in
* **BPF_F_ADJ_ROOM_NO_CSUM_RESET** flag would require one call
* to **bpf_csum_level**\ () with **BPF_CSUM_LEVEL_DEC** since
* the UDP header is removed. Similarly, an encap of the latter
* into the former could be accompanied by a helper call to
* **bpf_csum_level**\ () with **BPF_CSUM_LEVEL_INC** if the
* skb is still intended to be processed in higher layers of the
* stack instead of just egressing at tc.
*
* There are three supported level settings at this time:
*
* * **BPF_CSUM_LEVEL_INC**: Increases skb->csum_level for skbs
* with CHECKSUM_UNNECESSARY.
* * **BPF_CSUM_LEVEL_DEC**: Decreases skb->csum_level for skbs
* with CHECKSUM_UNNECESSARY.
* * **BPF_CSUM_LEVEL_RESET**: Resets skb->csum_level to 0 and
* sets CHECKSUM_NONE to force checksum validation by the stack.
* * **BPF_CSUM_LEVEL_QUERY**: No-op, returns the current
* skb->csum_level.
* Return
* 0 on success, or a negative error in case of failure. In the
* case of **BPF_CSUM_LEVEL_QUERY**, the current skb->csum_level
* is returned or the error code -EACCES in case the skb is not
* subject to CHECKSUM_UNNECESSARY.
*
* struct tcp6_sock *bpf_skc_to_tcp6_sock(void *sk)
* Description
* Dynamically cast a *sk* pointer to a *tcp6_sock* pointer.
* Return
* *sk* if casting is valid, or **NULL** otherwise.
*
* struct tcp_sock *bpf_skc_to_tcp_sock(void *sk)
* Description
* Dynamically cast a *sk* pointer to a *tcp_sock* pointer.
* Return
* *sk* if casting is valid, or **NULL** otherwise.
*
* struct tcp_timewait_sock *bpf_skc_to_tcp_timewait_sock(void *sk)
* Description
* Dynamically cast a *sk* pointer to a *tcp_timewait_sock* pointer.
* Return
* *sk* if casting is valid, or **NULL** otherwise.
*
* struct tcp_request_sock *bpf_skc_to_tcp_request_sock(void *sk)
* Description
* Dynamically cast a *sk* pointer to a *tcp_request_sock* pointer.
* Return
* *sk* if casting is valid, or **NULL** otherwise.
*
* struct udp6_sock *bpf_skc_to_udp6_sock(void *sk)
* Description
* Dynamically cast a *sk* pointer to a *udp6_sock* pointer.
* Return
* *sk* if casting is valid, or **NULL** otherwise.
*
* long bpf_get_task_stack(struct task_struct *task, void *buf, u32 size, u64 flags)
* Description
* Return a user or a kernel stack in bpf program provided buffer.
* Note: the user stack will only be populated if the *task* is
* the current task; all other tasks will return -EOPNOTSUPP.
* To achieve this, the helper needs *task*, which is a valid
* pointer to **struct task_struct**. To store the stacktrace, the
* bpf program provides *buf* with a nonnegative *size*.
*
* The last argument, *flags*, holds the number of stack frames to
* skip (from 0 to 255), masked with
* **BPF_F_SKIP_FIELD_MASK**. The next bits can be used to set
* the following flags:
*
* **BPF_F_USER_STACK**
* Collect a user space stack instead of a kernel stack.
* The *task* must be the current task.
* **BPF_F_USER_BUILD_ID**
* Collect buildid+offset instead of ips for user stack,
* only valid if **BPF_F_USER_STACK** is also specified.
*
* **bpf_get_task_stack**\ () can collect up to
* **PERF_MAX_STACK_DEPTH** both kernel and user frames, subject
* to sufficient large buffer size. Note that
* this limit can be controlled with the **sysctl** program, and
* that it should be manually increased in order to profile long
* user stacks (such as stacks for Java programs). To do so, use:
*
* ::
*
* # sysctl kernel.perf_event_max_stack=<new value>
* Return
* The non-negative copied *buf* length equal to or less than
* *size* on success, or a negative error in case of failure.
*
* long bpf_load_hdr_opt(struct bpf_sock_ops *skops, void *searchby_res, u32 len, u64 flags)
* Description
* Load header option. Support reading a particular TCP header
* option for bpf program (**BPF_PROG_TYPE_SOCK_OPS**).
*
* If *flags* is 0, it will search the option from the
* *skops*\ **->skb_data**. The comment in **struct bpf_sock_ops**
* has details on what skb_data contains under different
* *skops*\ **->op**.
*
* The first byte of the *searchby_res* specifies the
* kind that it wants to search.
*
* If the searching kind is an experimental kind
* (i.e. 253 or 254 according to RFC6994). It also
* needs to specify the "magic" which is either
* 2 bytes or 4 bytes. It then also needs to
* specify the size of the magic by using
* the 2nd byte which is "kind-length" of a TCP
* header option and the "kind-length" also
* includes the first 2 bytes "kind" and "kind-length"
* itself as a normal TCP header option also does.
*
* For example, to search experimental kind 254 with
* 2 byte magic 0xeB9F, the searchby_res should be
* [ 254, 4, 0xeB, 0x9F, 0, 0, .... 0 ].
*
* To search for the standard window scale option (3),
* the *searchby_res* should be [ 3, 0, 0, .... 0 ].
* Note, kind-length must be 0 for regular option.
*
* Searching for No-Op (0) and End-of-Option-List (1) are
* not supported.
*
* *len* must be at least 2 bytes which is the minimal size
* of a header option.
*
* Supported flags:
*
* * **BPF_LOAD_HDR_OPT_TCP_SYN** to search from the
* saved_syn packet or the just-received syn packet.
*
* Return
* > 0 when found, the header option is copied to *searchby_res*.
* The return value is the total length copied. On failure, a
* negative error code is returned:
*
* **-EINVAL** if a parameter is invalid.
*
* **-ENOMSG** if the option is not found.
*
* **-ENOENT** if no syn packet is available when
* **BPF_LOAD_HDR_OPT_TCP_SYN** is used.
*
* **-ENOSPC** if there is not enough space. Only *len* number of
* bytes are copied.
*
* **-EFAULT** on failure to parse the header options in the
* packet.
*
* **-EPERM** if the helper cannot be used under the current
* *skops*\ **->op**.
*
* long bpf_store_hdr_opt(struct bpf_sock_ops *skops, const void *from, u32 len, u64 flags)
* Description
* Store header option. The data will be copied
* from buffer *from* with length *len* to the TCP header.
*
* The buffer *from* should have the whole option that
* includes the kind, kind-length, and the actual
* option data. The *len* must be at least kind-length
* long. The kind-length does not have to be 4 byte
* aligned. The kernel will take care of the padding
* and setting the 4 bytes aligned value to th->doff.
*
* This helper will check for duplicated option
* by searching the same option in the outgoing skb.
*
* This helper can only be called during
* **BPF_SOCK_OPS_WRITE_HDR_OPT_CB**.
*
* Return
* 0 on success, or negative error in case of failure:
*
* **-EINVAL** If param is invalid.
*
* **-ENOSPC** if there is not enough space in the header.
* Nothing has been written
*
* **-EEXIST** if the option already exists.
*
* **-EFAULT** on failure to parse the existing header options.
*
* **-EPERM** if the helper cannot be used under the current
* *skops*\ **->op**.
*
* long bpf_reserve_hdr_opt(struct bpf_sock_ops *skops, u32 len, u64 flags)
* Description
* Reserve *len* bytes for the bpf header option. The
* space will be used by **bpf_store_hdr_opt**\ () later in
* **BPF_SOCK_OPS_WRITE_HDR_OPT_CB**.
*
* If **bpf_reserve_hdr_opt**\ () is called multiple times,
* the total number of bytes will be reserved.
*
* This helper can only be called during
* **BPF_SOCK_OPS_HDR_OPT_LEN_CB**.
*
* Return
* 0 on success, or negative error in case of failure:
*
* **-EINVAL** if a parameter is invalid.
*
* **-ENOSPC** if there is not enough space in the header.
*
* **-EPERM** if the helper cannot be used under the current
* *skops*\ **->op**.
*
* void *bpf_inode_storage_get(struct bpf_map *map, void *inode, void *value, u64 flags)
* Description
* Get a bpf_local_storage from an *inode*.
*
* Logically, it could be thought of as getting the value from
* a *map* with *inode* as the **key**. From this
* perspective, the usage is not much different from
* **bpf_map_lookup_elem**\ (*map*, **&**\ *inode*) except this
* helper enforces the key must be an inode and the map must also
* be a **BPF_MAP_TYPE_INODE_STORAGE**.
*
* Underneath, the value is stored locally at *inode* instead of
* the *map*. The *map* is used as the bpf-local-storage
* "type". The bpf-local-storage "type" (i.e. the *map*) is
* searched against all bpf_local_storage residing at *inode*.
*
* An optional *flags* (**BPF_LOCAL_STORAGE_GET_F_CREATE**) can be
* used such that a new bpf_local_storage will be
* created if one does not exist. *value* can be used
* together with **BPF_LOCAL_STORAGE_GET_F_CREATE** to specify
* the initial value of a bpf_local_storage. If *value* is
* **NULL**, the new bpf_local_storage will be zero initialized.
* Return
* A bpf_local_storage pointer is returned on success.
*
* **NULL** if not found or there was an error in adding
* a new bpf_local_storage.
*
* int bpf_inode_storage_delete(struct bpf_map *map, void *inode)
* Description
* Delete a bpf_local_storage from an *inode*.
* Return
* 0 on success.
*
* **-ENOENT** if the bpf_local_storage cannot be found.
*
* long bpf_d_path(const struct path *path, char *buf, u32 sz)
* Description
* Return full path for given **struct path** object, which
* needs to be the kernel BTF *path* object. The path is
* returned in the provided buffer *buf* of size *sz* and
* is zero terminated.
*
* Return
* On success, the strictly positive length of the string,
* including the trailing NUL character. On error, a negative
* value.
*
* long bpf_copy_from_user(void *dst, u32 size, const void *user_ptr)
* Description
* Read *size* bytes from user space address *user_ptr* and store
* the data in *dst*. This is a wrapper of **copy_from_user**\ ().
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_snprintf_btf(char *str, u32 str_size, struct btf_ptr *ptr, u32 btf_ptr_size, u64 flags)
* Description
* Use BTF to store a string representation of *ptr*->ptr in *str*,
* using *ptr*->type_id. This value should specify the type
* that *ptr*->ptr points to. LLVM __builtin_btf_type_id(type, 1)
* can be used to look up vmlinux BTF type ids. Traversing the
* data structure using BTF, the type information and values are
* stored in the first *str_size* - 1 bytes of *str*. Safe copy of
* the pointer data is carried out to avoid kernel crashes during
* operation. Smaller types can use string space on the stack;
* larger programs can use map data to store the string
* representation.
*
* The string can be subsequently shared with userspace via
* bpf_perf_event_output() or ring buffer interfaces.
* bpf_trace_printk() is to be avoided as it places too small
* a limit on string size to be useful.
*
* *flags* is a combination of
*
* **BTF_F_COMPACT**
* no formatting around type information
* **BTF_F_NONAME**
* no struct/union member names/types
* **BTF_F_PTR_RAW**
* show raw (unobfuscated) pointer values;
* equivalent to printk specifier %px.
* **BTF_F_ZERO**
* show zero-valued struct/union members; they
* are not displayed by default
*
* Return
* The number of bytes that were written (or would have been
* written if output had to be truncated due to string size),
* or a negative error in cases of failure.
*
* long bpf_seq_printf_btf(struct seq_file *m, struct btf_ptr *ptr, u32 ptr_size, u64 flags)
* Description
* Use BTF to write to seq_write a string representation of
* *ptr*->ptr, using *ptr*->type_id as per bpf_snprintf_btf().
* *flags* are identical to those used for bpf_snprintf_btf.
* Return
* 0 on success or a negative error in case of failure.
*
* u64 bpf_skb_cgroup_classid(struct sk_buff *skb)
* Description
* See **bpf_get_cgroup_classid**\ () for the main description.
* This helper differs from **bpf_get_cgroup_classid**\ () in that
* the cgroup v1 net_cls class is retrieved only from the *skb*'s
* associated socket instead of the current process.
* Return
* The id is returned or 0 in case the id could not be retrieved.
*
* long bpf_redirect_neigh(u32 ifindex, struct bpf_redir_neigh *params, int plen, u64 flags)
* Description
* Redirect the packet to another net device of index *ifindex*
* and fill in L2 addresses from neighboring subsystem. This helper
* is somewhat similar to **bpf_redirect**\ (), except that it
* populates L2 addresses as well, meaning, internally, the helper
* relies on the neighbor lookup for the L2 address of the nexthop.
*
* The helper will perform a FIB lookup based on the skb's
* networking header to get the address of the next hop, unless
* this is supplied by the caller in the *params* argument. The
* *plen* argument indicates the len of *params* and should be set
* to 0 if *params* is NULL.
*
* The *flags* argument is reserved and must be 0. The helper is
* currently only supported for tc BPF program types, and enabled
* for IPv4 and IPv6 protocols.
* Return
* The helper returns **TC_ACT_REDIRECT** on success or
* **TC_ACT_SHOT** on error.
*
* void *bpf_per_cpu_ptr(const void *percpu_ptr, u32 cpu)
* Description
* Take a pointer to a percpu ksym, *percpu_ptr*, and return a
* pointer to the percpu kernel variable on *cpu*. A ksym is an
* extern variable decorated with '__ksym'. For ksym, there is a
* global var (either static or global) defined of the same name
* in the kernel. The ksym is percpu if the global var is percpu.
* The returned pointer points to the global percpu var on *cpu*.
*
* bpf_per_cpu_ptr() has the same semantic as per_cpu_ptr() in the
* kernel, except that bpf_per_cpu_ptr() may return NULL. This
* happens if *cpu* is larger than nr_cpu_ids. The caller of
* bpf_per_cpu_ptr() must check the returned value.
* Return
* A pointer pointing to the kernel percpu variable on *cpu*, or
* NULL, if *cpu* is invalid.
*
* void *bpf_this_cpu_ptr(const void *percpu_ptr)
* Description
* Take a pointer to a percpu ksym, *percpu_ptr*, and return a
* pointer to the percpu kernel variable on this cpu. See the
* description of 'ksym' in **bpf_per_cpu_ptr**\ ().
*
* bpf_this_cpu_ptr() has the same semantic as this_cpu_ptr() in
* the kernel. Different from **bpf_per_cpu_ptr**\ (), it would
* never return NULL.
* Return
* A pointer pointing to the kernel percpu variable on this cpu.
*
* long bpf_redirect_peer(u32 ifindex, u64 flags)
* Description
* Redirect the packet to another net device of index *ifindex*.
* This helper is somewhat similar to **bpf_redirect**\ (), except
* that the redirection happens to the *ifindex*' peer device and
* the netns switch takes place from ingress to ingress without
* going through the CPU's backlog queue.
*
* *skb*\ **->mark** and *skb*\ **->tstamp** are not cleared during
* the netns switch.
*
* The *flags* argument is reserved and must be 0. The helper is
* currently only supported for tc BPF program types at the
* ingress hook and for veth and netkit target device types. The
* peer device must reside in a different network namespace.
* Return
* The helper returns **TC_ACT_REDIRECT** on success or
* **TC_ACT_SHOT** on error.
*
* void *bpf_task_storage_get(struct bpf_map *map, struct task_struct *task, void *value, u64 flags)
* Description
* Get a bpf_local_storage from the *task*.
*
* Logically, it could be thought of as getting the value from
* a *map* with *task* as the **key**. From this
* perspective, the usage is not much different from
* **bpf_map_lookup_elem**\ (*map*, **&**\ *task*) except this
* helper enforces the key must be a task_struct and the map must also
* be a **BPF_MAP_TYPE_TASK_STORAGE**.
*
* Underneath, the value is stored locally at *task* instead of
* the *map*. The *map* is used as the bpf-local-storage
* "type". The bpf-local-storage "type" (i.e. the *map*) is
* searched against all bpf_local_storage residing at *task*.
*
* An optional *flags* (**BPF_LOCAL_STORAGE_GET_F_CREATE**) can be
* used such that a new bpf_local_storage will be
* created if one does not exist. *value* can be used
* together with **BPF_LOCAL_STORAGE_GET_F_CREATE** to specify
* the initial value of a bpf_local_storage. If *value* is
* **NULL**, the new bpf_local_storage will be zero initialized.
* Return
* A bpf_local_storage pointer is returned on success.
*
* **NULL** if not found or there was an error in adding
* a new bpf_local_storage.
*
* long bpf_task_storage_delete(struct bpf_map *map, struct task_struct *task)
* Description
* Delete a bpf_local_storage from a *task*.
* Return
* 0 on success.
*
* **-ENOENT** if the bpf_local_storage cannot be found.
*
* struct task_struct *bpf_get_current_task_btf(void)
* Description
* Return a BTF pointer to the "current" task.
* This pointer can also be used in helpers that accept an
* *ARG_PTR_TO_BTF_ID* of type *task_struct*.
* Return
* Pointer to the current task.
*
* long bpf_bprm_opts_set(struct linux_binprm *bprm, u64 flags)
* Description
* Set or clear certain options on *bprm*:
*
* **BPF_F_BPRM_SECUREEXEC** Set the secureexec bit
* which sets the **AT_SECURE** auxv for glibc. The bit
* is cleared if the flag is not specified.
* Return
* **-EINVAL** if invalid *flags* are passed, zero otherwise.
*
* u64 bpf_ktime_get_coarse_ns(void)
* Description
* Return a coarse-grained version of the time elapsed since
* system boot, in nanoseconds. Does not include time the system
* was suspended.
*
* See: **clock_gettime**\ (**CLOCK_MONOTONIC_COARSE**)
* Return
* Current *ktime*.
*
* long bpf_ima_inode_hash(struct inode *inode, void *dst, u32 size)
* Description
* Returns the stored IMA hash of the *inode* (if it's available).
* If the hash is larger than *size*, then only *size*
* bytes will be copied to *dst*
* Return
* The **hash_algo** is returned on success,
* **-EOPNOTSUPP** if IMA is disabled or **-EINVAL** if
* invalid arguments are passed.
*
* struct socket *bpf_sock_from_file(struct file *file)
* Description
* If the given file represents a socket, returns the associated
* socket.
* Return
* A pointer to a struct socket on success or NULL if the file is
* not a socket.
*
* long bpf_check_mtu(void *ctx, u32 ifindex, u32 *mtu_len, s32 len_diff, u64 flags)
* Description
* Check packet size against exceeding MTU of net device (based
* on *ifindex*). This helper will likely be used in combination
* with helpers that adjust/change the packet size.
*
* The argument *len_diff* can be used for querying with a planned
* size change. This allows to check MTU prior to changing packet
* ctx. Providing a *len_diff* adjustment that is larger than the
* actual packet size (resulting in negative packet size) will in
* principle not exceed the MTU, which is why it is not considered
* a failure. Other BPF helpers are needed for performing the
* planned size change; therefore the responsibility for catching
* a negative packet size belongs in those helpers.
*
* Specifying *ifindex* zero means the MTU check is performed
* against the current net device. This is practical if this isn't
* used prior to redirect.
*
* On input *mtu_len* must be a valid pointer, else verifier will
* reject BPF program. If the value *mtu_len* is initialized to
* zero then the ctx packet size is use. When value *mtu_len* is
* provided as input this specify the L3 length that the MTU check
* is done against. Remember XDP and TC length operate at L2, but
* this value is L3 as this correlate to MTU and IP-header tot_len
* values which are L3 (similar behavior as bpf_fib_lookup).
*
* The Linux kernel route table can configure MTUs on a more
* specific per route level, which is not provided by this helper.
* For route level MTU checks use the **bpf_fib_lookup**\ ()
* helper.
*
* *ctx* is either **struct xdp_md** for XDP programs or
* **struct sk_buff** for tc cls_act programs.
*
* The *flags* argument can be a combination of one or more of the
* following values:
*
* **BPF_MTU_CHK_SEGS**
* This flag will only works for *ctx* **struct sk_buff**.
* If packet context contains extra packet segment buffers
* (often knows as GSO skb), then MTU check is harder to
* check at this point, because in transmit path it is
* possible for the skb packet to get re-segmented
* (depending on net device features). This could still be
* a MTU violation, so this flag enables performing MTU
* check against segments, with a different violation
* return code to tell it apart. Check cannot use len_diff.
*
* On return *mtu_len* pointer contains the MTU value of the net
* device. Remember the net device configured MTU is the L3 size,
* which is returned here and XDP and TC length operate at L2.
* Helper take this into account for you, but remember when using
* MTU value in your BPF-code.
*
* Return
* * 0 on success, and populate MTU value in *mtu_len* pointer.
*
* * < 0 if any input argument is invalid (*mtu_len* not updated)
*
* MTU violations return positive values, but also populate MTU
* value in *mtu_len* pointer, as this can be needed for
* implementing PMTU handing:
*
* * **BPF_MTU_CHK_RET_FRAG_NEEDED**
* * **BPF_MTU_CHK_RET_SEGS_TOOBIG**
*
* long bpf_for_each_map_elem(struct bpf_map *map, void *callback_fn, void *callback_ctx, u64 flags)
* Description
* For each element in **map**, call **callback_fn** function with
* **map**, **callback_ctx** and other map-specific parameters.
* The **callback_fn** should be a static function and
* the **callback_ctx** should be a pointer to the stack.
* The **flags** is used to control certain aspects of the helper.
* Currently, the **flags** must be 0.
*
* The following are a list of supported map types and their
* respective expected callback signatures:
*
* BPF_MAP_TYPE_HASH, BPF_MAP_TYPE_PERCPU_HASH,
* BPF_MAP_TYPE_LRU_HASH, BPF_MAP_TYPE_LRU_PERCPU_HASH,
* BPF_MAP_TYPE_ARRAY, BPF_MAP_TYPE_PERCPU_ARRAY
*
* long (\*callback_fn)(struct bpf_map \*map, const void \*key, void \*value, void \*ctx);
*
* For per_cpu maps, the map_value is the value on the cpu where the
* bpf_prog is running.
*
* If **callback_fn** return 0, the helper will continue to the next
* element. If return value is 1, the helper will skip the rest of
* elements and return. Other return values are not used now.
*
* Return
* The number of traversed map elements for success, **-EINVAL** for
* invalid **flags**.
*
* long bpf_snprintf(char *str, u32 str_size, const char *fmt, u64 *data, u32 data_len)
* Description
* Outputs a string into the **str** buffer of size **str_size**
* based on a format string stored in a read-only map pointed by
* **fmt**.
*
* Each format specifier in **fmt** corresponds to one u64 element
* in the **data** array. For strings and pointers where pointees
* are accessed, only the pointer values are stored in the *data*
* array. The *data_len* is the size of *data* in bytes - must be
* a multiple of 8.
*
* Formats **%s** and **%p{i,I}{4,6}** require to read kernel
* memory. Reading kernel memory may fail due to either invalid
* address or valid address but requiring a major memory fault. If
* reading kernel memory fails, the string for **%s** will be an
* empty string, and the ip address for **%p{i,I}{4,6}** will be 0.
* Not returning error to bpf program is consistent with what
* **bpf_trace_printk**\ () does for now.
*
* Return
* The strictly positive length of the formatted string, including
* the trailing zero character. If the return value is greater than
* **str_size**, **str** contains a truncated string, guaranteed to
* be zero-terminated except when **str_size** is 0.
*
* Or **-EBUSY** if the per-CPU memory copy buffer is busy.
*
* long bpf_sys_bpf(u32 cmd, void *attr, u32 attr_size)
* Description
* Execute bpf syscall with given arguments.
* Return
* A syscall result.
*
* long bpf_btf_find_by_name_kind(char *name, int name_sz, u32 kind, int flags)
* Description
* Find BTF type with given name and kind in vmlinux BTF or in module's BTFs.
* Return
* Returns btf_id and btf_obj_fd in lower and upper 32 bits.
*
* long bpf_sys_close(u32 fd)
* Description
* Execute close syscall for given FD.
* Return
* A syscall result.
*
* long bpf_timer_init(struct bpf_timer *timer, struct bpf_map *map, u64 flags)
* Description
* Initialize the timer.
* First 4 bits of *flags* specify clockid.
* Only CLOCK_MONOTONIC, CLOCK_REALTIME, CLOCK_BOOTTIME are allowed.
* All other bits of *flags* are reserved.
* The verifier will reject the program if *timer* is not from
* the same *map*.
* Return
* 0 on success.
* **-EBUSY** if *timer* is already initialized.
* **-EINVAL** if invalid *flags* are passed.
* **-EPERM** if *timer* is in a map that doesn't have any user references.
* The user space should either hold a file descriptor to a map with timers
* or pin such map in bpffs. When map is unpinned or file descriptor is
* closed all timers in the map will be cancelled and freed.
*
* long bpf_timer_set_callback(struct bpf_timer *timer, void *callback_fn)
* Description
* Configure the timer to call *callback_fn* static function.
* Return
* 0 on success.
* **-EINVAL** if *timer* was not initialized with bpf_timer_init() earlier.
* **-EPERM** if *timer* is in a map that doesn't have any user references.
* The user space should either hold a file descriptor to a map with timers
* or pin such map in bpffs. When map is unpinned or file descriptor is
* closed all timers in the map will be cancelled and freed.
*
* long bpf_timer_start(struct bpf_timer *timer, u64 nsecs, u64 flags)
* Description
* Set timer expiration N nanoseconds from the current time. The
* configured callback will be invoked in soft irq context on some cpu
* and will not repeat unless another bpf_timer_start() is made.
* In such case the next invocation can migrate to a different cpu.
* Since struct bpf_timer is a field inside map element the map
* owns the timer. The bpf_timer_set_callback() will increment refcnt
* of BPF program to make sure that callback_fn code stays valid.
* When user space reference to a map reaches zero all timers
* in a map are cancelled and corresponding program's refcnts are
* decremented. This is done to make sure that Ctrl-C of a user
* process doesn't leave any timers running. If map is pinned in
* bpffs the callback_fn can re-arm itself indefinitely.
* bpf_map_update/delete_elem() helpers and user space sys_bpf commands
* cancel and free the timer in the given map element.
* The map can contain timers that invoke callback_fn-s from different
* programs. The same callback_fn can serve different timers from
* different maps if key/value layout matches across maps.
* Every bpf_timer_set_callback() can have different callback_fn.
*
* *flags* can be one of:
*
* **BPF_F_TIMER_ABS**
* Start the timer in absolute expire value instead of the
* default relative one.
* **BPF_F_TIMER_CPU_PIN**
* Timer will be pinned to the CPU of the caller.
*
* Return
* 0 on success.
* **-EINVAL** if *timer* was not initialized with bpf_timer_init() earlier
* or invalid *flags* are passed.
*
* long bpf_timer_cancel(struct bpf_timer *timer)
* Description
* Cancel the timer and wait for callback_fn to finish if it was running.
* Return
* 0 if the timer was not active.
* 1 if the timer was active.
* **-EINVAL** if *timer* was not initialized with bpf_timer_init() earlier.
* **-EDEADLK** if callback_fn tried to call bpf_timer_cancel() on its
* own timer which would have led to a deadlock otherwise.
*
* u64 bpf_get_func_ip(void *ctx)
* Description
* Get address of the traced function (for tracing and kprobe programs).
*
* When called for kprobe program attached as uprobe it returns
* probe address for both entry and return uprobe.
*
* Return
* Address of the traced function for kprobe.
* 0 for kprobes placed within the function (not at the entry).
* Address of the probe for uprobe and return uprobe.
*
* u64 bpf_get_attach_cookie(void *ctx)
* Description
* Get bpf_cookie value provided (optionally) during the program
* attachment. It might be different for each individual
* attachment, even if BPF program itself is the same.
* Expects BPF program context *ctx* as a first argument.
*
* Supported for the following program types:
* - kprobe/uprobe;
* - tracepoint;
* - perf_event.
* Return
* Value specified by user at BPF link creation/attachment time
* or 0, if it was not specified.
*
* long bpf_task_pt_regs(struct task_struct *task)
* Description
* Get the struct pt_regs associated with **task**.
* Return
* A pointer to struct pt_regs.
*
* long bpf_get_branch_snapshot(void *entries, u32 size, u64 flags)
* Description
* Get branch trace from hardware engines like Intel LBR. The
* hardware engine is stopped shortly after the helper is
* called. Therefore, the user need to filter branch entries
* based on the actual use case. To capture branch trace
* before the trigger point of the BPF program, the helper
* should be called at the beginning of the BPF program.
*
* The data is stored as struct perf_branch_entry into output
* buffer *entries*. *size* is the size of *entries* in bytes.
* *flags* is reserved for now and must be zero.
*
* Return
* On success, number of bytes written to *buf*. On error, a
* negative value.
*
* **-EINVAL** if *flags* is not zero.
*
* **-ENOENT** if architecture does not support branch records.
*
* long bpf_trace_vprintk(const char *fmt, u32 fmt_size, const void *data, u32 data_len)
* Description
* Behaves like **bpf_trace_printk**\ () helper, but takes an array of u64
* to format and can handle more format args as a result.
*
* Arguments are to be used as in **bpf_seq_printf**\ () helper.
* Return
* The number of bytes written to the buffer, or a negative error
* in case of failure.
*
* struct unix_sock *bpf_skc_to_unix_sock(void *sk)
* Description
* Dynamically cast a *sk* pointer to a *unix_sock* pointer.
* Return
* *sk* if casting is valid, or **NULL** otherwise.
*
* long bpf_kallsyms_lookup_name(const char *name, int name_sz, int flags, u64 *res)
* Description
* Get the address of a kernel symbol, returned in *res*. *res* is
* set to 0 if the symbol is not found.
* Return
* On success, zero. On error, a negative value.
*
* **-EINVAL** if *flags* is not zero.
*
* **-EINVAL** if string *name* is not the same size as *name_sz*.
*
* **-ENOENT** if symbol is not found.
*
* **-EPERM** if caller does not have permission to obtain kernel address.
*
* long bpf_find_vma(struct task_struct *task, u64 addr, void *callback_fn, void *callback_ctx, u64 flags)
* Description
* Find vma of *task* that contains *addr*, call *callback_fn*
* function with *task*, *vma*, and *callback_ctx*.
* The *callback_fn* should be a static function and
* the *callback_ctx* should be a pointer to the stack.
* The *flags* is used to control certain aspects of the helper.
* Currently, the *flags* must be 0.
*
* The expected callback signature is
*
* long (\*callback_fn)(struct task_struct \*task, struct vm_area_struct \*vma, void \*callback_ctx);
*
* Return
* 0 on success.
* **-ENOENT** if *task->mm* is NULL, or no vma contains *addr*.
* **-EBUSY** if failed to try lock mmap_lock.
* **-EINVAL** for invalid **flags**.
*
* long bpf_loop(u32 nr_loops, void *callback_fn, void *callback_ctx, u64 flags)
* Description
* For **nr_loops**, call **callback_fn** function
* with **callback_ctx** as the context parameter.
* The **callback_fn** should be a static function and
* the **callback_ctx** should be a pointer to the stack.
* The **flags** is used to control certain aspects of the helper.
* Currently, the **flags** must be 0. Currently, nr_loops is
* limited to 1 << 23 (~8 million) loops.
*
* long (\*callback_fn)(u64 index, void \*ctx);
*
* where **index** is the current index in the loop. The index
* is zero-indexed.
*
* If **callback_fn** returns 0, the helper will continue to the next
* loop. If return value is 1, the helper will skip the rest of
* the loops and return. Other return values are not used now,
* and will be rejected by the verifier.
*
* Return
* The number of loops performed, **-EINVAL** for invalid **flags**,
* **-E2BIG** if **nr_loops** exceeds the maximum number of loops.
*
* long bpf_strncmp(const char *s1, u32 s1_sz, const char *s2)
* Description
* Do strncmp() between **s1** and **s2**. **s1** doesn't need
* to be null-terminated and **s1_sz** is the maximum storage
* size of **s1**. **s2** must be a read-only string.
* Return
* An integer less than, equal to, or greater than zero
* if the first **s1_sz** bytes of **s1** is found to be
* less than, to match, or be greater than **s2**.
*
* long bpf_get_func_arg(void *ctx, u32 n, u64 *value)
* Description
* Get **n**-th argument register (zero based) of the traced function (for tracing programs)
* returned in **value**.
*
* Return
* 0 on success.
* **-EINVAL** if n >= argument register count of traced function.
*
* long bpf_get_func_ret(void *ctx, u64 *value)
* Description
* Get return value of the traced function (for tracing programs)
* in **value**.
*
* Return
* 0 on success.
* **-EOPNOTSUPP** for tracing programs other than BPF_TRACE_FEXIT or BPF_MODIFY_RETURN.
*
* long bpf_get_func_arg_cnt(void *ctx)
* Description
* Get number of registers of the traced function (for tracing programs) where
* function arguments are stored in these registers.
*
* Return
* The number of argument registers of the traced function.
*
* int bpf_get_retval(void)
* Description
* Get the BPF program's return value that will be returned to the upper layers.
*
* This helper is currently supported by cgroup programs and only by the hooks
* where BPF program's return value is returned to the userspace via errno.
* Return
* The BPF program's return value.
*
* int bpf_set_retval(int retval)
* Description
* Set the BPF program's return value that will be returned to the upper layers.
*
* This helper is currently supported by cgroup programs and only by the hooks
* where BPF program's return value is returned to the userspace via errno.
*
* Note that there is the following corner case where the program exports an error
* via bpf_set_retval but signals success via 'return 1':
*
* bpf_set_retval(-EPERM);
* return 1;
*
* In this case, the BPF program's return value will use helper's -EPERM. This
* still holds true for cgroup/bind{4,6} which supports extra 'return 3' success case.
*
* Return
* 0 on success, or a negative error in case of failure.
*
* u64 bpf_xdp_get_buff_len(struct xdp_buff *xdp_md)
* Description
* Get the total size of a given xdp buff (linear and paged area)
* Return
* The total size of a given xdp buffer.
*
* long bpf_xdp_load_bytes(struct xdp_buff *xdp_md, u32 offset, void *buf, u32 len)
* Description
* This helper is provided as an easy way to load data from a
* xdp buffer. It can be used to load *len* bytes from *offset* from
* the frame associated to *xdp_md*, into the buffer pointed by
* *buf*.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_xdp_store_bytes(struct xdp_buff *xdp_md, u32 offset, void *buf, u32 len)
* Description
* Store *len* bytes from buffer *buf* into the frame
* associated to *xdp_md*, at *offset*.
* Return
* 0 on success, or a negative error in case of failure.
*
* long bpf_copy_from_user_task(void *dst, u32 size, const void *user_ptr, struct task_struct *tsk, u64 flags)
* Description
* Read *size* bytes from user space address *user_ptr* in *tsk*'s
* address space, and stores the data in *dst*. *flags* is not
* used yet and is provided for future extensibility. This helper
* can only be used by sleepable programs.
* Return
* 0 on success, or a negative error in case of failure. On error
* *dst* buffer is zeroed out.
*
* long bpf_skb_set_tstamp(struct sk_buff *skb, u64 tstamp, u32 tstamp_type)
* Description
* Change the __sk_buff->tstamp_type to *tstamp_type*
* and set *tstamp* to the __sk_buff->tstamp together.
*
* If there is no need to change the __sk_buff->tstamp_type,
* the tstamp value can be directly written to __sk_buff->tstamp
* instead.
*
* BPF_SKB_TSTAMP_DELIVERY_MONO is the only tstamp that
* will be kept during bpf_redirect_*(). A non zero
* *tstamp* must be used with the BPF_SKB_TSTAMP_DELIVERY_MONO
* *tstamp_type*.
*
* A BPF_SKB_TSTAMP_UNSPEC *tstamp_type* can only be used
* with a zero *tstamp*.
*
* Only IPv4 and IPv6 skb->protocol are supported.
*
* This function is most useful when it needs to set a
* mono delivery time to __sk_buff->tstamp and then
* bpf_redirect_*() to the egress of an iface. For example,
* changing the (rcv) timestamp in __sk_buff->tstamp at
* ingress to a mono delivery time and then bpf_redirect_*()
* to sch_fq@phy-dev.
* Return
* 0 on success.
* **-EINVAL** for invalid input
* **-EOPNOTSUPP** for unsupported protocol
*
* long bpf_ima_file_hash(struct file *file, void *dst, u32 size)
* Description
* Returns a calculated IMA hash of the *file*.
* If the hash is larger than *size*, then only *size*
* bytes will be copied to *dst*
* Return
* The **hash_algo** is returned on success,
* **-EOPNOTSUPP** if the hash calculation failed or **-EINVAL** if
* invalid arguments are passed.
*
* void *bpf_kptr_xchg(void *dst, void *ptr)
* Description
* Exchange kptr at pointer *dst* with *ptr*, and return the old value.
* *dst* can be map value or local kptr. *ptr* can be NULL, otherwise
* it must be a referenced pointer which will be released when this helper
* is called.
* Return
* The old value of kptr (which can be NULL). The returned pointer
* if not NULL, is a reference which must be released using its
* corresponding release function, or moved into a BPF map before
* program exit.
*
* void *bpf_map_lookup_percpu_elem(struct bpf_map *map, const void *key, u32 cpu)
* Description
* Perform a lookup in *percpu map* for an entry associated to
* *key* on *cpu*.
* Return
* Map value associated to *key* on *cpu*, or **NULL** if no entry
* was found or *cpu* is invalid.
*
* struct mptcp_sock *bpf_skc_to_mptcp_sock(void *sk)
* Description
* Dynamically cast a *sk* pointer to a *mptcp_sock* pointer.
* Return
* *sk* if casting is valid, or **NULL** otherwise.
*
* long bpf_dynptr_from_mem(void *data, u64 size, u64 flags, struct bpf_dynptr *ptr)
* Description
* Get a dynptr to local memory *data*.
*
* *data* must be a ptr to a map value.
* The maximum *size* supported is DYNPTR_MAX_SIZE.
* *flags* is currently unused.
* Return
* 0 on success, -E2BIG if the size exceeds DYNPTR_MAX_SIZE,
* -EINVAL if flags is not 0.
*
* long bpf_ringbuf_reserve_dynptr(void *ringbuf, u32 size, u64 flags, struct bpf_dynptr *ptr)
* Description
* Reserve *size* bytes of payload in a ring buffer *ringbuf*
* through the dynptr interface. *flags* must be 0.
*
* Please note that a corresponding bpf_ringbuf_submit_dynptr or
* bpf_ringbuf_discard_dynptr must be called on *ptr*, even if the
* reservation fails. This is enforced by the verifier.
* Return
* 0 on success, or a negative error in case of failure.
*
* void bpf_ringbuf_submit_dynptr(struct bpf_dynptr *ptr, u64 flags)
* Description
* Submit reserved ring buffer sample, pointed to by *data*,
* through the dynptr interface. This is a no-op if the dynptr is
* invalid/null.
*
* For more information on *flags*, please see
* 'bpf_ringbuf_submit'.
* Return
* Nothing. Always succeeds.
*
* void bpf_ringbuf_discard_dynptr(struct bpf_dynptr *ptr, u64 flags)
* Description
* Discard reserved ring buffer sample through the dynptr
* interface. This is a no-op if the dynptr is invalid/null.
*
* For more information on *flags*, please see
* 'bpf_ringbuf_discard'.
* Return
* Nothing. Always succeeds.
*
* long bpf_dynptr_read(void *dst, u64 len, const struct bpf_dynptr *src, u64 offset, u64 flags)
* Description
* Read *len* bytes from *src* into *dst*, starting from *offset*
* into *src*.
* *flags* is currently unused.
* Return
* 0 on success, -E2BIG if *offset* + *len* exceeds the length
* of *src*'s data, -EINVAL if *src* is an invalid dynptr or if
* *flags* is not 0.
*
* long bpf_dynptr_write(const struct bpf_dynptr *dst, u64 offset, void *src, u64 len, u64 flags)
* Description
* Write *len* bytes from *src* into *dst*, starting from *offset*
* into *dst*.
*
* *flags* must be 0 except for skb-type dynptrs.
*
* For skb-type dynptrs:
* * All data slices of the dynptr are automatically
* invalidated after **bpf_dynptr_write**\ (). This is
* because writing may pull the skb and change the
* underlying packet buffer.
*
* * For *flags*, please see the flags accepted by
* **bpf_skb_store_bytes**\ ().
* Return
* 0 on success, -E2BIG if *offset* + *len* exceeds the length
* of *dst*'s data, -EINVAL if *dst* is an invalid dynptr or if *dst*
* is a read-only dynptr or if *flags* is not correct. For skb-type dynptrs,
* other errors correspond to errors returned by **bpf_skb_store_bytes**\ ().
*
* void *bpf_dynptr_data(const struct bpf_dynptr *ptr, u64 offset, u64 len)
* Description
* Get a pointer to the underlying dynptr data.
*
* *len* must be a statically known value. The returned data slice
* is invalidated whenever the dynptr is invalidated.
*
* skb and xdp type dynptrs may not use bpf_dynptr_data. They should
* instead use bpf_dynptr_slice and bpf_dynptr_slice_rdwr.
* Return
* Pointer to the underlying dynptr data, NULL if the dynptr is
* read-only, if the dynptr is invalid, or if the offset and length
* is out of bounds.
*
* s64 bpf_tcp_raw_gen_syncookie_ipv4(struct iphdr *iph, struct tcphdr *th, u32 th_len)
* Description
* Try to issue a SYN cookie for the packet with corresponding
* IPv4/TCP headers, *iph* and *th*, without depending on a
* listening socket.
*
* *iph* points to the IPv4 header.
*
* *th* points to the start of the TCP header, while *th_len*
* contains the length of the TCP header (at least
* **sizeof**\ (**struct tcphdr**)).
* Return
* On success, lower 32 bits hold the generated SYN cookie in
* followed by 16 bits which hold the MSS value for that cookie,
* and the top 16 bits are unused.
*
* On failure, the returned value is one of the following:
*
* **-EINVAL** if *th_len* is invalid.
*
* s64 bpf_tcp_raw_gen_syncookie_ipv6(struct ipv6hdr *iph, struct tcphdr *th, u32 th_len)
* Description
* Try to issue a SYN cookie for the packet with corresponding
* IPv6/TCP headers, *iph* and *th*, without depending on a
* listening socket.
*
* *iph* points to the IPv6 header.
*
* *th* points to the start of the TCP header, while *th_len*
* contains the length of the TCP header (at least
* **sizeof**\ (**struct tcphdr**)).
* Return
* On success, lower 32 bits hold the generated SYN cookie in
* followed by 16 bits which hold the MSS value for that cookie,
* and the top 16 bits are unused.
*
* On failure, the returned value is one of the following:
*
* **-EINVAL** if *th_len* is invalid.
*
* **-EPROTONOSUPPORT** if CONFIG_IPV6 is not builtin.
*
* long bpf_tcp_raw_check_syncookie_ipv4(struct iphdr *iph, struct tcphdr *th)
* Description
* Check whether *iph* and *th* contain a valid SYN cookie ACK
* without depending on a listening socket.
*
* *iph* points to the IPv4 header.
*
* *th* points to the TCP header.
* Return
* 0 if *iph* and *th* are a valid SYN cookie ACK.
*
* On failure, the returned value is one of the following:
*
* **-EACCES** if the SYN cookie is not valid.
*
* long bpf_tcp_raw_check_syncookie_ipv6(struct ipv6hdr *iph, struct tcphdr *th)
* Description
* Check whether *iph* and *th* contain a valid SYN cookie ACK
* without depending on a listening socket.
*
* *iph* points to the IPv6 header.
*
* *th* points to the TCP header.
* Return
* 0 if *iph* and *th* are a valid SYN cookie ACK.
*
* On failure, the returned value is one of the following:
*
* **-EACCES** if the SYN cookie is not valid.
*
* **-EPROTONOSUPPORT** if CONFIG_IPV6 is not builtin.
*
* u64 bpf_ktime_get_tai_ns(void)
* Description
* A nonsettable system-wide clock derived from wall-clock time but
* ignoring leap seconds. This clock does not experience
* discontinuities and backwards jumps caused by NTP inserting leap
* seconds as CLOCK_REALTIME does.
*
* See: **clock_gettime**\ (**CLOCK_TAI**)
* Return
* Current *ktime*.
*
* long bpf_user_ringbuf_drain(struct bpf_map *map, void *callback_fn, void *ctx, u64 flags)
* Description
* Drain samples from the specified user ring buffer, and invoke
* the provided callback for each such sample:
*
* long (\*callback_fn)(const struct bpf_dynptr \*dynptr, void \*ctx);
*
* If **callback_fn** returns 0, the helper will continue to try
* and drain the next sample, up to a maximum of
* BPF_MAX_USER_RINGBUF_SAMPLES samples. If the return value is 1,
* the helper will skip the rest of the samples and return. Other
* return values are not used now, and will be rejected by the
* verifier.
* Return
* The number of drained samples if no error was encountered while
* draining samples, or 0 if no samples were present in the ring
* buffer. If a user-space producer was epoll-waiting on this map,
* and at least one sample was drained, they will receive an event
* notification notifying them of available space in the ring
* buffer. If the BPF_RB_NO_WAKEUP flag is passed to this
* function, no wakeup notification will be sent. If the
* BPF_RB_FORCE_WAKEUP flag is passed, a wakeup notification will
* be sent even if no sample was drained.
*
* On failure, the returned value is one of the following:
*
* **-EBUSY** if the ring buffer is contended, and another calling
* context was concurrently draining the ring buffer.
*
* **-EINVAL** if user-space is not properly tracking the ring
* buffer due to the producer position not being aligned to 8
* bytes, a sample not being aligned to 8 bytes, or the producer
* position not matching the advertised length of a sample.
*
* **-E2BIG** if user-space has tried to publish a sample which is
* larger than the size of the ring buffer, or which cannot fit
* within a struct bpf_dynptr.
*
* void *bpf_cgrp_storage_get(struct bpf_map *map, struct cgroup *cgroup, void *value, u64 flags)
* Description
* Get a bpf_local_storage from the *cgroup*.
*
* Logically, it could be thought of as getting the value from
* a *map* with *cgroup* as the **key**. From this
* perspective, the usage is not much different from
* **bpf_map_lookup_elem**\ (*map*, **&**\ *cgroup*) except this
* helper enforces the key must be a cgroup struct and the map must also
* be a **BPF_MAP_TYPE_CGRP_STORAGE**.
*
* In reality, the local-storage value is embedded directly inside of the
* *cgroup* object itself, rather than being located in the
* **BPF_MAP_TYPE_CGRP_STORAGE** map. When the local-storage value is
* queried for some *map* on a *cgroup* object, the kernel will perform an
* O(n) iteration over all of the live local-storage values for that
* *cgroup* object until the local-storage value for the *map* is found.
*
* An optional *flags* (**BPF_LOCAL_STORAGE_GET_F_CREATE**) can be
* used such that a new bpf_local_storage will be
* created if one does not exist. *value* can be used
* together with **BPF_LOCAL_STORAGE_GET_F_CREATE** to specify
* the initial value of a bpf_local_storage. If *value* is
* **NULL**, the new bpf_local_storage will be zero initialized.
* Return
* A bpf_local_storage pointer is returned on success.
*
* **NULL** if not found or there was an error in adding
* a new bpf_local_storage.
*
* long bpf_cgrp_storage_delete(struct bpf_map *map, struct cgroup *cgroup)
* Description
* Delete a bpf_local_storage from a *cgroup*.
* Return
* 0 on success.
*
* **-ENOENT** if the bpf_local_storage cannot be found.
*/
#define ___BPF_FUNC_MAPPER(FN, ctx...) \
FN(unspec, 0, ##ctx) \
FN(map_lookup_elem, 1, ##ctx) \
FN(map_update_elem, 2, ##ctx) \
FN(map_delete_elem, 3, ##ctx) \
FN(probe_read, 4, ##ctx) \
FN(ktime_get_ns, 5, ##ctx) \
FN(trace_printk, 6, ##ctx) \
FN(get_prandom_u32, 7, ##ctx) \
FN(get_smp_processor_id, 8, ##ctx) \
FN(skb_store_bytes, 9, ##ctx) \
FN(l3_csum_replace, 10, ##ctx) \
FN(l4_csum_replace, 11, ##ctx) \
FN(tail_call, 12, ##ctx) \
FN(clone_redirect, 13, ##ctx) \
FN(get_current_pid_tgid, 14, ##ctx) \
FN(get_current_uid_gid, 15, ##ctx) \
FN(get_current_comm, 16, ##ctx) \
FN(get_cgroup_classid, 17, ##ctx) \
FN(skb_vlan_push, 18, ##ctx) \
FN(skb_vlan_pop, 19, ##ctx) \
FN(skb_get_tunnel_key, 20, ##ctx) \
FN(skb_set_tunnel_key, 21, ##ctx) \
FN(perf_event_read, 22, ##ctx) \
FN(redirect, 23, ##ctx) \
FN(get_route_realm, 24, ##ctx) \
FN(perf_event_output, 25, ##ctx) \
FN(skb_load_bytes, 26, ##ctx) \
FN(get_stackid, 27, ##ctx) \
FN(csum_diff, 28, ##ctx) \
FN(skb_get_tunnel_opt, 29, ##ctx) \
FN(skb_set_tunnel_opt, 30, ##ctx) \
FN(skb_change_proto, 31, ##ctx) \
FN(skb_change_type, 32, ##ctx) \
FN(skb_under_cgroup, 33, ##ctx) \
FN(get_hash_recalc, 34, ##ctx) \
FN(get_current_task, 35, ##ctx) \
FN(probe_write_user, 36, ##ctx) \
FN(current_task_under_cgroup, 37, ##ctx) \
FN(skb_change_tail, 38, ##ctx) \
FN(skb_pull_data, 39, ##ctx) \
FN(csum_update, 40, ##ctx) \
FN(set_hash_invalid, 41, ##ctx) \
FN(get_numa_node_id, 42, ##ctx) \
FN(skb_change_head, 43, ##ctx) \
FN(xdp_adjust_head, 44, ##ctx) \
FN(probe_read_str, 45, ##ctx) \
FN(get_socket_cookie, 46, ##ctx) \
FN(get_socket_uid, 47, ##ctx) \
FN(set_hash, 48, ##ctx) \
FN(setsockopt, 49, ##ctx) \
FN(skb_adjust_room, 50, ##ctx) \
FN(redirect_map, 51, ##ctx) \
FN(sk_redirect_map, 52, ##ctx) \
FN(sock_map_update, 53, ##ctx) \
FN(xdp_adjust_meta, 54, ##ctx) \
FN(perf_event_read_value, 55, ##ctx) \
FN(perf_prog_read_value, 56, ##ctx) \
FN(getsockopt, 57, ##ctx) \
FN(override_return, 58, ##ctx) \
FN(sock_ops_cb_flags_set, 59, ##ctx) \
FN(msg_redirect_map, 60, ##ctx) \
FN(msg_apply_bytes, 61, ##ctx) \
FN(msg_cork_bytes, 62, ##ctx) \
FN(msg_pull_data, 63, ##ctx) \
FN(bind, 64, ##ctx) \
FN(xdp_adjust_tail, 65, ##ctx) \
FN(skb_get_xfrm_state, 66, ##ctx) \
FN(get_stack, 67, ##ctx) \
FN(skb_load_bytes_relative, 68, ##ctx) \
FN(fib_lookup, 69, ##ctx) \
FN(sock_hash_update, 70, ##ctx) \
FN(msg_redirect_hash, 71, ##ctx) \
FN(sk_redirect_hash, 72, ##ctx) \
FN(lwt_push_encap, 73, ##ctx) \
FN(lwt_seg6_store_bytes, 74, ##ctx) \
FN(lwt_seg6_adjust_srh, 75, ##ctx) \
FN(lwt_seg6_action, 76, ##ctx) \
FN(rc_repeat, 77, ##ctx) \
FN(rc_keydown, 78, ##ctx) \
FN(skb_cgroup_id, 79, ##ctx) \
FN(get_current_cgroup_id, 80, ##ctx) \
FN(get_local_storage, 81, ##ctx) \
FN(sk_select_reuseport, 82, ##ctx) \
FN(skb_ancestor_cgroup_id, 83, ##ctx) \
FN(sk_lookup_tcp, 84, ##ctx) \
FN(sk_lookup_udp, 85, ##ctx) \
FN(sk_release, 86, ##ctx) \
FN(map_push_elem, 87, ##ctx) \
FN(map_pop_elem, 88, ##ctx) \
FN(map_peek_elem, 89, ##ctx) \
FN(msg_push_data, 90, ##ctx) \
FN(msg_pop_data, 91, ##ctx) \
FN(rc_pointer_rel, 92, ##ctx) \
FN(spin_lock, 93, ##ctx) \
FN(spin_unlock, 94, ##ctx) \
FN(sk_fullsock, 95, ##ctx) \
FN(tcp_sock, 96, ##ctx) \
FN(skb_ecn_set_ce, 97, ##ctx) \
FN(get_listener_sock, 98, ##ctx) \
FN(skc_lookup_tcp, 99, ##ctx) \
FN(tcp_check_syncookie, 100, ##ctx) \
FN(sysctl_get_name, 101, ##ctx) \
FN(sysctl_get_current_value, 102, ##ctx) \
FN(sysctl_get_new_value, 103, ##ctx) \
FN(sysctl_set_new_value, 104, ##ctx) \
FN(strtol, 105, ##ctx) \
FN(strtoul, 106, ##ctx) \
FN(sk_storage_get, 107, ##ctx) \
FN(sk_storage_delete, 108, ##ctx) \
FN(send_signal, 109, ##ctx) \
FN(tcp_gen_syncookie, 110, ##ctx) \
FN(skb_output, 111, ##ctx) \
FN(probe_read_user, 112, ##ctx) \
FN(probe_read_kernel, 113, ##ctx) \
FN(probe_read_user_str, 114, ##ctx) \
FN(probe_read_kernel_str, 115, ##ctx) \
FN(tcp_send_ack, 116, ##ctx) \
FN(send_signal_thread, 117, ##ctx) \
FN(jiffies64, 118, ##ctx) \
FN(read_branch_records, 119, ##ctx) \
FN(get_ns_current_pid_tgid, 120, ##ctx) \
FN(xdp_output, 121, ##ctx) \
FN(get_netns_cookie, 122, ##ctx) \
FN(get_current_ancestor_cgroup_id, 123, ##ctx) \
FN(sk_assign, 124, ##ctx) \
FN(ktime_get_boot_ns, 125, ##ctx) \
FN(seq_printf, 126, ##ctx) \
FN(seq_write, 127, ##ctx) \
FN(sk_cgroup_id, 128, ##ctx) \
FN(sk_ancestor_cgroup_id, 129, ##ctx) \
FN(ringbuf_output, 130, ##ctx) \
FN(ringbuf_reserve, 131, ##ctx) \
FN(ringbuf_submit, 132, ##ctx) \
FN(ringbuf_discard, 133, ##ctx) \
FN(ringbuf_query, 134, ##ctx) \
FN(csum_level, 135, ##ctx) \
FN(skc_to_tcp6_sock, 136, ##ctx) \
FN(skc_to_tcp_sock, 137, ##ctx) \
FN(skc_to_tcp_timewait_sock, 138, ##ctx) \
FN(skc_to_tcp_request_sock, 139, ##ctx) \
FN(skc_to_udp6_sock, 140, ##ctx) \
FN(get_task_stack, 141, ##ctx) \
FN(load_hdr_opt, 142, ##ctx) \
FN(store_hdr_opt, 143, ##ctx) \
FN(reserve_hdr_opt, 144, ##ctx) \
FN(inode_storage_get, 145, ##ctx) \
FN(inode_storage_delete, 146, ##ctx) \
FN(d_path, 147, ##ctx) \
FN(copy_from_user, 148, ##ctx) \
FN(snprintf_btf, 149, ##ctx) \
FN(seq_printf_btf, 150, ##ctx) \
FN(skb_cgroup_classid, 151, ##ctx) \
FN(redirect_neigh, 152, ##ctx) \
FN(per_cpu_ptr, 153, ##ctx) \
FN(this_cpu_ptr, 154, ##ctx) \
FN(redirect_peer, 155, ##ctx) \
FN(task_storage_get, 156, ##ctx) \
FN(task_storage_delete, 157, ##ctx) \
FN(get_current_task_btf, 158, ##ctx) \
FN(bprm_opts_set, 159, ##ctx) \
FN(ktime_get_coarse_ns, 160, ##ctx) \
FN(ima_inode_hash, 161, ##ctx) \
FN(sock_from_file, 162, ##ctx) \
FN(check_mtu, 163, ##ctx) \
FN(for_each_map_elem, 164, ##ctx) \
FN(snprintf, 165, ##ctx) \
FN(sys_bpf, 166, ##ctx) \
FN(btf_find_by_name_kind, 167, ##ctx) \
FN(sys_close, 168, ##ctx) \
FN(timer_init, 169, ##ctx) \
FN(timer_set_callback, 170, ##ctx) \
FN(timer_start, 171, ##ctx) \
FN(timer_cancel, 172, ##ctx) \
FN(get_func_ip, 173, ##ctx) \
FN(get_attach_cookie, 174, ##ctx) \
FN(task_pt_regs, 175, ##ctx) \
FN(get_branch_snapshot, 176, ##ctx) \
FN(trace_vprintk, 177, ##ctx) \
FN(skc_to_unix_sock, 178, ##ctx) \
FN(kallsyms_lookup_name, 179, ##ctx) \
FN(find_vma, 180, ##ctx) \
FN(loop, 181, ##ctx) \
FN(strncmp, 182, ##ctx) \
FN(get_func_arg, 183, ##ctx) \
FN(get_func_ret, 184, ##ctx) \
FN(get_func_arg_cnt, 185, ##ctx) \
FN(get_retval, 186, ##ctx) \
FN(set_retval, 187, ##ctx) \
FN(xdp_get_buff_len, 188, ##ctx) \
FN(xdp_load_bytes, 189, ##ctx) \
FN(xdp_store_bytes, 190, ##ctx) \
FN(copy_from_user_task, 191, ##ctx) \
FN(skb_set_tstamp, 192, ##ctx) \
FN(ima_file_hash, 193, ##ctx) \
FN(kptr_xchg, 194, ##ctx) \
FN(map_lookup_percpu_elem, 195, ##ctx) \
FN(skc_to_mptcp_sock, 196, ##ctx) \
FN(dynptr_from_mem, 197, ##ctx) \
FN(ringbuf_reserve_dynptr, 198, ##ctx) \
FN(ringbuf_submit_dynptr, 199, ##ctx) \
FN(ringbuf_discard_dynptr, 200, ##ctx) \
FN(dynptr_read, 201, ##ctx) \
FN(dynptr_write, 202, ##ctx) \
FN(dynptr_data, 203, ##ctx) \
FN(tcp_raw_gen_syncookie_ipv4, 204, ##ctx) \
FN(tcp_raw_gen_syncookie_ipv6, 205, ##ctx) \
FN(tcp_raw_check_syncookie_ipv4, 206, ##ctx) \
FN(tcp_raw_check_syncookie_ipv6, 207, ##ctx) \
FN(ktime_get_tai_ns, 208, ##ctx) \
FN(user_ringbuf_drain, 209, ##ctx) \
FN(cgrp_storage_get, 210, ##ctx) \
FN(cgrp_storage_delete, 211, ##ctx) \
/* This helper list is effectively frozen. If you are trying to \
* add a new helper, you should add a kfunc instead which has \
* less stability guarantees. See Documentation/bpf/kfuncs.rst \
*/
/* backwards-compatibility macros for users of __BPF_FUNC_MAPPER that don't
* know or care about integer value that is now passed as second argument
*/
#define __BPF_FUNC_MAPPER_APPLY(name, value, FN) FN(name),
#define __BPF_FUNC_MAPPER(FN) ___BPF_FUNC_MAPPER(__BPF_FUNC_MAPPER_APPLY, FN)
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call
*/
#define __BPF_ENUM_FN(x, y) BPF_FUNC_ ## x = y,
enum bpf_func_id {
___BPF_FUNC_MAPPER(__BPF_ENUM_FN)
__BPF_FUNC_MAX_ID,
};
#undef __BPF_ENUM_FN
/* All flags used by eBPF helper functions, placed here. */
/* BPF_FUNC_skb_store_bytes flags. */
enum {
BPF_F_RECOMPUTE_CSUM = (1ULL << 0),
BPF_F_INVALIDATE_HASH = (1ULL << 1),
};
/* BPF_FUNC_l3_csum_replace and BPF_FUNC_l4_csum_replace flags.
* First 4 bits are for passing the header field size.
*/
enum {
BPF_F_HDR_FIELD_MASK = 0xfULL,
};
/* BPF_FUNC_l4_csum_replace flags. */
enum {
BPF_F_PSEUDO_HDR = (1ULL << 4),
BPF_F_MARK_MANGLED_0 = (1ULL << 5),
BPF_F_MARK_ENFORCE = (1ULL << 6),
BPF_F_IPV6 = (1ULL << 7),
};
/* BPF_FUNC_skb_set_tunnel_key and BPF_FUNC_skb_get_tunnel_key flags. */
enum {
BPF_F_TUNINFO_IPV6 = (1ULL << 0),
};
/* flags for both BPF_FUNC_get_stackid and BPF_FUNC_get_stack. */
enum {
BPF_F_SKIP_FIELD_MASK = 0xffULL,
BPF_F_USER_STACK = (1ULL << 8),
/* flags used by BPF_FUNC_get_stackid only. */
BPF_F_FAST_STACK_CMP = (1ULL << 9),
BPF_F_REUSE_STACKID = (1ULL << 10),
/* flags used by BPF_FUNC_get_stack only. */
BPF_F_USER_BUILD_ID = (1ULL << 11),
};
/* BPF_FUNC_skb_set_tunnel_key flags. */
enum {
BPF_F_ZERO_CSUM_TX = (1ULL << 1),
BPF_F_DONT_FRAGMENT = (1ULL << 2),
BPF_F_SEQ_NUMBER = (1ULL << 3),
BPF_F_NO_TUNNEL_KEY = (1ULL << 4),
};
/* BPF_FUNC_skb_get_tunnel_key flags. */
enum {
BPF_F_TUNINFO_FLAGS = (1ULL << 4),
};
/* BPF_FUNC_perf_event_output, BPF_FUNC_perf_event_read and
* BPF_FUNC_perf_event_read_value flags.
*/
enum {
BPF_F_INDEX_MASK = 0xffffffffULL,
BPF_F_CURRENT_CPU = BPF_F_INDEX_MASK,
/* BPF_FUNC_perf_event_output for sk_buff input context. */
BPF_F_CTXLEN_MASK = (0xfffffULL << 32),
};
/* Current network namespace */
enum {
BPF_F_CURRENT_NETNS = (-1L),
};
/* BPF_FUNC_csum_level level values. */
enum {
BPF_CSUM_LEVEL_QUERY,
BPF_CSUM_LEVEL_INC,
BPF_CSUM_LEVEL_DEC,
BPF_CSUM_LEVEL_RESET,
};
/* BPF_FUNC_skb_adjust_room flags. */
enum {
BPF_F_ADJ_ROOM_FIXED_GSO = (1ULL << 0),
BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 = (1ULL << 1),
BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 = (1ULL << 2),
BPF_F_ADJ_ROOM_ENCAP_L4_GRE = (1ULL << 3),
BPF_F_ADJ_ROOM_ENCAP_L4_UDP = (1ULL << 4),
BPF_F_ADJ_ROOM_NO_CSUM_RESET = (1ULL << 5),
BPF_F_ADJ_ROOM_ENCAP_L2_ETH = (1ULL << 6),
BPF_F_ADJ_ROOM_DECAP_L3_IPV4 = (1ULL << 7),
BPF_F_ADJ_ROOM_DECAP_L3_IPV6 = (1ULL << 8),
};
enum {
BPF_ADJ_ROOM_ENCAP_L2_MASK = 0xff,
BPF_ADJ_ROOM_ENCAP_L2_SHIFT = 56,
};
#define BPF_F_ADJ_ROOM_ENCAP_L2(len) (((__u64)len & \
BPF_ADJ_ROOM_ENCAP_L2_MASK) \
<< BPF_ADJ_ROOM_ENCAP_L2_SHIFT)
/* BPF_FUNC_sysctl_get_name flags. */
enum {
BPF_F_SYSCTL_BASE_NAME = (1ULL << 0),
};
/* BPF_FUNC_<kernel_obj>_storage_get flags */
enum {
BPF_LOCAL_STORAGE_GET_F_CREATE = (1ULL << 0),
/* BPF_SK_STORAGE_GET_F_CREATE is only kept for backward compatibility
* and BPF_LOCAL_STORAGE_GET_F_CREATE must be used instead.
*/
BPF_SK_STORAGE_GET_F_CREATE = BPF_LOCAL_STORAGE_GET_F_CREATE,
};
/* BPF_FUNC_read_branch_records flags. */
enum {
BPF_F_GET_BRANCH_RECORDS_SIZE = (1ULL << 0),
};
/* BPF_FUNC_bpf_ringbuf_commit, BPF_FUNC_bpf_ringbuf_discard, and
* BPF_FUNC_bpf_ringbuf_output flags.
*/
enum {
BPF_RB_NO_WAKEUP = (1ULL << 0),
BPF_RB_FORCE_WAKEUP = (1ULL << 1),
};
/* BPF_FUNC_bpf_ringbuf_query flags */
enum {
BPF_RB_AVAIL_DATA = 0,
BPF_RB_RING_SIZE = 1,
BPF_RB_CONS_POS = 2,
BPF_RB_PROD_POS = 3,
BPF_RB_OVERWRITE_POS = 4,
};
/* BPF ring buffer constants */
enum {
BPF_RINGBUF_BUSY_BIT = (1U << 31),
BPF_RINGBUF_DISCARD_BIT = (1U << 30),
BPF_RINGBUF_HDR_SZ = 8,
};
/* BPF_FUNC_sk_assign flags in bpf_sk_lookup context. */
enum {
BPF_SK_LOOKUP_F_REPLACE = (1ULL << 0),
BPF_SK_LOOKUP_F_NO_REUSEPORT = (1ULL << 1),
};
/* Mode for BPF_FUNC_skb_adjust_room helper. */
enum bpf_adj_room_mode {
BPF_ADJ_ROOM_NET,
BPF_ADJ_ROOM_MAC,
};
/* Mode for BPF_FUNC_skb_load_bytes_relative helper. */
enum bpf_hdr_start_off {
BPF_HDR_START_MAC,
BPF_HDR_START_NET,
};
/* Encapsulation type for BPF_FUNC_lwt_push_encap helper. */
enum bpf_lwt_encap_mode {
BPF_LWT_ENCAP_SEG6,
BPF_LWT_ENCAP_SEG6_INLINE,
BPF_LWT_ENCAP_IP,
};
/* Flags for bpf_bprm_opts_set helper */
enum {
BPF_F_BPRM_SECUREEXEC = (1ULL << 0),
};
/* Flags for bpf_redirect and bpf_redirect_map helpers */
enum {
BPF_F_INGRESS = (1ULL << 0), /* used for skb path */
BPF_F_BROADCAST = (1ULL << 3), /* used for XDP path */
BPF_F_EXCLUDE_INGRESS = (1ULL << 4), /* used for XDP path */
#define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS)
};
#define __bpf_md_ptr(type, name) \
union { \
type name; \
__u64 :64; \
} __attribute__((aligned(8)))
/* The enum used in skb->tstamp_type. It specifies the clock type
* of the time stored in the skb->tstamp.
*/
enum {
BPF_SKB_TSTAMP_UNSPEC = 0, /* DEPRECATED */
BPF_SKB_TSTAMP_DELIVERY_MONO = 1, /* DEPRECATED */
BPF_SKB_CLOCK_REALTIME = 0,
BPF_SKB_CLOCK_MONOTONIC = 1,
BPF_SKB_CLOCK_TAI = 2,
/* For any future BPF_SKB_CLOCK_* that the bpf prog cannot handle,
* the bpf prog can try to deduce it by ingress/egress/skb->sk->sk_clockid.
*/
};
/* user accessible mirror of in-kernel sk_buff.
* new fields can only be added to the end of this structure
*/
struct __sk_buff {
__u32 len;
__u32 pkt_type;
__u32 mark;
__u32 queue_mapping;
__u32 protocol;
__u32 vlan_present;
__u32 vlan_tci;
__u32 vlan_proto;
__u32 priority;
__u32 ingress_ifindex;
__u32 ifindex;
__u32 tc_index;
__u32 cb[5];
__u32 hash;
__u32 tc_classid;
__u32 data;
__u32 data_end;
__u32 napi_id;
/* Accessed by BPF_PROG_TYPE_sk_skb types from here to ... */
__u32 family;
__u32 remote_ip4; /* Stored in network byte order */
__u32 local_ip4; /* Stored in network byte order */
__u32 remote_ip6[4]; /* Stored in network byte order */
__u32 local_ip6[4]; /* Stored in network byte order */
__u32 remote_port; /* Stored in network byte order */
__u32 local_port; /* stored in host byte order */
/* ... here. */
__u32 data_meta;
__bpf_md_ptr(struct bpf_flow_keys *, flow_keys);
__u64 tstamp;
__u32 wire_len;
__u32 gso_segs;
__bpf_md_ptr(struct bpf_sock *, sk);
__u32 gso_size;
__u8 tstamp_type;
__u32 :24; /* Padding, future use. */
__u64 hwtstamp;
};
struct bpf_tunnel_key {
__u32 tunnel_id;
union {
__u32 remote_ipv4;
__u32 remote_ipv6[4];
};
__u8 tunnel_tos;
__u8 tunnel_ttl;
union {
__u16 tunnel_ext; /* compat */
__be16 tunnel_flags;
};
__u32 tunnel_label;
union {
__u32 local_ipv4;
__u32 local_ipv6[4];
};
};
/* user accessible mirror of in-kernel xfrm_state.
* new fields can only be added to the end of this structure
*/
struct bpf_xfrm_state {
__u32 reqid;
__u32 spi; /* Stored in network byte order */
__u16 family;
__u16 ext; /* Padding, future use. */
union {
__u32 remote_ipv4; /* Stored in network byte order */
__u32 remote_ipv6[4]; /* Stored in network byte order */
};
};
/* Generic BPF return codes which all BPF program types may support.
* The values are binary compatible with their TC_ACT_* counter-part to
* provide backwards compatibility with existing SCHED_CLS and SCHED_ACT
* programs.
*
* XDP is handled seprately, see XDP_*.
*/
enum bpf_ret_code {
BPF_OK = 0,
/* 1 reserved */
BPF_DROP = 2,
/* 3-6 reserved */
BPF_REDIRECT = 7,
/* >127 are reserved for prog type specific return codes.
*
* BPF_LWT_REROUTE: used by BPF_PROG_TYPE_LWT_IN and
* BPF_PROG_TYPE_LWT_XMIT to indicate that skb had been
* changed and should be routed based on its new L3 header.
* (This is an L3 redirect, as opposed to L2 redirect
* represented by BPF_REDIRECT above).
*/
BPF_LWT_REROUTE = 128,
/* BPF_FLOW_DISSECTOR_CONTINUE: used by BPF_PROG_TYPE_FLOW_DISSECTOR
* to indicate that no custom dissection was performed, and
* fallback to standard dissector is requested.
*/
BPF_FLOW_DISSECTOR_CONTINUE = 129,
};
struct bpf_sock {
__u32 bound_dev_if;
__u32 family;
__u32 type;
__u32 protocol;
__u32 mark;
__u32 priority;
/* IP address also allows 1 and 2 bytes access */
__u32 src_ip4;
__u32 src_ip6[4];
__u32 src_port; /* host byte order */
__be16 dst_port; /* network byte order */
__u16 :16; /* zero padding */
__u32 dst_ip4;
__u32 dst_ip6[4];
__u32 state;
__s32 rx_queue_mapping;
};
struct bpf_tcp_sock {
__u32 snd_cwnd; /* Sending congestion window */
__u32 srtt_us; /* smoothed round trip time << 3 in usecs */
__u32 rtt_min;
__u32 snd_ssthresh; /* Slow start size threshold */
__u32 rcv_nxt; /* What we want to receive next */
__u32 snd_nxt; /* Next sequence we send */
__u32 snd_una; /* First byte we want an ack for */
__u32 mss_cache; /* Cached effective mss, not including SACKS */
__u32 ecn_flags; /* ECN status bits. */
__u32 rate_delivered; /* saved rate sample: packets delivered */
__u32 rate_interval_us; /* saved rate sample: time elapsed */
__u32 packets_out; /* Packets which are "in flight" */
__u32 retrans_out; /* Retransmitted packets out */
__u32 total_retrans; /* Total retransmits for entire connection */
__u32 segs_in; /* RFC4898 tcpEStatsPerfSegsIn
* total number of segments in.
*/
__u32 data_segs_in; /* RFC4898 tcpEStatsPerfDataSegsIn
* total number of data segments in.
*/
__u32 segs_out; /* RFC4898 tcpEStatsPerfSegsOut
* The total number of segments sent.
*/
__u32 data_segs_out; /* RFC4898 tcpEStatsPerfDataSegsOut
* total number of data segments sent.
*/
__u32 lost_out; /* Lost packets */
__u32 sacked_out; /* SACK'd packets */
__u64 bytes_received; /* RFC4898 tcpEStatsAppHCThruOctetsReceived
* sum(delta(rcv_nxt)), or how many bytes
* were acked.
*/
__u64 bytes_acked; /* RFC4898 tcpEStatsAppHCThruOctetsAcked
* sum(delta(snd_una)), or how many bytes
* were acked.
*/
__u32 dsack_dups; /* RFC4898 tcpEStatsStackDSACKDups
* total number of DSACK blocks received
*/
__u32 delivered; /* Total data packets delivered incl. rexmits */
__u32 delivered_ce; /* Like the above but only ECE marked packets */
__u32 icsk_retransmits; /* Number of unrecovered [RTO] timeouts */
};
struct bpf_sock_tuple {
union {
struct {
__be32 saddr;
__be32 daddr;
__be16 sport;
__be16 dport;
} ipv4;
struct {
__be32 saddr[4];
__be32 daddr[4];
__be16 sport;
__be16 dport;
} ipv6;
};
};
/* (Simplified) user return codes for tcx prog type.
* A valid tcx program must return one of these defined values. All other
* return codes are reserved for future use. Must remain compatible with
* their TC_ACT_* counter-parts. For compatibility in behavior, unknown
* return codes are mapped to TCX_NEXT.
*/
enum tcx_action_base {
TCX_NEXT = -1,
TCX_PASS = 0,
TCX_DROP = 2,
TCX_REDIRECT = 7,
};
struct bpf_xdp_sock {
__u32 queue_id;
};
#define XDP_PACKET_HEADROOM 256
/* User return codes for XDP prog type.
* A valid XDP program must return one of these defined values. All other
* return codes are reserved for future use. Unknown return codes will
* result in packet drops and a warning via bpf_warn_invalid_xdp_action().
*/
enum xdp_action {
XDP_ABORTED = 0,
XDP_DROP,
XDP_PASS,
XDP_TX,
XDP_REDIRECT,
};
/* user accessible metadata for XDP packet hook
* new fields must be added to the end of this structure
*/
struct xdp_md {
__u32 data;
__u32 data_end;
__u32 data_meta;
/* Below access go through struct xdp_rxq_info */
__u32 ingress_ifindex; /* rxq->dev->ifindex */
__u32 rx_queue_index; /* rxq->queue_index */
__u32 egress_ifindex; /* txq->dev->ifindex */
};
/* DEVMAP map-value layout
*
* The struct data-layout of map-value is a configuration interface.
* New members can only be added to the end of this structure.
*/
struct bpf_devmap_val {
__u32 ifindex; /* device index */
union {
int fd; /* prog fd on map write */
__u32 id; /* prog id on map read */
} bpf_prog;
};
/* CPUMAP map-value layout
*
* The struct data-layout of map-value is a configuration interface.
* New members can only be added to the end of this structure.
*/
struct bpf_cpumap_val {
__u32 qsize; /* queue size to remote target CPU */
union {
int fd; /* prog fd on map write */
__u32 id; /* prog id on map read */
} bpf_prog;
};
enum sk_action {
SK_DROP = 0,
SK_PASS,
};
/* user accessible metadata for SK_MSG packet hook, new fields must
* be added to the end of this structure
*/
struct sk_msg_md {
__bpf_md_ptr(void *, data);
__bpf_md_ptr(void *, data_end);
__u32 family;
__u32 remote_ip4; /* Stored in network byte order */
__u32 local_ip4; /* Stored in network byte order */
__u32 remote_ip6[4]; /* Stored in network byte order */
__u32 local_ip6[4]; /* Stored in network byte order */
__u32 remote_port; /* Stored in network byte order */
__u32 local_port; /* stored in host byte order */
__u32 size; /* Total size of sk_msg */
__bpf_md_ptr(struct bpf_sock *, sk); /* current socket */
};
struct sk_reuseport_md {
/*
* Start of directly accessible data. It begins from
* the tcp/udp header.
*/
__bpf_md_ptr(void *, data);
/* End of directly accessible data */
__bpf_md_ptr(void *, data_end);
/*
* Total length of packet (starting from the tcp/udp header).
* Note that the directly accessible bytes (data_end - data)
* could be less than this "len". Those bytes could be
* indirectly read by a helper "bpf_skb_load_bytes()".
*/
__u32 len;
/*
* Eth protocol in the mac header (network byte order). e.g.
* ETH_P_IP(0x0800) and ETH_P_IPV6(0x86DD)
*/
__u32 eth_protocol;
__u32 ip_protocol; /* IP protocol. e.g. IPPROTO_TCP, IPPROTO_UDP */
__u32 bind_inany; /* Is sock bound to an INANY address? */
__u32 hash; /* A hash of the packet 4 tuples */
/* When reuse->migrating_sk is NULL, it is selecting a sk for the
* new incoming connection request (e.g. selecting a listen sk for
* the received SYN in the TCP case). reuse->sk is one of the sk
* in the reuseport group. The bpf prog can use reuse->sk to learn
* the local listening ip/port without looking into the skb.
*
* When reuse->migrating_sk is not NULL, reuse->sk is closed and
* reuse->migrating_sk is the socket that needs to be migrated
* to another listening socket. migrating_sk could be a fullsock
* sk that is fully established or a reqsk that is in-the-middle
* of 3-way handshake.
*/
__bpf_md_ptr(struct bpf_sock *, sk);
__bpf_md_ptr(struct bpf_sock *, migrating_sk);
};
#define BPF_TAG_SIZE 8
struct bpf_prog_info {
__u32 type;
__u32 id;
__u8 tag[BPF_TAG_SIZE];
__u32 jited_prog_len;
__u32 xlated_prog_len;
__aligned_u64 jited_prog_insns;
__aligned_u64 xlated_prog_insns;
__u64 load_time; /* ns since boottime */
__u32 created_by_uid;
__u32 nr_map_ids;
__aligned_u64 map_ids;
char name[BPF_OBJ_NAME_LEN];
__u32 ifindex;
__u32 gpl_compatible:1;
__u32 :31; /* alignment pad */
__u64 netns_dev;
__u64 netns_ino;
__u32 nr_jited_ksyms;
__u32 nr_jited_func_lens;
__aligned_u64 jited_ksyms;
__aligned_u64 jited_func_lens;
__u32 btf_id;
__u32 func_info_rec_size;
__aligned_u64 func_info;
__u32 nr_func_info;
__u32 nr_line_info;
__aligned_u64 line_info;
__aligned_u64 jited_line_info;
__u32 nr_jited_line_info;
__u32 line_info_rec_size;
__u32 jited_line_info_rec_size;
__u32 nr_prog_tags;
__aligned_u64 prog_tags;
__u64 run_time_ns;
__u64 run_cnt;
__u64 recursion_misses;
__u32 verified_insns;
__u32 attach_btf_obj_id;
__u32 attach_btf_id;
} __attribute__((aligned(8)));
struct bpf_map_info {
__u32 type;
__u32 id;
__u32 key_size;
__u32 value_size;
__u32 max_entries;
__u32 map_flags;
char name[BPF_OBJ_NAME_LEN];
__u32 ifindex;
__u32 btf_vmlinux_value_type_id;
__u64 netns_dev;
__u64 netns_ino;
__u32 btf_id;
__u32 btf_key_type_id;
__u32 btf_value_type_id;
__u32 btf_vmlinux_id;
__u64 map_extra;
__aligned_u64 hash;
__u32 hash_size;
} __attribute__((aligned(8)));
struct bpf_btf_info {
__aligned_u64 btf;
__u32 btf_size;
__u32 id;
__aligned_u64 name;
__u32 name_len;
__u32 kernel_btf;
} __attribute__((aligned(8)));
struct bpf_link_info {
__u32 type;
__u32 id;
__u32 prog_id;
union {
struct {
__aligned_u64 tp_name; /* in/out: tp_name buffer ptr */
__u32 tp_name_len; /* in/out: tp_name buffer len */
__u32 :32;
__u64 cookie;
} raw_tracepoint;
struct {
__u32 attach_type;
__u32 target_obj_id; /* prog_id for PROG_EXT, otherwise btf object id */
__u32 target_btf_id; /* BTF type id inside the object */
__u32 :32;
__u64 cookie;
} tracing;
struct {
__u64 cgroup_id;
__u32 attach_type;
} cgroup;
struct {
__aligned_u64 target_name; /* in/out: target_name buffer ptr */
__u32 target_name_len; /* in/out: target_name buffer len */
/* If the iter specific field is 32 bits, it can be put
* in the first or second union. Otherwise it should be
* put in the second union.
*/
union {
struct {
__u32 map_id;
} map;
};
union {
struct {
__u64 cgroup_id;
__u32 order;
} cgroup;
struct {
__u32 tid;
__u32 pid;
} task;
};
} iter;
struct {
__u32 netns_ino;
__u32 attach_type;
} netns;
struct {
__u32 ifindex;
} xdp;
struct {
__u32 map_id;
} struct_ops;
struct {
__u32 pf;
__u32 hooknum;
__s32 priority;
__u32 flags;
} netfilter;
struct {
__aligned_u64 addrs;
__u32 count; /* in/out: kprobe_multi function count */
__u32 flags;
__u64 missed;
__aligned_u64 cookies;
} kprobe_multi;
struct {
__aligned_u64 path;
__aligned_u64 offsets;
__aligned_u64 ref_ctr_offsets;
__aligned_u64 cookies;
__u32 path_size; /* in/out: real path size on success, including zero byte */
__u32 count; /* in/out: uprobe_multi offsets/ref_ctr_offsets/cookies count */
__u32 flags;
__u32 pid;
} uprobe_multi;
struct {
__u32 type; /* enum bpf_perf_event_type */
__u32 :32;
union {
struct {
__aligned_u64 file_name; /* in/out */
__u32 name_len;
__u32 offset; /* offset from file_name */
__u64 cookie;
__u64 ref_ctr_offset;
} uprobe; /* BPF_PERF_EVENT_UPROBE, BPF_PERF_EVENT_URETPROBE */
struct {
__aligned_u64 func_name; /* in/out */
__u32 name_len;
__u32 offset; /* offset from func_name */
__u64 addr;
__u64 missed;
__u64 cookie;
} kprobe; /* BPF_PERF_EVENT_KPROBE, BPF_PERF_EVENT_KRETPROBE */
struct {
__aligned_u64 tp_name; /* in/out */
__u32 name_len;
__u32 :32;
__u64 cookie;
} tracepoint; /* BPF_PERF_EVENT_TRACEPOINT */
struct {
__u64 config;
__u32 type;
__u32 :32;
__u64 cookie;
} event; /* BPF_PERF_EVENT_EVENT */
};
} perf_event;
struct {
__u32 ifindex;
__u32 attach_type;
} tcx;
struct {
__u32 ifindex;
__u32 attach_type;
} netkit;
struct {
__u32 map_id;
__u32 attach_type;
} sockmap;
};
} __attribute__((aligned(8)));
struct bpf_token_info {
__u64 allowed_cmds;
__u64 allowed_maps;
__u64 allowed_progs;
__u64 allowed_attachs;
} __attribute__((aligned(8)));
/* User bpf_sock_addr struct to access socket fields and sockaddr struct passed
* by user and intended to be used by socket (e.g. to bind to, depends on
* attach type).
*/
struct bpf_sock_addr {
__u32 user_family; /* Allows 4-byte read, but no write. */
__u32 user_ip4; /* Allows 1,2,4-byte read and 4-byte write.
* Stored in network byte order.
*/
__u32 user_ip6[4]; /* Allows 1,2,4,8-byte read and 4,8-byte write.
* Stored in network byte order.
*/
__u32 user_port; /* Allows 1,2,4-byte read and 4-byte write.
* Stored in network byte order
*/
__u32 family; /* Allows 4-byte read, but no write */
__u32 type; /* Allows 4-byte read, but no write */
__u32 protocol; /* Allows 4-byte read, but no write */
__u32 msg_src_ip4; /* Allows 1,2,4-byte read and 4-byte write.
* Stored in network byte order.
*/
__u32 msg_src_ip6[4]; /* Allows 1,2,4,8-byte read and 4,8-byte write.
* Stored in network byte order.
*/
__bpf_md_ptr(struct bpf_sock *, sk);
};
/* User bpf_sock_ops struct to access socket values and specify request ops
* and their replies.
* Some of this fields are in network (bigendian) byte order and may need
* to be converted before use (bpf_ntohl() defined in samples/bpf/bpf_endian.h).
* New fields can only be added at the end of this structure
*/
struct bpf_sock_ops {
__u32 op;
union {
__u32 args[4]; /* Optionally passed to bpf program */
__u32 reply; /* Returned by bpf program */
__u32 replylong[4]; /* Optionally returned by bpf prog */
};
__u32 family;
__u32 remote_ip4; /* Stored in network byte order */
__u32 local_ip4; /* Stored in network byte order */
__u32 remote_ip6[4]; /* Stored in network byte order */
__u32 local_ip6[4]; /* Stored in network byte order */
__u32 remote_port; /* Stored in network byte order */
__u32 local_port; /* stored in host byte order */
__u32 is_fullsock; /* Some TCP fields are only valid if
* there is a full socket. If not, the
* fields read as zero.
*/
__u32 snd_cwnd;
__u32 srtt_us; /* Averaged RTT << 3 in usecs */
__u32 bpf_sock_ops_cb_flags; /* flags defined in uapi/linux/tcp.h */
__u32 state;
__u32 rtt_min;
__u32 snd_ssthresh;
__u32 rcv_nxt;
__u32 snd_nxt;
__u32 snd_una;
__u32 mss_cache;
__u32 ecn_flags;
__u32 rate_delivered;
__u32 rate_interval_us;
__u32 packets_out;
__u32 retrans_out;
__u32 total_retrans;
__u32 segs_in;
__u32 data_segs_in;
__u32 segs_out;
__u32 data_segs_out;
__u32 lost_out;
__u32 sacked_out;
__u32 sk_txhash;
__u64 bytes_received;
__u64 bytes_acked;
__bpf_md_ptr(struct bpf_sock *, sk);
/* [skb_data, skb_data_end) covers the whole TCP header.
*
* BPF_SOCK_OPS_PARSE_HDR_OPT_CB: The packet received
* BPF_SOCK_OPS_HDR_OPT_LEN_CB: Not useful because the
* header has not been written.
* BPF_SOCK_OPS_WRITE_HDR_OPT_CB: The header and options have
* been written so far.
* BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB: The SYNACK that concludes
* the 3WHS.
* BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB: The ACK that concludes
* the 3WHS.
*
* bpf_load_hdr_opt() can also be used to read a particular option.
*/
__bpf_md_ptr(void *, skb_data);
__bpf_md_ptr(void *, skb_data_end);
__u32 skb_len; /* The total length of a packet.
* It includes the header, options,
* and payload.
*/
__u32 skb_tcp_flags; /* tcp_flags of the header. It provides
* an easy way to check for tcp_flags
* without parsing skb_data.
*
* In particular, the skb_tcp_flags
* will still be available in
* BPF_SOCK_OPS_HDR_OPT_LEN even though
* the outgoing header has not
* been written yet.
*/
__u64 skb_hwtstamp;
};
/* Definitions for bpf_sock_ops_cb_flags */
enum {
BPF_SOCK_OPS_RTO_CB_FLAG = (1<<0),
BPF_SOCK_OPS_RETRANS_CB_FLAG = (1<<1),
BPF_SOCK_OPS_STATE_CB_FLAG = (1<<2),
BPF_SOCK_OPS_RTT_CB_FLAG = (1<<3),
/* Call bpf for all received TCP headers. The bpf prog will be
* called under sock_ops->op == BPF_SOCK_OPS_PARSE_HDR_OPT_CB
*
* Please refer to the comment in BPF_SOCK_OPS_PARSE_HDR_OPT_CB
* for the header option related helpers that will be useful
* to the bpf programs.
*
* It could be used at the client/active side (i.e. connect() side)
* when the server told it that the server was in syncookie
* mode and required the active side to resend the bpf-written
* options. The active side can keep writing the bpf-options until
* it received a valid packet from the server side to confirm
* the earlier packet (and options) has been received. The later
* example patch is using it like this at the active side when the
* server is in syncookie mode.
*
* The bpf prog will usually turn this off in the common cases.
*/
BPF_SOCK_OPS_PARSE_ALL_HDR_OPT_CB_FLAG = (1<<4),
/* Call bpf when kernel has received a header option that
* the kernel cannot handle. The bpf prog will be called under
* sock_ops->op == BPF_SOCK_OPS_PARSE_HDR_OPT_CB.
*
* Please refer to the comment in BPF_SOCK_OPS_PARSE_HDR_OPT_CB
* for the header option related helpers that will be useful
* to the bpf programs.
*/
BPF_SOCK_OPS_PARSE_UNKNOWN_HDR_OPT_CB_FLAG = (1<<5),
/* Call bpf when the kernel is writing header options for the
* outgoing packet. The bpf prog will first be called
* to reserve space in a skb under
* sock_ops->op == BPF_SOCK_OPS_HDR_OPT_LEN_CB. Then
* the bpf prog will be called to write the header option(s)
* under sock_ops->op == BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
*
* Please refer to the comment in BPF_SOCK_OPS_HDR_OPT_LEN_CB
* and BPF_SOCK_OPS_WRITE_HDR_OPT_CB for the header option
* related helpers that will be useful to the bpf programs.
*
* The kernel gets its chance to reserve space and write
* options first before the BPF program does.
*/
BPF_SOCK_OPS_WRITE_HDR_OPT_CB_FLAG = (1<<6),
/* Mask of all currently supported cb flags */
BPF_SOCK_OPS_ALL_CB_FLAGS = 0x7F,
};
enum {
SK_BPF_CB_TX_TIMESTAMPING = 1<<0,
SK_BPF_CB_MASK = (SK_BPF_CB_TX_TIMESTAMPING - 1) |
SK_BPF_CB_TX_TIMESTAMPING
};
/* List of known BPF sock_ops operators.
* New entries can only be added at the end
*/
enum {
BPF_SOCK_OPS_VOID,
BPF_SOCK_OPS_TIMEOUT_INIT, /* Should return SYN-RTO value to use or
* -1 if default value should be used
*/
BPF_SOCK_OPS_RWND_INIT, /* Should return initial advertized
* window (in packets) or -1 if default
* value should be used
*/
BPF_SOCK_OPS_TCP_CONNECT_CB, /* Calls BPF program right before an
* active connection is initialized
*/
BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB, /* Calls BPF program when an
* active connection is
* established
*/
BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB, /* Calls BPF program when a
* passive connection is
* established
*/
BPF_SOCK_OPS_NEEDS_ECN, /* If connection's congestion control
* needs ECN
*/
BPF_SOCK_OPS_BASE_RTT, /* Get base RTT. The correct value is
* based on the path and may be
* dependent on the congestion control
* algorithm. In general it indicates
* a congestion threshold. RTTs above
* this indicate congestion
*/
BPF_SOCK_OPS_RTO_CB, /* Called when an RTO has triggered.
* Arg1: value of icsk_retransmits
* Arg2: value of icsk_rto
* Arg3: whether RTO has expired
*/
BPF_SOCK_OPS_RETRANS_CB, /* Called when skb is retransmitted.
* Arg1: sequence number of 1st byte
* Arg2: # segments
* Arg3: return value of
* tcp_transmit_skb (0 => success)
*/
BPF_SOCK_OPS_STATE_CB, /* Called when TCP changes state.
* Arg1: old_state
* Arg2: new_state
*/
BPF_SOCK_OPS_TCP_LISTEN_CB, /* Called on listen(2), right after
* socket transition to LISTEN state.
*/
BPF_SOCK_OPS_RTT_CB, /* Called on every RTT.
* Arg1: measured RTT input (mrtt)
* Arg2: updated srtt
*/
BPF_SOCK_OPS_PARSE_HDR_OPT_CB, /* Parse the header option.
* It will be called to handle
* the packets received at
* an already established
* connection.
*
* sock_ops->skb_data:
* Referring to the received skb.
* It covers the TCP header only.
*
* bpf_load_hdr_opt() can also
* be used to search for a
* particular option.
*/
BPF_SOCK_OPS_HDR_OPT_LEN_CB, /* Reserve space for writing the
* header option later in
* BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
* Arg1: bool want_cookie. (in
* writing SYNACK only)
*
* sock_ops->skb_data:
* Not available because no header has
* been written yet.
*
* sock_ops->skb_tcp_flags:
* The tcp_flags of the
* outgoing skb. (e.g. SYN, ACK, FIN).
*
* bpf_reserve_hdr_opt() should
* be used to reserve space.
*/
BPF_SOCK_OPS_WRITE_HDR_OPT_CB, /* Write the header options
* Arg1: bool want_cookie. (in
* writing SYNACK only)
*
* sock_ops->skb_data:
* Referring to the outgoing skb.
* It covers the TCP header
* that has already been written
* by the kernel and the
* earlier bpf-progs.
*
* sock_ops->skb_tcp_flags:
* The tcp_flags of the outgoing
* skb. (e.g. SYN, ACK, FIN).
*
* bpf_store_hdr_opt() should
* be used to write the
* option.
*
* bpf_load_hdr_opt() can also
* be used to search for a
* particular option that
* has already been written
* by the kernel or the
* earlier bpf-progs.
*/
BPF_SOCK_OPS_TSTAMP_SCHED_CB, /* Called when skb is passing
* through dev layer when
* SK_BPF_CB_TX_TIMESTAMPING
* feature is on.
*/
BPF_SOCK_OPS_TSTAMP_SND_SW_CB, /* Called when skb is about to send
* to the nic when SK_BPF_CB_TX_TIMESTAMPING
* feature is on.
*/
BPF_SOCK_OPS_TSTAMP_SND_HW_CB, /* Called in hardware phase when
* SK_BPF_CB_TX_TIMESTAMPING feature
* is on.
*/
BPF_SOCK_OPS_TSTAMP_ACK_CB, /* Called when all the skbs in the
* same sendmsg call are acked
* when SK_BPF_CB_TX_TIMESTAMPING
* feature is on.
*/
BPF_SOCK_OPS_TSTAMP_SENDMSG_CB, /* Called when every sendmsg syscall
* is triggered. It's used to correlate
* sendmsg timestamp with corresponding
* tskey.
*/
};
/* List of TCP states. There is a build check in net/ipv4/tcp.c to detect
* changes between the TCP and BPF versions. Ideally this should never happen.
* If it does, we need to add code to convert them before calling
* the BPF sock_ops function.
*/
enum {
BPF_TCP_ESTABLISHED = 1,
BPF_TCP_SYN_SENT,
BPF_TCP_SYN_RECV,
BPF_TCP_FIN_WAIT1,
BPF_TCP_FIN_WAIT2,
BPF_TCP_TIME_WAIT,
BPF_TCP_CLOSE,
BPF_TCP_CLOSE_WAIT,
BPF_TCP_LAST_ACK,
BPF_TCP_LISTEN,
BPF_TCP_CLOSING, /* Now a valid state */
BPF_TCP_NEW_SYN_RECV,
BPF_TCP_BOUND_INACTIVE,
BPF_TCP_MAX_STATES /* Leave at the end! */
};
enum {
TCP_BPF_IW = 1001, /* Set TCP initial congestion window */
TCP_BPF_SNDCWND_CLAMP = 1002, /* Set sndcwnd_clamp */
TCP_BPF_DELACK_MAX = 1003, /* Max delay ack in usecs */
TCP_BPF_RTO_MIN = 1004, /* Min delay ack in usecs */
/* Copy the SYN pkt to optval
*
* BPF_PROG_TYPE_SOCK_OPS only. It is similar to the
* bpf_getsockopt(TCP_SAVED_SYN) but it does not limit
* to only getting from the saved_syn. It can either get the
* syn packet from:
*
* 1. the just-received SYN packet (only available when writing the
* SYNACK). It will be useful when it is not necessary to
* save the SYN packet for latter use. It is also the only way
* to get the SYN during syncookie mode because the syn
* packet cannot be saved during syncookie.
*
* OR
*
* 2. the earlier saved syn which was done by
* bpf_setsockopt(TCP_SAVE_SYN).
*
* The bpf_getsockopt(TCP_BPF_SYN*) option will hide where the
* SYN packet is obtained.
*
* If the bpf-prog does not need the IP[46] header, the
* bpf-prog can avoid parsing the IP header by using
* TCP_BPF_SYN. Otherwise, the bpf-prog can get both
* IP[46] and TCP header by using TCP_BPF_SYN_IP.
*
* >0: Total number of bytes copied
* -ENOSPC: Not enough space in optval. Only optlen number of
* bytes is copied.
* -ENOENT: The SYN skb is not available now and the earlier SYN pkt
* is not saved by setsockopt(TCP_SAVE_SYN).
*/
TCP_BPF_SYN = 1005, /* Copy the TCP header */
TCP_BPF_SYN_IP = 1006, /* Copy the IP[46] and TCP header */
TCP_BPF_SYN_MAC = 1007, /* Copy the MAC, IP[46], and TCP header */
TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */
SK_BPF_CB_FLAGS = 1009, /* Get or set sock ops flags in socket */
SK_BPF_BYPASS_PROT_MEM = 1010, /* Get or Set sk->sk_bypass_prot_mem */
};
enum {
BPF_LOAD_HDR_OPT_TCP_SYN = (1ULL << 0),
};
/* args[0] value during BPF_SOCK_OPS_HDR_OPT_LEN_CB and
* BPF_SOCK_OPS_WRITE_HDR_OPT_CB.
*/
enum {
BPF_WRITE_HDR_TCP_CURRENT_MSS = 1, /* Kernel is finding the
* total option spaces
* required for an established
* sk in order to calculate the
* MSS. No skb is actually
* sent.
*/
BPF_WRITE_HDR_TCP_SYNACK_COOKIE = 2, /* Kernel is in syncookie mode
* when sending a SYN.
*/
};
struct bpf_perf_event_value {
__u64 counter;
__u64 enabled;
__u64 running;
};
enum {
BPF_DEVCG_ACC_MKNOD = (1ULL << 0),
BPF_DEVCG_ACC_READ = (1ULL << 1),
BPF_DEVCG_ACC_WRITE = (1ULL << 2),
};
enum {
BPF_DEVCG_DEV_BLOCK = (1ULL << 0),
BPF_DEVCG_DEV_CHAR = (1ULL << 1),
};
struct bpf_cgroup_dev_ctx {
/* access_type encoded as (BPF_DEVCG_ACC_* << 16) | BPF_DEVCG_DEV_* */
__u32 access_type;
__u32 major;
__u32 minor;
};
struct bpf_raw_tracepoint_args {
__u64 args[0];
};
/* DIRECT: Skip the FIB rules and go to FIB table associated with device
* OUTPUT: Do lookup from egress perspective; default is ingress
*/
enum {
BPF_FIB_LOOKUP_DIRECT = (1U << 0),
BPF_FIB_LOOKUP_OUTPUT = (1U << 1),
BPF_FIB_LOOKUP_SKIP_NEIGH = (1U << 2),
BPF_FIB_LOOKUP_TBID = (1U << 3),
BPF_FIB_LOOKUP_SRC = (1U << 4),
BPF_FIB_LOOKUP_MARK = (1U << 5),
};
enum {
BPF_FIB_LKUP_RET_SUCCESS, /* lookup successful */
BPF_FIB_LKUP_RET_BLACKHOLE, /* dest is blackholed; can be dropped */
BPF_FIB_LKUP_RET_UNREACHABLE, /* dest is unreachable; can be dropped */
BPF_FIB_LKUP_RET_PROHIBIT, /* dest not allowed; can be dropped */
BPF_FIB_LKUP_RET_NOT_FWDED, /* packet is not forwarded */
BPF_FIB_LKUP_RET_FWD_DISABLED, /* fwding is not enabled on ingress */
BPF_FIB_LKUP_RET_UNSUPP_LWT, /* fwd requires encapsulation */
BPF_FIB_LKUP_RET_NO_NEIGH, /* no neighbor entry for nh */
BPF_FIB_LKUP_RET_FRAG_NEEDED, /* fragmentation required to fwd */
BPF_FIB_LKUP_RET_NO_SRC_ADDR, /* failed to derive IP src addr */
};
struct bpf_fib_lookup {
/* input: network family for lookup (AF_INET, AF_INET6)
* output: network family of egress nexthop
*/
__u8 family;
/* set if lookup is to consider L4 data - e.g., FIB rules */
__u8 l4_protocol;
__be16 sport;
__be16 dport;
union { /* used for MTU check */
/* input to lookup */
__u16 tot_len; /* L3 length from network hdr (iph->tot_len) */
/* output: MTU value */
__u16 mtu_result;
} __attribute__((packed, aligned(2)));
/* input: L3 device index for lookup
* output: device index from FIB lookup
*/
__u32 ifindex;
union {
/* inputs to lookup */
__u8 tos; /* AF_INET */
__be32 flowinfo; /* AF_INET6, flow_label + priority */
/* output: metric of fib result (IPv4/IPv6 only) */
__u32 rt_metric;
};
/* input: source address to consider for lookup
* output: source address result from lookup
*/
union {
__be32 ipv4_src;
__u32 ipv6_src[4]; /* in6_addr; network order */
};
/* input to bpf_fib_lookup, ipv{4,6}_dst is destination address in
* network header. output: bpf_fib_lookup sets to gateway address
* if FIB lookup returns gateway route
*/
union {
__be32 ipv4_dst;
__u32 ipv6_dst[4]; /* in6_addr; network order */
};
union {
struct {
/* output */
__be16 h_vlan_proto;
__be16 h_vlan_TCI;
};
/* input: when accompanied with the
* 'BPF_FIB_LOOKUP_DIRECT | BPF_FIB_LOOKUP_TBID` flags, a
* specific routing table to use for the fib lookup.
*/
__u32 tbid;
};
union {
/* input */
struct {
__u32 mark; /* policy routing */
/* 2 4-byte holes for input */
};
/* output: source and dest mac */
struct {
__u8 smac[6]; /* ETH_ALEN */
__u8 dmac[6]; /* ETH_ALEN */
};
};
};
struct bpf_redir_neigh {
/* network family for lookup (AF_INET, AF_INET6) */
__u32 nh_family;
/* network address of nexthop; skips fib lookup to find gateway */
union {
__be32 ipv4_nh;
__u32 ipv6_nh[4]; /* in6_addr; network order */
};
};
/* bpf_check_mtu flags*/
enum bpf_check_mtu_flags {
BPF_MTU_CHK_SEGS = (1U << 0),
};
enum bpf_check_mtu_ret {
BPF_MTU_CHK_RET_SUCCESS, /* check and lookup successful */
BPF_MTU_CHK_RET_FRAG_NEEDED, /* fragmentation required to fwd */
BPF_MTU_CHK_RET_SEGS_TOOBIG, /* GSO re-segmentation needed to fwd */
};
enum bpf_task_fd_type {
BPF_FD_TYPE_RAW_TRACEPOINT, /* tp name */
BPF_FD_TYPE_TRACEPOINT, /* tp name */
BPF_FD_TYPE_KPROBE, /* (symbol + offset) or addr */
BPF_FD_TYPE_KRETPROBE, /* (symbol + offset) or addr */
BPF_FD_TYPE_UPROBE, /* filename + offset */
BPF_FD_TYPE_URETPROBE, /* filename + offset */
};
enum {
BPF_FLOW_DISSECTOR_F_PARSE_1ST_FRAG = (1U << 0),
BPF_FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL = (1U << 1),
BPF_FLOW_DISSECTOR_F_STOP_AT_ENCAP = (1U << 2),
};
struct bpf_flow_keys {
__u16 nhoff;
__u16 thoff;
__u16 addr_proto; /* ETH_P_* of valid addrs */
__u8 is_frag;
__u8 is_first_frag;
__u8 is_encap;
__u8 ip_proto;
__be16 n_proto;
__be16 sport;
__be16 dport;
union {
struct {
__be32 ipv4_src;
__be32 ipv4_dst;
};
struct {
__u32 ipv6_src[4]; /* in6_addr; network order */
__u32 ipv6_dst[4]; /* in6_addr; network order */
};
};
__u32 flags;
__be32 flow_label;
};
struct bpf_func_info {
__u32 insn_off;
__u32 type_id;
};
#define BPF_LINE_INFO_LINE_NUM(line_col) ((line_col) >> 10)
#define BPF_LINE_INFO_LINE_COL(line_col) ((line_col) & 0x3ff)
struct bpf_line_info {
__u32 insn_off;
__u32 file_name_off;
__u32 line_off;
__u32 line_col;
};
struct bpf_spin_lock {
__u32 val;
};
struct bpf_timer {
__u64 __opaque[2];
} __attribute__((aligned(8)));
struct bpf_task_work {
__u64 __opaque;
} __attribute__((aligned(8)));
struct bpf_wq {
__u64 __opaque[2];
} __attribute__((aligned(8)));
struct bpf_dynptr {
__u64 __opaque[2];
} __attribute__((aligned(8)));
struct bpf_list_head {
__u64 __opaque[2];
} __attribute__((aligned(8)));
struct bpf_list_node {
__u64 __opaque[3];
} __attribute__((aligned(8)));
struct bpf_rb_root {
__u64 __opaque[2];
} __attribute__((aligned(8)));
struct bpf_rb_node {
__u64 __opaque[4];
} __attribute__((aligned(8)));
struct bpf_refcount {
__u32 __opaque[1];
} __attribute__((aligned(4)));
struct bpf_sysctl {
__u32 write; /* Sysctl is being read (= 0) or written (= 1).
* Allows 1,2,4-byte read, but no write.
*/
__u32 file_pos; /* Sysctl file position to read from, write to.
* Allows 1,2,4-byte read an 4-byte write.
*/
};
struct bpf_sockopt {
__bpf_md_ptr(struct bpf_sock *, sk);
__bpf_md_ptr(void *, optval);
__bpf_md_ptr(void *, optval_end);
__s32 level;
__s32 optname;
__s32 optlen;
__s32 retval;
};
struct bpf_pidns_info {
__u32 pid;
__u32 tgid;
};
/* User accessible data for SK_LOOKUP programs. Add new fields at the end. */
struct bpf_sk_lookup {
union {
__bpf_md_ptr(struct bpf_sock *, sk); /* Selected socket */
__u64 cookie; /* Non-zero if socket was selected in PROG_TEST_RUN */
};
__u32 family; /* Protocol family (AF_INET, AF_INET6) */
__u32 protocol; /* IP protocol (IPPROTO_TCP, IPPROTO_UDP) */
__u32 remote_ip4; /* Network byte order */
__u32 remote_ip6[4]; /* Network byte order */
__be16 remote_port; /* Network byte order */
__u16 :16; /* Zero padding */
__u32 local_ip4; /* Network byte order */
__u32 local_ip6[4]; /* Network byte order */
__u32 local_port; /* Host byte order */
__u32 ingress_ifindex; /* The arriving interface. Determined by inet_iif. */
};
/*
* struct btf_ptr is used for typed pointer representation; the
* type id is used to render the pointer data as the appropriate type
* via the bpf_snprintf_btf() helper described above. A flags field -
* potentially to specify additional details about the BTF pointer
* (rather than its mode of display) - is included for future use.
* Display flags - BTF_F_* - are passed to bpf_snprintf_btf separately.
*/
struct btf_ptr {
void *ptr;
__u32 type_id;
__u32 flags; /* BTF ptr flags; unused at present. */
};
/*
* Flags to control bpf_snprintf_btf() behaviour.
* - BTF_F_COMPACT: no formatting around type information
* - BTF_F_NONAME: no struct/union member names/types
* - BTF_F_PTR_RAW: show raw (unobfuscated) pointer values;
* equivalent to %px.
* - BTF_F_ZERO: show zero-valued struct/union members; they
* are not displayed by default
*/
enum {
BTF_F_COMPACT = (1ULL << 0),
BTF_F_NONAME = (1ULL << 1),
BTF_F_PTR_RAW = (1ULL << 2),
BTF_F_ZERO = (1ULL << 3),
};
/* bpf_core_relo_kind encodes which aspect of captured field/type/enum value
* has to be adjusted by relocations. It is emitted by llvm and passed to
* libbpf and later to the kernel.
*/
enum bpf_core_relo_kind {
BPF_CORE_FIELD_BYTE_OFFSET = 0, /* field byte offset */
BPF_CORE_FIELD_BYTE_SIZE = 1, /* field size in bytes */
BPF_CORE_FIELD_EXISTS = 2, /* field existence in target kernel */
BPF_CORE_FIELD_SIGNED = 3, /* field signedness (0 - unsigned, 1 - signed) */
BPF_CORE_FIELD_LSHIFT_U64 = 4, /* bitfield-specific left bitshift */
BPF_CORE_FIELD_RSHIFT_U64 = 5, /* bitfield-specific right bitshift */
BPF_CORE_TYPE_ID_LOCAL = 6, /* type ID in local BPF object */
BPF_CORE_TYPE_ID_TARGET = 7, /* type ID in target kernel */
BPF_CORE_TYPE_EXISTS = 8, /* type existence in target kernel */
BPF_CORE_TYPE_SIZE = 9, /* type size in bytes */
BPF_CORE_ENUMVAL_EXISTS = 10, /* enum value existence in target kernel */
BPF_CORE_ENUMVAL_VALUE = 11, /* enum value integer value */
BPF_CORE_TYPE_MATCHES = 12, /* type match in target kernel */
};
/*
* "struct bpf_core_relo" is used to pass relocation data form LLVM to libbpf
* and from libbpf to the kernel.
*
* CO-RE relocation captures the following data:
* - insn_off - instruction offset (in bytes) within a BPF program that needs
* its insn->imm field to be relocated with actual field info;
* - type_id - BTF type ID of the "root" (containing) entity of a relocatable
* type or field;
* - access_str_off - offset into corresponding .BTF string section. String
* interpretation depends on specific relocation kind:
* - for field-based relocations, string encodes an accessed field using
* a sequence of field and array indices, separated by colon (:). It's
* conceptually very close to LLVM's getelementptr ([0]) instruction's
* arguments for identifying offset to a field.
* - for type-based relocations, strings is expected to be just "0";
* - for enum value-based relocations, string contains an index of enum
* value within its enum type;
* - kind - one of enum bpf_core_relo_kind;
*
* Example:
* struct sample {
* int a;
* struct {
* int b[10];
* };
* };
*
* struct sample *s = ...;
* int *x = &s->a; // encoded as "0:0" (a is field #0)
* int *y = &s->b[5]; // encoded as "0:1:0:5" (anon struct is field #1,
* // b is field #0 inside anon struct, accessing elem #5)
* int *z = &s[10]->b; // encoded as "10:1" (ptr is used as an array)
*
* type_id for all relocs in this example will capture BTF type id of
* `struct sample`.
*
* Such relocation is emitted when using __builtin_preserve_access_index()
* Clang built-in, passing expression that captures field address, e.g.:
*
* bpf_probe_read(&dst, sizeof(dst),
* __builtin_preserve_access_index(&src->a.b.c));
*
* In this case Clang will emit field relocation recording necessary data to
* be able to find offset of embedded `a.b.c` field within `src` struct.
*
* [0] https://llvm.org/docs/LangRef.html#getelementptr-instruction
*/
struct bpf_core_relo {
__u32 insn_off;
__u32 type_id;
__u32 access_str_off;
enum bpf_core_relo_kind kind;
};
/*
* Flags to control bpf_timer_start() behaviour.
* - BPF_F_TIMER_ABS: Timeout passed is absolute time, by default it is
* relative to current time.
* - BPF_F_TIMER_CPU_PIN: Timer will be pinned to the CPU of the caller.
*/
enum {
BPF_F_TIMER_ABS = (1ULL << 0),
BPF_F_TIMER_CPU_PIN = (1ULL << 1),
};
/* BPF numbers iterator state */
struct bpf_iter_num {
/* opaque iterator state; having __u64 here allows to preserve correct
* alignment requirements in vmlinux.h, generated from BTF
*/
__u64 __opaque[1];
} __attribute__((aligned(8)));
/*
* Flags to control BPF kfunc behaviour.
* - BPF_F_PAD_ZEROS: Pad destination buffer with zeros. (See the respective
* helper documentation for details.)
*/
enum bpf_kfunc_flags {
BPF_F_PAD_ZEROS = (1ULL << 0),
};
/*
* Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type.
*
* Before the map is used the orig_off field should point to an
* instruction inside the program being loaded. The other fields
* must be set to 0.
*
* After the program is loaded, the xlated_off will be adjusted
* by the verifier to point to the index of the original instruction
* in the xlated program. If the instruction is deleted, it will
* be set to (u32)-1. The jitted_off will be set to the corresponding
* offset in the jitted image of the program.
*/
struct bpf_insn_array_value {
__u32 orig_off;
__u32 xlated_off;
__u32 jitted_off;
__u32 :32;
};
#endif /* _UAPI__LINUX_BPF_H__ */
]
|
| 471/6 |
2026/05/18 19:27 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 160697
output: 34
thoughts: 637
Thoughts: **Analyzing `bpf_snprintf` and Potential Buffer Overflows**
Okay, so I'm diving into this `bpf_snprintf` function, and the documentation says it returns the length of the formatted string, including the trailing zero. This immediately makes me think about `bstr_printf` because I recall it *excludes* the trailing null. The docs are clear: the return value of `bpf_snprintf` should be the length, including the null terminator. I see in `kernel/bpf/helpers.c` that they're adding one to the `bstr_printf` return value before returning it from `bpf_snprintf`. That aligns perfectly! If `bstr_printf` returns `err`, then `bpf_snprintf` returning `err + 1` handles the trailing zero correctly, and if `err` is close to or bigger than the buffer size it indicates a truncation.
Now, I need to get back to the `bpf_bprintf_prepare` function because that's where things get interesting. I remember the `snprintf` return value is used for pointer arithmetic, and I *suspect* there's a problem there. Looking at it, the code uses `snprintf` to write to a `tmp_buf`. If `snprintf` truncates, the returned `err` indicates the *would-be* length. The code then increments `tmp_buf` by `err + 1`. If `err` is greater than or equal to the remaining buffer space, this means I'll be writing off the end of the buffer. Bingo, that's a buffer overflow.
So, I need to know if there's any *other* place in this `kernel/bpf/helpers.c` file where `snprintf` is used in a similar risky manner. I've already checked, and it looks like there are only two calls to `snprintf` in this file, the one I just analyzed and the call to `bpf_snprintf`, which uses a buffer size.
Now I need to be *sure* this is the only one, and it is a complete search. I should check for other "printf" variants, like `vsnprintf` and `sprintf`, and that I have.
Hmm, okay, time to expand the search. I'm going to look for any use of "printf" and similar formatting functions in the file.
|
| 472/6 |
2026/05/18 19:27 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:printf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 473/6 |
2026/05/18 19:27 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 161907
output: 23
thoughts: 14
Thoughts: Okay, here's my summary, tailored as if I'm an expert in the field thinking aloud:
**Code Inspection Strategy: Focused Search**
Alright, let's get down to it. My initial approach here is going to be targeted. I'm going to leverage my existing understanding of this code and the likely areas of concern. Specifically, I'm thinking I'll start by searching the file directly for the string `printf`. This is a classic indicator, isn't it? `printf` often exposes vulnerabilities or implementation issues, especially when it comes to format string bugs or potential for unexpected output in security-sensitive contexts. It's a quick and efficient way to pinpoint potential trouble spots or areas needing further scrutiny. I'll then examine the surrounding code to determine context and intention.
|
| 474/6 |
2026/05/18 19:28 |
tool |
read-file |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*/
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/bpf-cgroup.h>
#include <linux/cgroup.h>
#include <linux/rcupdate.h>
#include <linux/random.h>
#include <linux/smp.h>
#include <linux/topology.h>
#include <linux/ktime.h>
#include <linux/sched.h>
#include <linux/uidgid.h>
#include <linux/filter.h>
#include <linux/ctype.h>
#include <linux/jiffies.h>
#include <linux/pid_namespace.h>
#include <linux/poison.h>
#include <linux/proc_ns.h>
#include <linux/sched/task.h>
#include <linux/security.h>
#include <linux/btf_ids.h>
#include <linux/bpf_mem_alloc.h>
#include <linux/kasan.h>
#include <linux/bpf_verifier.h>
#include <linux/uaccess.h>
#include <linux/verification.h>
#include <linux/task_work.h>
#include <linux/irq_work.h>
#include <linux/buildid.h>
#include "../../lib/kstrtox.h"
/* If kernel subsystem is allowing eBPF programs to call this function,
* inside its own verifier_ops->get_func_proto() callback it should return
* bpf_map_lookup_elem_proto, so that verifier can properly check the arguments
*
* Different map implementations will rely on rcu in map methods
* lookup/update/delete, therefore eBPF programs must run under rcu lock
* if program is allowed to access maps, so check rcu_read_lock_held() or
* rcu_read_lock_trace_held() in all three functions.
*/
BPF_CALL_2(bpf_map_lookup_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_elem(map, key);
}
const struct bpf_func_proto bpf_map_lookup_elem_proto = {
.func = bpf_map_lookup_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_4(bpf_map_update_elem, struct bpf_map *, map, void *, key,
void *, value, u64, flags)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_update_elem(map, key, value, flags);
}
const struct bpf_func_proto bpf_map_update_elem_proto = {
.func = bpf_map_update_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_PTR_TO_MAP_VALUE,
.arg4_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_delete_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_delete_elem(map, key);
}
const struct bpf_func_proto bpf_map_delete_elem_proto = {
.func = bpf_map_delete_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_3(bpf_map_push_elem, struct bpf_map *, map, void *, value, u64, flags)
{
return map->ops->map_push_elem(map, value, flags);
}
const struct bpf_func_proto bpf_map_push_elem_proto = {
.func = bpf_map_push_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_pop_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_pop_elem(map, value);
}
const struct bpf_func_proto bpf_map_pop_elem_proto = {
.func = bpf_map_pop_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_2(bpf_map_peek_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_peek_elem(map, value);
}
const struct bpf_func_proto bpf_map_peek_elem_proto = {
.func = bpf_map_peek_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_3(bpf_map_lookup_percpu_elem, struct bpf_map *, map, void *, key, u32, cpu)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_percpu_elem(map, key, cpu);
}
const struct bpf_func_proto bpf_map_lookup_percpu_elem_proto = {
.func = bpf_map_lookup_percpu_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_ANYTHING,
};
const struct bpf_func_proto bpf_get_prandom_u32_proto = {
.func = bpf_user_rnd_u32,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_smp_processor_id)
{
return smp_processor_id();
}
const struct bpf_func_proto bpf_get_smp_processor_id_proto = {
.func = bpf_get_smp_processor_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.allow_fastcall = true,
};
BPF_CALL_0(bpf_get_numa_node_id)
{
return numa_node_id();
}
const struct bpf_func_proto bpf_get_numa_node_id_proto = {
.func = bpf_get_numa_node_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_ns)
{
/* NMI safe access to clock monotonic */
return ktime_get_mono_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_ns_proto = {
.func = bpf_ktime_get_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_boot_ns)
{
/* NMI safe access to clock boottime */
return ktime_get_boot_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_boot_ns_proto = {
.func = bpf_ktime_get_boot_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_coarse_ns)
{
return ktime_get_coarse_ns();
}
const struct bpf_func_proto bpf_ktime_get_coarse_ns_proto = {
.func = bpf_ktime_get_coarse_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_tai_ns)
{
/* NMI safe access to clock tai */
return ktime_get_tai_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_tai_ns_proto = {
.func = bpf_ktime_get_tai_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_pid_tgid)
{
struct task_struct *task = current;
if (unlikely(!task))
return -EINVAL;
return (u64) task->tgid << 32 | task->pid;
}
const struct bpf_func_proto bpf_get_current_pid_tgid_proto = {
.func = bpf_get_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_uid_gid)
{
struct task_struct *task = current;
kuid_t uid;
kgid_t gid;
if (unlikely(!task))
return -EINVAL;
current_uid_gid(&uid, &gid);
return (u64) from_kgid(&init_user_ns, gid) << 32 |
from_kuid(&init_user_ns, uid);
}
const struct bpf_func_proto bpf_get_current_uid_gid_proto = {
.func = bpf_get_current_uid_gid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_2(bpf_get_current_comm, char *, buf, u32, size)
{
struct task_struct *task = current;
if (unlikely(!task))
goto err_clear;
/* Verifier guarantees that size > 0 */
strscpy_pad(buf, task->comm, size);
return 0;
err_clear:
memset(buf, 0, size);
return -EINVAL;
}
const struct bpf_func_proto bpf_get_current_comm_proto = {
.func = bpf_get_current_comm,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE,
};
#if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK)
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
union {
__u32 val;
arch_spinlock_t lock;
} u = { .lock = __ARCH_SPIN_LOCK_UNLOCKED };
compiletime_assert(u.val == 0, "__ARCH_SPIN_LOCK_UNLOCKED not 0");
BUILD_BUG_ON(sizeof(*l) != sizeof(__u32));
BUILD_BUG_ON(sizeof(*lock) != sizeof(__u32));
preempt_disable();
arch_spin_lock(l);
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
arch_spin_unlock(l);
preempt_enable();
}
#else
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
BUILD_BUG_ON(sizeof(*l) != sizeof(*lock));
do {
atomic_cond_read_relaxed(l, !VAL);
} while (atomic_xchg(l, 1));
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
atomic_set_release(l, 0);
}
#endif
static DEFINE_PER_CPU(unsigned long, irqsave_flags);
static inline void __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
{
unsigned long flags;
local_irq_save(flags);
__bpf_spin_lock(lock);
__this_cpu_write(irqsave_flags, flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
{
__bpf_spin_lock_irqsave(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_lock_proto = {
.func = bpf_spin_lock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
{
unsigned long flags;
flags = __this_cpu_read(irqsave_flags);
__bpf_spin_unlock(lock);
local_irq_restore(flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
{
__bpf_spin_unlock_irqrestore(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_unlock_proto = {
.func = bpf_spin_unlock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
bool lock_src)
{
struct bpf_spin_lock *lock;
if (lock_src)
lock = src + map->record->spin_lock_off;
else
lock = dst + map->record->spin_lock_off;
preempt_disable();
__bpf_spin_lock_irqsave(lock);
copy_map_value(map, dst, src);
__bpf_spin_unlock_irqrestore(lock);
preempt_enable();
}
BPF_CALL_0(bpf_jiffies64)
{
return get_jiffies_64();
}
const struct bpf_func_proto bpf_jiffies64_proto = {
.func = bpf_jiffies64,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
#ifdef CONFIG_CGROUPS
BPF_CALL_0(bpf_get_current_cgroup_id)
{
struct cgroup *cgrp;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
cgrp_id = cgroup_id(cgrp);
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_cgroup_id_proto = {
.func = bpf_get_current_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_1(bpf_get_current_ancestor_cgroup_id, int, ancestor_level)
{
struct cgroup *cgrp;
struct cgroup *ancestor;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
ancestor = cgroup_ancestor(cgrp, ancestor_level);
cgrp_id = ancestor ? cgroup_id(ancestor) : 0;
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto = {
.func = bpf_get_current_ancestor_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
#endif /* CONFIG_CGROUPS */
#define BPF_STRTOX_BASE_MASK 0x1F
static int __bpf_strtoull(const char *buf, size_t buf_len, u64 flags,
unsigned long long *res, bool *is_negative)
{
unsigned int base = flags & BPF_STRTOX_BASE_MASK;
const char *cur_buf = buf;
size_t cur_len = buf_len;
unsigned int consumed;
size_t val_len;
char str[64];
if (!buf || !buf_len || !res || !is_negative)
return -EINVAL;
if (base != 0 && base != 8 && base != 10 && base != 16)
return -EINVAL;
if (flags & ~BPF_STRTOX_BASE_MASK)
return -EINVAL;
while (cur_buf < buf + buf_len && isspace(*cur_buf))
++cur_buf;
*is_negative = (cur_buf < buf + buf_len && *cur_buf == '-');
if (*is_negative)
++cur_buf;
consumed = cur_buf - buf;
cur_len -= consumed;
if (!cur_len)
return -EINVAL;
cur_len = min(cur_len, sizeof(str) - 1);
memcpy(str, cur_buf, cur_len);
str[cur_len] = '\0';
cur_buf = str;
cur_buf = _parse_integer_fixup_radix(cur_buf, &base);
val_len = _parse_integer(cur_buf, base, res);
if (val_len & KSTRTOX_OVERFLOW)
return -ERANGE;
if (val_len == 0)
return -EINVAL;
cur_buf += val_len;
consumed += cur_buf - str;
return consumed;
}
static int __bpf_strtoll(const char *buf, size_t buf_len, u64 flags,
long long *res)
{
unsigned long long _res;
bool is_negative;
int err;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative) {
if ((long long)-_res > 0)
return -ERANGE;
*res = -_res;
} else {
if ((long long)_res < 0)
return -ERANGE;
*res = _res;
}
return err;
}
BPF_CALL_4(bpf_strtol, const char *, buf, size_t, buf_len, u64, flags,
s64 *, res)
{
long long _res;
int err;
*res = 0;
err = __bpf_strtoll(buf, buf_len, flags, &_res);
if (err < 0)
return err;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtol_proto = {
.func = bpf_strtol,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(s64),
};
BPF_CALL_4(bpf_strtoul, const char *, buf, size_t, buf_len, u64, flags,
u64 *, res)
{
unsigned long long _res;
bool is_negative;
int err;
*res = 0;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative)
return -EINVAL;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtoul_proto = {
.func = bpf_strtoul,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(u64),
};
BPF_CALL_3(bpf_strncmp, const char *, s1, u32, s1_sz, const char *, s2)
{
return strncmp(s1, s2, s1_sz);
}
static const struct bpf_func_proto bpf_strncmp_proto = {
.func = bpf_strncmp,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_CONST_STR,
};
BPF_CALL_4(bpf_get_ns_current_pid_tgid, u64, dev, u64, ino,
struct bpf_pidns_info *, nsdata, u32, size)
{
struct task_struct *task = current;
struct pid_namespace *pidns;
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_pidns_info)))
goto clear;
if (unlikely((u64)(dev_t)dev != dev))
goto clear;
if (unlikely(!task))
goto clear;
pidns = task_active_pid_ns(task);
if (unlikely(!pidns)) {
err = -ENOENT;
goto clear;
}
if (!ns_match(&pidns->ns, (dev_t)dev, ino))
goto clear;
nsdata->pid = task_pid_nr_ns(task, pidns);
nsdata->tgid = task_tgid_nr_ns(task, pidns);
return 0;
clear:
memset((void *)nsdata, 0, (size_t) size);
return err;
}
const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = {
.func = bpf_get_ns_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
};
static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
.func = bpf_get_raw_cpu_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
}
const struct bpf_func_proto bpf_event_output_data_proto = {
.func = bpf_event_output_data,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
const void __user *, user_ptr)
{
int ret = copy_from_user(dst, user_ptr, size);
if (unlikely(ret)) {
memset(dst, 0, size);
ret = -EFAULT;
}
return ret;
}
const struct bpf_func_proto bpf_copy_from_user_proto = {
.func = bpf_copy_from_user,
.gpl_only = false,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_5(bpf_copy_from_user_task, void *, dst, u32, size,
const void __user *, user_ptr, struct task_struct *, tsk, u64, flags)
{
int ret;
/* flags is not used yet */
if (unlikely(flags))
return -EINVAL;
if (unlikely(!size))
return 0;
ret = access_process_vm(tsk, (unsigned long)user_ptr, dst, size, 0);
if (ret == size)
return 0;
memset(dst, 0, size);
/* Return -EFAULT for partial read */
return ret < 0 ? ret : -EFAULT;
}
const struct bpf_func_proto bpf_copy_from_user_task_proto = {
.func = bpf_copy_from_user_task,
.gpl_only = true,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_BTF_ID,
.arg4_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.arg5_type = ARG_ANYTHING
};
BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
{
if (cpu >= nr_cpu_ids)
return (unsigned long)NULL;
return (unsigned long)per_cpu_ptr((const void __percpu *)(const uintptr_t)ptr, cpu);
}
const struct bpf_func_proto bpf_per_cpu_ptr_proto = {
.func = bpf_per_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | PTR_MAYBE_NULL | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
.arg2_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_this_cpu_ptr, const void *, percpu_ptr)
{
return (unsigned long)this_cpu_ptr((const void __percpu *)(const uintptr_t)percpu_ptr);
}
const struct bpf_func_proto bpf_this_cpu_ptr_proto = {
.func = bpf_this_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
};
static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,
size_t bufsz)
{
void __user *user_ptr = (__force void __user *)unsafe_ptr;
buf[0] = 0;
switch (fmt_ptype) {
case 's':
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
if ((unsigned long)unsafe_ptr < TASK_SIZE)
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
fallthrough;
#endif
case 'k':
return strncpy_from_kernel_nofault(buf, unsafe_ptr, bufsz);
case 'u':
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
}
return -EINVAL;
}
/* Support executing three nested bprintf helper calls on a given CPU */
#define MAX_BPRINTF_NEST_LEVEL 3
static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs);
static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
{
int nest_level;
preempt_disable();
nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
return -EBUSY;
}
*bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
return 0;
}
void bpf_put_buffers(void)
{
if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
return;
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
}
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
{
if (!data->bin_args && !data->buf)
return;
bpf_put_buffers();
}
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)
{
if (map->map_type == BPF_MAP_TYPE_ARRAY) {
struct bpf_array *array = container_of(map, struct bpf_array, map);
*arr_idx = ((char *)value - array->value) / array->elem_size;
return arr_idx;
}
return (void *)value - round_up(map->key_size, 8);
}
enum bpf_async_type {
BPF_ASYNC_TYPE_TIMER = 0,
BPF_ASYNC_TYPE_WQ,
};
enum bpf_async_op {
BPF_ASYNC_START,
BPF_ASYNC_CANCEL
};
struct bpf_async_cmd {
struct llist_node node;
u64 nsec;
u32 mode;
enum bpf_async_op op;
};
struct bpf_async_cb {
struct bpf_map *map;
struct bpf_prog *prog;
void __rcu *callback_fn;
void *value;
struct rcu_head rcu;
u64 flags;
struct irq_work worker;
refcount_t refcnt;
enum bpf_async_type type;
struct llist_head async_cmds;
};
/* BPF map elements can contain 'struct bpf_timer'.
* Such map owns all of its BPF timers.
* 'struct bpf_timer' is allocated as part of map element allocation
* and it's zero initialized.
* That space is used to keep 'struct bpf_async_kern'.
* bpf_timer_init() allocates 'struct bpf_hrtimer', inits hrtimer, and
* remembers 'struct bpf_map *' pointer it's part of.
* bpf_timer_set_callback() increments prog refcnt and assign bpf callback_fn.
* bpf_timer_start() arms the timer.
* If user space reference to a map goes to zero at this point
* ops->map_release_uref callback is responsible for cancelling the timers,
* freeing their memory, and decrementing prog's refcnts.
* bpf_timer_cancel() cancels the timer and decrements prog's refcnt.
* Inner maps can contain bpf timers as well. ops->map_release_uref is
* freeing the timers when inner map is replaced or deleted by user space.
*/
struct bpf_hrtimer {
struct bpf_async_cb cb;
struct hrtimer timer;
atomic_t cancelling;
};
struct bpf_work {
struct bpf_async_cb cb;
struct work_struct work;
};
/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */
struct bpf_async_kern {
union {
struct bpf_async_cb *cb;
struct bpf_hrtimer *timer;
struct bpf_work *work;
};
} __attribute__((aligned(8)));
static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);
static void bpf_async_refcount_put(struct bpf_async_cb *cb);
static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer)
{
struct bpf_hrtimer *t = container_of(hrtimer, struct bpf_hrtimer, timer);
struct bpf_map *map = t->cb.map;
void *value = t->cb.value;
bpf_callback_t callback_fn;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_timer);
callback_fn = rcu_dereference_check(t->cb.callback_fn, rcu_read_lock_bh_held());
if (!callback_fn)
goto out;
/* bpf_timer_cb() runs in hrtimer_run_softirq. It doesn't migrate and
* cannot be preempted by another bpf_timer_cb() on the same cpu.
* Remember the timer this callback is servicing to prevent
* deadlock if callback_fn() calls bpf_timer_cancel() or
* bpf_map_delete_elem() on the same timer.
*/
this_cpu_write(hrtimer_running, t);
key = map_key_from_value(map, value, &idx);
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
/* The verifier checked that return value is zero. */
this_cpu_write(hrtimer_running, NULL);
out:
return HRTIMER_NORESTART;
}
static void bpf_wq_work(struct work_struct *work)
{
struct bpf_work *w = container_of(work, struct bpf_work, work);
struct bpf_async_cb *cb = &w->cb;
struct bpf_map *map = cb->map;
bpf_callback_t callback_fn;
void *value = cb->value;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_wq);
callback_fn = READ_ONCE(cb->callback_fn);
if (!callback_fn)
return;
key = map_key_from_value(map, value, &idx);
rcu_read_lock_trace();
migrate_disable();
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
migrate_enable();
rcu_read_unlock_trace();
}
static void bpf_async_cb_rcu_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
/*
* Drop the last reference to prog only after RCU GP, as set_callback()
* may race with cancel_and_free()
*/
if (cb->prog)
bpf_prog_put(cb->prog);
kfree_nolock(cb);
}
/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */
static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
bool retry = false;
/*
* bpf_async_cancel_and_free() tried to cancel timer/wq, but it
* could have raced with timer/wq_start. Now refcnt is zero and
* srcu/rcu GP completed. Cancel timer/wq again.
*/
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER:
if (hrtimer_try_to_cancel(&t->timer) < 0)
retry = true;
break;
case BPF_ASYNC_TYPE_WQ:
if (!cancel_work(&w->work) && work_busy(&w->work))
retry = true;
break;
}
if (retry) {
/*
* hrtimer or wq callback may still be running. It must be
* in rcu_tasks_trace or rcu CS, so wait for GP again.
* It won't retry forever, since refcnt zero prevents all
* operations on timer/wq.
*/
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
return;
}
/* RCU Tasks Trace grace period implies RCU grace period. */
bpf_async_cb_rcu_free(rcu);
}
static void worker_for_call_rcu(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
static void bpf_async_refcount_put(struct bpf_async_cb *cb)
{
if (!refcount_dec_and_test(&cb->refcnt))
return;
if (irqs_disabled()) {
cb->worker = IRQ_WORK_INIT(worker_for_call_rcu);
irq_work_queue(&cb->worker);
} else {
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async);
static void bpf_async_irq_worker(struct irq_work *work);
static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags,
enum bpf_async_type type)
{
struct bpf_async_cb *cb, *old_cb;
struct bpf_hrtimer *t;
struct bpf_work *w;
clockid_t clockid;
size_t size;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
size = sizeof(struct bpf_hrtimer);
break;
case BPF_ASYNC_TYPE_WQ:
size = sizeof(struct bpf_work);
break;
default:
return -EINVAL;
}
old_cb = READ_ONCE(async->cb);
if (old_cb)
return -EBUSY;
cb = bpf_map_kmalloc_nolock(map, size, 0, map->numa_node);
if (!cb)
return -ENOMEM;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
clockid = flags & (MAX_CLOCKS - 1);
t = (struct bpf_hrtimer *)cb;
atomic_set(&t->cancelling, 0);
hrtimer_setup(&t->timer, bpf_timer_cb, clockid, HRTIMER_MODE_REL_SOFT);
cb->value = (void *)async - map->record->timer_off;
break;
case BPF_ASYNC_TYPE_WQ:
w = (struct bpf_work *)cb;
INIT_WORK(&w->work, bpf_wq_work);
cb->value = (void *)async - map->record->wq_off;
break;
}
cb->map = map;
cb->prog = NULL;
cb->flags = flags;
cb->worker = IRQ_WORK_INIT(bpf_async_irq_worker);
init_llist_head(&cb->async_cmds);
refcount_set(&cb->refcnt, 1); /* map's reference */
cb->type = type;
rcu_assign_pointer(cb->callback_fn, NULL);
old_cb = cmpxchg(&async->cb, NULL, cb);
if (old_cb) {
/* Lost the race to initialize this bpf_async_kern, drop the allocated object */
kfree_nolock(cb);
return -EBUSY;
}
/* Guarantee the order between async->cb and map->usercnt. So
* when there are concurrent uref release and bpf timer init, either
* bpf_timer_cancel_and_free() called by uref release reads a no-NULL
* timer or atomic64_read() below returns a zero usercnt.
*/
smp_mb();
if (!atomic64_read(&map->usercnt)) {
/* maps with timers must be either held by user space
* or pinned in bpffs.
*/
bpf_async_cancel_and_free(async);
return -EPERM;
}
return 0;
}
BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
u64, flags)
{
clock_t clockid = flags & (MAX_CLOCKS - 1);
BUILD_BUG_ON(MAX_CLOCKS != 16);
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer));
if (flags >= MAX_CLOCKS ||
/* similar to timerfd except _ALARM variants are not supported */
(clockid != CLOCK_MONOTONIC &&
clockid != CLOCK_REALTIME &&
clockid != CLOCK_BOOTTIME))
return -EINVAL;
return __bpf_async_init(timer, map, flags, BPF_ASYNC_TYPE_TIMER);
}
static const struct bpf_func_proto bpf_timer_init_proto = {
.func = bpf_timer_init,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
static int bpf_async_update_prog_callback(struct bpf_async_cb *cb,
struct bpf_prog *prog,
void *callback_fn)
{
struct bpf_prog *prev;
/* Acquire a guard reference on prog to prevent it from being freed during the loop */
if (prog) {
prog = bpf_prog_inc_not_zero(prog);
if (IS_ERR(prog))
return PTR_ERR(prog);
}
do {
if (prog)
prog = bpf_prog_inc_not_zero(prog);
prev = xchg(&cb->prog, prog);
rcu_assign_pointer(cb->callback_fn, callback_fn);
/*
* Release previous prog, make sure that if other CPU is contending,
* to set bpf_prog, references are not leaked as each iteration acquires and
* releases one reference.
*/
if (prev)
bpf_prog_put(prev);
} while (READ_ONCE(cb->prog) != prog ||
(void __force *)READ_ONCE(cb->callback_fn) != callback_fn);
if (prog)
bpf_prog_put(prog);
return 0;
}
static DEFINE_PER_CPU(struct bpf_async_cb *, async_cb_running);
static int bpf_async_schedule_op(struct bpf_async_cb *cb, enum bpf_async_op op,
u64 nsec, u32 timer_mode)
{
/*
* Do not schedule another operation on this cpu if it's in irq_work
* callback that is processing async_cmds queue. Otherwise the following
* loop is possible:
* bpf_timer_start() -> bpf_async_schedule_op() -> irq_work_queue().
* irqrestore -> bpf_async_irq_worker() -> tracepoint -> bpf_timer_start().
*/
if (this_cpu_read(async_cb_running) == cb) {
bpf_async_refcount_put(cb);
return -EDEADLK;
}
struct bpf_async_cmd *cmd = kmalloc_nolock(sizeof(*cmd), 0, NUMA_NO_NODE);
if (!cmd) {
bpf_async_refcount_put(cb);
return -ENOMEM;
}
init_llist_node(&cmd->node);
cmd->nsec = nsec;
cmd->mode = timer_mode;
cmd->op = op;
if (llist_add(&cmd->node, &cb->async_cmds))
irq_work_queue(&cb->worker);
return 0;
}
static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback_fn,
struct bpf_prog *prog)
{
struct bpf_async_cb *cb;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
return bpf_async_update_prog_callback(cb, prog, callback_fn);
}
BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
struct bpf_prog_aux *, aux)
{
return __bpf_async_set_callback(timer, callback_fn, aux->prog);
}
static const struct bpf_func_proto bpf_timer_set_callback_proto = {
.func = bpf_timer_set_callback,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_PTR_TO_FUNC,
};
static bool defer_timer_wq_op(void)
{
return in_hardirq() || irqs_disabled();
}
BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags)
{
struct bpf_hrtimer *t;
u32 mode;
if (flags & ~(BPF_F_TIMER_ABS | BPF_F_TIMER_CPU_PIN))
return -EINVAL;
t = READ_ONCE(async->timer);
if (!t || !READ_ONCE(t->cb.prog))
return -EINVAL;
if (flags & BPF_F_TIMER_ABS)
mode = HRTIMER_MODE_ABS_SOFT;
else
mode = HRTIMER_MODE_REL_SOFT;
if (flags & BPF_F_TIMER_CPU_PIN)
mode |= HRTIMER_MODE_PINNED;
/*
* bpf_async_cancel_and_free() could have dropped refcnt to zero. In
* such case BPF progs are not allowed to arm the timer to prevent UAF.
*/
if (!refcount_inc_not_zero(&t->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
hrtimer_start(&t->timer, ns_to_ktime(nsecs), mode);
bpf_async_refcount_put(&t->cb);
return 0;
} else {
return bpf_async_schedule_op(&t->cb, BPF_ASYNC_START, nsecs, mode);
}
}
static const struct bpf_func_proto bpf_timer_start_proto = {
.func = bpf_timer_start,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async)
{
struct bpf_hrtimer *t, *cur_t;
bool inc = false;
int ret = 0;
if (defer_timer_wq_op())
return -EOPNOTSUPP;
t = READ_ONCE(async->timer);
if (!t)
return -EINVAL;
cur_t = this_cpu_read(hrtimer_running);
if (cur_t == t) {
/* If bpf callback_fn is trying to bpf_timer_cancel()
* its own timer the hrtimer_cancel() will deadlock
* since it waits for callback_fn to finish.
*/
return -EDEADLK;
}
/* Only account in-flight cancellations when invoked from a timer
* callback, since we want to avoid waiting only if other _callbacks_
* are waiting on us, to avoid introducing lockups. Non-callback paths
* are ok, since nobody would synchronously wait for their completion.
*/
if (!cur_t)
goto drop;
atomic_inc(&t->cancelling);
/* Need full barrier after relaxed atomic_inc */
smp_mb__after_atomic();
inc = true;
if (atomic_read(&cur_t->cancelling)) {
/* We're cancelling timer t, while some other timer callback is
* attempting to cancel us. In such a case, it might be possible
* that timer t belongs to the other callback, or some other
* callback waiting upon it (creating transitive dependencies
* upon us), and we will enter a deadlock if we continue
* cancelling and waiting for it synchronously, since it might
* do the same. Bail!
*/
atomic_dec(&t->cancelling);
return -EDEADLK;
}
drop:
bpf_async_update_prog_callback(&t->cb, NULL, NULL);
/* Cancel the timer and wait for associated callback to finish
* if it was running.
*/
ret = hrtimer_cancel(&t->timer);
if (inc)
atomic_dec(&t->cancelling);
return ret;
}
static const struct bpf_func_proto bpf_timer_cancel_proto = {
.func = bpf_timer_cancel,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
};
static void bpf_async_process_op(struct bpf_async_cb *cb, u32 op,
u64 timer_nsec, u32 timer_mode)
{
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER: {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
switch (op) {
case BPF_ASYNC_START:
hrtimer_start(&t->timer, ns_to_ktime(timer_nsec), timer_mode);
break;
case BPF_ASYNC_CANCEL:
hrtimer_try_to_cancel(&t->timer);
break;
}
break;
}
case BPF_ASYNC_TYPE_WQ: {
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
switch (op) {
case BPF_ASYNC_START:
schedule_work(&w->work);
break;
case BPF_ASYNC_CANCEL:
cancel_work(&w->work);
break;
}
break;
}
}
bpf_async_refcount_put(cb);
}
static void bpf_async_irq_worker(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
struct llist_node *pos, *n, *list;
list = llist_del_all(&cb->async_cmds);
if (!list)
return;
list = llist_reverse_order(list);
this_cpu_write(async_cb_running, cb);
llist_for_each_safe(pos, n, list) {
struct bpf_async_cmd *cmd;
cmd = container_of(pos, struct bpf_async_cmd, node);
bpf_async_process_op(cb, cmd->op, cmd->nsec, cmd->mode);
kfree_nolock(cmd);
}
this_cpu_write(async_cb_running, NULL);
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async)
{
struct bpf_async_cb *cb;
if (!READ_ONCE(async->cb))
return;
cb = xchg(&async->cb, NULL);
if (!cb)
return;
bpf_async_update_prog_callback(cb, NULL, NULL);
/*
* No refcount_inc_not_zero(&cb->refcnt) here. Dropping the last
* refcnt. Either synchronously or asynchronously in irq_work.
*/
if (!defer_timer_wq_op()) {
bpf_async_process_op(cb, BPF_ASYNC_CANCEL, 0, 0);
} else {
(void)bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
/*
* bpf_async_schedule_op() either enqueues allocated cmd into llist
* or fails with ENOMEM and drop the last refcnt.
* This is unlikely, but safe, since bpf_async_cb_rcu_tasks_trace_free()
* callback will do additional timer/wq_cancel due to races anyway.
*/
}
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_timer_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_wq_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
{
unsigned long *kptr = dst;
/* This helper may be inlined by verifier. */
return xchg(kptr, (unsigned long)ptr);
}
/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg()
* helper is determined dynamically by the verifier. Use BPF_PTR_POISON to
* denote type that verifier will determine.
*/
static const struct bpf_func_proto bpf_kptr_xchg_proto = {
.func = bpf_kptr_xchg,
.gpl_only = false,
.ret_type = RET_PTR_TO_BTF_ID_OR_NULL,
.ret_btf_id = BPF_PTR_POISON,
.arg1_type = ARG_KPTR_XCHG_DEST,
.arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL | OBJ_RELEASE,
.arg2_btf_id = BPF_PTR_POISON,
};
struct bpf_dynptr_file_impl {
struct freader freader;
/* 64 bit offset and size overriding 32 bit ones in bpf_dynptr_kern */
u64 offset;
u64 size;
};
/* Since the upper 8 bits of dynptr->size is reserved, the
* maximum supported size is 2^24 - 1.
*/
#define DYNPTR_MAX_SIZE ((1UL << 24) - 1)
#define DYNPTR_TYPE_SHIFT 28
#define DYNPTR_SIZE_MASK 0xFFFFFF
#define DYNPTR_RDONLY_BIT BIT(31)
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr)
{
return ptr->size & DYNPTR_RDONLY_BIT;
}
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
{
ptr->size |= DYNPTR_RDONLY_BIT;
}
static void bpf_dynptr_set_type(struct bpf_dynptr_kern *ptr, enum bpf_dynptr_type type)
{
ptr->size |= type << DYNPTR_TYPE_SHIFT;
}
static enum bpf_dynptr_type bpf_dynptr_get_type(const struct bpf_dynptr_kern *ptr)
{
return (ptr->size & ~(DYNPTR_RDONLY_BIT)) >> DYNPTR_TYPE_SHIFT;
}
u64 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
return df->size;
}
return ptr->size & DYNPTR_SIZE_MASK;
}
static void bpf_dynptr_advance_offset(struct bpf_dynptr_kern *ptr, u64 off)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->offset += off;
return;
}
ptr->offset += off;
}
static void bpf_dynptr_set_size(struct bpf_dynptr_kern *ptr, u64 new_size)
{
u32 metadata = ptr->size & ~DYNPTR_SIZE_MASK;
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->size = new_size;
return;
}
ptr->size = (u32)new_size | metadata;
}
int bpf_dynptr_check_size(u64 size)
{
return size > DYNPTR_MAX_SIZE ? -E2BIG : 0;
}
static int bpf_file_fetch_bytes(struct bpf_dynptr_file_impl *df, u64 offset, void *buf, u64 len)
{
const void *ptr;
if (!buf)
return -EINVAL;
df->freader.buf = buf;
df->freader.buf_sz = len;
ptr = freader_fetch(&df->freader, offset + df->offset, len);
if (!ptr)
return df->freader.err;
if (ptr != buf) /* Force copying into the buffer */
memcpy(buf, ptr, len);
return 0;
}
void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size)
{
ptr->data = data;
ptr->offset = offset;
ptr->size = size;
bpf_dynptr_set_type(ptr, type);
}
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)
{
memset(ptr, 0, sizeof(*ptr));
}
BPF_CALL_4(bpf_dynptr_from_mem, void *, data, u64, size, u64, flags, struct bpf_dynptr_kern *, ptr)
{
int err;
BTF_TYPE_EMIT(struct bpf_dynptr);
err = bpf_dynptr_check_size(size);
if (err)
goto error;
/* flags is currently unsupported */
if (flags) {
err = -EINVAL;
goto error;
}
bpf_dynptr_init(ptr, data, BPF_DYNPTR_TYPE_LOCAL, 0, size);
return 0;
error:
bpf_dynptr_set_null(ptr);
return err;
}
static const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
.func = bpf_dynptr_from_mem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL | MEM_UNINIT | MEM_WRITE,
};
static int __bpf_dynptr_read(void *dst, u64 len, const struct bpf_dynptr_kern *src,
u64 offset, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!src->data || flags)
return -EINVAL;
err = bpf_dynptr_check_off_len(src, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(src);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst, src->data + src->offset + offset, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_XDP:
return __bpf_xdp_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_SKB_META:
memmove(dst, bpf_skb_meta_pointer(src->data, src->offset + offset), len);
return 0;
case BPF_DYNPTR_TYPE_FILE:
return bpf_file_fetch_bytes(src->data, offset, dst, len);
default:
WARN_ONCE(true, "bpf_dynptr_read: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_read, void *, dst, u64, len, const struct bpf_dynptr_kern *, src,
u64, offset, u64, flags)
{
return __bpf_dynptr_read(dst, len, src, offset, flags);
}
static const struct bpf_func_proto bpf_dynptr_read_proto = {
.func = bpf_dynptr_read,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
int __bpf_dynptr_write(const struct bpf_dynptr_kern *dst, u64 offset, void *src,
u64 len, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!dst->data || __bpf_dynptr_is_rdonly(dst))
return -EINVAL;
err = bpf_dynptr_check_off_len(dst, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(dst);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
if (flags)
return -EINVAL;
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst->data + dst->offset + offset, src, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_store_bytes(dst->data, dst->offset + offset, src, len,
flags);
case BPF_DYNPTR_TYPE_XDP:
if (flags)
return -EINVAL;
return __bpf_xdp_store_bytes(dst->data, dst->offset + offset, src, len);
case BPF_DYNPTR_TYPE_SKB_META:
return __bpf_skb_meta_store_bytes(dst->data, dst->offset + offset, src,
len, flags);
default:
WARN_ONCE(true, "bpf_dynptr_write: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_write, const struct bpf_dynptr_kern *, dst, u64, offset, void *, src,
u64, len, u64, flags)
{
return __bpf_dynptr_write(dst, offset, src, len, flags);
}
static const struct bpf_func_proto bpf_dynptr_write_proto = {
.func = bpf_dynptr_write,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_dynptr_data, const struct bpf_dynptr_kern *, ptr, u64, offset, u64, len)
{
enum bpf_dynptr_type type;
int err;
if (!ptr->data)
return 0;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return 0;
if (__bpf_dynptr_is_rdonly(ptr))
return 0;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return (unsigned long)(ptr->data + ptr->offset + offset);
case BPF_DYNPTR_TYPE_SKB:
case BPF_DYNPTR_TYPE_XDP:
case BPF_DYNPTR_TYPE_SKB_META:
/* skb and xdp dynptrs should use bpf_dynptr_slice / bpf_dynptr_slice_rdwr */
return 0;
default:
WARN_ONCE(true, "bpf_dynptr_data: unknown dynptr type %d\n", type);
return 0;
}
}
static const struct bpf_func_proto bpf_dynptr_data_proto = {
.func = bpf_dynptr_data,
.gpl_only = false,
.ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
};
const struct bpf_func_proto bpf_get_current_task_proto __weak;
const struct bpf_func_proto bpf_get_current_task_btf_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_str_proto __weak;
const struct bpf_func_proto bpf_task_pt_regs_proto __weak;
const struct bpf_func_proto bpf_perf_event_read_proto __weak;
const struct bpf_func_proto bpf_send_signal_proto __weak;
const struct bpf_func_proto bpf_send_signal_thread_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_sleepable_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_proto __weak;
const struct bpf_func_proto bpf_get_branch_snapshot_proto __weak;
const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_map_lookup_elem:
return &bpf_map_lookup_elem_proto;
case BPF_FUNC_map_update_elem:
return &bpf_map_update_elem_proto;
case BPF_FUNC_map_delete_elem:
return &bpf_map_delete_elem_proto;
case BPF_FUNC_map_push_elem:
return &bpf_map_push_elem_proto;
case BPF_FUNC_map_pop_elem:
return &bpf_map_pop_elem_proto;
case BPF_FUNC_map_peek_elem:
return &bpf_map_peek_elem_proto;
case BPF_FUNC_map_lookup_percpu_elem:
return &bpf_map_lookup_percpu_elem_proto;
case BPF_FUNC_get_prandom_u32:
return &bpf_get_prandom_u32_proto;
case BPF_FUNC_get_smp_processor_id:
return &bpf_get_raw_smp_processor_id_proto;
case BPF_FUNC_get_numa_node_id:
return &bpf_get_numa_node_id_proto;
case BPF_FUNC_tail_call:
return &bpf_tail_call_proto;
case BPF_FUNC_ktime_get_ns:
return &bpf_ktime_get_ns_proto;
case BPF_FUNC_ktime_get_boot_ns:
return &bpf_ktime_get_boot_ns_proto;
case BPF_FUNC_ktime_get_tai_ns:
return &bpf_ktime_get_tai_ns_proto;
case BPF_FUNC_ringbuf_output:
return &bpf_ringbuf_output_proto;
case BPF_FUNC_ringbuf_reserve:
return &bpf_ringbuf_reserve_proto;
case BPF_FUNC_ringbuf_submit:
return &bpf_ringbuf_submit_proto;
case BPF_FUNC_ringbuf_discard:
return &bpf_ringbuf_discard_proto;
case BPF_FUNC_ringbuf_query:
return &bpf_ringbuf_query_proto;
case BPF_FUNC_strncmp:
return &bpf_strncmp_proto;
case BPF_FUNC_strtol:
return &bpf_strtol_proto;
case BPF_FUNC_strtoul:
return &bpf_strtoul_proto;
case BPF_FUNC_get_current_pid_tgid:
return &bpf_get_current_pid_tgid_proto;
case BPF_FUNC_get_ns_current_pid_tgid:
return &bpf_get_ns_current_pid_tgid_proto;
case BPF_FUNC_get_current_uid_gid:
return &bpf_get_current_uid_gid_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_BPF))
return NULL;
switch (func_id) {
case BPF_FUNC_spin_lock:
return &bpf_spin_lock_proto;
case BPF_FUNC_spin_unlock:
return &bpf_spin_unlock_proto;
case BPF_FUNC_jiffies64:
return &bpf_jiffies64_proto;
case BPF_FUNC_per_cpu_ptr:
return &bpf_per_cpu_ptr_proto;
case BPF_FUNC_this_cpu_ptr:
return &bpf_this_cpu_ptr_proto;
case BPF_FUNC_timer_init:
return &bpf_timer_init_proto;
case BPF_FUNC_timer_set_callback:
return &bpf_timer_set_callback_proto;
case BPF_FUNC_timer_start:
return &bpf_timer_start_proto;
case BPF_FUNC_timer_cancel:
return &bpf_timer_cancel_proto;
case BPF_FUNC_kptr_xchg:
return &bpf_kptr_xchg_proto;
case BPF_FUNC_for_each_map_elem:
return &bpf_for_each_map_elem_proto;
case BPF_FUNC_loop:
return &bpf_loop_proto;
case BPF_FUNC_user_ringbuf_drain:
return &bpf_user_ringbuf_drain_proto;
case BPF_FUNC_ringbuf_reserve_dynptr:
return &bpf_ringbuf_reserve_dynptr_proto;
case BPF_FUNC_ringbuf_submit_dynptr:
return &bpf_ringbuf_submit_dynptr_proto;
case BPF_FUNC_ringbuf_discard_dynptr:
return &bpf_ringbuf_discard_dynptr_proto;
case BPF_FUNC_dynptr_from_mem:
return &bpf_dynptr_from_mem_proto;
case BPF_FUNC_dynptr_read:
return &bpf_dynptr_read_proto;
case BPF_FUNC_dynptr_write:
return &bpf_dynptr_write_proto;
case BPF_FUNC_dynptr_data:
return &bpf_dynptr_data_proto;
#ifdef CONFIG_CGROUPS
case BPF_FUNC_cgrp_storage_get:
return &bpf_cgrp_storage_get_proto;
case BPF_FUNC_cgrp_storage_delete:
return &bpf_cgrp_storage_delete_proto;
case BPF_FUNC_get_current_cgroup_id:
return &bpf_get_current_cgroup_id_proto;
case BPF_FUNC_get_current_ancestor_cgroup_id:
return &bpf_get_current_ancestor_cgroup_id_proto;
case BPF_FUNC_current_task_under_cgroup:
return &bpf_current_task_under_cgroup_proto;
#endif
#ifdef CONFIG_CGROUP_NET_CLASSID
case BPF_FUNC_get_cgroup_classid:
return &bpf_get_cgroup_classid_curr_proto;
#endif
case BPF_FUNC_task_storage_get:
return &bpf_task_storage_get_proto;
case BPF_FUNC_task_storage_delete:
return &bpf_task_storage_delete_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_PERFMON))
return NULL;
switch (func_id) {
case BPF_FUNC_trace_printk:
return bpf_get_trace_printk_proto();
case BPF_FUNC_get_current_task:
return &bpf_get_current_task_proto;
case BPF_FUNC_get_current_task_btf:
return &bpf_get_current_task_btf_proto;
case BPF_FUNC_get_current_comm:
return &bpf_get_current_comm_proto;
case BPF_FUNC_probe_read_user:
return &bpf_probe_read_user_proto;
case BPF_FUNC_probe_read_kernel:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_proto;
case BPF_FUNC_probe_read_user_str:
return &bpf_probe_read_user_str_proto;
case BPF_FUNC_probe_read_kernel_str:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_str_proto;
case BPF_FUNC_copy_from_user:
return &bpf_copy_from_user_proto;
case BPF_FUNC_copy_from_user_task:
return &bpf_copy_from_user_task_proto;
case BPF_FUNC_snprintf_btf:
return &bpf_snprintf_btf_proto;
case BPF_FUNC_snprintf:
return &bpf_snprintf_proto;
case BPF_FUNC_task_pt_regs:
return &bpf_task_pt_regs_proto;
case BPF_FUNC_trace_vprintk:
return bpf_get_trace_vprintk_proto();
case BPF_FUNC_perf_event_read_value:
return bpf_get_perf_event_read_value_proto();
case BPF_FUNC_perf_event_read:
return &bpf_perf_event_read_proto;
case BPF_FUNC_send_signal:
return &bpf_send_signal_proto;
case BPF_FUNC_send_signal_thread:
return &bpf_send_signal_thread_proto;
case BPF_FUNC_get_task_stack:
return prog->sleepable ? &bpf_get_task_stack_sleepable_proto
: &bpf_get_task_stack_proto;
case BPF_FUNC_get_branch_snapshot:
return &bpf_get_branch_snapshot_proto;
case BPF_FUNC_find_vma:
return &bpf_find_vma_proto;
default:
return NULL;
}
}
EXPORT_SYMBOL_GPL(bpf_base_func_proto);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock)
{
struct list_head *head = list_head, *orig_head = list_head;
BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
/* Do the actual list draining outside the lock to not hold the lock for
* too long, and also prevent deadlocks if tracing programs end up
* executing on entry/exit of functions called inside the critical
* section, and end up doing map ops that call bpf_list_head_free for
* the same map value again.
*/
__bpf_spin_lock_irqsave(spin_lock);
if (!head->next || list_empty(head))
goto unlock;
head = head->next;
unlock:
INIT_LIST_HEAD(orig_head);
__bpf_spin_unlock_irqrestore(spin_lock);
while (head != orig_head) {
void *obj = head;
obj -= field->graph_root.node_offset;
head = head->next;
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
/* Like rbtree_postorder_for_each_entry_safe, but 'pos' and 'n' are
* 'rb_node *', so field name of rb_node within containing struct is not
* needed.
*
* Since bpf_rb_tree's node type has a corresponding struct btf_field with
* graph_root.node_offset, it's not necessary to know field name
* or type of node struct
*/
#define bpf_rbtree_postorder_for_each_entry_safe(pos, n, root) \
for (pos = rb_first_postorder(root); \
pos && ({ n = rb_next_postorder(pos); 1; }); \
pos = n)
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock)
{
struct rb_root_cached orig_root, *root = rb_root;
struct rb_node *pos, *n;
void *obj;
BUILD_BUG_ON(sizeof(struct rb_root_cached) > sizeof(struct bpf_rb_root));
BUILD_BUG_ON(__alignof__(struct rb_root_cached) > __alignof__(struct bpf_rb_root));
__bpf_spin_lock_irqsave(spin_lock);
orig_root = *root;
*root = RB_ROOT_CACHED;
__bpf_spin_unlock_irqrestore(spin_lock);
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
obj = pos;
obj -= field->graph_root.node_offset;
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
__bpf_kfunc_start_defs();
/**
* bpf_obj_new() - allocate an object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate an object of the type identified by @local_type_id__k and
* initialize its special fields. BPF programs can use
* bpf_core_type_id_local() to provide @local_type_id__k. The verifier
* rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
void *p;
p = bpf_mem_alloc(&bpf_global_ma, size);
if (!p)
return NULL;
if (meta)
bpf_obj_init(meta->record, p);
return p;
}
__bpf_kfunc void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_obj_new(local_type_id__k, meta__ign);
}
/**
* bpf_percpu_obj_new() - allocate a percpu object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate a percpu object of the type identified by @local_type_id__k. BPF
* programs can use bpf_core_type_id_local() to provide @local_type_id__k.
* The verifier rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated percpu object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_percpu_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
/* The verifier has ensured that meta must be NULL */
return bpf_mem_alloc(&bpf_global_percpu_ma, size);
}
__bpf_kfunc void *bpf_percpu_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_percpu_obj_new(local_type_id__k, meta__ign);
}
/* Must be called under migrate_disable(), as required by bpf_mem_free */
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu)
{
struct bpf_mem_alloc *ma;
if (rec && rec->refcount_off >= 0 &&
!refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) {
/* Object is refcounted and refcount_dec didn't result in 0
* refcount. Return without freeing the object
*/
return;
}
if (rec)
bpf_obj_free_fields(rec, p);
if (percpu)
ma = &bpf_global_percpu_ma;
else
ma = &bpf_global_ma;
bpf_mem_free_rcu(ma, p);
}
/**
* bpf_obj_drop() - drop a previously allocated object
* @p__alloc: object to free
* @meta: verifier-supplied struct metadata
*
* Destroy special fields in @p__alloc as needed and free the object. The
* verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
void *p = p__alloc;
__bpf_obj_drop_impl(p, meta ? meta->record : NULL, false);
}
__bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
{
return bpf_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_percpu_obj_drop() - drop a previously allocated percpu object
* @p__alloc: percpu object to free
* @meta: verifier-supplied struct metadata
*
* Free @p__alloc. The verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_percpu_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
/* The verifier has ensured that meta must be NULL */
bpf_mem_free_rcu(&bpf_global_percpu_ma, p__alloc);
}
__bpf_kfunc void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign)
{
bpf_percpu_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_refcount_acquire() - turn a local kptr into an owning reference
* @p__refcounted_kptr: non-owning local kptr
* @meta: verifier-supplied struct metadata
*
* Increment the refcount for @p__refcounted_kptr. The verifier rewrites
* @meta; BPF programs do not set it.
*
* Return: Owning reference to @p__refcounted_kptr, or %NULL on failure.
*/
__bpf_kfunc void *bpf_refcount_acquire(void *p__refcounted_kptr, struct btf_struct_meta *meta)
{
struct bpf_refcount *ref;
/* Could just cast directly to refcount_t *, but need some code using
* bpf_refcount type so that it is emitted in vmlinux BTF
*/
ref = (struct bpf_refcount *)(p__refcounted_kptr + meta->record->refcount_off);
if (!refcount_inc_not_zero((refcount_t *)ref))
return NULL;
/* Verifier strips KF_RET_NULL if input is owned ref, see is_kfunc_ret_null
* in verifier.c
*/
return (void *)p__refcounted_kptr;
}
__bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta__ign)
{
return bpf_refcount_acquire(p__refcounted_kptr, meta__ign);
}
static int __bpf_list_add(struct bpf_list_node_kern *node,
struct bpf_list_head *head,
bool tail, struct btf_record *rec, u64 off)
{
struct list_head *n = &node->list_head, *h = (void *)head;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
/* node->owner != NULL implies !list_empty(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
tail ? list_add_tail(n, h) : list_add(n, h);
WRITE_ONCE(node->owner, head);
return 0;
}
/**
* bpf_list_push_front() - add a node to the front of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the front of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_front(head, node, meta__ign, off);
}
/**
* bpf_list_push_back() - add a node to the back of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the back of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_back(head, node, meta__ign, off);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
{
struct list_head *n, *h = (void *)head;
struct bpf_list_node_kern *node;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
if (list_empty(h))
return NULL;
n = tail ? h->prev : h->next;
node = container_of(n, struct bpf_list_node_kern, list_head);
if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))
return NULL;
list_del_init(n);
WRITE_ONCE(node->owner, NULL);
return (struct bpf_list_node *)n;
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
{
return __bpf_list_del(head, false);
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
{
return __bpf_list_del(head, true);
}
__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->next;
}
__bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->prev;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
struct rb_root_cached *r = (struct rb_root_cached *)root;
struct rb_node *n = &node_internal->rb_node;
/* node_internal->owner != root implies either RB_EMPTY_NODE(n) or
* n is owned by some other tree. No need to check RB_EMPTY_NODE(n)
*/
if (READ_ONCE(node_internal->owner) != root)
return NULL;
rb_erase_cached(n, r);
RB_CLEAR_NODE(n);
WRITE_ONCE(node_internal->owner, NULL);
return (struct bpf_rb_node *)n;
}
/* Need to copy rbtree_add_cached's logic here because our 'less' is a BPF
* program
*/
static int __bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node_kern *node,
void *less, struct btf_record *rec, u64 off)
{
struct rb_node **link = &((struct rb_root_cached *)root)->rb_root.rb_node;
struct rb_node *parent = NULL, *n = &node->rb_node;
bpf_callback_t cb = (bpf_callback_t)less;
bool leftmost = true;
/* node->owner != NULL implies !RB_EMPTY_NODE(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
while (*link) {
parent = *link;
if (cb((uintptr_t)node, (uintptr_t)parent, 0, 0, 0)) {
link = &parent->rb_left;
} else {
link = &parent->rb_right;
leftmost = false;
}
}
rb_link_node(n, parent, link);
rb_insert_color_cached(n, (struct rb_root_cached *)root, leftmost);
WRITE_ONCE(node->owner, root);
return 0;
}
/**
* bpf_rbtree_add() - add a node to a BPF rbtree
* @root: tree root
* @node: node to insert
* @less: comparator used to order nodes
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node into @root using @less. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked in a tree.
*/
__bpf_kfunc int bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_rb_node_kern *n = (void *)node;
return __bpf_rbtree_add(root, n, (void *)less, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
void *meta__ign, u64 off)
{
return bpf_rbtree_add(root, node, less, meta__ign, off);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)rb_first_cached(r);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_root(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)r->rb_root.rb_node;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_left(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_left;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_right(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_right;
}
/**
* bpf_task_acquire - Acquire a reference to a task. A task acquired by this
* kfunc which is not stored in a map as a kptr, must be released by calling
* bpf_task_release().
* @p: The task on which a reference is being acquired.
*/
__bpf_kfunc struct task_struct *bpf_task_acquire(struct task_struct *p)
{
if (refcount_inc_not_zero(&p->rcu_users))
return p;
return NULL;
}
/**
* bpf_task_release - Release the reference acquired on a task.
* @p: The task on which a reference is being released.
*/
__bpf_kfunc void bpf_task_release(struct task_struct *p)
{
put_task_struct_rcu_user(p);
}
__bpf_kfunc void bpf_task_release_dtor(void *p)
{
put_task_struct_rcu_user(p);
}
CFI_NOSEAL(bpf_task_release_dtor);
#ifdef CONFIG_CGROUPS
/**
* bpf_cgroup_acquire - Acquire a reference to a cgroup. A cgroup acquired by
* this kfunc which is not stored in a map as a kptr, must be released by
* calling bpf_cgroup_release().
* @cgrp: The cgroup on which a reference is being acquired.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)
{
return cgroup_tryget(cgrp) ? cgrp : NULL;
}
/**
* bpf_cgroup_release - Release the reference acquired on a cgroup.
* If this kfunc is invoked in an RCU read region, the cgroup is guaranteed to
* not be freed until the current grace period has ended, even if its refcount
* drops to 0.
* @cgrp: The cgroup on which a reference is being released.
*/
__bpf_kfunc void bpf_cgroup_release(struct cgroup *cgrp)
{
cgroup_put(cgrp);
}
__bpf_kfunc void bpf_cgroup_release_dtor(void *cgrp)
{
cgroup_put(cgrp);
}
CFI_NOSEAL(bpf_cgroup_release_dtor);
/**
* bpf_cgroup_ancestor - Perform a lookup on an entry in a cgroup's ancestor
* array. A cgroup returned by this kfunc which is not subsequently stored in a
* map, must be released by calling bpf_cgroup_release().
* @cgrp: The cgroup for which we're performing a lookup.
* @level: The level of ancestor to look up.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)
{
struct cgroup *ancestor;
if (level > cgrp->level || level < 0)
return NULL;
/* cgrp's refcnt could be 0 here, but ancestors can still be accessed */
ancestor = cgrp->ancestors[level];
if (!cgroup_tryget(ancestor))
return NULL;
return ancestor;
}
/**
* bpf_cgroup_from_id - Find a cgroup from its ID. A cgroup returned by this
* kfunc which is not subsequently stored in a map, must be released by calling
* bpf_cgroup_release().
* @cgid: cgroup id.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_from_id(u64 cgid)
{
struct cgroup *cgrp;
cgrp = __cgroup_get_from_id(cgid);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
/**
* bpf_task_under_cgroup - wrap task_under_cgroup_hierarchy() as a kfunc, test
* task's membership of cgroup ancestry.
* @task: the task to be tested
* @ancestor: possible ancestor of @task's cgroup
*
* Tests whether @task's default cgroup hierarchy is a descendant of @ancestor.
* It follows all the same rules as cgroup_is_descendant, and only applies
* to the default hierarchy.
*/
__bpf_kfunc long bpf_task_under_cgroup(struct task_struct *task,
struct cgroup *ancestor)
{
long ret;
rcu_read_lock();
ret = task_under_cgroup_hierarchy(task, ancestor);
rcu_read_unlock();
return ret;
}
BPF_CALL_2(bpf_current_task_under_cgroup, struct bpf_map *, map, u32, idx)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
struct cgroup *cgrp;
if (unlikely(idx >= array->map.max_entries))
return -E2BIG;
cgrp = READ_ONCE(array->ptrs[idx]);
if (unlikely(!cgrp))
return -EAGAIN;
return task_under_cgroup_hierarchy(current, cgrp);
}
const struct bpf_func_proto bpf_current_task_under_cgroup_proto = {
.func = bpf_current_task_under_cgroup,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
};
/**
* bpf_task_get_cgroup1 - Acquires the associated cgroup of a task within a
* specific cgroup1 hierarchy. The cgroup1 hierarchy is identified by its
* hierarchy ID.
* @task: The target task
* @hierarchy_id: The ID of a cgroup1 hierarchy
*
* On success, the cgroup is returen. On failure, NULL is returned.
*/
__bpf_kfunc struct cgroup *
bpf_task_get_cgroup1(struct task_struct *task, int hierarchy_id)
{
struct cgroup *cgrp = task_get_cgroup1(task, hierarchy_id);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
#endif /* CONFIG_CGROUPS */
/**
* bpf_task_from_pid - Find a struct task_struct from its pid by looking it up
* in the root pid namespace idr. If a task is returned, it must either be
* stored in a map, or released with bpf_task_release().
* @pid: The pid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_pid_ns(pid, &init_pid_ns);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_task_from_vpid - Find a struct task_struct from its vpid by looking it up
* in the pid namespace of the current task. If a task is returned, it must
* either be stored in a map, or released with bpf_task_release().
* @vpid: The vpid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_vpid(vpid);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_dynptr_slice() - Obtain a read-only pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* If the intention is to write to the data slice, please use
* bpf_dynptr_slice_rdwr.
*
* The user must check that the returned pointer is not null before using it.
*
* Please note that in the case of skb and xdp dynptrs, bpf_dynptr_slice
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a read-only
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
enum bpf_dynptr_type type;
u64 len = buffer__szk;
int err;
if (!ptr->data)
return NULL;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return NULL;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return ptr->data + ptr->offset + offset;
case BPF_DYNPTR_TYPE_SKB:
if (buffer__nullable)
return skb_header_pointer(ptr->data, ptr->offset + offset, len, buffer__nullable);
else
return skb_pointer_if_linear(ptr->data, ptr->offset + offset, len);
case BPF_DYNPTR_TYPE_XDP:
{
void *xdp_ptr = bpf_xdp_pointer(ptr->data, ptr->offset + offset, len);
if (!IS_ERR_OR_NULL(xdp_ptr))
return xdp_ptr;
if (!buffer__nullable)
return NULL;
bpf_xdp_copy_buf(ptr->data, ptr->offset + offset, buffer__nullable, len, false);
return buffer__nullable;
}
case BPF_DYNPTR_TYPE_SKB_META:
return bpf_skb_meta_pointer(ptr->data, ptr->offset + offset);
case BPF_DYNPTR_TYPE_FILE:
err = bpf_file_fetch_bytes(ptr->data, offset, buffer__nullable, buffer__szk);
return err ? NULL : buffer__nullable;
default:
WARN_ONCE(true, "unknown dynptr type %d\n", type);
return NULL;
}
}
/**
* bpf_dynptr_slice_rdwr() - Obtain a writable pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* The returned pointer is writable and may point to either directly the dynptr
* data at the requested offset or to the buffer if unable to obtain a direct
* data pointer to (example: the requested slice is to the paged area of an skb
* packet). In the case where the returned pointer is to the buffer, the user
* is responsible for persisting writes through calling bpf_dynptr_write(). This
* usually looks something like this pattern:
*
* struct eth_hdr *eth = bpf_dynptr_slice_rdwr(&dynptr, 0, buffer, sizeof(buffer));
* if (!eth)
* return TC_ACT_SHOT;
*
* // mutate eth header //
*
* if (eth == buffer)
* bpf_dynptr_write(&ptr, 0, buffer, sizeof(buffer), 0);
*
* Please note that, as in the example above, the user must check that the
* returned pointer is not null before using it.
*
* Please also note that in the case of skb and xdp dynptrs, bpf_dynptr_slice_rdwr
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice_rdwr will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data || __bpf_dynptr_is_rdonly(ptr))
return NULL;
/* bpf_dynptr_slice_rdwr is the same logic as bpf_dynptr_slice.
*
* For skb-type dynptrs, it is safe to write into the returned pointer
* if the bpf program allows skb data writes. There are two possibilities
* that may occur when calling bpf_dynptr_slice_rdwr:
*
* 1) The requested slice is in the head of the skb. In this case, the
* returned pointer is directly to skb data, and if the skb is cloned, the
* verifier will have uncloned it (see bpf_unclone_prologue()) already.
* The pointer can be directly written into.
*
* 2) Some portion of the requested slice is in the paged buffer area.
* In this case, the requested data will be copied out into the buffer
* and the returned pointer will be a pointer to the buffer. The skb
* will not be pulled. To persist the write, the user will need to call
* bpf_dynptr_write(), which will pull the skb and commit the write.
*
* Similarly for xdp programs, if the requested slice is not across xdp
* fragments, then a direct pointer will be returned, otherwise the data
* will be copied out into the buffer and the user will need to call
* bpf_dynptr_write() to commit changes.
*/
return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
}
__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 size;
if (!ptr->data || start > end)
return -EINVAL;
size = __bpf_dynptr_size(ptr);
if (start > size || end > size)
return -ERANGE;
bpf_dynptr_advance_offset(ptr, start);
bpf_dynptr_set_size(ptr, end - start);
return 0;
}
__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
return !ptr->data;
}
__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return false;
return __bpf_dynptr_is_rdonly(ptr);
}
__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return -EINVAL;
return __bpf_dynptr_size(ptr);
}
__bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
struct bpf_dynptr *clone__uninit)
{
struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data) {
bpf_dynptr_set_null(clone);
return -EINVAL;
}
*clone = *ptr;
return 0;
}
/**
* bpf_dynptr_copy() - Copy data from one dynptr to another.
* @dst_ptr: Destination dynptr - where data should be copied to
* @dst_off: Offset into the destination dynptr
* @src_ptr: Source dynptr - where data should be copied from
* @src_off: Offset into the source dynptr
* @size: Length of the data to copy from source to destination
*
* Copies data from source dynptr to destination dynptr.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
{
struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
void *src_slice, *dst_slice;
char buf[256];
u64 off;
src_slice = bpf_dynptr_slice(src_ptr, src_off, NULL, size);
dst_slice = bpf_dynptr_slice_rdwr(dst_ptr, dst_off, NULL, size);
if (src_slice && dst_slice) {
memmove(dst_slice, src_slice, size);
return 0;
}
if (src_slice)
return __bpf_dynptr_write(dst, dst_off, src_slice, size, 0);
if (dst_slice)
return __bpf_dynptr_read(dst_slice, size, src, src_off, 0);
if (bpf_dynptr_check_off_len(dst, dst_off, size) ||
bpf_dynptr_check_off_len(src, src_off, size))
return -E2BIG;
off = 0;
while (off < size) {
u64 chunk_sz = min_t(u64, sizeof(buf), size - off);
int err;
err = __bpf_dynptr_read(buf, chunk_sz, src, src_off + off, 0);
if (err)
return err;
err = __bpf_dynptr_write(dst, dst_off + off, buf, chunk_sz, 0);
if (err)
return err;
off += chunk_sz;
}
return 0;
}
/**
* bpf_dynptr_memset() - Fill dynptr memory with a constant byte.
* @p: Destination dynptr - where data will be filled
* @offset: Offset into the dynptr to start filling from
* @size: Number of bytes to fill
* @val: Constant byte to fill the memory with
*
* Fills the @size bytes of the memory area pointed to by @p
* at @offset with the constant byte @val.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 chunk_sz, write_off;
char buf[256];
void* slice;
int err;
slice = bpf_dynptr_slice_rdwr(p, offset, NULL, size);
if (likely(slice)) {
memset(slice, val, size);
return 0;
}
if (__bpf_dynptr_is_rdonly(ptr))
return -EINVAL;
err = bpf_dynptr_check_off_len(ptr, offset, size);
if (err)
return err;
/* Non-linear data under the dynptr, write from a local buffer */
chunk_sz = min_t(u64, sizeof(buf), size);
memset(buf, val, chunk_sz);
for (write_off = 0; write_off < size; write_off += chunk_sz) {
chunk_sz = min_t(u64, sizeof(buf), size - write_off);
err = __bpf_dynptr_write(ptr, offset + write_off, buf, chunk_sz, 0);
if (err)
return err;
}
return 0;
}
__bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
{
return obj;
}
__bpf_kfunc void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k)
{
return (void *)obj__ign;
}
__bpf_kfunc void bpf_rcu_read_lock(void)
{
rcu_read_lock();
}
__bpf_kfunc void bpf_rcu_read_unlock(void)
{
rcu_read_unlock();
}
struct bpf_throw_ctx {
struct bpf_prog_aux *aux;
u64 sp;
u64 bp;
int cnt;
};
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
{
struct bpf_throw_ctx *ctx = cookie;
struct bpf_prog *prog;
/*
* The RCU read lock is held to safely traverse the latch tree, but we
* don't need its protection when accessing the prog, since it has an
* active stack frame on the current stack trace, and won't disappear.
*/
rcu_read_lock();
prog = bpf_prog_ksym_find(ip);
rcu_read_unlock();
if (!prog)
return !ctx->cnt;
ctx->cnt++;
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
ctx->sp = sp;
ctx->bp = bp;
return false;
}
__bpf_kfunc void bpf_throw(u64 cookie)
{
struct bpf_throw_ctx ctx = {};
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
WARN_ON_ONCE(!ctx.aux->exception_boundary);
WARN_ON_ONCE(!ctx.bp);
WARN_ON_ONCE(!ctx.cnt);
/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
* deeper stack depths than ctx.sp as we do not return from bpf_throw,
* which skips compiler generated instrumentation to do the same.
*/
kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_map *map = p__map;
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq));
if (flags)
return -EINVAL;
return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ);
}
__bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_work *w;
if (flags)
return -EINVAL;
w = READ_ONCE(async->work);
if (!w || !READ_ONCE(w->cb.prog))
return -EINVAL;
if (!refcount_inc_not_zero(&w->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
schedule_work(&w->work);
bpf_async_refcount_put(&w->cb);
return 0;
} else {
return bpf_async_schedule_op(&w->cb, BPF_ASYNC_START, 0, 0);
}
}
__bpf_kfunc int bpf_wq_set_callback(struct bpf_wq *wq,
int (callback_fn)(void *map, int *key, void *value),
unsigned int flags,
struct bpf_prog_aux *aux)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
if (flags)
return -EINVAL;
return __bpf_async_set_callback(async, callback_fn, aux->prog);
}
__bpf_kfunc void bpf_preempt_disable(void)
{
preempt_disable();
}
__bpf_kfunc void bpf_preempt_enable(void)
{
preempt_enable();
}
struct bpf_iter_bits {
__u64 __opaque[2];
} __aligned(8);
#define BITS_ITER_NR_WORDS_MAX 511
struct bpf_iter_bits_kern {
union {
__u64 *bits;
__u64 bits_copy;
};
int nr_bits;
int bit;
} __aligned(8);
/* On 64-bit hosts, unsigned long and u64 have the same size, so passing
* a u64 pointer and an unsigned long pointer to find_next_bit() will
* return the same result, as both point to the same 8-byte area.
*
* For 32-bit little-endian hosts, using a u64 pointer or unsigned long
* pointer also makes no difference. This is because the first iterated
* unsigned long is composed of bits 0-31 of the u64 and the second unsigned
* long is composed of bits 32-63 of the u64.
*
* However, for 32-bit big-endian hosts, this is not the case. The first
* iterated unsigned long will be bits 32-63 of the u64, so swap these two
* ulong values within the u64.
*/
static void swap_ulong_in_u64(u64 *bits, unsigned int nr)
{
#if (BITS_PER_LONG == 32) && defined(__BIG_ENDIAN)
unsigned int i;
for (i = 0; i < nr; i++)
bits[i] = (bits[i] >> 32) | ((u64)(u32)bits[i] << 32);
#endif
}
/**
* bpf_iter_bits_new() - Initialize a new bits iterator for a given memory area
* @it: The new bpf_iter_bits to be created
* @unsafe_ptr__ign: A pointer pointing to a memory area to be iterated over
* @nr_words: The size of the specified memory area, measured in 8-byte units.
* The maximum value of @nr_words is @BITS_ITER_NR_WORDS_MAX. This limit may be
* further reduced by the BPF memory allocator implementation.
*
* This function initializes a new bpf_iter_bits structure for iterating over
* a memory area which is specified by the @unsafe_ptr__ign and @nr_words. It
* copies the data of the memory area to the newly created bpf_iter_bits @it for
* subsequent iteration operations.
*
* On success, 0 is returned. On failure, ERR is returned.
*/
__bpf_kfunc int
bpf_iter_bits_new(struct bpf_iter_bits *it, const u64 *unsafe_ptr__ign, u32 nr_words)
{
struct bpf_iter_bits_kern *kit = (void *)it;
u32 nr_bytes = nr_words * sizeof(u64);
u32 nr_bits = BYTES_TO_BITS(nr_bytes);
int err;
BUILD_BUG_ON(sizeof(struct bpf_iter_bits_kern) != sizeof(struct bpf_iter_bits));
BUILD_BUG_ON(__alignof__(struct bpf_iter_bits_kern) !=
__alignof__(struct bpf_iter_bits));
kit->nr_bits = 0;
kit->bits_copy = 0;
kit->bit = -1;
if (!unsafe_ptr__ign || !nr_words)
return -EINVAL;
if (nr_words > BITS_ITER_NR_WORDS_MAX)
return -E2BIG;
/* Optimization for u64 mask */
if (nr_bits == 64) {
err = bpf_probe_read_kernel_common(&kit->bits_copy, nr_bytes, unsafe_ptr__ign);
if (err)
return -EFAULT;
swap_ulong_in_u64(&kit->bits_copy, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
if (bpf_mem_alloc_check_size(false, nr_bytes))
return -E2BIG;
/* Fallback to memalloc */
kit->bits = bpf_mem_alloc(&bpf_global_ma, nr_bytes);
if (!kit->bits)
return -ENOMEM;
err = bpf_probe_read_kernel_common(kit->bits, nr_bytes, unsafe_ptr__ign);
if (err) {
bpf_mem_free(&bpf_global_ma, kit->bits);
return err;
}
swap_ulong_in_u64(kit->bits, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
/**
* bpf_iter_bits_next() - Get the next bit in a bpf_iter_bits
* @it: The bpf_iter_bits to be checked
*
* This function returns a pointer to a number representing the value of the
* next bit in the bits.
*
* If there are no further bits available, it returns NULL.
*/
__bpf_kfunc int *bpf_iter_bits_next(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
int bit = kit->bit, nr_bits = kit->nr_bits;
const void *bits;
if (!nr_bits || bit >= nr_bits)
return NULL;
bits = nr_bits == 64 ? &kit->bits_copy : kit->bits;
bit = find_next_bit(bits, nr_bits, bit + 1);
if (bit >= nr_bits) {
kit->bit = bit;
return NULL;
}
kit->bit = bit;
return &kit->bit;
}
/**
* bpf_iter_bits_destroy() - Destroy a bpf_iter_bits
* @it: The bpf_iter_bits to be destroyed
*
* Destroy the resource associated with the bpf_iter_bits.
*/
__bpf_kfunc void bpf_iter_bits_destroy(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
if (kit->nr_bits <= 64)
return;
bpf_mem_free(&bpf_global_ma, kit->bits);
}
/**
* bpf_copy_from_user_str() - Copy a string from an unsafe user address
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address, in user space.
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL-terminated string from userspace to BPF space. If user string is
* too long this will still ensure zero termination in the dst buffer unless
* buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst to 0 on success and
* memset all of @dst on failure.
*/
__bpf_kfunc int bpf_copy_from_user_str(void *dst, u32 dst__sz, const void __user *unsafe_ptr__ign, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(!dst__sz))
return 0;
ret = strncpy_from_user(dst, unsafe_ptr__ign, dst__sz - 1);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst + ret, 0, dst__sz - ret);
else
((char *)dst)[ret] = '\0';
return ret + 1;
}
/**
* bpf_copy_from_user_task_str() - Copy a string from an task's address space
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address in the task's address space.
* @tsk: The task whose address space will be used
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL terminated string from a task's address space to @dst__sz
* buffer. If user string is too long this will still ensure zero termination
* in the @dst__sz buffer unless buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst__sz to 0 on success
* and memset all of @dst__sz on failure.
*
* Return: The number of copied bytes on success including the NUL terminator.
* A negative error code on failure.
*/
__bpf_kfunc int bpf_copy_from_user_task_str(void *dst, u32 dst__sz,
const void __user *unsafe_ptr__ign,
struct task_struct *tsk, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(dst__sz == 0))
return 0;
ret = copy_remote_vm_str(tsk, (unsigned long)unsafe_ptr__ign, dst, dst__sz, 0);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset(dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset(dst + ret, 0, dst__sz - ret);
return ret + 1;
}
/* Keep unsinged long in prototype so that kfunc is usable when emitted to
* vmlinux.h in BPF programs directly, but note that while in BPF prog, the
* unsigned long always points to 8-byte region on stack, the kernel may only
* read and write the 4-bytes on 32-bit.
*/
__bpf_kfunc void bpf_local_irq_save(unsigned long *flags__irq_flag)
{
local_irq_save(*flags__irq_flag);
}
__bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)
{
local_irq_restore(*flags__irq_flag);
}
__bpf_kfunc void __bpf_trap(void)
{
}
/*
* Kfuncs for string operations.
*
* Since strings are not necessarily %NUL-terminated, we cannot directly call
* in-kernel implementations. Instead, we open-code the implementations using
* __get_kernel_nofault instead of plain dereference to make them safe.
*/
static int __bpf_strncasecmp(const char *s1, const char *s2, bool ignore_case, size_t len)
{
char c1, c2;
int i;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < len && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c1, s1, char, err_out);
__get_kernel_nofault(&c2, s2, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 != c2)
return c1 < c2 ? -1 : 1;
if (c1 == '\0')
return 0;
s1++;
s2++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : 0;
err_out:
return -EFAULT;
}
/**
* bpf_strcmp - Compare two strings
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, false, XATTR_SIZE_MAX);
}
/**
* bpf_strcasecmp - Compare two strings, ignoring the case of the characters
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasecmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, XATTR_SIZE_MAX);
}
/*
* bpf_strncasecmp - Compare two length-limited strings, ignoring case
* @s1__ign: One string
* @s2__ign: Another string
* @len: The maximum number of characters to compare
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasecmp(const char *s1__ign, const char *s2__ign, size_t len)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, len);
}
/**
* bpf_strnchr - Find a character in a length limited string
* @s__ign: The string to be searched
* @count: The number of characters to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in the first @count characters of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnchr(const char *s__ign, size_t count, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
return i;
if (sc == '\0')
return -ENOENT;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : -ENOENT;
err_out:
return -EFAULT;
}
/**
* bpf_strchr - Find the first occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - The index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchr(const char *s__ign, char c)
{
return bpf_strnchr(s__ign, XATTR_SIZE_MAX, c);
}
/**
* bpf_strchrnul - Find and return a character in a string, or end of string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign or index of
* the null byte at the end of @s__ign when @c is not found
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchrnul(const char *s__ign, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == '\0' || sc == c)
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strrchr - Find the last occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the last occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strrchr(const char *s__ign, int c)
{
char sc;
int i, last = -ENOENT;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
last = i;
if (sc == '\0')
return last;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strnlen - Calculate the length of a length-limited string
* @s__ign: The string
* @count: The maximum number of characters to count
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnlen(const char *s__ign, size_t count)
{
char c;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c, s__ign, char, err_out);
if (c == '\0')
return i;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : i;
err_out:
return -EFAULT;
}
/**
* bpf_strlen - Calculate the length of a string
* @s__ign: The string
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strlen(const char *s__ign)
{
return bpf_strnlen(s__ign, XATTR_SIZE_MAX);
}
/**
* bpf_strspn - Calculate the length of the initial substring of @s__ign which
* only contains letters in @accept__ign
* @s__ign: The string to be searched
* @accept__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which only
* contains letters from @accept__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strspn(const char *s__ign, const char *accept__ign)
{
char cs, ca;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(accept__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&ca, accept__ign + j, char, err_out);
if (cs == ca || ca == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (ca == '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strcspn - Calculate the length of the initial substring of @s__ign which
* does not contain letters in @reject__ign
* @s__ign: The string to be searched
* @reject__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which does not
* contain letters from @reject__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcspn(const char *s__ign, const char *reject__ign)
{
char cs, cr;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(reject__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&cr, reject__ign + j, char, err_out);
if (cs == cr || cr == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (cr != '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
static int __bpf_strnstr(const char *s1, const char *s2, size_t len,
bool ignore_case)
{
char c1, c2;
int i, j;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
for (j = 0; i + j <= len && j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&c2, s2 + j, char, err_out);
if (c2 == '\0')
return i;
/*
* We allow reading an extra byte from s2 (note the
* `i + j <= len` above) to cover the case when s2 is
* a suffix of the first len chars of s1.
*/
if (i + j == len)
break;
__get_kernel_nofault(&c1, s1 + j, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 == '\0')
return -ENOENT;
if (c1 != c2)
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (i + j == len)
return -ENOENT;
s1++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strstr - Find the first substring in a string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strstr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, false);
}
/**
* bpf_strcasestr - Find the first substring in a string, ignoring the case of
* the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasestr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, true);
}
/**
* bpf_strnstr - Find the first substring in a length-limited string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strnstr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, false);
}
/**
* bpf_strncasestr - Find the first substring in a length-limited string,
* ignoring the case of the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasestr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, true);
}
#ifdef CONFIG_KEYS
/**
* bpf_lookup_user_key - lookup a key by its serial
* @serial: key handle serial number
* @flags: lookup-specific flags
*
* Search a key with a given *serial* and the provided *flags*.
* If found, increment the reference count of the key by one, and
* return it in the bpf_key structure.
*
* The bpf_key structure must be passed to bpf_key_put() when done
* with it, so that the key reference count is decremented and the
* bpf_key structure is freed.
*
* Permission checks are deferred to the time the key is used by
* one of the available key-specific kfuncs.
*
* Set *flags* with KEY_LOOKUP_CREATE, to attempt creating a requested
* special keyring (e.g. session keyring), if it doesn't yet exist.
* Set *flags* with KEY_LOOKUP_PARTIAL, to lookup a key without waiting
* for the key construction, and to retrieve uninstantiated keys (keys
* without data attached to them).
*
* Return: a bpf_key pointer with a valid key pointer if the key is found, a
* NULL pointer otherwise.
*/
__bpf_kfunc struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags)
{
key_ref_t key_ref;
struct bpf_key *bkey;
if (flags & ~KEY_LOOKUP_ALL)
return NULL;
/*
* Permission check is deferred until the key is used, as the
* intent of the caller is unknown here.
*/
key_ref = lookup_user_key(serial, flags, KEY_DEFER_PERM_CHECK);
if (IS_ERR(key_ref))
return NULL;
bkey = kmalloc_obj(*bkey);
if (!bkey) {
key_put(key_ref_to_ptr(key_ref));
return NULL;
}
bkey->key = key_ref_to_ptr(key_ref);
bkey->has_ref = true;
return bkey;
}
/**
* bpf_lookup_system_key - lookup a key by a system-defined ID
* @id: key ID
*
* Obtain a bpf_key structure with a key pointer set to the passed key ID.
* The key pointer is marked as invalid, to prevent bpf_key_put() from
* attempting to decrement the key reference count on that pointer. The key
* pointer set in such way is currently understood only by
* verify_pkcs7_signature().
*
* Set *id* to one of the values defined in include/linux/verification.h:
* 0 for the primary keyring (immutable keyring of system keys);
* VERIFY_USE_SECONDARY_KEYRING for both the primary and secondary keyring
* (where keys can be added only if they are vouched for by existing keys
* in those keyrings); VERIFY_USE_PLATFORM_KEYRING for the platform
* keyring (primarily used by the integrity subsystem to verify a kexec'ed
* kerned image and, possibly, the initramfs signature).
*
* Return: a bpf_key pointer with an invalid key pointer set from the
* pre-determined ID on success, a NULL pointer otherwise
*/
__bpf_kfunc struct bpf_key *bpf_lookup_system_key(u64 id)
{
struct bpf_key *bkey;
if (system_keyring_id_check(id) < 0)
return NULL;
bkey = kmalloc_obj(*bkey, GFP_ATOMIC);
if (!bkey)
return NULL;
bkey->key = (struct key *)(unsigned long)id;
bkey->has_ref = false;
return bkey;
}
/**
* bpf_key_put - decrement key reference count if key is valid and free bpf_key
* @bkey: bpf_key structure
*
* Decrement the reference count of the key inside *bkey*, if the pointer
* is valid, and free *bkey*.
*/
__bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
{
if (bkey->has_ref)
key_put(bkey->key);
kfree(bkey);
}
/**
* bpf_verify_pkcs7_signature - verify a PKCS#7 signature
* @data_p: data to verify
* @sig_p: signature of the data
* @trusted_keyring: keyring with keys trusted for signature verification
*
* Verify the PKCS#7 signature *sig_ptr* against the supplied *data_ptr*
* with keys in a keyring referenced by *trusted_keyring*.
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const void *data, *sig;
u32 data_len, sig_len;
int ret;
if (trusted_keyring->has_ref) {
/*
* Do the permission check deferred in bpf_lookup_user_key().
* See bpf_lookup_user_key() for more details.
*
* A call to key_task_permission() here would be redundant, as
* it is already done by keyring_search() called by
* find_asymmetric_key().
*/
ret = key_validate(trusted_keyring->key);
if (ret < 0)
return ret;
}
data_len = __bpf_dynptr_size(data_ptr);
data = __bpf_dynptr_data(data_ptr, data_len);
sig_len = __bpf_dynptr_size(sig_ptr);
sig = __bpf_dynptr_data(sig_ptr, sig_len);
return verify_pkcs7_signature(data, data_len, sig, sig_len,
trusted_keyring->key,
VERIFYING_BPF_SIGNATURE, NULL,
NULL);
#else
return -EOPNOTSUPP;
#endif /* CONFIG_SYSTEM_DATA_VERIFICATION */
}
#endif /* CONFIG_KEYS */
typedef int (*bpf_task_work_callback_t)(struct bpf_map *map, void *key, void *value);
enum bpf_task_work_state {
/* bpf_task_work is ready to be used */
BPF_TW_STANDBY = 0,
/* irq work scheduling in progress */
BPF_TW_PENDING,
/* task work scheduling in progress */
BPF_TW_SCHEDULING,
/* task work is scheduled successfully */
BPF_TW_SCHEDULED,
/* callback is running */
BPF_TW_RUNNING,
/* associated BPF map value is deleted */
BPF_TW_FREED,
};
struct bpf_task_work_ctx {
enum bpf_task_work_state state;
refcount_t refcnt;
struct callback_head work;
struct irq_work irq_work;
/* bpf_prog that schedules task work */
struct bpf_prog *prog;
/* task for which callback is scheduled */
struct task_struct *task;
/* the map and map value associated with this context */
struct bpf_map *map;
void *map_val;
enum task_work_notify_mode mode;
bpf_task_work_callback_t callback_fn;
struct rcu_head rcu;
} __aligned(8);
/* Actual type for struct bpf_task_work */
struct bpf_task_work_kern {
struct bpf_task_work_ctx *ctx;
};
static void bpf_task_work_ctx_reset(struct bpf_task_work_ctx *ctx)
{
if (ctx->prog) {
bpf_prog_put(ctx->prog);
ctx->prog = NULL;
}
if (ctx->task) {
bpf_task_release(ctx->task);
ctx->task = NULL;
}
}
static bool bpf_task_work_ctx_tryget(struct bpf_task_work_ctx *ctx)
{
return refcount_inc_not_zero(&ctx->refcnt);
}
static void bpf_task_work_destroy(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_ctx_reset(ctx);
kfree_rcu(ctx, rcu);
}
static void bpf_task_work_ctx_put(struct bpf_task_work_ctx *ctx)
{
if (!refcount_dec_and_test(&ctx->refcnt))
return;
if (irqs_disabled()) {
ctx->irq_work = IRQ_WORK_INIT(bpf_task_work_destroy);
irq_work_queue(&ctx->irq_work);
} else {
bpf_task_work_destroy(&ctx->irq_work);
}
}
static void bpf_task_work_cancel(struct bpf_task_work_ctx *ctx)
{
/*
* Scheduled task_work callback holds ctx ref, so if we successfully
* cancelled, we put that ref on callback's behalf. If we couldn't
* cancel, callback will inevitably run or has already completed
* running, and it would have taken care of its ctx ref itself.
*/
if (task_work_cancel(ctx->task, &ctx->work))
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_callback(struct callback_head *cb)
{
struct bpf_task_work_ctx *ctx = container_of(cb, struct bpf_task_work_ctx, work);
enum bpf_task_work_state state;
u32 idx;
void *key;
/* Read lock is needed to protect ctx and map key/value access */
guard(rcu_tasks_trace)();
/*
* This callback may start running before bpf_task_work_irq() switched to
* SCHEDULED state, so handle both transition variants SCHEDULING|SCHEDULED -> RUNNING.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_RUNNING);
if (state == BPF_TW_SCHEDULED)
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULED, BPF_TW_RUNNING);
if (state == BPF_TW_FREED) {
bpf_task_work_ctx_put(ctx);
return;
}
key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx);
migrate_disable();
ctx->callback_fn(ctx->map, key, ctx->map_val);
migrate_enable();
bpf_task_work_ctx_reset(ctx);
(void)cmpxchg(&ctx->state, BPF_TW_RUNNING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_irq(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
enum bpf_task_work_state state;
int err;
guard(rcu)();
if (cmpxchg(&ctx->state, BPF_TW_PENDING, BPF_TW_SCHEDULING) != BPF_TW_PENDING) {
bpf_task_work_ctx_put(ctx);
return;
}
err = task_work_add(ctx->task, &ctx->work, ctx->mode);
if (err) {
bpf_task_work_ctx_reset(ctx);
/*
* try to switch back to STANDBY for another task_work reuse, but we might have
* gone to FREED already, which is fine as we already cleaned up after ourselves
*/
(void)cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
return;
}
/*
* It's technically possible for just scheduled task_work callback to
* complete running by now, going SCHEDULING -> RUNNING and then
* dropping its ctx refcount. Instead of capturing an extra ref just
* to protect below ctx->state access, we rely on rcu_read_lock
* above to prevent kfree_rcu from freeing ctx before we return.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_SCHEDULED);
if (state == BPF_TW_FREED)
bpf_task_work_cancel(ctx); /* clean up if we switched into FREED state */
}
static struct bpf_task_work_ctx *bpf_task_work_fetch_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_kern *twk = (void *)tw;
struct bpf_task_work_ctx *ctx, *old_ctx;
ctx = READ_ONCE(twk->ctx);
if (ctx)
return ctx;
ctx = bpf_map_kmalloc_nolock(map, sizeof(*ctx), 0, NUMA_NO_NODE);
if (!ctx)
return ERR_PTR(-ENOMEM);
memset(ctx, 0, sizeof(*ctx));
refcount_set(&ctx->refcnt, 1); /* map's own ref */
ctx->state = BPF_TW_STANDBY;
old_ctx = cmpxchg(&twk->ctx, NULL, ctx);
if (old_ctx) {
/*
* tw->ctx is set by concurrent BPF program, release allocated
* memory and try to reuse already set context.
*/
kfree_nolock(ctx);
return old_ctx;
}
return ctx; /* Success */
}
static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_ctx *ctx;
/*
* Sleepable BPF programs hold rcu_read_lock_trace but not
* regular rcu_read_lock. Since kfree_rcu waits for regular
* RCU GP, the ctx can be freed while we're between reading
* the pointer and incrementing the refcount. Take regular
* rcu_read_lock to prevent kfree_rcu from freeing the ctx
* before we can tryget it.
*/
scoped_guard(rcu) {
ctx = bpf_task_work_fetch_ctx(tw, map);
if (IS_ERR(ctx))
return ctx;
/* try to get ref for task_work callback to hold */
if (!bpf_task_work_ctx_tryget(ctx))
return ERR_PTR(-EBUSY);
}
if (cmpxchg(&ctx->state, BPF_TW_STANDBY, BPF_TW_PENDING) != BPF_TW_STANDBY) {
/* lost acquiring race or map_release_uref() stole it from us, put ref and bail */
bpf_task_work_ctx_put(ctx);
return ERR_PTR(-EBUSY);
}
/*
* If no process or bpffs is holding a reference to the map, no new callbacks should be
* scheduled. This does not address any race or correctness issue, but rather is a policy
* choice: dropping user references should stop everything.
*/
if (!atomic64_read(&map->usercnt)) {
/* drop ref we just got for task_work callback itself */
bpf_task_work_ctx_put(ctx);
/* transfer map's ref into cancel_and_free() */
bpf_task_work_cancel_and_free(tw);
return ERR_PTR(-EBUSY);
}
return ctx;
}
static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw,
struct bpf_map *map, bpf_task_work_callback_t callback_fn,
struct bpf_prog_aux *aux, enum task_work_notify_mode mode)
{
struct bpf_prog *prog;
struct bpf_task_work_ctx *ctx;
int err;
BTF_TYPE_EMIT(struct bpf_task_work);
prog = bpf_prog_inc_not_zero(aux->prog);
if (IS_ERR(prog))
return -EBADF;
task = bpf_task_acquire(task);
if (!task) {
err = -EBADF;
goto release_prog;
}
ctx = bpf_task_work_acquire_ctx(tw, map);
if (IS_ERR(ctx)) {
err = PTR_ERR(ctx);
goto release_all;
}
ctx->task = task;
ctx->callback_fn = callback_fn;
ctx->prog = prog;
ctx->mode = mode;
ctx->map = map;
ctx->map_val = (void *)tw - map->record->task_work_off;
init_task_work(&ctx->work, bpf_task_work_callback);
init_irq_work(&ctx->irq_work, bpf_task_work_irq);
irq_work_queue(&ctx->irq_work);
return 0;
release_all:
bpf_task_release(task);
release_prog:
bpf_prog_put(prog);
return err;
}
/**
* bpf_task_work_schedule_signal - Schedule BPF callback using task_work_add with TWA_SIGNAL
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL);
}
/**
* bpf_task_work_schedule_resume - Schedule BPF callback using task_work_add with TWA_RESUME
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME);
}
static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
struct bpf_dynptr_kern *ptr)
{
struct bpf_dynptr_file_impl *state;
/* flags is currently unsupported */
if (flags) {
bpf_dynptr_set_null(ptr);
return -EINVAL;
}
state = kmalloc_nolock(sizeof(*state), 0, NUMA_NO_NODE);
if (!state) {
bpf_dynptr_set_null(ptr);
return -ENOMEM;
}
state->offset = 0;
state->size = U64_MAX; /* Don't restrict size, as file may change anyways */
freader_init_from_file(&state->freader, NULL, 0, file, may_sleep);
bpf_dynptr_init(ptr, state, BPF_DYNPTR_TYPE_FILE, 0, 0);
bpf_dynptr_set_rdonly(ptr);
return 0;
}
__bpf_kfunc int bpf_dynptr_from_file(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, false, (struct bpf_dynptr_kern *)ptr__uninit);
}
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, true, (struct bpf_dynptr_kern *)ptr__uninit);
}
__bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)dynptr;
struct bpf_dynptr_file_impl *df = ptr->data;
if (!df)
return 0;
freader_cleanup(&df->freader);
kfree_nolock(df);
bpf_dynptr_set_null(ptr);
return 0;
}
/**
* bpf_timer_cancel_async - try to deactivate a timer
* @timer: bpf_timer to stop
*
* Returns:
*
* * 0 when the timer was not active
* * 1 when the timer was active
* * -1 when the timer is currently executing the callback function and
* cannot be stopped
* * -ECANCELED when the timer will be cancelled asynchronously
* * -ENOMEM when out of memory
* * -EINVAL when the timer was not initialized
* * -ENOENT when this kfunc is racing with timer deletion
*/
__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer)
{
struct bpf_async_kern *async = (void *)timer;
struct bpf_async_cb *cb;
int ret;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
/*
* Unlike hrtimer_start() it's ok to synchronously call
* hrtimer_try_to_cancel() when refcnt reached zero, but deferring to
* irq_work is not, since irq callback may execute after RCU GP and
* cb could be freed at that time. Check for refcnt zero for
* consistency.
*/
if (!refcount_inc_not_zero(&cb->refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
ret = hrtimer_try_to_cancel(&t->timer);
bpf_async_refcount_put(cb);
return ret;
} else {
ret = bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
return ret ? ret : -ECANCELED;
}
}
__bpf_kfunc_end_defs();
static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_cancel(ctx); /* this might put task_work callback's ref */
bpf_task_work_ctx_put(ctx); /* and here we put map's own ref that was transferred to us */
}
void bpf_task_work_cancel_and_free(void *val)
{
struct bpf_task_work_kern *twk = val;
struct bpf_task_work_ctx *ctx;
enum bpf_task_work_state state;
ctx = xchg(&twk->ctx, NULL);
if (!ctx)
return;
state = xchg(&ctx->state, BPF_TW_FREED);
if (state == BPF_TW_SCHEDULED) {
/* run in irq_work to avoid locks in NMI */
init_irq_work(&ctx->irq_work, bpf_task_work_cancel_scheduled);
irq_work_queue(&ctx->irq_work);
return;
}
bpf_task_work_ctx_put(ctx); /* put bpf map's ref */
}
BTF_KFUNCS_START(generic_btf_ids)
#ifdef CONFIG_CRASH_DUMP
BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
#endif
BTF_ID_FLAGS(func, bpf_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_percpu_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_refcount_acquire, KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE | KF_RET_NULL | KF_RCU)
BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_front_impl)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_back_impl)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_rbtree_add_impl)
BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_cgroup_ancestor, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU)
BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
#endif
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_send_signal_task)
#endif
#ifdef CONFIG_KEYS
BTF_ID_FLAGS(func, bpf_lookup_user_key, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_lookup_system_key, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_key_put, KF_RELEASE)
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
BTF_ID_FLAGS(func, bpf_verify_pkcs7_signature, KF_SLEEPABLE)
#endif
#endif
#ifdef CONFIG_S390
BTF_ID_FLAGS(func, bpf_get_lowcore)
#endif
BTF_KFUNCS_END(generic_btf_ids)
static const struct btf_kfunc_id_set generic_kfunc_set = {
.owner = THIS_MODULE,
.set = &generic_btf_ids,
};
BTF_ID_LIST(generic_dtor_ids)
BTF_ID(struct, task_struct)
BTF_ID(func, bpf_task_release_dtor)
#ifdef CONFIG_CGROUPS
BTF_ID(struct, cgroup)
BTF_ID(func, bpf_cgroup_release_dtor)
#endif
BTF_KFUNCS_START(common_btf_ids)
BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rcu_read_lock)
BTF_ID_FLAGS(func, bpf_rcu_read_unlock)
BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_css_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_css_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_destroy, KF_ITER_DESTROY)
#endif
BTF_ID_FLAGS(func, bpf_iter_task_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_dynptr_adjust)
BTF_ID_FLAGS(func, bpf_dynptr_is_null)
BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
BTF_ID_FLAGS(func, bpf_dynptr_size)
BTF_ID_FLAGS(func, bpf_dynptr_clone)
BTF_ID_FLAGS(func, bpf_dynptr_copy)
BTF_ID_FLAGS(func, bpf_dynptr_memset)
#ifdef CONFIG_NET
BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
#endif
BTF_ID_FLAGS(func, bpf_wq_init)
BTF_ID_FLAGS(func, bpf_wq_set_callback, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_wq_start)
BTF_ID_FLAGS(func, bpf_preempt_disable)
BTF_ID_FLAGS(func, bpf_preempt_enable)
BTF_ID_FLAGS(func, bpf_iter_bits_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_bits_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_bits_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_copy_from_user_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_get_kmem_cache)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_local_irq_save)
BTF_ID_FLAGS(func, bpf_local_irq_restore)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr)
BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE)
#endif
#ifdef CONFIG_DMA_SHARED_BUFFER
BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
#endif
BTF_ID_FLAGS(func, __bpf_trap)
BTF_ID_FLAGS(func, bpf_strcmp);
BTF_ID_FLAGS(func, bpf_strcasecmp);
BTF_ID_FLAGS(func, bpf_strncasecmp);
BTF_ID_FLAGS(func, bpf_strchr);
BTF_ID_FLAGS(func, bpf_strchrnul);
BTF_ID_FLAGS(func, bpf_strnchr);
BTF_ID_FLAGS(func, bpf_strrchr);
BTF_ID_FLAGS(func, bpf_strlen);
BTF_ID_FLAGS(func, bpf_strnlen);
BTF_ID_FLAGS(func, bpf_strspn);
BTF_ID_FLAGS(func, bpf_strcspn);
BTF_ID_FLAGS(func, bpf_strstr);
BTF_ID_FLAGS(func, bpf_strcasestr);
BTF_ID_FLAGS(func, bpf_strnstr);
BTF_ID_FLAGS(func, bpf_strncasestr);
#if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS)
BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU)
#endif
BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_dynptr_from_file)
BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
BTF_ID_FLAGS(func, bpf_timer_cancel_async)
BTF_KFUNCS_END(common_btf_ids)
static const struct btf_kfunc_id_set common_kfunc_set = {
.owner = THIS_MODULE,
.set = &common_btf_ids,
};
static int __init kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc generic_dtors[] = {
{
.btf_id = generic_dtor_ids[0],
.kfunc_btf_id = generic_dtor_ids[1]
},
#ifdef CONFIG_CGROUPS
{
.btf_id = generic_dtor_ids[2],
.kfunc_btf_id = generic_dtor_ids[3]
},
#endif
};
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &generic_kfunc_set);
ret = ret ?: register_btf_id_dtor_kfuncs(generic_dtors,
ARRAY_SIZE(generic_dtors),
THIS_MODULE);
return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
}
late_initcall(kfunc_init);
/* Get a pointer to dynptr data up to len bytes for read only access. If
* the dynptr doesn't have continuous data up to len bytes, return NULL.
*/
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u64 len)
{
const struct bpf_dynptr *p = (struct bpf_dynptr *)ptr;
return bpf_dynptr_slice(p, 0, NULL, len);
}
/* Get a pointer to dynptr data up to len bytes for read write access. If
* the dynptr doesn't have continuous data up to len bytes, or the dynptr
* is read only, return NULL.
*/
void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u64 len)
{
if (__bpf_dynptr_is_rdonly(ptr))
return NULL;
return (void *)__bpf_dynptr_data(ptr, len);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *val)
{
if (btf_record_has_field(map->record, BPF_TIMER))
bpf_obj_free_timer(map->record, val);
if (btf_record_has_field(map->record, BPF_WORKQUEUE))
bpf_obj_free_workqueue(map->record, val);
if (btf_record_has_field(map->record, BPF_TASK_WORK))
bpf_obj_free_task_work(map->record, val);
}
]
|
| 475/6 |
2026/05/18 19:28 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 216669
output: 40
thoughts: 200
Thoughts: **BPF's `printf` Implementations: A Quick Dive**
Okay, so I'm digging into BPF's string formatting capabilities, specifically looking at how we've implemented something like `printf`. My initial search for "printf" turned up a few interesting leads. First, I noticed `bpf_seq_printf` defined in `include/uapi/linux/bpf.h`, but it doesn't seem to have a corresponding implementation directly in `helpers.c` – that's a bit odd. Then there's `bpf_snprintf`, which *is* implemented in `helpers.c`, along with `bstr_printf`, which seems to be used by `bpf_snprintf`. `bpf_trace_vprintk` surfaced, but that's in `kernel/trace/bpf_trace.c`, so it's probably using its own underlying mechanisms, even if it's based on `bstr_printf`.
Ah, there's `bpf_stream_vprintk` as well, but that's a kfunc ID. That's not what I'm looking for.
I'm going to shift gears and look at the implementation of `bpf_stream_vprintk` now.
Let's see where that takes me...
|
| 476/6 |
2026/05/18 19:28 |
tool |
codesearch-find-references |
0m
Args: map[Name:bpf_stream_vprintk SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 477/6 |
2026/05/18 19:28 |
llm |
codeexpert-compressor |
0m
Model:
Reply: The investigation into `kernel/bpf/helpers.c` regarding the improper use of `snprintf` has been completed. The research confirms that the vulnerability is localized to a specific function and identifies the exact lines of code involved.
### 1. Approaches and Results
* **Exhaustive Symbol Search:**
* The agent performed a targeted search for `snprintf` within `kernel/bpf/helpers.c`.
* **Result:** Only two instances of `snprintf` were found in the entire file. Both are located within the function `bpf_bprintf_prepare`.
* **Variant and Engine Analysis:**
* Searched for other formatting variants like `printf`, `sprintf`, and `vsnprintf`.
* **Result:** No direct calls to these functions exist in `helpers.c`. The primary string formatting for BPF helpers (like `bpf_snprintf`) is handled by `bstr_printf`.
* **Logic Verification of `bstr_printf`:**
* The agent examined the definition and comments for `bstr_printf` in `lib/vsprintf.c`.
* **Result:** Confirmed that `bstr_printf` follows C99 `vsnprintf` semantics (returning the number of characters that *would* have been written). However, the usage of `bstr_printf` in the `bpf_snprintf` helper (line 1085) is safe because it does not perform the dangerous pointer arithmetic seen in the preparation phase.
* **Kfunc and Lateral Investigation:**
* Checked for newer kfuncs like `bpf_stream_vprintk` to see if they introduced new formatting logic.
* **Result:** No references to `bpf_stream_vprintk` or similar logic were found within `kernel/bpf/helpers.c` that utilize `snprintf`.
* **Dead Ends:**
* A search for the generic `printf` symbol returned no results, confirming that the file relies strictly on specific kernel formatting utilities.
* Re-reading the file confirmed that while many helpers handle strings (e.g., `bpf_get_current_comm`, `bpf_copy_from_user_str`), they use `strscpy_pad` or `strncpy_from_user`, which do not share the `snprintf` return value pitfall.
### 2. Current Findings and Hypotheses
The hypothesis that the vulnerability is limited to the argument preparation logic is confirmed. There are **no other places** in `kernel/bpf/helpers.c` where `snprintf` is used.
**The Vulnerability Mechanism:**
The issue exists because `bpf_bprintf_prepare` treats the return value of `snprintf` as the number of bytes actually written to the buffer. In reality, `snprintf` returns the length the string *would have been* if enough space were available. If the output is truncated, the code advances the buffer pointer (`tmp_buf`) beyond the actual end of the buffer (`tmp_buf_end`), leading to a potential out-of-bounds write in subsequent loop iterations.
### 3. Critical File Paths and Code Snippets
* **File:** `kernel/bpf/helpers.c`
* **Function:** `bpf_bprintf_prepare`
* **Vulnerable Snippet 1 (Line 924):**
```c
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1); // Vulnerable: err can exceed remaining space
```
* **Vulnerable Snippet 2 (Line 965):**
```c
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1; // Vulnerable: err can exceed remaining space
```
### 4. Reasoning Loops and Observations
* **Repetitive File Access:** The agent read `kernel/bpf/helpers.c` twice. The second read was used to verify the absence of other formatting calls after the initial `snprintf` search, ensuring no macros or complex wrappers were missed.
* **Linear Investigation:** The investigation moved logically from identifying the known vulnerability to ruling out similar patterns in related helpers (`bpf_snprintf`) and kfuncs.
* **Conclusion:** No further tool calls are necessary. The scope of the issue in this specific file is fully mapped.
Tokens:
input: 213174
output: 1051
thoughts: 1145
|
| 478/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 3351
output: 35
thoughts: 372
Thoughts: **My Investigation into Potential `snprintf` Vulnerabilities in `kernel/bpf/helpers.c`**
Okay, so I need to verify this claim about `snprintf` usage in `kernel/bpf/helpers.c`. The previous summary indicates that only the instances within `bpf_bprintf_prepare` were found, which makes sense given the focus on this specific issue. But as a good practice I will verify that information thoroughly.
My first thought is to grab a listing of the file and perform a quick `snprintf` search. I should also check for variations like `vsnprintf`, `scnprintf`, and `sprintf` just to be absolutely thorough. The summary also mentioned `bstr_printf`, which was deemed safe due to its usage within `bpf_snprintf`. Still, I'll keep that in mind.
Wait, the prompt is specifically asking "Are there *other* places?". The summary states that the previous research "confirms that the vulnerability is localized" to `bpf_bprintf_prepare`. It seems like the investigation has already been done, but I still want to confirm this myself to solidify my understanding.
I'm starting with a direct search for `snprintf` in the `kernel/bpf/helpers.c` file and then expanding my search to include similar string formatting functions like `vsnprintf`, `scnprintf`, and `sprintf` to cover all my bases. I'm going to start there.
|
| 479/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Args: map[Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965]] TruncatedOutput:false]
|
| 480/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 4192
output: 105
thoughts: 53
Thoughts: Okay, here's my take on those thoughts, framed as my own:
**Investigating String Formatting in the BPF Context**
Right, let's see. My initial hunch about `snprintf` in `kernel/bpf/helpers.c` only turned up two instances within `bpf_bprintf_prepare`. That's not quite what I was expecting; perhaps my initial understanding was off. I need to be broader in my search. My next step is to cast the net wider. I should also investigate other string formatting function calls and associated code, and check for `scnprintf`, `vsnprintf`, and `sprintf` too. This might uncover any related implementations or alternative approaches the BPF infrastructure utilizes, as the low-level nature of this environment could require bespoke string formatting.
|
| 481/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Args: map[Name:scnprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 482/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Args: map[Name:vsnprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 483/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Args: map[Name:sprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 484/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 4565
output: 37
thoughts: 24
Thoughts: Okay, here's my interpretation of those thoughts, framed as my own:
**Initial Assessment: Absent Data, Targeted Search**
Alright, so the initial scan came up empty. No hits. Not surprising, but now we move to a more focused search. The summary mentioned `bstr_printf`, so that's the key term. Let's start there. This should be a direct lookup for a specific function we were looking for. The approach here is precise, surgical. Let's see if we can find it.
|
| 485/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Args: map[Name:bstr_printf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:____bpf_snprintf SourceFile:kernel/bpf/helpers.c SourceLine:1084]] TruncatedOutput:false]
|
| 486/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 4809
output: 43
thoughts: 23
Thoughts: Okay, let's dive into this. Here's my take on examining `____bpf_snprintf` in `kernel/bpf/helpers.c`:
**Dissecting `____bpf_snprintf`**
Right, so I'm focusing on `____bpf_snprintf` within `kernel/bpf/helpers.c`. I need to understand how this function is implemented and, more importantly, how it's used within the BPF context. I know this is a core utility. I'm immediately thinking about: what are the limitations imposed on this implementation in a BPF environment? It's not the full `snprintf` of userspace, so there are likely restrictions on buffer sizes, format specifiers, and potential for runtime errors. I'll need to look closely at memory management, especially within the BPF virtual machine. Is there any validation or bounds-checking to prevent buffer overflows or memory corruption? Are there specific error codes returned if the format string or output buffer is invalid? I'll also be considering its performance characteristics – how quickly does it execute, and are there any optimizations in place for common use cases? I'm anticipating that this is a critical helper, so it will need to be as efficient as possible. The goal is to fully understand its behaviour in the context of writing effective and robust BPF programs.
|
| 487/6 |
2026/05/18 19:29 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:____bpf_snprintf]
Results: map[SourceCode:1064: BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
1065: const void *, args, u32, data_len)
1066: {
1067: struct bpf_bprintf_data data = {
1068: .get_bin_args = true,
1069: };
1070: int err, num_args;
1071:
1072: if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
1073: (data_len && !args))
1074: return -EINVAL;
1075: num_args = data_len / 8;
1076:
1077: /* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
1078: * can safely give an unbounded size.
1079: */
1080: err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
1081: if (err < 0)
1082: return err;
1083:
1084: err = bstr_printf(str, str_size, fmt, data.bin_args);
1085:
1086: bpf_bprintf_cleanup(&data);
1087:
1088: return err + 1;
1089: }
SourceFile:kernel/bpf/helpers.c]
|
| 488/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 5598
output: 44
thoughts: 380
Thoughts: **My Analysis of the String Formatting Functions**
Okay, let me break this down. In `____bpf_snprintf`, the code calls `bstr_printf`. I see that the return value is `err + 1`. Now, `bstr_printf`, from what I recall, returns the number of characters that *would* have been printed. If the buffer size, `str_size`, is zero, then `err` itself becomes the length of the *formatted* string. Given that `bpf_snprintf` is documented to return the total number of bytes, *including* the null terminator, adding 1 to `err` makes sense here. It seems like the code is correctly accounting for the null termination. So, that part seems okay.
But the real concern is about using `snprintf` without proper return value checks. I need to focus on this.
Here, in `bpf_bprintf_prepare`, `snprintf` is used, and crucially, its return value is used to increment a pointer, `tmp_buf`. Specifically:
```c
tmp_buf += (err + 1);
```
The thing is, if `err` is *larger* than the available space (i.e., `tmp_buf_end - tmp_buf`), this means the incremented pointer, `tmp_buf`, will end up pointing *past* `tmp_buf_end`. That's a classic buffer overflow vulnerability waiting to happen. It looks like this increment should be checked to ensure that we are not going beyond the bounds of `tmp_buf`. I should probably examine the full context of `bpf_bprintf_prepare` to confirm this suspicion, and see if there are other potential issues in this function.
|
| 489/6 |
2026/05/18 19:29 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_prepare]
Results: map[SourceCode: 804: /*
805: * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
806: *
807: * Returns a negative value if fmt is an invalid format string or 0 otherwise.
808: *
809: * This can be used in two ways:
810: * - Format string verification only: when data->get_bin_args is false
811: * - Arguments preparation: in addition to the above verification, it writes in
812: * data->bin_args a binary representation of arguments usable by bstr_printf
813: * where pointers from BPF have been sanitized.
814: *
815: * In argument preparation mode, if 0 is returned, safe temporary buffers are
816: * allocated and bpf_bprintf_cleanup should be called to free them after use.
817: */
818: int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
819: u32 num_args, struct bpf_bprintf_data *data)
820: {
821: bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
822: char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
823: struct bpf_bprintf_buffers *buffers = NULL;
824: size_t sizeof_cur_arg, sizeof_cur_ip;
825: int err, i, num_spec = 0;
826: u64 cur_arg;
827: char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
828:
829: fmt_end = strnchr(fmt, fmt_size, 0);
830: if (!fmt_end)
831: return -EINVAL;
832: fmt_size = fmt_end - fmt;
833:
834: if (get_buffers && bpf_try_get_buffers(&buffers))
835: return -EBUSY;
836:
837: if (data->get_bin_args) {
838: if (num_args)
839: tmp_buf = buffers->bin_args;
840: tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
841: data->bin_args = (u32 *)tmp_buf;
842: }
843:
844: if (data->get_buf)
845: data->buf = buffers->buf;
846:
847: for (i = 0; i < fmt_size; i++) {
848: unsigned char c = fmt[i];
849:
850: /*
851: * Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
852: * through unchanged, while still rejecting ASCII control bytes.
853: */
854: if (isascii(c) && !isprint(c) && !isspace(c)) {
855: err = -EINVAL;
856: goto out;
857: }
858:
859: if (fmt[i] != '%')
860: continue;
861:
862: if (fmt[i + 1] == '%') {
863: i++;
864: continue;
865: }
866:
867: if (num_spec >= num_args) {
868: err = -EINVAL;
869: goto out;
870: }
871:
872: /* The string is zero-terminated so if fmt[i] != 0, we can
873: * always access fmt[i + 1], in the worst case it will be a 0
874: */
875: i++;
876: c = fmt[i];
877: /*
878: * The format parser below only understands ASCII conversion
879: * specifiers and modifiers, so reject non-ASCII after '%'.
880: */
881: if (!isascii(c)) {
882: err = -EINVAL;
883: goto out;
884: }
885:
886: /* skip optional "[0 +-][num]" width formatting field */
887: while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
888: fmt[i] == ' ')
889: i++;
890: if (fmt[i] >= '1' && fmt[i] <= '9') {
891: i++;
892: while (fmt[i] >= '0' && fmt[i] <= '9')
893: i++;
894: }
895:
896: if (fmt[i] == 'p') {
897: sizeof_cur_arg = sizeof(long);
898:
899: if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
900: ispunct(fmt[i + 1])) {
901: if (tmp_buf)
902: cur_arg = raw_args[num_spec];
903: goto nocopy_fmt;
904: }
905:
906: if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
907: fmt[i + 2] == 's') {
908: fmt_ptype = fmt[i + 1];
909: i += 2;
910: goto fmt_str;
911: }
912:
913: if (fmt[i + 1] == 'K' ||
914: fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
915: fmt[i + 1] == 'S') {
916: if (tmp_buf)
917: cur_arg = raw_args[num_spec];
918: i++;
919: goto nocopy_fmt;
920: }
921:
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
930:
931: i++;
932: num_spec++;
933: continue;
934: }
935:
936: /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
937: if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
938: (fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
939: err = -EINVAL;
940: goto out;
941: }
942:
943: i += 2;
944: if (!tmp_buf)
945: goto nocopy_fmt;
946:
947: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
948: if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
949: err = -ENOSPC;
950: goto out;
951: }
952:
953: unsafe_ptr = (char *)(long)raw_args[num_spec];
954: err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
955: sizeof_cur_ip);
956: if (err < 0)
957: memset(cur_ip, 0, sizeof_cur_ip);
958:
959: /* hack: bstr_printf expects IP addresses to be
960: * pre-formatted as strings, ironically, the easiest way
961: * to do that is to call snprintf.
962: */
963: ip_spec[2] = fmt[i - 1];
964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
969: num_spec++;
970:
971: continue;
972: } else if (fmt[i] == 's') {
973: fmt_ptype = fmt[i];
974: fmt_str:
975: if (fmt[i + 1] != 0 &&
976: !isspace(fmt[i + 1]) &&
977: !ispunct(fmt[i + 1])) {
978: err = -EINVAL;
979: goto out;
980: }
981:
982: if (!tmp_buf)
983: goto nocopy_fmt;
984:
985: if (tmp_buf_end == tmp_buf) {
986: err = -ENOSPC;
987: goto out;
988: }
989:
990: unsafe_ptr = (char *)(long)raw_args[num_spec];
991: err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
992: fmt_ptype,
993: tmp_buf_end - tmp_buf);
994: if (err < 0) {
995: tmp_buf[0] = '\0';
996: err = 1;
997: }
998:
999: tmp_buf += err;
1000: num_spec++;
1001:
1002: continue;
1003: } else if (fmt[i] == 'c') {
1004: if (!tmp_buf)
1005: goto nocopy_fmt;
1006:
1007: if (tmp_buf_end == tmp_buf) {
1008: err = -ENOSPC;
1009: goto out;
1010: }
1011:
1012: *tmp_buf = raw_args[num_spec];
1013: tmp_buf++;
1014: num_spec++;
1015:
1016: continue;
1017: }
1018:
1019: sizeof_cur_arg = sizeof(int);
1020:
1021: if (fmt[i] == 'l') {
1022: sizeof_cur_arg = sizeof(long);
1023: i++;
1024: }
1025: if (fmt[i] == 'l') {
1026: sizeof_cur_arg = sizeof(long long);
1027: i++;
1028: }
1029:
1030: if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
1031: fmt[i] != 'x' && fmt[i] != 'X') {
1032: err = -EINVAL;
1033: goto out;
1034: }
1035:
1036: if (tmp_buf)
1037: cur_arg = raw_args[num_spec];
1038: nocopy_fmt:
1039: if (tmp_buf) {
1040: tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
1041: if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
1042: err = -ENOSPC;
1043: goto out;
1044: }
1045:
1046: if (sizeof_cur_arg == 8) {
1047: *(u32 *)tmp_buf = *(u32 *)&cur_arg;
1048: *(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
1049: } else {
1050: *(u32 *)tmp_buf = (u32)(long)cur_arg;
1051: }
1052: tmp_buf += sizeof_cur_arg;
1053: }
1054: num_spec++;
1055: }
1056:
1057: err = 0;
1058: out:
1059: if (err)
1060: bpf_bprintf_cleanup(data);
1061: return err;
1062: }
SourceFile:kernel/bpf/helpers.c]
|
| 490/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 10233
output: 36
thoughts: 865
Thoughts: **Analyzing Potential Buffer Overflows in Format String Handling**
Okay, so I've been digging into this code, specifically looking for potential buffer overflows stemming from format string vulnerabilities. My focus is on how `snprintf` is used and if any subsequent pointer arithmetic could lead to issues.
I see two calls to `snprintf` that immediately raise my eyebrows. The first one, on line 924, uses `%pB` to print a pointer, and the second, on line 965, handles an IP address with a format specifier `ip_spec`. The critical thing here is how `err`, the return value of `snprintf` (the number of characters *that would have been written*, not necessarily *that were written*), is used. I'm noting that `tmp_buf` is incremented by `err + 1`, and if `err` is larger than the remaining buffer space (`tmp_buf_end - tmp_buf`), `tmp_buf` will advance beyond `tmp_buf_end`.
This is the key. The buffer overflow can happen when the pointer `tmp_buf` is advanced beyond the allowed buffer `tmp_buf_end` and then, later, dereferenced. In particular, I'm concerned about the pointer arithmetic at line 1040: `tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));` followed by the check `if (tmp_buf_end - tmp_buf < sizeof_cur_arg)`. Since I know that `tmp_buf` can be past `tmp_buf_end`, then the difference `tmp_buf_end - tmp_buf` will be a negative value. Since `sizeof_cur_arg` is a `size_t` (unsigned), this negative value is promoted to a very large unsigned value. This causes the comparison `(very_large_unsigned) < sizeof_cur_arg` to be *false*, and then the code proceeds to perform writes at lines 1047 and 1048. These writes dereference the out-of-bounds `tmp_buf` pointer, resulting in a buffer overflow.
Therefore, the critical question becomes: are there *any other* places where similar vulnerabilities might be present? I've already scrutinized the more common suspects like `snprintf`, `scnprintf`, `vsnprintf`, and `sprintf` and their usage in the code.
I'm now looking at other potential formatting functions. I've found `bstr_printf` and its usage in `____bpf_snprintf`. Here, the code calls `bstr_printf` but importantly, `err` (the return from `bstr_printf`) is *not* used in pointer arithmetic. The `bpf_snprintf` helper returns `err + 1`. This isn't immediately problematic because `bstr_printf` returns the *needed* length, and `bpf_snprintf` isn't using that result to advance a buffer pointer. So, that seems safe.
But I'm not stopping there; are there *any* other formatting functions I've missed? Like, say, `vscnprintf`? That's what I need to investigate next.
|
| 491/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Args: map[Name:vscnprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 492/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 11843
output: 36
thoughts: 55
Thoughts: Okay, let's see, my mind is jumping around a bit here, trying to get a grasp of this.
**Potential Parsing Approaches**
Alright, my initial thought is, "What about `sscanf`?" Probably not the ideal approach, given the context, but it's always good to consider the basics and rule them out. I mean, it's a standard C library function for parsing formatted input, right? But I'm thinking, this task probably involves something a bit more sophisticated, given the typical demands in this area.
Now, `seq_printf`... that's interesting. `helpers.c` often uses that for debugfs or similar reporting functions, right? But the problem I'm dealing with is likely related to BPF helpers, and I don't recall seeing `seq_printf` used in a common BPF helper context. BPF helpers tend to be focused on performance and low-level data access, so it's probably not the direct fit I'm looking for. Let's keep moving.
|
| 493/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Args: map[Name:seq_printf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 494/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 12172
output: 23
thoughts: 25
Thoughts: Okay, here's my take on those thoughts, framed as an expert's internal monologue:
**Checking for Related I/O Functions**
Alright, before I dive too deep, I need to make sure I haven't missed any other similar I/O functions besides the obvious `printf`. It's easy to get tunnel vision, especially when you're focusing on a specific instance. So, I'm going to run a quick search for `printf` as a substring. This will catch variations, custom implementations, or potentially even macros that expand into something printf-like. This approach should be efficient in identifying potentially missed code paths. Better to be thorough now and ensure I haven't overlooked something critical that would affect my overall analysis.
|
| 495/6 |
2026/05/18 19:29 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:[arena.c arraymap.c backtrack.c bloom_filter.c bpf_cgrp_storage.c bpf_inode_storage.c bpf_insn_array.c bpf_iter.c bpf_local_storage.c bpf_lru_list.c bpf_lru_list.h bpf_lsm.c bpf_lsm_proto.c bpf_struct_ops.c bpf_task_storage.c btf.c btf_iter.c btf_relocate.c cfg.c cgroup.c cgroup_iter.c check_btf.c const_fold.c core.c cpumap.c cpumask.c crypto.c devmap.c disasm.c disasm.h dispatcher.c dmabuf_iter.c fixups.c hashtab.c helpers.c inode.c kmem_cache_iter.c link_iter.c liveness.c local_storage.c log.c lpm_trie.c map_in_map.c map_in_map.h map_iter.c memalloc.c mmap_unlock_work.h mprog.c net_namespace.c offload.c percpu_freelist.c percpu_freelist.h prog_iter.c queue_stack_maps.c range_tree.c range_tree.h relo_core.c reuseport_array.c ringbuf.c rqspinlock.c rqspinlock.h stackmap.c states.c stream.c syscall.c sysfs_btf.c task_iter.c tcx.c tnum.c token.c trampoline.c verifier.c] Subdirs:[preload]]
|
| 496/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 12988
output: 34
thoughts: 26
Thoughts: Okay, here's my summary, presented as a stream of thought:
**Analyzing Kernel BPF Helper Function Usage**
Alright, I need to investigate the usage of `printf` within the kernel's BPF helpers file, specifically `kernel/bpf/helpers.c`. I need to understand exactly where these `printf` calls are being made within that specific file to assess their impact and if they're necessary. Time to get to the core of the matter: I'll quickly leverage `grep`. It’s the perfect tool for this type of surgical search. I will use `grep` to quickly and efficiently locate all instances of the string “`printf`” within the specified file. This will give me a precise list of lines to examine.
|
| 497/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:printf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 498/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 13227
output: 23
thoughts: 29
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Manual Symbol Reference Search**
Alright, so `codesearch-find-references` isn't going to cut it here. I need to be precise, and I suspect it's looking for something too general. The symbol name needs to be exact, and my current context suggests that a broader, more flexible search is necessary. So, I'm going to leverage `read-file` directly. This approach will allow me to target the specific text and manually parse the code. It might be slightly more involved than a fully automated process, but this offers superior control over the search's criteria and should be faster than fumbling with less precise tools.
|
| 499/6 |
2026/05/18 19:29 |
tool |
read-file |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*/
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/bpf-cgroup.h>
#include <linux/cgroup.h>
#include <linux/rcupdate.h>
#include <linux/random.h>
#include <linux/smp.h>
#include <linux/topology.h>
#include <linux/ktime.h>
#include <linux/sched.h>
#include <linux/uidgid.h>
#include <linux/filter.h>
#include <linux/ctype.h>
#include <linux/jiffies.h>
#include <linux/pid_namespace.h>
#include <linux/poison.h>
#include <linux/proc_ns.h>
#include <linux/sched/task.h>
#include <linux/security.h>
#include <linux/btf_ids.h>
#include <linux/bpf_mem_alloc.h>
#include <linux/kasan.h>
#include <linux/bpf_verifier.h>
#include <linux/uaccess.h>
#include <linux/verification.h>
#include <linux/task_work.h>
#include <linux/irq_work.h>
#include <linux/buildid.h>
#include "../../lib/kstrtox.h"
/* If kernel subsystem is allowing eBPF programs to call this function,
* inside its own verifier_ops->get_func_proto() callback it should return
* bpf_map_lookup_elem_proto, so that verifier can properly check the arguments
*
* Different map implementations will rely on rcu in map methods
* lookup/update/delete, therefore eBPF programs must run under rcu lock
* if program is allowed to access maps, so check rcu_read_lock_held() or
* rcu_read_lock_trace_held() in all three functions.
*/
BPF_CALL_2(bpf_map_lookup_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_elem(map, key);
}
const struct bpf_func_proto bpf_map_lookup_elem_proto = {
.func = bpf_map_lookup_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_4(bpf_map_update_elem, struct bpf_map *, map, void *, key,
void *, value, u64, flags)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_update_elem(map, key, value, flags);
}
const struct bpf_func_proto bpf_map_update_elem_proto = {
.func = bpf_map_update_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_PTR_TO_MAP_VALUE,
.arg4_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_delete_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_delete_elem(map, key);
}
const struct bpf_func_proto bpf_map_delete_elem_proto = {
.func = bpf_map_delete_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_3(bpf_map_push_elem, struct bpf_map *, map, void *, value, u64, flags)
{
return map->ops->map_push_elem(map, value, flags);
}
const struct bpf_func_proto bpf_map_push_elem_proto = {
.func = bpf_map_push_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_pop_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_pop_elem(map, value);
}
const struct bpf_func_proto bpf_map_pop_elem_proto = {
.func = bpf_map_pop_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_2(bpf_map_peek_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_peek_elem(map, value);
}
const struct bpf_func_proto bpf_map_peek_elem_proto = {
.func = bpf_map_peek_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_3(bpf_map_lookup_percpu_elem, struct bpf_map *, map, void *, key, u32, cpu)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_percpu_elem(map, key, cpu);
}
const struct bpf_func_proto bpf_map_lookup_percpu_elem_proto = {
.func = bpf_map_lookup_percpu_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_ANYTHING,
};
const struct bpf_func_proto bpf_get_prandom_u32_proto = {
.func = bpf_user_rnd_u32,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_smp_processor_id)
{
return smp_processor_id();
}
const struct bpf_func_proto bpf_get_smp_processor_id_proto = {
.func = bpf_get_smp_processor_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.allow_fastcall = true,
};
BPF_CALL_0(bpf_get_numa_node_id)
{
return numa_node_id();
}
const struct bpf_func_proto bpf_get_numa_node_id_proto = {
.func = bpf_get_numa_node_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_ns)
{
/* NMI safe access to clock monotonic */
return ktime_get_mono_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_ns_proto = {
.func = bpf_ktime_get_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_boot_ns)
{
/* NMI safe access to clock boottime */
return ktime_get_boot_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_boot_ns_proto = {
.func = bpf_ktime_get_boot_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_coarse_ns)
{
return ktime_get_coarse_ns();
}
const struct bpf_func_proto bpf_ktime_get_coarse_ns_proto = {
.func = bpf_ktime_get_coarse_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_tai_ns)
{
/* NMI safe access to clock tai */
return ktime_get_tai_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_tai_ns_proto = {
.func = bpf_ktime_get_tai_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_pid_tgid)
{
struct task_struct *task = current;
if (unlikely(!task))
return -EINVAL;
return (u64) task->tgid << 32 | task->pid;
}
const struct bpf_func_proto bpf_get_current_pid_tgid_proto = {
.func = bpf_get_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_uid_gid)
{
struct task_struct *task = current;
kuid_t uid;
kgid_t gid;
if (unlikely(!task))
return -EINVAL;
current_uid_gid(&uid, &gid);
return (u64) from_kgid(&init_user_ns, gid) << 32 |
from_kuid(&init_user_ns, uid);
}
const struct bpf_func_proto bpf_get_current_uid_gid_proto = {
.func = bpf_get_current_uid_gid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_2(bpf_get_current_comm, char *, buf, u32, size)
{
struct task_struct *task = current;
if (unlikely(!task))
goto err_clear;
/* Verifier guarantees that size > 0 */
strscpy_pad(buf, task->comm, size);
return 0;
err_clear:
memset(buf, 0, size);
return -EINVAL;
}
const struct bpf_func_proto bpf_get_current_comm_proto = {
.func = bpf_get_current_comm,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE,
};
#if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK)
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
union {
__u32 val;
arch_spinlock_t lock;
} u = { .lock = __ARCH_SPIN_LOCK_UNLOCKED };
compiletime_assert(u.val == 0, "__ARCH_SPIN_LOCK_UNLOCKED not 0");
BUILD_BUG_ON(sizeof(*l) != sizeof(__u32));
BUILD_BUG_ON(sizeof(*lock) != sizeof(__u32));
preempt_disable();
arch_spin_lock(l);
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
arch_spin_unlock(l);
preempt_enable();
}
#else
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
BUILD_BUG_ON(sizeof(*l) != sizeof(*lock));
do {
atomic_cond_read_relaxed(l, !VAL);
} while (atomic_xchg(l, 1));
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
atomic_set_release(l, 0);
}
#endif
static DEFINE_PER_CPU(unsigned long, irqsave_flags);
static inline void __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
{
unsigned long flags;
local_irq_save(flags);
__bpf_spin_lock(lock);
__this_cpu_write(irqsave_flags, flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
{
__bpf_spin_lock_irqsave(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_lock_proto = {
.func = bpf_spin_lock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
{
unsigned long flags;
flags = __this_cpu_read(irqsave_flags);
__bpf_spin_unlock(lock);
local_irq_restore(flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
{
__bpf_spin_unlock_irqrestore(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_unlock_proto = {
.func = bpf_spin_unlock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
bool lock_src)
{
struct bpf_spin_lock *lock;
if (lock_src)
lock = src + map->record->spin_lock_off;
else
lock = dst + map->record->spin_lock_off;
preempt_disable();
__bpf_spin_lock_irqsave(lock);
copy_map_value(map, dst, src);
__bpf_spin_unlock_irqrestore(lock);
preempt_enable();
}
BPF_CALL_0(bpf_jiffies64)
{
return get_jiffies_64();
}
const struct bpf_func_proto bpf_jiffies64_proto = {
.func = bpf_jiffies64,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
#ifdef CONFIG_CGROUPS
BPF_CALL_0(bpf_get_current_cgroup_id)
{
struct cgroup *cgrp;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
cgrp_id = cgroup_id(cgrp);
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_cgroup_id_proto = {
.func = bpf_get_current_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_1(bpf_get_current_ancestor_cgroup_id, int, ancestor_level)
{
struct cgroup *cgrp;
struct cgroup *ancestor;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
ancestor = cgroup_ancestor(cgrp, ancestor_level);
cgrp_id = ancestor ? cgroup_id(ancestor) : 0;
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto = {
.func = bpf_get_current_ancestor_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
#endif /* CONFIG_CGROUPS */
#define BPF_STRTOX_BASE_MASK 0x1F
static int __bpf_strtoull(const char *buf, size_t buf_len, u64 flags,
unsigned long long *res, bool *is_negative)
{
unsigned int base = flags & BPF_STRTOX_BASE_MASK;
const char *cur_buf = buf;
size_t cur_len = buf_len;
unsigned int consumed;
size_t val_len;
char str[64];
if (!buf || !buf_len || !res || !is_negative)
return -EINVAL;
if (base != 0 && base != 8 && base != 10 && base != 16)
return -EINVAL;
if (flags & ~BPF_STRTOX_BASE_MASK)
return -EINVAL;
while (cur_buf < buf + buf_len && isspace(*cur_buf))
++cur_buf;
*is_negative = (cur_buf < buf + buf_len && *cur_buf == '-');
if (*is_negative)
++cur_buf;
consumed = cur_buf - buf;
cur_len -= consumed;
if (!cur_len)
return -EINVAL;
cur_len = min(cur_len, sizeof(str) - 1);
memcpy(str, cur_buf, cur_len);
str[cur_len] = '\0';
cur_buf = str;
cur_buf = _parse_integer_fixup_radix(cur_buf, &base);
val_len = _parse_integer(cur_buf, base, res);
if (val_len & KSTRTOX_OVERFLOW)
return -ERANGE;
if (val_len == 0)
return -EINVAL;
cur_buf += val_len;
consumed += cur_buf - str;
return consumed;
}
static int __bpf_strtoll(const char *buf, size_t buf_len, u64 flags,
long long *res)
{
unsigned long long _res;
bool is_negative;
int err;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative) {
if ((long long)-_res > 0)
return -ERANGE;
*res = -_res;
} else {
if ((long long)_res < 0)
return -ERANGE;
*res = _res;
}
return err;
}
BPF_CALL_4(bpf_strtol, const char *, buf, size_t, buf_len, u64, flags,
s64 *, res)
{
long long _res;
int err;
*res = 0;
err = __bpf_strtoll(buf, buf_len, flags, &_res);
if (err < 0)
return err;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtol_proto = {
.func = bpf_strtol,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(s64),
};
BPF_CALL_4(bpf_strtoul, const char *, buf, size_t, buf_len, u64, flags,
u64 *, res)
{
unsigned long long _res;
bool is_negative;
int err;
*res = 0;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative)
return -EINVAL;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtoul_proto = {
.func = bpf_strtoul,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(u64),
};
BPF_CALL_3(bpf_strncmp, const char *, s1, u32, s1_sz, const char *, s2)
{
return strncmp(s1, s2, s1_sz);
}
static const struct bpf_func_proto bpf_strncmp_proto = {
.func = bpf_strncmp,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_CONST_STR,
};
BPF_CALL_4(bpf_get_ns_current_pid_tgid, u64, dev, u64, ino,
struct bpf_pidns_info *, nsdata, u32, size)
{
struct task_struct *task = current;
struct pid_namespace *pidns;
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_pidns_info)))
goto clear;
if (unlikely((u64)(dev_t)dev != dev))
goto clear;
if (unlikely(!task))
goto clear;
pidns = task_active_pid_ns(task);
if (unlikely(!pidns)) {
err = -ENOENT;
goto clear;
}
if (!ns_match(&pidns->ns, (dev_t)dev, ino))
goto clear;
nsdata->pid = task_pid_nr_ns(task, pidns);
nsdata->tgid = task_tgid_nr_ns(task, pidns);
return 0;
clear:
memset((void *)nsdata, 0, (size_t) size);
return err;
}
const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = {
.func = bpf_get_ns_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
};
static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
.func = bpf_get_raw_cpu_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
}
const struct bpf_func_proto bpf_event_output_data_proto = {
.func = bpf_event_output_data,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
const void __user *, user_ptr)
{
int ret = copy_from_user(dst, user_ptr, size);
if (unlikely(ret)) {
memset(dst, 0, size);
ret = -EFAULT;
}
return ret;
}
const struct bpf_func_proto bpf_copy_from_user_proto = {
.func = bpf_copy_from_user,
.gpl_only = false,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_5(bpf_copy_from_user_task, void *, dst, u32, size,
const void __user *, user_ptr, struct task_struct *, tsk, u64, flags)
{
int ret;
/* flags is not used yet */
if (unlikely(flags))
return -EINVAL;
if (unlikely(!size))
return 0;
ret = access_process_vm(tsk, (unsigned long)user_ptr, dst, size, 0);
if (ret == size)
return 0;
memset(dst, 0, size);
/* Return -EFAULT for partial read */
return ret < 0 ? ret : -EFAULT;
}
const struct bpf_func_proto bpf_copy_from_user_task_proto = {
.func = bpf_copy_from_user_task,
.gpl_only = true,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_BTF_ID,
.arg4_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.arg5_type = ARG_ANYTHING
};
BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
{
if (cpu >= nr_cpu_ids)
return (unsigned long)NULL;
return (unsigned long)per_cpu_ptr((const void __percpu *)(const uintptr_t)ptr, cpu);
}
const struct bpf_func_proto bpf_per_cpu_ptr_proto = {
.func = bpf_per_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | PTR_MAYBE_NULL | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
.arg2_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_this_cpu_ptr, const void *, percpu_ptr)
{
return (unsigned long)this_cpu_ptr((const void __percpu *)(const uintptr_t)percpu_ptr);
}
const struct bpf_func_proto bpf_this_cpu_ptr_proto = {
.func = bpf_this_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
};
static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,
size_t bufsz)
{
void __user *user_ptr = (__force void __user *)unsafe_ptr;
buf[0] = 0;
switch (fmt_ptype) {
case 's':
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
if ((unsigned long)unsafe_ptr < TASK_SIZE)
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
fallthrough;
#endif
case 'k':
return strncpy_from_kernel_nofault(buf, unsafe_ptr, bufsz);
case 'u':
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
}
return -EINVAL;
}
/* Support executing three nested bprintf helper calls on a given CPU */
#define MAX_BPRINTF_NEST_LEVEL 3
static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs);
static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
{
int nest_level;
preempt_disable();
nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
return -EBUSY;
}
*bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
return 0;
}
void bpf_put_buffers(void)
{
if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
return;
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
}
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
{
if (!data->bin_args && !data->buf)
return;
bpf_put_buffers();
}
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)
{
if (map->map_type == BPF_MAP_TYPE_ARRAY) {
struct bpf_array *array = container_of(map, struct bpf_array, map);
*arr_idx = ((char *)value - array->value) / array->elem_size;
return arr_idx;
}
return (void *)value - round_up(map->key_size, 8);
}
enum bpf_async_type {
BPF_ASYNC_TYPE_TIMER = 0,
BPF_ASYNC_TYPE_WQ,
};
enum bpf_async_op {
BPF_ASYNC_START,
BPF_ASYNC_CANCEL
};
struct bpf_async_cmd {
struct llist_node node;
u64 nsec;
u32 mode;
enum bpf_async_op op;
};
struct bpf_async_cb {
struct bpf_map *map;
struct bpf_prog *prog;
void __rcu *callback_fn;
void *value;
struct rcu_head rcu;
u64 flags;
struct irq_work worker;
refcount_t refcnt;
enum bpf_async_type type;
struct llist_head async_cmds;
};
/* BPF map elements can contain 'struct bpf_timer'.
* Such map owns all of its BPF timers.
* 'struct bpf_timer' is allocated as part of map element allocation
* and it's zero initialized.
* That space is used to keep 'struct bpf_async_kern'.
* bpf_timer_init() allocates 'struct bpf_hrtimer', inits hrtimer, and
* remembers 'struct bpf_map *' pointer it's part of.
* bpf_timer_set_callback() increments prog refcnt and assign bpf callback_fn.
* bpf_timer_start() arms the timer.
* If user space reference to a map goes to zero at this point
* ops->map_release_uref callback is responsible for cancelling the timers,
* freeing their memory, and decrementing prog's refcnts.
* bpf_timer_cancel() cancels the timer and decrements prog's refcnt.
* Inner maps can contain bpf timers as well. ops->map_release_uref is
* freeing the timers when inner map is replaced or deleted by user space.
*/
struct bpf_hrtimer {
struct bpf_async_cb cb;
struct hrtimer timer;
atomic_t cancelling;
};
struct bpf_work {
struct bpf_async_cb cb;
struct work_struct work;
};
/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */
struct bpf_async_kern {
union {
struct bpf_async_cb *cb;
struct bpf_hrtimer *timer;
struct bpf_work *work;
};
} __attribute__((aligned(8)));
static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);
static void bpf_async_refcount_put(struct bpf_async_cb *cb);
static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer)
{
struct bpf_hrtimer *t = container_of(hrtimer, struct bpf_hrtimer, timer);
struct bpf_map *map = t->cb.map;
void *value = t->cb.value;
bpf_callback_t callback_fn;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_timer);
callback_fn = rcu_dereference_check(t->cb.callback_fn, rcu_read_lock_bh_held());
if (!callback_fn)
goto out;
/* bpf_timer_cb() runs in hrtimer_run_softirq. It doesn't migrate and
* cannot be preempted by another bpf_timer_cb() on the same cpu.
* Remember the timer this callback is servicing to prevent
* deadlock if callback_fn() calls bpf_timer_cancel() or
* bpf_map_delete_elem() on the same timer.
*/
this_cpu_write(hrtimer_running, t);
key = map_key_from_value(map, value, &idx);
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
/* The verifier checked that return value is zero. */
this_cpu_write(hrtimer_running, NULL);
out:
return HRTIMER_NORESTART;
}
static void bpf_wq_work(struct work_struct *work)
{
struct bpf_work *w = container_of(work, struct bpf_work, work);
struct bpf_async_cb *cb = &w->cb;
struct bpf_map *map = cb->map;
bpf_callback_t callback_fn;
void *value = cb->value;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_wq);
callback_fn = READ_ONCE(cb->callback_fn);
if (!callback_fn)
return;
key = map_key_from_value(map, value, &idx);
rcu_read_lock_trace();
migrate_disable();
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
migrate_enable();
rcu_read_unlock_trace();
}
static void bpf_async_cb_rcu_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
/*
* Drop the last reference to prog only after RCU GP, as set_callback()
* may race with cancel_and_free()
*/
if (cb->prog)
bpf_prog_put(cb->prog);
kfree_nolock(cb);
}
/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */
static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
bool retry = false;
/*
* bpf_async_cancel_and_free() tried to cancel timer/wq, but it
* could have raced with timer/wq_start. Now refcnt is zero and
* srcu/rcu GP completed. Cancel timer/wq again.
*/
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER:
if (hrtimer_try_to_cancel(&t->timer) < 0)
retry = true;
break;
case BPF_ASYNC_TYPE_WQ:
if (!cancel_work(&w->work) && work_busy(&w->work))
retry = true;
break;
}
if (retry) {
/*
* hrtimer or wq callback may still be running. It must be
* in rcu_tasks_trace or rcu CS, so wait for GP again.
* It won't retry forever, since refcnt zero prevents all
* operations on timer/wq.
*/
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
return;
}
/* RCU Tasks Trace grace period implies RCU grace period. */
bpf_async_cb_rcu_free(rcu);
}
static void worker_for_call_rcu(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
static void bpf_async_refcount_put(struct bpf_async_cb *cb)
{
if (!refcount_dec_and_test(&cb->refcnt))
return;
if (irqs_disabled()) {
cb->worker = IRQ_WORK_INIT(worker_for_call_rcu);
irq_work_queue(&cb->worker);
} else {
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async);
static void bpf_async_irq_worker(struct irq_work *work);
static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags,
enum bpf_async_type type)
{
struct bpf_async_cb *cb, *old_cb;
struct bpf_hrtimer *t;
struct bpf_work *w;
clockid_t clockid;
size_t size;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
size = sizeof(struct bpf_hrtimer);
break;
case BPF_ASYNC_TYPE_WQ:
size = sizeof(struct bpf_work);
break;
default:
return -EINVAL;
}
old_cb = READ_ONCE(async->cb);
if (old_cb)
return -EBUSY;
cb = bpf_map_kmalloc_nolock(map, size, 0, map->numa_node);
if (!cb)
return -ENOMEM;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
clockid = flags & (MAX_CLOCKS - 1);
t = (struct bpf_hrtimer *)cb;
atomic_set(&t->cancelling, 0);
hrtimer_setup(&t->timer, bpf_timer_cb, clockid, HRTIMER_MODE_REL_SOFT);
cb->value = (void *)async - map->record->timer_off;
break;
case BPF_ASYNC_TYPE_WQ:
w = (struct bpf_work *)cb;
INIT_WORK(&w->work, bpf_wq_work);
cb->value = (void *)async - map->record->wq_off;
break;
}
cb->map = map;
cb->prog = NULL;
cb->flags = flags;
cb->worker = IRQ_WORK_INIT(bpf_async_irq_worker);
init_llist_head(&cb->async_cmds);
refcount_set(&cb->refcnt, 1); /* map's reference */
cb->type = type;
rcu_assign_pointer(cb->callback_fn, NULL);
old_cb = cmpxchg(&async->cb, NULL, cb);
if (old_cb) {
/* Lost the race to initialize this bpf_async_kern, drop the allocated object */
kfree_nolock(cb);
return -EBUSY;
}
/* Guarantee the order between async->cb and map->usercnt. So
* when there are concurrent uref release and bpf timer init, either
* bpf_timer_cancel_and_free() called by uref release reads a no-NULL
* timer or atomic64_read() below returns a zero usercnt.
*/
smp_mb();
if (!atomic64_read(&map->usercnt)) {
/* maps with timers must be either held by user space
* or pinned in bpffs.
*/
bpf_async_cancel_and_free(async);
return -EPERM;
}
return 0;
}
BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
u64, flags)
{
clock_t clockid = flags & (MAX_CLOCKS - 1);
BUILD_BUG_ON(MAX_CLOCKS != 16);
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer));
if (flags >= MAX_CLOCKS ||
/* similar to timerfd except _ALARM variants are not supported */
(clockid != CLOCK_MONOTONIC &&
clockid != CLOCK_REALTIME &&
clockid != CLOCK_BOOTTIME))
return -EINVAL;
return __bpf_async_init(timer, map, flags, BPF_ASYNC_TYPE_TIMER);
}
static const struct bpf_func_proto bpf_timer_init_proto = {
.func = bpf_timer_init,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
static int bpf_async_update_prog_callback(struct bpf_async_cb *cb,
struct bpf_prog *prog,
void *callback_fn)
{
struct bpf_prog *prev;
/* Acquire a guard reference on prog to prevent it from being freed during the loop */
if (prog) {
prog = bpf_prog_inc_not_zero(prog);
if (IS_ERR(prog))
return PTR_ERR(prog);
}
do {
if (prog)
prog = bpf_prog_inc_not_zero(prog);
prev = xchg(&cb->prog, prog);
rcu_assign_pointer(cb->callback_fn, callback_fn);
/*
* Release previous prog, make sure that if other CPU is contending,
* to set bpf_prog, references are not leaked as each iteration acquires and
* releases one reference.
*/
if (prev)
bpf_prog_put(prev);
} while (READ_ONCE(cb->prog) != prog ||
(void __force *)READ_ONCE(cb->callback_fn) != callback_fn);
if (prog)
bpf_prog_put(prog);
return 0;
}
static DEFINE_PER_CPU(struct bpf_async_cb *, async_cb_running);
static int bpf_async_schedule_op(struct bpf_async_cb *cb, enum bpf_async_op op,
u64 nsec, u32 timer_mode)
{
/*
* Do not schedule another operation on this cpu if it's in irq_work
* callback that is processing async_cmds queue. Otherwise the following
* loop is possible:
* bpf_timer_start() -> bpf_async_schedule_op() -> irq_work_queue().
* irqrestore -> bpf_async_irq_worker() -> tracepoint -> bpf_timer_start().
*/
if (this_cpu_read(async_cb_running) == cb) {
bpf_async_refcount_put(cb);
return -EDEADLK;
}
struct bpf_async_cmd *cmd = kmalloc_nolock(sizeof(*cmd), 0, NUMA_NO_NODE);
if (!cmd) {
bpf_async_refcount_put(cb);
return -ENOMEM;
}
init_llist_node(&cmd->node);
cmd->nsec = nsec;
cmd->mode = timer_mode;
cmd->op = op;
if (llist_add(&cmd->node, &cb->async_cmds))
irq_work_queue(&cb->worker);
return 0;
}
static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback_fn,
struct bpf_prog *prog)
{
struct bpf_async_cb *cb;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
return bpf_async_update_prog_callback(cb, prog, callback_fn);
}
BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
struct bpf_prog_aux *, aux)
{
return __bpf_async_set_callback(timer, callback_fn, aux->prog);
}
static const struct bpf_func_proto bpf_timer_set_callback_proto = {
.func = bpf_timer_set_callback,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_PTR_TO_FUNC,
};
static bool defer_timer_wq_op(void)
{
return in_hardirq() || irqs_disabled();
}
BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags)
{
struct bpf_hrtimer *t;
u32 mode;
if (flags & ~(BPF_F_TIMER_ABS | BPF_F_TIMER_CPU_PIN))
return -EINVAL;
t = READ_ONCE(async->timer);
if (!t || !READ_ONCE(t->cb.prog))
return -EINVAL;
if (flags & BPF_F_TIMER_ABS)
mode = HRTIMER_MODE_ABS_SOFT;
else
mode = HRTIMER_MODE_REL_SOFT;
if (flags & BPF_F_TIMER_CPU_PIN)
mode |= HRTIMER_MODE_PINNED;
/*
* bpf_async_cancel_and_free() could have dropped refcnt to zero. In
* such case BPF progs are not allowed to arm the timer to prevent UAF.
*/
if (!refcount_inc_not_zero(&t->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
hrtimer_start(&t->timer, ns_to_ktime(nsecs), mode);
bpf_async_refcount_put(&t->cb);
return 0;
} else {
return bpf_async_schedule_op(&t->cb, BPF_ASYNC_START, nsecs, mode);
}
}
static const struct bpf_func_proto bpf_timer_start_proto = {
.func = bpf_timer_start,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async)
{
struct bpf_hrtimer *t, *cur_t;
bool inc = false;
int ret = 0;
if (defer_timer_wq_op())
return -EOPNOTSUPP;
t = READ_ONCE(async->timer);
if (!t)
return -EINVAL;
cur_t = this_cpu_read(hrtimer_running);
if (cur_t == t) {
/* If bpf callback_fn is trying to bpf_timer_cancel()
* its own timer the hrtimer_cancel() will deadlock
* since it waits for callback_fn to finish.
*/
return -EDEADLK;
}
/* Only account in-flight cancellations when invoked from a timer
* callback, since we want to avoid waiting only if other _callbacks_
* are waiting on us, to avoid introducing lockups. Non-callback paths
* are ok, since nobody would synchronously wait for their completion.
*/
if (!cur_t)
goto drop;
atomic_inc(&t->cancelling);
/* Need full barrier after relaxed atomic_inc */
smp_mb__after_atomic();
inc = true;
if (atomic_read(&cur_t->cancelling)) {
/* We're cancelling timer t, while some other timer callback is
* attempting to cancel us. In such a case, it might be possible
* that timer t belongs to the other callback, or some other
* callback waiting upon it (creating transitive dependencies
* upon us), and we will enter a deadlock if we continue
* cancelling and waiting for it synchronously, since it might
* do the same. Bail!
*/
atomic_dec(&t->cancelling);
return -EDEADLK;
}
drop:
bpf_async_update_prog_callback(&t->cb, NULL, NULL);
/* Cancel the timer and wait for associated callback to finish
* if it was running.
*/
ret = hrtimer_cancel(&t->timer);
if (inc)
atomic_dec(&t->cancelling);
return ret;
}
static const struct bpf_func_proto bpf_timer_cancel_proto = {
.func = bpf_timer_cancel,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
};
static void bpf_async_process_op(struct bpf_async_cb *cb, u32 op,
u64 timer_nsec, u32 timer_mode)
{
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER: {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
switch (op) {
case BPF_ASYNC_START:
hrtimer_start(&t->timer, ns_to_ktime(timer_nsec), timer_mode);
break;
case BPF_ASYNC_CANCEL:
hrtimer_try_to_cancel(&t->timer);
break;
}
break;
}
case BPF_ASYNC_TYPE_WQ: {
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
switch (op) {
case BPF_ASYNC_START:
schedule_work(&w->work);
break;
case BPF_ASYNC_CANCEL:
cancel_work(&w->work);
break;
}
break;
}
}
bpf_async_refcount_put(cb);
}
static void bpf_async_irq_worker(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
struct llist_node *pos, *n, *list;
list = llist_del_all(&cb->async_cmds);
if (!list)
return;
list = llist_reverse_order(list);
this_cpu_write(async_cb_running, cb);
llist_for_each_safe(pos, n, list) {
struct bpf_async_cmd *cmd;
cmd = container_of(pos, struct bpf_async_cmd, node);
bpf_async_process_op(cb, cmd->op, cmd->nsec, cmd->mode);
kfree_nolock(cmd);
}
this_cpu_write(async_cb_running, NULL);
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async)
{
struct bpf_async_cb *cb;
if (!READ_ONCE(async->cb))
return;
cb = xchg(&async->cb, NULL);
if (!cb)
return;
bpf_async_update_prog_callback(cb, NULL, NULL);
/*
* No refcount_inc_not_zero(&cb->refcnt) here. Dropping the last
* refcnt. Either synchronously or asynchronously in irq_work.
*/
if (!defer_timer_wq_op()) {
bpf_async_process_op(cb, BPF_ASYNC_CANCEL, 0, 0);
} else {
(void)bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
/*
* bpf_async_schedule_op() either enqueues allocated cmd into llist
* or fails with ENOMEM and drop the last refcnt.
* This is unlikely, but safe, since bpf_async_cb_rcu_tasks_trace_free()
* callback will do additional timer/wq_cancel due to races anyway.
*/
}
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_timer_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_wq_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
{
unsigned long *kptr = dst;
/* This helper may be inlined by verifier. */
return xchg(kptr, (unsigned long)ptr);
}
/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg()
* helper is determined dynamically by the verifier. Use BPF_PTR_POISON to
* denote type that verifier will determine.
*/
static const struct bpf_func_proto bpf_kptr_xchg_proto = {
.func = bpf_kptr_xchg,
.gpl_only = false,
.ret_type = RET_PTR_TO_BTF_ID_OR_NULL,
.ret_btf_id = BPF_PTR_POISON,
.arg1_type = ARG_KPTR_XCHG_DEST,
.arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL | OBJ_RELEASE,
.arg2_btf_id = BPF_PTR_POISON,
};
struct bpf_dynptr_file_impl {
struct freader freader;
/* 64 bit offset and size overriding 32 bit ones in bpf_dynptr_kern */
u64 offset;
u64 size;
};
/* Since the upper 8 bits of dynptr->size is reserved, the
* maximum supported size is 2^24 - 1.
*/
#define DYNPTR_MAX_SIZE ((1UL << 24) - 1)
#define DYNPTR_TYPE_SHIFT 28
#define DYNPTR_SIZE_MASK 0xFFFFFF
#define DYNPTR_RDONLY_BIT BIT(31)
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr)
{
return ptr->size & DYNPTR_RDONLY_BIT;
}
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
{
ptr->size |= DYNPTR_RDONLY_BIT;
}
static void bpf_dynptr_set_type(struct bpf_dynptr_kern *ptr, enum bpf_dynptr_type type)
{
ptr->size |= type << DYNPTR_TYPE_SHIFT;
}
static enum bpf_dynptr_type bpf_dynptr_get_type(const struct bpf_dynptr_kern *ptr)
{
return (ptr->size & ~(DYNPTR_RDONLY_BIT)) >> DYNPTR_TYPE_SHIFT;
}
u64 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
return df->size;
}
return ptr->size & DYNPTR_SIZE_MASK;
}
static void bpf_dynptr_advance_offset(struct bpf_dynptr_kern *ptr, u64 off)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->offset += off;
return;
}
ptr->offset += off;
}
static void bpf_dynptr_set_size(struct bpf_dynptr_kern *ptr, u64 new_size)
{
u32 metadata = ptr->size & ~DYNPTR_SIZE_MASK;
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->size = new_size;
return;
}
ptr->size = (u32)new_size | metadata;
}
int bpf_dynptr_check_size(u64 size)
{
return size > DYNPTR_MAX_SIZE ? -E2BIG : 0;
}
static int bpf_file_fetch_bytes(struct bpf_dynptr_file_impl *df, u64 offset, void *buf, u64 len)
{
const void *ptr;
if (!buf)
return -EINVAL;
df->freader.buf = buf;
df->freader.buf_sz = len;
ptr = freader_fetch(&df->freader, offset + df->offset, len);
if (!ptr)
return df->freader.err;
if (ptr != buf) /* Force copying into the buffer */
memcpy(buf, ptr, len);
return 0;
}
void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size)
{
ptr->data = data;
ptr->offset = offset;
ptr->size = size;
bpf_dynptr_set_type(ptr, type);
}
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)
{
memset(ptr, 0, sizeof(*ptr));
}
BPF_CALL_4(bpf_dynptr_from_mem, void *, data, u64, size, u64, flags, struct bpf_dynptr_kern *, ptr)
{
int err;
BTF_TYPE_EMIT(struct bpf_dynptr);
err = bpf_dynptr_check_size(size);
if (err)
goto error;
/* flags is currently unsupported */
if (flags) {
err = -EINVAL;
goto error;
}
bpf_dynptr_init(ptr, data, BPF_DYNPTR_TYPE_LOCAL, 0, size);
return 0;
error:
bpf_dynptr_set_null(ptr);
return err;
}
static const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
.func = bpf_dynptr_from_mem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL | MEM_UNINIT | MEM_WRITE,
};
static int __bpf_dynptr_read(void *dst, u64 len, const struct bpf_dynptr_kern *src,
u64 offset, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!src->data || flags)
return -EINVAL;
err = bpf_dynptr_check_off_len(src, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(src);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst, src->data + src->offset + offset, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_XDP:
return __bpf_xdp_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_SKB_META:
memmove(dst, bpf_skb_meta_pointer(src->data, src->offset + offset), len);
return 0;
case BPF_DYNPTR_TYPE_FILE:
return bpf_file_fetch_bytes(src->data, offset, dst, len);
default:
WARN_ONCE(true, "bpf_dynptr_read: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_read, void *, dst, u64, len, const struct bpf_dynptr_kern *, src,
u64, offset, u64, flags)
{
return __bpf_dynptr_read(dst, len, src, offset, flags);
}
static const struct bpf_func_proto bpf_dynptr_read_proto = {
.func = bpf_dynptr_read,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
int __bpf_dynptr_write(const struct bpf_dynptr_kern *dst, u64 offset, void *src,
u64 len, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!dst->data || __bpf_dynptr_is_rdonly(dst))
return -EINVAL;
err = bpf_dynptr_check_off_len(dst, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(dst);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
if (flags)
return -EINVAL;
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst->data + dst->offset + offset, src, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_store_bytes(dst->data, dst->offset + offset, src, len,
flags);
case BPF_DYNPTR_TYPE_XDP:
if (flags)
return -EINVAL;
return __bpf_xdp_store_bytes(dst->data, dst->offset + offset, src, len);
case BPF_DYNPTR_TYPE_SKB_META:
return __bpf_skb_meta_store_bytes(dst->data, dst->offset + offset, src,
len, flags);
default:
WARN_ONCE(true, "bpf_dynptr_write: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_write, const struct bpf_dynptr_kern *, dst, u64, offset, void *, src,
u64, len, u64, flags)
{
return __bpf_dynptr_write(dst, offset, src, len, flags);
}
static const struct bpf_func_proto bpf_dynptr_write_proto = {
.func = bpf_dynptr_write,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_dynptr_data, const struct bpf_dynptr_kern *, ptr, u64, offset, u64, len)
{
enum bpf_dynptr_type type;
int err;
if (!ptr->data)
return 0;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return 0;
if (__bpf_dynptr_is_rdonly(ptr))
return 0;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return (unsigned long)(ptr->data + ptr->offset + offset);
case BPF_DYNPTR_TYPE_SKB:
case BPF_DYNPTR_TYPE_XDP:
case BPF_DYNPTR_TYPE_SKB_META:
/* skb and xdp dynptrs should use bpf_dynptr_slice / bpf_dynptr_slice_rdwr */
return 0;
default:
WARN_ONCE(true, "bpf_dynptr_data: unknown dynptr type %d\n", type);
return 0;
}
}
static const struct bpf_func_proto bpf_dynptr_data_proto = {
.func = bpf_dynptr_data,
.gpl_only = false,
.ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
};
const struct bpf_func_proto bpf_get_current_task_proto __weak;
const struct bpf_func_proto bpf_get_current_task_btf_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_str_proto __weak;
const struct bpf_func_proto bpf_task_pt_regs_proto __weak;
const struct bpf_func_proto bpf_perf_event_read_proto __weak;
const struct bpf_func_proto bpf_send_signal_proto __weak;
const struct bpf_func_proto bpf_send_signal_thread_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_sleepable_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_proto __weak;
const struct bpf_func_proto bpf_get_branch_snapshot_proto __weak;
const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_map_lookup_elem:
return &bpf_map_lookup_elem_proto;
case BPF_FUNC_map_update_elem:
return &bpf_map_update_elem_proto;
case BPF_FUNC_map_delete_elem:
return &bpf_map_delete_elem_proto;
case BPF_FUNC_map_push_elem:
return &bpf_map_push_elem_proto;
case BPF_FUNC_map_pop_elem:
return &bpf_map_pop_elem_proto;
case BPF_FUNC_map_peek_elem:
return &bpf_map_peek_elem_proto;
case BPF_FUNC_map_lookup_percpu_elem:
return &bpf_map_lookup_percpu_elem_proto;
case BPF_FUNC_get_prandom_u32:
return &bpf_get_prandom_u32_proto;
case BPF_FUNC_get_smp_processor_id:
return &bpf_get_raw_smp_processor_id_proto;
case BPF_FUNC_get_numa_node_id:
return &bpf_get_numa_node_id_proto;
case BPF_FUNC_tail_call:
return &bpf_tail_call_proto;
case BPF_FUNC_ktime_get_ns:
return &bpf_ktime_get_ns_proto;
case BPF_FUNC_ktime_get_boot_ns:
return &bpf_ktime_get_boot_ns_proto;
case BPF_FUNC_ktime_get_tai_ns:
return &bpf_ktime_get_tai_ns_proto;
case BPF_FUNC_ringbuf_output:
return &bpf_ringbuf_output_proto;
case BPF_FUNC_ringbuf_reserve:
return &bpf_ringbuf_reserve_proto;
case BPF_FUNC_ringbuf_submit:
return &bpf_ringbuf_submit_proto;
case BPF_FUNC_ringbuf_discard:
return &bpf_ringbuf_discard_proto;
case BPF_FUNC_ringbuf_query:
return &bpf_ringbuf_query_proto;
case BPF_FUNC_strncmp:
return &bpf_strncmp_proto;
case BPF_FUNC_strtol:
return &bpf_strtol_proto;
case BPF_FUNC_strtoul:
return &bpf_strtoul_proto;
case BPF_FUNC_get_current_pid_tgid:
return &bpf_get_current_pid_tgid_proto;
case BPF_FUNC_get_ns_current_pid_tgid:
return &bpf_get_ns_current_pid_tgid_proto;
case BPF_FUNC_get_current_uid_gid:
return &bpf_get_current_uid_gid_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_BPF))
return NULL;
switch (func_id) {
case BPF_FUNC_spin_lock:
return &bpf_spin_lock_proto;
case BPF_FUNC_spin_unlock:
return &bpf_spin_unlock_proto;
case BPF_FUNC_jiffies64:
return &bpf_jiffies64_proto;
case BPF_FUNC_per_cpu_ptr:
return &bpf_per_cpu_ptr_proto;
case BPF_FUNC_this_cpu_ptr:
return &bpf_this_cpu_ptr_proto;
case BPF_FUNC_timer_init:
return &bpf_timer_init_proto;
case BPF_FUNC_timer_set_callback:
return &bpf_timer_set_callback_proto;
case BPF_FUNC_timer_start:
return &bpf_timer_start_proto;
case BPF_FUNC_timer_cancel:
return &bpf_timer_cancel_proto;
case BPF_FUNC_kptr_xchg:
return &bpf_kptr_xchg_proto;
case BPF_FUNC_for_each_map_elem:
return &bpf_for_each_map_elem_proto;
case BPF_FUNC_loop:
return &bpf_loop_proto;
case BPF_FUNC_user_ringbuf_drain:
return &bpf_user_ringbuf_drain_proto;
case BPF_FUNC_ringbuf_reserve_dynptr:
return &bpf_ringbuf_reserve_dynptr_proto;
case BPF_FUNC_ringbuf_submit_dynptr:
return &bpf_ringbuf_submit_dynptr_proto;
case BPF_FUNC_ringbuf_discard_dynptr:
return &bpf_ringbuf_discard_dynptr_proto;
case BPF_FUNC_dynptr_from_mem:
return &bpf_dynptr_from_mem_proto;
case BPF_FUNC_dynptr_read:
return &bpf_dynptr_read_proto;
case BPF_FUNC_dynptr_write:
return &bpf_dynptr_write_proto;
case BPF_FUNC_dynptr_data:
return &bpf_dynptr_data_proto;
#ifdef CONFIG_CGROUPS
case BPF_FUNC_cgrp_storage_get:
return &bpf_cgrp_storage_get_proto;
case BPF_FUNC_cgrp_storage_delete:
return &bpf_cgrp_storage_delete_proto;
case BPF_FUNC_get_current_cgroup_id:
return &bpf_get_current_cgroup_id_proto;
case BPF_FUNC_get_current_ancestor_cgroup_id:
return &bpf_get_current_ancestor_cgroup_id_proto;
case BPF_FUNC_current_task_under_cgroup:
return &bpf_current_task_under_cgroup_proto;
#endif
#ifdef CONFIG_CGROUP_NET_CLASSID
case BPF_FUNC_get_cgroup_classid:
return &bpf_get_cgroup_classid_curr_proto;
#endif
case BPF_FUNC_task_storage_get:
return &bpf_task_storage_get_proto;
case BPF_FUNC_task_storage_delete:
return &bpf_task_storage_delete_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_PERFMON))
return NULL;
switch (func_id) {
case BPF_FUNC_trace_printk:
return bpf_get_trace_printk_proto();
case BPF_FUNC_get_current_task:
return &bpf_get_current_task_proto;
case BPF_FUNC_get_current_task_btf:
return &bpf_get_current_task_btf_proto;
case BPF_FUNC_get_current_comm:
return &bpf_get_current_comm_proto;
case BPF_FUNC_probe_read_user:
return &bpf_probe_read_user_proto;
case BPF_FUNC_probe_read_kernel:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_proto;
case BPF_FUNC_probe_read_user_str:
return &bpf_probe_read_user_str_proto;
case BPF_FUNC_probe_read_kernel_str:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_str_proto;
case BPF_FUNC_copy_from_user:
return &bpf_copy_from_user_proto;
case BPF_FUNC_copy_from_user_task:
return &bpf_copy_from_user_task_proto;
case BPF_FUNC_snprintf_btf:
return &bpf_snprintf_btf_proto;
case BPF_FUNC_snprintf:
return &bpf_snprintf_proto;
case BPF_FUNC_task_pt_regs:
return &bpf_task_pt_regs_proto;
case BPF_FUNC_trace_vprintk:
return bpf_get_trace_vprintk_proto();
case BPF_FUNC_perf_event_read_value:
return bpf_get_perf_event_read_value_proto();
case BPF_FUNC_perf_event_read:
return &bpf_perf_event_read_proto;
case BPF_FUNC_send_signal:
return &bpf_send_signal_proto;
case BPF_FUNC_send_signal_thread:
return &bpf_send_signal_thread_proto;
case BPF_FUNC_get_task_stack:
return prog->sleepable ? &bpf_get_task_stack_sleepable_proto
: &bpf_get_task_stack_proto;
case BPF_FUNC_get_branch_snapshot:
return &bpf_get_branch_snapshot_proto;
case BPF_FUNC_find_vma:
return &bpf_find_vma_proto;
default:
return NULL;
}
}
EXPORT_SYMBOL_GPL(bpf_base_func_proto);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock)
{
struct list_head *head = list_head, *orig_head = list_head;
BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
/* Do the actual list draining outside the lock to not hold the lock for
* too long, and also prevent deadlocks if tracing programs end up
* executing on entry/exit of functions called inside the critical
* section, and end up doing map ops that call bpf_list_head_free for
* the same map value again.
*/
__bpf_spin_lock_irqsave(spin_lock);
if (!head->next || list_empty(head))
goto unlock;
head = head->next;
unlock:
INIT_LIST_HEAD(orig_head);
__bpf_spin_unlock_irqrestore(spin_lock);
while (head != orig_head) {
void *obj = head;
obj -= field->graph_root.node_offset;
head = head->next;
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
/* Like rbtree_postorder_for_each_entry_safe, but 'pos' and 'n' are
* 'rb_node *', so field name of rb_node within containing struct is not
* needed.
*
* Since bpf_rb_tree's node type has a corresponding struct btf_field with
* graph_root.node_offset, it's not necessary to know field name
* or type of node struct
*/
#define bpf_rbtree_postorder_for_each_entry_safe(pos, n, root) \
for (pos = rb_first_postorder(root); \
pos && ({ n = rb_next_postorder(pos); 1; }); \
pos = n)
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock)
{
struct rb_root_cached orig_root, *root = rb_root;
struct rb_node *pos, *n;
void *obj;
BUILD_BUG_ON(sizeof(struct rb_root_cached) > sizeof(struct bpf_rb_root));
BUILD_BUG_ON(__alignof__(struct rb_root_cached) > __alignof__(struct bpf_rb_root));
__bpf_spin_lock_irqsave(spin_lock);
orig_root = *root;
*root = RB_ROOT_CACHED;
__bpf_spin_unlock_irqrestore(spin_lock);
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
obj = pos;
obj -= field->graph_root.node_offset;
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
__bpf_kfunc_start_defs();
/**
* bpf_obj_new() - allocate an object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate an object of the type identified by @local_type_id__k and
* initialize its special fields. BPF programs can use
* bpf_core_type_id_local() to provide @local_type_id__k. The verifier
* rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
void *p;
p = bpf_mem_alloc(&bpf_global_ma, size);
if (!p)
return NULL;
if (meta)
bpf_obj_init(meta->record, p);
return p;
}
__bpf_kfunc void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_obj_new(local_type_id__k, meta__ign);
}
/**
* bpf_percpu_obj_new() - allocate a percpu object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate a percpu object of the type identified by @local_type_id__k. BPF
* programs can use bpf_core_type_id_local() to provide @local_type_id__k.
* The verifier rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated percpu object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_percpu_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
/* The verifier has ensured that meta must be NULL */
return bpf_mem_alloc(&bpf_global_percpu_ma, size);
}
__bpf_kfunc void *bpf_percpu_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_percpu_obj_new(local_type_id__k, meta__ign);
}
/* Must be called under migrate_disable(), as required by bpf_mem_free */
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu)
{
struct bpf_mem_alloc *ma;
if (rec && rec->refcount_off >= 0 &&
!refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) {
/* Object is refcounted and refcount_dec didn't result in 0
* refcount. Return without freeing the object
*/
return;
}
if (rec)
bpf_obj_free_fields(rec, p);
if (percpu)
ma = &bpf_global_percpu_ma;
else
ma = &bpf_global_ma;
bpf_mem_free_rcu(ma, p);
}
/**
* bpf_obj_drop() - drop a previously allocated object
* @p__alloc: object to free
* @meta: verifier-supplied struct metadata
*
* Destroy special fields in @p__alloc as needed and free the object. The
* verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
void *p = p__alloc;
__bpf_obj_drop_impl(p, meta ? meta->record : NULL, false);
}
__bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
{
return bpf_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_percpu_obj_drop() - drop a previously allocated percpu object
* @p__alloc: percpu object to free
* @meta: verifier-supplied struct metadata
*
* Free @p__alloc. The verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_percpu_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
/* The verifier has ensured that meta must be NULL */
bpf_mem_free_rcu(&bpf_global_percpu_ma, p__alloc);
}
__bpf_kfunc void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign)
{
bpf_percpu_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_refcount_acquire() - turn a local kptr into an owning reference
* @p__refcounted_kptr: non-owning local kptr
* @meta: verifier-supplied struct metadata
*
* Increment the refcount for @p__refcounted_kptr. The verifier rewrites
* @meta; BPF programs do not set it.
*
* Return: Owning reference to @p__refcounted_kptr, or %NULL on failure.
*/
__bpf_kfunc void *bpf_refcount_acquire(void *p__refcounted_kptr, struct btf_struct_meta *meta)
{
struct bpf_refcount *ref;
/* Could just cast directly to refcount_t *, but need some code using
* bpf_refcount type so that it is emitted in vmlinux BTF
*/
ref = (struct bpf_refcount *)(p__refcounted_kptr + meta->record->refcount_off);
if (!refcount_inc_not_zero((refcount_t *)ref))
return NULL;
/* Verifier strips KF_RET_NULL if input is owned ref, see is_kfunc_ret_null
* in verifier.c
*/
return (void *)p__refcounted_kptr;
}
__bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta__ign)
{
return bpf_refcount_acquire(p__refcounted_kptr, meta__ign);
}
static int __bpf_list_add(struct bpf_list_node_kern *node,
struct bpf_list_head *head,
bool tail, struct btf_record *rec, u64 off)
{
struct list_head *n = &node->list_head, *h = (void *)head;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
/* node->owner != NULL implies !list_empty(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
tail ? list_add_tail(n, h) : list_add(n, h);
WRITE_ONCE(node->owner, head);
return 0;
}
/**
* bpf_list_push_front() - add a node to the front of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the front of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_front(head, node, meta__ign, off);
}
/**
* bpf_list_push_back() - add a node to the back of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the back of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_back(head, node, meta__ign, off);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
{
struct list_head *n, *h = (void *)head;
struct bpf_list_node_kern *node;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
if (list_empty(h))
return NULL;
n = tail ? h->prev : h->next;
node = container_of(n, struct bpf_list_node_kern, list_head);
if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))
return NULL;
list_del_init(n);
WRITE_ONCE(node->owner, NULL);
return (struct bpf_list_node *)n;
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
{
return __bpf_list_del(head, false);
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
{
return __bpf_list_del(head, true);
}
__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->next;
}
__bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->prev;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
struct rb_root_cached *r = (struct rb_root_cached *)root;
struct rb_node *n = &node_internal->rb_node;
/* node_internal->owner != root implies either RB_EMPTY_NODE(n) or
* n is owned by some other tree. No need to check RB_EMPTY_NODE(n)
*/
if (READ_ONCE(node_internal->owner) != root)
return NULL;
rb_erase_cached(n, r);
RB_CLEAR_NODE(n);
WRITE_ONCE(node_internal->owner, NULL);
return (struct bpf_rb_node *)n;
}
/* Need to copy rbtree_add_cached's logic here because our 'less' is a BPF
* program
*/
static int __bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node_kern *node,
void *less, struct btf_record *rec, u64 off)
{
struct rb_node **link = &((struct rb_root_cached *)root)->rb_root.rb_node;
struct rb_node *parent = NULL, *n = &node->rb_node;
bpf_callback_t cb = (bpf_callback_t)less;
bool leftmost = true;
/* node->owner != NULL implies !RB_EMPTY_NODE(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
while (*link) {
parent = *link;
if (cb((uintptr_t)node, (uintptr_t)parent, 0, 0, 0)) {
link = &parent->rb_left;
} else {
link = &parent->rb_right;
leftmost = false;
}
}
rb_link_node(n, parent, link);
rb_insert_color_cached(n, (struct rb_root_cached *)root, leftmost);
WRITE_ONCE(node->owner, root);
return 0;
}
/**
* bpf_rbtree_add() - add a node to a BPF rbtree
* @root: tree root
* @node: node to insert
* @less: comparator used to order nodes
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node into @root using @less. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked in a tree.
*/
__bpf_kfunc int bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_rb_node_kern *n = (void *)node;
return __bpf_rbtree_add(root, n, (void *)less, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
void *meta__ign, u64 off)
{
return bpf_rbtree_add(root, node, less, meta__ign, off);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)rb_first_cached(r);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_root(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)r->rb_root.rb_node;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_left(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_left;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_right(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_right;
}
/**
* bpf_task_acquire - Acquire a reference to a task. A task acquired by this
* kfunc which is not stored in a map as a kptr, must be released by calling
* bpf_task_release().
* @p: The task on which a reference is being acquired.
*/
__bpf_kfunc struct task_struct *bpf_task_acquire(struct task_struct *p)
{
if (refcount_inc_not_zero(&p->rcu_users))
return p;
return NULL;
}
/**
* bpf_task_release - Release the reference acquired on a task.
* @p: The task on which a reference is being released.
*/
__bpf_kfunc void bpf_task_release(struct task_struct *p)
{
put_task_struct_rcu_user(p);
}
__bpf_kfunc void bpf_task_release_dtor(void *p)
{
put_task_struct_rcu_user(p);
}
CFI_NOSEAL(bpf_task_release_dtor);
#ifdef CONFIG_CGROUPS
/**
* bpf_cgroup_acquire - Acquire a reference to a cgroup. A cgroup acquired by
* this kfunc which is not stored in a map as a kptr, must be released by
* calling bpf_cgroup_release().
* @cgrp: The cgroup on which a reference is being acquired.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)
{
return cgroup_tryget(cgrp) ? cgrp : NULL;
}
/**
* bpf_cgroup_release - Release the reference acquired on a cgroup.
* If this kfunc is invoked in an RCU read region, the cgroup is guaranteed to
* not be freed until the current grace period has ended, even if its refcount
* drops to 0.
* @cgrp: The cgroup on which a reference is being released.
*/
__bpf_kfunc void bpf_cgroup_release(struct cgroup *cgrp)
{
cgroup_put(cgrp);
}
__bpf_kfunc void bpf_cgroup_release_dtor(void *cgrp)
{
cgroup_put(cgrp);
}
CFI_NOSEAL(bpf_cgroup_release_dtor);
/**
* bpf_cgroup_ancestor - Perform a lookup on an entry in a cgroup's ancestor
* array. A cgroup returned by this kfunc which is not subsequently stored in a
* map, must be released by calling bpf_cgroup_release().
* @cgrp: The cgroup for which we're performing a lookup.
* @level: The level of ancestor to look up.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)
{
struct cgroup *ancestor;
if (level > cgrp->level || level < 0)
return NULL;
/* cgrp's refcnt could be 0 here, but ancestors can still be accessed */
ancestor = cgrp->ancestors[level];
if (!cgroup_tryget(ancestor))
return NULL;
return ancestor;
}
/**
* bpf_cgroup_from_id - Find a cgroup from its ID. A cgroup returned by this
* kfunc which is not subsequently stored in a map, must be released by calling
* bpf_cgroup_release().
* @cgid: cgroup id.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_from_id(u64 cgid)
{
struct cgroup *cgrp;
cgrp = __cgroup_get_from_id(cgid);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
/**
* bpf_task_under_cgroup - wrap task_under_cgroup_hierarchy() as a kfunc, test
* task's membership of cgroup ancestry.
* @task: the task to be tested
* @ancestor: possible ancestor of @task's cgroup
*
* Tests whether @task's default cgroup hierarchy is a descendant of @ancestor.
* It follows all the same rules as cgroup_is_descendant, and only applies
* to the default hierarchy.
*/
__bpf_kfunc long bpf_task_under_cgroup(struct task_struct *task,
struct cgroup *ancestor)
{
long ret;
rcu_read_lock();
ret = task_under_cgroup_hierarchy(task, ancestor);
rcu_read_unlock();
return ret;
}
BPF_CALL_2(bpf_current_task_under_cgroup, struct bpf_map *, map, u32, idx)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
struct cgroup *cgrp;
if (unlikely(idx >= array->map.max_entries))
return -E2BIG;
cgrp = READ_ONCE(array->ptrs[idx]);
if (unlikely(!cgrp))
return -EAGAIN;
return task_under_cgroup_hierarchy(current, cgrp);
}
const struct bpf_func_proto bpf_current_task_under_cgroup_proto = {
.func = bpf_current_task_under_cgroup,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
};
/**
* bpf_task_get_cgroup1 - Acquires the associated cgroup of a task within a
* specific cgroup1 hierarchy. The cgroup1 hierarchy is identified by its
* hierarchy ID.
* @task: The target task
* @hierarchy_id: The ID of a cgroup1 hierarchy
*
* On success, the cgroup is returen. On failure, NULL is returned.
*/
__bpf_kfunc struct cgroup *
bpf_task_get_cgroup1(struct task_struct *task, int hierarchy_id)
{
struct cgroup *cgrp = task_get_cgroup1(task, hierarchy_id);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
#endif /* CONFIG_CGROUPS */
/**
* bpf_task_from_pid - Find a struct task_struct from its pid by looking it up
* in the root pid namespace idr. If a task is returned, it must either be
* stored in a map, or released with bpf_task_release().
* @pid: The pid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_pid_ns(pid, &init_pid_ns);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_task_from_vpid - Find a struct task_struct from its vpid by looking it up
* in the pid namespace of the current task. If a task is returned, it must
* either be stored in a map, or released with bpf_task_release().
* @vpid: The vpid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_vpid(vpid);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_dynptr_slice() - Obtain a read-only pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* If the intention is to write to the data slice, please use
* bpf_dynptr_slice_rdwr.
*
* The user must check that the returned pointer is not null before using it.
*
* Please note that in the case of skb and xdp dynptrs, bpf_dynptr_slice
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a read-only
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
enum bpf_dynptr_type type;
u64 len = buffer__szk;
int err;
if (!ptr->data)
return NULL;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return NULL;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return ptr->data + ptr->offset + offset;
case BPF_DYNPTR_TYPE_SKB:
if (buffer__nullable)
return skb_header_pointer(ptr->data, ptr->offset + offset, len, buffer__nullable);
else
return skb_pointer_if_linear(ptr->data, ptr->offset + offset, len);
case BPF_DYNPTR_TYPE_XDP:
{
void *xdp_ptr = bpf_xdp_pointer(ptr->data, ptr->offset + offset, len);
if (!IS_ERR_OR_NULL(xdp_ptr))
return xdp_ptr;
if (!buffer__nullable)
return NULL;
bpf_xdp_copy_buf(ptr->data, ptr->offset + offset, buffer__nullable, len, false);
return buffer__nullable;
}
case BPF_DYNPTR_TYPE_SKB_META:
return bpf_skb_meta_pointer(ptr->data, ptr->offset + offset);
case BPF_DYNPTR_TYPE_FILE:
err = bpf_file_fetch_bytes(ptr->data, offset, buffer__nullable, buffer__szk);
return err ? NULL : buffer__nullable;
default:
WARN_ONCE(true, "unknown dynptr type %d\n", type);
return NULL;
}
}
/**
* bpf_dynptr_slice_rdwr() - Obtain a writable pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* The returned pointer is writable and may point to either directly the dynptr
* data at the requested offset or to the buffer if unable to obtain a direct
* data pointer to (example: the requested slice is to the paged area of an skb
* packet). In the case where the returned pointer is to the buffer, the user
* is responsible for persisting writes through calling bpf_dynptr_write(). This
* usually looks something like this pattern:
*
* struct eth_hdr *eth = bpf_dynptr_slice_rdwr(&dynptr, 0, buffer, sizeof(buffer));
* if (!eth)
* return TC_ACT_SHOT;
*
* // mutate eth header //
*
* if (eth == buffer)
* bpf_dynptr_write(&ptr, 0, buffer, sizeof(buffer), 0);
*
* Please note that, as in the example above, the user must check that the
* returned pointer is not null before using it.
*
* Please also note that in the case of skb and xdp dynptrs, bpf_dynptr_slice_rdwr
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice_rdwr will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data || __bpf_dynptr_is_rdonly(ptr))
return NULL;
/* bpf_dynptr_slice_rdwr is the same logic as bpf_dynptr_slice.
*
* For skb-type dynptrs, it is safe to write into the returned pointer
* if the bpf program allows skb data writes. There are two possibilities
* that may occur when calling bpf_dynptr_slice_rdwr:
*
* 1) The requested slice is in the head of the skb. In this case, the
* returned pointer is directly to skb data, and if the skb is cloned, the
* verifier will have uncloned it (see bpf_unclone_prologue()) already.
* The pointer can be directly written into.
*
* 2) Some portion of the requested slice is in the paged buffer area.
* In this case, the requested data will be copied out into the buffer
* and the returned pointer will be a pointer to the buffer. The skb
* will not be pulled. To persist the write, the user will need to call
* bpf_dynptr_write(), which will pull the skb and commit the write.
*
* Similarly for xdp programs, if the requested slice is not across xdp
* fragments, then a direct pointer will be returned, otherwise the data
* will be copied out into the buffer and the user will need to call
* bpf_dynptr_write() to commit changes.
*/
return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
}
__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 size;
if (!ptr->data || start > end)
return -EINVAL;
size = __bpf_dynptr_size(ptr);
if (start > size || end > size)
return -ERANGE;
bpf_dynptr_advance_offset(ptr, start);
bpf_dynptr_set_size(ptr, end - start);
return 0;
}
__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
return !ptr->data;
}
__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return false;
return __bpf_dynptr_is_rdonly(ptr);
}
__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return -EINVAL;
return __bpf_dynptr_size(ptr);
}
__bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
struct bpf_dynptr *clone__uninit)
{
struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data) {
bpf_dynptr_set_null(clone);
return -EINVAL;
}
*clone = *ptr;
return 0;
}
/**
* bpf_dynptr_copy() - Copy data from one dynptr to another.
* @dst_ptr: Destination dynptr - where data should be copied to
* @dst_off: Offset into the destination dynptr
* @src_ptr: Source dynptr - where data should be copied from
* @src_off: Offset into the source dynptr
* @size: Length of the data to copy from source to destination
*
* Copies data from source dynptr to destination dynptr.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
{
struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
void *src_slice, *dst_slice;
char buf[256];
u64 off;
src_slice = bpf_dynptr_slice(src_ptr, src_off, NULL, size);
dst_slice = bpf_dynptr_slice_rdwr(dst_ptr, dst_off, NULL, size);
if (src_slice && dst_slice) {
memmove(dst_slice, src_slice, size);
return 0;
}
if (src_slice)
return __bpf_dynptr_write(dst, dst_off, src_slice, size, 0);
if (dst_slice)
return __bpf_dynptr_read(dst_slice, size, src, src_off, 0);
if (bpf_dynptr_check_off_len(dst, dst_off, size) ||
bpf_dynptr_check_off_len(src, src_off, size))
return -E2BIG;
off = 0;
while (off < size) {
u64 chunk_sz = min_t(u64, sizeof(buf), size - off);
int err;
err = __bpf_dynptr_read(buf, chunk_sz, src, src_off + off, 0);
if (err)
return err;
err = __bpf_dynptr_write(dst, dst_off + off, buf, chunk_sz, 0);
if (err)
return err;
off += chunk_sz;
}
return 0;
}
/**
* bpf_dynptr_memset() - Fill dynptr memory with a constant byte.
* @p: Destination dynptr - where data will be filled
* @offset: Offset into the dynptr to start filling from
* @size: Number of bytes to fill
* @val: Constant byte to fill the memory with
*
* Fills the @size bytes of the memory area pointed to by @p
* at @offset with the constant byte @val.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 chunk_sz, write_off;
char buf[256];
void* slice;
int err;
slice = bpf_dynptr_slice_rdwr(p, offset, NULL, size);
if (likely(slice)) {
memset(slice, val, size);
return 0;
}
if (__bpf_dynptr_is_rdonly(ptr))
return -EINVAL;
err = bpf_dynptr_check_off_len(ptr, offset, size);
if (err)
return err;
/* Non-linear data under the dynptr, write from a local buffer */
chunk_sz = min_t(u64, sizeof(buf), size);
memset(buf, val, chunk_sz);
for (write_off = 0; write_off < size; write_off += chunk_sz) {
chunk_sz = min_t(u64, sizeof(buf), size - write_off);
err = __bpf_dynptr_write(ptr, offset + write_off, buf, chunk_sz, 0);
if (err)
return err;
}
return 0;
}
__bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
{
return obj;
}
__bpf_kfunc void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k)
{
return (void *)obj__ign;
}
__bpf_kfunc void bpf_rcu_read_lock(void)
{
rcu_read_lock();
}
__bpf_kfunc void bpf_rcu_read_unlock(void)
{
rcu_read_unlock();
}
struct bpf_throw_ctx {
struct bpf_prog_aux *aux;
u64 sp;
u64 bp;
int cnt;
};
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
{
struct bpf_throw_ctx *ctx = cookie;
struct bpf_prog *prog;
/*
* The RCU read lock is held to safely traverse the latch tree, but we
* don't need its protection when accessing the prog, since it has an
* active stack frame on the current stack trace, and won't disappear.
*/
rcu_read_lock();
prog = bpf_prog_ksym_find(ip);
rcu_read_unlock();
if (!prog)
return !ctx->cnt;
ctx->cnt++;
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
ctx->sp = sp;
ctx->bp = bp;
return false;
}
__bpf_kfunc void bpf_throw(u64 cookie)
{
struct bpf_throw_ctx ctx = {};
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
WARN_ON_ONCE(!ctx.aux->exception_boundary);
WARN_ON_ONCE(!ctx.bp);
WARN_ON_ONCE(!ctx.cnt);
/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
* deeper stack depths than ctx.sp as we do not return from bpf_throw,
* which skips compiler generated instrumentation to do the same.
*/
kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_map *map = p__map;
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq));
if (flags)
return -EINVAL;
return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ);
}
__bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_work *w;
if (flags)
return -EINVAL;
w = READ_ONCE(async->work);
if (!w || !READ_ONCE(w->cb.prog))
return -EINVAL;
if (!refcount_inc_not_zero(&w->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
schedule_work(&w->work);
bpf_async_refcount_put(&w->cb);
return 0;
} else {
return bpf_async_schedule_op(&w->cb, BPF_ASYNC_START, 0, 0);
}
}
__bpf_kfunc int bpf_wq_set_callback(struct bpf_wq *wq,
int (callback_fn)(void *map, int *key, void *value),
unsigned int flags,
struct bpf_prog_aux *aux)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
if (flags)
return -EINVAL;
return __bpf_async_set_callback(async, callback_fn, aux->prog);
}
__bpf_kfunc void bpf_preempt_disable(void)
{
preempt_disable();
}
__bpf_kfunc void bpf_preempt_enable(void)
{
preempt_enable();
}
struct bpf_iter_bits {
__u64 __opaque[2];
} __aligned(8);
#define BITS_ITER_NR_WORDS_MAX 511
struct bpf_iter_bits_kern {
union {
__u64 *bits;
__u64 bits_copy;
};
int nr_bits;
int bit;
} __aligned(8);
/* On 64-bit hosts, unsigned long and u64 have the same size, so passing
* a u64 pointer and an unsigned long pointer to find_next_bit() will
* return the same result, as both point to the same 8-byte area.
*
* For 32-bit little-endian hosts, using a u64 pointer or unsigned long
* pointer also makes no difference. This is because the first iterated
* unsigned long is composed of bits 0-31 of the u64 and the second unsigned
* long is composed of bits 32-63 of the u64.
*
* However, for 32-bit big-endian hosts, this is not the case. The first
* iterated unsigned long will be bits 32-63 of the u64, so swap these two
* ulong values within the u64.
*/
static void swap_ulong_in_u64(u64 *bits, unsigned int nr)
{
#if (BITS_PER_LONG == 32) && defined(__BIG_ENDIAN)
unsigned int i;
for (i = 0; i < nr; i++)
bits[i] = (bits[i] >> 32) | ((u64)(u32)bits[i] << 32);
#endif
}
/**
* bpf_iter_bits_new() - Initialize a new bits iterator for a given memory area
* @it: The new bpf_iter_bits to be created
* @unsafe_ptr__ign: A pointer pointing to a memory area to be iterated over
* @nr_words: The size of the specified memory area, measured in 8-byte units.
* The maximum value of @nr_words is @BITS_ITER_NR_WORDS_MAX. This limit may be
* further reduced by the BPF memory allocator implementation.
*
* This function initializes a new bpf_iter_bits structure for iterating over
* a memory area which is specified by the @unsafe_ptr__ign and @nr_words. It
* copies the data of the memory area to the newly created bpf_iter_bits @it for
* subsequent iteration operations.
*
* On success, 0 is returned. On failure, ERR is returned.
*/
__bpf_kfunc int
bpf_iter_bits_new(struct bpf_iter_bits *it, const u64 *unsafe_ptr__ign, u32 nr_words)
{
struct bpf_iter_bits_kern *kit = (void *)it;
u32 nr_bytes = nr_words * sizeof(u64);
u32 nr_bits = BYTES_TO_BITS(nr_bytes);
int err;
BUILD_BUG_ON(sizeof(struct bpf_iter_bits_kern) != sizeof(struct bpf_iter_bits));
BUILD_BUG_ON(__alignof__(struct bpf_iter_bits_kern) !=
__alignof__(struct bpf_iter_bits));
kit->nr_bits = 0;
kit->bits_copy = 0;
kit->bit = -1;
if (!unsafe_ptr__ign || !nr_words)
return -EINVAL;
if (nr_words > BITS_ITER_NR_WORDS_MAX)
return -E2BIG;
/* Optimization for u64 mask */
if (nr_bits == 64) {
err = bpf_probe_read_kernel_common(&kit->bits_copy, nr_bytes, unsafe_ptr__ign);
if (err)
return -EFAULT;
swap_ulong_in_u64(&kit->bits_copy, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
if (bpf_mem_alloc_check_size(false, nr_bytes))
return -E2BIG;
/* Fallback to memalloc */
kit->bits = bpf_mem_alloc(&bpf_global_ma, nr_bytes);
if (!kit->bits)
return -ENOMEM;
err = bpf_probe_read_kernel_common(kit->bits, nr_bytes, unsafe_ptr__ign);
if (err) {
bpf_mem_free(&bpf_global_ma, kit->bits);
return err;
}
swap_ulong_in_u64(kit->bits, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
/**
* bpf_iter_bits_next() - Get the next bit in a bpf_iter_bits
* @it: The bpf_iter_bits to be checked
*
* This function returns a pointer to a number representing the value of the
* next bit in the bits.
*
* If there are no further bits available, it returns NULL.
*/
__bpf_kfunc int *bpf_iter_bits_next(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
int bit = kit->bit, nr_bits = kit->nr_bits;
const void *bits;
if (!nr_bits || bit >= nr_bits)
return NULL;
bits = nr_bits == 64 ? &kit->bits_copy : kit->bits;
bit = find_next_bit(bits, nr_bits, bit + 1);
if (bit >= nr_bits) {
kit->bit = bit;
return NULL;
}
kit->bit = bit;
return &kit->bit;
}
/**
* bpf_iter_bits_destroy() - Destroy a bpf_iter_bits
* @it: The bpf_iter_bits to be destroyed
*
* Destroy the resource associated with the bpf_iter_bits.
*/
__bpf_kfunc void bpf_iter_bits_destroy(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
if (kit->nr_bits <= 64)
return;
bpf_mem_free(&bpf_global_ma, kit->bits);
}
/**
* bpf_copy_from_user_str() - Copy a string from an unsafe user address
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address, in user space.
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL-terminated string from userspace to BPF space. If user string is
* too long this will still ensure zero termination in the dst buffer unless
* buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst to 0 on success and
* memset all of @dst on failure.
*/
__bpf_kfunc int bpf_copy_from_user_str(void *dst, u32 dst__sz, const void __user *unsafe_ptr__ign, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(!dst__sz))
return 0;
ret = strncpy_from_user(dst, unsafe_ptr__ign, dst__sz - 1);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst + ret, 0, dst__sz - ret);
else
((char *)dst)[ret] = '\0';
return ret + 1;
}
/**
* bpf_copy_from_user_task_str() - Copy a string from an task's address space
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address in the task's address space.
* @tsk: The task whose address space will be used
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL terminated string from a task's address space to @dst__sz
* buffer. If user string is too long this will still ensure zero termination
* in the @dst__sz buffer unless buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst__sz to 0 on success
* and memset all of @dst__sz on failure.
*
* Return: The number of copied bytes on success including the NUL terminator.
* A negative error code on failure.
*/
__bpf_kfunc int bpf_copy_from_user_task_str(void *dst, u32 dst__sz,
const void __user *unsafe_ptr__ign,
struct task_struct *tsk, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(dst__sz == 0))
return 0;
ret = copy_remote_vm_str(tsk, (unsigned long)unsafe_ptr__ign, dst, dst__sz, 0);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset(dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset(dst + ret, 0, dst__sz - ret);
return ret + 1;
}
/* Keep unsinged long in prototype so that kfunc is usable when emitted to
* vmlinux.h in BPF programs directly, but note that while in BPF prog, the
* unsigned long always points to 8-byte region on stack, the kernel may only
* read and write the 4-bytes on 32-bit.
*/
__bpf_kfunc void bpf_local_irq_save(unsigned long *flags__irq_flag)
{
local_irq_save(*flags__irq_flag);
}
__bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)
{
local_irq_restore(*flags__irq_flag);
}
__bpf_kfunc void __bpf_trap(void)
{
}
/*
* Kfuncs for string operations.
*
* Since strings are not necessarily %NUL-terminated, we cannot directly call
* in-kernel implementations. Instead, we open-code the implementations using
* __get_kernel_nofault instead of plain dereference to make them safe.
*/
static int __bpf_strncasecmp(const char *s1, const char *s2, bool ignore_case, size_t len)
{
char c1, c2;
int i;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < len && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c1, s1, char, err_out);
__get_kernel_nofault(&c2, s2, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 != c2)
return c1 < c2 ? -1 : 1;
if (c1 == '\0')
return 0;
s1++;
s2++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : 0;
err_out:
return -EFAULT;
}
/**
* bpf_strcmp - Compare two strings
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, false, XATTR_SIZE_MAX);
}
/**
* bpf_strcasecmp - Compare two strings, ignoring the case of the characters
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasecmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, XATTR_SIZE_MAX);
}
/*
* bpf_strncasecmp - Compare two length-limited strings, ignoring case
* @s1__ign: One string
* @s2__ign: Another string
* @len: The maximum number of characters to compare
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasecmp(const char *s1__ign, const char *s2__ign, size_t len)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, len);
}
/**
* bpf_strnchr - Find a character in a length limited string
* @s__ign: The string to be searched
* @count: The number of characters to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in the first @count characters of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnchr(const char *s__ign, size_t count, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
return i;
if (sc == '\0')
return -ENOENT;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : -ENOENT;
err_out:
return -EFAULT;
}
/**
* bpf_strchr - Find the first occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - The index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchr(const char *s__ign, char c)
{
return bpf_strnchr(s__ign, XATTR_SIZE_MAX, c);
}
/**
* bpf_strchrnul - Find and return a character in a string, or end of string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign or index of
* the null byte at the end of @s__ign when @c is not found
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchrnul(const char *s__ign, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == '\0' || sc == c)
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strrchr - Find the last occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the last occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strrchr(const char *s__ign, int c)
{
char sc;
int i, last = -ENOENT;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
last = i;
if (sc == '\0')
return last;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strnlen - Calculate the length of a length-limited string
* @s__ign: The string
* @count: The maximum number of characters to count
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnlen(const char *s__ign, size_t count)
{
char c;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c, s__ign, char, err_out);
if (c == '\0')
return i;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : i;
err_out:
return -EFAULT;
}
/**
* bpf_strlen - Calculate the length of a string
* @s__ign: The string
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strlen(const char *s__ign)
{
return bpf_strnlen(s__ign, XATTR_SIZE_MAX);
}
/**
* bpf_strspn - Calculate the length of the initial substring of @s__ign which
* only contains letters in @accept__ign
* @s__ign: The string to be searched
* @accept__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which only
* contains letters from @accept__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strspn(const char *s__ign, const char *accept__ign)
{
char cs, ca;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(accept__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&ca, accept__ign + j, char, err_out);
if (cs == ca || ca == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (ca == '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strcspn - Calculate the length of the initial substring of @s__ign which
* does not contain letters in @reject__ign
* @s__ign: The string to be searched
* @reject__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which does not
* contain letters from @reject__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcspn(const char *s__ign, const char *reject__ign)
{
char cs, cr;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(reject__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&cr, reject__ign + j, char, err_out);
if (cs == cr || cr == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (cr != '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
static int __bpf_strnstr(const char *s1, const char *s2, size_t len,
bool ignore_case)
{
char c1, c2;
int i, j;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
for (j = 0; i + j <= len && j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&c2, s2 + j, char, err_out);
if (c2 == '\0')
return i;
/*
* We allow reading an extra byte from s2 (note the
* `i + j <= len` above) to cover the case when s2 is
* a suffix of the first len chars of s1.
*/
if (i + j == len)
break;
__get_kernel_nofault(&c1, s1 + j, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 == '\0')
return -ENOENT;
if (c1 != c2)
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (i + j == len)
return -ENOENT;
s1++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strstr - Find the first substring in a string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strstr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, false);
}
/**
* bpf_strcasestr - Find the first substring in a string, ignoring the case of
* the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasestr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, true);
}
/**
* bpf_strnstr - Find the first substring in a length-limited string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strnstr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, false);
}
/**
* bpf_strncasestr - Find the first substring in a length-limited string,
* ignoring the case of the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasestr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, true);
}
#ifdef CONFIG_KEYS
/**
* bpf_lookup_user_key - lookup a key by its serial
* @serial: key handle serial number
* @flags: lookup-specific flags
*
* Search a key with a given *serial* and the provided *flags*.
* If found, increment the reference count of the key by one, and
* return it in the bpf_key structure.
*
* The bpf_key structure must be passed to bpf_key_put() when done
* with it, so that the key reference count is decremented and the
* bpf_key structure is freed.
*
* Permission checks are deferred to the time the key is used by
* one of the available key-specific kfuncs.
*
* Set *flags* with KEY_LOOKUP_CREATE, to attempt creating a requested
* special keyring (e.g. session keyring), if it doesn't yet exist.
* Set *flags* with KEY_LOOKUP_PARTIAL, to lookup a key without waiting
* for the key construction, and to retrieve uninstantiated keys (keys
* without data attached to them).
*
* Return: a bpf_key pointer with a valid key pointer if the key is found, a
* NULL pointer otherwise.
*/
__bpf_kfunc struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags)
{
key_ref_t key_ref;
struct bpf_key *bkey;
if (flags & ~KEY_LOOKUP_ALL)
return NULL;
/*
* Permission check is deferred until the key is used, as the
* intent of the caller is unknown here.
*/
key_ref = lookup_user_key(serial, flags, KEY_DEFER_PERM_CHECK);
if (IS_ERR(key_ref))
return NULL;
bkey = kmalloc_obj(*bkey);
if (!bkey) {
key_put(key_ref_to_ptr(key_ref));
return NULL;
}
bkey->key = key_ref_to_ptr(key_ref);
bkey->has_ref = true;
return bkey;
}
/**
* bpf_lookup_system_key - lookup a key by a system-defined ID
* @id: key ID
*
* Obtain a bpf_key structure with a key pointer set to the passed key ID.
* The key pointer is marked as invalid, to prevent bpf_key_put() from
* attempting to decrement the key reference count on that pointer. The key
* pointer set in such way is currently understood only by
* verify_pkcs7_signature().
*
* Set *id* to one of the values defined in include/linux/verification.h:
* 0 for the primary keyring (immutable keyring of system keys);
* VERIFY_USE_SECONDARY_KEYRING for both the primary and secondary keyring
* (where keys can be added only if they are vouched for by existing keys
* in those keyrings); VERIFY_USE_PLATFORM_KEYRING for the platform
* keyring (primarily used by the integrity subsystem to verify a kexec'ed
* kerned image and, possibly, the initramfs signature).
*
* Return: a bpf_key pointer with an invalid key pointer set from the
* pre-determined ID on success, a NULL pointer otherwise
*/
__bpf_kfunc struct bpf_key *bpf_lookup_system_key(u64 id)
{
struct bpf_key *bkey;
if (system_keyring_id_check(id) < 0)
return NULL;
bkey = kmalloc_obj(*bkey, GFP_ATOMIC);
if (!bkey)
return NULL;
bkey->key = (struct key *)(unsigned long)id;
bkey->has_ref = false;
return bkey;
}
/**
* bpf_key_put - decrement key reference count if key is valid and free bpf_key
* @bkey: bpf_key structure
*
* Decrement the reference count of the key inside *bkey*, if the pointer
* is valid, and free *bkey*.
*/
__bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
{
if (bkey->has_ref)
key_put(bkey->key);
kfree(bkey);
}
/**
* bpf_verify_pkcs7_signature - verify a PKCS#7 signature
* @data_p: data to verify
* @sig_p: signature of the data
* @trusted_keyring: keyring with keys trusted for signature verification
*
* Verify the PKCS#7 signature *sig_ptr* against the supplied *data_ptr*
* with keys in a keyring referenced by *trusted_keyring*.
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const void *data, *sig;
u32 data_len, sig_len;
int ret;
if (trusted_keyring->has_ref) {
/*
* Do the permission check deferred in bpf_lookup_user_key().
* See bpf_lookup_user_key() for more details.
*
* A call to key_task_permission() here would be redundant, as
* it is already done by keyring_search() called by
* find_asymmetric_key().
*/
ret = key_validate(trusted_keyring->key);
if (ret < 0)
return ret;
}
data_len = __bpf_dynptr_size(data_ptr);
data = __bpf_dynptr_data(data_ptr, data_len);
sig_len = __bpf_dynptr_size(sig_ptr);
sig = __bpf_dynptr_data(sig_ptr, sig_len);
return verify_pkcs7_signature(data, data_len, sig, sig_len,
trusted_keyring->key,
VERIFYING_BPF_SIGNATURE, NULL,
NULL);
#else
return -EOPNOTSUPP;
#endif /* CONFIG_SYSTEM_DATA_VERIFICATION */
}
#endif /* CONFIG_KEYS */
typedef int (*bpf_task_work_callback_t)(struct bpf_map *map, void *key, void *value);
enum bpf_task_work_state {
/* bpf_task_work is ready to be used */
BPF_TW_STANDBY = 0,
/* irq work scheduling in progress */
BPF_TW_PENDING,
/* task work scheduling in progress */
BPF_TW_SCHEDULING,
/* task work is scheduled successfully */
BPF_TW_SCHEDULED,
/* callback is running */
BPF_TW_RUNNING,
/* associated BPF map value is deleted */
BPF_TW_FREED,
};
struct bpf_task_work_ctx {
enum bpf_task_work_state state;
refcount_t refcnt;
struct callback_head work;
struct irq_work irq_work;
/* bpf_prog that schedules task work */
struct bpf_prog *prog;
/* task for which callback is scheduled */
struct task_struct *task;
/* the map and map value associated with this context */
struct bpf_map *map;
void *map_val;
enum task_work_notify_mode mode;
bpf_task_work_callback_t callback_fn;
struct rcu_head rcu;
} __aligned(8);
/* Actual type for struct bpf_task_work */
struct bpf_task_work_kern {
struct bpf_task_work_ctx *ctx;
};
static void bpf_task_work_ctx_reset(struct bpf_task_work_ctx *ctx)
{
if (ctx->prog) {
bpf_prog_put(ctx->prog);
ctx->prog = NULL;
}
if (ctx->task) {
bpf_task_release(ctx->task);
ctx->task = NULL;
}
}
static bool bpf_task_work_ctx_tryget(struct bpf_task_work_ctx *ctx)
{
return refcount_inc_not_zero(&ctx->refcnt);
}
static void bpf_task_work_destroy(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_ctx_reset(ctx);
kfree_rcu(ctx, rcu);
}
static void bpf_task_work_ctx_put(struct bpf_task_work_ctx *ctx)
{
if (!refcount_dec_and_test(&ctx->refcnt))
return;
if (irqs_disabled()) {
ctx->irq_work = IRQ_WORK_INIT(bpf_task_work_destroy);
irq_work_queue(&ctx->irq_work);
} else {
bpf_task_work_destroy(&ctx->irq_work);
}
}
static void bpf_task_work_cancel(struct bpf_task_work_ctx *ctx)
{
/*
* Scheduled task_work callback holds ctx ref, so if we successfully
* cancelled, we put that ref on callback's behalf. If we couldn't
* cancel, callback will inevitably run or has already completed
* running, and it would have taken care of its ctx ref itself.
*/
if (task_work_cancel(ctx->task, &ctx->work))
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_callback(struct callback_head *cb)
{
struct bpf_task_work_ctx *ctx = container_of(cb, struct bpf_task_work_ctx, work);
enum bpf_task_work_state state;
u32 idx;
void *key;
/* Read lock is needed to protect ctx and map key/value access */
guard(rcu_tasks_trace)();
/*
* This callback may start running before bpf_task_work_irq() switched to
* SCHEDULED state, so handle both transition variants SCHEDULING|SCHEDULED -> RUNNING.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_RUNNING);
if (state == BPF_TW_SCHEDULED)
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULED, BPF_TW_RUNNING);
if (state == BPF_TW_FREED) {
bpf_task_work_ctx_put(ctx);
return;
}
key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx);
migrate_disable();
ctx->callback_fn(ctx->map, key, ctx->map_val);
migrate_enable();
bpf_task_work_ctx_reset(ctx);
(void)cmpxchg(&ctx->state, BPF_TW_RUNNING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_irq(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
enum bpf_task_work_state state;
int err;
guard(rcu)();
if (cmpxchg(&ctx->state, BPF_TW_PENDING, BPF_TW_SCHEDULING) != BPF_TW_PENDING) {
bpf_task_work_ctx_put(ctx);
return;
}
err = task_work_add(ctx->task, &ctx->work, ctx->mode);
if (err) {
bpf_task_work_ctx_reset(ctx);
/*
* try to switch back to STANDBY for another task_work reuse, but we might have
* gone to FREED already, which is fine as we already cleaned up after ourselves
*/
(void)cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
return;
}
/*
* It's technically possible for just scheduled task_work callback to
* complete running by now, going SCHEDULING -> RUNNING and then
* dropping its ctx refcount. Instead of capturing an extra ref just
* to protect below ctx->state access, we rely on rcu_read_lock
* above to prevent kfree_rcu from freeing ctx before we return.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_SCHEDULED);
if (state == BPF_TW_FREED)
bpf_task_work_cancel(ctx); /* clean up if we switched into FREED state */
}
static struct bpf_task_work_ctx *bpf_task_work_fetch_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_kern *twk = (void *)tw;
struct bpf_task_work_ctx *ctx, *old_ctx;
ctx = READ_ONCE(twk->ctx);
if (ctx)
return ctx;
ctx = bpf_map_kmalloc_nolock(map, sizeof(*ctx), 0, NUMA_NO_NODE);
if (!ctx)
return ERR_PTR(-ENOMEM);
memset(ctx, 0, sizeof(*ctx));
refcount_set(&ctx->refcnt, 1); /* map's own ref */
ctx->state = BPF_TW_STANDBY;
old_ctx = cmpxchg(&twk->ctx, NULL, ctx);
if (old_ctx) {
/*
* tw->ctx is set by concurrent BPF program, release allocated
* memory and try to reuse already set context.
*/
kfree_nolock(ctx);
return old_ctx;
}
return ctx; /* Success */
}
static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_ctx *ctx;
/*
* Sleepable BPF programs hold rcu_read_lock_trace but not
* regular rcu_read_lock. Since kfree_rcu waits for regular
* RCU GP, the ctx can be freed while we're between reading
* the pointer and incrementing the refcount. Take regular
* rcu_read_lock to prevent kfree_rcu from freeing the ctx
* before we can tryget it.
*/
scoped_guard(rcu) {
ctx = bpf_task_work_fetch_ctx(tw, map);
if (IS_ERR(ctx))
return ctx;
/* try to get ref for task_work callback to hold */
if (!bpf_task_work_ctx_tryget(ctx))
return ERR_PTR(-EBUSY);
}
if (cmpxchg(&ctx->state, BPF_TW_STANDBY, BPF_TW_PENDING) != BPF_TW_STANDBY) {
/* lost acquiring race or map_release_uref() stole it from us, put ref and bail */
bpf_task_work_ctx_put(ctx);
return ERR_PTR(-EBUSY);
}
/*
* If no process or bpffs is holding a reference to the map, no new callbacks should be
* scheduled. This does not address any race or correctness issue, but rather is a policy
* choice: dropping user references should stop everything.
*/
if (!atomic64_read(&map->usercnt)) {
/* drop ref we just got for task_work callback itself */
bpf_task_work_ctx_put(ctx);
/* transfer map's ref into cancel_and_free() */
bpf_task_work_cancel_and_free(tw);
return ERR_PTR(-EBUSY);
}
return ctx;
}
static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw,
struct bpf_map *map, bpf_task_work_callback_t callback_fn,
struct bpf_prog_aux *aux, enum task_work_notify_mode mode)
{
struct bpf_prog *prog;
struct bpf_task_work_ctx *ctx;
int err;
BTF_TYPE_EMIT(struct bpf_task_work);
prog = bpf_prog_inc_not_zero(aux->prog);
if (IS_ERR(prog))
return -EBADF;
task = bpf_task_acquire(task);
if (!task) {
err = -EBADF;
goto release_prog;
}
ctx = bpf_task_work_acquire_ctx(tw, map);
if (IS_ERR(ctx)) {
err = PTR_ERR(ctx);
goto release_all;
}
ctx->task = task;
ctx->callback_fn = callback_fn;
ctx->prog = prog;
ctx->mode = mode;
ctx->map = map;
ctx->map_val = (void *)tw - map->record->task_work_off;
init_task_work(&ctx->work, bpf_task_work_callback);
init_irq_work(&ctx->irq_work, bpf_task_work_irq);
irq_work_queue(&ctx->irq_work);
return 0;
release_all:
bpf_task_release(task);
release_prog:
bpf_prog_put(prog);
return err;
}
/**
* bpf_task_work_schedule_signal - Schedule BPF callback using task_work_add with TWA_SIGNAL
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL);
}
/**
* bpf_task_work_schedule_resume - Schedule BPF callback using task_work_add with TWA_RESUME
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME);
}
static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
struct bpf_dynptr_kern *ptr)
{
struct bpf_dynptr_file_impl *state;
/* flags is currently unsupported */
if (flags) {
bpf_dynptr_set_null(ptr);
return -EINVAL;
}
state = kmalloc_nolock(sizeof(*state), 0, NUMA_NO_NODE);
if (!state) {
bpf_dynptr_set_null(ptr);
return -ENOMEM;
}
state->offset = 0;
state->size = U64_MAX; /* Don't restrict size, as file may change anyways */
freader_init_from_file(&state->freader, NULL, 0, file, may_sleep);
bpf_dynptr_init(ptr, state, BPF_DYNPTR_TYPE_FILE, 0, 0);
bpf_dynptr_set_rdonly(ptr);
return 0;
}
__bpf_kfunc int bpf_dynptr_from_file(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, false, (struct bpf_dynptr_kern *)ptr__uninit);
}
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, true, (struct bpf_dynptr_kern *)ptr__uninit);
}
__bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)dynptr;
struct bpf_dynptr_file_impl *df = ptr->data;
if (!df)
return 0;
freader_cleanup(&df->freader);
kfree_nolock(df);
bpf_dynptr_set_null(ptr);
return 0;
}
/**
* bpf_timer_cancel_async - try to deactivate a timer
* @timer: bpf_timer to stop
*
* Returns:
*
* * 0 when the timer was not active
* * 1 when the timer was active
* * -1 when the timer is currently executing the callback function and
* cannot be stopped
* * -ECANCELED when the timer will be cancelled asynchronously
* * -ENOMEM when out of memory
* * -EINVAL when the timer was not initialized
* * -ENOENT when this kfunc is racing with timer deletion
*/
__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer)
{
struct bpf_async_kern *async = (void *)timer;
struct bpf_async_cb *cb;
int ret;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
/*
* Unlike hrtimer_start() it's ok to synchronously call
* hrtimer_try_to_cancel() when refcnt reached zero, but deferring to
* irq_work is not, since irq callback may execute after RCU GP and
* cb could be freed at that time. Check for refcnt zero for
* consistency.
*/
if (!refcount_inc_not_zero(&cb->refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
ret = hrtimer_try_to_cancel(&t->timer);
bpf_async_refcount_put(cb);
return ret;
} else {
ret = bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
return ret ? ret : -ECANCELED;
}
}
__bpf_kfunc_end_defs();
static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_cancel(ctx); /* this might put task_work callback's ref */
bpf_task_work_ctx_put(ctx); /* and here we put map's own ref that was transferred to us */
}
void bpf_task_work_cancel_and_free(void *val)
{
struct bpf_task_work_kern *twk = val;
struct bpf_task_work_ctx *ctx;
enum bpf_task_work_state state;
ctx = xchg(&twk->ctx, NULL);
if (!ctx)
return;
state = xchg(&ctx->state, BPF_TW_FREED);
if (state == BPF_TW_SCHEDULED) {
/* run in irq_work to avoid locks in NMI */
init_irq_work(&ctx->irq_work, bpf_task_work_cancel_scheduled);
irq_work_queue(&ctx->irq_work);
return;
}
bpf_task_work_ctx_put(ctx); /* put bpf map's ref */
}
BTF_KFUNCS_START(generic_btf_ids)
#ifdef CONFIG_CRASH_DUMP
BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
#endif
BTF_ID_FLAGS(func, bpf_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_percpu_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_refcount_acquire, KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE | KF_RET_NULL | KF_RCU)
BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_front_impl)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_back_impl)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_rbtree_add_impl)
BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_cgroup_ancestor, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU)
BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
#endif
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_send_signal_task)
#endif
#ifdef CONFIG_KEYS
BTF_ID_FLAGS(func, bpf_lookup_user_key, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_lookup_system_key, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_key_put, KF_RELEASE)
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
BTF_ID_FLAGS(func, bpf_verify_pkcs7_signature, KF_SLEEPABLE)
#endif
#endif
#ifdef CONFIG_S390
BTF_ID_FLAGS(func, bpf_get_lowcore)
#endif
BTF_KFUNCS_END(generic_btf_ids)
static const struct btf_kfunc_id_set generic_kfunc_set = {
.owner = THIS_MODULE,
.set = &generic_btf_ids,
};
BTF_ID_LIST(generic_dtor_ids)
BTF_ID(struct, task_struct)
BTF_ID(func, bpf_task_release_dtor)
#ifdef CONFIG_CGROUPS
BTF_ID(struct, cgroup)
BTF_ID(func, bpf_cgroup_release_dtor)
#endif
BTF_KFUNCS_START(common_btf_ids)
BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rcu_read_lock)
BTF_ID_FLAGS(func, bpf_rcu_read_unlock)
BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_css_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_css_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_destroy, KF_ITER_DESTROY)
#endif
BTF_ID_FLAGS(func, bpf_iter_task_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_dynptr_adjust)
BTF_ID_FLAGS(func, bpf_dynptr_is_null)
BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
BTF_ID_FLAGS(func, bpf_dynptr_size)
BTF_ID_FLAGS(func, bpf_dynptr_clone)
BTF_ID_FLAGS(func, bpf_dynptr_copy)
BTF_ID_FLAGS(func, bpf_dynptr_memset)
#ifdef CONFIG_NET
BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
#endif
BTF_ID_FLAGS(func, bpf_wq_init)
BTF_ID_FLAGS(func, bpf_wq_set_callback, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_wq_start)
BTF_ID_FLAGS(func, bpf_preempt_disable)
BTF_ID_FLAGS(func, bpf_preempt_enable)
BTF_ID_FLAGS(func, bpf_iter_bits_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_bits_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_bits_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_copy_from_user_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_get_kmem_cache)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_local_irq_save)
BTF_ID_FLAGS(func, bpf_local_irq_restore)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr)
BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE)
#endif
#ifdef CONFIG_DMA_SHARED_BUFFER
BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
#endif
BTF_ID_FLAGS(func, __bpf_trap)
BTF_ID_FLAGS(func, bpf_strcmp);
BTF_ID_FLAGS(func, bpf_strcasecmp);
BTF_ID_FLAGS(func, bpf_strncasecmp);
BTF_ID_FLAGS(func, bpf_strchr);
BTF_ID_FLAGS(func, bpf_strchrnul);
BTF_ID_FLAGS(func, bpf_strnchr);
BTF_ID_FLAGS(func, bpf_strrchr);
BTF_ID_FLAGS(func, bpf_strlen);
BTF_ID_FLAGS(func, bpf_strnlen);
BTF_ID_FLAGS(func, bpf_strspn);
BTF_ID_FLAGS(func, bpf_strcspn);
BTF_ID_FLAGS(func, bpf_strstr);
BTF_ID_FLAGS(func, bpf_strcasestr);
BTF_ID_FLAGS(func, bpf_strnstr);
BTF_ID_FLAGS(func, bpf_strncasestr);
#if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS)
BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU)
#endif
BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_dynptr_from_file)
BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
BTF_ID_FLAGS(func, bpf_timer_cancel_async)
BTF_KFUNCS_END(common_btf_ids)
static const struct btf_kfunc_id_set common_kfunc_set = {
.owner = THIS_MODULE,
.set = &common_btf_ids,
};
static int __init kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc generic_dtors[] = {
{
.btf_id = generic_dtor_ids[0],
.kfunc_btf_id = generic_dtor_ids[1]
},
#ifdef CONFIG_CGROUPS
{
.btf_id = generic_dtor_ids[2],
.kfunc_btf_id = generic_dtor_ids[3]
},
#endif
};
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &generic_kfunc_set);
ret = ret ?: register_btf_id_dtor_kfuncs(generic_dtors,
ARRAY_SIZE(generic_dtors),
THIS_MODULE);
return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
}
late_initcall(kfunc_init);
/* Get a pointer to dynptr data up to len bytes for read only access. If
* the dynptr doesn't have continuous data up to len bytes, return NULL.
*/
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u64 len)
{
const struct bpf_dynptr *p = (struct bpf_dynptr *)ptr;
return bpf_dynptr_slice(p, 0, NULL, len);
}
/* Get a pointer to dynptr data up to len bytes for read write access. If
* the dynptr doesn't have continuous data up to len bytes, or the dynptr
* is read only, return NULL.
*/
void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u64 len)
{
if (__bpf_dynptr_is_rdonly(ptr))
return NULL;
return (void *)__bpf_dynptr_data(ptr, len);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *val)
{
if (btf_record_has_field(map->record, BPF_TIMER))
bpf_obj_free_timer(map->record, val);
if (btf_record_has_field(map->record, BPF_WORKQUEUE))
bpf_obj_free_workqueue(map->record, val);
if (btf_record_has_field(map->record, BPF_TASK_WORK))
bpf_obj_free_task_work(map->record, val);
}
]
|
| 500/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 67991
output: 32
thoughts: 1341
Thoughts: **Analyzing Potential Buffer Overflows in kernel/bpf/helpers.c**
Okay, I've thoroughly reviewed the `kernel/bpf/helpers.c` file and, as a precaution, I've double-checked my initial scan for `snprintf` instances to make sure I haven't missed anything. I've also done a quick sweep for `sprintf`, `vsnprintf`, `scnprintf`, and `vscnprintf`, but `codesearch-find-references` turned up only the two instances within `bpf_bprintf_prepare`.
I started by looking at other formatting functions. I noticed `bstr_printf` is called by `____bpf_snprintf`. `bpf_trace_printk` calls `bpf_trace_printk_proto.func` which is a BPF helper, and I confirmed that it's defined elsewhere, likely in `kernel/trace/bpf_trace.c`. I also recalled `bpf_stream_vprintk`, which I need to investigate further. I'm going to look for its definition since it wasn't present in this file.
Focusing specifically on `kernel/bpf/helpers.c`, the critical area is the usage of `snprintf` within `bpf_bprintf_prepare`. I need to re-verify the two `snprintf` calls there.
In both of the `snprintf` instances:
```c
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf, ...);
tmp_buf += (err + 1);
```
If `err` is greater than or equal to `(tmp_buf_end - tmp_buf)`, then incrementing `tmp_buf` by `err + 1` could lead to `tmp_buf` exceeding `tmp_buf_end`, potentially leading to a buffer overflow on the next loop iteration or subsequent code execution.
I did a thorough search through this file, as well as the entire directory to be absolutely sure, confirming that there were no other instances of `snprintf`. I then went on to check `strscpy`, which I confirmed is safe.
Then I considered `bpf_trace_copy_string`. It uses `strncpy_from_kernel_nofault` (or user), and its return value is the length of the copied string. My initial assumption about that was incorrect. The code is structured as:
```c
err = bpf_trace_copy_string(tmp_buf, ..., tmp_buf_end - tmp_buf);
tmp_buf += err;
```
If the string is truncated, `err` will be the size of the *entire* buffer ( `tmp_buf_end - tmp_buf` ). This seems safe because the code will advance the pointer all the way to the end of the buffer, which would get caught in a subsequent check at line 985 or 1007.
Returning to `snprintf`, the key problem with `snprintf` is that it returns the number of characters *that would have been* written, not the number that actually *were* written. If this return value is larger than the available space, incrementing `tmp_buf` based on this value can overflow the buffer.
Now, I'll search for `snprintf` calls across the whole `/kernel/bpf/` directory just to be absolutely sure that I did not miss anything.
|
| 501/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Args: map[Name:snprintf SourceTreePrefix:kernel/bpf/]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__func_get_name SourceFile:kernel/bpf/disasm.c SourceLine:36] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__func_get_name SourceFile:kernel/bpf/disasm.c SourceLine:38] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__func_imm_name SourceFile:kernel/bpf/disasm.c SourceLine:50] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_fmt_stack_mask SourceFile:kernel/bpf/backtrack.c SourceLine:239] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_prog_ksym_set_name SourceFile:kernel/bpf/core.c SourceLine:581] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_prog_ksym_set_name SourceFile:kernel/bpf/core.c SourceLine:589] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_prog_ksym_set_name SourceFile:kernel/bpf/core.c SourceLine:594] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_struct_ops_ksym_init SourceFile:kernel/bpf/bpf_struct_ops.c SourceLine:647] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_tramp_image_alloc SourceFile:kernel/bpf/trampoline.c SourceLine:592] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_check_iter_kfuncs SourceFile:kernel/bpf/btf.c SourceLine:8662] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_nested_type_is_trusted SourceFile:kernel/bpf/btf.c SourceLine:9592] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_show_name SourceFile:kernel/bpf/btf.c SourceLine:1215] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_show_name SourceFile:kernel/bpf/btf.c SourceLine:1218] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:find_kfunc_impl_proto SourceFile:kernel/bpf/verifier.c SourceLine:2912] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:fmt_instance SourceFile:kernel/bpf/liveness.c SourceLine:419] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:fmt_reg_mask SourceFile:kernel/bpf/backtrack.c SourceLine:220] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:fmt_subprog SourceFile:kernel/bpf/liveness.c SourceLine:412] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:format_callchain SourceFile:kernel/bpf/states.c SourceLine:144] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:format_callchain SourceFile:kernel/bpf/states.c SourceLine:148] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:format_callchain SourceFile:kernel/bpf/states.c SourceLine:151] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:reg_type_str SourceFile:kernel/bpf/log.c SourceLine:435] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:reg_type_str SourceFile:kernel/bpf/log.c SourceLine:445] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:543] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:545] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:547] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:549]] TruncatedOutput:false]
|
| 502/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 71581
output: 40
thoughts: 118
Thoughts: **Investigating `snprintf` Usage in BPF**
Okay, so the initial search was pretty straightforward. It looks like `snprintf` is solely employed within `bpf_bprintf_prepare` in `kernel/bpf/helpers.c`. But hold on... something feels incomplete. I need to be more thorough here.
The summary referenced `bpf_snprintf_btf`. Now that's intriguing. I have to find the associated function and definition. Let's delve deeper and track down `bpf_snprintf_btf_proto` and unravel what's going on with this entry. I want to see how this fits into the overall picture.
|
| 503/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf_proto SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 504/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 71904
output: 42
thoughts: 155
Thoughts: **Tracing the Elusive `bpf_snprintf_btf_proto` Definition**
Okay, I need to understand where this `bpf_snprintf_btf_proto` is defined. I remember seeing it in the `bpf_helpers.c` file; I saw the case statement:
```c
2225: case BPF_FUNC_snprintf_btf:
2226: return &bpf_snprintf_btf_proto;
```
Right, but that's just using the function. Then, I went back to the same file and remembered seeing the declaration on line 2147:
```c
2147: const struct bpf_func_proto bpf_snprintf_btf_proto __weak;
```
Aha! It's `__weak`. That means it's likely *defined* elsewhere, likely in a separate compilation unit. Given the context and the nature of BTF-related functions, my intuition tells me it's probably in `kernel/bpf/btf.c`. Time to verify that guess and look for the definition of `bpf_snprintf_btf` in that file.
|
| 505/6 |
2026/05/18 19:29 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf SourceTreePrefix:kernel/bpf/btf.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 506/6 |
2026/05/18 19:29 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 72384
output: 38
thoughts: 27
Thoughts: Okay, here's my take on those thoughts, framed as an expert's internal monologue:
**Investigating `bpf_snprintf_btf` Within the BPF Subsystem**
Alright, time to dive in. I need to understand how `bpf_snprintf_btf` is being used. Given that function name, it's almost certainly related to BPF (Berkeley Packet Filter) and probably BTF (BPF Type Format) as well. My initial hunch is that it's handling some sort of formatted output or debugging information specifically related to BTF data structures within the BPF subsystem.
Therefore, the most logical first step is a focused search. Let's start with a comprehensive search across the `kernel/bpf/` directory. This is the heart of the BPF implementation in the kernel, so this search should reveal how, and where, this particular function is getting called, and provide the usage context. I expect that will yield the vital clues needed to understand this function’s behaviour.
|
| 507/6 |
2026/05/18 19:30 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 508/6 |
2026/05/18 19:30 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 72686
output: 39
thoughts: 12
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:
**BPF Function Call Investigation**
Alright, so I'm thinking about this BPF thing. The question that has popped into my head is how this particular function is defined. My first instinct is to check if it's being declared or invoked via a `BPF_CALL`. That's where I'd start digging. Considering the context, it's pretty likely that's how this is set up, given that's the most common way to link eBPF programs, or to call another function from an eBPF program. It would be my starting point in any debugging or reverse-engineering efforts to understand this function's purpose or the data its handling.
|
| 509/6 |
2026/05/18 19:30 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:____bpf_snprintf_btf SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 510/6 |
2026/05/18 19:30 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 72926
output: 28
thoughts: 26
Thoughts: Okay, here's my interpretation of that thought process, as if it were my own:
**Investigating a Kernel BPF Function**
Alright, time to dive in. I need to understand this specific function, `bpf_snprintf_btf`. Seems like it's likely related to BPF and BTF, the core of efficient tracing and debugging. My first move will be to locate the definition of this function. Given the naming convention and its relationship to BPF, it's highly probable that it resides within the `kernel/bpf/` directory. So, let's start with a focused search within that area. A simple string search for "bpf_snprintf_btf" within `kernel/bpf/` should quickly point me to the relevant source file. From there, I can begin to unravel its functionality and its relationship to BTF.
|
| 511/6 |
2026/05/18 19:30 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf_proto]
Results: map[References:<nil> TruncatedOutput:false]
|
| 512/6 |
2026/05/18 19:30 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 73186
output: 24
thoughts: 75
Thoughts: **Navigating the BTF Landscape**
Okay, so I need to understand the usage of a specific symbol within `btf.c`. My initial instinct is to leverage `grep` through `codesearch-find-references` – a tried and true method for quickly pinpointing occurrences. However, a different approach occurs to me – a potentially faster route. I think, if I'm already pretty sure about the file I can skip the `find-references` step. I can probably use `codesearch-dir-index` followed by `read-file` directly, potentially skipping the wider search and getting to the heart of the matter quicker. Given my familiarity with the kernel structure, I suspect the target file is `kernel/bpf/btf.c`. Let's see if this hunch is correct and save some processing time.
|
| 513/6 |
2026/05/18 19:30 |
tool |
read-file |
0m
Args: map[File:kernel/bpf/btf.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2018 Facebook */
#include <uapi/linux/btf.h>
#include <uapi/linux/bpf.h>
#include <uapi/linux/bpf_perf_event.h>
#include <uapi/linux/types.h>
#include <linux/seq_file.h>
#include <linux/compiler.h>
#include <linux/ctype.h>
#include <linux/errno.h>
#include <linux/slab.h>
#include <linux/anon_inodes.h>
#include <linux/file.h>
#include <linux/uaccess.h>
#include <linux/kernel.h>
#include <linux/idr.h>
#include <linux/sort.h>
#include <linux/bpf_verifier.h>
#include <linux/btf.h>
#include <linux/btf_ids.h>
#include <linux/bpf.h>
#include <linux/bpf_lsm.h>
#include <linux/skmsg.h>
#include <linux/perf_event.h>
#include <linux/bsearch.h>
#include <linux/kobject.h>
#include <linux/string.h>
#include <linux/sysfs.h>
#include <linux/overflow.h>
#include <net/netfilter/nf_bpf_link.h>
#include <net/sock.h>
#include <net/xdp.h>
#include "../tools/lib/bpf/relo_core.h"
/* BTF (BPF Type Format) is the meta data format which describes
* the data types of BPF program/map. Hence, it basically focus
* on the C programming language which the modern BPF is primary
* using.
*
* ELF Section:
* ~~~~~~~~~~~
* The BTF data is stored under the ".BTF" ELF section
*
* struct btf_type:
* ~~~~~~~~~~~~~~~
* Each 'struct btf_type' object describes a C data type.
* Depending on the type it is describing, a 'struct btf_type'
* object may be followed by more data. F.e.
* To describe an array, 'struct btf_type' is followed by
* 'struct btf_array'.
*
* 'struct btf_type' and any extra data following it are
* 4 bytes aligned.
*
* Type section:
* ~~~~~~~~~~~~~
* The BTF type section contains a list of 'struct btf_type' objects.
* Each one describes a C type. Recall from the above section
* that a 'struct btf_type' object could be immediately followed by extra
* data in order to describe some particular C types.
*
* type_id:
* ~~~~~~~
* Each btf_type object is identified by a type_id. The type_id
* is implicitly implied by the location of the btf_type object in
* the BTF type section. The first one has type_id 1. The second
* one has type_id 2...etc. Hence, an earlier btf_type has
* a smaller type_id.
*
* A btf_type object may refer to another btf_type object by using
* type_id (i.e. the "type" in the "struct btf_type").
*
* NOTE that we cannot assume any reference-order.
* A btf_type object can refer to an earlier btf_type object
* but it can also refer to a later btf_type object.
*
* For example, to describe "const void *". A btf_type
* object describing "const" may refer to another btf_type
* object describing "void *". This type-reference is done
* by specifying type_id:
*
* [1] CONST (anon) type_id=2
* [2] PTR (anon) type_id=0
*
* The above is the btf_verifier debug log:
* - Each line started with "[?]" is a btf_type object
* - [?] is the type_id of the btf_type object.
* - CONST/PTR is the BTF_KIND_XXX
* - "(anon)" is the name of the type. It just
* happens that CONST and PTR has no name.
* - type_id=XXX is the 'u32 type' in btf_type
*
* NOTE: "void" has type_id 0
*
* String section:
* ~~~~~~~~~~~~~~
* The BTF string section contains the names used by the type section.
* Each string is referred by an "offset" from the beginning of the
* string section.
*
* Each string is '\0' terminated.
*
* The first character in the string section must be '\0'
* which is used to mean 'anonymous'. Some btf_type may not
* have a name.
*/
/* BTF verification:
*
* To verify BTF data, two passes are needed.
*
* Pass #1
* ~~~~~~~
* The first pass is to collect all btf_type objects to
* an array: "btf->types".
*
* Depending on the C type that a btf_type is describing,
* a btf_type may be followed by extra data. We don't know
* how many btf_type is there, and more importantly we don't
* know where each btf_type is located in the type section.
*
* Without knowing the location of each type_id, most verifications
* cannot be done. e.g. an earlier btf_type may refer to a later
* btf_type (recall the "const void *" above), so we cannot
* check this type-reference in the first pass.
*
* In the first pass, it still does some verifications (e.g.
* checking the name is a valid offset to the string section).
*
* Pass #2
* ~~~~~~~
* The main focus is to resolve a btf_type that is referring
* to another type.
*
* We have to ensure the referring type:
* 1) does exist in the BTF (i.e. in btf->types[])
* 2) does not cause a loop:
* struct A {
* struct B b;
* };
*
* struct B {
* struct A a;
* };
*
* btf_type_needs_resolve() decides if a btf_type needs
* to be resolved.
*
* The needs_resolve type implements the "resolve()" ops which
* essentially does a DFS and detects backedge.
*
* During resolve (or DFS), different C types have different
* "RESOLVED" conditions.
*
* When resolving a BTF_KIND_STRUCT, we need to resolve all its
* members because a member is always referring to another
* type. A struct's member can be treated as "RESOLVED" if
* it is referring to a BTF_KIND_PTR. Otherwise, the
* following valid C struct would be rejected:
*
* struct A {
* int m;
* struct A *a;
* };
*
* When resolving a BTF_KIND_PTR, it needs to keep resolving if
* it is referring to another BTF_KIND_PTR. Otherwise, we cannot
* detect a pointer loop, e.g.:
* BTF_KIND_CONST -> BTF_KIND_PTR -> BTF_KIND_CONST -> BTF_KIND_PTR +
* ^ |
* +-----------------------------------------+
*
*/
#define BITS_PER_U128 (sizeof(u64) * BITS_PER_BYTE * 2)
#define BITS_PER_BYTE_MASK (BITS_PER_BYTE - 1)
#define BITS_PER_BYTE_MASKED(bits) ((bits) & BITS_PER_BYTE_MASK)
#define BITS_ROUNDDOWN_BYTES(bits) ((bits) >> 3)
#define BITS_ROUNDUP_BYTES(bits) \
(BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits))
#define BTF_INFO_MASK 0x9f00ffff
#define BTF_INT_MASK 0x0fffffff
#define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE)
#define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET)
/* 16MB for 64k structs and each has 16 members and
* a few MB spaces for the string section.
* The hard limit is S32_MAX.
*/
#define BTF_MAX_SIZE (16 * 1024 * 1024)
#define for_each_member_from(i, from, struct_type, member) \
for (i = from, member = btf_type_member(struct_type) + from; \
i < btf_type_vlen(struct_type); \
i++, member++)
#define for_each_vsi_from(i, from, struct_type, member) \
for (i = from, member = btf_type_var_secinfo(struct_type) + from; \
i < btf_type_vlen(struct_type); \
i++, member++)
DEFINE_IDR(btf_idr);
DEFINE_SPINLOCK(btf_idr_lock);
enum btf_kfunc_hook {
BTF_KFUNC_HOOK_COMMON,
BTF_KFUNC_HOOK_XDP,
BTF_KFUNC_HOOK_TC,
BTF_KFUNC_HOOK_STRUCT_OPS,
BTF_KFUNC_HOOK_TRACING,
BTF_KFUNC_HOOK_SYSCALL,
BTF_KFUNC_HOOK_FMODRET,
BTF_KFUNC_HOOK_CGROUP,
BTF_KFUNC_HOOK_SCHED_ACT,
BTF_KFUNC_HOOK_SK_SKB,
BTF_KFUNC_HOOK_SOCKET_FILTER,
BTF_KFUNC_HOOK_LWT,
BTF_KFUNC_HOOK_NETFILTER,
BTF_KFUNC_HOOK_KPROBE,
BTF_KFUNC_HOOK_MAX,
};
enum {
BTF_KFUNC_SET_MAX_CNT = 256,
BTF_DTOR_KFUNC_MAX_CNT = 256,
BTF_KFUNC_FILTER_MAX_CNT = 16,
};
struct btf_kfunc_hook_filter {
btf_kfunc_filter_t filters[BTF_KFUNC_FILTER_MAX_CNT];
u32 nr_filters;
};
struct btf_kfunc_set_tab {
struct btf_id_set8 *sets[BTF_KFUNC_HOOK_MAX];
struct btf_kfunc_hook_filter hook_filters[BTF_KFUNC_HOOK_MAX];
};
struct btf_id_dtor_kfunc_tab {
u32 cnt;
struct btf_id_dtor_kfunc dtors[];
};
struct btf_struct_ops_tab {
u32 cnt;
u32 capacity;
struct bpf_struct_ops_desc ops[];
};
struct btf {
void *data;
struct btf_type **types;
u32 *resolved_ids;
u32 *resolved_sizes;
const char *strings;
void *nohdr_data;
struct btf_header hdr;
u32 nr_types; /* includes VOID for base BTF */
u32 named_start_id;
u32 types_size;
u32 data_size;
refcount_t refcnt;
u32 id;
struct rcu_head rcu;
struct btf_kfunc_set_tab *kfunc_set_tab;
struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab;
struct btf_struct_metas *struct_meta_tab;
struct btf_struct_ops_tab *struct_ops_tab;
struct btf_layout *layout;
/* split BTF support */
struct btf *base_btf;
u32 start_id; /* first type ID in this BTF (0 for base BTF) */
u32 start_str_off; /* first string offset (0 for base BTF) */
char name[MODULE_NAME_LEN];
bool kernel_btf;
__u32 *base_id_map; /* map from distilled base BTF -> vmlinux BTF ids */
};
enum verifier_phase {
CHECK_META,
CHECK_TYPE,
};
struct resolve_vertex {
const struct btf_type *t;
u32 type_id;
u16 next_member;
};
enum visit_state {
NOT_VISITED,
VISITED,
RESOLVED,
};
enum resolve_mode {
RESOLVE_TBD, /* To Be Determined */
RESOLVE_PTR, /* Resolving for Pointer */
RESOLVE_STRUCT_OR_ARRAY, /* Resolving for struct/union
* or array
*/
};
#define MAX_RESOLVE_DEPTH 32
struct btf_sec_info {
u32 off;
u32 len;
};
struct btf_verifier_env {
struct btf *btf;
u8 *visit_states;
struct resolve_vertex stack[MAX_RESOLVE_DEPTH];
struct bpf_verifier_log log;
u32 log_type_id;
u32 top_stack;
enum verifier_phase phase;
enum resolve_mode resolve_mode;
};
static const char * const btf_kind_str[NR_BTF_KINDS] = {
[BTF_KIND_UNKN] = "UNKNOWN",
[BTF_KIND_INT] = "INT",
[BTF_KIND_PTR] = "PTR",
[BTF_KIND_ARRAY] = "ARRAY",
[BTF_KIND_STRUCT] = "STRUCT",
[BTF_KIND_UNION] = "UNION",
[BTF_KIND_ENUM] = "ENUM",
[BTF_KIND_FWD] = "FWD",
[BTF_KIND_TYPEDEF] = "TYPEDEF",
[BTF_KIND_VOLATILE] = "VOLATILE",
[BTF_KIND_CONST] = "CONST",
[BTF_KIND_RESTRICT] = "RESTRICT",
[BTF_KIND_FUNC] = "FUNC",
[BTF_KIND_FUNC_PROTO] = "FUNC_PROTO",
[BTF_KIND_VAR] = "VAR",
[BTF_KIND_DATASEC] = "DATASEC",
[BTF_KIND_FLOAT] = "FLOAT",
[BTF_KIND_DECL_TAG] = "DECL_TAG",
[BTF_KIND_TYPE_TAG] = "TYPE_TAG",
[BTF_KIND_ENUM64] = "ENUM64",
};
const char *btf_type_str(const struct btf_type *t)
{
return btf_kind_str[BTF_INFO_KIND(t->info)];
}
/* Chunk size we use in safe copy of data to be shown. */
#define BTF_SHOW_OBJ_SAFE_SIZE 32
/*
* This is the maximum size of a base type value (equivalent to a
* 128-bit int); if we are at the end of our safe buffer and have
* less than 16 bytes space we can't be assured of being able
* to copy the next type safely, so in such cases we will initiate
* a new copy.
*/
#define BTF_SHOW_OBJ_BASE_TYPE_SIZE 16
/* Type name size */
#define BTF_SHOW_NAME_SIZE 80
/*
* The suffix of a type that indicates it cannot alias another type when
* comparing BTF IDs for kfunc invocations.
*/
#define NOCAST_ALIAS_SUFFIX "___init"
/*
* Common data to all BTF show operations. Private show functions can add
* their own data to a structure containing a struct btf_show and consult it
* in the show callback. See btf_type_show() below.
*
* One challenge with showing nested data is we want to skip 0-valued
* data, but in order to figure out whether a nested object is all zeros
* we need to walk through it. As a result, we need to make two passes
* when handling structs, unions and arrays; the first path simply looks
* for nonzero data, while the second actually does the display. The first
* pass is signalled by show->state.depth_check being set, and if we
* encounter a non-zero value we set show->state.depth_to_show to
* the depth at which we encountered it. When we have completed the
* first pass, we will know if anything needs to be displayed if
* depth_to_show > depth. See btf_[struct,array]_show() for the
* implementation of this.
*
* Another problem is we want to ensure the data for display is safe to
* access. To support this, the anonymous "struct {} obj" tracks the data
* object and our safe copy of it. We copy portions of the data needed
* to the object "copy" buffer, but because its size is limited to
* BTF_SHOW_OBJ_COPY_LEN bytes, multiple copies may be required as we
* traverse larger objects for display.
*
* The various data type show functions all start with a call to
* btf_show_start_type() which returns a pointer to the safe copy
* of the data needed (or if BTF_SHOW_UNSAFE is specified, to the
* raw data itself). btf_show_obj_safe() is responsible for
* using copy_from_kernel_nofault() to update the safe data if necessary
* as we traverse the object's data. skbuff-like semantics are
* used:
*
* - obj.head points to the start of the toplevel object for display
* - obj.size is the size of the toplevel object
* - obj.data points to the current point in the original data at
* which our safe data starts. obj.data will advance as we copy
* portions of the data.
*
* In most cases a single copy will suffice, but larger data structures
* such as "struct task_struct" will require many copies. The logic in
* btf_show_obj_safe() handles the logic that determines if a new
* copy_from_kernel_nofault() is needed.
*/
struct btf_show {
u64 flags;
void *target; /* target of show operation (seq file, buffer) */
__printf(2, 0) void (*showfn)(struct btf_show *show, const char *fmt, va_list args);
const struct btf *btf;
/* below are used during iteration */
struct {
u8 depth;
u8 depth_to_show;
u8 depth_check;
u8 array_member:1,
array_terminated:1;
u16 array_encoding;
u32 type_id;
int status; /* non-zero for error */
const struct btf_type *type;
const struct btf_member *member;
char name[BTF_SHOW_NAME_SIZE]; /* space for member name/type */
} state;
struct {
u32 size;
void *head;
void *data;
u8 safe[BTF_SHOW_OBJ_SAFE_SIZE];
} obj;
};
struct btf_kind_operations {
s32 (*check_meta)(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left);
int (*resolve)(struct btf_verifier_env *env,
const struct resolve_vertex *v);
int (*check_member)(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type);
int (*check_kflag_member)(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type);
void (*log_details)(struct btf_verifier_env *env,
const struct btf_type *t);
void (*show)(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offsets,
struct btf_show *show);
};
static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS];
static struct btf_type btf_void;
static int btf_resolve(struct btf_verifier_env *env,
const struct btf_type *t, u32 type_id);
static int btf_func_check(struct btf_verifier_env *env,
const struct btf_type *t);
static bool btf_type_is_modifier(const struct btf_type *t)
{
/* Some of them is not strictly a C modifier
* but they are grouped into the same bucket
* for BTF concern:
* A type (t) that refers to another
* type through t->type AND its size cannot
* be determined without following the t->type.
*
* ptr does not fall into this bucket
* because its size is always sizeof(void *).
*/
switch (BTF_INFO_KIND(t->info)) {
case BTF_KIND_TYPEDEF:
case BTF_KIND_VOLATILE:
case BTF_KIND_CONST:
case BTF_KIND_RESTRICT:
case BTF_KIND_TYPE_TAG:
return true;
}
return false;
}
static int btf_start_id(const struct btf *btf)
{
return btf->start_id + (btf->base_btf ? 0 : 1);
}
bool btf_type_is_void(const struct btf_type *t)
{
return t == &btf_void;
}
static bool btf_type_is_datasec(const struct btf_type *t)
{
return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC;
}
static bool btf_type_is_decl_tag(const struct btf_type *t)
{
return BTF_INFO_KIND(t->info) == BTF_KIND_DECL_TAG;
}
static bool btf_type_nosize(const struct btf_type *t)
{
return btf_type_is_void(t) || btf_type_is_fwd(t) ||
btf_type_is_func(t) || btf_type_is_func_proto(t) ||
btf_type_is_decl_tag(t);
}
static bool btf_type_nosize_or_null(const struct btf_type *t)
{
return !t || btf_type_nosize(t);
}
static bool btf_type_is_decl_tag_target(const struct btf_type *t)
{
return btf_type_is_func(t) || btf_type_is_struct(t) ||
btf_type_is_var(t) || btf_type_is_typedef(t);
}
bool btf_is_vmlinux(const struct btf *btf)
{
return btf->kernel_btf && !btf->base_btf;
}
u32 btf_nr_types(const struct btf *btf)
{
u32 total = 0;
while (btf) {
total += btf->nr_types;
btf = btf->base_btf;
}
return total;
}
/*
* Note that vmlinux and kernel module BTFs are always sorted
* during the building phase.
*/
static void btf_check_sorted(struct btf *btf)
{
u32 i, n, named_start_id = 0;
n = btf_nr_types(btf);
if (btf_is_vmlinux(btf)) {
for (i = btf_start_id(btf); i < n; i++) {
const struct btf_type *t = btf_type_by_id(btf, i);
const char *n = btf_name_by_offset(btf, t->name_off);
if (n[0] != '\0') {
btf->named_start_id = i;
return;
}
}
return;
}
for (i = btf_start_id(btf) + 1; i < n; i++) {
const struct btf_type *ta = btf_type_by_id(btf, i - 1);
const struct btf_type *tb = btf_type_by_id(btf, i);
const char *na = btf_name_by_offset(btf, ta->name_off);
const char *nb = btf_name_by_offset(btf, tb->name_off);
if (strcmp(na, nb) > 0)
return;
if (named_start_id == 0 && na[0] != '\0')
named_start_id = i - 1;
if (named_start_id == 0 && nb[0] != '\0')
named_start_id = i;
}
if (named_start_id)
btf->named_start_id = named_start_id;
}
/*
* btf_named_start_id - Get the named starting ID for the BTF
* @btf: Pointer to the target BTF object
* @own: Flag indicating whether to query only the current BTF (true = current BTF only,
* false = recursively traverse the base BTF chain)
*
* Return value rules:
* 1. For a sorted btf, return its named_start_id
* 2. Else for a split BTF, return its start_id
* 3. Else for a base BTF, return 1
*/
u32 btf_named_start_id(const struct btf *btf, bool own)
{
const struct btf *base_btf = btf;
while (!own && base_btf->base_btf)
base_btf = base_btf->base_btf;
return base_btf->named_start_id ?: (base_btf->start_id ?: 1);
}
static s32 btf_find_by_name_kind_bsearch(const struct btf *btf, const char *name)
{
const struct btf_type *t;
const char *tname;
s32 l, r, m;
l = btf_named_start_id(btf, true);
r = btf_nr_types(btf) - 1;
while (l <= r) {
m = l + (r - l) / 2;
t = btf_type_by_id(btf, m);
tname = btf_name_by_offset(btf, t->name_off);
if (strcmp(tname, name) >= 0) {
if (l == r)
return r;
r = m;
} else {
l = m + 1;
}
}
return btf_nr_types(btf);
}
s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind)
{
const struct btf *base_btf = btf_base_btf(btf);
const struct btf_type *t;
const char *tname;
s32 id, total;
if (base_btf) {
id = btf_find_by_name_kind(base_btf, name, kind);
if (id > 0)
return id;
}
total = btf_nr_types(btf);
if (btf->named_start_id > 0 && name[0]) {
id = btf_find_by_name_kind_bsearch(btf, name);
for (; id < total; id++) {
t = btf_type_by_id(btf, id);
tname = btf_name_by_offset(btf, t->name_off);
if (strcmp(tname, name) != 0)
return -ENOENT;
if (BTF_INFO_KIND(t->info) == kind)
return id;
}
} else {
for (id = btf_start_id(btf); id < total; id++) {
t = btf_type_by_id(btf, id);
if (BTF_INFO_KIND(t->info) != kind)
continue;
tname = btf_name_by_offset(btf, t->name_off);
if (strcmp(tname, name) == 0)
return id;
}
}
return -ENOENT;
}
s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p)
{
struct btf *btf;
s32 ret;
int id;
btf = bpf_get_btf_vmlinux();
if (IS_ERR(btf))
return PTR_ERR(btf);
if (!btf)
return -EINVAL;
ret = btf_find_by_name_kind(btf, name, kind);
/* ret is never zero, since btf_find_by_name_kind returns
* positive btf_id or negative error.
*/
if (ret > 0) {
btf_get(btf);
*btf_p = btf;
return ret;
}
/* If name is not found in vmlinux's BTF then search in module's BTFs */
spin_lock_bh(&btf_idr_lock);
idr_for_each_entry(&btf_idr, btf, id) {
if (!btf_is_module(btf))
continue;
/* linear search could be slow hence unlock/lock
* the IDR to avoiding holding it for too long
*/
btf_get(btf);
spin_unlock_bh(&btf_idr_lock);
ret = btf_find_by_name_kind(btf, name, kind);
if (ret > 0) {
*btf_p = btf;
return ret;
}
btf_put(btf);
spin_lock_bh(&btf_idr_lock);
}
spin_unlock_bh(&btf_idr_lock);
return ret;
}
EXPORT_SYMBOL_GPL(bpf_find_btf_id);
const struct btf_type *btf_type_skip_modifiers(const struct btf *btf,
u32 id, u32 *res_id)
{
const struct btf_type *t = btf_type_by_id(btf, id);
while (btf_type_is_modifier(t)) {
id = t->type;
t = btf_type_by_id(btf, t->type);
}
if (res_id)
*res_id = id;
return t;
}
const struct btf_type *btf_type_resolve_ptr(const struct btf *btf,
u32 id, u32 *res_id)
{
const struct btf_type *t;
t = btf_type_skip_modifiers(btf, id, NULL);
if (!btf_type_is_ptr(t))
return NULL;
return btf_type_skip_modifiers(btf, t->type, res_id);
}
const struct btf_type *btf_type_resolve_func_ptr(const struct btf *btf,
u32 id, u32 *res_id)
{
const struct btf_type *ptype;
ptype = btf_type_resolve_ptr(btf, id, res_id);
if (ptype && btf_type_is_func_proto(ptype))
return ptype;
return NULL;
}
/* Types that act only as a source, not sink or intermediate
* type when resolving.
*/
static bool btf_type_is_resolve_source_only(const struct btf_type *t)
{
return btf_type_is_var(t) ||
btf_type_is_decl_tag(t) ||
btf_type_is_datasec(t);
}
/* What types need to be resolved?
*
* btf_type_is_modifier() is an obvious one.
*
* btf_type_is_struct() because its member refers to
* another type (through member->type).
*
* btf_type_is_var() because the variable refers to
* another type. btf_type_is_datasec() holds multiple
* btf_type_is_var() types that need resolving.
*
* btf_type_is_array() because its element (array->type)
* refers to another type. Array can be thought of a
* special case of struct while array just has the same
* member-type repeated by array->nelems of times.
*/
static bool btf_type_needs_resolve(const struct btf_type *t)
{
return btf_type_is_modifier(t) ||
btf_type_is_ptr(t) ||
btf_type_is_struct(t) ||
btf_type_is_array(t) ||
btf_type_is_var(t) ||
btf_type_is_func(t) ||
btf_type_is_decl_tag(t) ||
btf_type_is_datasec(t);
}
/* t->size can be used */
static bool btf_type_has_size(const struct btf_type *t)
{
switch (BTF_INFO_KIND(t->info)) {
case BTF_KIND_INT:
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
case BTF_KIND_ENUM:
case BTF_KIND_DATASEC:
case BTF_KIND_FLOAT:
case BTF_KIND_ENUM64:
return true;
}
return false;
}
static const char *btf_int_encoding_str(u8 encoding)
{
if (encoding == 0)
return "(none)";
else if (encoding == BTF_INT_SIGNED)
return "SIGNED";
else if (encoding == BTF_INT_CHAR)
return "CHAR";
else if (encoding == BTF_INT_BOOL)
return "BOOL";
else
return "UNKN";
}
static u32 btf_type_int(const struct btf_type *t)
{
return *(u32 *)(t + 1);
}
static const struct btf_array *btf_type_array(const struct btf_type *t)
{
return (const struct btf_array *)(t + 1);
}
static const struct btf_enum *btf_type_enum(const struct btf_type *t)
{
return (const struct btf_enum *)(t + 1);
}
static const struct btf_var *btf_type_var(const struct btf_type *t)
{
return (const struct btf_var *)(t + 1);
}
static const struct btf_decl_tag *btf_type_decl_tag(const struct btf_type *t)
{
return (const struct btf_decl_tag *)(t + 1);
}
static const struct btf_enum64 *btf_type_enum64(const struct btf_type *t)
{
return (const struct btf_enum64 *)(t + 1);
}
static const struct btf_kind_operations *btf_type_ops(const struct btf_type *t)
{
return kind_ops[BTF_INFO_KIND(t->info)];
}
static bool btf_name_offset_valid(const struct btf *btf, u32 offset)
{
if (!BTF_STR_OFFSET_VALID(offset))
return false;
while (offset < btf->start_str_off)
btf = btf->base_btf;
offset -= btf->start_str_off;
return offset < btf->hdr.str_len;
}
static bool __btf_name_char_ok(char c, bool first)
{
if ((first ? !isalpha(c) :
!isalnum(c)) &&
c != '_' &&
c != '.')
return false;
return true;
}
const char *btf_str_by_offset(const struct btf *btf, u32 offset)
{
while (offset < btf->start_str_off)
btf = btf->base_btf;
offset -= btf->start_str_off;
if (offset < btf->hdr.str_len)
return &btf->strings[offset];
return NULL;
}
static bool btf_name_valid_identifier(const struct btf *btf, u32 offset)
{
/* offset must be valid */
const char *src = btf_str_by_offset(btf, offset);
const char *src_limit;
if (!__btf_name_char_ok(*src, true))
return false;
/* set a limit on identifier length */
src_limit = src + KSYM_NAME_LEN;
src++;
while (*src && src < src_limit) {
if (!__btf_name_char_ok(*src, false))
return false;
src++;
}
return !*src;
}
/* Allow any printable character in DATASEC names */
static bool btf_name_valid_section(const struct btf *btf, u32 offset)
{
/* offset must be valid */
const char *src = btf_str_by_offset(btf, offset);
const char *src_limit;
if (!*src)
return false;
/* set a limit on identifier length */
src_limit = src + KSYM_NAME_LEN;
while (*src && src < src_limit) {
if (!isprint(*src))
return false;
src++;
}
return !*src;
}
static const char *__btf_name_by_offset(const struct btf *btf, u32 offset)
{
const char *name;
if (!offset)
return "(anon)";
name = btf_str_by_offset(btf, offset);
return name ?: "(invalid-name-offset)";
}
const char *btf_name_by_offset(const struct btf *btf, u32 offset)
{
return btf_str_by_offset(btf, offset);
}
const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id)
{
while (type_id < btf->start_id)
btf = btf->base_btf;
type_id -= btf->start_id;
if (type_id >= btf->nr_types)
return NULL;
return btf->types[type_id];
}
EXPORT_SYMBOL_GPL(btf_type_by_id);
/*
* Check that the type @t is a regular int. This means that @t is not
* a bit field and it has the same size as either of u8/u16/u32/u64
* or __int128. If @expected_size is not zero, then size of @t should
* be the same. A caller should already have checked that the type @t
* is an integer.
*/
static bool __btf_type_int_is_regular(const struct btf_type *t, size_t expected_size)
{
u32 int_data = btf_type_int(t);
u8 nr_bits = BTF_INT_BITS(int_data);
u8 nr_bytes = BITS_ROUNDUP_BYTES(nr_bits);
return BITS_PER_BYTE_MASKED(nr_bits) == 0 &&
BTF_INT_OFFSET(int_data) == 0 &&
(nr_bytes <= 16 && is_power_of_2(nr_bytes)) &&
(expected_size == 0 || nr_bytes == expected_size);
}
static bool btf_type_int_is_regular(const struct btf_type *t)
{
return __btf_type_int_is_regular(t, 0);
}
bool btf_type_is_i32(const struct btf_type *t)
{
return btf_type_is_int(t) && __btf_type_int_is_regular(t, 4);
}
bool btf_type_is_i64(const struct btf_type *t)
{
return btf_type_is_int(t) && __btf_type_int_is_regular(t, 8);
}
bool btf_type_is_primitive(const struct btf_type *t)
{
return (btf_type_is_int(t) && btf_type_int_is_regular(t)) ||
btf_is_any_enum(t);
}
/*
* Check that given struct member is a regular int with expected
* offset and size.
*/
bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
const struct btf_member *m,
u32 expected_offset, u32 expected_size)
{
const struct btf_type *t;
u32 id, int_data;
u8 nr_bits;
id = m->type;
t = btf_type_id_size(btf, &id, NULL);
if (!t || !btf_type_is_int(t))
return false;
int_data = btf_type_int(t);
nr_bits = BTF_INT_BITS(int_data);
if (btf_type_kflag(s)) {
u32 bitfield_size = BTF_MEMBER_BITFIELD_SIZE(m->offset);
u32 bit_offset = BTF_MEMBER_BIT_OFFSET(m->offset);
/* if kflag set, int should be a regular int and
* bit offset should be at byte boundary.
*/
return !bitfield_size &&
BITS_ROUNDUP_BYTES(bit_offset) == expected_offset &&
BITS_ROUNDUP_BYTES(nr_bits) == expected_size;
}
if (BTF_INT_OFFSET(int_data) ||
BITS_PER_BYTE_MASKED(m->offset) ||
BITS_ROUNDUP_BYTES(m->offset) != expected_offset ||
BITS_PER_BYTE_MASKED(nr_bits) ||
BITS_ROUNDUP_BYTES(nr_bits) != expected_size)
return false;
return true;
}
/* Similar to btf_type_skip_modifiers() but does not skip typedefs. */
static const struct btf_type *btf_type_skip_qualifiers(const struct btf *btf,
u32 id)
{
const struct btf_type *t = btf_type_by_id(btf, id);
while (btf_type_is_modifier(t) &&
BTF_INFO_KIND(t->info) != BTF_KIND_TYPEDEF) {
t = btf_type_by_id(btf, t->type);
}
return t;
}
#define BTF_SHOW_MAX_ITER 10
#define BTF_KIND_BIT(kind) (1ULL << kind)
/*
* Populate show->state.name with type name information.
* Format of type name is
*
* [.member_name = ] (type_name)
*/
static const char *btf_show_name(struct btf_show *show)
{
/* BTF_MAX_ITER array suffixes "[]" */
const char *array_suffixes = "[][][][][][][][][][]";
const char *array_suffix = &array_suffixes[strlen(array_suffixes)];
/* BTF_MAX_ITER pointer suffixes "*" */
const char *ptr_suffixes = "**********";
const char *ptr_suffix = &ptr_suffixes[strlen(ptr_suffixes)];
const char *name = NULL, *prefix = "", *parens = "";
const struct btf_member *m = show->state.member;
const struct btf_type *t;
const struct btf_array *array;
u32 id = show->state.type_id;
const char *member = NULL;
bool show_member = false;
u64 kinds = 0;
int i;
show->state.name[0] = '\0';
/*
* Don't show type name if we're showing an array member;
* in that case we show the array type so don't need to repeat
* ourselves for each member.
*/
if (show->state.array_member)
return "";
/* Retrieve member name, if any. */
if (m) {
member = btf_name_by_offset(show->btf, m->name_off);
show_member = strlen(member) > 0;
id = m->type;
}
/*
* Start with type_id, as we have resolved the struct btf_type *
* via btf_modifier_show() past the parent typedef to the child
* struct, int etc it is defined as. In such cases, the type_id
* still represents the starting type while the struct btf_type *
* in our show->state points at the resolved type of the typedef.
*/
t = btf_type_by_id(show->btf, id);
if (!t)
return "";
/*
* The goal here is to build up the right number of pointer and
* array suffixes while ensuring the type name for a typedef
* is represented. Along the way we accumulate a list of
* BTF kinds we have encountered, since these will inform later
* display; for example, pointer types will not require an
* opening "{" for struct, we will just display the pointer value.
*
* We also want to accumulate the right number of pointer or array
* indices in the format string while iterating until we get to
* the typedef/pointee/array member target type.
*
* We start by pointing at the end of pointer and array suffix
* strings; as we accumulate pointers and arrays we move the pointer
* or array string backwards so it will show the expected number of
* '*' or '[]' for the type. BTF_SHOW_MAX_ITER of nesting of pointers
* and/or arrays and typedefs are supported as a precaution.
*
* We also want to get typedef name while proceeding to resolve
* type it points to so that we can add parentheses if it is a
* "typedef struct" etc.
*/
for (i = 0; i < BTF_SHOW_MAX_ITER; i++) {
switch (BTF_INFO_KIND(t->info)) {
case BTF_KIND_TYPEDEF:
if (!name)
name = btf_name_by_offset(show->btf,
t->name_off);
kinds |= BTF_KIND_BIT(BTF_KIND_TYPEDEF);
id = t->type;
break;
case BTF_KIND_ARRAY:
kinds |= BTF_KIND_BIT(BTF_KIND_ARRAY);
parens = "[";
if (!t)
return "";
array = btf_type_array(t);
if (array_suffix > array_suffixes)
array_suffix -= 2;
id = array->type;
break;
case BTF_KIND_PTR:
kinds |= BTF_KIND_BIT(BTF_KIND_PTR);
if (ptr_suffix > ptr_suffixes)
ptr_suffix -= 1;
id = t->type;
break;
default:
id = 0;
break;
}
if (!id)
break;
t = btf_type_skip_qualifiers(show->btf, id);
}
/* We may not be able to represent this type; bail to be safe */
if (i == BTF_SHOW_MAX_ITER)
return "";
if (!name)
name = btf_name_by_offset(show->btf, t->name_off);
switch (BTF_INFO_KIND(t->info)) {
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
prefix = BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT ?
"struct" : "union";
/* if it's an array of struct/union, parens is already set */
if (!(kinds & (BTF_KIND_BIT(BTF_KIND_ARRAY))))
parens = "{";
break;
case BTF_KIND_ENUM:
case BTF_KIND_ENUM64:
prefix = "enum";
break;
default:
break;
}
/* pointer does not require parens */
if (kinds & BTF_KIND_BIT(BTF_KIND_PTR))
parens = "";
/* typedef does not require struct/union/enum prefix */
if (kinds & BTF_KIND_BIT(BTF_KIND_TYPEDEF))
prefix = "";
if (!name)
name = "";
/* Even if we don't want type name info, we want parentheses etc */
if (show->flags & BTF_SHOW_NONAME)
snprintf(show->state.name, sizeof(show->state.name), "%s",
parens);
else
snprintf(show->state.name, sizeof(show->state.name),
"%s%s%s(%s%s%s%s%s%s)%s",
/* first 3 strings comprise ".member = " */
show_member ? "." : "",
show_member ? member : "",
show_member ? " = " : "",
/* ...next is our prefix (struct, enum, etc) */
prefix,
strlen(prefix) > 0 && strlen(name) > 0 ? " " : "",
/* ...this is the type name itself */
name,
/* ...suffixed by the appropriate '*', '[]' suffixes */
strlen(ptr_suffix) > 0 ? " " : "", ptr_suffix,
array_suffix, parens);
return show->state.name;
}
static const char *__btf_show_indent(struct btf_show *show)
{
const char *indents = " ";
const char *indent = &indents[strlen(indents)];
if ((indent - show->state.depth) >= indents)
return indent - show->state.depth;
return indents;
}
static const char *btf_show_indent(struct btf_show *show)
{
return show->flags & BTF_SHOW_COMPACT ? "" : __btf_show_indent(show);
}
static const char *btf_show_newline(struct btf_show *show)
{
return show->flags & BTF_SHOW_COMPACT ? "" : "\n";
}
static const char *btf_show_delim(struct btf_show *show)
{
if (show->state.depth == 0)
return "";
if ((show->flags & BTF_SHOW_COMPACT) && show->state.type &&
BTF_INFO_KIND(show->state.type->info) == BTF_KIND_UNION)
return "|";
return ",";
}
__printf(2, 3) static void btf_show(struct btf_show *show, const char *fmt, ...)
{
va_list args;
if (!show->state.depth_check) {
va_start(args, fmt);
show->showfn(show, fmt, args);
va_end(args);
}
}
/* Macros are used here as btf_show_type_value[s]() prepends and appends
* format specifiers to the format specifier passed in; these do the work of
* adding indentation, delimiters etc while the caller simply has to specify
* the type value(s) in the format specifier + value(s).
*/
#define btf_show_type_value(show, fmt, value) \
do { \
if ((value) != (__typeof__(value))0 || \
(show->flags & BTF_SHOW_ZERO) || \
show->state.depth == 0) { \
btf_show(show, "%s%s" fmt "%s%s", \
btf_show_indent(show), \
btf_show_name(show), \
value, btf_show_delim(show), \
btf_show_newline(show)); \
if (show->state.depth > show->state.depth_to_show) \
show->state.depth_to_show = show->state.depth; \
} \
} while (0)
#define btf_show_type_values(show, fmt, ...) \
do { \
btf_show(show, "%s%s" fmt "%s%s", btf_show_indent(show), \
btf_show_name(show), \
__VA_ARGS__, btf_show_delim(show), \
btf_show_newline(show)); \
if (show->state.depth > show->state.depth_to_show) \
show->state.depth_to_show = show->state.depth; \
} while (0)
/* How much is left to copy to safe buffer after @data? */
static int btf_show_obj_size_left(struct btf_show *show, void *data)
{
return show->obj.head + show->obj.size - data;
}
/* Is object pointed to by @data of @size already copied to our safe buffer? */
static bool btf_show_obj_is_safe(struct btf_show *show, void *data, int size)
{
return data >= show->obj.data &&
(data + size) < (show->obj.data + BTF_SHOW_OBJ_SAFE_SIZE);
}
/*
* If object pointed to by @data of @size falls within our safe buffer, return
* the equivalent pointer to the same safe data. Assumes
* copy_from_kernel_nofault() has already happened and our safe buffer is
* populated.
*/
static void *__btf_show_obj_safe(struct btf_show *show, void *data, int size)
{
if (btf_show_obj_is_safe(show, data, size))
return show->obj.safe + (data - show->obj.data);
return NULL;
}
/*
* Return a safe-to-access version of data pointed to by @data.
* We do this by copying the relevant amount of information
* to the struct btf_show obj.safe buffer using copy_from_kernel_nofault().
*
* If BTF_SHOW_UNSAFE is specified, just return data as-is; no
* safe copy is needed.
*
* Otherwise we need to determine if we have the required amount
* of data (determined by the @data pointer and the size of the
* largest base type we can encounter (represented by
* BTF_SHOW_OBJ_BASE_TYPE_SIZE). Having that much data ensures
* that we will be able to print some of the current object,
* and if more is needed a copy will be triggered.
* Some objects such as structs will not fit into the buffer;
* in such cases additional copies when we iterate over their
* members may be needed.
*
* btf_show_obj_safe() is used to return a safe buffer for
* btf_show_start_type(); this ensures that as we recurse into
* nested types we always have safe data for the given type.
* This approach is somewhat wasteful; it's possible for example
* that when iterating over a large union we'll end up copying the
* same data repeatedly, but the goal is safety not performance.
* We use stack data as opposed to per-CPU buffers because the
* iteration over a type can take some time, and preemption handling
* would greatly complicate use of the safe buffer.
*/
static void *btf_show_obj_safe(struct btf_show *show,
const struct btf_type *t,
void *data)
{
const struct btf_type *rt;
int size_left, size;
void *safe = NULL;
if (show->flags & BTF_SHOW_UNSAFE)
return data;
rt = btf_resolve_size(show->btf, t, &size);
if (IS_ERR(rt)) {
show->state.status = PTR_ERR(rt);
return NULL;
}
/*
* Is this toplevel object? If so, set total object size and
* initialize pointers. Otherwise check if we still fall within
* our safe object data.
*/
if (show->state.depth == 0) {
show->obj.size = size;
show->obj.head = data;
} else {
/*
* If the size of the current object is > our remaining
* safe buffer we _may_ need to do a new copy. However
* consider the case of a nested struct; it's size pushes
* us over the safe buffer limit, but showing any individual
* struct members does not. In such cases, we don't need
* to initiate a fresh copy yet; however we definitely need
* at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes left
* in our buffer, regardless of the current object size.
* The logic here is that as we resolve types we will
* hit a base type at some point, and we need to be sure
* the next chunk of data is safely available to display
* that type info safely. We cannot rely on the size of
* the current object here because it may be much larger
* than our current buffer (e.g. task_struct is 8k).
* All we want to do here is ensure that we can print the
* next basic type, which we can if either
* - the current type size is within the safe buffer; or
* - at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes are left in
* the safe buffer.
*/
safe = __btf_show_obj_safe(show, data,
min(size,
BTF_SHOW_OBJ_BASE_TYPE_SIZE));
}
/*
* We need a new copy to our safe object, either because we haven't
* yet copied and are initializing safe data, or because the data
* we want falls outside the boundaries of the safe object.
*/
if (!safe) {
size_left = btf_show_obj_size_left(show, data);
if (size_left > BTF_SHOW_OBJ_SAFE_SIZE)
size_left = BTF_SHOW_OBJ_SAFE_SIZE;
show->state.status = copy_from_kernel_nofault(show->obj.safe,
data, size_left);
if (!show->state.status) {
show->obj.data = data;
safe = show->obj.safe;
}
}
return safe;
}
/*
* Set the type we are starting to show and return a safe data pointer
* to be used for showing the associated data.
*/
static void *btf_show_start_type(struct btf_show *show,
const struct btf_type *t,
u32 type_id, void *data)
{
show->state.type = t;
show->state.type_id = type_id;
show->state.name[0] = '\0';
return btf_show_obj_safe(show, t, data);
}
static void btf_show_end_type(struct btf_show *show)
{
show->state.type = NULL;
show->state.type_id = 0;
show->state.name[0] = '\0';
}
static void *btf_show_start_aggr_type(struct btf_show *show,
const struct btf_type *t,
u32 type_id, void *data)
{
void *safe_data = btf_show_start_type(show, t, type_id, data);
if (!safe_data)
return safe_data;
btf_show(show, "%s%s%s", btf_show_indent(show),
btf_show_name(show),
btf_show_newline(show));
show->state.depth++;
return safe_data;
}
static void btf_show_end_aggr_type(struct btf_show *show,
const char *suffix)
{
show->state.depth--;
btf_show(show, "%s%s%s%s", btf_show_indent(show), suffix,
btf_show_delim(show), btf_show_newline(show));
btf_show_end_type(show);
}
static void btf_show_start_member(struct btf_show *show,
const struct btf_member *m)
{
show->state.member = m;
}
static void btf_show_start_array_member(struct btf_show *show)
{
show->state.array_member = 1;
btf_show_start_member(show, NULL);
}
static void btf_show_end_member(struct btf_show *show)
{
show->state.member = NULL;
}
static void btf_show_end_array_member(struct btf_show *show)
{
show->state.array_member = 0;
btf_show_end_member(show);
}
static void *btf_show_start_array_type(struct btf_show *show,
const struct btf_type *t,
u32 type_id,
u16 array_encoding,
void *data)
{
show->state.array_encoding = array_encoding;
show->state.array_terminated = 0;
return btf_show_start_aggr_type(show, t, type_id, data);
}
static void btf_show_end_array_type(struct btf_show *show)
{
show->state.array_encoding = 0;
show->state.array_terminated = 0;
btf_show_end_aggr_type(show, "]");
}
static void *btf_show_start_struct_type(struct btf_show *show,
const struct btf_type *t,
u32 type_id,
void *data)
{
return btf_show_start_aggr_type(show, t, type_id, data);
}
static void btf_show_end_struct_type(struct btf_show *show)
{
btf_show_end_aggr_type(show, "}");
}
__printf(2, 3) static void __btf_verifier_log(struct bpf_verifier_log *log,
const char *fmt, ...)
{
va_list args;
va_start(args, fmt);
bpf_verifier_vlog(log, fmt, args);
va_end(args);
}
__printf(2, 3) static void btf_verifier_log(struct btf_verifier_env *env,
const char *fmt, ...)
{
struct bpf_verifier_log *log = &env->log;
va_list args;
if (!bpf_verifier_log_needed(log))
return;
va_start(args, fmt);
bpf_verifier_vlog(log, fmt, args);
va_end(args);
}
__printf(4, 5) static void __btf_verifier_log_type(struct btf_verifier_env *env,
const struct btf_type *t,
bool log_details,
const char *fmt, ...)
{
struct bpf_verifier_log *log = &env->log;
struct btf *btf = env->btf;
va_list args;
if (!bpf_verifier_log_needed(log))
return;
if (log->level == BPF_LOG_KERNEL) {
/* btf verifier prints all types it is processing via
* btf_verifier_log_type(..., fmt = NULL).
* Skip those prints for in-kernel BTF verification.
*/
if (!fmt)
return;
/* Skip logging when loading module BTF with mismatches permitted */
if (env->btf->base_btf && IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH))
return;
}
__btf_verifier_log(log, "[%u] %s %s%s",
env->log_type_id,
btf_type_str(t),
__btf_name_by_offset(btf, t->name_off),
log_details ? " " : "");
if (log_details)
btf_type_ops(t)->log_details(env, t);
if (fmt && *fmt) {
__btf_verifier_log(log, " ");
va_start(args, fmt);
bpf_verifier_vlog(log, fmt, args);
va_end(args);
}
__btf_verifier_log(log, "\n");
}
#define btf_verifier_log_type(env, t, ...) \
__btf_verifier_log_type((env), (t), true, __VA_ARGS__)
#define btf_verifier_log_basic(env, t, ...) \
__btf_verifier_log_type((env), (t), false, __VA_ARGS__)
__printf(4, 5)
static void btf_verifier_log_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const char *fmt, ...)
{
struct bpf_verifier_log *log = &env->log;
struct btf *btf = env->btf;
va_list args;
if (!bpf_verifier_log_needed(log))
return;
if (log->level == BPF_LOG_KERNEL) {
if (!fmt)
return;
/* Skip logging when loading module BTF with mismatches permitted */
if (env->btf->base_btf && IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH))
return;
}
/* The CHECK_META phase already did a btf dump.
*
* If member is logged again, it must hit an error in
* parsing this member. It is useful to print out which
* struct this member belongs to.
*/
if (env->phase != CHECK_META)
btf_verifier_log_type(env, struct_type, NULL);
if (btf_type_kflag(struct_type))
__btf_verifier_log(log,
"\t%s type_id=%u bitfield_size=%u bits_offset=%u",
__btf_name_by_offset(btf, member->name_off),
member->type,
BTF_MEMBER_BITFIELD_SIZE(member->offset),
BTF_MEMBER_BIT_OFFSET(member->offset));
else
__btf_verifier_log(log, "\t%s type_id=%u bits_offset=%u",
__btf_name_by_offset(btf, member->name_off),
member->type, member->offset);
if (fmt && *fmt) {
__btf_verifier_log(log, " ");
va_start(args, fmt);
bpf_verifier_vlog(log, fmt, args);
va_end(args);
}
__btf_verifier_log(log, "\n");
}
__printf(4, 5)
static void btf_verifier_log_vsi(struct btf_verifier_env *env,
const struct btf_type *datasec_type,
const struct btf_var_secinfo *vsi,
const char *fmt, ...)
{
struct bpf_verifier_log *log = &env->log;
va_list args;
if (!bpf_verifier_log_needed(log))
return;
if (log->level == BPF_LOG_KERNEL && !fmt)
return;
if (env->phase != CHECK_META)
btf_verifier_log_type(env, datasec_type, NULL);
__btf_verifier_log(log, "\t type_id=%u offset=%u size=%u",
vsi->type, vsi->offset, vsi->size);
if (fmt && *fmt) {
__btf_verifier_log(log, " ");
va_start(args, fmt);
bpf_verifier_vlog(log, fmt, args);
va_end(args);
}
__btf_verifier_log(log, "\n");
}
static void btf_verifier_log_hdr(struct btf_verifier_env *env,
u32 btf_data_size)
{
struct bpf_verifier_log *log = &env->log;
const struct btf *btf = env->btf;
const struct btf_header *hdr;
if (!bpf_verifier_log_needed(log))
return;
if (log->level == BPF_LOG_KERNEL)
return;
hdr = &btf->hdr;
__btf_verifier_log(log, "magic: 0x%x\n", hdr->magic);
__btf_verifier_log(log, "version: %u\n", hdr->version);
__btf_verifier_log(log, "flags: 0x%x\n", hdr->flags);
__btf_verifier_log(log, "hdr_len: %u\n", hdr->hdr_len);
__btf_verifier_log(log, "type_off: %u\n", hdr->type_off);
__btf_verifier_log(log, "type_len: %u\n", hdr->type_len);
__btf_verifier_log(log, "str_off: %u\n", hdr->str_off);
__btf_verifier_log(log, "str_len: %u\n", hdr->str_len);
if (hdr->hdr_len >= sizeof(struct btf_header) &&
btf_data_size >= hdr->hdr_len) {
__btf_verifier_log(log, "layout_off: %u\n", hdr->layout_off);
__btf_verifier_log(log, "layout_len: %u\n", hdr->layout_len);
}
__btf_verifier_log(log, "btf_total_size: %u\n", btf_data_size);
}
static int btf_add_type(struct btf_verifier_env *env, struct btf_type *t)
{
struct btf *btf = env->btf;
if (btf->types_size == btf->nr_types) {
/* Expand 'types' array */
struct btf_type **new_types;
u32 expand_by, new_size;
if (btf->start_id + btf->types_size == BTF_MAX_TYPE) {
btf_verifier_log(env, "Exceeded max num of types");
return -E2BIG;
}
expand_by = max_t(u32, btf->types_size >> 2, 16);
new_size = min_t(u32, BTF_MAX_TYPE,
btf->types_size + expand_by);
new_types = kvzalloc_objs(*new_types, new_size,
GFP_KERNEL | __GFP_NOWARN);
if (!new_types)
return -ENOMEM;
if (btf->nr_types == 0) {
if (!btf->base_btf) {
/* lazily init VOID type */
new_types[0] = &btf_void;
btf->nr_types++;
}
} else {
memcpy(new_types, btf->types,
sizeof(*btf->types) * btf->nr_types);
}
kvfree(btf->types);
btf->types = new_types;
btf->types_size = new_size;
}
btf->types[btf->nr_types++] = t;
return 0;
}
static int btf_alloc_id(struct btf *btf)
{
int id;
idr_preload(GFP_KERNEL);
spin_lock_bh(&btf_idr_lock);
id = idr_alloc_cyclic(&btf_idr, btf, 1, INT_MAX, GFP_ATOMIC);
if (id > 0)
btf->id = id;
spin_unlock_bh(&btf_idr_lock);
idr_preload_end();
if (WARN_ON_ONCE(!id))
return -ENOSPC;
return id > 0 ? 0 : id;
}
static void btf_free_id(struct btf *btf)
{
unsigned long flags;
/*
* In map-in-map, calling map_delete_elem() on outer
* map will call bpf_map_put on the inner map.
* It will then eventually call btf_free_id()
* on the inner map. Some of the map_delete_elem()
* implementation may have irq disabled, so
* we need to use the _irqsave() version instead
* of the _bh() version.
*/
spin_lock_irqsave(&btf_idr_lock, flags);
if (btf->id) {
idr_remove(&btf_idr, btf->id);
/*
* Clear the id here to make this function idempotent, since it will get
* called a couple of times for module BTFs: on module unload, and then
* the final btf_put(). btf_alloc_id() starts IDs with 1, so we can use
* 0 as sentinel value.
*/
WRITE_ONCE(btf->id, 0);
}
spin_unlock_irqrestore(&btf_idr_lock, flags);
}
static void btf_free_kfunc_set_tab(struct btf *btf)
{
struct btf_kfunc_set_tab *tab = btf->kfunc_set_tab;
int hook;
if (!tab)
return;
for (hook = 0; hook < ARRAY_SIZE(tab->sets); hook++)
kfree(tab->sets[hook]);
kfree(tab);
btf->kfunc_set_tab = NULL;
}
static void btf_free_dtor_kfunc_tab(struct btf *btf)
{
struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab;
if (!tab)
return;
kfree(tab);
btf->dtor_kfunc_tab = NULL;
}
static void btf_struct_metas_free(struct btf_struct_metas *tab)
{
int i;
if (!tab)
return;
for (i = 0; i < tab->cnt; i++)
btf_record_free(tab->types[i].record);
kfree(tab);
}
static void btf_free_struct_meta_tab(struct btf *btf)
{
struct btf_struct_metas *tab = btf->struct_meta_tab;
btf_struct_metas_free(tab);
btf->struct_meta_tab = NULL;
}
static void btf_free_struct_ops_tab(struct btf *btf)
{
struct btf_struct_ops_tab *tab = btf->struct_ops_tab;
u32 i;
if (!tab)
return;
for (i = 0; i < tab->cnt; i++)
bpf_struct_ops_desc_release(&tab->ops[i]);
kfree(tab);
btf->struct_ops_tab = NULL;
}
static void btf_free(struct btf *btf)
{
btf_free_struct_meta_tab(btf);
btf_free_dtor_kfunc_tab(btf);
btf_free_kfunc_set_tab(btf);
btf_free_struct_ops_tab(btf);
kvfree(btf->types);
kvfree(btf->resolved_sizes);
kvfree(btf->resolved_ids);
/* vmlinux does not allocate btf->data, it simply points it at
* __start_BTF.
*/
if (!btf_is_vmlinux(btf))
kvfree(btf->data);
kvfree(btf->base_id_map);
kfree(btf);
}
static void btf_free_rcu(struct rcu_head *rcu)
{
struct btf *btf = container_of(rcu, struct btf, rcu);
btf_free(btf);
}
const char *btf_get_name(const struct btf *btf)
{
return btf->name;
}
void btf_get(struct btf *btf)
{
refcount_inc(&btf->refcnt);
}
void btf_put(struct btf *btf)
{
if (btf && refcount_dec_and_test(&btf->refcnt)) {
btf_free_id(btf);
call_rcu(&btf->rcu, btf_free_rcu);
}
}
struct btf *btf_base_btf(const struct btf *btf)
{
return btf->base_btf;
}
const struct btf_header *btf_header(const struct btf *btf)
{
return &btf->hdr;
}
void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
{
btf->base_btf = (struct btf *)base_btf;
btf->start_id = btf_nr_types(base_btf);
btf->start_str_off = base_btf->hdr.str_len;
}
static int env_resolve_init(struct btf_verifier_env *env)
{
struct btf *btf = env->btf;
u32 nr_types = btf->nr_types;
u32 *resolved_sizes = NULL;
u32 *resolved_ids = NULL;
u8 *visit_states = NULL;
resolved_sizes = kvcalloc(nr_types, sizeof(*resolved_sizes),
GFP_KERNEL | __GFP_NOWARN);
if (!resolved_sizes)
goto nomem;
resolved_ids = kvcalloc(nr_types, sizeof(*resolved_ids),
GFP_KERNEL | __GFP_NOWARN);
if (!resolved_ids)
goto nomem;
visit_states = kvcalloc(nr_types, sizeof(*visit_states),
GFP_KERNEL | __GFP_NOWARN);
if (!visit_states)
goto nomem;
btf->resolved_sizes = resolved_sizes;
btf->resolved_ids = resolved_ids;
env->visit_states = visit_states;
return 0;
nomem:
kvfree(resolved_sizes);
kvfree(resolved_ids);
kvfree(visit_states);
return -ENOMEM;
}
static void btf_verifier_env_free(struct btf_verifier_env *env)
{
kvfree(env->visit_states);
kfree(env);
}
static bool env_type_is_resolve_sink(const struct btf_verifier_env *env,
const struct btf_type *next_type)
{
switch (env->resolve_mode) {
case RESOLVE_TBD:
/* int, enum or void is a sink */
return !btf_type_needs_resolve(next_type);
case RESOLVE_PTR:
/* int, enum, void, struct, array, func or func_proto is a sink
* for ptr
*/
return !btf_type_is_modifier(next_type) &&
!btf_type_is_ptr(next_type);
case RESOLVE_STRUCT_OR_ARRAY:
/* int, enum, void, ptr, func or func_proto is a sink
* for struct and array
*/
return !btf_type_is_modifier(next_type) &&
!btf_type_is_array(next_type) &&
!btf_type_is_struct(next_type);
default:
BUG();
}
}
static bool env_type_is_resolved(const struct btf_verifier_env *env,
u32 type_id)
{
/* base BTF types should be resolved by now */
if (type_id < env->btf->start_id)
return true;
return env->visit_states[type_id - env->btf->start_id] == RESOLVED;
}
static int env_stack_push(struct btf_verifier_env *env,
const struct btf_type *t, u32 type_id)
{
const struct btf *btf = env->btf;
struct resolve_vertex *v;
if (env->top_stack == MAX_RESOLVE_DEPTH)
return -E2BIG;
if (type_id < btf->start_id
|| env->visit_states[type_id - btf->start_id] != NOT_VISITED)
return -EEXIST;
env->visit_states[type_id - btf->start_id] = VISITED;
v = &env->stack[env->top_stack++];
v->t = t;
v->type_id = type_id;
v->next_member = 0;
if (env->resolve_mode == RESOLVE_TBD) {
if (btf_type_is_ptr(t))
env->resolve_mode = RESOLVE_PTR;
else if (btf_type_is_struct(t) || btf_type_is_array(t))
env->resolve_mode = RESOLVE_STRUCT_OR_ARRAY;
}
return 0;
}
static void env_stack_set_next_member(struct btf_verifier_env *env,
u16 next_member)
{
env->stack[env->top_stack - 1].next_member = next_member;
}
static void env_stack_pop_resolved(struct btf_verifier_env *env,
u32 resolved_type_id,
u32 resolved_size)
{
u32 type_id = env->stack[--(env->top_stack)].type_id;
struct btf *btf = env->btf;
type_id -= btf->start_id; /* adjust to local type id */
btf->resolved_sizes[type_id] = resolved_size;
btf->resolved_ids[type_id] = resolved_type_id;
env->visit_states[type_id] = RESOLVED;
}
static const struct resolve_vertex *env_stack_peak(struct btf_verifier_env *env)
{
return env->top_stack ? &env->stack[env->top_stack - 1] : NULL;
}
/* Resolve the size of a passed-in "type"
*
* type: is an array (e.g. u32 array[x][y])
* return type: type "u32[x][y]", i.e. BTF_KIND_ARRAY,
* *type_size: (x * y * sizeof(u32)). Hence, *type_size always
* corresponds to the return type.
* *elem_type: u32
* *elem_id: id of u32
* *total_nelems: (x * y). Hence, individual elem size is
* (*type_size / *total_nelems)
* *type_id: id of type if it's changed within the function, 0 if not
*
* type: is not an array (e.g. const struct X)
* return type: type "struct X"
* *type_size: sizeof(struct X)
* *elem_type: same as return type ("struct X")
* *elem_id: 0
* *total_nelems: 1
* *type_id: id of type if it's changed within the function, 0 if not
*/
static const struct btf_type *
__btf_resolve_size(const struct btf *btf, const struct btf_type *type,
u32 *type_size, const struct btf_type **elem_type,
u32 *elem_id, u32 *total_nelems, u32 *type_id)
{
const struct btf_type *array_type = NULL;
const struct btf_array *array = NULL;
u32 i, size, nelems = 1, id = 0;
for (i = 0; i < MAX_RESOLVE_DEPTH; i++) {
switch (BTF_INFO_KIND(type->info)) {
/* type->size can be used */
case BTF_KIND_INT:
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
case BTF_KIND_ENUM:
case BTF_KIND_FLOAT:
case BTF_KIND_ENUM64:
size = type->size;
goto resolved;
case BTF_KIND_PTR:
size = sizeof(void *);
goto resolved;
/* Modifiers */
case BTF_KIND_TYPEDEF:
case BTF_KIND_VOLATILE:
case BTF_KIND_CONST:
case BTF_KIND_RESTRICT:
case BTF_KIND_TYPE_TAG:
id = type->type;
type = btf_type_by_id(btf, type->type);
break;
case BTF_KIND_ARRAY:
if (!array_type)
array_type = type;
array = btf_type_array(type);
if (nelems && array->nelems > U32_MAX / nelems)
return ERR_PTR(-EINVAL);
nelems *= array->nelems;
type = btf_type_by_id(btf, array->type);
break;
/* type without size */
default:
return ERR_PTR(-EINVAL);
}
}
return ERR_PTR(-EINVAL);
resolved:
if (nelems && size > U32_MAX / nelems)
return ERR_PTR(-EINVAL);
*type_size = nelems * size;
if (total_nelems)
*total_nelems = nelems;
if (elem_type)
*elem_type = type;
if (elem_id)
*elem_id = array ? array->type : 0;
if (type_id && id)
*type_id = id;
return array_type ? : type;
}
const struct btf_type *
btf_resolve_size(const struct btf *btf, const struct btf_type *type,
u32 *type_size)
{
return __btf_resolve_size(btf, type, type_size, NULL, NULL, NULL, NULL);
}
static u32 btf_resolved_type_id(const struct btf *btf, u32 type_id)
{
while (type_id < btf->start_id)
btf = btf->base_btf;
return btf->resolved_ids[type_id - btf->start_id];
}
/* The input param "type_id" must point to a needs_resolve type */
static const struct btf_type *btf_type_id_resolve(const struct btf *btf,
u32 *type_id)
{
*type_id = btf_resolved_type_id(btf, *type_id);
return btf_type_by_id(btf, *type_id);
}
static u32 btf_resolved_type_size(const struct btf *btf, u32 type_id)
{
while (type_id < btf->start_id)
btf = btf->base_btf;
return btf->resolved_sizes[type_id - btf->start_id];
}
const struct btf_type *btf_type_id_size(const struct btf *btf,
u32 *type_id, u32 *ret_size)
{
const struct btf_type *size_type;
u32 size_type_id = *type_id;
u32 size = 0;
size_type = btf_type_by_id(btf, size_type_id);
if (btf_type_nosize_or_null(size_type))
return NULL;
if (btf_type_has_size(size_type)) {
size = size_type->size;
} else if (btf_type_is_array(size_type)) {
size = btf_resolved_type_size(btf, size_type_id);
} else if (btf_type_is_ptr(size_type)) {
size = sizeof(void *);
} else {
if (WARN_ON_ONCE(!btf_type_is_modifier(size_type) &&
!btf_type_is_var(size_type)))
return NULL;
size_type_id = btf_resolved_type_id(btf, size_type_id);
size_type = btf_type_by_id(btf, size_type_id);
if (btf_type_nosize_or_null(size_type))
return NULL;
else if (btf_type_has_size(size_type))
size = size_type->size;
else if (btf_type_is_array(size_type))
size = btf_resolved_type_size(btf, size_type_id);
else if (btf_type_is_ptr(size_type))
size = sizeof(void *);
else
return NULL;
}
*type_id = size_type_id;
if (ret_size)
*ret_size = size;
return size_type;
}
static int btf_df_check_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
btf_verifier_log_basic(env, struct_type,
"Unsupported check_member");
return -EINVAL;
}
static int btf_df_check_kflag_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
btf_verifier_log_basic(env, struct_type,
"Unsupported check_kflag_member");
return -EINVAL;
}
/* Used for ptr, array struct/union and float type members.
* int, enum and modifier types have their specific callback functions.
*/
static int btf_generic_check_kflag_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
if (BTF_MEMBER_BITFIELD_SIZE(member->offset)) {
btf_verifier_log_member(env, struct_type, member,
"Invalid member bitfield_size");
return -EINVAL;
}
/* bitfield size is 0, so member->offset represents bit offset only.
* It is safe to call non kflag check_member variants.
*/
return btf_type_ops(member_type)->check_member(env, struct_type,
member,
member_type);
}
static int btf_df_resolve(struct btf_verifier_env *env,
const struct resolve_vertex *v)
{
btf_verifier_log_basic(env, v->t, "Unsupported resolve");
return -EINVAL;
}
static void btf_df_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offsets,
struct btf_show *show)
{
btf_show(show, "<unsupported kind:%u>", BTF_INFO_KIND(t->info));
}
static int btf_int_check_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
u32 int_data = btf_type_int(member_type);
u32 struct_bits_off = member->offset;
u32 struct_size = struct_type->size;
u32 nr_copy_bits;
u32 bytes_offset;
if (U32_MAX - struct_bits_off < BTF_INT_OFFSET(int_data)) {
btf_verifier_log_member(env, struct_type, member,
"bits_offset exceeds U32_MAX");
return -EINVAL;
}
struct_bits_off += BTF_INT_OFFSET(int_data);
bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
nr_copy_bits = BTF_INT_BITS(int_data) +
BITS_PER_BYTE_MASKED(struct_bits_off);
if (nr_copy_bits > BITS_PER_U128) {
btf_verifier_log_member(env, struct_type, member,
"nr_copy_bits exceeds 128");
return -EINVAL;
}
if (struct_size < bytes_offset ||
struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
btf_verifier_log_member(env, struct_type, member,
"Member exceeds struct_size");
return -EINVAL;
}
return 0;
}
static int btf_int_check_kflag_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
u32 struct_bits_off, nr_bits, nr_int_data_bits, bytes_offset;
u32 int_data = btf_type_int(member_type);
u32 struct_size = struct_type->size;
u32 nr_copy_bits;
/* a regular int type is required for the kflag int member */
if (!btf_type_int_is_regular(member_type)) {
btf_verifier_log_member(env, struct_type, member,
"Invalid member base type");
return -EINVAL;
}
/* check sanity of bitfield size */
nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
nr_int_data_bits = BTF_INT_BITS(int_data);
if (!nr_bits) {
/* Not a bitfield member, member offset must be at byte
* boundary.
*/
if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
btf_verifier_log_member(env, struct_type, member,
"Invalid member offset");
return -EINVAL;
}
nr_bits = nr_int_data_bits;
} else if (nr_bits > nr_int_data_bits) {
btf_verifier_log_member(env, struct_type, member,
"Invalid member bitfield_size");
return -EINVAL;
}
bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
nr_copy_bits = nr_bits + BITS_PER_BYTE_MASKED(struct_bits_off);
if (nr_copy_bits > BITS_PER_U128) {
btf_verifier_log_member(env, struct_type, member,
"nr_copy_bits exceeds 128");
return -EINVAL;
}
if (struct_size < bytes_offset ||
struct_size - bytes_offset < BITS_ROUNDUP_BYTES(nr_copy_bits)) {
btf_verifier_log_member(env, struct_type, member,
"Member exceeds struct_size");
return -EINVAL;
}
return 0;
}
static s32 btf_int_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
u32 int_data, nr_bits, meta_needed = sizeof(int_data);
u16 encoding;
if (meta_left < meta_needed) {
btf_verifier_log_basic(env, t,
"meta_left:%u meta_needed:%u",
meta_left, meta_needed);
return -EINVAL;
}
if (btf_type_vlen(t)) {
btf_verifier_log_type(env, t, "vlen != 0");
return -EINVAL;
}
if (btf_type_kflag(t)) {
btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
return -EINVAL;
}
int_data = btf_type_int(t);
if (int_data & ~BTF_INT_MASK) {
btf_verifier_log_basic(env, t, "Invalid int_data:%x",
int_data);
return -EINVAL;
}
nr_bits = BTF_INT_BITS(int_data) + BTF_INT_OFFSET(int_data);
if (nr_bits > BITS_PER_U128) {
btf_verifier_log_type(env, t, "nr_bits exceeds %zu",
BITS_PER_U128);
return -EINVAL;
}
if (BITS_ROUNDUP_BYTES(nr_bits) > t->size) {
btf_verifier_log_type(env, t, "nr_bits exceeds type_size");
return -EINVAL;
}
/*
* Only one of the encoding bits is allowed and it
* should be sufficient for the pretty print purpose (i.e. decoding).
* Multiple bits can be allowed later if it is found
* to be insufficient.
*/
encoding = BTF_INT_ENCODING(int_data);
if (encoding &&
encoding != BTF_INT_SIGNED &&
encoding != BTF_INT_CHAR &&
encoding != BTF_INT_BOOL) {
btf_verifier_log_type(env, t, "Unsupported encoding");
return -ENOTSUPP;
}
btf_verifier_log_type(env, t, NULL);
return meta_needed;
}
static void btf_int_log(struct btf_verifier_env *env,
const struct btf_type *t)
{
int int_data = btf_type_int(t);
btf_verifier_log(env,
"size=%u bits_offset=%u nr_bits=%u encoding=%s",
t->size, BTF_INT_OFFSET(int_data),
BTF_INT_BITS(int_data),
btf_int_encoding_str(BTF_INT_ENCODING(int_data)));
}
static void btf_int128_print(struct btf_show *show, void *data)
{
/* data points to a __int128 number.
* Suppose
* int128_num = *(__int128 *)data;
* The below formulas shows what upper_num and lower_num represents:
* upper_num = int128_num >> 64;
* lower_num = int128_num & 0xffffffffFFFFFFFFULL;
*/
u64 upper_num, lower_num;
#ifdef __BIG_ENDIAN_BITFIELD
upper_num = *(u64 *)data;
lower_num = *(u64 *)(data + 8);
#else
upper_num = *(u64 *)(data + 8);
lower_num = *(u64 *)data;
#endif
if (upper_num == 0)
btf_show_type_value(show, "0x%llx", lower_num);
else
btf_show_type_values(show, "0x%llx%016llx", upper_num,
lower_num);
}
static void btf_int128_shift(u64 *print_num, u16 left_shift_bits,
u16 right_shift_bits)
{
u64 upper_num, lower_num;
#ifdef __BIG_ENDIAN_BITFIELD
upper_num = print_num[0];
lower_num = print_num[1];
#else
upper_num = print_num[1];
lower_num = print_num[0];
#endif
/* shake out un-needed bits by shift/or operations */
if (left_shift_bits >= 64) {
upper_num = lower_num << (left_shift_bits - 64);
lower_num = 0;
} else {
upper_num = (upper_num << left_shift_bits) |
(lower_num >> (64 - left_shift_bits));
lower_num = lower_num << left_shift_bits;
}
if (right_shift_bits >= 64) {
lower_num = upper_num >> (right_shift_bits - 64);
upper_num = 0;
} else {
lower_num = (lower_num >> right_shift_bits) |
(upper_num << (64 - right_shift_bits));
upper_num = upper_num >> right_shift_bits;
}
#ifdef __BIG_ENDIAN_BITFIELD
print_num[0] = upper_num;
print_num[1] = lower_num;
#else
print_num[0] = lower_num;
print_num[1] = upper_num;
#endif
}
static void btf_bitfield_show(void *data, u8 bits_offset,
u8 nr_bits, struct btf_show *show)
{
u16 left_shift_bits, right_shift_bits;
u8 nr_copy_bytes;
u8 nr_copy_bits;
u64 print_num[2] = {};
nr_copy_bits = nr_bits + bits_offset;
nr_copy_bytes = BITS_ROUNDUP_BYTES(nr_copy_bits);
memcpy(print_num, data, nr_copy_bytes);
#ifdef __BIG_ENDIAN_BITFIELD
left_shift_bits = bits_offset;
#else
left_shift_bits = BITS_PER_U128 - nr_copy_bits;
#endif
right_shift_bits = BITS_PER_U128 - nr_bits;
btf_int128_shift(print_num, left_shift_bits, right_shift_bits);
btf_int128_print(show, print_num);
}
static void btf_int_bits_show(const struct btf *btf,
const struct btf_type *t,
void *data, u8 bits_offset,
struct btf_show *show)
{
u32 int_data = btf_type_int(t);
u8 nr_bits = BTF_INT_BITS(int_data);
u8 total_bits_offset;
/*
* bits_offset is at most 7.
* BTF_INT_OFFSET() cannot exceed 128 bits.
*/
total_bits_offset = bits_offset + BTF_INT_OFFSET(int_data);
data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
bits_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
btf_bitfield_show(data, bits_offset, nr_bits, show);
}
static void btf_int_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct btf_show *show)
{
u32 int_data = btf_type_int(t);
u8 encoding = BTF_INT_ENCODING(int_data);
bool sign = encoding & BTF_INT_SIGNED;
u8 nr_bits = BTF_INT_BITS(int_data);
void *safe_data;
safe_data = btf_show_start_type(show, t, type_id, data);
if (!safe_data)
return;
if (bits_offset || BTF_INT_OFFSET(int_data) ||
BITS_PER_BYTE_MASKED(nr_bits)) {
btf_int_bits_show(btf, t, safe_data, bits_offset, show);
goto out;
}
switch (nr_bits) {
case 128:
btf_int128_print(show, safe_data);
break;
case 64:
if (sign)
btf_show_type_value(show, "%lld", *(s64 *)safe_data);
else
btf_show_type_value(show, "%llu", *(u64 *)safe_data);
break;
case 32:
if (sign)
btf_show_type_value(show, "%d", *(s32 *)safe_data);
else
btf_show_type_value(show, "%u", *(u32 *)safe_data);
break;
case 16:
if (sign)
btf_show_type_value(show, "%d", *(s16 *)safe_data);
else
btf_show_type_value(show, "%u", *(u16 *)safe_data);
break;
case 8:
if (show->state.array_encoding == BTF_INT_CHAR) {
/* check for null terminator */
if (show->state.array_terminated)
break;
if (*(char *)data == '\0') {
show->state.array_terminated = 1;
break;
}
if (isprint(*(char *)data)) {
btf_show_type_value(show, "'%c'",
*(char *)safe_data);
break;
}
}
if (sign)
btf_show_type_value(show, "%d", *(s8 *)safe_data);
else
btf_show_type_value(show, "%u", *(u8 *)safe_data);
break;
default:
btf_int_bits_show(btf, t, safe_data, bits_offset, show);
break;
}
out:
btf_show_end_type(show);
}
static const struct btf_kind_operations int_ops = {
.check_meta = btf_int_check_meta,
.resolve = btf_df_resolve,
.check_member = btf_int_check_member,
.check_kflag_member = btf_int_check_kflag_member,
.log_details = btf_int_log,
.show = btf_int_show,
};
static int btf_modifier_check_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
const struct btf_type *resolved_type;
u32 resolved_type_id = member->type;
struct btf_member resolved_member;
struct btf *btf = env->btf;
resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
if (!resolved_type) {
btf_verifier_log_member(env, struct_type, member,
"Invalid member");
return -EINVAL;
}
resolved_member = *member;
resolved_member.type = resolved_type_id;
return btf_type_ops(resolved_type)->check_member(env, struct_type,
&resolved_member,
resolved_type);
}
static int btf_modifier_check_kflag_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
const struct btf_type *resolved_type;
u32 resolved_type_id = member->type;
struct btf_member resolved_member;
struct btf *btf = env->btf;
resolved_type = btf_type_id_size(btf, &resolved_type_id, NULL);
if (!resolved_type) {
btf_verifier_log_member(env, struct_type, member,
"Invalid member");
return -EINVAL;
}
resolved_member = *member;
resolved_member.type = resolved_type_id;
return btf_type_ops(resolved_type)->check_kflag_member(env, struct_type,
&resolved_member,
resolved_type);
}
static int btf_ptr_check_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
u32 struct_size, struct_bits_off, bytes_offset;
struct_size = struct_type->size;
struct_bits_off = member->offset;
bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
btf_verifier_log_member(env, struct_type, member,
"Member is not byte aligned");
return -EINVAL;
}
if (struct_size - bytes_offset < sizeof(void *)) {
btf_verifier_log_member(env, struct_type, member,
"Member exceeds struct_size");
return -EINVAL;
}
return 0;
}
static int btf_ref_type_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
const char *value;
if (btf_type_vlen(t)) {
btf_verifier_log_type(env, t, "vlen != 0");
return -EINVAL;
}
if (btf_type_kflag(t) && !btf_type_is_type_tag(t)) {
btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
return -EINVAL;
}
if (!BTF_TYPE_ID_VALID(t->type)) {
btf_verifier_log_type(env, t, "Invalid type_id");
return -EINVAL;
}
/* typedef/type_tag type must have a valid name, and other ref types,
* volatile, const, restrict, should have a null name.
*/
if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPEDEF) {
if (!t->name_off ||
!btf_name_valid_identifier(env->btf, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
} else if (BTF_INFO_KIND(t->info) == BTF_KIND_TYPE_TAG) {
value = btf_name_by_offset(env->btf, t->name_off);
if (!value || !value[0]) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
} else {
if (t->name_off) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
}
btf_verifier_log_type(env, t, NULL);
return 0;
}
static int btf_modifier_resolve(struct btf_verifier_env *env,
const struct resolve_vertex *v)
{
const struct btf_type *t = v->t;
const struct btf_type *next_type;
u32 next_type_id = t->type;
struct btf *btf = env->btf;
next_type = btf_type_by_id(btf, next_type_id);
if (!next_type || btf_type_is_resolve_source_only(next_type)) {
btf_verifier_log_type(env, v->t, "Invalid type_id");
return -EINVAL;
}
if (!env_type_is_resolve_sink(env, next_type) &&
!env_type_is_resolved(env, next_type_id))
return env_stack_push(env, next_type, next_type_id);
/* Figure out the resolved next_type_id with size.
* They will be stored in the current modifier's
* resolved_ids and resolved_sizes such that it can
* save us a few type-following when we use it later (e.g. in
* pretty print).
*/
if (!btf_type_id_size(btf, &next_type_id, NULL)) {
if (env_type_is_resolved(env, next_type_id))
next_type = btf_type_id_resolve(btf, &next_type_id);
/* "typedef void new_void", "const void"...etc */
if (!btf_type_is_void(next_type) &&
!btf_type_is_fwd(next_type) &&
!btf_type_is_func_proto(next_type)) {
btf_verifier_log_type(env, v->t, "Invalid type_id");
return -EINVAL;
}
}
env_stack_pop_resolved(env, next_type_id, 0);
return 0;
}
static int btf_var_resolve(struct btf_verifier_env *env,
const struct resolve_vertex *v)
{
const struct btf_type *next_type;
const struct btf_type *t = v->t;
u32 next_type_id = t->type;
struct btf *btf = env->btf;
next_type = btf_type_by_id(btf, next_type_id);
if (!next_type || btf_type_is_resolve_source_only(next_type)) {
btf_verifier_log_type(env, v->t, "Invalid type_id");
return -EINVAL;
}
if (!env_type_is_resolve_sink(env, next_type) &&
!env_type_is_resolved(env, next_type_id))
return env_stack_push(env, next_type, next_type_id);
if (btf_type_is_modifier(next_type)) {
const struct btf_type *resolved_type;
u32 resolved_type_id;
resolved_type_id = next_type_id;
resolved_type = btf_type_id_resolve(btf, &resolved_type_id);
if (btf_type_is_ptr(resolved_type) &&
!env_type_is_resolve_sink(env, resolved_type) &&
!env_type_is_resolved(env, resolved_type_id))
return env_stack_push(env, resolved_type,
resolved_type_id);
}
/* We must resolve to something concrete at this point, no
* forward types or similar that would resolve to size of
* zero is allowed.
*/
if (!btf_type_id_size(btf, &next_type_id, NULL)) {
btf_verifier_log_type(env, v->t, "Invalid type_id");
return -EINVAL;
}
env_stack_pop_resolved(env, next_type_id, 0);
return 0;
}
static int btf_ptr_resolve(struct btf_verifier_env *env,
const struct resolve_vertex *v)
{
const struct btf_type *next_type;
const struct btf_type *t = v->t;
u32 next_type_id = t->type;
struct btf *btf = env->btf;
next_type = btf_type_by_id(btf, next_type_id);
if (!next_type || btf_type_is_resolve_source_only(next_type)) {
btf_verifier_log_type(env, v->t, "Invalid type_id");
return -EINVAL;
}
if (!env_type_is_resolve_sink(env, next_type) &&
!env_type_is_resolved(env, next_type_id))
return env_stack_push(env, next_type, next_type_id);
/* If the modifier was RESOLVED during RESOLVE_STRUCT_OR_ARRAY,
* the modifier may have stopped resolving when it was resolved
* to a ptr (last-resolved-ptr).
*
* We now need to continue from the last-resolved-ptr to
* ensure the last-resolved-ptr will not referring back to
* the current ptr (t).
*/
if (btf_type_is_modifier(next_type)) {
const struct btf_type *resolved_type;
u32 resolved_type_id;
resolved_type_id = next_type_id;
resolved_type = btf_type_id_resolve(btf, &resolved_type_id);
if (btf_type_is_ptr(resolved_type) &&
!env_type_is_resolve_sink(env, resolved_type) &&
!env_type_is_resolved(env, resolved_type_id))
return env_stack_push(env, resolved_type,
resolved_type_id);
}
if (!btf_type_id_size(btf, &next_type_id, NULL)) {
if (env_type_is_resolved(env, next_type_id))
next_type = btf_type_id_resolve(btf, &next_type_id);
if (!btf_type_is_void(next_type) &&
!btf_type_is_fwd(next_type) &&
!btf_type_is_func_proto(next_type)) {
btf_verifier_log_type(env, v->t, "Invalid type_id");
return -EINVAL;
}
}
env_stack_pop_resolved(env, next_type_id, 0);
return 0;
}
static void btf_modifier_show(const struct btf *btf,
const struct btf_type *t,
u32 type_id, void *data,
u8 bits_offset, struct btf_show *show)
{
if (btf->resolved_ids)
t = btf_type_id_resolve(btf, &type_id);
else
t = btf_type_skip_modifiers(btf, type_id, NULL);
btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
}
static void btf_var_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct btf_show *show)
{
t = btf_type_id_resolve(btf, &type_id);
btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
}
static void btf_ptr_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct btf_show *show)
{
void *safe_data;
safe_data = btf_show_start_type(show, t, type_id, data);
if (!safe_data)
return;
/* It is a hashed value unless BTF_SHOW_PTR_RAW is specified */
if (show->flags & BTF_SHOW_PTR_RAW)
btf_show_type_value(show, "0x%px", *(void **)safe_data);
else
btf_show_type_value(show, "0x%p", *(void **)safe_data);
btf_show_end_type(show);
}
static void btf_ref_type_log(struct btf_verifier_env *env,
const struct btf_type *t)
{
btf_verifier_log(env, "type_id=%u", t->type);
}
static const struct btf_kind_operations modifier_ops = {
.check_meta = btf_ref_type_check_meta,
.resolve = btf_modifier_resolve,
.check_member = btf_modifier_check_member,
.check_kflag_member = btf_modifier_check_kflag_member,
.log_details = btf_ref_type_log,
.show = btf_modifier_show,
};
static const struct btf_kind_operations ptr_ops = {
.check_meta = btf_ref_type_check_meta,
.resolve = btf_ptr_resolve,
.check_member = btf_ptr_check_member,
.check_kflag_member = btf_generic_check_kflag_member,
.log_details = btf_ref_type_log,
.show = btf_ptr_show,
};
static s32 btf_fwd_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
if (btf_type_vlen(t)) {
btf_verifier_log_type(env, t, "vlen != 0");
return -EINVAL;
}
if (t->type) {
btf_verifier_log_type(env, t, "type != 0");
return -EINVAL;
}
/* fwd type must have a valid name */
if (!t->name_off ||
!btf_name_valid_identifier(env->btf, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
return 0;
}
static void btf_fwd_type_log(struct btf_verifier_env *env,
const struct btf_type *t)
{
btf_verifier_log(env, "%s", btf_type_kflag(t) ? "union" : "struct");
}
static const struct btf_kind_operations fwd_ops = {
.check_meta = btf_fwd_check_meta,
.resolve = btf_df_resolve,
.check_member = btf_df_check_member,
.check_kflag_member = btf_df_check_kflag_member,
.log_details = btf_fwd_type_log,
.show = btf_df_show,
};
static int btf_array_check_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
u32 struct_bits_off = member->offset;
u32 struct_size, bytes_offset;
u32 array_type_id, array_size;
struct btf *btf = env->btf;
if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
btf_verifier_log_member(env, struct_type, member,
"Member is not byte aligned");
return -EINVAL;
}
array_type_id = member->type;
btf_type_id_size(btf, &array_type_id, &array_size);
struct_size = struct_type->size;
bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
if (struct_size - bytes_offset < array_size) {
btf_verifier_log_member(env, struct_type, member,
"Member exceeds struct_size");
return -EINVAL;
}
return 0;
}
static s32 btf_array_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
const struct btf_array *array = btf_type_array(t);
u32 meta_needed = sizeof(*array);
if (meta_left < meta_needed) {
btf_verifier_log_basic(env, t,
"meta_left:%u meta_needed:%u",
meta_left, meta_needed);
return -EINVAL;
}
/* array type should not have a name */
if (t->name_off) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
if (btf_type_vlen(t)) {
btf_verifier_log_type(env, t, "vlen != 0");
return -EINVAL;
}
if (btf_type_kflag(t)) {
btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
return -EINVAL;
}
if (t->size) {
btf_verifier_log_type(env, t, "size != 0");
return -EINVAL;
}
/* Array elem type and index type cannot be in type void,
* so !array->type and !array->index_type are not allowed.
*/
if (!array->type || !BTF_TYPE_ID_VALID(array->type)) {
btf_verifier_log_type(env, t, "Invalid elem");
return -EINVAL;
}
if (!array->index_type || !BTF_TYPE_ID_VALID(array->index_type)) {
btf_verifier_log_type(env, t, "Invalid index");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
return meta_needed;
}
static int btf_array_resolve(struct btf_verifier_env *env,
const struct resolve_vertex *v)
{
const struct btf_array *array = btf_type_array(v->t);
const struct btf_type *elem_type, *index_type;
u32 elem_type_id, index_type_id;
struct btf *btf = env->btf;
u32 elem_size;
/* Check array->index_type */
index_type_id = array->index_type;
index_type = btf_type_by_id(btf, index_type_id);
if (btf_type_nosize_or_null(index_type) ||
btf_type_is_resolve_source_only(index_type)) {
btf_verifier_log_type(env, v->t, "Invalid index");
return -EINVAL;
}
if (!env_type_is_resolve_sink(env, index_type) &&
!env_type_is_resolved(env, index_type_id))
return env_stack_push(env, index_type, index_type_id);
index_type = btf_type_id_size(btf, &index_type_id, NULL);
if (!index_type || !btf_type_is_int(index_type) ||
!btf_type_int_is_regular(index_type)) {
btf_verifier_log_type(env, v->t, "Invalid index");
return -EINVAL;
}
/* Check array->type */
elem_type_id = array->type;
elem_type = btf_type_by_id(btf, elem_type_id);
if (btf_type_nosize_or_null(elem_type) ||
btf_type_is_resolve_source_only(elem_type)) {
btf_verifier_log_type(env, v->t,
"Invalid elem");
return -EINVAL;
}
if (!env_type_is_resolve_sink(env, elem_type) &&
!env_type_is_resolved(env, elem_type_id))
return env_stack_push(env, elem_type, elem_type_id);
elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
if (!elem_type) {
btf_verifier_log_type(env, v->t, "Invalid elem");
return -EINVAL;
}
if (btf_type_is_int(elem_type) && !btf_type_int_is_regular(elem_type)) {
btf_verifier_log_type(env, v->t, "Invalid array of int");
return -EINVAL;
}
if (array->nelems && elem_size > U32_MAX / array->nelems) {
btf_verifier_log_type(env, v->t,
"Array size overflows U32_MAX");
return -EINVAL;
}
env_stack_pop_resolved(env, elem_type_id, elem_size * array->nelems);
return 0;
}
static void btf_array_log(struct btf_verifier_env *env,
const struct btf_type *t)
{
const struct btf_array *array = btf_type_array(t);
btf_verifier_log(env, "type_id=%u index_type_id=%u nr_elems=%u",
array->type, array->index_type, array->nelems);
}
static void __btf_array_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct btf_show *show)
{
const struct btf_array *array = btf_type_array(t);
const struct btf_kind_operations *elem_ops;
const struct btf_type *elem_type;
u32 i, elem_size = 0, elem_type_id;
u16 encoding = 0;
elem_type_id = array->type;
elem_type = btf_type_skip_modifiers(btf, elem_type_id, NULL);
if (elem_type && btf_type_has_size(elem_type))
elem_size = elem_type->size;
if (elem_type && btf_type_is_int(elem_type)) {
u32 int_type = btf_type_int(elem_type);
encoding = BTF_INT_ENCODING(int_type);
/*
* BTF_INT_CHAR encoding never seems to be set for
* char arrays, so if size is 1 and element is
* printable as a char, we'll do that.
*/
if (elem_size == 1)
encoding = BTF_INT_CHAR;
}
if (!btf_show_start_array_type(show, t, type_id, encoding, data))
return;
if (!elem_type)
goto out;
elem_ops = btf_type_ops(elem_type);
for (i = 0; i < array->nelems; i++) {
btf_show_start_array_member(show);
elem_ops->show(btf, elem_type, elem_type_id, data,
bits_offset, show);
data += elem_size;
btf_show_end_array_member(show);
if (show->state.array_terminated)
break;
}
out:
btf_show_end_array_type(show);
}
static void btf_array_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct btf_show *show)
{
const struct btf_member *m = show->state.member;
/*
* First check if any members would be shown (are non-zero).
* See comments above "struct btf_show" definition for more
* details on how this works at a high-level.
*/
if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
if (!show->state.depth_check) {
show->state.depth_check = show->state.depth + 1;
show->state.depth_to_show = 0;
}
__btf_array_show(btf, t, type_id, data, bits_offset, show);
show->state.member = m;
if (show->state.depth_check != show->state.depth + 1)
return;
show->state.depth_check = 0;
if (show->state.depth_to_show <= show->state.depth)
return;
/*
* Reaching here indicates we have recursed and found
* non-zero array member(s).
*/
}
__btf_array_show(btf, t, type_id, data, bits_offset, show);
}
static const struct btf_kind_operations array_ops = {
.check_meta = btf_array_check_meta,
.resolve = btf_array_resolve,
.check_member = btf_array_check_member,
.check_kflag_member = btf_generic_check_kflag_member,
.log_details = btf_array_log,
.show = btf_array_show,
};
static int btf_struct_check_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
u32 struct_bits_off = member->offset;
u32 struct_size, bytes_offset;
if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
btf_verifier_log_member(env, struct_type, member,
"Member is not byte aligned");
return -EINVAL;
}
struct_size = struct_type->size;
bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
if (struct_size - bytes_offset < member_type->size) {
btf_verifier_log_member(env, struct_type, member,
"Member exceeds struct_size");
return -EINVAL;
}
return 0;
}
static s32 btf_struct_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
bool is_union = BTF_INFO_KIND(t->info) == BTF_KIND_UNION;
const struct btf_member *member;
u32 meta_needed, last_offset;
struct btf *btf = env->btf;
u32 struct_size = t->size;
u32 offset;
u16 i;
meta_needed = btf_type_vlen(t) * sizeof(*member);
if (meta_left < meta_needed) {
btf_verifier_log_basic(env, t,
"meta_left:%u meta_needed:%u",
meta_left, meta_needed);
return -EINVAL;
}
/* struct type either no name or a valid one */
if (t->name_off &&
!btf_name_valid_identifier(env->btf, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
last_offset = 0;
for_each_member(i, t, member) {
if (!btf_name_offset_valid(btf, member->name_off)) {
btf_verifier_log_member(env, t, member,
"Invalid member name_offset:%u",
member->name_off);
return -EINVAL;
}
/* struct member either no name or a valid one */
if (member->name_off &&
!btf_name_valid_identifier(btf, member->name_off)) {
btf_verifier_log_member(env, t, member, "Invalid name");
return -EINVAL;
}
/* A member cannot be in type void */
if (!member->type || !BTF_TYPE_ID_VALID(member->type)) {
btf_verifier_log_member(env, t, member,
"Invalid type_id");
return -EINVAL;
}
offset = __btf_member_bit_offset(t, member);
if (is_union && offset) {
btf_verifier_log_member(env, t, member,
"Invalid member bits_offset");
return -EINVAL;
}
/*
* ">" instead of ">=" because the last member could be
* "char a[0];"
*/
if (last_offset > offset) {
btf_verifier_log_member(env, t, member,
"Invalid member bits_offset");
return -EINVAL;
}
if (BITS_ROUNDUP_BYTES(offset) > struct_size) {
btf_verifier_log_member(env, t, member,
"Member bits_offset exceeds its struct size");
return -EINVAL;
}
btf_verifier_log_member(env, t, member, NULL);
last_offset = offset;
}
return meta_needed;
}
static int btf_struct_resolve(struct btf_verifier_env *env,
const struct resolve_vertex *v)
{
const struct btf_member *member;
int err;
u16 i;
/* Before continue resolving the next_member,
* ensure the last member is indeed resolved to a
* type with size info.
*/
if (v->next_member) {
const struct btf_type *last_member_type;
const struct btf_member *last_member;
u32 last_member_type_id;
last_member = btf_type_member(v->t) + v->next_member - 1;
last_member_type_id = last_member->type;
if (WARN_ON_ONCE(!env_type_is_resolved(env,
last_member_type_id)))
return -EINVAL;
last_member_type = btf_type_by_id(env->btf,
last_member_type_id);
if (btf_type_kflag(v->t))
err = btf_type_ops(last_member_type)->check_kflag_member(env, v->t,
last_member,
last_member_type);
else
err = btf_type_ops(last_member_type)->check_member(env, v->t,
last_member,
last_member_type);
if (err)
return err;
}
for_each_member_from(i, v->next_member, v->t, member) {
u32 member_type_id = member->type;
const struct btf_type *member_type = btf_type_by_id(env->btf,
member_type_id);
if (btf_type_nosize_or_null(member_type) ||
btf_type_is_resolve_source_only(member_type)) {
btf_verifier_log_member(env, v->t, member,
"Invalid member");
return -EINVAL;
}
if (!env_type_is_resolve_sink(env, member_type) &&
!env_type_is_resolved(env, member_type_id)) {
env_stack_set_next_member(env, i + 1);
return env_stack_push(env, member_type, member_type_id);
}
if (btf_type_kflag(v->t))
err = btf_type_ops(member_type)->check_kflag_member(env, v->t,
member,
member_type);
else
err = btf_type_ops(member_type)->check_member(env, v->t,
member,
member_type);
if (err)
return err;
}
env_stack_pop_resolved(env, 0, 0);
return 0;
}
static void btf_struct_log(struct btf_verifier_env *env,
const struct btf_type *t)
{
btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
}
enum {
BTF_FIELD_IGNORE = 0,
BTF_FIELD_FOUND = 1,
};
struct btf_field_info {
enum btf_field_type type;
u32 off;
union {
struct {
u32 type_id;
} kptr;
struct {
const char *node_name;
u32 value_btf_id;
} graph_root;
};
};
static int btf_find_struct(const struct btf *btf, const struct btf_type *t,
u32 off, int sz, enum btf_field_type field_type,
struct btf_field_info *info)
{
if (!__btf_type_is_struct(t))
return BTF_FIELD_IGNORE;
if (t->size != sz)
return BTF_FIELD_IGNORE;
info->type = field_type;
info->off = off;
return BTF_FIELD_FOUND;
}
static int btf_find_kptr(const struct btf *btf, const struct btf_type *t,
u32 off, int sz, struct btf_field_info *info, u32 field_mask)
{
enum btf_field_type type;
const char *tag_value;
bool is_type_tag;
u32 res_id;
/* Permit modifiers on the pointer itself */
if (btf_type_is_volatile(t))
t = btf_type_by_id(btf, t->type);
/* For PTR, sz is always == 8 */
if (!btf_type_is_ptr(t))
return BTF_FIELD_IGNORE;
t = btf_type_by_id(btf, t->type);
is_type_tag = btf_type_is_type_tag(t) && !btf_type_kflag(t);
if (!is_type_tag)
return BTF_FIELD_IGNORE;
/* Reject extra tags */
if (btf_type_is_type_tag(btf_type_by_id(btf, t->type)))
return -EINVAL;
tag_value = __btf_name_by_offset(btf, t->name_off);
if (!strcmp("kptr_untrusted", tag_value))
type = BPF_KPTR_UNREF;
else if (!strcmp("kptr", tag_value))
type = BPF_KPTR_REF;
else if (!strcmp("percpu_kptr", tag_value))
type = BPF_KPTR_PERCPU;
else if (!strcmp("uptr", tag_value))
type = BPF_UPTR;
else
return -EINVAL;
if (!(type & field_mask))
return BTF_FIELD_IGNORE;
/* Get the base type */
t = btf_type_skip_modifiers(btf, t->type, &res_id);
/* Only pointer to struct is allowed */
if (!__btf_type_is_struct(t))
return -EINVAL;
info->type = type;
info->off = off;
info->kptr.type_id = res_id;
return BTF_FIELD_FOUND;
}
int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *pt,
int comp_idx, const char *tag_key, int last_id)
{
int len = strlen(tag_key);
int i, n;
for (i = last_id + 1, n = btf_nr_types(btf); i < n; i++) {
const struct btf_type *t = btf_type_by_id(btf, i);
if (!btf_type_is_decl_tag(t))
continue;
if (pt != btf_type_by_id(btf, t->type))
continue;
if (btf_type_decl_tag(t)->component_idx != comp_idx)
continue;
if (strncmp(__btf_name_by_offset(btf, t->name_off), tag_key, len))
continue;
return i;
}
return -ENOENT;
}
const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt,
int comp_idx, const char *tag_key)
{
const char *value = NULL;
const struct btf_type *t;
int len, id;
id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key,
btf_named_start_id(btf, false) - 1);
if (id < 0)
return ERR_PTR(id);
t = btf_type_by_id(btf, id);
len = strlen(tag_key);
value = __btf_name_by_offset(btf, t->name_off) + len;
/* Prevent duplicate entries for same type */
id = btf_find_next_decl_tag(btf, pt, comp_idx, tag_key, id);
if (id >= 0)
return ERR_PTR(-EEXIST);
return value;
}
static int
btf_find_graph_root(const struct btf *btf, const struct btf_type *pt,
const struct btf_type *t, int comp_idx, u32 off,
int sz, struct btf_field_info *info,
enum btf_field_type head_type)
{
const char *node_field_name;
const char *value_type;
s32 id;
if (!__btf_type_is_struct(t))
return BTF_FIELD_IGNORE;
if (t->size != sz)
return BTF_FIELD_IGNORE;
value_type = btf_find_decl_tag_value(btf, pt, comp_idx, "contains:");
if (IS_ERR(value_type))
return -EINVAL;
node_field_name = strstr(value_type, ":");
if (!node_field_name)
return -EINVAL;
value_type = kstrndup(value_type, node_field_name - value_type,
GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
if (!value_type)
return -ENOMEM;
id = btf_find_by_name_kind(btf, value_type, BTF_KIND_STRUCT);
kfree(value_type);
if (id < 0)
return id;
node_field_name++;
if (str_is_empty(node_field_name))
return -EINVAL;
info->type = head_type;
info->off = off;
info->graph_root.value_btf_id = id;
info->graph_root.node_name = node_field_name;
return BTF_FIELD_FOUND;
}
static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_type,
u32 field_mask, u32 *seen_mask, int *align, int *sz)
{
const struct {
enum btf_field_type type;
const char *const name;
const bool is_unique;
} field_types[] = {
{ BPF_SPIN_LOCK, "bpf_spin_lock", true },
{ BPF_RES_SPIN_LOCK, "bpf_res_spin_lock", true },
{ BPF_TIMER, "bpf_timer", true },
{ BPF_WORKQUEUE, "bpf_wq", true },
{ BPF_TASK_WORK, "bpf_task_work", true },
{ BPF_LIST_HEAD, "bpf_list_head", false },
{ BPF_LIST_NODE, "bpf_list_node", false },
{ BPF_RB_ROOT, "bpf_rb_root", false },
{ BPF_RB_NODE, "bpf_rb_node", false },
{ BPF_REFCOUNT, "bpf_refcount", false },
};
int type = 0, i;
const char *name = __btf_name_by_offset(btf, var_type->name_off);
const char *field_type_name;
enum btf_field_type field_type;
bool is_unique;
for (i = 0; i < ARRAY_SIZE(field_types); ++i) {
field_type = field_types[i].type;
field_type_name = field_types[i].name;
is_unique = field_types[i].is_unique;
if (!(field_mask & field_type) || strcmp(name, field_type_name))
continue;
if (is_unique) {
if (*seen_mask & field_type)
return -E2BIG;
*seen_mask |= field_type;
}
type = field_type;
goto end;
}
/* Only return BPF_KPTR when all other types with matchable names fail */
if (field_mask & (BPF_KPTR | BPF_UPTR) && !__btf_type_is_struct(var_type)) {
type = BPF_KPTR_REF;
goto end;
}
return 0;
end:
*sz = btf_field_type_size(type);
*align = btf_field_type_align(type);
return type;
}
/* Repeat a number of fields for a specified number of times.
*
* Copy the fields starting from the first field and repeat them for
* repeat_cnt times. The fields are repeated by adding the offset of each
* field with
* (i + 1) * elem_size
* where i is the repeat index and elem_size is the size of an element.
*/
static int btf_repeat_fields(struct btf_field_info *info, int info_cnt,
u32 field_cnt, u32 repeat_cnt, u32 elem_size)
{
u32 i, j;
u32 cur;
/* Ensure not repeating fields that should not be repeated. */
for (i = 0; i < field_cnt; i++) {
switch (info[i].type) {
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
case BPF_KPTR_PERCPU:
case BPF_UPTR:
case BPF_LIST_HEAD:
case BPF_RB_ROOT:
break;
default:
return -EINVAL;
}
}
/* The type of struct size or variable size is u32,
* so the multiplication will not overflow.
*/
if (field_cnt * (repeat_cnt + 1) > info_cnt)
return -E2BIG;
cur = field_cnt;
for (i = 0; i < repeat_cnt; i++) {
memcpy(&info[cur], &info[0], field_cnt * sizeof(info[0]));
for (j = 0; j < field_cnt; j++)
info[cur++].off += (i + 1) * elem_size;
}
return 0;
}
static int btf_find_struct_field(const struct btf *btf,
const struct btf_type *t, u32 field_mask,
struct btf_field_info *info, int info_cnt,
u32 level);
/* Find special fields in the struct type of a field.
*
* This function is used to find fields of special types that is not a
* global variable or a direct field of a struct type. It also handles the
* repetition if it is the element type of an array.
*/
static int btf_find_nested_struct(const struct btf *btf, const struct btf_type *t,
u32 off, u32 nelems,
u32 field_mask, struct btf_field_info *info,
int info_cnt, u32 level)
{
int ret, err, i;
level++;
if (level >= MAX_RESOLVE_DEPTH)
return -E2BIG;
ret = btf_find_struct_field(btf, t, field_mask, info, info_cnt, level);
if (ret <= 0)
return ret;
/* Shift the offsets of the nested struct fields to the offsets
* related to the container.
*/
for (i = 0; i < ret; i++)
info[i].off += off;
if (nelems > 1) {
err = btf_repeat_fields(info, info_cnt, ret, nelems - 1, t->size);
if (err == 0)
ret *= nelems;
else
ret = err;
}
return ret;
}
static int btf_find_field_one(const struct btf *btf,
const struct btf_type *var,
const struct btf_type *var_type,
int var_idx,
u32 off, u32 expected_size,
u32 field_mask, u32 *seen_mask,
struct btf_field_info *info, int info_cnt,
u32 level)
{
int ret, align, sz, field_type;
struct btf_field_info tmp;
const struct btf_array *array;
u32 i, nelems = 1;
/* Walk into array types to find the element type and the number of
* elements in the (flattened) array.
*/
for (i = 0; i < MAX_RESOLVE_DEPTH && btf_type_is_array(var_type); i++) {
array = btf_array(var_type);
nelems *= array->nelems;
var_type = btf_type_by_id(btf, array->type);
}
if (i == MAX_RESOLVE_DEPTH)
return -E2BIG;
if (nelems == 0)
return 0;
field_type = btf_get_field_type(btf, var_type,
field_mask, seen_mask, &align, &sz);
/* Look into variables of struct types */
if (!field_type && __btf_type_is_struct(var_type)) {
sz = var_type->size;
if (expected_size && expected_size != sz * nelems)
return 0;
ret = btf_find_nested_struct(btf, var_type, off, nelems, field_mask,
&info[0], info_cnt, level);
return ret;
}
if (field_type == 0)
return 0;
if (field_type < 0)
return field_type;
if (expected_size && expected_size != sz * nelems)
return 0;
if (off % align)
return 0;
switch (field_type) {
case BPF_SPIN_LOCK:
case BPF_RES_SPIN_LOCK:
case BPF_TIMER:
case BPF_WORKQUEUE:
case BPF_LIST_NODE:
case BPF_RB_NODE:
case BPF_REFCOUNT:
case BPF_TASK_WORK:
ret = btf_find_struct(btf, var_type, off, sz, field_type,
info_cnt ? &info[0] : &tmp);
if (ret < 0)
return ret;
break;
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
case BPF_KPTR_PERCPU:
case BPF_UPTR:
ret = btf_find_kptr(btf, var_type, off, sz,
info_cnt ? &info[0] : &tmp, field_mask);
if (ret < 0)
return ret;
break;
case BPF_LIST_HEAD:
case BPF_RB_ROOT:
ret = btf_find_graph_root(btf, var, var_type,
var_idx, off, sz,
info_cnt ? &info[0] : &tmp,
field_type);
if (ret < 0)
return ret;
break;
default:
return -EFAULT;
}
if (ret == BTF_FIELD_IGNORE)
return 0;
if (!info_cnt)
return -E2BIG;
if (nelems > 1) {
ret = btf_repeat_fields(info, info_cnt, 1, nelems - 1, sz);
if (ret < 0)
return ret;
}
return nelems;
}
static int btf_find_struct_field(const struct btf *btf,
const struct btf_type *t, u32 field_mask,
struct btf_field_info *info, int info_cnt,
u32 level)
{
int ret, idx = 0;
const struct btf_member *member;
u32 i, off, seen_mask = 0;
for_each_member(i, t, member) {
const struct btf_type *member_type = btf_type_by_id(btf,
member->type);
off = __btf_member_bit_offset(t, member);
if (off % 8)
/* valid C code cannot generate such BTF */
return -EINVAL;
off /= 8;
ret = btf_find_field_one(btf, t, member_type, i,
off, 0,
field_mask, &seen_mask,
&info[idx], info_cnt - idx, level);
if (ret < 0)
return ret;
idx += ret;
}
return idx;
}
static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t,
u32 field_mask, struct btf_field_info *info,
int info_cnt, u32 level)
{
int ret, idx = 0;
const struct btf_var_secinfo *vsi;
u32 i, off, seen_mask = 0;
for_each_vsi(i, t, vsi) {
const struct btf_type *var = btf_type_by_id(btf, vsi->type);
const struct btf_type *var_type = btf_type_by_id(btf, var->type);
off = vsi->offset;
ret = btf_find_field_one(btf, var, var_type, -1, off, vsi->size,
field_mask, &seen_mask,
&info[idx], info_cnt - idx,
level);
if (ret < 0)
return ret;
idx += ret;
}
return idx;
}
static int btf_find_field(const struct btf *btf, const struct btf_type *t,
u32 field_mask, struct btf_field_info *info,
int info_cnt)
{
if (__btf_type_is_struct(t))
return btf_find_struct_field(btf, t, field_mask, info, info_cnt, 0);
else if (btf_type_is_datasec(t))
return btf_find_datasec_var(btf, t, field_mask, info, info_cnt, 0);
return -EINVAL;
}
/* Callers have to ensure the life cycle of btf if it is program BTF */
static int btf_parse_kptr(const struct btf *btf, struct btf_field *field,
struct btf_field_info *info)
{
struct module *mod = NULL;
const struct btf_type *t;
/* If a matching btf type is found in kernel or module BTFs, kptr_ref
* is that BTF, otherwise it's program BTF
*/
struct btf *kptr_btf;
int ret;
s32 id;
/* Find type in map BTF, and use it to look up the matching type
* in vmlinux or module BTFs, by name and kind.
*/
t = btf_type_by_id(btf, info->kptr.type_id);
id = bpf_find_btf_id(__btf_name_by_offset(btf, t->name_off), BTF_INFO_KIND(t->info),
&kptr_btf);
if (id == -ENOENT) {
/* btf_parse_kptr should only be called w/ btf = program BTF */
WARN_ON_ONCE(btf_is_kernel(btf));
/* Type exists only in program BTF. Assume that it's a MEM_ALLOC
* kptr allocated via bpf_obj_new
*/
field->kptr.dtor = NULL;
id = info->kptr.type_id;
kptr_btf = (struct btf *)btf;
goto found_dtor;
}
if (id < 0)
return id;
/* Find and stash the function pointer for the destruction function that
* needs to be eventually invoked from the map free path.
*/
if (info->type == BPF_KPTR_REF) {
const struct btf_type *dtor_func;
const char *dtor_func_name;
unsigned long addr;
s32 dtor_btf_id;
/* This call also serves as a whitelist of allowed objects that
* can be used as a referenced pointer and be stored in a map at
* the same time.
*/
dtor_btf_id = btf_find_dtor_kfunc(kptr_btf, id);
if (dtor_btf_id < 0) {
ret = dtor_btf_id;
goto end_btf;
}
dtor_func = btf_type_by_id(kptr_btf, dtor_btf_id);
if (!dtor_func) {
ret = -ENOENT;
goto end_btf;
}
if (btf_is_module(kptr_btf)) {
mod = btf_try_get_module(kptr_btf);
if (!mod) {
ret = -ENXIO;
goto end_btf;
}
}
/* We already verified dtor_func to be btf_type_is_func
* in register_btf_id_dtor_kfuncs.
*/
dtor_func_name = __btf_name_by_offset(kptr_btf, dtor_func->name_off);
addr = kallsyms_lookup_name(dtor_func_name);
if (!addr) {
ret = -EINVAL;
goto end_mod;
}
field->kptr.dtor = (void *)addr;
}
found_dtor:
field->kptr.btf_id = id;
field->kptr.btf = kptr_btf;
field->kptr.module = mod;
return 0;
end_mod:
module_put(mod);
end_btf:
btf_put(kptr_btf);
return ret;
}
static int btf_parse_graph_root(const struct btf *btf,
struct btf_field *field,
struct btf_field_info *info,
const char *node_type_name,
size_t node_type_align)
{
const struct btf_type *t, *n = NULL;
const struct btf_member *member;
u32 offset;
int i;
t = btf_type_by_id(btf, info->graph_root.value_btf_id);
/* We've already checked that value_btf_id is a struct type. We
* just need to figure out the offset of the list_node, and
* verify its type.
*/
for_each_member(i, t, member) {
if (strcmp(info->graph_root.node_name,
__btf_name_by_offset(btf, member->name_off)))
continue;
/* Invalid BTF, two members with same name */
if (n)
return -EINVAL;
n = btf_type_by_id(btf, member->type);
if (!__btf_type_is_struct(n))
return -EINVAL;
if (strcmp(node_type_name, __btf_name_by_offset(btf, n->name_off)))
return -EINVAL;
offset = __btf_member_bit_offset(n, member);
if (offset % 8)
return -EINVAL;
offset /= 8;
if (offset % node_type_align)
return -EINVAL;
field->graph_root.btf = (struct btf *)btf;
field->graph_root.value_btf_id = info->graph_root.value_btf_id;
field->graph_root.node_offset = offset;
}
if (!n)
return -ENOENT;
return 0;
}
static int btf_parse_list_head(const struct btf *btf, struct btf_field *field,
struct btf_field_info *info)
{
return btf_parse_graph_root(btf, field, info, "bpf_list_node",
__alignof__(struct bpf_list_node));
}
static int btf_parse_rb_root(const struct btf *btf, struct btf_field *field,
struct btf_field_info *info)
{
return btf_parse_graph_root(btf, field, info, "bpf_rb_node",
__alignof__(struct bpf_rb_node));
}
static int btf_field_cmp(const void *_a, const void *_b, const void *priv)
{
const struct btf_field *a = (const struct btf_field *)_a;
const struct btf_field *b = (const struct btf_field *)_b;
if (a->offset < b->offset)
return -1;
else if (a->offset > b->offset)
return 1;
return 0;
}
struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type *t,
u32 field_mask, u32 value_size)
{
struct btf_field_info info_arr[BTF_FIELDS_MAX];
u32 next_off = 0, field_type_size;
struct btf_record *rec;
int ret, i, cnt;
ret = btf_find_field(btf, t, field_mask, info_arr, ARRAY_SIZE(info_arr));
if (ret < 0)
return ERR_PTR(ret);
if (!ret)
return NULL;
cnt = ret;
/* This needs to be kzalloc to zero out padding and unused fields, see
* comment in btf_record_equal.
*/
rec = kzalloc_flex(*rec, fields, cnt, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
if (!rec)
return ERR_PTR(-ENOMEM);
rec->spin_lock_off = -EINVAL;
rec->res_spin_lock_off = -EINVAL;
rec->timer_off = -EINVAL;
rec->wq_off = -EINVAL;
rec->refcount_off = -EINVAL;
rec->task_work_off = -EINVAL;
for (i = 0; i < cnt; i++) {
field_type_size = btf_field_type_size(info_arr[i].type);
if (info_arr[i].off + field_type_size > value_size) {
WARN_ONCE(1, "verifier bug off %d size %d", info_arr[i].off, value_size);
ret = -EFAULT;
goto end;
}
if (info_arr[i].off < next_off) {
ret = -EEXIST;
goto end;
}
next_off = info_arr[i].off + field_type_size;
rec->field_mask |= info_arr[i].type;
rec->fields[i].offset = info_arr[i].off;
rec->fields[i].type = info_arr[i].type;
rec->fields[i].size = field_type_size;
switch (info_arr[i].type) {
case BPF_SPIN_LOCK:
WARN_ON_ONCE(rec->spin_lock_off >= 0);
/* Cache offset for faster lookup at runtime */
rec->spin_lock_off = rec->fields[i].offset;
break;
case BPF_RES_SPIN_LOCK:
WARN_ON_ONCE(rec->spin_lock_off >= 0);
/* Cache offset for faster lookup at runtime */
rec->res_spin_lock_off = rec->fields[i].offset;
break;
case BPF_TIMER:
WARN_ON_ONCE(rec->timer_off >= 0);
/* Cache offset for faster lookup at runtime */
rec->timer_off = rec->fields[i].offset;
break;
case BPF_WORKQUEUE:
WARN_ON_ONCE(rec->wq_off >= 0);
/* Cache offset for faster lookup at runtime */
rec->wq_off = rec->fields[i].offset;
break;
case BPF_TASK_WORK:
WARN_ON_ONCE(rec->task_work_off >= 0);
rec->task_work_off = rec->fields[i].offset;
break;
case BPF_REFCOUNT:
WARN_ON_ONCE(rec->refcount_off >= 0);
/* Cache offset for faster lookup at runtime */
rec->refcount_off = rec->fields[i].offset;
break;
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
case BPF_KPTR_PERCPU:
case BPF_UPTR:
ret = btf_parse_kptr(btf, &rec->fields[i], &info_arr[i]);
if (ret < 0)
goto end;
break;
case BPF_LIST_HEAD:
ret = btf_parse_list_head(btf, &rec->fields[i], &info_arr[i]);
if (ret < 0)
goto end;
break;
case BPF_RB_ROOT:
ret = btf_parse_rb_root(btf, &rec->fields[i], &info_arr[i]);
if (ret < 0)
goto end;
break;
case BPF_LIST_NODE:
case BPF_RB_NODE:
break;
default:
ret = -EFAULT;
goto end;
}
rec->cnt++;
}
if (rec->spin_lock_off >= 0 && rec->res_spin_lock_off >= 0) {
ret = -EINVAL;
goto end;
}
/* bpf_{list_head, rb_node} require bpf_spin_lock */
if ((btf_record_has_field(rec, BPF_LIST_HEAD) ||
btf_record_has_field(rec, BPF_RB_ROOT)) &&
(rec->spin_lock_off < 0 && rec->res_spin_lock_off < 0)) {
ret = -EINVAL;
goto end;
}
if (rec->refcount_off < 0 &&
btf_record_has_field(rec, BPF_LIST_NODE) &&
btf_record_has_field(rec, BPF_RB_NODE)) {
ret = -EINVAL;
goto end;
}
sort_r(rec->fields, rec->cnt, sizeof(struct btf_field), btf_field_cmp,
NULL, rec);
return rec;
end:
btf_record_free(rec);
return ERR_PTR(ret);
}
int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec)
{
int i;
/* There are three types that signify ownership of some other type:
* kptr_ref, bpf_list_head, bpf_rb_root.
* kptr_ref only supports storing kernel types, which can't store
* references to program allocated local types.
*
* Hence we only need to ensure that bpf_{list_head,rb_root} ownership
* does not form cycles.
*/
if (IS_ERR_OR_NULL(rec) || !(rec->field_mask & (BPF_GRAPH_ROOT | BPF_UPTR)))
return 0;
for (i = 0; i < rec->cnt; i++) {
struct btf_struct_meta *meta;
const struct btf_type *t;
u32 btf_id;
if (rec->fields[i].type == BPF_UPTR) {
/* The uptr only supports pinning one page and cannot
* point to a kernel struct
*/
if (btf_is_kernel(rec->fields[i].kptr.btf))
return -EINVAL;
t = btf_type_by_id(rec->fields[i].kptr.btf,
rec->fields[i].kptr.btf_id);
if (!t->size)
return -EINVAL;
if (t->size > PAGE_SIZE)
return -E2BIG;
continue;
}
if (!(rec->fields[i].type & BPF_GRAPH_ROOT))
continue;
btf_id = rec->fields[i].graph_root.value_btf_id;
meta = btf_find_struct_meta(btf, btf_id);
if (!meta)
return -EFAULT;
rec->fields[i].graph_root.value_rec = meta->record;
/* We need to set value_rec for all root types, but no need
* to check ownership cycle for a type unless it's also a
* node type.
*/
if (!(rec->field_mask & BPF_GRAPH_NODE))
continue;
/* We need to ensure ownership acyclicity among all types. The
* proper way to do it would be to topologically sort all BTF
* IDs based on the ownership edges, since there can be multiple
* bpf_{list_head,rb_node} in a type. Instead, we use the
* following resaoning:
*
* - A type can only be owned by another type in user BTF if it
* has a bpf_{list,rb}_node. Let's call these node types.
* - A type can only _own_ another type in user BTF if it has a
* bpf_{list_head,rb_root}. Let's call these root types.
*
* We ensure that if a type is both a root and node, its
* element types cannot be root types.
*
* To ensure acyclicity:
*
* When A is an root type but not a node, its ownership
* chain can be:
* A -> B -> C
* Where:
* - A is an root, e.g. has bpf_rb_root.
* - B is both a root and node, e.g. has bpf_rb_node and
* bpf_list_head.
* - C is only an root, e.g. has bpf_list_node
*
* When A is both a root and node, some other type already
* owns it in the BTF domain, hence it can not own
* another root type through any of the ownership edges.
* A -> B
* Where:
* - A is both an root and node.
* - B is only an node.
*/
if (meta->record->field_mask & BPF_GRAPH_ROOT)
return -ELOOP;
}
return 0;
}
static void __btf_struct_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct btf_show *show)
{
const struct btf_member *member;
void *safe_data;
u32 i;
safe_data = btf_show_start_struct_type(show, t, type_id, data);
if (!safe_data)
return;
for_each_member(i, t, member) {
const struct btf_type *member_type = btf_type_by_id(btf,
member->type);
const struct btf_kind_operations *ops;
u32 member_offset, bitfield_size;
u32 bytes_offset;
u8 bits8_offset;
btf_show_start_member(show, member);
member_offset = __btf_member_bit_offset(t, member);
bitfield_size = __btf_member_bitfield_size(t, member);
bytes_offset = BITS_ROUNDDOWN_BYTES(member_offset);
bits8_offset = BITS_PER_BYTE_MASKED(member_offset);
if (bitfield_size) {
safe_data = btf_show_start_type(show, member_type,
member->type,
data + bytes_offset);
if (safe_data)
btf_bitfield_show(safe_data,
bits8_offset,
bitfield_size, show);
btf_show_end_type(show);
} else {
ops = btf_type_ops(member_type);
ops->show(btf, member_type, member->type,
data + bytes_offset, bits8_offset, show);
}
btf_show_end_member(show);
}
btf_show_end_struct_type(show);
}
static void btf_struct_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct btf_show *show)
{
const struct btf_member *m = show->state.member;
/*
* First check if any members would be shown (are non-zero).
* See comments above "struct btf_show" definition for more
* details on how this works at a high-level.
*/
if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
if (!show->state.depth_check) {
show->state.depth_check = show->state.depth + 1;
show->state.depth_to_show = 0;
}
__btf_struct_show(btf, t, type_id, data, bits_offset, show);
/* Restore saved member data here */
show->state.member = m;
if (show->state.depth_check != show->state.depth + 1)
return;
show->state.depth_check = 0;
if (show->state.depth_to_show <= show->state.depth)
return;
/*
* Reaching here indicates we have recursed and found
* non-zero child values.
*/
}
__btf_struct_show(btf, t, type_id, data, bits_offset, show);
}
static const struct btf_kind_operations struct_ops = {
.check_meta = btf_struct_check_meta,
.resolve = btf_struct_resolve,
.check_member = btf_struct_check_member,
.check_kflag_member = btf_generic_check_kflag_member,
.log_details = btf_struct_log,
.show = btf_struct_show,
};
static int btf_enum_check_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
u32 struct_bits_off = member->offset;
u32 struct_size, bytes_offset;
if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
btf_verifier_log_member(env, struct_type, member,
"Member is not byte aligned");
return -EINVAL;
}
struct_size = struct_type->size;
bytes_offset = BITS_ROUNDDOWN_BYTES(struct_bits_off);
if (struct_size - bytes_offset < member_type->size) {
btf_verifier_log_member(env, struct_type, member,
"Member exceeds struct_size");
return -EINVAL;
}
return 0;
}
static int btf_enum_check_kflag_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
u32 struct_bits_off, nr_bits, bytes_end, struct_size;
u32 int_bitsize = sizeof(int) * BITS_PER_BYTE;
struct_bits_off = BTF_MEMBER_BIT_OFFSET(member->offset);
nr_bits = BTF_MEMBER_BITFIELD_SIZE(member->offset);
if (!nr_bits) {
if (BITS_PER_BYTE_MASKED(struct_bits_off)) {
btf_verifier_log_member(env, struct_type, member,
"Member is not byte aligned");
return -EINVAL;
}
nr_bits = int_bitsize;
} else if (nr_bits > int_bitsize) {
btf_verifier_log_member(env, struct_type, member,
"Invalid member bitfield_size");
return -EINVAL;
}
struct_size = struct_type->size;
bytes_end = BITS_ROUNDUP_BYTES(struct_bits_off + nr_bits);
if (struct_size < bytes_end) {
btf_verifier_log_member(env, struct_type, member,
"Member exceeds struct_size");
return -EINVAL;
}
return 0;
}
static s32 btf_enum_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
const struct btf_enum *enums = btf_type_enum(t);
struct btf *btf = env->btf;
const char *fmt_str;
u16 i, nr_enums;
u32 meta_needed;
nr_enums = btf_type_vlen(t);
meta_needed = nr_enums * sizeof(*enums);
if (meta_left < meta_needed) {
btf_verifier_log_basic(env, t,
"meta_left:%u meta_needed:%u",
meta_left, meta_needed);
return -EINVAL;
}
if (t->size > 8 || !is_power_of_2(t->size)) {
btf_verifier_log_type(env, t, "Unexpected size");
return -EINVAL;
}
/* enum type either no name or a valid one */
if (t->name_off &&
!btf_name_valid_identifier(env->btf, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
for (i = 0; i < nr_enums; i++) {
if (!btf_name_offset_valid(btf, enums[i].name_off)) {
btf_verifier_log(env, "\tInvalid name_offset:%u",
enums[i].name_off);
return -EINVAL;
}
/* enum member must have a valid name */
if (!enums[i].name_off ||
!btf_name_valid_identifier(btf, enums[i].name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
if (env->log.level == BPF_LOG_KERNEL)
continue;
fmt_str = btf_type_kflag(t) ? "\t%s val=%d\n" : "\t%s val=%u\n";
btf_verifier_log(env, fmt_str,
__btf_name_by_offset(btf, enums[i].name_off),
enums[i].val);
}
return meta_needed;
}
static void btf_enum_log(struct btf_verifier_env *env,
const struct btf_type *t)
{
btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
}
static void btf_enum_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct btf_show *show)
{
const struct btf_enum *enums = btf_type_enum(t);
u32 i, nr_enums = btf_type_vlen(t);
void *safe_data;
int v;
safe_data = btf_show_start_type(show, t, type_id, data);
if (!safe_data)
return;
v = *(int *)safe_data;
for (i = 0; i < nr_enums; i++) {
if (v != enums[i].val)
continue;
btf_show_type_value(show, "%s",
__btf_name_by_offset(btf,
enums[i].name_off));
btf_show_end_type(show);
return;
}
if (btf_type_kflag(t))
btf_show_type_value(show, "%d", v);
else
btf_show_type_value(show, "%u", v);
btf_show_end_type(show);
}
static const struct btf_kind_operations enum_ops = {
.check_meta = btf_enum_check_meta,
.resolve = btf_df_resolve,
.check_member = btf_enum_check_member,
.check_kflag_member = btf_enum_check_kflag_member,
.log_details = btf_enum_log,
.show = btf_enum_show,
};
static s32 btf_enum64_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
const struct btf_enum64 *enums = btf_type_enum64(t);
struct btf *btf = env->btf;
const char *fmt_str;
u16 i, nr_enums;
u32 meta_needed;
nr_enums = btf_type_vlen(t);
meta_needed = nr_enums * sizeof(*enums);
if (meta_left < meta_needed) {
btf_verifier_log_basic(env, t,
"meta_left:%u meta_needed:%u",
meta_left, meta_needed);
return -EINVAL;
}
if (t->size > 8 || !is_power_of_2(t->size)) {
btf_verifier_log_type(env, t, "Unexpected size");
return -EINVAL;
}
/* enum type either no name or a valid one */
if (t->name_off &&
!btf_name_valid_identifier(env->btf, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
for (i = 0; i < nr_enums; i++) {
if (!btf_name_offset_valid(btf, enums[i].name_off)) {
btf_verifier_log(env, "\tInvalid name_offset:%u",
enums[i].name_off);
return -EINVAL;
}
/* enum member must have a valid name */
if (!enums[i].name_off ||
!btf_name_valid_identifier(btf, enums[i].name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
if (env->log.level == BPF_LOG_KERNEL)
continue;
fmt_str = btf_type_kflag(t) ? "\t%s val=%lld\n" : "\t%s val=%llu\n";
btf_verifier_log(env, fmt_str,
__btf_name_by_offset(btf, enums[i].name_off),
btf_enum64_value(enums + i));
}
return meta_needed;
}
static void btf_enum64_show(const struct btf *btf, const struct btf_type *t,
u32 type_id, void *data, u8 bits_offset,
struct btf_show *show)
{
const struct btf_enum64 *enums = btf_type_enum64(t);
u32 i, nr_enums = btf_type_vlen(t);
void *safe_data;
s64 v;
safe_data = btf_show_start_type(show, t, type_id, data);
if (!safe_data)
return;
v = *(u64 *)safe_data;
for (i = 0; i < nr_enums; i++) {
if (v != btf_enum64_value(enums + i))
continue;
btf_show_type_value(show, "%s",
__btf_name_by_offset(btf,
enums[i].name_off));
btf_show_end_type(show);
return;
}
if (btf_type_kflag(t))
btf_show_type_value(show, "%lld", v);
else
btf_show_type_value(show, "%llu", v);
btf_show_end_type(show);
}
static const struct btf_kind_operations enum64_ops = {
.check_meta = btf_enum64_check_meta,
.resolve = btf_df_resolve,
.check_member = btf_enum_check_member,
.check_kflag_member = btf_enum_check_kflag_member,
.log_details = btf_enum_log,
.show = btf_enum64_show,
};
static s32 btf_func_proto_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
u32 meta_needed = btf_type_vlen(t) * sizeof(struct btf_param);
if (meta_left < meta_needed) {
btf_verifier_log_basic(env, t,
"meta_left:%u meta_needed:%u",
meta_left, meta_needed);
return -EINVAL;
}
if (t->name_off) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
if (btf_type_kflag(t)) {
btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
return meta_needed;
}
static void btf_func_proto_log(struct btf_verifier_env *env,
const struct btf_type *t)
{
const struct btf_param *args = (const struct btf_param *)(t + 1);
u16 nr_args = btf_type_vlen(t), i;
btf_verifier_log(env, "return=%u args=(", t->type);
if (!nr_args) {
btf_verifier_log(env, "void");
goto done;
}
if (nr_args == 1 && !args[0].type) {
/* Only one vararg */
btf_verifier_log(env, "vararg");
goto done;
}
btf_verifier_log(env, "%u %s", args[0].type,
__btf_name_by_offset(env->btf,
args[0].name_off));
for (i = 1; i < nr_args - 1; i++)
btf_verifier_log(env, ", %u %s", args[i].type,
__btf_name_by_offset(env->btf,
args[i].name_off));
if (nr_args > 1) {
const struct btf_param *last_arg = &args[nr_args - 1];
if (last_arg->type)
btf_verifier_log(env, ", %u %s", last_arg->type,
__btf_name_by_offset(env->btf,
last_arg->name_off));
else
btf_verifier_log(env, ", vararg");
}
done:
btf_verifier_log(env, ")");
}
static const struct btf_kind_operations func_proto_ops = {
.check_meta = btf_func_proto_check_meta,
.resolve = btf_df_resolve,
/*
* BTF_KIND_FUNC_PROTO cannot be directly referred by
* a struct's member.
*
* It should be a function pointer instead.
* (i.e. struct's member -> BTF_KIND_PTR -> BTF_KIND_FUNC_PROTO)
*
* Hence, there is no btf_func_check_member().
*/
.check_member = btf_df_check_member,
.check_kflag_member = btf_df_check_kflag_member,
.log_details = btf_func_proto_log,
.show = btf_df_show,
};
static s32 btf_func_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
if (!t->name_off ||
!btf_name_valid_identifier(env->btf, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
if (btf_type_vlen(t) > BTF_FUNC_GLOBAL) {
btf_verifier_log_type(env, t, "Invalid func linkage");
return -EINVAL;
}
if (btf_type_kflag(t)) {
btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
return 0;
}
static int btf_func_resolve(struct btf_verifier_env *env,
const struct resolve_vertex *v)
{
const struct btf_type *t = v->t;
u32 next_type_id = t->type;
int err;
err = btf_func_check(env, t);
if (err)
return err;
env_stack_pop_resolved(env, next_type_id, 0);
return 0;
}
static const struct btf_kind_operations func_ops = {
.check_meta = btf_func_check_meta,
.resolve = btf_func_resolve,
.check_member = btf_df_check_member,
.check_kflag_member = btf_df_check_kflag_member,
.log_details = btf_ref_type_log,
.show = btf_df_show,
};
static s32 btf_var_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
const struct btf_var *var;
u32 meta_needed = sizeof(*var);
if (meta_left < meta_needed) {
btf_verifier_log_basic(env, t,
"meta_left:%u meta_needed:%u",
meta_left, meta_needed);
return -EINVAL;
}
if (btf_type_vlen(t)) {
btf_verifier_log_type(env, t, "vlen != 0");
return -EINVAL;
}
if (btf_type_kflag(t)) {
btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
return -EINVAL;
}
if (!t->name_off ||
!btf_name_valid_identifier(env->btf, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
/* A var cannot be in type void */
if (!t->type || !BTF_TYPE_ID_VALID(t->type)) {
btf_verifier_log_type(env, t, "Invalid type_id");
return -EINVAL;
}
var = btf_type_var(t);
if (var->linkage != BTF_VAR_STATIC &&
var->linkage != BTF_VAR_GLOBAL_ALLOCATED) {
btf_verifier_log_type(env, t, "Linkage not supported");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
return meta_needed;
}
static void btf_var_log(struct btf_verifier_env *env, const struct btf_type *t)
{
const struct btf_var *var = btf_type_var(t);
btf_verifier_log(env, "type_id=%u linkage=%u", t->type, var->linkage);
}
static const struct btf_kind_operations var_ops = {
.check_meta = btf_var_check_meta,
.resolve = btf_var_resolve,
.check_member = btf_df_check_member,
.check_kflag_member = btf_df_check_kflag_member,
.log_details = btf_var_log,
.show = btf_var_show,
};
static s32 btf_datasec_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
const struct btf_var_secinfo *vsi;
u64 last_vsi_end_off = 0, sum = 0;
u32 i, meta_needed;
meta_needed = btf_type_vlen(t) * sizeof(*vsi);
if (meta_left < meta_needed) {
btf_verifier_log_basic(env, t,
"meta_left:%u meta_needed:%u",
meta_left, meta_needed);
return -EINVAL;
}
if (!t->size) {
btf_verifier_log_type(env, t, "size == 0");
return -EINVAL;
}
if (btf_type_kflag(t)) {
btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
return -EINVAL;
}
if (!t->name_off ||
!btf_name_valid_section(env->btf, t->name_off)) {
btf_verifier_log_type(env, t, "Invalid name");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
for_each_vsi(i, t, vsi) {
/* A var cannot be in type void */
if (!vsi->type || !BTF_TYPE_ID_VALID(vsi->type)) {
btf_verifier_log_vsi(env, t, vsi,
"Invalid type_id");
return -EINVAL;
}
if (vsi->offset < last_vsi_end_off || vsi->offset >= t->size) {
btf_verifier_log_vsi(env, t, vsi,
"Invalid offset");
return -EINVAL;
}
if (!vsi->size || vsi->size > t->size) {
btf_verifier_log_vsi(env, t, vsi,
"Invalid size");
return -EINVAL;
}
last_vsi_end_off = vsi->offset + vsi->size;
if (last_vsi_end_off > t->size) {
btf_verifier_log_vsi(env, t, vsi,
"Invalid offset+size");
return -EINVAL;
}
btf_verifier_log_vsi(env, t, vsi, NULL);
sum += vsi->size;
}
if (t->size < sum) {
btf_verifier_log_type(env, t, "Invalid btf_info size");
return -EINVAL;
}
return meta_needed;
}
static int btf_datasec_resolve(struct btf_verifier_env *env,
const struct resolve_vertex *v)
{
const struct btf_var_secinfo *vsi;
struct btf *btf = env->btf;
u16 i;
env->resolve_mode = RESOLVE_TBD;
for_each_vsi_from(i, v->next_member, v->t, vsi) {
u32 var_type_id = vsi->type, type_id, type_size = 0;
const struct btf_type *var_type = btf_type_by_id(env->btf,
var_type_id);
if (!var_type || !btf_type_is_var(var_type)) {
btf_verifier_log_vsi(env, v->t, vsi,
"Not a VAR kind member");
return -EINVAL;
}
if (!env_type_is_resolve_sink(env, var_type) &&
!env_type_is_resolved(env, var_type_id)) {
env_stack_set_next_member(env, i + 1);
return env_stack_push(env, var_type, var_type_id);
}
type_id = var_type->type;
if (!btf_type_id_size(btf, &type_id, &type_size)) {
btf_verifier_log_vsi(env, v->t, vsi, "Invalid type");
return -EINVAL;
}
if (vsi->size < type_size) {
btf_verifier_log_vsi(env, v->t, vsi, "Invalid size");
return -EINVAL;
}
}
env_stack_pop_resolved(env, 0, 0);
return 0;
}
static void btf_datasec_log(struct btf_verifier_env *env,
const struct btf_type *t)
{
btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
}
static void btf_datasec_show(const struct btf *btf,
const struct btf_type *t, u32 type_id,
void *data, u8 bits_offset,
struct btf_show *show)
{
const struct btf_var_secinfo *vsi;
const struct btf_type *var;
u32 i;
if (!btf_show_start_type(show, t, type_id, data))
return;
btf_show_type_value(show, "section (\"%s\") = {",
__btf_name_by_offset(btf, t->name_off));
for_each_vsi(i, t, vsi) {
var = btf_type_by_id(btf, vsi->type);
if (i)
btf_show(show, ",");
btf_type_ops(var)->show(btf, var, vsi->type,
data + vsi->offset, bits_offset, show);
}
btf_show_end_type(show);
}
static const struct btf_kind_operations datasec_ops = {
.check_meta = btf_datasec_check_meta,
.resolve = btf_datasec_resolve,
.check_member = btf_df_check_member,
.check_kflag_member = btf_df_check_kflag_member,
.log_details = btf_datasec_log,
.show = btf_datasec_show,
};
static s32 btf_float_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
if (btf_type_vlen(t)) {
btf_verifier_log_type(env, t, "vlen != 0");
return -EINVAL;
}
if (btf_type_kflag(t)) {
btf_verifier_log_type(env, t, "Invalid btf_info kind_flag");
return -EINVAL;
}
if (t->size != 2 && t->size != 4 && t->size != 8 && t->size != 12 &&
t->size != 16) {
btf_verifier_log_type(env, t, "Invalid type_size");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
return 0;
}
static int btf_float_check_member(struct btf_verifier_env *env,
const struct btf_type *struct_type,
const struct btf_member *member,
const struct btf_type *member_type)
{
u64 start_offset_bytes;
u64 end_offset_bytes;
u64 misalign_bits;
u64 align_bytes;
u64 align_bits;
/* Different architectures have different alignment requirements, so
* here we check only for the reasonable minimum. This way we ensure
* that types after CO-RE can pass the kernel BTF verifier.
*/
align_bytes = min_t(u64, sizeof(void *), member_type->size);
align_bits = align_bytes * BITS_PER_BYTE;
div64_u64_rem(member->offset, align_bits, &misalign_bits);
if (misalign_bits) {
btf_verifier_log_member(env, struct_type, member,
"Member is not properly aligned");
return -EINVAL;
}
start_offset_bytes = member->offset / BITS_PER_BYTE;
end_offset_bytes = start_offset_bytes + member_type->size;
if (end_offset_bytes > struct_type->size) {
btf_verifier_log_member(env, struct_type, member,
"Member exceeds struct_size");
return -EINVAL;
}
return 0;
}
static void btf_float_log(struct btf_verifier_env *env,
const struct btf_type *t)
{
btf_verifier_log(env, "size=%u", t->size);
}
static const struct btf_kind_operations float_ops = {
.check_meta = btf_float_check_meta,
.resolve = btf_df_resolve,
.check_member = btf_float_check_member,
.check_kflag_member = btf_generic_check_kflag_member,
.log_details = btf_float_log,
.show = btf_df_show,
};
static s32 btf_decl_tag_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
const struct btf_decl_tag *tag;
u32 meta_needed = sizeof(*tag);
s32 component_idx;
const char *value;
if (meta_left < meta_needed) {
btf_verifier_log_basic(env, t,
"meta_left:%u meta_needed:%u",
meta_left, meta_needed);
return -EINVAL;
}
value = btf_name_by_offset(env->btf, t->name_off);
if (!value || !value[0]) {
btf_verifier_log_type(env, t, "Invalid value");
return -EINVAL;
}
if (btf_type_vlen(t)) {
btf_verifier_log_type(env, t, "vlen != 0");
return -EINVAL;
}
component_idx = btf_type_decl_tag(t)->component_idx;
if (component_idx < -1) {
btf_verifier_log_type(env, t, "Invalid component_idx");
return -EINVAL;
}
btf_verifier_log_type(env, t, NULL);
return meta_needed;
}
static int btf_decl_tag_resolve(struct btf_verifier_env *env,
const struct resolve_vertex *v)
{
const struct btf_type *next_type;
const struct btf_type *t = v->t;
u32 next_type_id = t->type;
struct btf *btf = env->btf;
s32 component_idx;
u32 vlen;
next_type = btf_type_by_id(btf, next_type_id);
if (!next_type || !btf_type_is_decl_tag_target(next_type)) {
btf_verifier_log_type(env, v->t, "Invalid type_id");
return -EINVAL;
}
if (!env_type_is_resolve_sink(env, next_type) &&
!env_type_is_resolved(env, next_type_id))
return env_stack_push(env, next_type, next_type_id);
component_idx = btf_type_decl_tag(t)->component_idx;
if (component_idx != -1) {
if (btf_type_is_var(next_type) || btf_type_is_typedef(next_type)) {
btf_verifier_log_type(env, v->t, "Invalid component_idx");
return -EINVAL;
}
if (btf_type_is_struct(next_type)) {
vlen = btf_type_vlen(next_type);
} else {
/* next_type should be a function */
next_type = btf_type_by_id(btf, next_type->type);
vlen = btf_type_vlen(next_type);
}
if ((u32)component_idx >= vlen) {
btf_verifier_log_type(env, v->t, "Invalid component_idx");
return -EINVAL;
}
}
env_stack_pop_resolved(env, next_type_id, 0);
return 0;
}
static void btf_decl_tag_log(struct btf_verifier_env *env, const struct btf_type *t)
{
btf_verifier_log(env, "type=%u component_idx=%d", t->type,
btf_type_decl_tag(t)->component_idx);
}
static const struct btf_kind_operations decl_tag_ops = {
.check_meta = btf_decl_tag_check_meta,
.resolve = btf_decl_tag_resolve,
.check_member = btf_df_check_member,
.check_kflag_member = btf_df_check_kflag_member,
.log_details = btf_decl_tag_log,
.show = btf_df_show,
};
static int btf_func_proto_check(struct btf_verifier_env *env,
const struct btf_type *t)
{
const struct btf_type *ret_type;
const struct btf_param *args;
const struct btf *btf;
u16 nr_args, i;
int err;
btf = env->btf;
args = (const struct btf_param *)(t + 1);
nr_args = btf_type_vlen(t);
/* Check func return type which could be "void" (t->type == 0) */
if (t->type) {
u32 ret_type_id = t->type;
ret_type = btf_type_by_id(btf, ret_type_id);
if (!ret_type) {
btf_verifier_log_type(env, t, "Invalid return type");
return -EINVAL;
}
if (btf_type_is_resolve_source_only(ret_type)) {
btf_verifier_log_type(env, t, "Invalid return type");
return -EINVAL;
}
if (btf_type_needs_resolve(ret_type) &&
!env_type_is_resolved(env, ret_type_id)) {
err = btf_resolve(env, ret_type, ret_type_id);
if (err)
return err;
}
/* Ensure the return type is a type that has a size */
if (!btf_type_id_size(btf, &ret_type_id, NULL)) {
btf_verifier_log_type(env, t, "Invalid return type");
return -EINVAL;
}
}
if (!nr_args)
return 0;
/* Last func arg type_id could be 0 if it is a vararg */
if (!args[nr_args - 1].type) {
if (args[nr_args - 1].name_off) {
btf_verifier_log_type(env, t, "Invalid arg#%u",
nr_args);
return -EINVAL;
}
nr_args--;
}
for (i = 0; i < nr_args; i++) {
const struct btf_type *arg_type;
u32 arg_type_id;
arg_type_id = args[i].type;
arg_type = btf_type_by_id(btf, arg_type_id);
if (!arg_type) {
btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
return -EINVAL;
}
if (btf_type_is_resolve_source_only(arg_type)) {
btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
return -EINVAL;
}
if (args[i].name_off &&
(!btf_name_offset_valid(btf, args[i].name_off) ||
!btf_name_valid_identifier(btf, args[i].name_off))) {
btf_verifier_log_type(env, t,
"Invalid arg#%u", i + 1);
return -EINVAL;
}
if (btf_type_needs_resolve(arg_type) &&
!env_type_is_resolved(env, arg_type_id)) {
err = btf_resolve(env, arg_type, arg_type_id);
if (err)
return err;
}
if (!btf_type_id_size(btf, &arg_type_id, NULL)) {
btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
return -EINVAL;
}
}
return 0;
}
static int btf_func_check(struct btf_verifier_env *env,
const struct btf_type *t)
{
const struct btf_type *proto_type;
const struct btf_param *args;
const struct btf *btf;
u16 nr_args, i;
btf = env->btf;
proto_type = btf_type_by_id(btf, t->type);
if (!proto_type || !btf_type_is_func_proto(proto_type)) {
btf_verifier_log_type(env, t, "Invalid type_id");
return -EINVAL;
}
args = (const struct btf_param *)(proto_type + 1);
nr_args = btf_type_vlen(proto_type);
for (i = 0; i < nr_args; i++) {
if (!args[i].name_off && args[i].type) {
btf_verifier_log_type(env, t, "Invalid arg#%u", i + 1);
return -EINVAL;
}
}
return 0;
}
static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS] = {
[BTF_KIND_INT] = &int_ops,
[BTF_KIND_PTR] = &ptr_ops,
[BTF_KIND_ARRAY] = &array_ops,
[BTF_KIND_STRUCT] = &struct_ops,
[BTF_KIND_UNION] = &struct_ops,
[BTF_KIND_ENUM] = &enum_ops,
[BTF_KIND_FWD] = &fwd_ops,
[BTF_KIND_TYPEDEF] = &modifier_ops,
[BTF_KIND_VOLATILE] = &modifier_ops,
[BTF_KIND_CONST] = &modifier_ops,
[BTF_KIND_RESTRICT] = &modifier_ops,
[BTF_KIND_FUNC] = &func_ops,
[BTF_KIND_FUNC_PROTO] = &func_proto_ops,
[BTF_KIND_VAR] = &var_ops,
[BTF_KIND_DATASEC] = &datasec_ops,
[BTF_KIND_FLOAT] = &float_ops,
[BTF_KIND_DECL_TAG] = &decl_tag_ops,
[BTF_KIND_TYPE_TAG] = &modifier_ops,
[BTF_KIND_ENUM64] = &enum64_ops,
};
static s32 btf_check_meta(struct btf_verifier_env *env,
const struct btf_type *t,
u32 meta_left)
{
u32 saved_meta_left = meta_left;
s32 var_meta_size;
if (meta_left < sizeof(*t)) {
btf_verifier_log(env, "[%u] meta_left:%u meta_needed:%zu",
env->log_type_id, meta_left, sizeof(*t));
return -EINVAL;
}
meta_left -= sizeof(*t);
if (t->info & ~BTF_INFO_MASK) {
btf_verifier_log(env, "[%u] Invalid btf_info:%x",
env->log_type_id, t->info);
return -EINVAL;
}
if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX ||
BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) {
btf_verifier_log(env, "[%u] Invalid kind:%u",
env->log_type_id, BTF_INFO_KIND(t->info));
return -EINVAL;
}
if (!btf_name_offset_valid(env->btf, t->name_off)) {
btf_verifier_log(env, "[%u] Invalid name_offset:%u",
env->log_type_id, t->name_off);
return -EINVAL;
}
var_meta_size = btf_type_ops(t)->check_meta(env, t, meta_left);
if (var_meta_size < 0)
return var_meta_size;
meta_left -= var_meta_size;
return saved_meta_left - meta_left;
}
static int btf_check_all_metas(struct btf_verifier_env *env)
{
struct btf *btf = env->btf;
struct btf_header *hdr;
void *cur, *end;
hdr = &btf->hdr;
cur = btf->nohdr_data + hdr->type_off;
end = cur + hdr->type_len;
env->log_type_id = btf->base_btf ? btf->start_id : 1;
while (cur < end) {
struct btf_type *t = cur;
s32 meta_size;
meta_size = btf_check_meta(env, t, end - cur);
if (meta_size < 0)
return meta_size;
btf_add_type(env, t);
cur += meta_size;
env->log_type_id++;
}
return 0;
}
static bool btf_resolve_valid(struct btf_verifier_env *env,
const struct btf_type *t,
u32 type_id)
{
struct btf *btf = env->btf;
if (!env_type_is_resolved(env, type_id))
return false;
if (btf_type_is_struct(t) || btf_type_is_datasec(t))
return !btf_resolved_type_id(btf, type_id) &&
!btf_resolved_type_size(btf, type_id);
if (btf_type_is_decl_tag(t) || btf_type_is_func(t))
return btf_resolved_type_id(btf, type_id) &&
!btf_resolved_type_size(btf, type_id);
if (btf_type_is_modifier(t) || btf_type_is_ptr(t) ||
btf_type_is_var(t)) {
t = btf_type_id_resolve(btf, &type_id);
return t &&
!btf_type_is_modifier(t) &&
!btf_type_is_var(t) &&
!btf_type_is_datasec(t);
}
if (btf_type_is_array(t)) {
const struct btf_array *array = btf_type_array(t);
const struct btf_type *elem_type;
u32 elem_type_id = array->type;
u32 elem_size;
elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
return elem_type && !btf_type_is_modifier(elem_type) &&
(array->nelems * elem_size ==
btf_resolved_type_size(btf, type_id));
}
return false;
}
static int btf_resolve(struct btf_verifier_env *env,
const struct btf_type *t, u32 type_id)
{
u32 save_log_type_id = env->log_type_id;
const struct resolve_vertex *v;
int err = 0;
env->resolve_mode = RESOLVE_TBD;
env_stack_push(env, t, type_id);
while (!err && (v = env_stack_peak(env))) {
env->log_type_id = v->type_id;
err = btf_type_ops(v->t)->resolve(env, v);
}
env->log_type_id = type_id;
if (err == -E2BIG) {
btf_verifier_log_type(env, t,
"Exceeded max resolving depth:%u",
MAX_RESOLVE_DEPTH);
} else if (err == -EEXIST) {
btf_verifier_log_type(env, t, "Loop detected");
}
/* Final sanity check */
if (!err && !btf_resolve_valid(env, t, type_id)) {
btf_verifier_log_type(env, t, "Invalid resolve state");
err = -EINVAL;
}
env->log_type_id = save_log_type_id;
return err;
}
static int btf_check_all_types(struct btf_verifier_env *env)
{
struct btf *btf = env->btf;
const struct btf_type *t;
u32 type_id, i;
int err;
err = env_resolve_init(env);
if (err)
return err;
env->phase++;
for (i = btf->base_btf ? 0 : 1; i < btf->nr_types; i++) {
type_id = btf->start_id + i;
t = btf_type_by_id(btf, type_id);
env->log_type_id = type_id;
if (btf_type_needs_resolve(t) &&
!env_type_is_resolved(env, type_id)) {
err = btf_resolve(env, t, type_id);
if (err)
return err;
}
if (btf_type_is_func_proto(t)) {
err = btf_func_proto_check(env, t);
if (err)
return err;
}
}
return 0;
}
static int btf_parse_type_sec(struct btf_verifier_env *env)
{
const struct btf_header *hdr = &env->btf->hdr;
int err;
/* Type section must align to 4 bytes */
if (hdr->type_off & (sizeof(u32) - 1)) {
btf_verifier_log(env, "Unaligned type_off");
return -EINVAL;
}
if (!env->btf->base_btf && !hdr->type_len) {
btf_verifier_log(env, "No type found");
return -EINVAL;
}
err = btf_check_all_metas(env);
if (err)
return err;
return btf_check_all_types(env);
}
static int btf_parse_str_sec(struct btf_verifier_env *env)
{
const struct btf_header *hdr;
struct btf *btf = env->btf;
const char *start, *end;
hdr = &btf->hdr;
start = btf->nohdr_data + hdr->str_off;
end = start + hdr->str_len;
if (hdr->hdr_len < sizeof(struct btf_header) &&
end != btf->data + btf->data_size) {
btf_verifier_log(env, "String section is not at the end");
return -EINVAL;
}
btf->strings = start;
if (btf->base_btf && !hdr->str_len)
return 0;
if (!hdr->str_len || hdr->str_len - 1 > BTF_MAX_NAME_OFFSET || end[-1]) {
btf_verifier_log(env, "Invalid string section");
return -EINVAL;
}
if (!btf->base_btf && start[0]) {
btf_verifier_log(env, "Invalid string section");
return -EINVAL;
}
return 0;
}
static int btf_parse_layout_sec(struct btf_verifier_env *env)
{
const struct btf_header *hdr = &env->btf->hdr;
struct btf *btf = env->btf;
void *start, *end;
if (hdr->hdr_len < sizeof(struct btf_header) ||
hdr->layout_len == 0)
return 0;
/* Layout section must align to 4 bytes */
if (hdr->layout_off & (sizeof(u32) - 1)) {
btf_verifier_log(env, "Unaligned layout_off");
return -EINVAL;
}
start = btf->nohdr_data + hdr->layout_off;
end = start + hdr->layout_len;
if (hdr->layout_len < sizeof(struct btf_layout)) {
btf_verifier_log(env, "Layout section is too small");
return -EINVAL;
}
if (hdr->layout_len % sizeof(struct btf_layout) != 0) {
btf_verifier_log(env, "layout_len is not multiple of %zu",
sizeof(struct btf_layout));
return -EINVAL;
}
if (end > btf->data + btf->data_size) {
btf_verifier_log(env, "Layout section is too big");
return -EINVAL;
}
btf->layout = start;
return 0;
}
static const size_t btf_sec_info_offset[] = {
offsetof(struct btf_header, type_off),
offsetof(struct btf_header, str_off),
offsetof(struct btf_header, layout_off)
};
static int btf_sec_info_cmp(const void *a, const void *b)
{
const struct btf_sec_info *x = a;
const struct btf_sec_info *y = b;
return (int)(x->off - y->off) ? : (int)(x->len - y->len);
}
static int btf_check_sec_info(struct btf_verifier_env *env,
u32 btf_data_size)
{
struct btf_sec_info secs[ARRAY_SIZE(btf_sec_info_offset)];
u32 total, expected_total, i;
u32 nr_secs = ARRAY_SIZE(btf_sec_info_offset);
const struct btf_header *hdr;
const struct btf *btf;
btf = env->btf;
hdr = &btf->hdr;
if (hdr->hdr_len < sizeof(struct btf_header) || hdr->layout_len == 0)
nr_secs--;
/* Populate the secs from hdr */
for (i = 0; i < nr_secs; i++)
secs[i] = *(struct btf_sec_info *)((void *)hdr +
btf_sec_info_offset[i]);
sort(secs, nr_secs,
sizeof(struct btf_sec_info), btf_sec_info_cmp, NULL);
/* Check for gaps and overlap among sections */
total = 0;
expected_total = btf_data_size - hdr->hdr_len;
for (i = 0; i < nr_secs; i++) {
if (expected_total < secs[i].off) {
btf_verifier_log(env, "Invalid section offset");
return -EINVAL;
}
if (total < secs[i].off) {
/* gap */
btf_verifier_log(env, "Unsupported section found");
return -EINVAL;
}
if (total > secs[i].off) {
btf_verifier_log(env, "Section overlap found");
return -EINVAL;
}
if (expected_total - total < secs[i].len) {
btf_verifier_log(env,
"Total section length too long");
return -EINVAL;
}
total += secs[i].len;
}
/* There is data other than hdr and known sections */
if (expected_total != total) {
btf_verifier_log(env, "Unsupported section found");
return -EINVAL;
}
return 0;
}
static int btf_parse_hdr(struct btf_verifier_env *env)
{
u32 hdr_len, hdr_copy, btf_data_size;
const struct btf_header *hdr;
struct btf *btf;
btf = env->btf;
btf_data_size = btf->data_size;
if (btf_data_size < offsetofend(struct btf_header, hdr_len)) {
btf_verifier_log(env, "hdr_len not found");
return -EINVAL;
}
hdr = btf->data;
hdr_len = hdr->hdr_len;
if (btf_data_size < hdr_len) {
btf_verifier_log(env, "btf_header not found");
return -EINVAL;
}
/* Ensure the unsupported header fields are zero */
if (hdr_len > sizeof(btf->hdr)) {
u8 *expected_zero = btf->data + sizeof(btf->hdr);
u8 *end = btf->data + hdr_len;
for (; expected_zero < end; expected_zero++) {
if (*expected_zero) {
btf_verifier_log(env, "Unsupported btf_header");
return -E2BIG;
}
}
}
hdr_copy = min_t(u32, hdr_len, sizeof(btf->hdr));
memcpy(&btf->hdr, btf->data, hdr_copy);
hdr = &btf->hdr;
btf_verifier_log_hdr(env, btf_data_size);
if (hdr->magic != BTF_MAGIC) {
btf_verifier_log(env, "Invalid magic");
return -EINVAL;
}
if (hdr->version != BTF_VERSION) {
btf_verifier_log(env, "Unsupported version");
return -ENOTSUPP;
}
if (hdr->flags) {
btf_verifier_log(env, "Unsupported flags");
return -ENOTSUPP;
}
if (!btf->base_btf && btf_data_size == hdr->hdr_len) {
btf_verifier_log(env, "No data");
return -EINVAL;
}
return btf_check_sec_info(env, btf_data_size);
}
static const char *alloc_obj_fields[] = {
"bpf_spin_lock",
"bpf_list_head",
"bpf_list_node",
"bpf_rb_root",
"bpf_rb_node",
"bpf_refcount",
};
static struct btf_struct_metas *
btf_parse_struct_metas(struct bpf_verifier_log *log, struct btf *btf)
{
struct btf_struct_metas *tab = NULL;
struct btf_id_set *aof;
int i, n, id, ret;
BUILD_BUG_ON(offsetof(struct btf_id_set, cnt) != 0);
BUILD_BUG_ON(sizeof(struct btf_id_set) != sizeof(u32));
aof = kmalloc_obj(*aof, GFP_KERNEL | __GFP_NOWARN);
if (!aof)
return ERR_PTR(-ENOMEM);
aof->cnt = 0;
for (i = 0; i < ARRAY_SIZE(alloc_obj_fields); i++) {
/* Try to find whether this special type exists in user BTF, and
* if so remember its ID so we can easily find it among members
* of structs that we iterate in the next loop.
*/
struct btf_id_set *new_aof;
id = btf_find_by_name_kind(btf, alloc_obj_fields[i], BTF_KIND_STRUCT);
if (id < 0)
continue;
new_aof = krealloc(aof, struct_size(new_aof, ids, aof->cnt + 1),
GFP_KERNEL | __GFP_NOWARN);
if (!new_aof) {
ret = -ENOMEM;
goto free_aof;
}
aof = new_aof;
aof->ids[aof->cnt++] = id;
}
n = btf_nr_types(btf);
for (i = 1; i < n; i++) {
/* Try to find if there are kptrs in user BTF and remember their ID */
struct btf_id_set *new_aof;
struct btf_field_info tmp;
const struct btf_type *t;
t = btf_type_by_id(btf, i);
if (!t) {
ret = -EINVAL;
goto free_aof;
}
ret = btf_find_kptr(btf, t, 0, 0, &tmp, BPF_KPTR);
if (ret != BTF_FIELD_FOUND)
continue;
new_aof = krealloc(aof, struct_size(new_aof, ids, aof->cnt + 1),
GFP_KERNEL | __GFP_NOWARN);
if (!new_aof) {
ret = -ENOMEM;
goto free_aof;
}
aof = new_aof;
aof->ids[aof->cnt++] = i;
}
if (!aof->cnt) {
kfree(aof);
return NULL;
}
sort(&aof->ids, aof->cnt, sizeof(aof->ids[0]), btf_id_cmp_func, NULL);
for (i = 1; i < n; i++) {
struct btf_struct_metas *new_tab;
const struct btf_member *member;
struct btf_struct_meta *type;
struct btf_record *record;
const struct btf_type *t;
int j, tab_cnt;
t = btf_type_by_id(btf, i);
if (!__btf_type_is_struct(t))
continue;
cond_resched();
for_each_member(j, t, member) {
if (btf_id_set_contains(aof, member->type))
goto parse;
}
continue;
parse:
tab_cnt = tab ? tab->cnt : 0;
new_tab = krealloc(tab, struct_size(new_tab, types, tab_cnt + 1),
GFP_KERNEL | __GFP_NOWARN);
if (!new_tab) {
ret = -ENOMEM;
goto free;
}
if (!tab)
new_tab->cnt = 0;
tab = new_tab;
type = &tab->types[tab->cnt];
type->btf_id = i;
record = btf_parse_fields(btf, t, BPF_SPIN_LOCK | BPF_RES_SPIN_LOCK | BPF_LIST_HEAD | BPF_LIST_NODE |
BPF_RB_ROOT | BPF_RB_NODE | BPF_REFCOUNT |
BPF_KPTR, t->size);
/* The record cannot be unset, treat it as an error if so */
if (IS_ERR_OR_NULL(record)) {
ret = PTR_ERR_OR_ZERO(record) ?: -EFAULT;
goto free;
}
type->record = record;
tab->cnt++;
}
kfree(aof);
return tab;
free:
btf_struct_metas_free(tab);
free_aof:
kfree(aof);
return ERR_PTR(ret);
}
struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf, u32 btf_id)
{
struct btf_struct_metas *tab;
BUILD_BUG_ON(offsetof(struct btf_struct_meta, btf_id) != 0);
tab = btf->struct_meta_tab;
if (!tab)
return NULL;
return bsearch(&btf_id, tab->types, tab->cnt, sizeof(tab->types[0]), btf_id_cmp_func);
}
static int btf_check_type_tags(struct btf_verifier_env *env,
struct btf *btf, int start_id)
{
int i, n, good_id = start_id - 1;
bool in_tags;
n = btf_nr_types(btf);
for (i = start_id; i < n; i++) {
const struct btf_type *t;
int chain_limit = 32;
u32 cur_id = i;
t = btf_type_by_id(btf, i);
if (!t)
return -EINVAL;
if (!btf_type_is_modifier(t))
continue;
cond_resched();
in_tags = btf_type_is_type_tag(t);
while (btf_type_is_modifier(t)) {
if (!chain_limit--) {
btf_verifier_log(env, "Max chain length or cycle detected");
return -ELOOP;
}
if (btf_type_is_type_tag(t)) {
if (!in_tags) {
btf_verifier_log(env, "Type tags don't precede modifiers");
return -EINVAL;
}
} else if (in_tags) {
in_tags = false;
}
if (cur_id <= good_id)
break;
/* Move to next type */
cur_id = t->type;
t = btf_type_by_id(btf, cur_id);
if (!t)
return -EINVAL;
}
good_id = i;
}
return 0;
}
static int finalize_log(struct bpf_verifier_log *log, bpfptr_t uattr, u32 uattr_size)
{
u32 log_true_size;
int err;
err = bpf_vlog_finalize(log, &log_true_size);
if (uattr_size >= offsetofend(union bpf_attr, btf_log_true_size) &&
copy_to_bpfptr_offset(uattr, offsetof(union bpf_attr, btf_log_true_size),
&log_true_size, sizeof(log_true_size)))
err = -EFAULT;
return err;
}
static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
{
bpfptr_t btf_data = make_bpfptr(attr->btf, uattr.is_kernel);
char __user *log_ubuf = u64_to_user_ptr(attr->btf_log_buf);
struct btf_struct_metas *struct_meta_tab;
struct btf_verifier_env *env = NULL;
struct btf *btf = NULL;
u8 *data;
int err, ret;
if (attr->btf_size > BTF_MAX_SIZE)
return ERR_PTR(-E2BIG);
env = kzalloc_obj(*env, GFP_KERNEL | __GFP_NOWARN);
if (!env)
return ERR_PTR(-ENOMEM);
/* user could have requested verbose verifier output
* and supplied buffer to store the verification trace
*/
err = bpf_vlog_init(&env->log, attr->btf_log_level,
log_ubuf, attr->btf_log_size);
if (err)
goto errout_free;
btf = kzalloc_obj(*btf, GFP_KERNEL | __GFP_NOWARN);
if (!btf) {
err = -ENOMEM;
goto errout;
}
env->btf = btf;
btf->named_start_id = 0;
data = kvmalloc(attr->btf_size, GFP_KERNEL | __GFP_NOWARN);
if (!data) {
err = -ENOMEM;
goto errout;
}
btf->data = data;
btf->data_size = attr->btf_size;
if (copy_from_bpfptr(data, btf_data, attr->btf_size)) {
err = -EFAULT;
goto errout;
}
err = btf_parse_hdr(env);
if (err)
goto errout;
btf->nohdr_data = btf->data + btf->hdr.hdr_len;
err = btf_parse_str_sec(env);
if (err)
goto errout;
err = btf_parse_layout_sec(env);
if (err)
goto errout;
err = btf_parse_type_sec(env);
if (err)
goto errout;
err = btf_check_type_tags(env, btf, 1);
if (err)
goto errout;
struct_meta_tab = btf_parse_struct_metas(&env->log, btf);
if (IS_ERR(struct_meta_tab)) {
err = PTR_ERR(struct_meta_tab);
goto errout;
}
btf->struct_meta_tab = struct_meta_tab;
if (struct_meta_tab) {
int i;
for (i = 0; i < struct_meta_tab->cnt; i++) {
err = btf_check_and_fixup_fields(btf, struct_meta_tab->types[i].record);
if (err < 0)
goto errout_meta;
}
}
err = finalize_log(&env->log, uattr, uattr_size);
if (err)
goto errout_free;
btf_verifier_env_free(env);
refcount_set(&btf->refcnt, 1);
return btf;
errout_meta:
btf_free_struct_meta_tab(btf);
errout:
/* overwrite err with -ENOSPC or -EFAULT */
ret = finalize_log(&env->log, uattr, uattr_size);
if (ret)
err = ret;
errout_free:
btf_verifier_env_free(env);
if (btf)
btf_free(btf);
return ERR_PTR(err);
}
extern char __start_BTF[];
extern char __stop_BTF[];
extern struct btf *btf_vmlinux;
#define BPF_MAP_TYPE(_id, _ops)
#define BPF_LINK_TYPE(_id, _name)
static union {
struct bpf_ctx_convert {
#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
prog_ctx_type _id##_prog; \
kern_ctx_type _id##_kern;
#include <linux/bpf_types.h>
#undef BPF_PROG_TYPE
} *__t;
/* 't' is written once under lock. Read many times. */
const struct btf_type *t;
} bpf_ctx_convert;
enum {
#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
__ctx_convert##_id,
#include <linux/bpf_types.h>
#undef BPF_PROG_TYPE
__ctx_convert_unused, /* to avoid empty enum in extreme .config */
};
static u8 bpf_ctx_convert_map[] = {
#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
[_id] = __ctx_convert##_id,
#include <linux/bpf_types.h>
#undef BPF_PROG_TYPE
0, /* avoid empty array */
};
#undef BPF_MAP_TYPE
#undef BPF_LINK_TYPE
static const struct btf_type *find_canonical_prog_ctx_type(enum bpf_prog_type prog_type)
{
const struct btf_type *conv_struct;
const struct btf_member *ctx_type;
conv_struct = bpf_ctx_convert.t;
if (!conv_struct)
return NULL;
/* prog_type is valid bpf program type. No need for bounds check. */
ctx_type = btf_type_member(conv_struct) + bpf_ctx_convert_map[prog_type] * 2;
/* ctx_type is a pointer to prog_ctx_type in vmlinux.
* Like 'struct __sk_buff'
*/
return btf_type_by_id(btf_vmlinux, ctx_type->type);
}
static int find_kern_ctx_type_id(enum bpf_prog_type prog_type)
{
const struct btf_type *conv_struct;
const struct btf_member *ctx_type;
conv_struct = bpf_ctx_convert.t;
if (!conv_struct)
return -EFAULT;
/* prog_type is valid bpf program type. No need for bounds check. */
ctx_type = btf_type_member(conv_struct) + bpf_ctx_convert_map[prog_type] * 2 + 1;
/* ctx_type is a pointer to prog_ctx_type in vmlinux.
* Like 'struct sk_buff'
*/
return ctx_type->type;
}
bool btf_is_projection_of(const char *pname, const char *tname)
{
if (strcmp(pname, "__sk_buff") == 0 && strcmp(tname, "sk_buff") == 0)
return true;
if (strcmp(pname, "xdp_md") == 0 && strcmp(tname, "xdp_buff") == 0)
return true;
return false;
}
bool btf_is_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg)
{
const struct btf_type *ctx_type;
const char *tname, *ctx_tname;
t = btf_type_by_id(btf, t->type);
/* KPROBE programs allow bpf_user_pt_regs_t typedef, which we need to
* check before we skip all the typedef below.
*/
if (prog_type == BPF_PROG_TYPE_KPROBE) {
while (btf_type_is_modifier(t) && !btf_type_is_typedef(t))
t = btf_type_by_id(btf, t->type);
if (btf_type_is_typedef(t)) {
tname = btf_name_by_offset(btf, t->name_off);
if (tname && strcmp(tname, "bpf_user_pt_regs_t") == 0)
return true;
}
}
while (btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);
if (!btf_type_is_struct(t)) {
/* Only pointer to struct is supported for now.
* That means that BPF_PROG_TYPE_TRACEPOINT with BTF
* is not supported yet.
* BPF_PROG_TYPE_RAW_TRACEPOINT is fine.
*/
return false;
}
tname = btf_name_by_offset(btf, t->name_off);
if (!tname) {
bpf_log(log, "arg#%d struct doesn't have a name\n", arg);
return false;
}
ctx_type = find_canonical_prog_ctx_type(prog_type);
if (!ctx_type) {
bpf_log(log, "btf_vmlinux is malformed\n");
/* should not happen */
return false;
}
again:
ctx_tname = btf_name_by_offset(btf_vmlinux, ctx_type->name_off);
if (!ctx_tname) {
/* should not happen */
bpf_log(log, "Please fix kernel include/linux/bpf_types.h\n");
return false;
}
/* program types without named context types work only with arg:ctx tag */
if (ctx_tname[0] == '\0')
return false;
/* only compare that prog's ctx type name is the same as
* kernel expects. No need to compare field by field.
* It's ok for bpf prog to do:
* struct __sk_buff {};
* int socket_filter_bpf_prog(struct __sk_buff *skb)
* { // no fields of skb are ever used }
*/
if (btf_is_projection_of(ctx_tname, tname))
return true;
if (strcmp(ctx_tname, tname)) {
/* bpf_user_pt_regs_t is a typedef, so resolve it to
* underlying struct and check name again
*/
if (!btf_type_is_modifier(ctx_type))
return false;
while (btf_type_is_modifier(ctx_type))
ctx_type = btf_type_by_id(btf_vmlinux, ctx_type->type);
goto again;
}
return true;
}
/* forward declarations for arch-specific underlying types of
* bpf_user_pt_regs_t; this avoids the need for arch-specific #ifdef
* compilation guards below for BPF_PROG_TYPE_PERF_EVENT checks, but still
* works correctly with __builtin_types_compatible_p() on respective
* architectures
*/
struct user_regs_struct;
struct user_pt_regs;
static int btf_validate_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, int arg,
enum bpf_prog_type prog_type,
enum bpf_attach_type attach_type)
{
const struct btf_type *ctx_type;
const char *tname, *ctx_tname;
if (!btf_is_ptr(t)) {
bpf_log(log, "arg#%d type isn't a pointer\n", arg);
return -EINVAL;
}
t = btf_type_by_id(btf, t->type);
/* KPROBE and PERF_EVENT programs allow bpf_user_pt_regs_t typedef */
if (prog_type == BPF_PROG_TYPE_KPROBE || prog_type == BPF_PROG_TYPE_PERF_EVENT) {
while (btf_type_is_modifier(t) && !btf_type_is_typedef(t))
t = btf_type_by_id(btf, t->type);
if (btf_type_is_typedef(t)) {
tname = btf_name_by_offset(btf, t->name_off);
if (tname && strcmp(tname, "bpf_user_pt_regs_t") == 0)
return 0;
}
}
/* all other program types don't use typedefs for context type */
while (btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);
/* `void *ctx __arg_ctx` is always valid */
if (btf_type_is_void(t))
return 0;
tname = btf_name_by_offset(btf, t->name_off);
if (str_is_empty(tname)) {
bpf_log(log, "arg#%d type doesn't have a name\n", arg);
return -EINVAL;
}
/* special cases */
switch (prog_type) {
case BPF_PROG_TYPE_KPROBE:
if (__btf_type_is_struct(t) && strcmp(tname, "pt_regs") == 0)
return 0;
break;
case BPF_PROG_TYPE_PERF_EVENT:
if (__builtin_types_compatible_p(bpf_user_pt_regs_t, struct pt_regs) &&
__btf_type_is_struct(t) && strcmp(tname, "pt_regs") == 0)
return 0;
if (__builtin_types_compatible_p(bpf_user_pt_regs_t, struct user_pt_regs) &&
__btf_type_is_struct(t) && strcmp(tname, "user_pt_regs") == 0)
return 0;
if (__builtin_types_compatible_p(bpf_user_pt_regs_t, struct user_regs_struct) &&
__btf_type_is_struct(t) && strcmp(tname, "user_regs_struct") == 0)
return 0;
break;
case BPF_PROG_TYPE_RAW_TRACEPOINT:
case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:
/* allow u64* as ctx */
if (btf_is_int(t) && t->size == 8)
return 0;
break;
case BPF_PROG_TYPE_TRACING:
switch (attach_type) {
case BPF_TRACE_RAW_TP:
/* tp_btf program is TRACING, so need special case here */
if (__btf_type_is_struct(t) &&
strcmp(tname, "bpf_raw_tracepoint_args") == 0)
return 0;
/* allow u64* as ctx */
if (btf_is_int(t) && t->size == 8)
return 0;
break;
case BPF_TRACE_ITER:
/* allow struct bpf_iter__xxx types only */
if (__btf_type_is_struct(t) &&
strncmp(tname, "bpf_iter__", sizeof("bpf_iter__") - 1) == 0)
return 0;
break;
case BPF_TRACE_FENTRY:
case BPF_TRACE_FEXIT:
case BPF_MODIFY_RETURN:
case BPF_TRACE_FSESSION:
/* allow u64* as ctx */
if (btf_is_int(t) && t->size == 8)
return 0;
break;
default:
break;
}
break;
case BPF_PROG_TYPE_LSM:
case BPF_PROG_TYPE_STRUCT_OPS:
/* allow u64* as ctx */
if (btf_is_int(t) && t->size == 8)
return 0;
break;
case BPF_PROG_TYPE_TRACEPOINT:
case BPF_PROG_TYPE_SYSCALL:
case BPF_PROG_TYPE_EXT:
return 0; /* anything goes */
default:
break;
}
ctx_type = find_canonical_prog_ctx_type(prog_type);
if (!ctx_type) {
/* should not happen */
bpf_log(log, "btf_vmlinux is malformed\n");
return -EINVAL;
}
/* resolve typedefs and check that underlying structs are matching as well */
while (btf_type_is_modifier(ctx_type))
ctx_type = btf_type_by_id(btf_vmlinux, ctx_type->type);
/* if program type doesn't have distinctly named struct type for
* context, then __arg_ctx argument can only be `void *`, which we
* already checked above
*/
if (!__btf_type_is_struct(ctx_type)) {
bpf_log(log, "arg#%d should be void pointer\n", arg);
return -EINVAL;
}
ctx_tname = btf_name_by_offset(btf_vmlinux, ctx_type->name_off);
if (!__btf_type_is_struct(t) || strcmp(ctx_tname, tname) != 0) {
bpf_log(log, "arg#%d should be `struct %s *`\n", arg, ctx_tname);
return -EINVAL;
}
return 0;
}
static int btf_translate_to_vmlinux(struct bpf_verifier_log *log,
struct btf *btf,
const struct btf_type *t,
enum bpf_prog_type prog_type,
int arg)
{
if (!btf_is_prog_ctx_type(log, btf, t, prog_type, arg))
return -ENOENT;
return find_kern_ctx_type_id(prog_type);
}
int get_kern_ctx_btf_id(struct bpf_verifier_log *log, enum bpf_prog_type prog_type)
{
const struct btf_member *kctx_member;
const struct btf_type *conv_struct;
const struct btf_type *kctx_type;
u32 kctx_type_id;
conv_struct = bpf_ctx_convert.t;
/* get member for kernel ctx type */
kctx_member = btf_type_member(conv_struct) + bpf_ctx_convert_map[prog_type] * 2 + 1;
kctx_type_id = kctx_member->type;
kctx_type = btf_type_by_id(btf_vmlinux, kctx_type_id);
if (!btf_type_is_struct(kctx_type)) {
bpf_log(log, "kern ctx type id %u is not a struct\n", kctx_type_id);
return -EINVAL;
}
return kctx_type_id;
}
BTF_ID_LIST_SINGLE(bpf_ctx_convert_btf_id, struct, bpf_ctx_convert)
static struct btf *btf_parse_base(struct btf_verifier_env *env, const char *name,
void *data, unsigned int data_size)
{
struct btf *btf = NULL;
int err;
if (!IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
return ERR_PTR(-ENOENT);
btf = kzalloc_obj(*btf, GFP_KERNEL | __GFP_NOWARN);
if (!btf) {
err = -ENOMEM;
goto errout;
}
env->btf = btf;
btf->data = data;
btf->data_size = data_size;
btf->kernel_btf = true;
btf->named_start_id = 0;
strscpy(btf->name, name);
err = btf_parse_hdr(env);
if (err)
goto errout;
btf->nohdr_data = btf->data + btf->hdr.hdr_len;
err = btf_parse_str_sec(env);
if (err)
goto errout;
err = btf_check_all_metas(env);
if (err)
goto errout;
err = btf_check_type_tags(env, btf, 1);
if (err)
goto errout;
btf_check_sorted(btf);
refcount_set(&btf->refcnt, 1);
return btf;
errout:
if (btf) {
kvfree(btf->types);
kfree(btf);
}
return ERR_PTR(err);
}
struct btf *btf_parse_vmlinux(void)
{
struct btf_verifier_env *env = NULL;
struct bpf_verifier_log *log;
struct btf *btf;
int err;
env = kzalloc_obj(*env, GFP_KERNEL | __GFP_NOWARN);
if (!env)
return ERR_PTR(-ENOMEM);
log = &env->log;
log->level = BPF_LOG_KERNEL;
btf = btf_parse_base(env, "vmlinux", __start_BTF, __stop_BTF - __start_BTF);
if (IS_ERR(btf))
goto err_out;
/* btf_parse_vmlinux() runs under bpf_verifier_lock */
bpf_ctx_convert.t = btf_type_by_id(btf, bpf_ctx_convert_btf_id[0]);
err = btf_alloc_id(btf);
if (err) {
btf_free(btf);
btf = ERR_PTR(err);
}
err_out:
btf_verifier_env_free(env);
return btf;
}
/* If .BTF_ids section was created with distilled base BTF, both base and
* split BTF ids will need to be mapped to actual base/split ids for
* BTF now that it has been relocated.
*/
static __u32 btf_relocate_id(const struct btf *btf, __u32 id)
{
if (!btf->base_btf || !btf->base_id_map)
return id;
return btf->base_id_map[id];
}
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
static struct btf *btf_parse_module(const char *module_name, const void *data,
unsigned int data_size, void *base_data,
unsigned int base_data_size)
{
struct btf *btf = NULL, *vmlinux_btf, *base_btf = NULL;
struct btf_verifier_env *env = NULL;
struct bpf_verifier_log *log;
int err = 0;
vmlinux_btf = bpf_get_btf_vmlinux();
if (IS_ERR(vmlinux_btf))
return vmlinux_btf;
if (!vmlinux_btf)
return ERR_PTR(-EINVAL);
env = kzalloc_obj(*env, GFP_KERNEL | __GFP_NOWARN);
if (!env)
return ERR_PTR(-ENOMEM);
log = &env->log;
log->level = BPF_LOG_KERNEL;
if (base_data) {
base_btf = btf_parse_base(env, ".BTF.base", base_data, base_data_size);
if (IS_ERR(base_btf)) {
err = PTR_ERR(base_btf);
goto errout;
}
} else {
base_btf = vmlinux_btf;
}
btf = kzalloc_obj(*btf, GFP_KERNEL | __GFP_NOWARN);
if (!btf) {
err = -ENOMEM;
goto errout;
}
env->btf = btf;
btf->base_btf = base_btf;
btf->start_id = base_btf->nr_types;
btf->start_str_off = base_btf->hdr.str_len;
btf->kernel_btf = true;
btf->named_start_id = 0;
strscpy(btf->name, module_name);
btf->data = kvmemdup(data, data_size, GFP_KERNEL | __GFP_NOWARN);
if (!btf->data) {
err = -ENOMEM;
goto errout;
}
btf->data_size = data_size;
err = btf_parse_hdr(env);
if (err)
goto errout;
btf->nohdr_data = btf->data + btf->hdr.hdr_len;
err = btf_parse_str_sec(env);
if (err)
goto errout;
err = btf_check_all_metas(env);
if (err)
goto errout;
err = btf_check_type_tags(env, btf, btf_nr_types(base_btf));
if (err)
goto errout;
if (base_btf != vmlinux_btf) {
err = btf_relocate(btf, vmlinux_btf, &btf->base_id_map);
if (err)
goto errout;
btf_free(base_btf);
base_btf = vmlinux_btf;
}
btf_verifier_env_free(env);
btf_check_sorted(btf);
refcount_set(&btf->refcnt, 1);
return btf;
errout:
btf_verifier_env_free(env);
if (!IS_ERR(base_btf) && base_btf != vmlinux_btf)
btf_free(base_btf);
if (btf) {
kvfree(btf->data);
kvfree(btf->types);
kfree(btf);
}
return ERR_PTR(err);
}
#endif /* CONFIG_DEBUG_INFO_BTF_MODULES */
struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog)
{
struct bpf_prog *tgt_prog = prog->aux->dst_prog;
if (tgt_prog)
return tgt_prog->aux->btf;
else
return prog->aux->attach_btf;
}
u32 btf_ctx_arg_idx(struct btf *btf, const struct btf_type *func_proto,
int off)
{
const struct btf_param *args;
const struct btf_type *t;
u32 offset = 0, nr_args;
int i;
if (!func_proto)
return off / 8;
nr_args = btf_type_vlen(func_proto);
args = (const struct btf_param *)(func_proto + 1);
for (i = 0; i < nr_args; i++) {
t = btf_type_skip_modifiers(btf, args[i].type, NULL);
offset += btf_type_is_ptr(t) ? 8 : roundup(t->size, 8);
if (off < offset)
return i;
}
t = btf_type_skip_modifiers(btf, func_proto->type, NULL);
offset += btf_type_is_ptr(t) ? 8 : roundup(t->size, 8);
if (off < offset)
return nr_args;
return nr_args + 1;
}
static bool prog_args_trusted(const struct bpf_prog *prog)
{
enum bpf_attach_type atype = prog->expected_attach_type;
switch (prog->type) {
case BPF_PROG_TYPE_TRACING:
return atype == BPF_TRACE_RAW_TP || atype == BPF_TRACE_ITER;
case BPF_PROG_TYPE_LSM:
return bpf_lsm_is_trusted(prog);
case BPF_PROG_TYPE_STRUCT_OPS:
return true;
default:
return false;
}
}
int btf_ctx_arg_offset(const struct btf *btf, const struct btf_type *func_proto,
u32 arg_no)
{
const struct btf_param *args;
const struct btf_type *t;
int off = 0, i;
u32 sz;
args = btf_params(func_proto);
for (i = 0; i < arg_no; i++) {
t = btf_type_by_id(btf, args[i].type);
t = btf_resolve_size(btf, t, &sz);
if (IS_ERR(t))
return PTR_ERR(t);
off += roundup(sz, 8);
}
return off;
}
struct bpf_raw_tp_null_args {
const char *func;
u64 mask;
};
static const struct bpf_raw_tp_null_args raw_tp_null_args[] = {
/* sched */
{ "sched_pi_setprio", 0x10 },
/* ... from sched_numa_pair_template event class */
{ "sched_stick_numa", 0x100 },
{ "sched_swap_numa", 0x100 },
/* afs */
{ "afs_make_fs_call", 0x10 },
{ "afs_make_fs_calli", 0x10 },
{ "afs_make_fs_call1", 0x10 },
{ "afs_make_fs_call2", 0x10 },
{ "afs_protocol_error", 0x1 },
{ "afs_flock_ev", 0x10 },
/* cachefiles */
{ "cachefiles_lookup", 0x1 | 0x200 },
{ "cachefiles_unlink", 0x1 },
{ "cachefiles_rename", 0x1 },
{ "cachefiles_prep_read", 0x1 },
{ "cachefiles_mark_active", 0x1 },
{ "cachefiles_mark_failed", 0x1 },
{ "cachefiles_mark_inactive", 0x1 },
{ "cachefiles_vfs_error", 0x1 },
{ "cachefiles_io_error", 0x1 },
{ "cachefiles_ondemand_open", 0x1 },
{ "cachefiles_ondemand_copen", 0x1 },
{ "cachefiles_ondemand_close", 0x1 },
{ "cachefiles_ondemand_read", 0x1 },
{ "cachefiles_ondemand_cread", 0x1 },
{ "cachefiles_ondemand_fd_write", 0x1 },
{ "cachefiles_ondemand_fd_release", 0x1 },
/* ext4, from ext4__mballoc event class */
{ "ext4_mballoc_discard", 0x10 },
{ "ext4_mballoc_free", 0x10 },
/* fib */
{ "fib_table_lookup", 0x100 },
/* filelock */
/* ... from filelock_lock event class */
{ "posix_lock_inode", 0x10 },
{ "fcntl_setlk", 0x10 },
{ "locks_remove_posix", 0x10 },
{ "flock_lock_inode", 0x10 },
/* ... from filelock_lease event class */
{ "break_lease_noblock", 0x10 },
{ "break_lease_block", 0x10 },
{ "break_lease_unblock", 0x10 },
{ "generic_delete_lease", 0x10 },
{ "time_out_leases", 0x10 },
/* host1x */
{ "host1x_cdma_push_gather", 0x10000 },
/* huge_memory */
{ "mm_khugepaged_scan_pmd", 0x10 },
{ "mm_collapse_huge_page_isolate", 0x1 },
{ "mm_khugepaged_scan_file", 0x10 },
{ "mm_khugepaged_collapse_file", 0x10 },
/* kmem */
{ "mm_page_alloc", 0x1 },
{ "mm_page_pcpu_drain", 0x1 },
/* .. from mm_page event class */
{ "mm_page_alloc_zone_locked", 0x1 },
/* netfs */
{ "netfs_failure", 0x10 },
/* power */
{ "device_pm_callback_start", 0x10 },
/* qdisc */
{ "qdisc_dequeue", 0x1000 },
/* rxrpc */
{ "rxrpc_recvdata", 0x1 },
{ "rxrpc_resend", 0x10 },
{ "rxrpc_tq", 0x10 },
{ "rxrpc_client", 0x1 },
/* skb */
{"kfree_skb", 0x1000},
/* sunrpc */
{ "xs_stream_read_data", 0x1 },
/* ... from xprt_cong_event event class */
{ "xprt_reserve_cong", 0x10 },
{ "xprt_release_cong", 0x10 },
{ "xprt_get_cong", 0x10 },
{ "xprt_put_cong", 0x10 },
/* tcp */
{ "tcp_send_reset", 0x11 },
{ "tcp_sendmsg_locked", 0x100 },
/* tegra_apb_dma */
{ "tegra_dma_tx_status", 0x100 },
/* timer_migration */
{ "tmigr_update_events", 0x1 },
/* writeback, from writeback_folio_template event class */
{ "writeback_dirty_folio", 0x10 },
{ "folio_wait_writeback", 0x10 },
/* rdma */
{ "mr_integ_alloc", 0x2000 },
/* bpf_testmod */
{ "bpf_testmod_test_read", 0x0 },
/* amdgpu */
{ "amdgpu_vm_bo_map", 0x1 },
{ "amdgpu_vm_bo_unmap", 0x1 },
/* netfs */
{ "netfs_folioq", 0x1 },
/* xfs from xfs_defer_pending_class */
{ "xfs_defer_create_intent", 0x1 },
{ "xfs_defer_cancel_list", 0x1 },
{ "xfs_defer_pending_finish", 0x1 },
{ "xfs_defer_pending_abort", 0x1 },
{ "xfs_defer_relog_intent", 0x1 },
{ "xfs_defer_isolate_paused", 0x1 },
{ "xfs_defer_item_pause", 0x1 },
{ "xfs_defer_item_unpause", 0x1 },
/* xfs from xfs_defer_pending_item_class */
{ "xfs_defer_add_item", 0x1 },
{ "xfs_defer_cancel_item", 0x1 },
{ "xfs_defer_finish_item", 0x1 },
/* xfs from xfs_icwalk_class */
{ "xfs_ioc_free_eofblocks", 0x10 },
{ "xfs_blockgc_free_space", 0x10 },
/* xfs from xfs_btree_cur_class */
{ "xfs_btree_updkeys", 0x100 },
{ "xfs_btree_overlapped_query_range", 0x100 },
/* xfs from xfs_imap_class*/
{ "xfs_map_blocks_found", 0x10000 },
{ "xfs_map_blocks_alloc", 0x10000 },
{ "xfs_iomap_alloc", 0x1000 },
{ "xfs_iomap_found", 0x1000 },
/* xfs from xfs_fs_class */
{ "xfs_inodegc_flush", 0x1 },
{ "xfs_inodegc_push", 0x1 },
{ "xfs_inodegc_start", 0x1 },
{ "xfs_inodegc_stop", 0x1 },
{ "xfs_inodegc_queue", 0x1 },
{ "xfs_inodegc_throttle", 0x1 },
{ "xfs_fs_sync_fs", 0x1 },
{ "xfs_blockgc_start", 0x1 },
{ "xfs_blockgc_stop", 0x1 },
{ "xfs_blockgc_worker", 0x1 },
{ "xfs_blockgc_flush_all", 0x1 },
/* xfs_scrub */
{ "xchk_nlinks_live_update", 0x10 },
/* xfs_scrub from xchk_metapath_class */
{ "xchk_metapath_lookup", 0x100 },
/* nfsd */
{ "nfsd_dirent", 0x1 },
{ "nfsd_file_acquire", 0x1001 },
{ "nfsd_file_insert_err", 0x1 },
{ "nfsd_file_cons_err", 0x1 },
/* nfs4 */
{ "nfs4_setup_sequence", 0x1 },
{ "pnfs_update_layout", 0x10000 },
{ "nfs4_inode_callback_event", 0x200 },
{ "nfs4_inode_stateid_callback_event", 0x200 },
/* nfs from pnfs_layout_event */
{ "pnfs_mds_fallback_pg_init_read", 0x10000 },
{ "pnfs_mds_fallback_pg_init_write", 0x10000 },
{ "pnfs_mds_fallback_pg_get_mirror_count", 0x10000 },
{ "pnfs_mds_fallback_read_done", 0x10000 },
{ "pnfs_mds_fallback_write_done", 0x10000 },
{ "pnfs_mds_fallback_read_pagelist", 0x10000 },
{ "pnfs_mds_fallback_write_pagelist", 0x10000 },
/* coda */
{ "coda_dec_pic_run", 0x10 },
{ "coda_dec_pic_done", 0x10 },
/* cfg80211 */
{ "cfg80211_scan_done", 0x11 },
{ "rdev_set_coalesce", 0x10 },
{ "cfg80211_report_wowlan_wakeup", 0x100 },
{ "cfg80211_inform_bss_frame", 0x100 },
{ "cfg80211_michael_mic_failure", 0x10000 },
/* cfg80211 from wiphy_work_event */
{ "wiphy_work_queue", 0x10 },
{ "wiphy_work_run", 0x10 },
{ "wiphy_work_cancel", 0x10 },
{ "wiphy_work_flush", 0x10 },
/* hugetlbfs */
{ "hugetlbfs_alloc_inode", 0x10 },
/* spufs */
{ "spufs_context", 0x10 },
/* kvm_hv */
{ "kvm_page_fault_enter", 0x100 },
/* dpu */
{ "dpu_crtc_setup_mixer", 0x100 },
/* binder */
{ "binder_transaction", 0x100 },
/* bcachefs */
{ "btree_path_free", 0x100 },
/* hfi1_tx */
{ "hfi1_sdma_progress", 0x1000 },
/* iptfs */
{ "iptfs_ingress_postq_event", 0x1000 },
/* neigh */
{ "neigh_update", 0x10 },
/* snd_firewire_lib */
{ "amdtp_packet", 0x100 },
};
bool btf_ctx_access(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
const struct btf_type *t = prog->aux->attach_func_proto;
struct bpf_prog *tgt_prog = prog->aux->dst_prog;
struct btf *btf = bpf_prog_get_target_btf(prog);
const char *tname = prog->aux->attach_func_name;
struct bpf_verifier_log *log = info->log;
const struct btf_param *args;
bool ptr_err_raw_tp = false;
const char *tag_value;
u32 nr_args, arg;
int i, ret;
if (off % 8) {
bpf_log(log, "func '%s' offset %d is not multiple of 8\n",
tname, off);
return false;
}
arg = btf_ctx_arg_idx(btf, t, off);
args = (const struct btf_param *)(t + 1);
/* if (t == NULL) Fall back to default BPF prog with
* MAX_BPF_FUNC_REG_ARGS u64 arguments.
*/
nr_args = t ? btf_type_vlen(t) : MAX_BPF_FUNC_REG_ARGS;
if (prog->aux->attach_btf_trace) {
/* skip first 'void *__data' argument in btf_trace_##name typedef */
args++;
nr_args--;
}
if (arg > nr_args) {
bpf_log(log, "func '%s' doesn't have %d-th argument\n",
tname, arg + 1);
return false;
}
if (arg == nr_args) {
switch (prog->expected_attach_type) {
case BPF_LSM_MAC:
/* mark we are accessing the return value */
info->is_retval = true;
fallthrough;
case BPF_LSM_CGROUP:
case BPF_TRACE_FEXIT:
case BPF_TRACE_FSESSION:
/* When LSM programs are attached to void LSM hooks
* they use FEXIT trampolines and when attached to
* int LSM hooks, they use MODIFY_RETURN trampolines.
*
* While the LSM programs are BPF_MODIFY_RETURN-like
* the check:
*
* if (ret_type != 'int')
* return -EINVAL;
*
* is _not_ done here. This is still safe as LSM hooks
* have only void and int return types.
*/
if (!t)
return true;
t = btf_type_by_id(btf, t->type);
break;
case BPF_MODIFY_RETURN:
/* For now the BPF_MODIFY_RETURN can only be attached to
* functions that return an int.
*/
if (!t)
return false;
t = btf_type_skip_modifiers(btf, t->type, NULL);
if (!btf_type_is_small_int(t)) {
bpf_log(log,
"ret type %s not allowed for fmod_ret\n",
btf_type_str(t));
return false;
}
break;
default:
bpf_log(log, "func '%s' doesn't have %d-th argument\n",
tname, arg + 1);
return false;
}
} else {
if (!t)
/* Default prog with MAX_BPF_FUNC_REG_ARGS args */
return true;
t = btf_type_by_id(btf, args[arg].type);
}
/* skip modifiers */
while (btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);
if (btf_type_is_small_int(t) || btf_is_any_enum(t) || btf_type_is_struct(t))
/* accessing a scalar */
return true;
if (!btf_type_is_ptr(t)) {
bpf_log(log,
"func '%s' arg%d '%s' has type %s. Only pointer access is allowed\n",
tname, arg,
__btf_name_by_offset(btf, t->name_off),
btf_type_str(t));
return false;
}
if (size != sizeof(u64)) {
bpf_log(log, "func '%s' size %d must be 8\n",
tname, size);
return false;
}
/* check for PTR_TO_RDONLY_BUF_OR_NULL or PTR_TO_RDWR_BUF_OR_NULL */
for (i = 0; i < prog->aux->ctx_arg_info_size; i++) {
const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];
u32 type, flag;
type = base_type(ctx_arg_info->reg_type);
flag = type_flag(ctx_arg_info->reg_type);
if (ctx_arg_info->offset == off && type == PTR_TO_BUF &&
(flag & PTR_MAYBE_NULL)) {
info->reg_type = ctx_arg_info->reg_type;
return true;
}
}
/*
* If it's a single or multilevel pointer, except a pointer
* to a structure, it's the same as scalar from the verifier
* safety POV. Multilevel pointers to structures are treated as
* scalars. The verifier lacks the context to infer the size of
* their target memory regions. Either way, no further pointer
* walking is allowed.
*/
if (!btf_type_is_struct_ptr(btf, t))
return true;
/* this is a pointer to another type */
for (i = 0; i < prog->aux->ctx_arg_info_size; i++) {
const struct bpf_ctx_arg_aux *ctx_arg_info = &prog->aux->ctx_arg_info[i];
if (ctx_arg_info->offset == off) {
if (!ctx_arg_info->btf_id) {
bpf_log(log,"invalid btf_id for context argument offset %u\n", off);
return false;
}
info->reg_type = ctx_arg_info->reg_type;
info->btf = ctx_arg_info->btf ? : btf_vmlinux;
info->btf_id = ctx_arg_info->btf_id;
info->ref_obj_id = ctx_arg_info->ref_obj_id;
return true;
}
}
info->reg_type = PTR_TO_BTF_ID;
if (prog_args_trusted(prog))
info->reg_type |= PTR_TRUSTED;
if (btf_param_match_suffix(btf, &args[arg], "__nullable"))
info->reg_type |= PTR_MAYBE_NULL;
if (prog->expected_attach_type == BPF_TRACE_RAW_TP) {
struct btf *btf = prog->aux->attach_btf;
const struct btf_type *t;
const char *tname;
/* BTF lookups cannot fail, return false on error */
t = btf_type_by_id(btf, prog->aux->attach_btf_id);
if (!t)
return false;
tname = btf_name_by_offset(btf, t->name_off);
if (!tname)
return false;
/* Checked by bpf_check_attach_target */
tname += sizeof("btf_trace_") - 1;
for (i = 0; i < ARRAY_SIZE(raw_tp_null_args); i++) {
/* Is this a func with potential NULL args? */
if (strcmp(tname, raw_tp_null_args[i].func))
continue;
if (raw_tp_null_args[i].mask & (0x1ULL << (arg * 4)))
info->reg_type |= PTR_MAYBE_NULL;
/* Is the current arg IS_ERR? */
if (raw_tp_null_args[i].mask & (0x2ULL << (arg * 4)))
ptr_err_raw_tp = true;
break;
}
/* If we don't know NULL-ness specification and the tracepoint
* is coming from a loadable module, be conservative and mark
* argument as PTR_MAYBE_NULL.
*/
if (i == ARRAY_SIZE(raw_tp_null_args) && btf_is_module(btf))
info->reg_type |= PTR_MAYBE_NULL;
}
if (tgt_prog) {
enum bpf_prog_type tgt_type;
if (tgt_prog->type == BPF_PROG_TYPE_EXT)
tgt_type = tgt_prog->aux->saved_dst_prog_type;
else
tgt_type = tgt_prog->type;
ret = btf_translate_to_vmlinux(log, btf, t, tgt_type, arg);
if (ret > 0) {
info->btf = btf_vmlinux;
info->btf_id = ret;
return true;
} else {
return false;
}
}
info->btf = btf;
info->btf_id = t->type;
t = btf_type_by_id(btf, t->type);
if (btf_type_is_type_tag(t) && !btf_type_kflag(t)) {
tag_value = __btf_name_by_offset(btf, t->name_off);
if (strcmp(tag_value, "user") == 0)
info->reg_type |= MEM_USER;
if (strcmp(tag_value, "percpu") == 0)
info->reg_type |= MEM_PERCPU;
}
/* skip modifiers */
while (btf_type_is_modifier(t)) {
info->btf_id = t->type;
t = btf_type_by_id(btf, t->type);
}
if (!btf_type_is_struct(t)) {
bpf_log(log,
"func '%s' arg%d type %s is not a struct\n",
tname, arg, btf_type_str(t));
return false;
}
bpf_log(log, "func '%s' arg%d has btf_id %d type %s '%s'\n",
tname, arg, info->btf_id, btf_type_str(t),
__btf_name_by_offset(btf, t->name_off));
/* Perform all checks on the validity of type for this argument, but if
* we know it can be IS_ERR at runtime, scrub pointer type and mark as
* scalar.
*/
if (ptr_err_raw_tp) {
bpf_log(log, "marking pointer arg%d as scalar as it may encode error", arg);
info->reg_type = SCALAR_VALUE;
}
return true;
}
EXPORT_SYMBOL_GPL(btf_ctx_access);
enum bpf_struct_walk_result {
/* < 0 error */
WALK_SCALAR = 0,
WALK_PTR,
WALK_PTR_UNTRUSTED,
WALK_STRUCT,
};
static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, int off, int size,
u32 *next_btf_id, enum bpf_type_flag *flag,
const char **field_name)
{
u32 i, moff, mtrue_end, msize = 0, total_nelems = 0;
const struct btf_type *mtype, *elem_type = NULL;
const struct btf_member *member;
const char *tname, *mname, *tag_value;
u32 vlen, elem_id, mid;
again:
if (btf_type_is_modifier(t))
t = btf_type_skip_modifiers(btf, t->type, NULL);
tname = __btf_name_by_offset(btf, t->name_off);
if (!btf_type_is_struct(t)) {
bpf_log(log, "Type '%s' is not a struct\n", tname);
return -EINVAL;
}
vlen = btf_type_vlen(t);
if (BTF_INFO_KIND(t->info) == BTF_KIND_UNION && vlen != 1 && !(*flag & PTR_UNTRUSTED))
/*
* walking unions yields untrusted pointers
* with exception of __bpf_md_ptr and other
* unions with a single member
*/
*flag |= PTR_UNTRUSTED;
if (off + size > t->size) {
/* If the last element is a variable size array, we may
* need to relax the rule.
*/
struct btf_array *array_elem;
if (vlen == 0)
goto error;
member = btf_type_member(t) + vlen - 1;
mtype = btf_type_skip_modifiers(btf, member->type,
NULL);
if (!btf_type_is_array(mtype))
goto error;
array_elem = (struct btf_array *)(mtype + 1);
if (array_elem->nelems != 0)
goto error;
moff = __btf_member_bit_offset(t, member) / 8;
if (off < moff)
goto error;
/* allow structure and integer */
t = btf_type_skip_modifiers(btf, array_elem->type,
NULL);
if (btf_type_is_int(t))
return WALK_SCALAR;
if (!btf_type_is_struct(t))
goto error;
off = (off - moff) % t->size;
goto again;
error:
bpf_log(log, "access beyond struct %s at off %u size %u\n",
tname, off, size);
return -EACCES;
}
for_each_member(i, t, member) {
/* offset of the field in bytes */
moff = __btf_member_bit_offset(t, member) / 8;
if (off + size <= moff)
/* won't find anything, field is already too far */
break;
if (__btf_member_bitfield_size(t, member)) {
u32 end_bit = __btf_member_bit_offset(t, member) +
__btf_member_bitfield_size(t, member);
/* off <= moff instead of off == moff because clang
* does not generate a BTF member for anonymous
* bitfield like the ":16" here:
* struct {
* int :16;
* int x:8;
* };
*/
if (off <= moff &&
BITS_ROUNDUP_BYTES(end_bit) <= off + size)
return WALK_SCALAR;
/* off may be accessing a following member
*
* or
*
* Doing partial access at either end of this
* bitfield. Continue on this case also to
* treat it as not accessing this bitfield
* and eventually error out as field not
* found to keep it simple.
* It could be relaxed if there was a legit
* partial access case later.
*/
continue;
}
/* In case of "off" is pointing to holes of a struct */
if (off < moff)
break;
/* type of the field */
mid = member->type;
mtype = btf_type_by_id(btf, member->type);
mname = __btf_name_by_offset(btf, member->name_off);
mtype = __btf_resolve_size(btf, mtype, &msize,
&elem_type, &elem_id, &total_nelems,
&mid);
if (IS_ERR(mtype)) {
bpf_log(log, "field %s doesn't have size\n", mname);
return -EFAULT;
}
mtrue_end = moff + msize;
if (off >= mtrue_end)
/* no overlap with member, keep iterating */
continue;
if (btf_type_is_array(mtype)) {
u32 elem_idx;
/* __btf_resolve_size() above helps to
* linearize a multi-dimensional array.
*
* The logic here is treating an array
* in a struct as the following way:
*
* struct outer {
* struct inner array[2][2];
* };
*
* looks like:
*
* struct outer {
* struct inner array_elem0;
* struct inner array_elem1;
* struct inner array_elem2;
* struct inner array_elem3;
* };
*
* When accessing outer->array[1][0], it moves
* moff to "array_elem2", set mtype to
* "struct inner", and msize also becomes
* sizeof(struct inner). Then most of the
* remaining logic will fall through without
* caring the current member is an array or
* not.
*
* Unlike mtype/msize/moff, mtrue_end does not
* change. The naming difference ("_true") tells
* that it is not always corresponding to
* the current mtype/msize/moff.
* It is the true end of the current
* member (i.e. array in this case). That
* will allow an int array to be accessed like
* a scratch space,
* i.e. allow access beyond the size of
* the array's element as long as it is
* within the mtrue_end boundary.
*/
/* skip empty array */
if (moff == mtrue_end)
continue;
msize /= total_nelems;
elem_idx = (off - moff) / msize;
moff += elem_idx * msize;
mtype = elem_type;
mid = elem_id;
}
/* the 'off' we're looking for is either equal to start
* of this field or inside of this struct
*/
if (btf_type_is_struct(mtype)) {
/* our field must be inside that union or struct */
t = mtype;
/* return if the offset matches the member offset */
if (off == moff) {
*next_btf_id = mid;
return WALK_STRUCT;
}
/* adjust offset we're looking for */
off -= moff;
goto again;
}
if (btf_type_is_ptr(mtype)) {
const struct btf_type *stype, *t;
enum bpf_type_flag tmp_flag = 0;
u32 id;
if (msize != size || off != moff) {
bpf_log(log,
"cannot access ptr member %s with moff %u in struct %s with off %u size %u\n",
mname, moff, tname, off, size);
return -EACCES;
}
/* check type tag */
t = btf_type_by_id(btf, mtype->type);
if (btf_type_is_type_tag(t) && !btf_type_kflag(t)) {
tag_value = __btf_name_by_offset(btf, t->name_off);
/* check __user tag */
if (strcmp(tag_value, "user") == 0)
tmp_flag = MEM_USER;
/* check __percpu tag */
if (strcmp(tag_value, "percpu") == 0)
tmp_flag = MEM_PERCPU;
/* check __rcu tag */
if (strcmp(tag_value, "rcu") == 0)
tmp_flag = MEM_RCU;
}
stype = btf_type_skip_modifiers(btf, mtype->type, &id);
if (btf_type_is_struct(stype)) {
*next_btf_id = id;
*flag |= tmp_flag;
if (field_name)
*field_name = mname;
return WALK_PTR;
}
return WALK_PTR_UNTRUSTED;
}
/* Allow more flexible access within an int as long as
* it is within mtrue_end.
* Since mtrue_end could be the end of an array,
* that also allows using an array of int as a scratch
* space. e.g. skb->cb[].
*/
if (off + size > mtrue_end && !(*flag & PTR_UNTRUSTED)) {
bpf_log(log,
"access beyond the end of member %s (mend:%u) in struct %s with off %u size %u\n",
mname, mtrue_end, tname, off, size);
return -EACCES;
}
return WALK_SCALAR;
}
bpf_log(log, "struct %s doesn't have field at offset %d\n", tname, off);
return -EINVAL;
}
int btf_struct_access(struct bpf_verifier_log *log,
const struct bpf_reg_state *reg,
int off, int size, enum bpf_access_type atype __maybe_unused,
u32 *next_btf_id, enum bpf_type_flag *flag,
const char **field_name)
{
const struct btf *btf = reg->btf;
enum bpf_type_flag tmp_flag = 0;
const struct btf_type *t;
u32 id = reg->btf_id;
int err;
while (type_is_alloc(reg->type)) {
struct btf_struct_meta *meta;
struct btf_record *rec;
int i;
meta = btf_find_struct_meta(btf, id);
if (!meta)
break;
rec = meta->record;
for (i = 0; i < rec->cnt; i++) {
struct btf_field *field = &rec->fields[i];
u32 offset = field->offset;
if (off < offset + field->size && offset < off + size) {
bpf_log(log,
"direct access to %s is disallowed\n",
btf_field_type_name(field->type));
return -EACCES;
}
}
break;
}
t = btf_type_by_id(btf, id);
do {
err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag, field_name);
switch (err) {
case WALK_PTR:
/* For local types, the destination register cannot
* become a pointer again.
*/
if (type_is_alloc(reg->type))
return SCALAR_VALUE;
/* If we found the pointer or scalar on t+off,
* we're done.
*/
*next_btf_id = id;
*flag = tmp_flag;
return PTR_TO_BTF_ID;
case WALK_PTR_UNTRUSTED:
*flag = MEM_RDONLY | PTR_UNTRUSTED;
return PTR_TO_MEM;
case WALK_SCALAR:
return SCALAR_VALUE;
case WALK_STRUCT:
/* We found nested struct, so continue the search
* by diving in it. At this point the offset is
* aligned with the new type, so set it to 0.
*/
t = btf_type_by_id(btf, id);
off = 0;
break;
default:
/* It's either error or unknown return value..
* scream and leave.
*/
if (WARN_ONCE(err > 0, "unknown btf_struct_walk return value"))
return -EINVAL;
return err;
}
} while (t);
return -EINVAL;
}
/* Check that two BTF types, each specified as an BTF object + id, are exactly
* the same. Trivial ID check is not enough due to module BTFs, because we can
* end up with two different module BTFs, but IDs point to the common type in
* vmlinux BTF.
*/
bool btf_types_are_same(const struct btf *btf1, u32 id1,
const struct btf *btf2, u32 id2)
{
if (id1 != id2)
return false;
if (btf1 == btf2)
return true;
return btf_type_by_id(btf1, id1) == btf_type_by_id(btf2, id2);
}
bool btf_struct_ids_match(struct bpf_verifier_log *log,
const struct btf *btf, u32 id, int off,
const struct btf *need_btf, u32 need_type_id,
bool strict)
{
const struct btf_type *type;
enum bpf_type_flag flag = 0;
int err;
/* Are we already done? */
if (off == 0 && btf_types_are_same(btf, id, need_btf, need_type_id))
return true;
/* In case of strict type match, we do not walk struct, the top level
* type match must succeed. When strict is true, off should have already
* been 0.
*/
if (strict)
return false;
again:
type = btf_type_by_id(btf, id);
if (!type)
return false;
err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL);
if (err != WALK_STRUCT)
return false;
/* We found nested struct object. If it matches
* the requested ID, we're done. Otherwise let's
* continue the search with offset 0 in the new
* type.
*/
if (!btf_types_are_same(btf, id, need_btf, need_type_id)) {
off = 0;
goto again;
}
return true;
}
static int __get_type_size(struct btf *btf, u32 btf_id,
const struct btf_type **ret_type)
{
const struct btf_type *t;
*ret_type = btf_type_by_id(btf, 0);
if (!btf_id)
/* void */
return 0;
t = btf_type_by_id(btf, btf_id);
while (t && btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);
if (!t)
return -EINVAL;
*ret_type = t;
if (btf_type_is_ptr(t))
/* kernel size of pointer. Not BPF's size of pointer*/
return sizeof(void *);
if (btf_type_is_int(t) || btf_is_any_enum(t) || btf_type_is_struct(t))
return t->size;
return -EINVAL;
}
static u8 __get_type_fmodel_flags(const struct btf_type *t)
{
u8 flags = 0;
if (btf_type_is_struct(t))
flags |= BTF_FMODEL_STRUCT_ARG;
if (btf_type_is_signed_int(t))
flags |= BTF_FMODEL_SIGNED_ARG;
return flags;
}
int btf_distill_func_proto(struct bpf_verifier_log *log,
struct btf *btf,
const struct btf_type *func,
const char *tname,
struct btf_func_model *m)
{
const struct btf_param *args;
const struct btf_type *t;
u32 i, nargs;
int ret;
if (!func) {
/* BTF function prototype doesn't match the verifier types.
* Fall back to MAX_BPF_FUNC_REG_ARGS u64 args.
*/
for (i = 0; i < MAX_BPF_FUNC_REG_ARGS; i++) {
m->arg_size[i] = 8;
m->arg_flags[i] = 0;
}
m->ret_size = 8;
m->ret_flags = 0;
m->nr_args = MAX_BPF_FUNC_REG_ARGS;
return 0;
}
args = (const struct btf_param *)(func + 1);
nargs = btf_type_vlen(func);
if (nargs > MAX_BPF_FUNC_ARGS) {
bpf_log(log,
"The function %s has %d arguments. Too many.\n",
tname, nargs);
return -EINVAL;
}
ret = __get_type_size(btf, func->type, &t);
if (ret < 0 || btf_type_is_struct(t)) {
bpf_log(log,
"The function %s return type %s is unsupported.\n",
tname, btf_type_str(t));
return -EINVAL;
}
m->ret_size = ret;
m->ret_flags = __get_type_fmodel_flags(t);
for (i = 0; i < nargs; i++) {
if (i == nargs - 1 && args[i].type == 0) {
bpf_log(log,
"The function %s with variable args is unsupported.\n",
tname);
return -EINVAL;
}
ret = __get_type_size(btf, args[i].type, &t);
/* No support of struct argument size greater than 16 bytes */
if (ret < 0 || ret > 16) {
bpf_log(log,
"The function %s arg%d type %s is unsupported.\n",
tname, i, btf_type_str(t));
return -EINVAL;
}
if (ret == 0) {
bpf_log(log,
"The function %s has malformed void argument.\n",
tname);
return -EINVAL;
}
m->arg_size[i] = ret;
m->arg_flags[i] = __get_type_fmodel_flags(t);
}
m->nr_args = nargs;
return 0;
}
/* Compare BTFs of two functions assuming only scalars and pointers to context.
* t1 points to BTF_KIND_FUNC in btf1
* t2 points to BTF_KIND_FUNC in btf2
* Returns:
* EINVAL - function prototype mismatch
* EFAULT - verifier bug
* 0 - 99% match. The last 1% is validated by the verifier.
*/
static int btf_check_func_type_match(struct bpf_verifier_log *log,
struct btf *btf1, const struct btf_type *t1,
struct btf *btf2, const struct btf_type *t2)
{
const struct btf_param *args1, *args2;
const char *fn1, *fn2, *s1, *s2;
u32 nargs1, nargs2, i;
fn1 = btf_name_by_offset(btf1, t1->name_off);
fn2 = btf_name_by_offset(btf2, t2->name_off);
if (btf_func_linkage(t1) != BTF_FUNC_GLOBAL) {
bpf_log(log, "%s() is not a global function\n", fn1);
return -EINVAL;
}
if (btf_func_linkage(t2) != BTF_FUNC_GLOBAL) {
bpf_log(log, "%s() is not a global function\n", fn2);
return -EINVAL;
}
t1 = btf_type_by_id(btf1, t1->type);
if (!t1 || !btf_type_is_func_proto(t1))
return -EFAULT;
t2 = btf_type_by_id(btf2, t2->type);
if (!t2 || !btf_type_is_func_proto(t2))
return -EFAULT;
args1 = (const struct btf_param *)(t1 + 1);
nargs1 = btf_type_vlen(t1);
args2 = (const struct btf_param *)(t2 + 1);
nargs2 = btf_type_vlen(t2);
if (nargs1 != nargs2) {
bpf_log(log, "%s() has %d args while %s() has %d args\n",
fn1, nargs1, fn2, nargs2);
return -EINVAL;
}
t1 = btf_type_skip_modifiers(btf1, t1->type, NULL);
t2 = btf_type_skip_modifiers(btf2, t2->type, NULL);
if (t1->info != t2->info) {
bpf_log(log,
"Return type %s of %s() doesn't match type %s of %s()\n",
btf_type_str(t1), fn1,
btf_type_str(t2), fn2);
return -EINVAL;
}
for (i = 0; i < nargs1; i++) {
t1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL);
t2 = btf_type_skip_modifiers(btf2, args2[i].type, NULL);
if (t1->info != t2->info) {
bpf_log(log, "arg%d in %s() is %s while %s() has %s\n",
i, fn1, btf_type_str(t1),
fn2, btf_type_str(t2));
return -EINVAL;
}
if (btf_type_has_size(t1) && t1->size != t2->size) {
bpf_log(log,
"arg%d in %s() has size %d while %s() has %d\n",
i, fn1, t1->size,
fn2, t2->size);
return -EINVAL;
}
/* global functions are validated with scalars and pointers
* to context only. And only global functions can be replaced.
* Hence type check only those types.
*/
if (btf_type_is_int(t1) || btf_is_any_enum(t1))
continue;
if (!btf_type_is_ptr(t1)) {
bpf_log(log,
"arg%d in %s() has unrecognized type\n",
i, fn1);
return -EINVAL;
}
t1 = btf_type_skip_modifiers(btf1, t1->type, NULL);
t2 = btf_type_skip_modifiers(btf2, t2->type, NULL);
if (!btf_type_is_struct(t1)) {
bpf_log(log,
"arg%d in %s() is not a pointer to context\n",
i, fn1);
return -EINVAL;
}
if (!btf_type_is_struct(t2)) {
bpf_log(log,
"arg%d in %s() is not a pointer to context\n",
i, fn2);
return -EINVAL;
}
/* This is an optional check to make program writing easier.
* Compare names of structs and report an error to the user.
* btf_prepare_func_args() already checked that t2 struct
* is a context type. btf_prepare_func_args() will check
* later that t1 struct is a context type as well.
*/
s1 = btf_name_by_offset(btf1, t1->name_off);
s2 = btf_name_by_offset(btf2, t2->name_off);
if (strcmp(s1, s2)) {
bpf_log(log,
"arg%d %s(struct %s *) doesn't match %s(struct %s *)\n",
i, fn1, s1, fn2, s2);
return -EINVAL;
}
}
return 0;
}
/* Compare BTFs of given program with BTF of target program */
int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog,
struct btf *btf2, const struct btf_type *t2)
{
struct btf *btf1 = prog->aux->btf;
const struct btf_type *t1;
u32 btf_id = 0;
if (!prog->aux->func_info) {
bpf_log(log, "Program extension requires BTF\n");
return -EINVAL;
}
btf_id = prog->aux->func_info[0].type_id;
if (!btf_id)
return -EFAULT;
t1 = btf_type_by_id(btf1, btf_id);
if (!t1 || !btf_type_is_func(t1))
return -EFAULT;
return btf_check_func_type_match(log, btf1, t1, btf2, t2);
}
static bool btf_is_dynptr_ptr(const struct btf *btf, const struct btf_type *t)
{
const char *name;
t = btf_type_by_id(btf, t->type); /* skip PTR */
while (btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);
/* allow either struct or struct forward declaration */
if (btf_type_is_struct(t) ||
(btf_type_is_fwd(t) && btf_type_kflag(t) == 0)) {
name = btf_str_by_offset(btf, t->name_off);
return name && strcmp(name, "bpf_dynptr") == 0;
}
return false;
}
struct bpf_cand_cache {
const char *name;
u32 name_len;
u16 kind;
u16 cnt;
struct {
const struct btf *btf;
u32 id;
} cands[];
};
static DEFINE_MUTEX(cand_cache_mutex);
static struct bpf_cand_cache *
bpf_core_find_cands(struct bpf_core_ctx *ctx, u32 local_type_id);
static int btf_get_ptr_to_btf_id(struct bpf_verifier_log *log, int arg_idx,
const struct btf *btf, const struct btf_type *t)
{
struct bpf_cand_cache *cc;
struct bpf_core_ctx ctx = {
.btf = btf,
.log = log,
};
u32 kern_type_id, type_id;
int err = 0;
/* skip PTR and modifiers */
type_id = t->type;
t = btf_type_by_id(btf, t->type);
while (btf_type_is_modifier(t)) {
type_id = t->type;
t = btf_type_by_id(btf, t->type);
}
mutex_lock(&cand_cache_mutex);
cc = bpf_core_find_cands(&ctx, type_id);
if (IS_ERR(cc)) {
err = PTR_ERR(cc);
bpf_log(log, "arg#%d reference type('%s %s') candidate matching error: %d\n",
arg_idx, btf_type_str(t), __btf_name_by_offset(btf, t->name_off),
err);
goto cand_cache_unlock;
}
if (cc->cnt != 1) {
bpf_log(log, "arg#%d reference type('%s %s') %s\n",
arg_idx, btf_type_str(t), __btf_name_by_offset(btf, t->name_off),
cc->cnt == 0 ? "has no matches" : "is ambiguous");
err = cc->cnt == 0 ? -ENOENT : -ESRCH;
goto cand_cache_unlock;
}
if (btf_is_module(cc->cands[0].btf)) {
bpf_log(log, "arg#%d reference type('%s %s') points to kernel module type (unsupported)\n",
arg_idx, btf_type_str(t), __btf_name_by_offset(btf, t->name_off));
err = -EOPNOTSUPP;
goto cand_cache_unlock;
}
kern_type_id = cc->cands[0].id;
cand_cache_unlock:
mutex_unlock(&cand_cache_mutex);
if (err)
return err;
return kern_type_id;
}
enum btf_arg_tag {
ARG_TAG_CTX = BIT_ULL(0),
ARG_TAG_NONNULL = BIT_ULL(1),
ARG_TAG_TRUSTED = BIT_ULL(2),
ARG_TAG_UNTRUSTED = BIT_ULL(3),
ARG_TAG_NULLABLE = BIT_ULL(4),
ARG_TAG_ARENA = BIT_ULL(5),
};
/* Process BTF of a function to produce high-level expectation of function
* arguments (like ARG_PTR_TO_CTX, or ARG_PTR_TO_MEM, etc). This information
* is cached in subprog info for reuse.
* Returns:
* EFAULT - there is a verifier bug. Abort verification.
* EINVAL - cannot convert BTF.
* 0 - Successfully processed BTF and constructed argument expectations.
*/
int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
{
bool is_global = subprog_aux(env, subprog)->linkage == BTF_FUNC_GLOBAL;
struct bpf_subprog_info *sub = subprog_info(env, subprog);
struct bpf_verifier_log *log = &env->log;
struct bpf_prog *prog = env->prog;
enum bpf_prog_type prog_type = prog->type;
struct btf *btf = prog->aux->btf;
const struct btf_param *args;
const struct btf_type *t, *ref_t, *fn_t;
u32 i, nargs, btf_id;
const char *tname;
if (sub->args_cached)
return 0;
if (!prog->aux->func_info) {
verifier_bug(env, "func_info undefined");
return -EFAULT;
}
btf_id = prog->aux->func_info[subprog].type_id;
if (!btf_id) {
if (!is_global) /* not fatal for static funcs */
return -EINVAL;
bpf_log(log, "Global functions need valid BTF\n");
return -EFAULT;
}
fn_t = btf_type_by_id(btf, btf_id);
if (!fn_t || !btf_type_is_func(fn_t)) {
/* These checks were already done by the verifier while loading
* struct bpf_func_info
*/
bpf_log(log, "BTF of func#%d doesn't point to KIND_FUNC\n",
subprog);
return -EFAULT;
}
tname = btf_name_by_offset(btf, fn_t->name_off);
if (prog->aux->func_info_aux[subprog].unreliable) {
verifier_bug(env, "unreliable BTF for function %s()", tname);
return -EFAULT;
}
if (prog_type == BPF_PROG_TYPE_EXT)
prog_type = prog->aux->dst_prog->type;
t = btf_type_by_id(btf, fn_t->type);
if (!t || !btf_type_is_func_proto(t)) {
bpf_log(log, "Invalid type of function %s()\n", tname);
return -EFAULT;
}
args = (const struct btf_param *)(t + 1);
nargs = btf_type_vlen(t);
if (nargs > MAX_BPF_FUNC_REG_ARGS) {
if (!is_global)
return -EINVAL;
bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n",
tname, nargs, MAX_BPF_FUNC_REG_ARGS);
return -EINVAL;
}
/* check that function is void or returns int, exception cb also requires this */
t = btf_type_by_id(btf, t->type);
while (btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);
if (!btf_type_is_void(t) && !btf_type_is_int(t) && !btf_is_any_enum(t)) {
if (!is_global)
return -EINVAL;
bpf_log(log,
"Global function %s() return value not void or scalar. "
"Only those are supported.\n",
tname);
return -EINVAL;
}
/* Convert BTF function arguments into verifier types.
* Only PTR_TO_CTX and SCALAR are supported atm.
*/
for (i = 0; i < nargs; i++) {
u32 tags = 0;
int id = btf_named_start_id(btf, false) - 1;
/* 'arg:<tag>' decl_tag takes precedence over derivation of
* register type from BTF type itself
*/
while ((id = btf_find_next_decl_tag(btf, fn_t, i, "arg:", id)) > 0) {
const struct btf_type *tag_t = btf_type_by_id(btf, id);
const char *tag = __btf_name_by_offset(btf, tag_t->name_off) + 4;
/* disallow arg tags in static subprogs */
if (!is_global) {
bpf_log(log, "arg#%d type tag is not supported in static functions\n", i);
return -EOPNOTSUPP;
}
if (strcmp(tag, "ctx") == 0) {
tags |= ARG_TAG_CTX;
} else if (strcmp(tag, "trusted") == 0) {
tags |= ARG_TAG_TRUSTED;
} else if (strcmp(tag, "untrusted") == 0) {
tags |= ARG_TAG_UNTRUSTED;
} else if (strcmp(tag, "nonnull") == 0) {
tags |= ARG_TAG_NONNULL;
} else if (strcmp(tag, "nullable") == 0) {
tags |= ARG_TAG_NULLABLE;
} else if (strcmp(tag, "arena") == 0) {
tags |= ARG_TAG_ARENA;
} else {
bpf_log(log, "arg#%d has unsupported set of tags\n", i);
return -EOPNOTSUPP;
}
}
if (id != -ENOENT) {
bpf_log(log, "arg#%d type tag fetching failure: %d\n", i, id);
return id;
}
t = btf_type_by_id(btf, args[i].type);
while (btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);
if (!btf_type_is_ptr(t))
goto skip_pointer;
if ((tags & ARG_TAG_CTX) || btf_is_prog_ctx_type(log, btf, t, prog_type, i)) {
if (tags & ~ARG_TAG_CTX) {
bpf_log(log, "arg#%d has invalid combination of tags\n", i);
return -EINVAL;
}
if ((tags & ARG_TAG_CTX) &&
btf_validate_prog_ctx_type(log, btf, t, i, prog_type,
prog->expected_attach_type))
return -EINVAL;
sub->args[i].arg_type = ARG_PTR_TO_CTX;
continue;
}
if (btf_is_dynptr_ptr(btf, t)) {
if (tags) {
bpf_log(log, "arg#%d has invalid combination of tags\n", i);
return -EINVAL;
}
sub->args[i].arg_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY;
continue;
}
if (tags & ARG_TAG_TRUSTED) {
int kern_type_id;
if (tags & ARG_TAG_NONNULL) {
bpf_log(log, "arg#%d has invalid combination of tags\n", i);
return -EINVAL;
}
kern_type_id = btf_get_ptr_to_btf_id(log, i, btf, t);
if (kern_type_id < 0)
return kern_type_id;
sub->args[i].arg_type = ARG_PTR_TO_BTF_ID | PTR_TRUSTED;
if (tags & ARG_TAG_NULLABLE)
sub->args[i].arg_type |= PTR_MAYBE_NULL;
sub->args[i].btf_id = kern_type_id;
continue;
}
if (tags & ARG_TAG_UNTRUSTED) {
struct btf *vmlinux_btf;
int kern_type_id;
if (tags & ~ARG_TAG_UNTRUSTED) {
bpf_log(log, "arg#%d untrusted cannot be combined with any other tags\n", i);
return -EINVAL;
}
ref_t = btf_type_skip_modifiers(btf, t->type, NULL);
if (btf_type_is_void(ref_t) || btf_type_is_primitive(ref_t)) {
sub->args[i].arg_type = ARG_PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED;
sub->args[i].mem_size = 0;
continue;
}
kern_type_id = btf_get_ptr_to_btf_id(log, i, btf, t);
if (kern_type_id < 0)
return kern_type_id;
vmlinux_btf = bpf_get_btf_vmlinux();
ref_t = btf_type_by_id(vmlinux_btf, kern_type_id);
if (!btf_type_is_struct(ref_t)) {
tname = __btf_name_by_offset(vmlinux_btf, t->name_off);
bpf_log(log, "arg#%d has type %s '%s', but only struct or primitive types are allowed\n",
i, btf_type_str(ref_t), tname);
return -EINVAL;
}
sub->args[i].arg_type = ARG_PTR_TO_BTF_ID | PTR_UNTRUSTED;
sub->args[i].btf_id = kern_type_id;
continue;
}
if (tags & ARG_TAG_ARENA) {
if (tags & ~ARG_TAG_ARENA) {
bpf_log(log, "arg#%d arena cannot be combined with any other tags\n", i);
return -EINVAL;
}
sub->args[i].arg_type = ARG_PTR_TO_ARENA;
continue;
}
if (is_global) { /* generic user data pointer */
u32 mem_size;
if (tags & ARG_TAG_NULLABLE) {
bpf_log(log, "arg#%d has invalid combination of tags\n", i);
return -EINVAL;
}
t = btf_type_skip_modifiers(btf, t->type, NULL);
ref_t = btf_resolve_size(btf, t, &mem_size);
if (IS_ERR(ref_t)) {
bpf_log(log, "arg#%d reference type('%s %s') size cannot be determined: %ld\n",
i, btf_type_str(t), btf_name_by_offset(btf, t->name_off),
PTR_ERR(ref_t));
return -EINVAL;
}
sub->args[i].arg_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL;
if (tags & ARG_TAG_NONNULL)
sub->args[i].arg_type &= ~PTR_MAYBE_NULL;
sub->args[i].mem_size = mem_size;
continue;
}
skip_pointer:
if (tags) {
bpf_log(log, "arg#%d has pointer tag, but is not a pointer type\n", i);
return -EINVAL;
}
if (btf_type_is_int(t) || btf_is_any_enum(t)) {
sub->args[i].arg_type = ARG_ANYTHING;
continue;
}
if (!is_global)
return -EINVAL;
bpf_log(log, "Arg#%d type %s in %s() is not supported yet.\n",
i, btf_type_str(t), tname);
return -EINVAL;
}
sub->arg_cnt = nargs;
sub->args_cached = true;
return 0;
}
static void btf_type_show(const struct btf *btf, u32 type_id, void *obj,
struct btf_show *show)
{
const struct btf_type *t = btf_type_by_id(btf, type_id);
show->btf = btf;
memset(&show->state, 0, sizeof(show->state));
memset(&show->obj, 0, sizeof(show->obj));
btf_type_ops(t)->show(btf, t, type_id, obj, 0, show);
}
__printf(2, 0) static void btf_seq_show(struct btf_show *show, const char *fmt,
va_list args)
{
seq_vprintf((struct seq_file *)show->target, fmt, args);
}
int btf_type_seq_show_flags(const struct btf *btf, u32 type_id,
void *obj, struct seq_file *m, u64 flags)
{
struct btf_show sseq;
sseq.target = m;
sseq.showfn = btf_seq_show;
sseq.flags = flags;
btf_type_show(btf, type_id, obj, &sseq);
return sseq.state.status;
}
void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,
struct seq_file *m)
{
(void) btf_type_seq_show_flags(btf, type_id, obj, m,
BTF_SHOW_NONAME | BTF_SHOW_COMPACT |
BTF_SHOW_ZERO | BTF_SHOW_UNSAFE);
}
struct btf_show_snprintf {
struct btf_show show;
int len_left; /* space left in string */
int len; /* length we would have written */
};
__printf(2, 0) static void btf_snprintf_show(struct btf_show *show, const char *fmt,
va_list args)
{
struct btf_show_snprintf *ssnprintf = (struct btf_show_snprintf *)show;
int len;
len = vsnprintf(show->target, ssnprintf->len_left, fmt, args);
if (len < 0) {
ssnprintf->len_left = 0;
ssnprintf->len = len;
} else if (len >= ssnprintf->len_left) {
/* no space, drive on to get length we would have written */
ssnprintf->len_left = 0;
ssnprintf->len += len;
} else {
ssnprintf->len_left -= len;
ssnprintf->len += len;
show->target += len;
}
}
int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
char *buf, int len, u64 flags)
{
struct btf_show_snprintf ssnprintf;
ssnprintf.show.target = buf;
ssnprintf.show.flags = flags;
ssnprintf.show.showfn = btf_snprintf_show;
ssnprintf.len_left = len;
ssnprintf.len = 0;
btf_type_show(btf, type_id, obj, (struct btf_show *)&ssnprintf);
/* If we encountered an error, return it. */
if (ssnprintf.show.state.status)
return ssnprintf.show.state.status;
/* Otherwise return length we would have written */
return ssnprintf.len;
}
#ifdef CONFIG_PROC_FS
static void bpf_btf_show_fdinfo(struct seq_file *m, struct file *filp)
{
const struct btf *btf = filp->private_data;
seq_printf(m, "btf_id:\t%u\n", READ_ONCE(btf->id));
}
#endif
static int btf_release(struct inode *inode, struct file *filp)
{
btf_put(filp->private_data);
return 0;
}
const struct file_operations btf_fops = {
#ifdef CONFIG_PROC_FS
.show_fdinfo = bpf_btf_show_fdinfo,
#endif
.release = btf_release,
};
static int __btf_new_fd(struct btf *btf)
{
return anon_inode_getfd("btf", &btf_fops, btf, O_RDONLY | O_CLOEXEC);
}
int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
{
struct btf *btf;
int ret;
btf = btf_parse(attr, uattr, uattr_size);
if (IS_ERR(btf))
return PTR_ERR(btf);
ret = btf_alloc_id(btf);
if (ret) {
btf_free(btf);
return ret;
}
/*
* The BTF ID is published to the userspace.
* All BTF free must go through call_rcu() from
* now on (i.e. free by calling btf_put()).
*/
ret = __btf_new_fd(btf);
if (ret < 0)
btf_put(btf);
return ret;
}
struct btf *btf_get_by_fd(int fd)
{
struct btf *btf;
CLASS(fd, f)(fd);
btf = __btf_get_by_fd(f);
if (!IS_ERR(btf))
refcount_inc(&btf->refcnt);
return btf;
}
int btf_get_info_by_fd(const struct btf *btf,
const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
struct bpf_btf_info __user *uinfo;
struct bpf_btf_info info;
u32 info_copy, btf_copy;
void __user *ubtf;
char __user *uname;
u32 uinfo_len, uname_len, name_len;
int ret = 0;
uinfo = u64_to_user_ptr(attr->info.info);
uinfo_len = attr->info.info_len;
info_copy = min_t(u32, uinfo_len, sizeof(info));
memset(&info, 0, sizeof(info));
if (copy_from_user(&info, uinfo, info_copy))
return -EFAULT;
info.id = READ_ONCE(btf->id);
ubtf = u64_to_user_ptr(info.btf);
btf_copy = min_t(u32, btf->data_size, info.btf_size);
if (copy_to_user(ubtf, btf->data, btf_copy))
return -EFAULT;
info.btf_size = btf->data_size;
info.kernel_btf = btf->kernel_btf;
uname = u64_to_user_ptr(info.name);
uname_len = info.name_len;
if (!uname ^ !uname_len)
return -EINVAL;
name_len = strlen(btf->name);
info.name_len = name_len;
if (uname) {
if (uname_len >= name_len + 1) {
if (copy_to_user(uname, btf->name, name_len + 1))
return -EFAULT;
} else {
char zero = '\0';
if (copy_to_user(uname, btf->name, uname_len - 1))
return -EFAULT;
if (put_user(zero, uname + uname_len - 1))
return -EFAULT;
/* let user-space know about too short buffer */
ret = -ENOSPC;
}
}
if (copy_to_user(uinfo, &info, info_copy) ||
put_user(info_copy, &uattr->info.info_len))
return -EFAULT;
return ret;
}
int btf_get_fd_by_id(u32 id)
{
struct btf *btf;
int fd;
rcu_read_lock();
btf = idr_find(&btf_idr, id);
if (!btf || !refcount_inc_not_zero(&btf->refcnt))
btf = ERR_PTR(-ENOENT);
rcu_read_unlock();
if (IS_ERR(btf))
return PTR_ERR(btf);
fd = __btf_new_fd(btf);
if (fd < 0)
btf_put(btf);
return fd;
}
u32 btf_obj_id(const struct btf *btf)
{
return READ_ONCE(btf->id);
}
bool btf_is_kernel(const struct btf *btf)
{
return btf->kernel_btf;
}
bool btf_is_module(const struct btf *btf)
{
return btf->kernel_btf && strcmp(btf->name, "vmlinux") != 0;
}
enum {
BTF_MODULE_F_LIVE = (1 << 0),
};
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
struct btf_module {
struct list_head list;
struct module *module;
struct btf *btf;
struct bin_attribute *sysfs_attr;
int flags;
};
static LIST_HEAD(btf_modules);
static DEFINE_MUTEX(btf_module_mutex);
static void purge_cand_cache(struct btf *btf);
static int btf_module_notify(struct notifier_block *nb, unsigned long op,
void *module)
{
struct btf_module *btf_mod, *tmp;
struct module *mod = module;
struct btf *btf;
int err = 0;
if (mod->btf_data_size == 0 ||
(op != MODULE_STATE_COMING && op != MODULE_STATE_LIVE &&
op != MODULE_STATE_GOING))
goto out;
switch (op) {
case MODULE_STATE_COMING:
btf_mod = kzalloc_obj(*btf_mod);
if (!btf_mod) {
err = -ENOMEM;
goto out;
}
btf = btf_parse_module(mod->name, mod->btf_data, mod->btf_data_size,
mod->btf_base_data, mod->btf_base_data_size);
if (IS_ERR(btf)) {
kfree(btf_mod);
if (!IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH)) {
pr_warn("failed to validate module [%s] BTF: %ld\n",
mod->name, PTR_ERR(btf));
err = PTR_ERR(btf);
} else {
pr_warn_once("Kernel module BTF mismatch detected, BTF debug info may be unavailable for some modules\n");
}
goto out;
}
err = btf_alloc_id(btf);
if (err) {
btf_free(btf);
kfree(btf_mod);
goto out;
}
purge_cand_cache(NULL);
mutex_lock(&btf_module_mutex);
btf_mod->module = module;
btf_mod->btf = btf;
list_add(&btf_mod->list, &btf_modules);
mutex_unlock(&btf_module_mutex);
if (IS_ENABLED(CONFIG_SYSFS)) {
struct bin_attribute *attr;
attr = kzalloc_obj(*attr);
if (!attr)
goto out;
sysfs_bin_attr_init(attr);
attr->attr.name = btf->name;
attr->attr.mode = 0444;
attr->size = btf->data_size;
attr->private = btf->data;
attr->read = sysfs_bin_attr_simple_read;
err = sysfs_create_bin_file(btf_kobj, attr);
if (err) {
pr_warn("failed to register module [%s] BTF in sysfs: %d\n",
mod->name, err);
kfree(attr);
err = 0;
goto out;
}
btf_mod->sysfs_attr = attr;
}
break;
case MODULE_STATE_LIVE:
mutex_lock(&btf_module_mutex);
list_for_each_entry_safe(btf_mod, tmp, &btf_modules, list) {
if (btf_mod->module != module)
continue;
btf_mod->flags |= BTF_MODULE_F_LIVE;
break;
}
mutex_unlock(&btf_module_mutex);
break;
case MODULE_STATE_GOING:
mutex_lock(&btf_module_mutex);
list_for_each_entry_safe(btf_mod, tmp, &btf_modules, list) {
if (btf_mod->module != module)
continue;
/*
* For modules, we do the freeing of BTF IDR as soon as
* module goes away to disable BTF discovery, since the
* btf_try_get_module() on such BTFs will fail. This may
* be called again on btf_put(), but it's ok to do so.
*/
btf_free_id(btf_mod->btf);
list_del(&btf_mod->list);
if (btf_mod->sysfs_attr)
sysfs_remove_bin_file(btf_kobj, btf_mod->sysfs_attr);
purge_cand_cache(btf_mod->btf);
btf_put(btf_mod->btf);
kfree(btf_mod->sysfs_attr);
kfree(btf_mod);
break;
}
mutex_unlock(&btf_module_mutex);
break;
}
out:
return notifier_from_errno(err);
}
static struct notifier_block btf_module_nb = {
.notifier_call = btf_module_notify,
};
static int __init btf_module_init(void)
{
register_module_notifier(&btf_module_nb);
return 0;
}
fs_initcall(btf_module_init);
#endif /* CONFIG_DEBUG_INFO_BTF_MODULES */
struct module *btf_try_get_module(const struct btf *btf)
{
struct module *res = NULL;
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
struct btf_module *btf_mod, *tmp;
mutex_lock(&btf_module_mutex);
list_for_each_entry_safe(btf_mod, tmp, &btf_modules, list) {
if (btf_mod->btf != btf)
continue;
/* We must only consider module whose __init routine has
* finished, hence we must check for BTF_MODULE_F_LIVE flag,
* which is set from the notifier callback for
* MODULE_STATE_LIVE.
*/
if ((btf_mod->flags & BTF_MODULE_F_LIVE) && try_module_get(btf_mod->module))
res = btf_mod->module;
break;
}
mutex_unlock(&btf_module_mutex);
#endif
return res;
}
/* Returns struct btf corresponding to the struct module.
* This function can return NULL or ERR_PTR.
*/
static struct btf *btf_get_module_btf(const struct module *module)
{
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
struct btf_module *btf_mod, *tmp;
#endif
struct btf *btf = NULL;
if (!module) {
btf = bpf_get_btf_vmlinux();
if (!IS_ERR_OR_NULL(btf))
btf_get(btf);
return btf;
}
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
mutex_lock(&btf_module_mutex);
list_for_each_entry_safe(btf_mod, tmp, &btf_modules, list) {
if (btf_mod->module != module)
continue;
btf_get(btf_mod->btf);
btf = btf_mod->btf;
break;
}
mutex_unlock(&btf_module_mutex);
#endif
return btf;
}
static int check_btf_kconfigs(const struct module *module, const char *feature)
{
if (!module && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
pr_err("missing vmlinux BTF, cannot register %s\n", feature);
return -ENOENT;
}
if (module && IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES))
pr_warn("missing module BTF, cannot register %s\n", feature);
return 0;
}
BPF_CALL_4(bpf_btf_find_by_name_kind, char *, name, int, name_sz, u32, kind, int, flags)
{
struct btf *btf = NULL;
int btf_obj_fd = 0;
long ret;
if (flags)
return -EINVAL;
if (name_sz <= 1 || name[name_sz - 1])
return -EINVAL;
ret = bpf_find_btf_id(name, kind, &btf);
if (ret > 0 && btf_is_module(btf)) {
btf_obj_fd = __btf_new_fd(btf);
if (btf_obj_fd < 0) {
btf_put(btf);
return btf_obj_fd;
}
return ret | (((u64)btf_obj_fd) << 32);
}
if (ret > 0)
btf_put(btf);
return ret;
}
const struct bpf_func_proto bpf_btf_find_by_name_kind_proto = {
.func = bpf_btf_find_by_name_kind,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_ANYTHING,
};
BTF_ID_LIST_GLOBAL(btf_tracing_ids, MAX_BTF_TRACING_TYPE)
#define BTF_TRACING_TYPE(name, type) BTF_ID(struct, type)
BTF_TRACING_TYPE_xxx
#undef BTF_TRACING_TYPE
/* Validate well-formedness of iter argument type.
* On success, return positive BTF ID of iter state's STRUCT type.
* On error, negative error is returned.
*/
int btf_check_iter_arg(struct btf *btf, const struct btf_type *func, int arg_idx)
{
const struct btf_param *arg;
const struct btf_type *t;
const char *name;
int btf_id;
if (btf_type_vlen(func) <= arg_idx)
return -EINVAL;
arg = &btf_params(func)[arg_idx];
t = btf_type_skip_modifiers(btf, arg->type, NULL);
if (!t || !btf_type_is_ptr(t))
return -EINVAL;
t = btf_type_skip_modifiers(btf, t->type, &btf_id);
if (!t || !__btf_type_is_struct(t))
return -EINVAL;
name = btf_name_by_offset(btf, t->name_off);
if (!name || strncmp(name, ITER_PREFIX, sizeof(ITER_PREFIX) - 1))
return -EINVAL;
return btf_id;
}
static int btf_check_iter_kfuncs(struct btf *btf, const char *func_name,
const struct btf_type *func, u32 func_flags)
{
u32 flags = func_flags & (KF_ITER_NEW | KF_ITER_NEXT | KF_ITER_DESTROY);
const char *sfx, *iter_name;
const struct btf_type *t;
char exp_name[128];
u32 nr_args;
int btf_id;
/* exactly one of KF_ITER_{NEW,NEXT,DESTROY} can be set */
if (!flags || (flags & (flags - 1)))
return -EINVAL;
/* any BPF iter kfunc should have `struct bpf_iter_<type> *` first arg */
nr_args = btf_type_vlen(func);
if (nr_args < 1)
return -EINVAL;
btf_id = btf_check_iter_arg(btf, func, 0);
if (btf_id < 0)
return btf_id;
/* sizeof(struct bpf_iter_<type>) should be a multiple of 8 to
* fit nicely in stack slots
*/
t = btf_type_by_id(btf, btf_id);
if (t->size == 0 || (t->size % 8))
return -EINVAL;
/* validate bpf_iter_<type>_{new,next,destroy}(struct bpf_iter_<type> *)
* naming pattern
*/
iter_name = btf_name_by_offset(btf, t->name_off) + sizeof(ITER_PREFIX) - 1;
if (flags & KF_ITER_NEW)
sfx = "new";
else if (flags & KF_ITER_NEXT)
sfx = "next";
else /* (flags & KF_ITER_DESTROY) */
sfx = "destroy";
snprintf(exp_name, sizeof(exp_name), "bpf_iter_%s_%s", iter_name, sfx);
if (strcmp(func_name, exp_name))
return -EINVAL;
/* only iter constructor should have extra arguments */
if (!(flags & KF_ITER_NEW) && nr_args != 1)
return -EINVAL;
if (flags & KF_ITER_NEXT) {
/* bpf_iter_<type>_next() should return pointer */
t = btf_type_skip_modifiers(btf, func->type, NULL);
if (!t || !btf_type_is_ptr(t))
return -EINVAL;
}
if (flags & KF_ITER_DESTROY) {
/* bpf_iter_<type>_destroy() should return void */
t = btf_type_by_id(btf, func->type);
if (!t || !btf_type_is_void(t))
return -EINVAL;
}
return 0;
}
static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags)
{
const struct btf_type *func;
const char *func_name;
int err;
/* any kfunc should be FUNC -> FUNC_PROTO */
func = btf_type_by_id(btf, func_id);
if (!func || !btf_type_is_func(func))
return -EINVAL;
/* sanity check kfunc name */
func_name = btf_name_by_offset(btf, func->name_off);
if (!func_name || !func_name[0])
return -EINVAL;
func = btf_type_by_id(btf, func->type);
if (!func || !btf_type_is_func_proto(func))
return -EINVAL;
if (func_flags & (KF_ITER_NEW | KF_ITER_NEXT | KF_ITER_DESTROY)) {
err = btf_check_iter_kfuncs(btf, func_name, func, func_flags);
if (err)
return err;
}
return 0;
}
/* Kernel Function (kfunc) BTF ID set registration API */
static int btf_populate_kfunc_set(struct btf *btf, enum btf_kfunc_hook hook,
const struct btf_kfunc_id_set *kset)
{
struct btf_kfunc_hook_filter *hook_filter;
struct btf_id_set8 *add_set = kset->set;
bool vmlinux_set = !btf_is_module(btf);
bool add_filter = !!kset->filter;
struct btf_kfunc_set_tab *tab;
struct btf_id_set8 *set;
u32 set_cnt, i;
int ret;
if (hook >= BTF_KFUNC_HOOK_MAX) {
ret = -EINVAL;
goto end;
}
if (!add_set->cnt)
return 0;
tab = btf->kfunc_set_tab;
if (tab && add_filter) {
u32 i;
hook_filter = &tab->hook_filters[hook];
for (i = 0; i < hook_filter->nr_filters; i++) {
if (hook_filter->filters[i] == kset->filter) {
add_filter = false;
break;
}
}
if (add_filter && hook_filter->nr_filters == BTF_KFUNC_FILTER_MAX_CNT) {
ret = -E2BIG;
goto end;
}
}
if (!tab) {
tab = kzalloc_obj(*tab, GFP_KERNEL | __GFP_NOWARN);
if (!tab)
return -ENOMEM;
btf->kfunc_set_tab = tab;
}
set = tab->sets[hook];
/* Warn when register_btf_kfunc_id_set is called twice for the same hook
* for module sets.
*/
if (WARN_ON_ONCE(set && !vmlinux_set)) {
ret = -EINVAL;
goto end;
}
/* In case of vmlinux sets, there may be more than one set being
* registered per hook. To create a unified set, we allocate a new set
* and concatenate all individual sets being registered. While each set
* is individually sorted, they may become unsorted when concatenated,
* hence re-sorting the final set again is required to make binary
* searching the set using btf_id_set8_contains function work.
*
* For module sets, we need to allocate as we may need to relocate
* BTF ids.
*/
set_cnt = set ? set->cnt : 0;
if (set_cnt > U32_MAX - add_set->cnt) {
ret = -EOVERFLOW;
goto end;
}
if (set_cnt + add_set->cnt > BTF_KFUNC_SET_MAX_CNT) {
ret = -E2BIG;
goto end;
}
/* Grow set */
set = krealloc(tab->sets[hook],
struct_size(set, pairs, set_cnt + add_set->cnt),
GFP_KERNEL | __GFP_NOWARN);
if (!set) {
ret = -ENOMEM;
goto end;
}
/* For newly allocated set, initialize set->cnt to 0 */
if (!tab->sets[hook])
set->cnt = 0;
tab->sets[hook] = set;
/* Concatenate the two sets */
memcpy(set->pairs + set->cnt, add_set->pairs, add_set->cnt * sizeof(set->pairs[0]));
/* Now that the set is copied, update with relocated BTF ids */
for (i = set->cnt; i < set->cnt + add_set->cnt; i++)
set->pairs[i].id = btf_relocate_id(btf, set->pairs[i].id);
set->cnt += add_set->cnt;
sort(set->pairs, set->cnt, sizeof(set->pairs[0]), btf_id_cmp_func, NULL);
if (add_filter) {
hook_filter = &tab->hook_filters[hook];
hook_filter->filters[hook_filter->nr_filters++] = kset->filter;
}
return 0;
end:
btf_free_kfunc_set_tab(btf);
return ret;
}
static u32 *btf_kfunc_id_set_contains(const struct btf *btf,
enum btf_kfunc_hook hook,
u32 kfunc_btf_id)
{
struct btf_id_set8 *set;
u32 *id;
if (hook >= BTF_KFUNC_HOOK_MAX)
return NULL;
if (!btf->kfunc_set_tab)
return NULL;
set = btf->kfunc_set_tab->sets[hook];
if (!set)
return NULL;
id = btf_id_set8_contains(set, kfunc_btf_id);
if (!id)
return NULL;
/* The flags for BTF ID are located next to it */
return id + 1;
}
static bool __btf_kfunc_is_allowed(const struct btf *btf,
enum btf_kfunc_hook hook,
u32 kfunc_btf_id,
const struct bpf_prog *prog)
{
struct btf_kfunc_hook_filter *hook_filter;
int i;
if (hook >= BTF_KFUNC_HOOK_MAX)
return false;
if (!btf->kfunc_set_tab)
return false;
hook_filter = &btf->kfunc_set_tab->hook_filters[hook];
for (i = 0; i < hook_filter->nr_filters; i++) {
if (hook_filter->filters[i](prog, kfunc_btf_id))
return false;
}
return true;
}
static int bpf_prog_type_to_kfunc_hook(enum bpf_prog_type prog_type)
{
switch (prog_type) {
case BPF_PROG_TYPE_UNSPEC:
return BTF_KFUNC_HOOK_COMMON;
case BPF_PROG_TYPE_XDP:
return BTF_KFUNC_HOOK_XDP;
case BPF_PROG_TYPE_SCHED_CLS:
return BTF_KFUNC_HOOK_TC;
case BPF_PROG_TYPE_STRUCT_OPS:
return BTF_KFUNC_HOOK_STRUCT_OPS;
case BPF_PROG_TYPE_TRACING:
case BPF_PROG_TYPE_TRACEPOINT:
case BPF_PROG_TYPE_RAW_TRACEPOINT:
case BPF_PROG_TYPE_PERF_EVENT:
case BPF_PROG_TYPE_LSM:
return BTF_KFUNC_HOOK_TRACING;
case BPF_PROG_TYPE_SYSCALL:
return BTF_KFUNC_HOOK_SYSCALL;
case BPF_PROG_TYPE_CGROUP_SKB:
case BPF_PROG_TYPE_CGROUP_SOCK:
case BPF_PROG_TYPE_CGROUP_DEVICE:
case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
case BPF_PROG_TYPE_CGROUP_SOCKOPT:
case BPF_PROG_TYPE_CGROUP_SYSCTL:
case BPF_PROG_TYPE_SOCK_OPS:
return BTF_KFUNC_HOOK_CGROUP;
case BPF_PROG_TYPE_SCHED_ACT:
return BTF_KFUNC_HOOK_SCHED_ACT;
case BPF_PROG_TYPE_SK_SKB:
return BTF_KFUNC_HOOK_SK_SKB;
case BPF_PROG_TYPE_SOCKET_FILTER:
return BTF_KFUNC_HOOK_SOCKET_FILTER;
case BPF_PROG_TYPE_LWT_OUT:
case BPF_PROG_TYPE_LWT_IN:
case BPF_PROG_TYPE_LWT_XMIT:
case BPF_PROG_TYPE_LWT_SEG6LOCAL:
return BTF_KFUNC_HOOK_LWT;
case BPF_PROG_TYPE_NETFILTER:
return BTF_KFUNC_HOOK_NETFILTER;
case BPF_PROG_TYPE_KPROBE:
return BTF_KFUNC_HOOK_KPROBE;
default:
return BTF_KFUNC_HOOK_MAX;
}
}
bool btf_kfunc_is_allowed(const struct btf *btf,
u32 kfunc_btf_id,
const struct bpf_prog *prog)
{
enum bpf_prog_type prog_type = resolve_prog_type(prog);
enum btf_kfunc_hook hook;
u32 *kfunc_flags;
kfunc_flags = btf_kfunc_id_set_contains(btf, BTF_KFUNC_HOOK_COMMON, kfunc_btf_id);
if (kfunc_flags && __btf_kfunc_is_allowed(btf, BTF_KFUNC_HOOK_COMMON, kfunc_btf_id, prog))
return true;
hook = bpf_prog_type_to_kfunc_hook(prog_type);
kfunc_flags = btf_kfunc_id_set_contains(btf, hook, kfunc_btf_id);
if (kfunc_flags && __btf_kfunc_is_allowed(btf, hook, kfunc_btf_id, prog))
return true;
return false;
}
/* Caution:
* Reference to the module (obtained using btf_try_get_module) corresponding to
* the struct btf *MUST* be held when calling this function from verifier
* context. This is usually true as we stash references in prog's kfunc_btf_tab;
* keeping the reference for the duration of the call provides the necessary
* protection for looking up a well-formed btf->kfunc_set_tab.
*/
u32 *btf_kfunc_flags(const struct btf *btf, u32 kfunc_btf_id, const struct bpf_prog *prog)
{
enum bpf_prog_type prog_type = resolve_prog_type(prog);
enum btf_kfunc_hook hook;
u32 *kfunc_flags;
kfunc_flags = btf_kfunc_id_set_contains(btf, BTF_KFUNC_HOOK_COMMON, kfunc_btf_id);
if (kfunc_flags)
return kfunc_flags;
hook = bpf_prog_type_to_kfunc_hook(prog_type);
return btf_kfunc_id_set_contains(btf, hook, kfunc_btf_id);
}
u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id,
const struct bpf_prog *prog)
{
if (!__btf_kfunc_is_allowed(btf, BTF_KFUNC_HOOK_FMODRET, kfunc_btf_id, prog))
return NULL;
return btf_kfunc_id_set_contains(btf, BTF_KFUNC_HOOK_FMODRET, kfunc_btf_id);
}
static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook,
const struct btf_kfunc_id_set *kset)
{
struct btf *btf;
int ret, i;
btf = btf_get_module_btf(kset->owner);
if (!btf)
return check_btf_kconfigs(kset->owner, "kfunc");
if (IS_ERR(btf))
return PTR_ERR(btf);
for (i = 0; i < kset->set->cnt; i++) {
ret = btf_check_kfunc_protos(btf, btf_relocate_id(btf, kset->set->pairs[i].id),
kset->set->pairs[i].flags);
if (ret)
goto err_out;
}
ret = btf_populate_kfunc_set(btf, hook, kset);
err_out:
btf_put(btf);
return ret;
}
/* This function must be invoked only from initcalls/module init functions */
int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,
const struct btf_kfunc_id_set *kset)
{
enum btf_kfunc_hook hook;
/* All kfuncs need to be tagged as such in BTF.
* WARN() for initcall registrations that do not check errors.
*/
if (!(kset->set->flags & BTF_SET8_KFUNCS)) {
WARN_ON(!kset->owner);
return -EINVAL;
}
hook = bpf_prog_type_to_kfunc_hook(prog_type);
return __register_btf_kfunc_id_set(hook, kset);
}
EXPORT_SYMBOL_GPL(register_btf_kfunc_id_set);
/* This function must be invoked only from initcalls/module init functions */
int register_btf_fmodret_id_set(const struct btf_kfunc_id_set *kset)
{
return __register_btf_kfunc_id_set(BTF_KFUNC_HOOK_FMODRET, kset);
}
EXPORT_SYMBOL_GPL(register_btf_fmodret_id_set);
s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id)
{
struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab;
struct btf_id_dtor_kfunc *dtor;
if (!tab)
return -ENOENT;
/* Even though the size of tab->dtors[0] is > sizeof(u32), we only need
* to compare the first u32 with btf_id, so we can reuse btf_id_cmp_func.
*/
BUILD_BUG_ON(offsetof(struct btf_id_dtor_kfunc, btf_id) != 0);
dtor = bsearch(&btf_id, tab->dtors, tab->cnt, sizeof(tab->dtors[0]), btf_id_cmp_func);
if (!dtor)
return -ENOENT;
return dtor->kfunc_btf_id;
}
static int btf_check_dtor_kfuncs(struct btf *btf, const struct btf_id_dtor_kfunc *dtors, u32 cnt)
{
const struct btf_type *dtor_func, *dtor_func_proto, *t;
const struct btf_param *args;
s32 dtor_btf_id;
u32 nr_args, i;
for (i = 0; i < cnt; i++) {
dtor_btf_id = btf_relocate_id(btf, dtors[i].kfunc_btf_id);
dtor_func = btf_type_by_id(btf, dtor_btf_id);
if (!dtor_func || !btf_type_is_func(dtor_func))
return -EINVAL;
dtor_func_proto = btf_type_by_id(btf, dtor_func->type);
if (!dtor_func_proto || !btf_type_is_func_proto(dtor_func_proto))
return -EINVAL;
/* Make sure the prototype of the destructor kfunc is 'void func(type *)' */
t = btf_type_by_id(btf, dtor_func_proto->type);
if (!t || !btf_type_is_void(t))
return -EINVAL;
nr_args = btf_type_vlen(dtor_func_proto);
if (nr_args != 1)
return -EINVAL;
args = btf_params(dtor_func_proto);
t = btf_type_by_id(btf, args[0].type);
/* Allow any pointer type, as width on targets Linux supports
* will be same for all pointer types (i.e. sizeof(void *))
*/
if (!t || !btf_type_is_ptr(t))
return -EINVAL;
if (IS_ENABLED(CONFIG_CFI)) {
/* Ensure the destructor kfunc type matches btf_dtor_kfunc_t */
t = btf_type_by_id(btf, t->type);
if (!btf_type_is_void(t))
return -EINVAL;
}
}
return 0;
}
/* This function must be invoked only from initcalls/module init functions */
int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,
struct module *owner)
{
struct btf_id_dtor_kfunc_tab *tab;
struct btf *btf;
u32 tab_cnt, i;
int ret;
btf = btf_get_module_btf(owner);
if (!btf)
return check_btf_kconfigs(owner, "dtor kfuncs");
if (IS_ERR(btf))
return PTR_ERR(btf);
if (add_cnt >= BTF_DTOR_KFUNC_MAX_CNT) {
pr_err("cannot register more than %d kfunc destructors\n", BTF_DTOR_KFUNC_MAX_CNT);
ret = -E2BIG;
goto end;
}
/* Ensure that the prototype of dtor kfuncs being registered is sane */
ret = btf_check_dtor_kfuncs(btf, dtors, add_cnt);
if (ret < 0)
goto end;
tab = btf->dtor_kfunc_tab;
/* Only one call allowed for modules */
if (WARN_ON_ONCE(tab && btf_is_module(btf))) {
ret = -EINVAL;
goto end;
}
tab_cnt = tab ? tab->cnt : 0;
if (tab_cnt > U32_MAX - add_cnt) {
ret = -EOVERFLOW;
goto end;
}
if (tab_cnt + add_cnt >= BTF_DTOR_KFUNC_MAX_CNT) {
pr_err("cannot register more than %d kfunc destructors\n", BTF_DTOR_KFUNC_MAX_CNT);
ret = -E2BIG;
goto end;
}
tab = krealloc(btf->dtor_kfunc_tab,
struct_size(tab, dtors, tab_cnt + add_cnt),
GFP_KERNEL | __GFP_NOWARN);
if (!tab) {
ret = -ENOMEM;
goto end;
}
if (!btf->dtor_kfunc_tab)
tab->cnt = 0;
btf->dtor_kfunc_tab = tab;
memcpy(tab->dtors + tab->cnt, dtors, add_cnt * sizeof(tab->dtors[0]));
/* remap BTF ids based on BTF relocation (if any) */
for (i = tab_cnt; i < tab_cnt + add_cnt; i++) {
tab->dtors[i].btf_id = btf_relocate_id(btf, tab->dtors[i].btf_id);
tab->dtors[i].kfunc_btf_id = btf_relocate_id(btf, tab->dtors[i].kfunc_btf_id);
}
tab->cnt += add_cnt;
sort(tab->dtors, tab->cnt, sizeof(tab->dtors[0]), btf_id_cmp_func, NULL);
end:
if (ret)
btf_free_dtor_kfunc_tab(btf);
btf_put(btf);
return ret;
}
EXPORT_SYMBOL_GPL(register_btf_id_dtor_kfuncs);
#define MAX_TYPES_ARE_COMPAT_DEPTH 2
/* Check local and target types for compatibility. This check is used for
* type-based CO-RE relocations and follow slightly different rules than
* field-based relocations. This function assumes that root types were already
* checked for name match. Beyond that initial root-level name check, names
* are completely ignored. Compatibility rules are as follows:
* - any two STRUCTs/UNIONs/FWDs/ENUMs/INTs/ENUM64s are considered compatible, but
* kind should match for local and target types (i.e., STRUCT is not
* compatible with UNION);
* - for ENUMs/ENUM64s, the size is ignored;
* - for INT, size and signedness are ignored;
* - for ARRAY, dimensionality is ignored, element types are checked for
* compatibility recursively;
* - CONST/VOLATILE/RESTRICT modifiers are ignored;
* - TYPEDEFs/PTRs are compatible if types they pointing to are compatible;
* - FUNC_PROTOs are compatible if they have compatible signature: same
* number of input args and compatible return and argument types.
* These rules are not set in stone and probably will be adjusted as we get
* more experience with using BPF CO-RE relocations.
*/
int bpf_core_types_are_compat(const struct btf *local_btf, __u32 local_id,
const struct btf *targ_btf, __u32 targ_id)
{
return __bpf_core_types_are_compat(local_btf, local_id, targ_btf, targ_id,
MAX_TYPES_ARE_COMPAT_DEPTH);
}
#define MAX_TYPES_MATCH_DEPTH 2
int bpf_core_types_match(const struct btf *local_btf, u32 local_id,
const struct btf *targ_btf, u32 targ_id)
{
return __bpf_core_types_match(local_btf, local_id, targ_btf, targ_id, false,
MAX_TYPES_MATCH_DEPTH);
}
static bool bpf_core_is_flavor_sep(const char *s)
{
/* check X___Y name pattern, where X and Y are not underscores */
return s[0] != '_' && /* X */
s[1] == '_' && s[2] == '_' && s[3] == '_' && /* ___ */
s[4] != '_'; /* Y */
}
size_t bpf_core_essential_name_len(const char *name)
{
size_t n = strlen(name);
int i;
for (i = n - 5; i >= 0; i--) {
if (bpf_core_is_flavor_sep(name + i))
return i + 1;
}
return n;
}
static void bpf_free_cands(struct bpf_cand_cache *cands)
{
if (!cands->cnt)
/* empty candidate array was allocated on stack */
return;
kfree(cands);
}
static void bpf_free_cands_from_cache(struct bpf_cand_cache *cands)
{
kfree(cands->name);
kfree(cands);
}
#define VMLINUX_CAND_CACHE_SIZE 31
static struct bpf_cand_cache *vmlinux_cand_cache[VMLINUX_CAND_CACHE_SIZE];
#define MODULE_CAND_CACHE_SIZE 31
static struct bpf_cand_cache *module_cand_cache[MODULE_CAND_CACHE_SIZE];
static void __print_cand_cache(struct bpf_verifier_log *log,
struct bpf_cand_cache **cache,
int cache_size)
{
struct bpf_cand_cache *cc;
int i, j;
for (i = 0; i < cache_size; i++) {
cc = cache[i];
if (!cc)
continue;
bpf_log(log, "[%d]%s(", i, cc->name);
for (j = 0; j < cc->cnt; j++) {
bpf_log(log, "%d", cc->cands[j].id);
if (j < cc->cnt - 1)
bpf_log(log, " ");
}
bpf_log(log, "), ");
}
}
static void print_cand_cache(struct bpf_verifier_log *log)
{
mutex_lock(&cand_cache_mutex);
bpf_log(log, "vmlinux_cand_cache:");
__print_cand_cache(log, vmlinux_cand_cache, VMLINUX_CAND_CACHE_SIZE);
bpf_log(log, "\nmodule_cand_cache:");
__print_cand_cache(log, module_cand_cache, MODULE_CAND_CACHE_SIZE);
bpf_log(log, "\n");
mutex_unlock(&cand_cache_mutex);
}
static u32 hash_cands(struct bpf_cand_cache *cands)
{
return jhash(cands->name, cands->name_len, 0);
}
static struct bpf_cand_cache *check_cand_cache(struct bpf_cand_cache *cands,
struct bpf_cand_cache **cache,
int cache_size)
{
struct bpf_cand_cache *cc = cache[hash_cands(cands) % cache_size];
if (cc && cc->name_len == cands->name_len &&
!strncmp(cc->name, cands->name, cands->name_len))
return cc;
return NULL;
}
static size_t sizeof_cands(int cnt)
{
return offsetof(struct bpf_cand_cache, cands[cnt]);
}
static struct bpf_cand_cache *populate_cand_cache(struct bpf_cand_cache *cands,
struct bpf_cand_cache **cache,
int cache_size)
{
struct bpf_cand_cache **cc = &cache[hash_cands(cands) % cache_size], *new_cands;
if (*cc) {
bpf_free_cands_from_cache(*cc);
*cc = NULL;
}
new_cands = kmemdup(cands, sizeof_cands(cands->cnt), GFP_KERNEL_ACCOUNT);
if (!new_cands) {
bpf_free_cands(cands);
return ERR_PTR(-ENOMEM);
}
/* strdup the name, since it will stay in cache.
* the cands->name points to strings in prog's BTF and the prog can be unloaded.
*/
new_cands->name = kmemdup_nul(cands->name, cands->name_len, GFP_KERNEL_ACCOUNT);
bpf_free_cands(cands);
if (!new_cands->name) {
kfree(new_cands);
return ERR_PTR(-ENOMEM);
}
*cc = new_cands;
return new_cands;
}
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
static void __purge_cand_cache(struct btf *btf, struct bpf_cand_cache **cache,
int cache_size)
{
struct bpf_cand_cache *cc;
int i, j;
for (i = 0; i < cache_size; i++) {
cc = cache[i];
if (!cc)
continue;
if (!btf) {
/* when new module is loaded purge all of module_cand_cache,
* since new module might have candidates with the name
* that matches cached cands.
*/
bpf_free_cands_from_cache(cc);
cache[i] = NULL;
continue;
}
/* when module is unloaded purge cache entries
* that match module's btf
*/
for (j = 0; j < cc->cnt; j++)
if (cc->cands[j].btf == btf) {
bpf_free_cands_from_cache(cc);
cache[i] = NULL;
break;
}
}
}
static void purge_cand_cache(struct btf *btf)
{
mutex_lock(&cand_cache_mutex);
__purge_cand_cache(btf, module_cand_cache, MODULE_CAND_CACHE_SIZE);
mutex_unlock(&cand_cache_mutex);
}
#endif
static struct bpf_cand_cache *
bpf_core_add_cands(struct bpf_cand_cache *cands, const struct btf *targ_btf,
int targ_start_id)
{
struct bpf_cand_cache *new_cands;
const struct btf_type *t;
const char *targ_name;
size_t targ_essent_len;
int n, i;
n = btf_nr_types(targ_btf);
for (i = targ_start_id; i < n; i++) {
t = btf_type_by_id(targ_btf, i);
if (btf_kind(t) != cands->kind)
continue;
targ_name = btf_name_by_offset(targ_btf, t->name_off);
if (!targ_name)
continue;
/* the resched point is before strncmp to make sure that search
* for non-existing name will have a chance to schedule().
*/
cond_resched();
if (strncmp(cands->name, targ_name, cands->name_len) != 0)
continue;
targ_essent_len = bpf_core_essential_name_len(targ_name);
if (targ_essent_len != cands->name_len)
continue;
/* most of the time there is only one candidate for a given kind+name pair */
new_cands = kmalloc(sizeof_cands(cands->cnt + 1), GFP_KERNEL_ACCOUNT);
if (!new_cands) {
bpf_free_cands(cands);
return ERR_PTR(-ENOMEM);
}
memcpy(new_cands, cands, sizeof_cands(cands->cnt));
bpf_free_cands(cands);
cands = new_cands;
cands->cands[cands->cnt].btf = targ_btf;
cands->cands[cands->cnt].id = i;
cands->cnt++;
}
return cands;
}
static struct bpf_cand_cache *
bpf_core_find_cands(struct bpf_core_ctx *ctx, u32 local_type_id)
{
struct bpf_cand_cache *cands, *cc, local_cand = {};
const struct btf *local_btf = ctx->btf;
const struct btf_type *local_type;
const struct btf *main_btf;
size_t local_essent_len;
struct btf *mod_btf;
const char *name;
int id;
main_btf = bpf_get_btf_vmlinux();
if (IS_ERR(main_btf))
return ERR_CAST(main_btf);
if (!main_btf)
return ERR_PTR(-EINVAL);
local_type = btf_type_by_id(local_btf, local_type_id);
if (!local_type)
return ERR_PTR(-EINVAL);
name = btf_name_by_offset(local_btf, local_type->name_off);
if (str_is_empty(name))
return ERR_PTR(-EINVAL);
local_essent_len = bpf_core_essential_name_len(name);
cands = &local_cand;
cands->name = name;
cands->kind = btf_kind(local_type);
cands->name_len = local_essent_len;
cc = check_cand_cache(cands, vmlinux_cand_cache, VMLINUX_CAND_CACHE_SIZE);
/* cands is a pointer to stack here */
if (cc) {
if (cc->cnt)
return cc;
goto check_modules;
}
/* Attempt to find target candidates in vmlinux BTF first */
cands = bpf_core_add_cands(cands, main_btf, btf_named_start_id(main_btf, true));
if (IS_ERR(cands))
return ERR_CAST(cands);
/* cands is a pointer to kmalloced memory here if cands->cnt > 0 */
/* populate cache even when cands->cnt == 0 */
cc = populate_cand_cache(cands, vmlinux_cand_cache, VMLINUX_CAND_CACHE_SIZE);
if (IS_ERR(cc))
return ERR_CAST(cc);
/* if vmlinux BTF has any candidate, don't go for module BTFs */
if (cc->cnt)
return cc;
check_modules:
/* cands is a pointer to stack here and cands->cnt == 0 */
cc = check_cand_cache(cands, module_cand_cache, MODULE_CAND_CACHE_SIZE);
if (cc)
/* if cache has it return it even if cc->cnt == 0 */
return cc;
/* If candidate is not found in vmlinux's BTF then search in module's BTFs */
spin_lock_bh(&btf_idr_lock);
idr_for_each_entry(&btf_idr, mod_btf, id) {
if (!btf_is_module(mod_btf))
continue;
/* linear search could be slow hence unlock/lock
* the IDR to avoiding holding it for too long
*/
btf_get(mod_btf);
spin_unlock_bh(&btf_idr_lock);
cands = bpf_core_add_cands(cands, mod_btf, btf_named_start_id(mod_btf, true));
btf_put(mod_btf);
if (IS_ERR(cands))
return ERR_CAST(cands);
spin_lock_bh(&btf_idr_lock);
}
spin_unlock_bh(&btf_idr_lock);
/* cands is a pointer to kmalloced memory here if cands->cnt > 0
* or pointer to stack if cands->cnd == 0.
* Copy it into the cache even when cands->cnt == 0 and
* return the result.
*/
return populate_cand_cache(cands, module_cand_cache, MODULE_CAND_CACHE_SIZE);
}
int bpf_core_apply(struct bpf_core_ctx *ctx, const struct bpf_core_relo *relo,
int relo_idx, void *insn)
{
bool need_cands = relo->kind != BPF_CORE_TYPE_ID_LOCAL;
struct bpf_core_cand_list cands = {};
struct bpf_core_relo_res targ_res;
struct bpf_core_spec *specs;
const struct btf_type *type;
int err;
/* ~4k of temp memory necessary to convert LLVM spec like "0:1:0:5"
* into arrays of btf_ids of struct fields and array indices.
*/
specs = kzalloc_objs(*specs, 3, GFP_KERNEL_ACCOUNT);
if (!specs)
return -ENOMEM;
type = btf_type_by_id(ctx->btf, relo->type_id);
if (!type) {
bpf_log(ctx->log, "relo #%u: bad type id %u\n",
relo_idx, relo->type_id);
kfree(specs);
return -EINVAL;
}
if (need_cands) {
struct bpf_cand_cache *cc;
int i;
mutex_lock(&cand_cache_mutex);
cc = bpf_core_find_cands(ctx, relo->type_id);
if (IS_ERR(cc)) {
bpf_log(ctx->log, "target candidate search failed for %d\n",
relo->type_id);
err = PTR_ERR(cc);
goto out;
}
if (cc->cnt) {
cands.cands = kzalloc_objs(*cands.cands, cc->cnt,
GFP_KERNEL_ACCOUNT);
if (!cands.cands) {
err = -ENOMEM;
goto out;
}
}
for (i = 0; i < cc->cnt; i++) {
bpf_log(ctx->log,
"CO-RE relocating %s %s: found target candidate [%d]\n",
btf_kind_str[cc->kind], cc->name, cc->cands[i].id);
cands.cands[i].btf = cc->cands[i].btf;
cands.cands[i].id = cc->cands[i].id;
}
cands.len = cc->cnt;
/* cand_cache_mutex needs to span the cache lookup and
* copy of btf pointer into bpf_core_cand_list,
* since module can be unloaded while bpf_core_calc_relo_insn
* is working with module's btf.
*/
}
err = bpf_core_calc_relo_insn((void *)ctx->log, relo, relo_idx, ctx->btf, &cands, specs,
&targ_res);
if (err)
goto out;
err = bpf_core_patch_insn((void *)ctx->log, insn, relo->insn_off / 8, relo, relo_idx,
&targ_res);
out:
kfree(specs);
if (need_cands) {
kfree(cands.cands);
mutex_unlock(&cand_cache_mutex);
if (ctx->log->level & BPF_LOG_LEVEL2)
print_cand_cache(ctx->log);
}
return err;
}
bool btf_nested_type_is_trusted(struct bpf_verifier_log *log,
const struct bpf_reg_state *reg,
const char *field_name, u32 btf_id, const char *suffix)
{
struct btf *btf = reg->btf;
const struct btf_type *walk_type, *safe_type;
const char *tname;
char safe_tname[64];
long ret, safe_id;
const struct btf_member *member;
u32 i;
walk_type = btf_type_by_id(btf, reg->btf_id);
if (!walk_type)
return false;
tname = btf_name_by_offset(btf, walk_type->name_off);
ret = snprintf(safe_tname, sizeof(safe_tname), "%s%s", tname, suffix);
if (ret >= sizeof(safe_tname))
return false;
safe_id = btf_find_by_name_kind(btf, safe_tname, BTF_INFO_KIND(walk_type->info));
if (safe_id < 0)
return false;
safe_type = btf_type_by_id(btf, safe_id);
if (!safe_type)
return false;
for_each_member(i, safe_type, member) {
const char *m_name = __btf_name_by_offset(btf, member->name_off);
const struct btf_type *mtype = btf_type_by_id(btf, member->type);
u32 id;
if (!btf_type_is_ptr(mtype))
continue;
btf_type_skip_modifiers(btf, mtype->type, &id);
/* If we match on both type and name, the field is considered trusted. */
if (btf_id == id && !strcmp(field_name, m_name))
return true;
}
return false;
}
bool btf_type_ids_nocast_alias(struct bpf_verifier_log *log,
const struct btf *reg_btf, u32 reg_id,
const struct btf *arg_btf, u32 arg_id)
{
const char *reg_name, *arg_name, *search_needle;
const struct btf_type *reg_type, *arg_type;
int reg_len, arg_len, cmp_len;
size_t pattern_len = sizeof(NOCAST_ALIAS_SUFFIX) - sizeof(char);
reg_type = btf_type_by_id(reg_btf, reg_id);
if (!reg_type)
return false;
arg_type = btf_type_by_id(arg_btf, arg_id);
if (!arg_type)
return false;
reg_name = btf_name_by_offset(reg_btf, reg_type->name_off);
arg_name = btf_name_by_offset(arg_btf, arg_type->name_off);
reg_len = strlen(reg_name);
arg_len = strlen(arg_name);
/* Exactly one of the two type names may be suffixed with ___init, so
* if the strings are the same size, they can't possibly be no-cast
* aliases of one another. If you have two of the same type names, e.g.
* they're both nf_conn___init, it would be improper to return true
* because they are _not_ no-cast aliases, they are the same type.
*/
if (reg_len == arg_len)
return false;
/* Either of the two names must be the other name, suffixed with ___init. */
if ((reg_len != arg_len + pattern_len) &&
(arg_len != reg_len + pattern_len))
return false;
if (reg_len < arg_len) {
search_needle = strstr(arg_name, NOCAST_ALIAS_SUFFIX);
cmp_len = reg_len;
} else {
search_needle = strstr(reg_name, NOCAST_ALIAS_SUFFIX);
cmp_len = arg_len;
}
if (!search_needle)
return false;
/* ___init suffix must come at the end of the name */
if (*(search_needle + pattern_len) != '\0')
return false;
return !strncmp(reg_name, arg_name, cmp_len);
}
#ifdef CONFIG_BPF_JIT
static int
btf_add_struct_ops(struct btf *btf, struct bpf_struct_ops *st_ops,
struct bpf_verifier_log *log)
{
struct btf_struct_ops_tab *tab, *new_tab;
int i, err;
tab = btf->struct_ops_tab;
if (!tab) {
tab = kzalloc_flex(*tab, ops, 4);
if (!tab)
return -ENOMEM;
tab->capacity = 4;
btf->struct_ops_tab = tab;
}
for (i = 0; i < tab->cnt; i++)
if (tab->ops[i].st_ops == st_ops)
return -EEXIST;
if (tab->cnt == tab->capacity) {
new_tab = krealloc(tab,
struct_size(tab, ops, tab->capacity * 2),
GFP_KERNEL);
if (!new_tab)
return -ENOMEM;
tab = new_tab;
tab->capacity *= 2;
btf->struct_ops_tab = tab;
}
tab->ops[btf->struct_ops_tab->cnt].st_ops = st_ops;
err = bpf_struct_ops_desc_init(&tab->ops[btf->struct_ops_tab->cnt], btf, log);
if (err)
return err;
btf->struct_ops_tab->cnt++;
return 0;
}
const struct bpf_struct_ops_desc *
bpf_struct_ops_find_value(struct btf *btf, u32 value_id)
{
const struct bpf_struct_ops_desc *st_ops_list;
unsigned int i;
u32 cnt;
if (!value_id)
return NULL;
if (!btf->struct_ops_tab)
return NULL;
cnt = btf->struct_ops_tab->cnt;
st_ops_list = btf->struct_ops_tab->ops;
for (i = 0; i < cnt; i++) {
if (st_ops_list[i].value_id == value_id)
return &st_ops_list[i];
}
return NULL;
}
const struct bpf_struct_ops_desc *
bpf_struct_ops_find(struct btf *btf, u32 type_id)
{
const struct bpf_struct_ops_desc *st_ops_list;
unsigned int i;
u32 cnt;
if (!type_id)
return NULL;
if (!btf->struct_ops_tab)
return NULL;
cnt = btf->struct_ops_tab->cnt;
st_ops_list = btf->struct_ops_tab->ops;
for (i = 0; i < cnt; i++) {
if (st_ops_list[i].type_id == type_id)
return &st_ops_list[i];
}
return NULL;
}
int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)
{
struct bpf_verifier_log *log;
struct btf *btf;
int err = 0;
btf = btf_get_module_btf(st_ops->owner);
if (!btf)
return check_btf_kconfigs(st_ops->owner, "struct_ops");
if (IS_ERR(btf))
return PTR_ERR(btf);
log = kzalloc_obj(*log, GFP_KERNEL | __GFP_NOWARN);
if (!log) {
err = -ENOMEM;
goto errout;
}
log->level = BPF_LOG_KERNEL;
err = btf_add_struct_ops(btf, st_ops, log);
errout:
kfree(log);
btf_put(btf);
return err;
}
EXPORT_SYMBOL_GPL(__register_bpf_struct_ops);
#endif
bool btf_param_match_suffix(const struct btf *btf,
const struct btf_param *arg,
const char *suffix)
{
int suffix_len = strlen(suffix), len;
const char *param_name;
/* In the future, this can be ported to use BTF tagging */
param_name = btf_name_by_offset(btf, arg->name_off);
if (str_is_empty(param_name))
return false;
len = strlen(param_name);
if (len <= suffix_len)
return false;
param_name += len - suffix_len;
return !strncmp(param_name, suffix, suffix_len);
}
]
|
| 514/6 |
2026/05/18 19:30 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 179546
output: 42
thoughts: 516
Thoughts: **Analyzing `btf_snprintf_show` and Hunting for `bpf_snprintf_btf`**
Okay, I've dived into `kernel/bpf/btf.c` and found `btf_type_snprintf_show`, which uses `btf_snprintf_show`. Let me take a look at the latter, specifically how it handles potential buffer overflows, which is always a crucial consideration.
Looking at the code for `btf_snprintf_show`:
```c
7125: __printf(2, 0) static void btf_snprintf_show(struct btf_show *show, const char *fmt,
7126: va_list args)
7127: {
7128: struct btf_show_snprintf *ssnprintf = (struct btf_show_snprintf *)show;
7129: int len;
7130:
7131: len = vsnprintf(show->target, ssnprintf->len_left, fmt, args);
7132:
7133: if (len < 0) {
7134: ssnprintf->len_left = 0;
7135: ssnprintf->len = len;
7136: } else if (len >= ssnprintf->len_left) {
7137: /* no space, drive on to get length we would have written */
7138: ssnprintf->len_left = 0;
7139: ssnprintf->len += len;
7140: } else {
7141: ssnprintf->len_left -= len;
7142: ssnprintf->len += len;
7143: show->target += len;
7144: }
7145: }
```
Right, the code *seems* to handle the buffer overflow case correctly. If `len` (the return from `vsnprintf`) is greater than or equal to `ssnprintf->len_left`, the code does the right thing. It sets `len_left` to 0 and stops advancing `show->target`. This will avoid any writes outside of the allocated buffer. I'd give this a preliminary "safe" rating, but I need to ensure this is used correctly and consistently.
Now, I want to see how this function is *used*, specifically I'm looking for the function `bpf_snprintf_btf`. I thought it was in this file, so I need to check.
Oh, it's not defined here. I'll search again, just to be sure, I could be mistaken.
|
| 515/6 |
2026/05/18 19:30 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf SourceTreePrefix:kernel/bpf/btf.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 516/6 |
2026/05/18 19:30 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 180767
output: 28
thoughts: 100
Thoughts: **Tracing the `bpf_snprintf_btf_proto` Function**
Okay, so I'm trying to figure out where `bpf_snprintf_btf_proto` is defined and used. My initial thought was to check in `kernel/trace/bpf_trace.c`, but let's take a look. Ah, here it is, found in `helpers.c`:
```c
2147: const struct bpf_func_proto bpf_snprintf_btf_proto __weak;
```
Now, I recall it being used within `bpf_base_func_proto`. Given that `__weak` declaration, I need to understand where this specific function prototype is actually *implemented* and fully used. Best to now search the entire tree for `bpf_snprintf_btf_proto` to get the complete picture. Let's see what the broader search reveals.
|
| 517/6 |
2026/05/18 19:30 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf_proto]
Results: map[References:<nil> TruncatedOutput:false]
|
| 518/6 |
2026/05/18 19:30 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 181116
output: 40
thoughts: 24
Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience and written in the first person:
**Investigating a Code Oddity**
Hmm, that's peculiar. I'm seeing something here that feels off. Let me see... I need to dig into this further. My initial instinct is to search for the specific string "bpf_snprintf_btf_proto." This seems like a promising starting point. I suspect it's likely related to BPF, and perhaps involves formatting something related to BTF (BPF Type Format) prototypes. I'll see what the code reveals about its usage and context. It’s likely related to either debugging, inspection, or perhaps generating metadata. I anticipate this search will lead me to the heart of the issue quickly.
|
| 519/6 |
2026/05/18 19:30 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf_proto SourceTreePrefix:kernel/bpf/]
Results: map[References:<nil> TruncatedOutput:false]
|
| 520/6 |
2026/05/18 19:30 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 181367
output: 23
thoughts: 47
Thoughts: Okay, here's my summarized thought process:
**Efficient Code Exploration: Initial Plan and Refinement**
Alright, first instinct is to locate this `snprintf_btf` function using `grep` through `codesearch-find-references`. That's usually my go-to when I'm chasing down a specific symbol. It's fast and effective for getting an overview.
But wait, given the likely specificity of the function, I think I'll skip the more targeted `codesearch-find-references`. I'll go directly for a straight-up search for `snprintf_btf` in the entire codebase. That should be quick enough, and it will hopefully give me all the locations without needing that extra layer of indirection. That'll cut down on the time it takes.
|
| 521/6 |
2026/05/18 19:30 |
tool |
codesearch-find-references |
0m
Error:
Args:
Results: map[References:<nil> TruncatedOutput:false]
|
| 522/6 |
2026/05/18 19:30 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 181628
output: 26
thoughts: 20
Thoughts: Okay, here's my attempt at summarizing that thought process, keeping in mind I'm speaking to a fellow expert:
**Investigating `bpf_snprintf_btf`**
Alright, let's start with a basic approach. I'm going to kick things off by searching the code base directly for the string `bpf_snprintf_btf`. Often, a simple string search reveals the most relevant usages, especially if it's a relatively recent addition or something specific to a certain feature. This should quickly give me an overview of where this function is being called and how it's being used within the larger context of the BTF implementation and BPF tooling. From there, I can begin to unpack its role and how it’s contributing to the overall design. Let's see what that search turns up!
|
| 523/6 |
2026/05/18 19:30 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf]
Results: map[References:<nil> TruncatedOutput:false]
|
| 524/6 |
2026/05/18 19:30 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 181870
output: 28
thoughts: 109
Thoughts: **Locating the Definition**
Alright, let's get to work. My initial thought is to simply locate the definition of `bpf_snprintf_btf_proto`. My first instinct, as always, is to just search all the files for the string itself. However, I need to keep in mind the limitations of the current toolset. If this weren't a recognized symbol, a simple string search might be my only option, but it's crucial to acknowledge that wouldn't be ideal.
Fortunately, I recall that `bpf_snprintf_btf_proto` *is* a properly defined symbol. This opens up more efficient possibilities. That means a more targeted symbol-based search is the way to go. So, let's use `codesearch-find-references` with the query `Name: "bpf_snprintf_btf_proto"` and no prefix. This should pinpoint the exact definition and any references with far more precision. This method should provide the best results given the nature of the task.
|
| 525/6 |
2026/05/18 19:31 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf_proto]
Results: map[References:<nil> TruncatedOutput:false]
|
| 526/6 |
2026/05/18 19:31 |
llm |
codeexpert |
1m
Model:
Tokens:
input: 182247
output: 47
thoughts: 25
Thoughts: Okay, here's my summary, taking on the persona you've described:
**Investigating `bpf_snprintf_btf_proto` Definition**
Alright, let's see... I need to find the definition source for `bpf_snprintf_btf_proto`. My first instinct, and the most logical place to start given my experience with these kinds of investigations, is to leverage the `codesearch-definition-source` tool. It's usually the quickest way to zero in on the precise location of a function definition within a large codebase. It'll be interesting to see what it turns up and what context it provides. Hopefully, this search will be straightforward. I'm expecting this to be a fairly common debugging pattern, nothing out of the ordinary. Let's see what the tool can dig up!
|
| 527/6 |
2026/05/18 19:32 |
tool |
codesearch-definition-source |
0m
Error:
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_snprintf_btf_proto]
Results: map[SourceCode: SourceFile:]
|
| 528/6 |
2026/05/18 19:32 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 182516
output: 27
thoughts: 87
Thoughts: Here's my thought process, as it unfolded:
**Debugging the bpf_snprintf_btf_proto Mystery**
Okay, I *think* I recall seeing something related to `bpf_snprintf_btf_proto` in the kernel source, specifically `kernel/bpf/helpers.c`. I need to confirm that. Yes, there it is! I saw it! It's right there on line 2147. But... why?
Aha! Now I'm wondering if its declaration as `__weak` is the key. Is that causing it to be handled differently, or perhaps to be excluded under certain configurations? That's a strong hypothesis, and I need to investigate. I'll need to figure out why this is being declared with that modifier.
To confirm this, I should be able to query the source to see how this symbol is used. Let me quickly use `codesearch-file-index` to search for this symbol, focusing on the `kernel/bpf/helpers.c` file itself. This should show me all uses of this particular definition. I should be able to see who uses the `bpf_snprintf_btf_proto` and hopefully determine how that is being used.
|
| 529/6 |
2026/05/18 19:32 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/bpf/helpers.c]
Results: map[Entities:[map[Kind:function Name:____bpf_copy_from_user] map[Kind:function Name:____bpf_copy_from_user_task] map[Kind:function Name:____bpf_current_task_under_cgroup] map[Kind:function Name:____bpf_dynptr_data] map[Kind:function Name:____bpf_dynptr_from_mem] map[Kind:function Name:____bpf_dynptr_read] map[Kind:function Name:____bpf_dynptr_write] map[Kind:function Name:____bpf_event_output_data] map[Kind:function Name:____bpf_get_current_ancestor_cgroup_id] map[Kind:function Name:____bpf_get_current_cgroup_id] map[Kind:function Name:____bpf_get_current_comm] map[Kind:function Name:____bpf_get_current_pid_tgid] map[Kind:function Name:____bpf_get_current_uid_gid] map[Kind:function Name:____bpf_get_ns_current_pid_tgid] map[Kind:function Name:____bpf_get_numa_node_id] map[Kind:function Name:____bpf_get_smp_processor_id] map[Kind:function Name:____bpf_jiffies64] map[Kind:function Name:____bpf_kptr_xchg] map[Kind:function Name:____bpf_ktime_get_boot_ns] map[Kind:function Name:____bpf_ktime_get_coarse_ns] map[Kind:function Name:____bpf_ktime_get_ns] map[Kind:function Name:____bpf_ktime_get_tai_ns] map[Kind:function Name:____bpf_map_delete_elem] map[Kind:function Name:____bpf_map_lookup_elem] map[Kind:function Name:____bpf_map_lookup_percpu_elem] map[Kind:function Name:____bpf_map_peek_elem] map[Kind:function Name:____bpf_map_pop_elem] map[Kind:function Name:____bpf_map_push_elem] map[Kind:function Name:____bpf_map_update_elem] map[Kind:function Name:____bpf_per_cpu_ptr] map[Kind:function Name:____bpf_snprintf] map[Kind:function Name:____bpf_spin_lock] map[Kind:function Name:____bpf_spin_unlock] map[Kind:function Name:____bpf_strncmp] map[Kind:function Name:____bpf_strtol] map[Kind:function Name:____bpf_strtoul] map[Kind:function Name:____bpf_this_cpu_ptr] map[Kind:function Name:____bpf_timer_cancel] map[Kind:function Name:____bpf_timer_init] map[Kind:function Name:____bpf_timer_set_callback] map[Kind:function Name:____bpf_timer_start] map[Kind:function Name:__bpf_async_init] map[Kind:function Name:__bpf_async_set_callback] map[Kind:function Name:__bpf_dynptr_data] map[Kind:function Name:__bpf_dynptr_data_rw] map[Kind:function Name:__bpf_dynptr_is_rdonly] map[Kind:function Name:__bpf_dynptr_read] map[Kind:function Name:__bpf_dynptr_size] map[Kind:function Name:__bpf_dynptr_write] map[Kind:function Name:__bpf_list_add] map[Kind:function Name:__bpf_list_del] map[Kind:function Name:__bpf_obj_drop_impl] map[Kind:function Name:__bpf_rbtree_add] map[Kind:function Name:__bpf_spin_lock] map[Kind:function Name:__bpf_spin_lock_irqsave] map[Kind:function Name:__bpf_spin_unlock] map[Kind:function Name:__bpf_spin_unlock_irqrestore] map[Kind:function Name:__bpf_strncasecmp] map[Kind:function Name:__bpf_strnstr] map[Kind:function Name:__bpf_strtoll] map[Kind:function Name:__bpf_strtoull] map[Kind:function Name:__bpf_trap] map[Kind:function Name:bpf_async_cancel_and_free] map[Kind:function Name:bpf_async_cb_rcu_free] map[Kind:function Name:bpf_async_cb_rcu_tasks_trace_free] map[Kind:function Name:bpf_async_irq_worker] map[Kind:function Name:bpf_async_process_op] map[Kind:function Name:bpf_async_refcount_put] map[Kind:function Name:bpf_async_schedule_op] map[Kind:function Name:bpf_async_update_prog_callback] map[Kind:function Name:bpf_base_func_proto] map[Kind:function Name:bpf_bprintf_cleanup] map[Kind:function Name:bpf_bprintf_prepare] map[Kind:function Name:bpf_cast_to_kern_ctx] map[Kind:function Name:bpf_cgroup_acquire] map[Kind:function Name:bpf_cgroup_ancestor] map[Kind:function Name:bpf_cgroup_from_id] map[Kind:function Name:bpf_cgroup_release] map[Kind:function Name:bpf_cgroup_release_dtor] map[Kind:function Name:bpf_copy_from_user] map[Kind:function Name:bpf_copy_from_user_str] map[Kind:function Name:bpf_copy_from_user_task] map[Kind:function Name:bpf_copy_from_user_task_str] map[Kind:function Name:bpf_current_task_under_cgroup] map[Kind:function Name:bpf_dynptr_adjust] map[Kind:function Name:bpf_dynptr_advance_offset] map[Kind:function Name:bpf_dynptr_check_size] map[Kind:function Name:bpf_dynptr_clone] map[Kind:function Name:bpf_dynptr_copy] map[Kind:function Name:bpf_dynptr_data] map[Kind:function Name:bpf_dynptr_file_discard] map[Kind:function Name:bpf_dynptr_from_file] map[Kind:function Name:bpf_dynptr_from_file_sleepable] map[Kind:function Name:bpf_dynptr_from_mem] map[Kind:function Name:bpf_dynptr_get_type] map[Kind:function Name:bpf_dynptr_init] map[Kind:function Name:bpf_dynptr_is_null] map[Kind:function Name:bpf_dynptr_is_rdonly] map[Kind:function Name:bpf_dynptr_memset] map[Kind:function Name:bpf_dynptr_read] map[Kind:function Name:bpf_dynptr_set_null] map[Kind:function Name:bpf_dynptr_set_rdonly] map[Kind:function Name:bpf_dynptr_set_size] map[Kind:function Name:bpf_dynptr_set_type] map[Kind:function Name:bpf_dynptr_size] map[Kind:function Name:bpf_dynptr_slice] map[Kind:function Name:bpf_dynptr_slice_rdwr] map[Kind:function Name:bpf_dynptr_write] map[Kind:function Name:bpf_event_output_data] map[Kind:function Name:bpf_file_fetch_bytes] map[Kind:function Name:bpf_get_current_ancestor_cgroup_id] map[Kind:function Name:bpf_get_current_cgroup_id] map[Kind:function Name:bpf_get_current_comm] map[Kind:function Name:bpf_get_current_pid_tgid] map[Kind:function Name:bpf_get_current_uid_gid] map[Kind:function Name:bpf_get_ns_current_pid_tgid] map[Kind:function Name:bpf_get_numa_node_id] map[Kind:function Name:bpf_get_smp_processor_id] map[Kind:function Name:bpf_iter_bits_destroy] map[Kind:function Name:bpf_iter_bits_new] map[Kind:function Name:bpf_iter_bits_next] map[Kind:function Name:bpf_jiffies64] map[Kind:function Name:bpf_key_put] map[Kind:function Name:bpf_kptr_xchg] map[Kind:function Name:bpf_ktime_get_boot_ns] map[Kind:function Name:bpf_ktime_get_coarse_ns] map[Kind:function Name:bpf_ktime_get_ns] map[Kind:function Name:bpf_ktime_get_tai_ns] map[Kind:function Name:bpf_list_back] map[Kind:function Name:bpf_list_front] map[Kind:function Name:bpf_list_head_free] map[Kind:function Name:bpf_list_pop_back] map[Kind:function Name:bpf_list_pop_front] map[Kind:function Name:bpf_list_push_back] map[Kind:function Name:bpf_list_push_back_impl] map[Kind:function Name:bpf_list_push_front] map[Kind:function Name:bpf_list_push_front_impl] map[Kind:function Name:bpf_local_irq_restore] map[Kind:function Name:bpf_local_irq_save] map[Kind:function Name:bpf_lookup_system_key] map[Kind:function Name:bpf_lookup_user_key] map[Kind:function Name:bpf_map_delete_elem] map[Kind:function Name:bpf_map_free_internal_structs] map[Kind:function Name:bpf_map_lookup_elem] map[Kind:function Name:bpf_map_lookup_percpu_elem] map[Kind:function Name:bpf_map_peek_elem] map[Kind:function Name:bpf_map_pop_elem] map[Kind:function Name:bpf_map_push_elem] map[Kind:function Name:bpf_map_update_elem] map[Kind:function Name:bpf_obj_drop] map[Kind:function Name:bpf_obj_drop_impl] map[Kind:function Name:bpf_obj_new] map[Kind:function Name:bpf_obj_new_impl] map[Kind:function Name:bpf_per_cpu_ptr] map[Kind:function Name:bpf_percpu_obj_drop] map[Kind:function Name:bpf_percpu_obj_drop_impl] map[Kind:function Name:bpf_percpu_obj_new] map[Kind:function Name:bpf_percpu_obj_new_impl] map[Kind:function Name:bpf_preempt_disable] map[Kind:function Name:bpf_preempt_enable] map[Kind:function Name:bpf_put_buffers] map[Kind:function Name:bpf_rb_root_free] map[Kind:function Name:bpf_rbtree_add] map[Kind:function Name:bpf_rbtree_add_impl] map[Kind:function Name:bpf_rbtree_first] map[Kind:function Name:bpf_rbtree_left] map[Kind:function Name:bpf_rbtree_remove] map[Kind:function Name:bpf_rbtree_right] map[Kind:function Name:bpf_rbtree_root] map[Kind:function Name:bpf_rcu_read_lock] map[Kind:function Name:bpf_rcu_read_unlock] map[Kind:function Name:bpf_rdonly_cast] map[Kind:function Name:bpf_refcount_acquire] map[Kind:function Name:bpf_refcount_acquire_impl] map[Kind:function Name:bpf_snprintf] map[Kind:function Name:bpf_spin_lock] map[Kind:function Name:bpf_spin_unlock] map[Kind:function Name:bpf_stack_walker] map[Kind:function Name:bpf_strcasecmp] map[Kind:function Name:bpf_strcasestr] map[Kind:function Name:bpf_strchr] map[Kind:function Name:bpf_strchrnul] map[Kind:function Name:bpf_strcmp] map[Kind:function Name:bpf_strcspn] map[Kind:function Name:bpf_strlen] map[Kind:function Name:bpf_strncasecmp] map[Kind:function Name:bpf_strncasestr] map[Kind:function Name:bpf_strnchr] map[Kind:function Name:bpf_strncmp] map[Kind:function Name:bpf_strnlen] map[Kind:function Name:bpf_strnstr] map[Kind:function Name:bpf_strrchr] map[Kind:function Name:bpf_strspn] map[Kind:function Name:bpf_strstr] map[Kind:function Name:bpf_strtol] map[Kind:function Name:bpf_strtoul] map[Kind:function Name:bpf_task_acquire] map[Kind:function Name:bpf_task_from_pid] map[Kind:function Name:bpf_task_from_vpid] map[Kind:function Name:bpf_task_get_cgroup1] map[Kind:function Name:bpf_task_release] map[Kind:function Name:bpf_task_release_dtor] map[Kind:function Name:bpf_task_under_cgroup] map[Kind:function Name:bpf_task_work_acquire_ctx] map[Kind:function Name:bpf_task_work_callback] map[Kind:function Name:bpf_task_work_cancel] map[Kind:function Name:bpf_task_work_cancel_and_free] map[Kind:function Name:bpf_task_work_cancel_scheduled] map[Kind:function Name:bpf_task_work_ctx_put] map[Kind:function Name:bpf_task_work_ctx_reset] map[Kind:function Name:bpf_task_work_ctx_tryget] map[Kind:function Name:bpf_task_work_destroy] map[Kind:function Name:bpf_task_work_fetch_ctx] map[Kind:function Name:bpf_task_work_irq] map[Kind:function Name:bpf_task_work_schedule] map[Kind:function Name:bpf_task_work_schedule_resume] map[Kind:function Name:bpf_task_work_schedule_signal] map[Kind:function Name:bpf_this_cpu_ptr] map[Kind:function Name:bpf_throw] map[Kind:function Name:bpf_timer_cancel] map[Kind:function Name:bpf_timer_cancel_and_free] map[Kind:function Name:bpf_timer_cancel_async] map[Kind:function Name:bpf_timer_cb] map[Kind:function Name:bpf_timer_init] map[Kind:function Name:bpf_timer_set_callback] map[Kind:function Name:bpf_timer_start] map[Kind:function Name:bpf_trace_copy_string] map[Kind:function Name:bpf_try_get_buffers] map[Kind:function Name:bpf_verify_pkcs7_signature] map[Kind:function Name:bpf_wq_cancel_and_free] map[Kind:function Name:bpf_wq_init] map[Kind:function Name:bpf_wq_set_callback] map[Kind:function Name:bpf_wq_start] map[Kind:function Name:bpf_wq_work] map[Kind:function Name:copy_map_value_locked] map[Kind:function Name:defer_timer_wq_op] map[Kind:function Name:kfunc_init] map[Kind:function Name:make_file_dynptr] map[Kind:function Name:map_key_from_value] map[Kind:function Name:swap_ulong_in_u64] map[Kind:function Name:worker_for_call_rcu] map[Kind:struct Name:bpf_async_cb] map[Kind:struct Name:bpf_async_cmd] map[Kind:struct Name:bpf_async_kern] map[Kind:struct Name:bpf_dynptr_file_impl] map[Kind:struct Name:bpf_hrtimer] map[Kind:struct Name:bpf_iter_bits] map[Kind:struct Name:bpf_iter_bits_kern] map[Kind:struct Name:bpf_task_work_ctx] map[Kind:struct Name:bpf_task_work_kern] map[Kind:struct Name:bpf_throw_ctx] map[Kind:struct Name:bpf_work] map[Kind:global_variable Name:__UNIQUE_ID_addressable_bpf_base_func_proto_789] map[Kind:global_variable Name:__UNIQUE_ID_addressable_kfunc_init_832] map[Kind:global_variable Name:bpf_copy_from_user_proto] map[Kind:global_variable Name:bpf_copy_from_user_task_proto] map[Kind:global_variable Name:bpf_current_task_under_cgroup_proto] map[Kind:global_variable Name:bpf_dynptr_data_proto] map[Kind:global_variable Name:bpf_dynptr_from_mem_proto] map[Kind:global_variable Name:bpf_dynptr_read_proto] map[Kind:global_variable Name:bpf_dynptr_write_proto] map[Kind:global_variable Name:bpf_event_output_data_proto] map[Kind:global_variable Name:bpf_get_current_ancestor_cgroup_id_proto] map[Kind:global_variable Name:bpf_get_current_cgroup_id_proto] map[Kind:global_variable Name:bpf_get_current_comm_proto] map[Kind:global_variable Name:bpf_get_current_pid_tgid_proto] map[Kind:global_variable Name:bpf_get_current_uid_gid_proto] map[Kind:global_variable Name:bpf_get_ns_current_pid_tgid_proto] map[Kind:global_variable Name:bpf_get_numa_node_id_proto] map[Kind:global_variable Name:bpf_get_prandom_u32_proto] map[Kind:global_variable Name:bpf_get_raw_smp_processor_id_proto] map[Kind:global_variable Name:bpf_get_smp_processor_id_proto] map[Kind:global_variable Name:bpf_jiffies64_proto] map[Kind:global_variable Name:bpf_kptr_xchg_proto] map[Kind:global_variable Name:bpf_ktime_get_boot_ns_proto] map[Kind:global_variable Name:bpf_ktime_get_coarse_ns_proto] map[Kind:global_variable Name:bpf_ktime_get_ns_proto] map[Kind:global_variable Name:bpf_ktime_get_tai_ns_proto] map[Kind:global_variable Name:bpf_map_delete_elem_proto] map[Kind:global_variable Name:bpf_map_lookup_elem_proto] map[Kind:global_variable Name:bpf_map_lookup_percpu_elem_proto] map[Kind:global_variable Name:bpf_map_peek_elem_proto] map[Kind:global_variable Name:bpf_map_pop_elem_proto] map[Kind:global_variable Name:bpf_map_push_elem_proto] map[Kind:global_variable Name:bpf_map_update_elem_proto] map[Kind:global_variable Name:bpf_per_cpu_ptr_proto] map[Kind:global_variable Name:bpf_snprintf_proto] map[Kind:global_variable Name:bpf_spin_lock_proto] map[Kind:global_variable Name:bpf_spin_unlock_proto] map[Kind:global_variable Name:bpf_strncmp_proto] map[Kind:global_variable Name:bpf_strtol_proto] map[Kind:global_variable Name:bpf_strtoul_proto] map[Kind:global_variable Name:bpf_this_cpu_ptr_proto] map[Kind:global_variable Name:bpf_timer_cancel_proto] map[Kind:global_variable Name:bpf_timer_init_proto] map[Kind:global_variable Name:bpf_timer_set_callback_proto] map[Kind:global_variable Name:bpf_timer_start_proto] map[Kind:global_variable Name:common_btf_ids] map[Kind:global_variable Name:common_kfunc_set] map[Kind:global_variable Name:generic_btf_ids] map[Kind:global_variable Name:generic_kfunc_set] map[Kind:enum Name:bpf_async_op] map[Kind:enum Name:bpf_async_type] map[Kind:enum Name:bpf_task_work_state] map[Kind:typedef Name:bpf_task_work_callback_t] map[Kind:typedef Name:btf_bpf_copy_from_user] map[Kind:typedef Name:btf_bpf_copy_from_user_task] map[Kind:typedef Name:btf_bpf_current_task_under_cgroup] map[Kind:typedef Name:btf_bpf_dynptr_data] map[Kind:typedef Name:btf_bpf_dynptr_from_mem] map[Kind:typedef Name:btf_bpf_dynptr_read] map[Kind:typedef Name:btf_bpf_dynptr_write] map[Kind:typedef Name:btf_bpf_event_output_data] map[Kind:typedef Name:btf_bpf_get_current_ancestor_cgroup_id] map[Kind:typedef Name:btf_bpf_get_current_cgroup_id] map[Kind:typedef Name:btf_bpf_get_current_comm] map[Kind:typedef Name:btf_bpf_get_current_pid_tgid] map[Kind:typedef Name:btf_bpf_get_current_uid_gid] map[Kind:typedef Name:btf_bpf_get_ns_current_pid_tgid] map[Kind:typedef Name:btf_bpf_get_numa_node_id] map[Kind:typedef Name:btf_bpf_get_smp_processor_id] map[Kind:typedef Name:btf_bpf_jiffies64] map[Kind:typedef Name:btf_bpf_kptr_xchg] map[Kind:typedef Name:btf_bpf_ktime_get_boot_ns] map[Kind:typedef Name:btf_bpf_ktime_get_coarse_ns] map[Kind:typedef Name:btf_bpf_ktime_get_ns] map[Kind:typedef Name:btf_bpf_ktime_get_tai_ns] map[Kind:typedef Name:btf_bpf_map_delete_elem] map[Kind:typedef Name:btf_bpf_map_lookup_elem] map[Kind:typedef Name:btf_bpf_map_lookup_percpu_elem] map[Kind:typedef Name:btf_bpf_map_peek_elem] map[Kind:typedef Name:btf_bpf_map_pop_elem] map[Kind:typedef Name:btf_bpf_map_push_elem] map[Kind:typedef Name:btf_bpf_map_update_elem] map[Kind:typedef Name:btf_bpf_per_cpu_ptr] map[Kind:typedef Name:btf_bpf_snprintf] map[Kind:typedef Name:btf_bpf_spin_lock] map[Kind:typedef Name:btf_bpf_spin_unlock] map[Kind:typedef Name:btf_bpf_strncmp] map[Kind:typedef Name:btf_bpf_strtol] map[Kind:typedef Name:btf_bpf_strtoul] map[Kind:typedef Name:btf_bpf_this_cpu_ptr] map[Kind:typedef Name:btf_bpf_timer_cancel] map[Kind:typedef Name:btf_bpf_timer_init] map[Kind:typedef Name:btf_bpf_timer_set_callback] map[Kind:typedef Name:btf_bpf_timer_start]]]
|
| 530/6 |
2026/05/18 19:32 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 189929
output: 34
thoughts: 316
Thoughts: **Investigating BPF `snprintf` Usage**
Okay, so the file index isn't showing `bpf_snprintf_btf_proto`. That's odd. Wait, I see `bpf_snprintf_proto`, but not the BTF version. But, I *know* I saw it in the file content, right there:
```c
2147: const struct bpf_func_proto bpf_snprintf_btf_proto __weak;
```
Right, it's probably not indexed because it's just a declaration of a weak symbol. That makes sense.
Anyway, I've confirmed that `snprintf` is only used in `bpf_bprintf_prepare` in `kernel/bpf/helpers.c`. Good.
Now, let's double-check if there are any other formatting functions that might have similar potential issues. Let's look at `bstr_printf`. I see it's used in `____bpf_snprintf`. `bstr_printf` returns the number of characters that *would* have been written, and in `____bpf_snprintf` I see:
```c
1084: err = bstr_printf(str, str_size, fmt, data.bin_args);
...
1088: return err + 1;
```
Okay, this looks safe because it doesn't use `err` to advance a pointer and write more data. Good.
Are there any other places where `bstr_printf` is used in the `kernel/bpf/` directory? Let's search for it.
|
| 531/6 |
2026/05/18 19:32 |
tool |
codesearch-find-references |
0m
Args: map[Name:bstr_printf SourceTreePrefix:kernel/bpf/]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:____bpf_snprintf SourceFile:kernel/bpf/helpers.c SourceLine:1084] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_stream_vprintk SourceFile:kernel/bpf/stream.c SourceLine:240]] TruncatedOutput:false]
|
| 532/6 |
2026/05/18 19:32 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 190763
output: 44
thoughts: 24
Thoughts: Here's my take on the thought process, as if I were the one thinking it:
**Diving into `bpf_stream_vprintk`**
Alright, time to dig into this. I need to understand how this `bpf_stream_vprintk` function is behaving, specifically, I want to see what's happening within `kernel/bpf/stream.c`. Let's pull up the source and get right to it. I want to see how this logging mechanism is implemented. Maybe I can spot some interesting performance characteristics, security considerations, or interaction points with other BPF components. Let's see how they're handling formatting the output, any potential buffer management issues, and how it interacts with the stream data structures. I'm hoping to get a clear picture of its role in the bigger BPF picture.
|
| 533/6 |
2026/05/18 19:32 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/stream.c IncludeLines:true Name:bpf_stream_vprintk]
Results: map[SourceCode: 211: /*
212: * Avoid using enum bpf_stream_id so that kfunc users don't have to pull in the
213: * enum in headers.
214: */
215: __bpf_kfunc int bpf_stream_vprintk(int stream_id, const char *fmt__str, const void *args,
216: u32 len__sz, struct bpf_prog_aux *aux)
217: {
218: struct bpf_bprintf_data data = {
219: .get_bin_args = true,
220: .get_buf = true,
221: };
222: u32 fmt_size = strlen(fmt__str) + 1;
223: struct bpf_stream *stream;
224: u32 data_len = len__sz;
225: int ret, num_args;
226:
227: stream = bpf_stream_get(stream_id, aux);
228: if (!stream)
229: return -ENOENT;
230:
231: if (data_len & 7 || data_len > MAX_BPRINTF_VARARGS * 8 ||
232: (data_len && !args))
233: return -EINVAL;
234: num_args = data_len / 8;
235:
236: ret = bpf_bprintf_prepare(fmt__str, fmt_size, args, num_args, &data);
237: if (ret < 0)
238: return ret;
239:
240: ret = bstr_printf(data.buf, MAX_BPRINTF_BUF, fmt__str, data.bin_args);
241: /* Exclude NULL byte during push. */
242: ret = bpf_stream_push_str(stream, data.buf, ret);
243: bpf_bprintf_cleanup(&data);
244:
245: return ret;
246: }
SourceFile:kernel/bpf/stream.c]
|
| 534/6 |
2026/05/18 19:32 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 191627
output: 45
thoughts: 201
Thoughts: **Analyzing a Potential Vulnerability in BPF Stream Printing**
Okay, let's take a look at this `bpf_stream_vprintk` function. I see it's using `bstr_printf` to format a string into a buffer named `data.buf`. The crucial thing here is that `bstr_printf` returns the *projected* number of characters that would be written, not necessarily the actual number that were written, due to buffer limitations. Specifically, if `ret` (the return value of `bstr_printf`) is greater than or equal to `MAX_BPRINTF_BUF`, then the string in `data.buf` is going to be truncated.
Now, here's where my alarm bells start ringing. The code then calls `bpf_stream_push_str`, passing `ret`, which could be *larger* than the actual size of the string in `data.buf` if truncation occurred. If that's the case, then `bpf_stream_push_str` might try to read past the end of the buffer, leading to a possible out-of-bounds read. This is a classic potential vulnerability, especially in BPF contexts where we're very careful about memory access.
I think it's time to check out how `bpf_stream_push_str` is implemented. I need to understand exactly how it uses the `ret` value and the buffer `data.buf` to see if my concern is valid. This warrants a closer look, especially considering the potential for exploitation in the kernel.
|
| 535/6 |
2026/05/18 19:33 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/stream.c IncludeLines:true Name:bpf_stream_push_str]
Results: map[SourceCode: 78: static int bpf_stream_push_str(struct bpf_stream *stream, const char *str, int len)
79: {
80: int ret = bpf_stream_consume_capacity(stream, len);
81:
82: return ret ?: __bpf_stream_push_str(&stream->log, str, len);
83: }
SourceFile:kernel/bpf/stream.c]
|
| 536/6 |
2026/05/18 19:33 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 192338
output: 46
thoughts: 12
Thoughts: Okay, here's my attempt at summarizing the thought process, as requested, targeting an expert audience and using the first person:
**Decoding that `__bpf_stream_push_str`**
Alright, let's break this down. Seeing `__bpf_stream_push_str` immediately flags a few things. First, this is undoubtedly BPF, specifically dealing with stream-based data manipulation. The function name itself screams "pushing a string" onto some kind of stream structure, likely within the BPF context. My mind immediately jumps to potential use cases: are we talking about a network stream, a file stream, or something entirely internal to a particular kernel module?
The presence of this function tells me someone is writing a BPF program designed to either modify or collect string data as part of its operation. I'm already mentally forming a few hypotheses:
* **Network Packet Inspection:** Could be processing headers or payloads, likely sniffing or filtering.
* **System Call Tracing:** Possibly capturing or injecting string arguments to system calls.
* **Performance Monitoring:** Maybe instrumenting the string representation of certain kernel or user space variables.
* **Security Auditing:** Conceivably inspecting user input or system events for malicious patterns.
I'm thinking about the potential stream structure. Is it a linked list, a circular buffer, or something more specialized for BPF's memory constraints? Does this stream function handle memory allocation and deallocation itself, or does the calling BPF program manage that? These are all key details that would influence the program's efficiency and reliability. The specific implementation of `__bpf_stream_push_str` is critical for understanding its resource implications.
I want to know the context of this function call. Where is it being called from? What's the surrounding code? Is it part of a larger BPF program, or is it a standalone utility? Access to surrounding code is essential to get a full understanding. Without this context, I'm left to make assumptions based on common BPF programming practices.
I would need to see the function's signature and the overall program structure to properly evaluate its functionality, potential bottlenecks, and any security considerations.
|
| 537/6 |
2026/05/18 19:33 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/stream.c IncludeLines:true Name:__bpf_stream_push_str]
Results: map[SourceCode: 42: static int __bpf_stream_push_str(struct llist_head *log, const char *str, int len)
43: {
44: struct bpf_stream_elem *elem = NULL;
45:
46: /*
47: * Allocate a bpf_prog_stream_elem and push it to the bpf_prog_stream
48: * log, elements will be popped at once and reversed to print the log.
49: */
50: elem = bpf_stream_elem_alloc(len);
51: if (!elem)
52: return -ENOMEM;
53:
54: memcpy(elem->str, str, len);
55: llist_add(&elem->node, log);
56:
57: return 0;
58: }
SourceFile:kernel/bpf/stream.c]
|
| 538/6 |
2026/05/18 19:33 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 193120
output: 44
thoughts: 306
Thoughts: **Analyzing a Potential Buffer Overflow in `bpf_stream_vprintk`**
Okay, so I'm looking at this section in `bpf_stream_vprintk`, specifically these lines:
```c
240: ret = bstr_printf(data.buf, MAX_BPRINTF_BUF, fmt__str, data.bin_args);
241: /* Exclude NULL byte during push. */
242: ret = bpf_stream_push_str(stream, data.buf, ret);
```
My immediate concern is that if `bstr_printf` returns a value `ret` that's greater than or equal to `MAX_BPRINTF_BUF`, then the subsequent `bpf_stream_push_str` will attempt to copy data from `data.buf` using `memcpy(elem->str, str, len)`, but `len` would be greater than the allocated size of `data.buf`. That clearly screams buffer overflow. `data.buf` is sized at `MAX_BPRINTF_BUF`, so a large `ret` could overrun the buffer.
Now, where is `MAX_BPRINTF_BUF` defined? I need to find the source. My instinct is that it's likely defined in `include/linux/bpf.h` or something similar, but I should probably confirm that in `kernel/bpf/helpers.c`, to be sure it's not some weird compile-time artifact. Wait, `bpf_bprintf_prepare` uses `buffers->buf`:
```c
845: data->buf = buffers->buf;
```
Interesting. So, `data->buf` is actually coming from `buffers->buf`. Okay, so `struct bpf_bprintf_buffers` is defined in `include/linux/bpf.h`.
I'm going to dig into that `struct bpf_bprintf_buffers` definition to get the whole picture.
|
| 539/6 |
2026/05/18 19:33 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_buffers]
Results: map[SourceCode:3877: struct bpf_bprintf_buffers {
3878: char bin_args[MAX_BPRINTF_BIN_ARGS];
3879: char buf[MAX_BPRINTF_BUF];
3880: };
SourceFile:include/linux/bpf.h]
|
| 540/6 |
2026/05/18 19:33 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 193996
output: 25
thoughts: 10
Thoughts: Okay, here's my attempt at summarizing the thought process, tailored for an expert audience, and written in the first person:
**Concerning `MAX_BPRINTF_BUF` and Buffer Management**
Right, so I'm thinking about `MAX_BPRINTF_BUF`. It's that critical constant, isn't it? The one that sets the upper bound on the size of the buffer used in, presumably, `bprintf` or a similar buffered printing function. And it's making me consider a few key aspects, given its implications.
Firstly, the *trade-offs*. A larger buffer, theoretically, allows for more complex formatting and larger output strings before potential truncation or overflow. It buys us flexibility. But, on the other hand, a significantly larger `MAX_BPRINTF_BUF` directly impacts memory usage. Every instance of this buffer, potentially, consumes that much RAM, even if it's mostly unused. In memory-constrained environments, or where there are many concurrent uses of the function, this is a very real concern. And, of course, memory allocation at runtime might be avoided.
Then there's the question of how this constant is *derived*. Is it a purely arbitrary number? Is it based on empirical analysis of typical output string sizes? Or, perhaps, is it calculated according to available memory? Is it configurable at compile time and runtime? Can it be increased, and should it be? Because a static value can either be too restrictive, causing premature truncation, or too generous, wasting memory. There’s a certain elegance in an automatic sizing system if feasible.
I'm also considering the *error handling*. What happens if the output exceeds `MAX_BPRINTF_BUF`? Does the function return an error code? Truncate silently? Assert? Log a warning? The choice here is crucial. Silent truncation can lead to subtle bugs and missed debugging opportunities, but an overly aggressive error handling scheme might be too disruptive. And what if memory allocation fails? Is there a fallback mechanism or an assertion?
Finally, I'm thinking about the *impact on performance*. Buffer management can become a bottleneck, especially if there's a lot of copying or reallocation involved. Could an alternative, streaming approach to printing offer advantages? Would pre-allocating the buffer, or using a circular buffer, be more efficient in certain scenarios? The mechanics of the buffer are as critical as its size.
So, `MAX_BPRINTF_BUF` is more than just a number; it's a critical design decision with wide-ranging consequences for memory usage, error handling, and performance. We need to be confident that it's sized appropriately for the expected workloads and that the underlying buffer management strategy is efficient and robust.
|
| 541/6 |
2026/05/18 19:33 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:MAX_BPRINTF_BUF]
Results: map[References:<nil> TruncatedOutput:false]
|
| 542/6 |
2026/05/18 19:33 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 194625
output: 23
thoughts: 17
Thoughts: Here's my thought process, summarized as you requested:
**Investigating `MAX_BPRINTF_BUF`**
Okay, so I need to understand what `MAX_BPRINTF_BUF` refers to. My first instinct is to dig directly into its definition. I'll search for it, probably looking for a header file or documentation related to the specific codebase I'm working with. This will likely involve a quick `grep` or `find` if I'm navigating a large project, or a direct code search if it's a smaller, more familiar system. I'm expecting this to be a constant or macro that defines the maximum buffer size used by a `bprintf`-like function. The context will probably give me more clues. I'll be looking for things like the units it's defined in (bytes, words, etc.) and what implications its value has on the overall system's memory management and potential buffer overflow vulnerabilities. Knowing the context, I'll then be able to assess it, and relate it to the project’s objectives.
|
| 543/6 |
2026/05/18 19:33 |
tool |
read-file |
0m
Args: map[File:include/linux/bpf.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0-only */
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*/
#ifndef _LINUX_BPF_H
#define _LINUX_BPF_H 1
#include <uapi/linux/bpf.h>
#include <uapi/linux/filter.h>
#include <crypto/sha2.h>
#include <linux/workqueue.h>
#include <linux/file.h>
#include <linux/percpu.h>
#include <linux/err.h>
#include <linux/rbtree_latch.h>
#include <linux/numa.h>
#include <linux/mm_types.h>
#include <linux/wait.h>
#include <linux/refcount.h>
#include <linux/mutex.h>
#include <linux/module.h>
#include <linux/kallsyms.h>
#include <linux/capability.h>
#include <linux/sched/mm.h>
#include <linux/slab.h>
#include <linux/percpu-refcount.h>
#include <linux/stddef.h>
#include <linux/bpfptr.h>
#include <linux/btf.h>
#include <linux/rcupdate_trace.h>
#include <linux/static_call.h>
#include <linux/memcontrol.h>
#include <linux/cfi.h>
#include <asm/rqspinlock.h>
struct bpf_verifier_env;
struct bpf_verifier_log;
struct perf_event;
struct bpf_prog;
struct bpf_prog_aux;
struct bpf_map;
struct bpf_arena;
struct sock;
struct seq_file;
struct btf;
struct btf_type;
struct exception_table_entry;
struct seq_operations;
struct bpf_iter_aux_info;
struct bpf_local_storage;
struct bpf_local_storage_map;
struct kobject;
struct mem_cgroup;
struct module;
struct bpf_func_state;
struct ftrace_ops;
struct cgroup;
struct bpf_token;
struct user_namespace;
struct super_block;
struct inode;
extern struct idr btf_idr;
extern spinlock_t btf_idr_lock;
extern struct kobject *btf_kobj;
extern struct bpf_mem_alloc bpf_global_ma, bpf_global_percpu_ma;
extern bool bpf_global_ma_set;
typedef u64 (*bpf_callback_t)(u64, u64, u64, u64, u64);
typedef int (*bpf_iter_init_seq_priv_t)(void *private_data,
struct bpf_iter_aux_info *aux);
typedef void (*bpf_iter_fini_seq_priv_t)(void *private_data);
typedef unsigned int (*bpf_func_t)(const void *,
const struct bpf_insn *);
struct bpf_iter_seq_info {
const struct seq_operations *seq_ops;
bpf_iter_init_seq_priv_t init_seq_private;
bpf_iter_fini_seq_priv_t fini_seq_private;
u32 seq_priv_size;
};
/* map is generic key/value storage optionally accessible by eBPF programs */
struct bpf_map_ops {
/* funcs callable from userspace (via syscall) */
int (*map_alloc_check)(union bpf_attr *attr);
struct bpf_map *(*map_alloc)(union bpf_attr *attr);
void (*map_release)(struct bpf_map *map, struct file *map_file);
void (*map_free)(struct bpf_map *map);
int (*map_get_next_key)(struct bpf_map *map, void *key, void *next_key);
void (*map_release_uref)(struct bpf_map *map);
void *(*map_lookup_elem_sys_only)(struct bpf_map *map, void *key);
int (*map_lookup_batch)(struct bpf_map *map, const union bpf_attr *attr,
union bpf_attr __user *uattr);
int (*map_lookup_and_delete_elem)(struct bpf_map *map, void *key,
void *value, u64 flags);
int (*map_lookup_and_delete_batch)(struct bpf_map *map,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
int (*map_update_batch)(struct bpf_map *map, struct file *map_file,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
int (*map_delete_batch)(struct bpf_map *map, const union bpf_attr *attr,
union bpf_attr __user *uattr);
/* funcs callable from userspace and from eBPF programs */
void *(*map_lookup_elem)(struct bpf_map *map, void *key);
long (*map_update_elem)(struct bpf_map *map, void *key, void *value, u64 flags);
long (*map_delete_elem)(struct bpf_map *map, void *key);
long (*map_push_elem)(struct bpf_map *map, void *value, u64 flags);
long (*map_pop_elem)(struct bpf_map *map, void *value);
long (*map_peek_elem)(struct bpf_map *map, void *value);
void *(*map_lookup_percpu_elem)(struct bpf_map *map, void *key, u32 cpu);
int (*map_get_hash)(struct bpf_map *map, u32 hash_buf_size, void *hash_buf);
/* funcs called by prog_array and perf_event_array map */
void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file,
int fd);
/* If need_defer is true, the implementation should guarantee that
* the to-be-put element is still alive before the bpf program, which
* may manipulate it, exists.
*/
void (*map_fd_put_ptr)(struct bpf_map *map, void *ptr, bool need_defer);
int (*map_gen_lookup)(struct bpf_map *map, struct bpf_insn *insn_buf);
u32 (*map_fd_sys_lookup_elem)(void *ptr);
void (*map_seq_show_elem)(struct bpf_map *map, void *key,
struct seq_file *m);
int (*map_check_btf)(struct bpf_map *map,
const struct btf *btf,
const struct btf_type *key_type,
const struct btf_type *value_type);
/* Prog poke tracking helpers. */
int (*map_poke_track)(struct bpf_map *map, struct bpf_prog_aux *aux);
void (*map_poke_untrack)(struct bpf_map *map, struct bpf_prog_aux *aux);
void (*map_poke_run)(struct bpf_map *map, u32 key, struct bpf_prog *old,
struct bpf_prog *new);
/* Direct value access helpers. */
int (*map_direct_value_addr)(const struct bpf_map *map,
u64 *imm, u32 off);
int (*map_direct_value_meta)(const struct bpf_map *map,
u64 imm, u32 *off);
int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma);
__poll_t (*map_poll)(struct bpf_map *map, struct file *filp,
struct poll_table_struct *pts);
unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr,
unsigned long len, unsigned long pgoff,
unsigned long flags);
/* Functions called by bpf_local_storage maps */
int (*map_local_storage_charge)(struct bpf_local_storage_map *smap,
void *owner, u32 size);
void (*map_local_storage_uncharge)(struct bpf_local_storage_map *smap,
void *owner, u32 size);
struct bpf_local_storage __rcu ** (*map_owner_storage_ptr)(void *owner);
/* Misc helpers.*/
long (*map_redirect)(struct bpf_map *map, u64 key, u64 flags);
/* map_meta_equal must be implemented for maps that can be
* used as an inner map. It is a runtime check to ensure
* an inner map can be inserted to an outer map.
*
* Some properties of the inner map has been used during the
* verification time. When inserting an inner map at the runtime,
* map_meta_equal has to ensure the inserting map has the same
* properties that the verifier has used earlier.
*/
bool (*map_meta_equal)(const struct bpf_map *meta0,
const struct bpf_map *meta1);
int (*map_set_for_each_callback_args)(struct bpf_verifier_env *env,
struct bpf_func_state *caller,
struct bpf_func_state *callee);
long (*map_for_each_callback)(struct bpf_map *map,
bpf_callback_t callback_fn,
void *callback_ctx, u64 flags);
u64 (*map_mem_usage)(const struct bpf_map *map);
/* BTF id of struct allocated by map_alloc */
int *map_btf_id;
/* bpf_iter info used to open a seq_file */
const struct bpf_iter_seq_info *iter_seq_info;
};
enum {
/* Support at most 11 fields in a BTF type */
BTF_FIELDS_MAX = 11,
};
enum btf_field_type {
BPF_SPIN_LOCK = (1 << 0),
BPF_TIMER = (1 << 1),
BPF_KPTR_UNREF = (1 << 2),
BPF_KPTR_REF = (1 << 3),
BPF_KPTR_PERCPU = (1 << 4),
BPF_KPTR = BPF_KPTR_UNREF | BPF_KPTR_REF | BPF_KPTR_PERCPU,
BPF_LIST_HEAD = (1 << 5),
BPF_LIST_NODE = (1 << 6),
BPF_RB_ROOT = (1 << 7),
BPF_RB_NODE = (1 << 8),
BPF_GRAPH_NODE = BPF_RB_NODE | BPF_LIST_NODE,
BPF_GRAPH_ROOT = BPF_RB_ROOT | BPF_LIST_HEAD,
BPF_REFCOUNT = (1 << 9),
BPF_WORKQUEUE = (1 << 10),
BPF_UPTR = (1 << 11),
BPF_RES_SPIN_LOCK = (1 << 12),
BPF_TASK_WORK = (1 << 13),
};
enum bpf_cgroup_storage_type {
BPF_CGROUP_STORAGE_SHARED,
BPF_CGROUP_STORAGE_PERCPU,
__BPF_CGROUP_STORAGE_MAX
#define MAX_BPF_CGROUP_STORAGE_TYPE __BPF_CGROUP_STORAGE_MAX
};
#ifdef CONFIG_CGROUP_BPF
# define for_each_cgroup_storage_type(stype) \
for (stype = 0; stype < MAX_BPF_CGROUP_STORAGE_TYPE; stype++)
#else
# define for_each_cgroup_storage_type(stype) for (; false; )
#endif /* CONFIG_CGROUP_BPF */
typedef void (*btf_dtor_kfunc_t)(void *);
struct btf_field_kptr {
struct btf *btf;
struct module *module;
/* dtor used if btf_is_kernel(btf), otherwise the type is
* program-allocated, dtor is NULL, and __bpf_obj_drop_impl is used
*/
btf_dtor_kfunc_t dtor;
u32 btf_id;
};
struct btf_field_graph_root {
struct btf *btf;
u32 value_btf_id;
u32 node_offset;
struct btf_record *value_rec;
};
struct btf_field {
u32 offset;
u32 size;
enum btf_field_type type;
union {
struct btf_field_kptr kptr;
struct btf_field_graph_root graph_root;
};
};
struct btf_record {
u32 cnt;
u32 field_mask;
int spin_lock_off;
int res_spin_lock_off;
int timer_off;
int wq_off;
int refcount_off;
int task_work_off;
struct btf_field fields[];
};
/* Non-opaque version of bpf_rb_node in uapi/linux/bpf.h */
struct bpf_rb_node_kern {
struct rb_node rb_node;
void *owner;
} __attribute__((aligned(8)));
/* Non-opaque version of bpf_list_node in uapi/linux/bpf.h */
struct bpf_list_node_kern {
struct list_head list_head;
void *owner;
} __attribute__((aligned(8)));
/* 'Ownership' of program-containing map is claimed by the first program
* that is going to use this map or by the first program which FD is
* stored in the map to make sure that all callers and callees have the
* same prog type, JITed flag and xdp_has_frags flag.
*/
struct bpf_map_owner {
enum bpf_prog_type type;
bool jited;
bool xdp_has_frags;
bool sleepable;
u64 storage_cookie[MAX_BPF_CGROUP_STORAGE_TYPE];
const struct btf_type *attach_func_proto;
enum bpf_attach_type expected_attach_type;
};
struct bpf_map {
u8 sha[SHA256_DIGEST_SIZE];
const struct bpf_map_ops *ops;
struct bpf_map *inner_map_meta;
#ifdef CONFIG_SECURITY
void *security;
#endif
enum bpf_map_type map_type;
u32 key_size;
u32 value_size;
u32 max_entries;
u64 map_extra; /* any per-map-type extra fields */
u32 map_flags;
u32 id;
struct btf_record *record;
int numa_node;
u32 btf_key_type_id;
u32 btf_value_type_id;
u32 btf_vmlinux_value_type_id;
struct btf *btf;
#ifdef CONFIG_MEMCG
struct obj_cgroup *objcg;
#endif
char name[BPF_OBJ_NAME_LEN];
struct mutex freeze_mutex;
atomic64_t refcnt;
atomic64_t usercnt;
/* rcu is used before freeing and work is only used during freeing */
union {
struct work_struct work;
struct rcu_head rcu;
};
atomic64_t writecnt;
spinlock_t owner_lock;
struct bpf_map_owner *owner;
bool bypass_spec_v1;
bool frozen; /* write-once; write-protected by freeze_mutex */
bool free_after_mult_rcu_gp;
bool free_after_rcu_gp;
atomic64_t sleepable_refcnt;
s64 __percpu *elem_count;
u64 cookie; /* write-once */
char *excl_prog_sha;
};
static inline const char *btf_field_type_name(enum btf_field_type type)
{
switch (type) {
case BPF_SPIN_LOCK:
return "bpf_spin_lock";
case BPF_RES_SPIN_LOCK:
return "bpf_res_spin_lock";
case BPF_TIMER:
return "bpf_timer";
case BPF_WORKQUEUE:
return "bpf_wq";
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
return "kptr";
case BPF_KPTR_PERCPU:
return "percpu_kptr";
case BPF_UPTR:
return "uptr";
case BPF_LIST_HEAD:
return "bpf_list_head";
case BPF_LIST_NODE:
return "bpf_list_node";
case BPF_RB_ROOT:
return "bpf_rb_root";
case BPF_RB_NODE:
return "bpf_rb_node";
case BPF_REFCOUNT:
return "bpf_refcount";
case BPF_TASK_WORK:
return "bpf_task_work";
default:
WARN_ON_ONCE(1);
return "unknown";
}
}
#if IS_ENABLED(CONFIG_DEBUG_KERNEL)
#define BPF_WARN_ONCE(cond, format...) WARN_ONCE(cond, format)
#else
#define BPF_WARN_ONCE(cond, format...) BUILD_BUG_ON_INVALID(cond)
#endif
static inline u32 btf_field_type_size(enum btf_field_type type)
{
switch (type) {
case BPF_SPIN_LOCK:
return sizeof(struct bpf_spin_lock);
case BPF_RES_SPIN_LOCK:
return sizeof(struct bpf_res_spin_lock);
case BPF_TIMER:
return sizeof(struct bpf_timer);
case BPF_WORKQUEUE:
return sizeof(struct bpf_wq);
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
case BPF_KPTR_PERCPU:
case BPF_UPTR:
return sizeof(u64);
case BPF_LIST_HEAD:
return sizeof(struct bpf_list_head);
case BPF_LIST_NODE:
return sizeof(struct bpf_list_node);
case BPF_RB_ROOT:
return sizeof(struct bpf_rb_root);
case BPF_RB_NODE:
return sizeof(struct bpf_rb_node);
case BPF_REFCOUNT:
return sizeof(struct bpf_refcount);
case BPF_TASK_WORK:
return sizeof(struct bpf_task_work);
default:
WARN_ON_ONCE(1);
return 0;
}
}
static inline u32 btf_field_type_align(enum btf_field_type type)
{
switch (type) {
case BPF_SPIN_LOCK:
return __alignof__(struct bpf_spin_lock);
case BPF_RES_SPIN_LOCK:
return __alignof__(struct bpf_res_spin_lock);
case BPF_TIMER:
return __alignof__(struct bpf_timer);
case BPF_WORKQUEUE:
return __alignof__(struct bpf_wq);
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
case BPF_KPTR_PERCPU:
case BPF_UPTR:
return __alignof__(u64);
case BPF_LIST_HEAD:
return __alignof__(struct bpf_list_head);
case BPF_LIST_NODE:
return __alignof__(struct bpf_list_node);
case BPF_RB_ROOT:
return __alignof__(struct bpf_rb_root);
case BPF_RB_NODE:
return __alignof__(struct bpf_rb_node);
case BPF_REFCOUNT:
return __alignof__(struct bpf_refcount);
case BPF_TASK_WORK:
return __alignof__(struct bpf_task_work);
default:
WARN_ON_ONCE(1);
return 0;
}
}
static inline void bpf_obj_init_field(const struct btf_field *field, void *addr)
{
memset(addr, 0, field->size);
switch (field->type) {
case BPF_REFCOUNT:
refcount_set((refcount_t *)addr, 1);
break;
case BPF_RB_NODE:
RB_CLEAR_NODE((struct rb_node *)addr);
break;
case BPF_LIST_HEAD:
case BPF_LIST_NODE:
INIT_LIST_HEAD((struct list_head *)addr);
break;
case BPF_RB_ROOT:
/* RB_ROOT_CACHED 0-inits, no need to do anything after memset */
case BPF_SPIN_LOCK:
case BPF_RES_SPIN_LOCK:
case BPF_TIMER:
case BPF_WORKQUEUE:
case BPF_KPTR_UNREF:
case BPF_KPTR_REF:
case BPF_KPTR_PERCPU:
case BPF_UPTR:
case BPF_TASK_WORK:
break;
default:
WARN_ON_ONCE(1);
return;
}
}
static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_field_type type)
{
if (IS_ERR_OR_NULL(rec))
return false;
return rec->field_mask & type;
}
static inline void bpf_obj_init(const struct btf_record *rec, void *obj)
{
int i;
if (IS_ERR_OR_NULL(rec))
return;
for (i = 0; i < rec->cnt; i++)
bpf_obj_init_field(&rec->fields[i], obj + rec->fields[i].offset);
}
/* 'dst' must be a temporary buffer and should not point to memory that is being
* used in parallel by a bpf program or bpf syscall, otherwise the access from
* the bpf program or bpf syscall may be corrupted by the reinitialization,
* leading to weird problems. Even 'dst' is newly-allocated from bpf memory
* allocator, it is still possible for 'dst' to be used in parallel by a bpf
* program or bpf syscall.
*/
static inline void check_and_init_map_value(struct bpf_map *map, void *dst)
{
bpf_obj_init(map->record, dst);
}
/* memcpy that is used with 8-byte aligned pointers, power-of-8 size and
* forced to use 'long' read/writes to try to atomically copy long counters.
* Best-effort only. No barriers here, since it _will_ race with concurrent
* updates from BPF programs. Called from bpf syscall and mostly used with
* size 8 or 16 bytes, so ask compiler to inline it.
*/
static inline void bpf_long_memcpy(void *dst, const void *src, u32 size)
{
const long *lsrc = src;
long *ldst = dst;
size /= sizeof(long);
while (size--)
data_race(*ldst++ = *lsrc++);
}
/* copy everything but bpf_spin_lock, bpf_timer, and kptrs. There could be one of each. */
static inline void bpf_obj_memcpy(struct btf_record *rec,
void *dst, void *src, u32 size,
bool long_memcpy)
{
u32 curr_off = 0;
int i;
if (IS_ERR_OR_NULL(rec)) {
if (long_memcpy)
bpf_long_memcpy(dst, src, round_up(size, 8));
else
memcpy(dst, src, size);
return;
}
for (i = 0; i < rec->cnt; i++) {
u32 next_off = rec->fields[i].offset;
u32 sz = next_off - curr_off;
memcpy(dst + curr_off, src + curr_off, sz);
curr_off += rec->fields[i].size + sz;
}
memcpy(dst + curr_off, src + curr_off, size - curr_off);
}
static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
{
bpf_obj_memcpy(map->record, dst, src, map->value_size, false);
}
static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *src)
{
bpf_obj_memcpy(map->record, dst, src, map->value_size, true);
}
static inline void bpf_obj_swap_uptrs(const struct btf_record *rec, void *dst, void *src)
{
unsigned long *src_uptr, *dst_uptr;
const struct btf_field *field;
int i;
if (!btf_record_has_field(rec, BPF_UPTR))
return;
for (i = 0, field = rec->fields; i < rec->cnt; i++, field++) {
if (field->type != BPF_UPTR)
continue;
src_uptr = src + field->offset;
dst_uptr = dst + field->offset;
swap(*src_uptr, *dst_uptr);
}
}
static inline void bpf_obj_memzero(struct btf_record *rec, void *dst, u32 size)
{
u32 curr_off = 0;
int i;
if (IS_ERR_OR_NULL(rec)) {
memset(dst, 0, size);
return;
}
for (i = 0; i < rec->cnt; i++) {
u32 next_off = rec->fields[i].offset;
u32 sz = next_off - curr_off;
memset(dst + curr_off, 0, sz);
curr_off += rec->fields[i].size + sz;
}
memset(dst + curr_off, 0, size - curr_off);
}
static inline void zero_map_value(struct bpf_map *map, void *dst)
{
bpf_obj_memzero(map->record, dst, map->value_size);
}
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
bool lock_src);
void bpf_timer_cancel_and_free(void *timer);
void bpf_wq_cancel_and_free(void *timer);
void bpf_task_work_cancel_and_free(void *timer);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock);
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock);
u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);
u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);
int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
struct bpf_offload_dev;
struct bpf_offloaded_map;
struct bpf_map_dev_ops {
int (*map_get_next_key)(struct bpf_offloaded_map *map,
void *key, void *next_key);
int (*map_lookup_elem)(struct bpf_offloaded_map *map,
void *key, void *value);
int (*map_update_elem)(struct bpf_offloaded_map *map,
void *key, void *value, u64 flags);
int (*map_delete_elem)(struct bpf_offloaded_map *map, void *key);
};
struct bpf_offloaded_map {
struct bpf_map map;
struct net_device *netdev;
const struct bpf_map_dev_ops *dev_ops;
void *dev_priv;
struct list_head offloads;
};
static inline struct bpf_offloaded_map *map_to_offmap(struct bpf_map *map)
{
return container_of(map, struct bpf_offloaded_map, map);
}
static inline bool bpf_map_offload_neutral(const struct bpf_map *map)
{
return map->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY;
}
static inline bool bpf_map_support_seq_show(const struct bpf_map *map)
{
return (map->btf_value_type_id || map->btf_vmlinux_value_type_id) &&
map->ops->map_seq_show_elem;
}
int map_check_no_btf(struct bpf_map *map,
const struct btf *btf,
const struct btf_type *key_type,
const struct btf_type *value_type);
bool bpf_map_meta_equal(const struct bpf_map *meta0,
const struct bpf_map *meta1);
static inline bool bpf_map_has_internal_structs(struct bpf_map *map)
{
return btf_record_has_field(map->record, BPF_TIMER | BPF_WORKQUEUE | BPF_TASK_WORK);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags,
struct bpf_dynptr *ptr__uninit);
#if defined(CONFIG_MMU) && defined(CONFIG_64BIT)
void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id,
u64 flags);
void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt);
#else
static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt,
int node_id, u64 flags)
{
return NULL;
}
static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt)
{
}
#endif
extern const struct bpf_map_ops bpf_map_offload_ops;
/* bpf_type_flag contains a set of flags that are applicable to the values of
* arg_type, ret_type and reg_type. For example, a pointer value may be null,
* or a memory is read-only. We classify types into two categories: base types
* and extended types. Extended types are base types combined with a type flag.
*
* Currently there are no more than 32 base types in arg_type, ret_type and
* reg_types.
*/
#define BPF_BASE_TYPE_BITS 8
enum bpf_type_flag {
/* PTR may be NULL. */
PTR_MAYBE_NULL = BIT(0 + BPF_BASE_TYPE_BITS),
/* MEM is read-only. When applied on bpf_arg, it indicates the arg is
* compatible with both mutable and immutable memory.
*/
MEM_RDONLY = BIT(1 + BPF_BASE_TYPE_BITS),
/* MEM points to BPF ring buffer reservation. */
MEM_RINGBUF = BIT(2 + BPF_BASE_TYPE_BITS),
/* MEM is in user address space. */
MEM_USER = BIT(3 + BPF_BASE_TYPE_BITS),
/* MEM is a percpu memory. MEM_PERCPU tags PTR_TO_BTF_ID. When tagged
* with MEM_PERCPU, PTR_TO_BTF_ID _cannot_ be directly accessed. In
* order to drop this tag, it must be passed into bpf_per_cpu_ptr()
* or bpf_this_cpu_ptr(), which will return the pointer corresponding
* to the specified cpu.
*/
MEM_PERCPU = BIT(4 + BPF_BASE_TYPE_BITS),
/* Indicates that the argument will be released. */
OBJ_RELEASE = BIT(5 + BPF_BASE_TYPE_BITS),
/* PTR is not trusted. This is only used with PTR_TO_BTF_ID, to mark
* unreferenced and referenced kptr loaded from map value using a load
* instruction, so that they can only be dereferenced but not escape the
* BPF program into the kernel (i.e. cannot be passed as arguments to
* kfunc or bpf helpers).
*/
PTR_UNTRUSTED = BIT(6 + BPF_BASE_TYPE_BITS),
/* MEM can be uninitialized. */
MEM_UNINIT = BIT(7 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to memory local to the bpf program. */
DYNPTR_TYPE_LOCAL = BIT(8 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to a kernel-produced ringbuf record. */
DYNPTR_TYPE_RINGBUF = BIT(9 + BPF_BASE_TYPE_BITS),
/* Size is known at compile time. */
MEM_FIXED_SIZE = BIT(10 + BPF_BASE_TYPE_BITS),
/* MEM is of an allocated object of type in program BTF. This is used to
* tag PTR_TO_BTF_ID allocated using bpf_obj_new.
*/
MEM_ALLOC = BIT(11 + BPF_BASE_TYPE_BITS),
/* PTR was passed from the kernel in a trusted context, and may be
* passed to kfuncs or BPF helper functions.
* Confusingly, this is _not_ the opposite of PTR_UNTRUSTED above.
* PTR_UNTRUSTED refers to a kptr that was read directly from a map
* without invoking bpf_kptr_xchg(). What we really need to know is
* whether a pointer is safe to pass to a kfunc or BPF helper function.
* While PTR_UNTRUSTED pointers are unsafe to pass to kfuncs and BPF
* helpers, they do not cover all possible instances of unsafe
* pointers. For example, a pointer that was obtained from walking a
* struct will _not_ get the PTR_UNTRUSTED type modifier, despite the
* fact that it may be NULL, invalid, etc. This is due to backwards
* compatibility requirements, as this was the behavior that was first
* introduced when kptrs were added. The behavior is now considered
* deprecated, and PTR_UNTRUSTED will eventually be removed.
*
* PTR_TRUSTED, on the other hand, is a pointer that the kernel
* guarantees to be valid and safe to pass to kfuncs and BPF helpers.
* For example, pointers passed to tracepoint arguments are considered
* PTR_TRUSTED, as are pointers that are passed to struct_ops
* callbacks. As alluded to above, pointers that are obtained from
* walking PTR_TRUSTED pointers are _not_ trusted. For example, if a
* struct task_struct *task is PTR_TRUSTED, then accessing
* task->last_wakee will lose the PTR_TRUSTED modifier when it's stored
* in a BPF register. Similarly, pointers passed to certain programs
* types such as kretprobes are not guaranteed to be valid, as they may
* for example contain an object that was recently freed.
*/
PTR_TRUSTED = BIT(12 + BPF_BASE_TYPE_BITS),
/* MEM is tagged with rcu and memory access needs rcu_read_lock protection. */
MEM_RCU = BIT(13 + BPF_BASE_TYPE_BITS),
/* Used to tag PTR_TO_BTF_ID | MEM_ALLOC references which are non-owning.
* Currently only valid for linked-list and rbtree nodes. If the nodes
* have a bpf_refcount_field, they must be tagged MEM_RCU as well.
*/
NON_OWN_REF = BIT(14 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to sk_buff */
DYNPTR_TYPE_SKB = BIT(15 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to xdp_buff */
DYNPTR_TYPE_XDP = BIT(16 + BPF_BASE_TYPE_BITS),
/* Memory must be aligned on some architectures, used in combination with
* MEM_FIXED_SIZE.
*/
MEM_ALIGNED = BIT(17 + BPF_BASE_TYPE_BITS),
/* MEM is being written to, often combined with MEM_UNINIT. Non-presence
* of MEM_WRITE means that MEM is only being read. MEM_WRITE without the
* MEM_UNINIT means that memory needs to be initialized since it is also
* read.
*/
MEM_WRITE = BIT(18 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to skb_metadata_end()-skb_metadata_len() */
DYNPTR_TYPE_SKB_META = BIT(19 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to file */
DYNPTR_TYPE_FILE = BIT(20 + BPF_BASE_TYPE_BITS),
__BPF_TYPE_FLAG_MAX,
__BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1,
};
#define DYNPTR_TYPE_FLAG_MASK (DYNPTR_TYPE_LOCAL | DYNPTR_TYPE_RINGBUF | DYNPTR_TYPE_SKB \
| DYNPTR_TYPE_XDP | DYNPTR_TYPE_SKB_META | DYNPTR_TYPE_FILE)
/* Max number of base types. */
#define BPF_BASE_TYPE_LIMIT (1UL << BPF_BASE_TYPE_BITS)
/* Max number of all types. */
#define BPF_TYPE_LIMIT (__BPF_TYPE_LAST_FLAG | (__BPF_TYPE_LAST_FLAG - 1))
/* function argument constraints */
enum bpf_arg_type {
ARG_DONTCARE = 0, /* unused argument in helper function */
/* the following constraints used to prototype
* bpf_map_lookup/update/delete_elem() functions
*/
ARG_CONST_MAP_PTR, /* const argument used as pointer to bpf_map */
ARG_PTR_TO_MAP_KEY, /* pointer to stack used as map key */
ARG_PTR_TO_MAP_VALUE, /* pointer to stack used as map value */
/* Used to prototype bpf_memcmp() and other functions that access data
* on eBPF program stack
*/
ARG_PTR_TO_MEM, /* pointer to valid memory (stack, packet, map value) */
ARG_PTR_TO_ARENA,
ARG_CONST_SIZE, /* number of bytes accessed from memory */
ARG_CONST_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */
ARG_PTR_TO_CTX, /* pointer to context */
ARG_ANYTHING, /* any (initialized) argument is ok */
ARG_PTR_TO_SPIN_LOCK, /* pointer to bpf_spin_lock */
ARG_PTR_TO_SOCK_COMMON, /* pointer to sock_common */
ARG_PTR_TO_SOCKET, /* pointer to bpf_sock (fullsock) */
ARG_PTR_TO_BTF_ID, /* pointer to in-kernel struct */
ARG_PTR_TO_RINGBUF_MEM, /* pointer to dynamically reserved ringbuf memory */
ARG_CONST_ALLOC_SIZE_OR_ZERO, /* number of allocated bytes requested */
ARG_PTR_TO_BTF_ID_SOCK_COMMON, /* pointer to in-kernel sock_common or bpf-mirrored bpf_sock */
ARG_PTR_TO_PERCPU_BTF_ID, /* pointer to in-kernel percpu type */
ARG_PTR_TO_FUNC, /* pointer to a bpf program function */
ARG_PTR_TO_STACK, /* pointer to stack */
ARG_PTR_TO_CONST_STR, /* pointer to a null terminated read-only string */
ARG_PTR_TO_TIMER, /* pointer to bpf_timer */
ARG_KPTR_XCHG_DEST, /* pointer to destination that kptrs are bpf_kptr_xchg'd into */
ARG_PTR_TO_DYNPTR, /* pointer to bpf_dynptr. See bpf_type_flag for dynptr type */
__BPF_ARG_TYPE_MAX,
/* Extended arg_types. */
ARG_PTR_TO_MAP_VALUE_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_MAP_VALUE,
ARG_PTR_TO_MEM_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_MEM,
ARG_PTR_TO_CTX_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_CTX,
ARG_PTR_TO_SOCKET_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_SOCKET,
ARG_PTR_TO_STACK_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_STACK,
ARG_PTR_TO_BTF_ID_OR_NULL = PTR_MAYBE_NULL | ARG_PTR_TO_BTF_ID,
/* Pointer to memory does not need to be initialized, since helper function
* fills all bytes or clears them in error case.
*/
ARG_PTR_TO_UNINIT_MEM = MEM_UNINIT | MEM_WRITE | ARG_PTR_TO_MEM,
/* Pointer to valid memory of size known at compile time. */
ARG_PTR_TO_FIXED_SIZE_MEM = MEM_FIXED_SIZE | ARG_PTR_TO_MEM,
/* This must be the last entry. Its purpose is to ensure the enum is
* wide enough to hold the higher bits reserved for bpf_type_flag.
*/
__BPF_ARG_TYPE_LIMIT = BPF_TYPE_LIMIT,
};
static_assert(__BPF_ARG_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
/* type of values returned from helper functions */
enum bpf_return_type {
RET_INTEGER, /* function returns integer */
RET_VOID, /* function doesn't return anything */
RET_PTR_TO_MAP_VALUE, /* returns a pointer to map elem value */
RET_PTR_TO_SOCKET, /* returns a pointer to a socket */
RET_PTR_TO_TCP_SOCK, /* returns a pointer to a tcp_sock */
RET_PTR_TO_SOCK_COMMON, /* returns a pointer to a sock_common */
RET_PTR_TO_MEM, /* returns a pointer to memory */
RET_PTR_TO_MEM_OR_BTF_ID, /* returns a pointer to a valid memory or a btf_id */
RET_PTR_TO_BTF_ID, /* returns a pointer to a btf_id */
__BPF_RET_TYPE_MAX,
/* Extended ret_types. */
RET_PTR_TO_MAP_VALUE_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_MAP_VALUE,
RET_PTR_TO_SOCKET_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_SOCKET,
RET_PTR_TO_TCP_SOCK_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_TCP_SOCK,
RET_PTR_TO_SOCK_COMMON_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_SOCK_COMMON,
RET_PTR_TO_RINGBUF_MEM_OR_NULL = PTR_MAYBE_NULL | MEM_RINGBUF | RET_PTR_TO_MEM,
RET_PTR_TO_DYNPTR_MEM_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_MEM,
RET_PTR_TO_BTF_ID_OR_NULL = PTR_MAYBE_NULL | RET_PTR_TO_BTF_ID,
RET_PTR_TO_BTF_ID_TRUSTED = PTR_TRUSTED | RET_PTR_TO_BTF_ID,
/* This must be the last entry. Its purpose is to ensure the enum is
* wide enough to hold the higher bits reserved for bpf_type_flag.
*/
__BPF_RET_TYPE_LIMIT = BPF_TYPE_LIMIT,
};
static_assert(__BPF_RET_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
/* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs
* to in-kernel helper functions and for adjusting imm32 field in BPF_CALL
* instructions after verifying
*/
struct bpf_func_proto {
u64 (*func)(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
bool gpl_only;
bool pkt_access;
bool might_sleep;
/* set to true if helper follows contract for llvm
* attribute bpf_fastcall:
* - void functions do not scratch r0
* - functions taking N arguments scratch only registers r1-rN
*/
bool allow_fastcall;
enum bpf_return_type ret_type;
union {
struct {
enum bpf_arg_type arg1_type;
enum bpf_arg_type arg2_type;
enum bpf_arg_type arg3_type;
enum bpf_arg_type arg4_type;
enum bpf_arg_type arg5_type;
};
enum bpf_arg_type arg_type[5];
};
union {
struct {
u32 *arg1_btf_id;
u32 *arg2_btf_id;
u32 *arg3_btf_id;
u32 *arg4_btf_id;
u32 *arg5_btf_id;
};
u32 *arg_btf_id[5];
struct {
size_t arg1_size;
size_t arg2_size;
size_t arg3_size;
size_t arg4_size;
size_t arg5_size;
};
size_t arg_size[5];
};
int *ret_btf_id; /* return value btf_id */
bool (*allowed)(const struct bpf_prog *prog);
};
/* bpf_context is intentionally undefined structure. Pointer to bpf_context is
* the first argument to eBPF programs.
* For socket filters: 'struct bpf_context *' == 'struct sk_buff *'
*/
struct bpf_context;
enum bpf_access_type {
BPF_READ = 1,
BPF_WRITE = 2
};
/* types of values stored in eBPF registers */
/* Pointer types represent:
* pointer
* pointer + imm
* pointer + (u16) var
* pointer + (u16) var + imm
* if (range > 0) then [ptr, ptr + range - off) is safe to access
* if (id > 0) means that some 'var' was added
* if (off > 0) means that 'imm' was added
*/
enum bpf_reg_type {
NOT_INIT = 0, /* nothing was written into register */
SCALAR_VALUE, /* reg doesn't contain a valid pointer */
PTR_TO_CTX, /* reg points to bpf_context */
CONST_PTR_TO_MAP, /* reg points to struct bpf_map */
PTR_TO_MAP_VALUE, /* reg points to map element value */
PTR_TO_MAP_KEY, /* reg points to a map element key */
PTR_TO_STACK, /* reg == frame_pointer + offset */
PTR_TO_PACKET_META, /* skb->data - meta_len */
PTR_TO_PACKET, /* reg points to skb->data */
PTR_TO_PACKET_END, /* skb->data + headlen */
PTR_TO_FLOW_KEYS, /* reg points to bpf_flow_keys */
PTR_TO_SOCKET, /* reg points to struct bpf_sock */
PTR_TO_SOCK_COMMON, /* reg points to sock_common */
PTR_TO_TCP_SOCK, /* reg points to struct tcp_sock */
PTR_TO_TP_BUFFER, /* reg points to a writable raw tp's buffer */
PTR_TO_XDP_SOCK, /* reg points to struct xdp_sock */
/* PTR_TO_BTF_ID points to a kernel struct that does not need
* to be null checked by the BPF program. This does not imply the
* pointer is _not_ null and in practice this can easily be a null
* pointer when reading pointer chains. The assumption is program
* context will handle null pointer dereference typically via fault
* handling. The verifier must keep this in mind and can make no
* assumptions about null or non-null when doing branch analysis.
* Further, when passed into helpers the helpers can not, without
* additional context, assume the value is non-null.
*/
PTR_TO_BTF_ID,
PTR_TO_MEM, /* reg points to valid memory region */
PTR_TO_ARENA,
PTR_TO_BUF, /* reg points to a read/write buffer */
PTR_TO_FUNC, /* reg points to a bpf program function */
PTR_TO_INSN, /* reg points to a bpf program instruction */
CONST_PTR_TO_DYNPTR, /* reg points to a const struct bpf_dynptr */
__BPF_REG_TYPE_MAX,
/* Extended reg_types. */
PTR_TO_MAP_VALUE_OR_NULL = PTR_MAYBE_NULL | PTR_TO_MAP_VALUE,
PTR_TO_SOCKET_OR_NULL = PTR_MAYBE_NULL | PTR_TO_SOCKET,
PTR_TO_SOCK_COMMON_OR_NULL = PTR_MAYBE_NULL | PTR_TO_SOCK_COMMON,
PTR_TO_TCP_SOCK_OR_NULL = PTR_MAYBE_NULL | PTR_TO_TCP_SOCK,
/* PTR_TO_BTF_ID_OR_NULL points to a kernel struct that has not
* been checked for null. Used primarily to inform the verifier
* an explicit null check is required for this struct.
*/
PTR_TO_BTF_ID_OR_NULL = PTR_MAYBE_NULL | PTR_TO_BTF_ID,
/* This must be the last entry. Its purpose is to ensure the enum is
* wide enough to hold the higher bits reserved for bpf_type_flag.
*/
__BPF_REG_TYPE_LIMIT = BPF_TYPE_LIMIT,
};
static_assert(__BPF_REG_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
/* The information passed from prog-specific *_is_valid_access
* back to the verifier.
*/
struct bpf_insn_access_aux {
enum bpf_reg_type reg_type;
bool is_ldsx;
union {
int ctx_field_size;
struct {
struct btf *btf;
u32 btf_id;
u32 ref_obj_id;
};
};
struct bpf_verifier_log *log; /* for verbose logs */
bool is_retval; /* is accessing function return value ? */
};
static inline void
bpf_ctx_record_field_size(struct bpf_insn_access_aux *aux, u32 size)
{
aux->ctx_field_size = size;
}
static bool bpf_is_ldimm64(const struct bpf_insn *insn)
{
return insn->code == (BPF_LD | BPF_IMM | BPF_DW);
}
static inline bool bpf_pseudo_func(const struct bpf_insn *insn)
{
return bpf_is_ldimm64(insn) && insn->src_reg == BPF_PSEUDO_FUNC;
}
/* Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an
* atomic load or store, and false if it is a read-modify-write instruction.
*/
static inline bool
bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn)
{
switch (atomic_insn->imm) {
case BPF_LOAD_ACQ:
case BPF_STORE_REL:
return true;
default:
return false;
}
}
struct bpf_prog_ops {
int (*test_run)(struct bpf_prog *prog, const union bpf_attr *kattr,
union bpf_attr __user *uattr);
};
struct bpf_reg_state;
struct bpf_verifier_ops {
/* return eBPF function prototype for verification */
const struct bpf_func_proto *
(*get_func_proto)(enum bpf_func_id func_id,
const struct bpf_prog *prog);
/* return true if 'size' wide access at offset 'off' within bpf_context
* with 'type' (read or write) is allowed
*/
bool (*is_valid_access)(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info);
int (*gen_prologue)(struct bpf_insn *insn, bool direct_write,
const struct bpf_prog *prog);
int (*gen_epilogue)(struct bpf_insn *insn, const struct bpf_prog *prog,
s16 ctx_stack_off);
int (*gen_ld_abs)(const struct bpf_insn *orig,
struct bpf_insn *insn_buf);
u32 (*convert_ctx_access)(enum bpf_access_type type,
const struct bpf_insn *src,
struct bpf_insn *dst,
struct bpf_prog *prog, u32 *target_size);
int (*btf_struct_access)(struct bpf_verifier_log *log,
const struct bpf_reg_state *reg,
int off, int size);
};
struct bpf_prog_offload_ops {
/* verifier basic callbacks */
int (*insn_hook)(struct bpf_verifier_env *env,
int insn_idx, int prev_insn_idx);
int (*finalize)(struct bpf_verifier_env *env);
/* verifier optimization callbacks (called after .finalize) */
int (*replace_insn)(struct bpf_verifier_env *env, u32 off,
struct bpf_insn *insn);
int (*remove_insns)(struct bpf_verifier_env *env, u32 off, u32 cnt);
/* program management callbacks */
int (*prepare)(struct bpf_prog *prog);
int (*translate)(struct bpf_prog *prog);
void (*destroy)(struct bpf_prog *prog);
};
struct bpf_prog_offload {
struct bpf_prog *prog;
struct net_device *netdev;
struct bpf_offload_dev *offdev;
void *dev_priv;
struct list_head offloads;
bool dev_state;
bool opt_failed;
void *jited_image;
u32 jited_len;
};
/* The longest tracepoint has 12 args.
* See include/trace/bpf_probe.h
*/
#define MAX_BPF_FUNC_ARGS 12
/* The maximum number of arguments passed through registers
* a single function may have.
*/
#define MAX_BPF_FUNC_REG_ARGS 5
/* The argument is a structure or a union. */
#define BTF_FMODEL_STRUCT_ARG BIT(0)
/* The argument is signed. */
#define BTF_FMODEL_SIGNED_ARG BIT(1)
struct btf_func_model {
u8 ret_size;
u8 ret_flags;
u8 nr_args;
u8 arg_size[MAX_BPF_FUNC_ARGS];
u8 arg_flags[MAX_BPF_FUNC_ARGS];
};
/* Restore arguments before returning from trampoline to let original function
* continue executing. This flag is used for fentry progs when there are no
* fexit progs.
*/
#define BPF_TRAMP_F_RESTORE_REGS BIT(0)
/* Call original function after fentry progs, but before fexit progs.
* Makes sense for fentry/fexit, normal calls and indirect calls.
*/
#define BPF_TRAMP_F_CALL_ORIG BIT(1)
/* Skip current frame and return to parent. Makes sense for fentry/fexit
* programs only. Should not be used with normal calls and indirect calls.
*/
#define BPF_TRAMP_F_SKIP_FRAME BIT(2)
/* Store IP address of the caller on the trampoline stack,
* so it's available for trampoline's programs.
*/
#define BPF_TRAMP_F_IP_ARG BIT(3)
/* Return the return value of fentry prog. Only used by bpf_struct_ops. */
#define BPF_TRAMP_F_RET_FENTRY_RET BIT(4)
/* Get original function from stack instead of from provided direct address.
* Makes sense for trampolines with fexit or fmod_ret programs.
*/
#define BPF_TRAMP_F_ORIG_STACK BIT(5)
/* This trampoline is on a function with another ftrace_ops with IPMODIFY,
* e.g., a live patch. This flag is set and cleared by ftrace call backs,
*/
#define BPF_TRAMP_F_SHARE_IPMODIFY BIT(6)
/* Indicate that current trampoline is in a tail call context. Then, it has to
* cache and restore tail_call_cnt to avoid infinite tail call loop.
*/
#define BPF_TRAMP_F_TAIL_CALL_CTX BIT(7)
/*
* Indicate the trampoline should be suitable to receive indirect calls;
* without this indirectly calling the generated code can result in #UD/#CP,
* depending on the CFI options.
*
* Used by bpf_struct_ops.
*
* Incompatible with FENTRY usage, overloads @func_addr argument.
*/
#define BPF_TRAMP_F_INDIRECT BIT(8)
/* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50
* bytes on x86.
*/
enum {
#if defined(__s390x__)
BPF_MAX_TRAMP_LINKS = 27,
#else
BPF_MAX_TRAMP_LINKS = 38,
#endif
};
#define BPF_TRAMP_COOKIE_INDEX_SHIFT 8
#define BPF_TRAMP_IS_RETURN_SHIFT 63
struct bpf_tramp_links {
struct bpf_tramp_link *links[BPF_MAX_TRAMP_LINKS];
int nr_links;
};
struct bpf_tramp_run_ctx;
/* Different use cases for BPF trampoline:
* 1. replace nop at the function entry (kprobe equivalent)
* flags = BPF_TRAMP_F_RESTORE_REGS
* fentry = a set of programs to run before returning from trampoline
*
* 2. replace nop at the function entry (kprobe + kretprobe equivalent)
* flags = BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_SKIP_FRAME
* orig_call = fentry_ip + MCOUNT_INSN_SIZE
* fentry = a set of program to run before calling original function
* fexit = a set of program to run after original function
*
* 3. replace direct call instruction anywhere in the function body
* or assign a function pointer for indirect call (like tcp_congestion_ops->cong_avoid)
* With flags = 0
* fentry = a set of programs to run before returning from trampoline
* With flags = BPF_TRAMP_F_CALL_ORIG
* orig_call = original callback addr or direct function addr
* fentry = a set of program to run before calling original function
* fexit = a set of program to run after original function
*/
struct bpf_tramp_image;
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,
const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks,
void *func_addr);
void *arch_alloc_bpf_trampoline(unsigned int size);
void arch_free_bpf_trampoline(void *image, unsigned int size);
int __must_check arch_protect_bpf_trampoline(void *image, unsigned int size);
int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
struct bpf_tramp_links *tlinks, void *func_addr);
u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog,
struct bpf_tramp_run_ctx *run_ctx);
void notrace __bpf_prog_exit_sleepable_recur(struct bpf_prog *prog, u64 start,
struct bpf_tramp_run_ctx *run_ctx);
void notrace __bpf_tramp_enter(struct bpf_tramp_image *tr);
void notrace __bpf_tramp_exit(struct bpf_tramp_image *tr);
typedef u64 (*bpf_trampoline_enter_t)(struct bpf_prog *prog,
struct bpf_tramp_run_ctx *run_ctx);
typedef void (*bpf_trampoline_exit_t)(struct bpf_prog *prog, u64 start,
struct bpf_tramp_run_ctx *run_ctx);
bpf_trampoline_enter_t bpf_trampoline_enter(const struct bpf_prog *prog);
bpf_trampoline_exit_t bpf_trampoline_exit(const struct bpf_prog *prog);
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_JMP
static inline bool bpf_trampoline_use_jmp(u64 flags)
{
return flags & BPF_TRAMP_F_CALL_ORIG && !(flags & BPF_TRAMP_F_SKIP_FRAME);
}
#else
static inline bool bpf_trampoline_use_jmp(u64 flags)
{
return false;
}
#endif
struct bpf_ksym {
unsigned long start;
unsigned long end;
char name[KSYM_NAME_LEN];
struct list_head lnode;
struct latch_tree_node tnode;
bool prog;
u32 fp_start;
u32 fp_end;
};
enum bpf_tramp_prog_type {
BPF_TRAMP_FENTRY,
BPF_TRAMP_FEXIT,
BPF_TRAMP_MODIFY_RETURN,
BPF_TRAMP_MAX,
BPF_TRAMP_REPLACE, /* more than MAX */
BPF_TRAMP_FSESSION,
};
struct bpf_tramp_image {
void *image;
int size;
struct bpf_ksym ksym;
struct percpu_ref pcref;
void *ip_after_call;
void *ip_epilogue;
union {
struct rcu_head rcu;
struct work_struct work;
};
};
struct bpf_trampoline {
/* hlist for trampoline_key_table */
struct hlist_node hlist_key;
/* hlist for trampoline_ip_table */
struct hlist_node hlist_ip;
struct ftrace_ops *fops;
/* serializes access to fields of this trampoline */
struct mutex mutex;
refcount_t refcnt;
u32 flags;
u64 key;
unsigned long ip;
struct {
struct btf_func_model model;
void *addr;
bool ftrace_managed;
} func;
/* if !NULL this is BPF_PROG_TYPE_EXT program that extends another BPF
* program by replacing one of its functions. func.addr is the address
* of the function it replaced.
*/
struct bpf_prog *extension_prog;
/* list of BPF programs using this trampoline */
struct hlist_head progs_hlist[BPF_TRAMP_MAX];
/* Number of attached programs. A counter per kind. */
int progs_cnt[BPF_TRAMP_MAX];
/* Executable image of trampoline */
struct bpf_tramp_image *cur_image;
};
struct bpf_attach_target_info {
struct btf_func_model fmodel;
long tgt_addr;
struct module *tgt_mod;
const char *tgt_name;
const struct btf_type *tgt_type;
};
#define BPF_DISPATCHER_MAX 48 /* Fits in 2048B */
struct bpf_dispatcher_prog {
struct bpf_prog *prog;
refcount_t users;
};
struct bpf_dispatcher {
/* dispatcher mutex */
struct mutex mutex;
void *func;
struct bpf_dispatcher_prog progs[BPF_DISPATCHER_MAX];
int num_progs;
void *image;
void *rw_image;
u32 image_off;
struct bpf_ksym ksym;
#ifdef CONFIG_HAVE_STATIC_CALL
struct static_call_key *sc_key;
void *sc_tramp;
#endif
};
#ifndef __bpfcall
#define __bpfcall __nocfi
#endif
static __always_inline __bpfcall unsigned int bpf_dispatcher_nop_func(
const void *ctx,
const struct bpf_insn *insnsi,
bpf_func_t bpf_func)
{
return bpf_func(ctx, insnsi);
}
/* the implementation of the opaque uapi struct bpf_dynptr */
struct bpf_dynptr_kern {
void *data;
/* Size represents the number of usable bytes of dynptr data.
* If for example the offset is at 4 for a local dynptr whose data is
* of type u64, the number of usable bytes is 4.
*
* The upper 8 bits are reserved. It is as follows:
* Bits 0 - 23 = size
* Bits 24 - 30 = dynptr type
* Bit 31 = whether dynptr is read-only
*/
u32 size;
u32 offset;
} __aligned(8);
enum bpf_dynptr_type {
BPF_DYNPTR_TYPE_INVALID,
/* Points to memory that is local to the bpf program */
BPF_DYNPTR_TYPE_LOCAL,
/* Underlying data is a ringbuf record */
BPF_DYNPTR_TYPE_RINGBUF,
/* Underlying data is a sk_buff */
BPF_DYNPTR_TYPE_SKB,
/* Underlying data is a xdp_buff */
BPF_DYNPTR_TYPE_XDP,
/* Points to skb_metadata_end()-skb_metadata_len() */
BPF_DYNPTR_TYPE_SKB_META,
/* Underlying data is a file */
BPF_DYNPTR_TYPE_FILE,
};
int bpf_dynptr_check_size(u64 size);
u64 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr);
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u64 len);
void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u64 len);
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr);
int __bpf_dynptr_write(const struct bpf_dynptr_kern *dst, u64 offset,
void *src, u64 len, u64 flags);
void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk);
static inline int bpf_dynptr_check_off_len(const struct bpf_dynptr_kern *ptr, u64 offset, u64 len)
{
u64 size = __bpf_dynptr_size(ptr);
if (len > size || offset > size - len)
return -E2BIG;
return 0;
}
#ifdef CONFIG_BPF_JIT
int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog);
int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog);
struct bpf_trampoline *bpf_trampoline_get(u64 key,
struct bpf_attach_target_info *tgt_info);
void bpf_trampoline_put(struct bpf_trampoline *tr);
int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
/*
* When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
* indirection with a direct call to the bpf program. If the architecture does
* not have STATIC_CALL, avoid a double-indirection.
*/
#ifdef CONFIG_HAVE_STATIC_CALL
#define __BPF_DISPATCHER_SC_INIT(_name) \
.sc_key = &STATIC_CALL_KEY(_name), \
.sc_tramp = STATIC_CALL_TRAMP_ADDR(_name),
#define __BPF_DISPATCHER_SC(name) \
DEFINE_STATIC_CALL(bpf_dispatcher_##name##_call, bpf_dispatcher_nop_func)
#define __BPF_DISPATCHER_CALL(name) \
static_call(bpf_dispatcher_##name##_call)(ctx, insnsi, bpf_func)
#define __BPF_DISPATCHER_UPDATE(_d, _new) \
__static_call_update((_d)->sc_key, (_d)->sc_tramp, (_new))
#else
#define __BPF_DISPATCHER_SC_INIT(name)
#define __BPF_DISPATCHER_SC(name)
#define __BPF_DISPATCHER_CALL(name) bpf_func(ctx, insnsi)
#define __BPF_DISPATCHER_UPDATE(_d, _new)
#endif
#define BPF_DISPATCHER_INIT(_name) { \
.mutex = __MUTEX_INITIALIZER(_name.mutex), \
.func = &_name##_func, \
.progs = {}, \
.num_progs = 0, \
.image = NULL, \
.image_off = 0, \
.ksym = { \
.name = #_name, \
.lnode = LIST_HEAD_INIT(_name.ksym.lnode), \
}, \
__BPF_DISPATCHER_SC_INIT(_name##_call) \
}
#define DEFINE_BPF_DISPATCHER(name) \
__BPF_DISPATCHER_SC(name); \
noinline __bpfcall unsigned int bpf_dispatcher_##name##_func( \
const void *ctx, \
const struct bpf_insn *insnsi, \
bpf_func_t bpf_func) \
{ \
return __BPF_DISPATCHER_CALL(name); \
} \
EXPORT_SYMBOL(bpf_dispatcher_##name##_func); \
struct bpf_dispatcher bpf_dispatcher_##name = \
BPF_DISPATCHER_INIT(bpf_dispatcher_##name);
#define DECLARE_BPF_DISPATCHER(name) \
unsigned int bpf_dispatcher_##name##_func( \
const void *ctx, \
const struct bpf_insn *insnsi, \
bpf_func_t bpf_func); \
extern struct bpf_dispatcher bpf_dispatcher_##name;
#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_##name##_func
#define BPF_DISPATCHER_PTR(name) (&bpf_dispatcher_##name)
void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
struct bpf_prog *to);
/* Called only from JIT-enabled code, so there's no need for stubs. */
void bpf_image_ksym_init(void *data, unsigned int size, struct bpf_ksym *ksym);
void bpf_image_ksym_add(struct bpf_ksym *ksym);
void bpf_image_ksym_del(struct bpf_ksym *ksym);
void bpf_ksym_add(struct bpf_ksym *ksym);
void bpf_ksym_del(struct bpf_ksym *ksym);
bool bpf_has_frame_pointer(unsigned long ip);
int bpf_jit_charge_modmem(u32 size);
void bpf_jit_uncharge_modmem(u32 size);
bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog,
int insn_idx);
#else
static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
return -ENOTSUPP;
}
static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
struct bpf_trampoline *tr,
struct bpf_prog *tgt_prog)
{
return -ENOTSUPP;
}
static inline struct bpf_trampoline *bpf_trampoline_get(u64 key,
struct bpf_attach_target_info *tgt_info)
{
return NULL;
}
static inline void bpf_trampoline_put(struct bpf_trampoline *tr) {}
#define DEFINE_BPF_DISPATCHER(name)
#define DECLARE_BPF_DISPATCHER(name)
#define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_nop_func
#define BPF_DISPATCHER_PTR(name) NULL
static inline void bpf_dispatcher_change_prog(struct bpf_dispatcher *d,
struct bpf_prog *from,
struct bpf_prog *to) {}
static inline bool is_bpf_image_address(unsigned long address)
{
return false;
}
static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog)
{
return false;
}
#endif
struct bpf_func_info_aux {
u16 linkage;
bool unreliable;
bool called : 1;
bool verified : 1;
};
enum bpf_jit_poke_reason {
BPF_POKE_REASON_TAIL_CALL,
};
/* Descriptor of pokes pointing /into/ the JITed image. */
struct bpf_jit_poke_descriptor {
void *tailcall_target;
void *tailcall_bypass;
void *bypass_addr;
void *aux;
union {
struct {
struct bpf_map *map;
u32 key;
} tail_call;
};
bool tailcall_target_stable;
u8 adj_off;
u16 reason;
u32 insn_idx;
};
/* reg_type info for ctx arguments */
struct bpf_ctx_arg_aux {
u32 offset;
enum bpf_reg_type reg_type;
struct btf *btf;
u32 btf_id;
u32 ref_obj_id;
bool refcounted;
};
struct btf_mod_pair {
struct btf *btf;
struct module *module;
};
struct bpf_kfunc_desc_tab;
enum bpf_stream_id {
BPF_STDOUT = 1,
BPF_STDERR = 2,
};
struct bpf_stream_elem {
struct llist_node node;
int total_len;
int consumed_len;
char str[];
};
enum {
/* 100k bytes */
BPF_STREAM_MAX_CAPACITY = 100000ULL,
};
struct bpf_stream {
atomic_t capacity;
struct llist_head log; /* list of in-flight stream elements in LIFO order */
struct mutex lock; /* lock protecting backlog_{head,tail} */
struct llist_node *backlog_head; /* list of in-flight stream elements in FIFO order */
struct llist_node *backlog_tail; /* tail of the list above */
};
struct bpf_stream_stage {
struct llist_head log;
int len;
};
struct bpf_prog_aux {
atomic64_t refcnt;
u32 used_map_cnt;
u32 used_btf_cnt;
u32 max_ctx_offset;
u32 max_pkt_offset;
u32 max_tp_access;
u32 stack_depth;
u32 id;
u32 func_cnt; /* used by non-func prog as the number of func progs */
u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */
u32 func_idx; /* 0 for non-func prog, the index in func array for func prog */
u32 attach_btf_id; /* in-kernel BTF type id to attach to */
u32 attach_st_ops_member_off;
u32 ctx_arg_info_size;
u32 max_rdonly_access;
u32 max_rdwr_access;
u32 subprog_start;
struct btf *attach_btf;
struct bpf_ctx_arg_aux *ctx_arg_info;
void __percpu *priv_stack_ptr;
struct mutex dst_mutex; /* protects dst_* pointers below, *after* prog becomes visible */
struct bpf_prog *dst_prog;
struct bpf_trampoline *dst_trampoline;
enum bpf_prog_type saved_dst_prog_type;
enum bpf_attach_type saved_dst_attach_type;
bool verifier_zext; /* Zero extensions has been inserted by verifier. */
bool dev_bound; /* Program is bound to the netdev. */
bool offload_requested; /* Program is bound and offloaded to the netdev. */
bool attach_btf_trace; /* true if attaching to BTF-enabled raw tp */
bool attach_tracing_prog; /* true if tracing another tracing program */
bool func_proto_unreliable;
bool tail_call_reachable;
bool xdp_has_frags;
bool exception_cb;
bool exception_boundary;
bool is_extended; /* true if extended by freplace program */
bool jits_use_priv_stack;
bool priv_stack_requested;
bool changes_pkt_data;
bool might_sleep;
bool kprobe_write_ctx;
u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */
struct bpf_arena *arena;
void (*recursion_detected)(struct bpf_prog *prog); /* callback if recursion is detected */
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
/* function name for valid attach_btf_id */
const char *attach_func_name;
struct bpf_prog **func;
struct bpf_prog_aux *main_prog_aux;
void *jit_data; /* JIT specific data. arch dependent */
struct bpf_jit_poke_descriptor *poke_tab;
struct bpf_kfunc_desc_tab *kfunc_tab;
struct bpf_kfunc_btf_tab *kfunc_btf_tab;
u32 size_poke_tab;
#ifdef CONFIG_FINEIBT
struct bpf_ksym ksym_prefix;
#endif
struct bpf_ksym ksym;
const struct bpf_prog_ops *ops;
const struct bpf_struct_ops *st_ops;
struct bpf_map **used_maps;
struct mutex used_maps_mutex; /* mutex for used_maps and used_map_cnt */
struct btf_mod_pair *used_btfs;
struct bpf_prog *prog;
struct user_struct *user;
u64 load_time; /* ns since boottime */
u32 verified_insns;
int cgroup_atype; /* enum cgroup_bpf_attach_type */
struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
char name[BPF_OBJ_NAME_LEN];
u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
#ifdef CONFIG_SECURITY
void *security;
#endif
struct bpf_token *token;
struct bpf_prog_offload *offload;
struct btf *btf;
struct bpf_func_info *func_info;
struct bpf_func_info_aux *func_info_aux;
/* bpf_line_info loaded from userspace. linfo->insn_off
* has the xlated insn offset.
* Both the main and sub prog share the same linfo.
* The subprog can access its first linfo by
* using the linfo_idx.
*/
struct bpf_line_info *linfo;
/* jited_linfo is the jited addr of the linfo. It has a
* one to one mapping to linfo:
* jited_linfo[i] is the jited addr for the linfo[i]->insn_off.
* Both the main and sub prog share the same jited_linfo.
* The subprog can access its first jited_linfo by
* using the linfo_idx.
*/
void **jited_linfo;
u32 func_info_cnt;
u32 nr_linfo;
/* subprog can use linfo_idx to access its first linfo and
* jited_linfo.
* main prog always has linfo_idx == 0
*/
u32 linfo_idx;
struct module *mod;
u32 num_exentries;
struct exception_table_entry *extable;
union {
struct work_struct work;
struct rcu_head rcu;
};
struct bpf_stream stream[2];
struct mutex st_ops_assoc_mutex;
struct bpf_map __rcu *st_ops_assoc;
};
#define BPF_NR_CONTEXTS 4 /* normal, softirq, hardirq, NMI */
struct bpf_prog {
u16 pages; /* Number of allocated pages */
u16 jited:1, /* Is our filter JIT'ed? */
jit_requested:1,/* archs need to JIT the prog */
gpl_compatible:1, /* Is filter GPL compatible? */
cb_access:1, /* Is control block accessed? */
dst_needed:1, /* Do we need dst entry? */
blinding_requested:1, /* needs constant blinding */
blinded:1, /* Was blinded */
is_func:1, /* program is a bpf function */
kprobe_override:1, /* Do we override a kprobe? */
has_callchain_buf:1, /* callchain buffer allocated? */
enforce_expected_attach_type:1, /* Enforce expected_attach_type checking at attach time */
call_get_stack:1, /* Do we call bpf_get_stack() or bpf_get_stackid() */
call_get_func_ip:1, /* Do we call get_func_ip() */
call_session_cookie:1, /* Do we call bpf_session_cookie() */
tstamp_type_access:1, /* Accessed __sk_buff->tstamp_type */
sleepable:1; /* BPF program is sleepable */
enum bpf_prog_type type; /* Type of BPF program */
enum bpf_attach_type expected_attach_type; /* For some prog types */
u32 len; /* Number of filter blocks */
u32 jited_len; /* Size of jited insns in bytes */
union {
u8 digest[SHA256_DIGEST_SIZE];
u8 tag[BPF_TAG_SIZE];
};
struct bpf_prog_stats __percpu *stats;
u8 __percpu *active; /* u8[BPF_NR_CONTEXTS] for recursion protection */
unsigned int (*bpf_func)(const void *ctx,
const struct bpf_insn *insn);
struct bpf_prog_aux *aux; /* Auxiliary fields */
struct sock_fprog_kern *orig_prog; /* Original BPF program */
/* Instructions for interpreter */
union {
DECLARE_FLEX_ARRAY(struct sock_filter, insns);
DECLARE_FLEX_ARRAY(struct bpf_insn, insnsi);
};
};
struct bpf_array_aux {
/* Programs with direct jumps into programs part of this array. */
struct list_head poke_progs;
struct bpf_map *map;
struct mutex poke_mutex;
struct work_struct work;
};
struct bpf_link {
atomic64_t refcnt;
u32 id;
enum bpf_link_type type;
const struct bpf_link_ops *ops;
struct bpf_prog *prog;
u32 flags;
enum bpf_attach_type attach_type;
/* rcu is used before freeing, work can be used to schedule that
* RCU-based freeing before that, so they never overlap
*/
union {
struct rcu_head rcu;
struct work_struct work;
};
/* whether BPF link itself has "sleepable" semantics, which can differ
* from underlying BPF program having a "sleepable" semantics, as BPF
* link's semantics is determined by target attach hook
*/
bool sleepable;
};
struct bpf_link_ops {
void (*release)(struct bpf_link *link);
/* deallocate link resources callback, called without RCU grace period
* waiting
*/
void (*dealloc)(struct bpf_link *link);
/* deallocate link resources callback, called after RCU grace period;
* if either the underlying BPF program is sleepable or BPF link's
* target hook is sleepable, we'll go through tasks trace RCU GP and
* then "classic" RCU GP; this need for chaining tasks trace and
* classic RCU GPs is designated by setting bpf_link->sleepable flag
*
* For non-sleepable tracepoint links we go through SRCU gp instead,
* since RCU is not used in that case. Sleepable tracepoints still
* follow the scheme above.
*/
void (*dealloc_deferred)(struct bpf_link *link);
int (*detach)(struct bpf_link *link);
int (*update_prog)(struct bpf_link *link, struct bpf_prog *new_prog,
struct bpf_prog *old_prog);
void (*show_fdinfo)(const struct bpf_link *link, struct seq_file *seq);
int (*fill_link_info)(const struct bpf_link *link,
struct bpf_link_info *info);
int (*update_map)(struct bpf_link *link, struct bpf_map *new_map,
struct bpf_map *old_map);
__poll_t (*poll)(struct file *file, struct poll_table_struct *pts);
};
struct bpf_tramp_link {
struct bpf_link link;
struct hlist_node tramp_hlist;
u64 cookie;
};
struct bpf_shim_tramp_link {
struct bpf_tramp_link link;
struct bpf_trampoline *trampoline;
};
struct bpf_tracing_link {
struct bpf_tramp_link link;
struct bpf_trampoline *trampoline;
struct bpf_prog *tgt_prog;
};
struct bpf_fsession_link {
struct bpf_tracing_link link;
struct bpf_tramp_link fexit;
};
struct bpf_raw_tp_link {
struct bpf_link link;
struct bpf_raw_event_map *btp;
u64 cookie;
};
struct bpf_link_primer {
struct bpf_link *link;
struct file *file;
int fd;
u32 id;
};
struct bpf_mount_opts {
kuid_t uid;
kgid_t gid;
umode_t mode;
/* BPF token-related delegation options */
u64 delegate_cmds;
u64 delegate_maps;
u64 delegate_progs;
u64 delegate_attachs;
};
struct bpf_token {
struct work_struct work;
atomic64_t refcnt;
struct user_namespace *userns;
u64 allowed_cmds;
u64 allowed_maps;
u64 allowed_progs;
u64 allowed_attachs;
#ifdef CONFIG_SECURITY
void *security;
#endif
};
struct bpf_struct_ops_value;
struct btf_member;
#define BPF_STRUCT_OPS_MAX_NR_MEMBERS 64
/**
* struct bpf_struct_ops - A structure of callbacks allowing a subsystem to
* define a BPF_MAP_TYPE_STRUCT_OPS map type composed
* of BPF_PROG_TYPE_STRUCT_OPS progs.
* @verifier_ops: A structure of callbacks that are invoked by the verifier
* when determining whether the struct_ops progs in the
* struct_ops map are valid.
* @init: A callback that is invoked a single time, and before any other
* callback, to initialize the structure. A nonzero return value means
* the subsystem could not be initialized.
* @check_member: When defined, a callback invoked by the verifier to allow
* the subsystem to determine if an entry in the struct_ops map
* is valid. A nonzero return value means that the map is
* invalid and should be rejected by the verifier.
* @init_member: A callback that is invoked for each member of the struct_ops
* map to allow the subsystem to initialize the member. A nonzero
* value means the member could not be initialized. This callback
* is exclusive with the @type, @type_id, @value_type, and
* @value_id fields.
* @reg: A callback that is invoked when the struct_ops map has been
* initialized and is being attached to. Zero means the struct_ops map
* has been successfully registered and is live. A nonzero return value
* means the struct_ops map could not be registered.
* @unreg: A callback that is invoked when the struct_ops map should be
* unregistered.
* @update: A callback that is invoked when the live struct_ops map is being
* updated to contain new values. This callback is only invoked when
* the struct_ops map is loaded with BPF_F_LINK. If not defined, the
* it is assumed that the struct_ops map cannot be updated.
* @validate: A callback that is invoked after all of the members have been
* initialized. This callback should perform static checks on the
* map, meaning that it should either fail or succeed
* deterministically. A struct_ops map that has been validated may
* not necessarily succeed in being registered if the call to @reg
* fails. For example, a valid struct_ops map may be loaded, but
* then fail to be registered due to there being another active
* struct_ops map on the system in the subsystem already. For this
* reason, if this callback is not defined, the check is skipped as
* the struct_ops map will have final verification performed in
* @reg.
* @cfi_stubs: Pointer to a structure of stub functions for CFI. These stubs
* provide the correct Control Flow Integrity hashes for the
* trampolines generated by BPF struct_ops.
* @owner: The module that owns this struct_ops. Used for module reference
* counting to ensure the module providing the struct_ops cannot be
* unloaded while in use.
* @name: The name of the struct bpf_struct_ops object.
* @func_models: Func models
*/
struct bpf_struct_ops {
const struct bpf_verifier_ops *verifier_ops;
int (*init)(struct btf *btf);
int (*check_member)(const struct btf_type *t,
const struct btf_member *member,
const struct bpf_prog *prog);
int (*init_member)(const struct btf_type *t,
const struct btf_member *member,
void *kdata, const void *udata);
int (*reg)(void *kdata, struct bpf_link *link);
void (*unreg)(void *kdata, struct bpf_link *link);
int (*update)(void *kdata, void *old_kdata, struct bpf_link *link);
int (*validate)(void *kdata);
void *cfi_stubs;
struct module *owner;
const char *name;
struct btf_func_model func_models[BPF_STRUCT_OPS_MAX_NR_MEMBERS];
};
/* Every member of a struct_ops type has an instance even a member is not
* an operator (function pointer). The "info" field will be assigned to
* prog->aux->ctx_arg_info of BPF struct_ops programs to provide the
* argument information required by the verifier to verify the program.
*
* btf_ctx_access() will lookup prog->aux->ctx_arg_info to find the
* corresponding entry for an given argument.
*/
struct bpf_struct_ops_arg_info {
struct bpf_ctx_arg_aux *info;
u32 cnt;
};
struct bpf_struct_ops_desc {
struct bpf_struct_ops *st_ops;
const struct btf_type *type;
const struct btf_type *value_type;
u32 type_id;
u32 value_id;
/* Collection of argument information for each member */
struct bpf_struct_ops_arg_info *arg_info;
};
enum bpf_struct_ops_state {
BPF_STRUCT_OPS_STATE_INIT,
BPF_STRUCT_OPS_STATE_INUSE,
BPF_STRUCT_OPS_STATE_TOBEFREE,
BPF_STRUCT_OPS_STATE_READY,
};
struct bpf_struct_ops_common_value {
refcount_t refcnt;
enum bpf_struct_ops_state state;
};
static inline bool bpf_prog_get_recursion_context(struct bpf_prog *prog)
{
#ifdef CONFIG_ARM64
u8 rctx = interrupt_context_level();
u8 *active = this_cpu_ptr(prog->active);
u32 val;
preempt_disable();
active[rctx]++;
val = le32_to_cpu(*(__le32 *)active);
preempt_enable();
if (val != BIT(rctx * 8))
return false;
return true;
#else
return this_cpu_inc_return(*(int __percpu *)(prog->active)) == 1;
#endif
}
static inline void bpf_prog_put_recursion_context(struct bpf_prog *prog)
{
#ifdef CONFIG_ARM64
u8 rctx = interrupt_context_level();
u8 *active = this_cpu_ptr(prog->active);
preempt_disable();
active[rctx]--;
preempt_enable();
#else
this_cpu_dec(*(int __percpu *)(prog->active));
#endif
}
#if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL)
/* This macro helps developer to register a struct_ops type and generate
* type information correctly. Developers should use this macro to register
* a struct_ops type instead of calling __register_bpf_struct_ops() directly.
*/
#define register_bpf_struct_ops(st_ops, type) \
({ \
struct bpf_struct_ops_##type { \
struct bpf_struct_ops_common_value common; \
struct type data ____cacheline_aligned_in_smp; \
}; \
BTF_TYPE_EMIT(struct bpf_struct_ops_##type); \
__register_bpf_struct_ops(st_ops); \
})
#define BPF_MODULE_OWNER ((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA))
bool bpf_struct_ops_get(const void *kdata);
void bpf_struct_ops_put(const void *kdata);
int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff);
int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key,
void *value);
int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
struct bpf_tramp_link *link,
const struct btf_func_model *model,
void *stub_func,
void **image, u32 *image_off,
bool allow_alloc);
void bpf_struct_ops_image_free(void *image);
static inline bool bpf_try_module_get(const void *data, struct module *owner)
{
if (owner == BPF_MODULE_OWNER)
return bpf_struct_ops_get(data);
else
return try_module_get(owner);
}
static inline void bpf_module_put(const void *data, struct module *owner)
{
if (owner == BPF_MODULE_OWNER)
bpf_struct_ops_put(data);
else
module_put(owner);
}
int bpf_struct_ops_link_create(union bpf_attr *attr);
int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map);
void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog);
void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux);
u32 bpf_struct_ops_id(const void *kdata);
#ifdef CONFIG_NET
/* Define it here to avoid the use of forward declaration */
struct bpf_dummy_ops_state {
int val;
};
struct bpf_dummy_ops {
int (*test_1)(struct bpf_dummy_ops_state *cb);
int (*test_2)(struct bpf_dummy_ops_state *cb, int a1, unsigned short a2,
char a3, unsigned long a4);
int (*test_sleepable)(struct bpf_dummy_ops_state *cb);
};
int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
union bpf_attr __user *uattr);
#endif
int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
struct btf *btf,
struct bpf_verifier_log *log);
void bpf_map_struct_ops_info_fill(struct bpf_map_info *info, struct bpf_map *map);
void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_ops_desc);
#else
#define register_bpf_struct_ops(st_ops, type) ({ (void *)(st_ops); 0; })
static inline bool bpf_try_module_get(const void *data, struct module *owner)
{
return try_module_get(owner);
}
static inline void bpf_module_put(const void *data, struct module *owner)
{
module_put(owner);
}
static inline int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff)
{
return -ENOTSUPP;
}
static inline int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map,
void *key,
void *value)
{
return -EINVAL;
}
static inline int bpf_struct_ops_link_create(union bpf_attr *attr)
{
return -EOPNOTSUPP;
}
static inline int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map)
{
return -EOPNOTSUPP;
}
static inline void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog)
{
}
static inline void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux)
{
return NULL;
}
static inline void bpf_map_struct_ops_info_fill(struct bpf_map_info *info, struct bpf_map *map)
{
}
static inline void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_ops_desc)
{
}
#endif
static inline int bpf_fsession_cnt(struct bpf_tramp_links *links)
{
struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];
int cnt = 0;
for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {
if (fentries.links[i]->link.prog->expected_attach_type == BPF_TRACE_FSESSION)
cnt++;
}
return cnt;
}
static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_link *link)
{
return link->link.prog->call_session_cookie;
}
static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_links *links)
{
struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY];
int cnt = 0;
for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) {
if (bpf_prog_calls_session_cookie(fentries.links[i]))
cnt++;
}
return cnt;
}
int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
const struct bpf_ctx_arg_aux *info, u32 cnt);
#if defined(CONFIG_CGROUP_BPF) && defined(CONFIG_BPF_LSM)
int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
int cgroup_atype,
enum bpf_attach_type attach_type);
void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog);
#else
static inline int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
int cgroup_atype,
enum bpf_attach_type attach_type)
{
return -EOPNOTSUPP;
}
static inline void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog)
{
}
#endif
struct bpf_array {
struct bpf_map map;
u32 elem_size;
u32 index_mask;
struct bpf_array_aux *aux;
union {
DECLARE_FLEX_ARRAY(char, value) __aligned(8);
DECLARE_FLEX_ARRAY(void *, ptrs) __aligned(8);
DECLARE_FLEX_ARRAY(void __percpu *, pptrs) __aligned(8);
};
};
/*
* The bpf_array_get_next_key() function may be used for all array-like
* maps, i.e., maps with u32 keys with range [0 ,..., max_entries)
*/
int bpf_array_get_next_key(struct bpf_map *map, void *key, void *next_key);
#define BPF_COMPLEXITY_LIMIT_INSNS 1000000 /* yes. 1M insns */
#define MAX_TAIL_CALL_CNT 33
/* Maximum number of loops for bpf_loop and bpf_iter_num.
* It's enum to expose it (and thus make it discoverable) through BTF.
*/
enum {
BPF_MAX_LOOPS = 8 * 1024 * 1024,
BPF_MAX_TIMED_LOOPS = 0xffff,
};
#define BPF_F_ACCESS_MASK (BPF_F_RDONLY | \
BPF_F_RDONLY_PROG | \
BPF_F_WRONLY | \
BPF_F_WRONLY_PROG)
#define BPF_MAP_CAN_READ BIT(0)
#define BPF_MAP_CAN_WRITE BIT(1)
/* Maximum number of user-producer ring buffer samples that can be drained in
* a call to bpf_user_ringbuf_drain().
*/
#define BPF_MAX_USER_RINGBUF_SAMPLES (128 * 1024)
static inline u32 bpf_map_flags_to_cap(struct bpf_map *map)
{
u32 access_flags = map->map_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG);
/* Combination of BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG is
* not possible.
*/
if (access_flags & BPF_F_RDONLY_PROG)
return BPF_MAP_CAN_READ;
else if (access_flags & BPF_F_WRONLY_PROG)
return BPF_MAP_CAN_WRITE;
else
return BPF_MAP_CAN_READ | BPF_MAP_CAN_WRITE;
}
static inline bool bpf_map_flags_access_ok(u32 access_flags)
{
return (access_flags & (BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG)) !=
(BPF_F_RDONLY_PROG | BPF_F_WRONLY_PROG);
}
static inline struct bpf_map_owner *bpf_map_owner_alloc(struct bpf_map *map)
{
return kzalloc_obj(*map->owner, GFP_ATOMIC);
}
static inline void bpf_map_owner_free(struct bpf_map *map)
{
kfree(map->owner);
}
struct bpf_event_entry {
struct perf_event *event;
struct file *perf_file;
struct file *map_file;
struct rcu_head rcu;
};
static inline bool map_type_contains_progs(struct bpf_map *map)
{
return map->map_type == BPF_MAP_TYPE_PROG_ARRAY ||
map->map_type == BPF_MAP_TYPE_DEVMAP ||
map->map_type == BPF_MAP_TYPE_CPUMAP;
}
bool bpf_prog_map_compatible(struct bpf_map *map, const struct bpf_prog *fp);
int bpf_prog_calc_tag(struct bpf_prog *fp);
const struct bpf_func_proto *bpf_get_trace_printk_proto(void);
const struct bpf_func_proto *bpf_get_trace_vprintk_proto(void);
const struct bpf_func_proto *bpf_get_perf_event_read_value_proto(void);
typedef unsigned long (*bpf_ctx_copy_t)(void *dst, const void *src,
unsigned long off, unsigned long len);
typedef u32 (*bpf_convert_ctx_access_t)(enum bpf_access_type type,
const struct bpf_insn *src,
struct bpf_insn *dst,
struct bpf_prog *prog,
u32 *target_size);
u64 bpf_event_output(struct bpf_map *map, u64 flags, void *meta, u64 meta_size,
void *ctx, u64 ctx_size, bpf_ctx_copy_t ctx_copy);
/* an array of programs to be executed under rcu_lock.
*
* Typical usage:
* ret = bpf_prog_run_array(rcu_dereference(&bpf_prog_array), ctx, bpf_prog_run);
*
* the structure returned by bpf_prog_array_alloc() should be populated
* with program pointers and the last pointer must be NULL.
* The user has to keep refcnt on the program and make sure the program
* is removed from the array before bpf_prog_put().
* The 'struct bpf_prog_array *' should only be replaced with xchg()
* since other cpus are walking the array of pointers in parallel.
*/
struct bpf_prog_array_item {
struct bpf_prog *prog;
union {
struct bpf_cgroup_storage *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
u64 bpf_cookie;
};
};
struct bpf_prog_array {
struct rcu_head rcu;
struct bpf_prog_array_item items[];
};
/* to avoid allocating empty bpf_prog_array for cgroups that
* don't have bpf program attached use one global 'bpf_empty_prog_array'
* It will not be modified the caller of bpf_prog_array_alloc()
* (since caller requested prog_cnt == 0)
* that pointer should be 'freed' by bpf_prog_array_free()
*/
extern struct bpf_prog_array bpf_empty_prog_array;
struct bpf_prog_array *bpf_prog_array_alloc(u32 prog_cnt, gfp_t flags);
void bpf_prog_array_free(struct bpf_prog_array *progs);
/* Use when traversal over the bpf_prog_array uses tasks_trace rcu */
void bpf_prog_array_free_sleepable(struct bpf_prog_array *progs);
int bpf_prog_array_length(struct bpf_prog_array *progs);
bool bpf_prog_array_is_empty(struct bpf_prog_array *array);
int bpf_prog_array_copy_to_user(struct bpf_prog_array *progs,
__u32 __user *prog_ids, u32 cnt);
void bpf_prog_array_delete_safe(struct bpf_prog_array *progs,
struct bpf_prog *old_prog);
int bpf_prog_array_delete_safe_at(struct bpf_prog_array *array, int index);
int bpf_prog_array_update_at(struct bpf_prog_array *array, int index,
struct bpf_prog *prog);
int bpf_prog_array_copy_info(struct bpf_prog_array *array,
u32 *prog_ids, u32 request_cnt,
u32 *prog_cnt);
int bpf_prog_array_copy(struct bpf_prog_array *old_array,
struct bpf_prog *exclude_prog,
struct bpf_prog *include_prog,
u64 bpf_cookie,
struct bpf_prog_array **new_array);
struct bpf_run_ctx {};
struct bpf_cg_run_ctx {
struct bpf_run_ctx run_ctx;
const struct bpf_prog_array_item *prog_item;
int retval;
};
struct bpf_trace_run_ctx {
struct bpf_run_ctx run_ctx;
u64 bpf_cookie;
bool is_uprobe;
};
struct bpf_tramp_run_ctx {
struct bpf_run_ctx run_ctx;
u64 bpf_cookie;
struct bpf_run_ctx *saved_run_ctx;
};
static inline struct bpf_run_ctx *bpf_set_run_ctx(struct bpf_run_ctx *new_ctx)
{
struct bpf_run_ctx *old_ctx = NULL;
#ifdef CONFIG_BPF_SYSCALL
old_ctx = current->bpf_ctx;
current->bpf_ctx = new_ctx;
#endif
return old_ctx;
}
static inline void bpf_reset_run_ctx(struct bpf_run_ctx *old_ctx)
{
#ifdef CONFIG_BPF_SYSCALL
current->bpf_ctx = old_ctx;
#endif
}
/* BPF program asks to bypass CAP_NET_BIND_SERVICE in bind. */
#define BPF_RET_BIND_NO_CAP_NET_BIND_SERVICE (1 << 0)
/* BPF program asks to set CN on the packet. */
#define BPF_RET_SET_CN (1 << 0)
typedef u32 (*bpf_prog_run_fn)(const struct bpf_prog *prog, const void *ctx);
static __always_inline u32
bpf_prog_run_array(const struct bpf_prog_array *array,
const void *ctx, bpf_prog_run_fn run_prog)
{
const struct bpf_prog_array_item *item;
const struct bpf_prog *prog;
struct bpf_run_ctx *old_run_ctx;
struct bpf_trace_run_ctx run_ctx;
u32 ret = 1;
RCU_LOCKDEP_WARN(!rcu_read_lock_held(), "no rcu lock held");
if (unlikely(!array))
return ret;
run_ctx.is_uprobe = false;
migrate_disable();
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
item = &array->items[0];
while ((prog = READ_ONCE(item->prog))) {
run_ctx.bpf_cookie = item->bpf_cookie;
ret &= run_prog(prog, ctx);
item++;
}
bpf_reset_run_ctx(old_run_ctx);
migrate_enable();
return ret;
}
/* Notes on RCU design for bpf_prog_arrays containing sleepable programs:
*
* We use the tasks_trace rcu flavor read section to protect the bpf_prog_array
* overall. As a result, we must use the bpf_prog_array_free_sleepable
* in order to use the tasks_trace rcu grace period.
*
* When a non-sleepable program is inside the array, we take the rcu read
* section and disable preemption for that program alone, so it can access
* rcu-protected dynamically sized maps.
*/
static __always_inline u32
bpf_prog_run_array_uprobe(const struct bpf_prog_array *array,
const void *ctx, bpf_prog_run_fn run_prog)
{
const struct bpf_prog_array_item *item;
const struct bpf_prog *prog;
struct bpf_run_ctx *old_run_ctx;
struct bpf_trace_run_ctx run_ctx;
u32 ret = 1;
might_fault();
RCU_LOCKDEP_WARN(!rcu_read_lock_trace_held(), "no rcu lock held");
if (unlikely(!array))
return ret;
migrate_disable();
run_ctx.is_uprobe = true;
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
item = &array->items[0];
while ((prog = READ_ONCE(item->prog))) {
if (!prog->sleepable)
rcu_read_lock();
run_ctx.bpf_cookie = item->bpf_cookie;
ret &= run_prog(prog, ctx);
item++;
if (!prog->sleepable)
rcu_read_unlock();
}
bpf_reset_run_ctx(old_run_ctx);
migrate_enable();
return ret;
}
bool bpf_jit_bypass_spec_v1(void);
bool bpf_jit_bypass_spec_v4(void);
#define bpf_rcu_lock_held() \
(rcu_read_lock_held() || rcu_read_lock_trace_held() || rcu_read_lock_bh_held())
#ifdef CONFIG_BPF_SYSCALL
DECLARE_PER_CPU(int, bpf_prog_active);
extern struct mutex bpf_stats_enabled_mutex;
/*
* Block execution of BPF programs attached to instrumentation (perf,
* kprobes, tracepoints) to prevent deadlocks on map operations as any of
* these events can happen inside a region which holds a map bucket lock
* and can deadlock on it.
*/
static inline void bpf_disable_instrumentation(void)
{
migrate_disable();
this_cpu_inc(bpf_prog_active);
}
static inline void bpf_enable_instrumentation(void)
{
this_cpu_dec(bpf_prog_active);
migrate_enable();
}
extern const struct super_operations bpf_super_ops;
extern const struct file_operations bpf_map_fops;
extern const struct file_operations bpf_prog_fops;
extern const struct file_operations bpf_iter_fops;
extern const struct file_operations bpf_token_fops;
#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
extern const struct bpf_prog_ops _name ## _prog_ops; \
extern const struct bpf_verifier_ops _name ## _verifier_ops;
#define BPF_MAP_TYPE(_id, _ops) \
extern const struct bpf_map_ops _ops;
#define BPF_LINK_TYPE(_id, _name)
#include <linux/bpf_types.h>
#undef BPF_PROG_TYPE
#undef BPF_MAP_TYPE
#undef BPF_LINK_TYPE
extern const struct bpf_prog_ops bpf_offload_prog_ops;
extern const struct bpf_verifier_ops tc_cls_act_analyzer_ops;
extern const struct bpf_verifier_ops xdp_analyzer_ops;
struct bpf_prog *bpf_prog_get(u32 ufd);
struct bpf_prog *bpf_prog_get_type_dev(u32 ufd, enum bpf_prog_type type,
bool attach_drv);
void bpf_prog_add(struct bpf_prog *prog, int i);
void bpf_prog_sub(struct bpf_prog *prog, int i);
void bpf_prog_inc(struct bpf_prog *prog);
struct bpf_prog * __must_check bpf_prog_inc_not_zero(struct bpf_prog *prog);
void bpf_prog_put(struct bpf_prog *prog);
void bpf_prog_free_id(struct bpf_prog *prog);
void bpf_map_free_id(struct bpf_map *map);
struct btf_field *btf_record_find(const struct btf_record *rec,
u32 offset, u32 field_mask);
void btf_record_free(struct btf_record *rec);
void bpf_map_free_record(struct bpf_map *map);
struct btf_record *btf_record_dup(const struct btf_record *rec);
bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *rec_b);
void bpf_obj_free_timer(const struct btf_record *rec, void *obj);
void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj);
void bpf_obj_free_task_work(const struct btf_record *rec, void *obj);
void bpf_obj_free_fields(const struct btf_record *rec, void *obj);
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu);
struct bpf_map *bpf_map_get(u32 ufd);
struct bpf_map *bpf_map_get_with_uref(u32 ufd);
/*
* The __bpf_map_get() and __btf_get_by_fd() functions parse a file
* descriptor and return a corresponding map or btf object.
* Their names are double underscored to emphasize the fact that they
* do not increase refcnt. To also increase refcnt use corresponding
* bpf_map_get() and btf_get_by_fd() functions.
*/
static inline struct bpf_map *__bpf_map_get(struct fd f)
{
if (fd_empty(f))
return ERR_PTR(-EBADF);
if (unlikely(fd_file(f)->f_op != &bpf_map_fops))
return ERR_PTR(-EINVAL);
return fd_file(f)->private_data;
}
static inline struct btf *__btf_get_by_fd(struct fd f)
{
if (fd_empty(f))
return ERR_PTR(-EBADF);
if (unlikely(fd_file(f)->f_op != &btf_fops))
return ERR_PTR(-EINVAL);
return fd_file(f)->private_data;
}
void bpf_map_inc(struct bpf_map *map);
void bpf_map_inc_with_uref(struct bpf_map *map);
struct bpf_map *__bpf_map_inc_not_zero(struct bpf_map *map, bool uref);
struct bpf_map * __must_check bpf_map_inc_not_zero(struct bpf_map *map);
void bpf_map_put_with_uref(struct bpf_map *map);
void bpf_map_put(struct bpf_map *map);
void *bpf_map_area_alloc(u64 size, int numa_node);
void *bpf_map_area_mmapable_alloc(u64 size, int numa_node);
void bpf_map_area_free(void *base);
bool bpf_map_write_active(const struct bpf_map *map);
void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr);
int generic_map_lookup_batch(struct bpf_map *map,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
int generic_map_update_batch(struct bpf_map *map, struct file *map_file,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
int generic_map_delete_batch(struct bpf_map *map,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
struct bpf_map *bpf_map_get_curr_or_next(u32 *id);
struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id);
int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
unsigned long nr_pages, struct page **page_array);
#ifdef CONFIG_MEMCG
void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg,
struct mem_cgroup **new_memcg);
void bpf_map_memcg_exit(struct mem_cgroup *old_memcg,
struct mem_cgroup *memcg);
void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags,
int node);
void *bpf_map_kmalloc_nolock(const struct bpf_map *map, size_t size, gfp_t flags,
int node);
void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags);
void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size,
gfp_t flags);
void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size,
size_t align, gfp_t flags);
#else
/*
* These specialized allocators have to be macros for their allocations to be
* accounted separately (to have separate alloc_tag).
*/
#define bpf_map_kmalloc_node(_map, _size, _flags, _node) \
kmalloc_node(_size, _flags, _node)
#define bpf_map_kmalloc_nolock(_map, _size, _flags, _node) \
kmalloc_nolock(_size, _flags, _node)
#define bpf_map_kzalloc(_map, _size, _flags) \
kzalloc(_size, _flags)
#define bpf_map_kvcalloc(_map, _n, _size, _flags) \
kvcalloc(_n, _size, _flags)
#define bpf_map_alloc_percpu(_map, _size, _align, _flags) \
__alloc_percpu_gfp(_size, _align, _flags)
static inline void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg,
struct mem_cgroup **new_memcg)
{
*new_memcg = NULL;
*old_memcg = NULL;
}
static inline void bpf_map_memcg_exit(struct mem_cgroup *old_memcg,
struct mem_cgroup *memcg)
{
}
#endif
static inline int
bpf_map_init_elem_count(struct bpf_map *map)
{
size_t size = sizeof(*map->elem_count), align = size;
gfp_t flags = GFP_USER | __GFP_NOWARN;
map->elem_count = bpf_map_alloc_percpu(map, size, align, flags);
if (!map->elem_count)
return -ENOMEM;
return 0;
}
static inline void
bpf_map_free_elem_count(struct bpf_map *map)
{
free_percpu(map->elem_count);
}
static inline void bpf_map_inc_elem_count(struct bpf_map *map)
{
this_cpu_inc(*map->elem_count);
}
static inline void bpf_map_dec_elem_count(struct bpf_map *map)
{
this_cpu_dec(*map->elem_count);
}
extern int sysctl_unprivileged_bpf_disabled;
bool bpf_token_capable(const struct bpf_token *token, int cap);
static inline bool bpf_allow_ptr_leaks(const struct bpf_token *token)
{
return bpf_token_capable(token, CAP_PERFMON);
}
static inline bool bpf_allow_uninit_stack(const struct bpf_token *token)
{
return bpf_token_capable(token, CAP_PERFMON);
}
static inline bool bpf_bypass_spec_v1(const struct bpf_token *token)
{
return bpf_jit_bypass_spec_v1() ||
cpu_mitigations_off() ||
bpf_token_capable(token, CAP_PERFMON);
}
static inline bool bpf_bypass_spec_v4(const struct bpf_token *token)
{
return bpf_jit_bypass_spec_v4() ||
cpu_mitigations_off() ||
bpf_token_capable(token, CAP_PERFMON);
}
int bpf_map_new_fd(struct bpf_map *map, int flags);
int bpf_prog_new_fd(struct bpf_prog *prog);
void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type);
void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, bool sleepable);
int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer);
int bpf_link_settle(struct bpf_link_primer *primer);
void bpf_link_cleanup(struct bpf_link_primer *primer);
void bpf_link_inc(struct bpf_link *link);
struct bpf_link *bpf_link_inc_not_zero(struct bpf_link *link);
void bpf_link_put(struct bpf_link *link);
int bpf_link_new_fd(struct bpf_link *link);
struct bpf_link *bpf_link_get_from_fd(u32 ufd);
struct bpf_link *bpf_link_get_curr_or_next(u32 *id);
void bpf_token_inc(struct bpf_token *token);
void bpf_token_put(struct bpf_token *token);
int bpf_token_create(union bpf_attr *attr);
struct bpf_token *bpf_token_get_from_fd(u32 ufd);
int bpf_token_get_info_by_fd(struct bpf_token *token,
const union bpf_attr *attr,
union bpf_attr __user *uattr);
bool bpf_token_allow_cmd(const struct bpf_token *token, enum bpf_cmd cmd);
bool bpf_token_allow_map_type(const struct bpf_token *token, enum bpf_map_type type);
bool bpf_token_allow_prog_type(const struct bpf_token *token,
enum bpf_prog_type prog_type,
enum bpf_attach_type attach_type);
int bpf_obj_pin_user(u32 ufd, int path_fd, const char __user *pathname);
int bpf_obj_get_user(int path_fd, const char __user *pathname, int flags);
struct inode *bpf_get_inode(struct super_block *sb, const struct inode *dir,
umode_t mode);
#define BPF_ITER_FUNC_PREFIX "bpf_iter_"
#define DEFINE_BPF_ITER_FUNC(target, args...) \
extern int bpf_iter_ ## target(args); \
int __init bpf_iter_ ## target(args) { return 0; }
/*
* The task type of iterators.
*
* For BPF task iterators, they can be parameterized with various
* parameters to visit only some of tasks.
*
* BPF_TASK_ITER_ALL (default)
* Iterate over resources of every task.
*
* BPF_TASK_ITER_TID
* Iterate over resources of a task/tid.
*
* BPF_TASK_ITER_TGID
* Iterate over resources of every task of a process / task group.
*/
enum bpf_iter_task_type {
BPF_TASK_ITER_ALL = 0,
BPF_TASK_ITER_TID,
BPF_TASK_ITER_TGID,
};
struct bpf_iter_aux_info {
/* for map_elem iter */
struct bpf_map *map;
/* for cgroup iter */
struct {
struct cgroup *start; /* starting cgroup */
enum bpf_cgroup_iter_order order;
} cgroup;
struct {
enum bpf_iter_task_type type;
u32 pid;
} task;
};
typedef int (*bpf_iter_attach_target_t)(struct bpf_prog *prog,
union bpf_iter_link_info *linfo,
struct bpf_iter_aux_info *aux);
typedef void (*bpf_iter_detach_target_t)(struct bpf_iter_aux_info *aux);
typedef void (*bpf_iter_show_fdinfo_t) (const struct bpf_iter_aux_info *aux,
struct seq_file *seq);
typedef int (*bpf_iter_fill_link_info_t)(const struct bpf_iter_aux_info *aux,
struct bpf_link_info *info);
typedef const struct bpf_func_proto *
(*bpf_iter_get_func_proto_t)(enum bpf_func_id func_id,
const struct bpf_prog *prog);
enum bpf_iter_feature {
BPF_ITER_RESCHED = BIT(0),
};
#define BPF_ITER_CTX_ARG_MAX 2
struct bpf_iter_reg {
const char *target;
bpf_iter_attach_target_t attach_target;
bpf_iter_detach_target_t detach_target;
bpf_iter_show_fdinfo_t show_fdinfo;
bpf_iter_fill_link_info_t fill_link_info;
bpf_iter_get_func_proto_t get_func_proto;
u32 ctx_arg_info_size;
u32 feature;
struct bpf_ctx_arg_aux ctx_arg_info[BPF_ITER_CTX_ARG_MAX];
const struct bpf_iter_seq_info *seq_info;
};
struct bpf_iter_meta {
__bpf_md_ptr(struct seq_file *, seq);
u64 session_id;
u64 seq_num;
};
struct bpf_iter__bpf_map_elem {
__bpf_md_ptr(struct bpf_iter_meta *, meta);
__bpf_md_ptr(struct bpf_map *, map);
__bpf_md_ptr(void *, key);
__bpf_md_ptr(void *, value);
};
int bpf_iter_reg_target(const struct bpf_iter_reg *reg_info);
void bpf_iter_unreg_target(const struct bpf_iter_reg *reg_info);
int bpf_iter_prog_supported(struct bpf_prog *prog);
const struct bpf_func_proto *
bpf_iter_get_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
int bpf_iter_link_attach(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_prog *prog);
int bpf_iter_new_fd(struct bpf_link *link);
bool bpf_link_is_iter(struct bpf_link *link);
struct bpf_prog *bpf_iter_get_info(struct bpf_iter_meta *meta, bool in_stop);
int bpf_iter_run_prog(struct bpf_prog *prog, void *ctx);
void bpf_iter_map_show_fdinfo(const struct bpf_iter_aux_info *aux,
struct seq_file *seq);
int bpf_iter_map_fill_link_info(const struct bpf_iter_aux_info *aux,
struct bpf_link_info *info);
int map_set_for_each_callback_args(struct bpf_verifier_env *env,
struct bpf_func_state *caller,
struct bpf_func_state *callee);
int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value, u64 flags);
int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value, u64 flags);
int bpf_percpu_hash_update(struct bpf_map *map, void *key, void *value,
u64 flags);
int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
u64 flags);
int bpf_stackmap_extract(struct bpf_map *map, void *key, void *value, bool delete);
int bpf_fd_array_map_update_elem(struct bpf_map *map, struct file *map_file,
void *key, void *value, u64 map_flags);
int bpf_fd_array_map_lookup_elem(struct bpf_map *map, void *key, u32 *value);
int bpf_fd_htab_map_update_elem(struct bpf_map *map, struct file *map_file,
void *key, void *value, u64 map_flags);
int bpf_fd_htab_map_lookup_elem(struct bpf_map *map, void *key, u32 *value);
int bpf_get_file_flag(int flags);
int bpf_check_uarg_tail_zero(bpfptr_t uaddr, size_t expected_size,
size_t actual_size);
/* verify correctness of eBPF program */
int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size);
#ifndef CONFIG_BPF_JIT_ALWAYS_ON
void bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth);
#endif
struct btf *bpf_get_btf_vmlinux(void);
/* Map specifics */
struct xdp_frame;
struct sk_buff;
struct bpf_dtab_netdev;
struct bpf_cpu_map_entry;
void __dev_flush(struct list_head *flush_list);
int dev_xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf,
struct net_device *dev_rx);
int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_frame *xdpf,
struct net_device *dev_rx);
int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx,
struct bpf_map *map, bool exclude_ingress);
int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb,
const struct bpf_prog *xdp_prog);
int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb,
const struct bpf_prog *xdp_prog,
struct bpf_map *map, bool exclude_ingress);
void __cpu_map_flush(struct list_head *flush_list);
int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf,
struct net_device *dev_rx);
int cpu_map_generic_redirect(struct bpf_cpu_map_entry *rcpu,
struct sk_buff *skb);
/* Return map's numa specified by userspace */
static inline int bpf_map_attr_numa_node(const union bpf_attr *attr)
{
return (attr->map_flags & BPF_F_NUMA_NODE) ?
attr->numa_node : NUMA_NO_NODE;
}
struct bpf_prog *bpf_prog_get_type_path(const char *name, enum bpf_prog_type type);
int array_map_alloc_check(union bpf_attr *attr);
int bpf_prog_test_run_xdp(struct bpf_prog *prog, const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_tracing(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_raw_tp(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_sk_lookup(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int bpf_prog_test_run_nf(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
bool btf_ctx_access(int off, int size, enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info);
static inline bool bpf_tracing_ctx_access(int off, int size,
enum bpf_access_type type)
{
if (off < 0 || off >= sizeof(__u64) * MAX_BPF_FUNC_ARGS)
return false;
if (type != BPF_READ)
return false;
if (off % size != 0)
return false;
return true;
}
static inline bool bpf_tracing_btf_ctx_access(int off, int size,
enum bpf_access_type type,
const struct bpf_prog *prog,
struct bpf_insn_access_aux *info)
{
if (!bpf_tracing_ctx_access(off, size, type))
return false;
return btf_ctx_access(off, size, type, prog, info);
}
int btf_struct_access(struct bpf_verifier_log *log,
const struct bpf_reg_state *reg,
int off, int size, enum bpf_access_type atype,
u32 *next_btf_id, enum bpf_type_flag *flag, const char **field_name);
bool btf_struct_ids_match(struct bpf_verifier_log *log,
const struct btf *btf, u32 id, int off,
const struct btf *need_btf, u32 need_type_id,
bool strict);
int btf_distill_func_proto(struct bpf_verifier_log *log,
struct btf *btf,
const struct btf_type *func_proto,
const char *func_name,
struct btf_func_model *m);
struct bpf_reg_state;
int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog);
int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog,
struct btf *btf, const struct btf_type *t);
const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt,
int comp_idx, const char *tag_key);
int btf_find_next_decl_tag(const struct btf *btf, const struct btf_type *pt,
int comp_idx, const char *tag_key, int last_id);
struct bpf_prog *bpf_prog_by_id(u32 id);
struct bpf_link *bpf_link_by_id(u32 id);
const struct bpf_func_proto *bpf_base_func_proto(enum bpf_func_id func_id,
const struct bpf_prog *prog);
void bpf_task_storage_free(struct task_struct *task);
void bpf_cgrp_storage_free(struct cgroup *cgroup);
bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog);
const struct btf_func_model *
bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
const struct bpf_insn *insn);
int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
u16 btf_fd_idx, u8 **func_addr);
struct bpf_core_ctx {
struct bpf_verifier_log *log;
const struct btf *btf;
};
bool btf_nested_type_is_trusted(struct bpf_verifier_log *log,
const struct bpf_reg_state *reg,
const char *field_name, u32 btf_id, const char *suffix);
bool btf_type_ids_nocast_alias(struct bpf_verifier_log *log,
const struct btf *reg_btf, u32 reg_id,
const struct btf *arg_btf, u32 arg_id);
int bpf_core_apply(struct bpf_core_ctx *ctx, const struct bpf_core_relo *relo,
int relo_idx, void *insn);
static inline bool unprivileged_ebpf_enabled(void)
{
return !sysctl_unprivileged_bpf_disabled;
}
/* Not all bpf prog type has the bpf_ctx.
* For the bpf prog type that has initialized the bpf_ctx,
* this function can be used to decide if a kernel function
* is called by a bpf program.
*/
static inline bool has_current_bpf_ctx(void)
{
return !!current->bpf_ctx;
}
void notrace bpf_prog_inc_misses_counter(struct bpf_prog *prog);
void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size);
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr);
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr);
void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip);
#else /* !CONFIG_BPF_SYSCALL */
static inline struct bpf_prog *bpf_prog_get(u32 ufd)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline struct bpf_prog *bpf_prog_get_type_dev(u32 ufd,
enum bpf_prog_type type,
bool attach_drv)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline void bpf_prog_add(struct bpf_prog *prog, int i)
{
}
static inline void bpf_prog_sub(struct bpf_prog *prog, int i)
{
}
static inline void bpf_prog_put(struct bpf_prog *prog)
{
}
static inline void bpf_prog_inc(struct bpf_prog *prog)
{
}
static inline struct bpf_prog *__must_check
bpf_prog_inc_not_zero(struct bpf_prog *prog)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops,
struct bpf_prog *prog, enum bpf_attach_type attach_type)
{
}
static inline void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog,
enum bpf_attach_type attach_type, bool sleepable)
{
}
static inline int bpf_link_prime(struct bpf_link *link,
struct bpf_link_primer *primer)
{
return -EOPNOTSUPP;
}
static inline int bpf_link_settle(struct bpf_link_primer *primer)
{
return -EOPNOTSUPP;
}
static inline void bpf_link_cleanup(struct bpf_link_primer *primer)
{
}
static inline void bpf_link_inc(struct bpf_link *link)
{
}
static inline struct bpf_link *bpf_link_inc_not_zero(struct bpf_link *link)
{
return NULL;
}
static inline void bpf_link_put(struct bpf_link *link)
{
}
static inline int bpf_obj_get_user(const char __user *pathname, int flags)
{
return -EOPNOTSUPP;
}
static inline bool bpf_token_capable(const struct bpf_token *token, int cap)
{
return capable(cap) || (cap != CAP_SYS_ADMIN && capable(CAP_SYS_ADMIN));
}
static inline void bpf_token_inc(struct bpf_token *token)
{
}
static inline void bpf_token_put(struct bpf_token *token)
{
}
static inline struct bpf_token *bpf_token_get_from_fd(u32 ufd)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline int bpf_token_get_info_by_fd(struct bpf_token *token,
const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
return -EOPNOTSUPP;
}
static inline void __dev_flush(struct list_head *flush_list)
{
}
struct xdp_frame;
struct bpf_dtab_netdev;
struct bpf_cpu_map_entry;
static inline
int dev_xdp_enqueue(struct net_device *dev, struct xdp_frame *xdpf,
struct net_device *dev_rx)
{
return 0;
}
static inline
int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_frame *xdpf,
struct net_device *dev_rx)
{
return 0;
}
static inline
int dev_map_enqueue_multi(struct xdp_frame *xdpf, struct net_device *dev_rx,
struct bpf_map *map, bool exclude_ingress)
{
return 0;
}
struct sk_buff;
static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst,
struct sk_buff *skb,
const struct bpf_prog *xdp_prog)
{
return 0;
}
static inline
int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb,
const struct bpf_prog *xdp_prog,
struct bpf_map *map, bool exclude_ingress)
{
return 0;
}
static inline void __cpu_map_flush(struct list_head *flush_list)
{
}
static inline int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu,
struct xdp_frame *xdpf,
struct net_device *dev_rx)
{
return 0;
}
static inline int cpu_map_generic_redirect(struct bpf_cpu_map_entry *rcpu,
struct sk_buff *skb)
{
return -EOPNOTSUPP;
}
static inline struct bpf_prog *bpf_prog_get_type_path(const char *name,
enum bpf_prog_type type)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline int bpf_prog_test_run_xdp(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
static inline int bpf_prog_test_run_skb(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
static inline int bpf_prog_test_run_tracing(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
static inline int bpf_prog_test_run_flow_dissector(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
static inline int bpf_prog_test_run_sk_lookup(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
static inline void bpf_map_put(struct bpf_map *map)
{
}
static inline struct bpf_prog *bpf_prog_by_id(u32 id)
{
return ERR_PTR(-ENOTSUPP);
}
static inline int btf_struct_access(struct bpf_verifier_log *log,
const struct bpf_reg_state *reg,
int off, int size, enum bpf_access_type atype,
u32 *next_btf_id, enum bpf_type_flag *flag,
const char **field_name)
{
return -EACCES;
}
static inline const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
return NULL;
}
static inline void bpf_task_storage_free(struct task_struct *task)
{
}
static inline bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog)
{
return false;
}
static inline const struct btf_func_model *
bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
const struct bpf_insn *insn)
{
return NULL;
}
static inline int
bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
u16 btf_fd_idx, u8 **func_addr)
{
return -ENOTSUPP;
}
static inline bool unprivileged_ebpf_enabled(void)
{
return false;
}
static inline bool has_current_bpf_ctx(void)
{
return false;
}
static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)
{
}
static inline void bpf_cgrp_storage_free(struct cgroup *cgroup)
{
}
static inline void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size)
{
}
static inline void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)
{
}
static inline void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
{
}
static inline void bpf_prog_report_arena_violation(bool write, unsigned long addr,
unsigned long fault_ip)
{
}
#endif /* CONFIG_BPF_SYSCALL */
static inline bool bpf_net_capable(void)
{
return capable(CAP_NET_ADMIN) || capable(CAP_SYS_ADMIN);
}
static __always_inline int
bpf_probe_read_kernel_common(void *dst, u32 size, const void *unsafe_ptr)
{
int ret = -EFAULT;
if (IS_ENABLED(CONFIG_BPF_EVENTS))
ret = copy_from_kernel_nofault(dst, unsafe_ptr, size);
if (unlikely(ret < 0))
memset(dst, 0, size);
return ret;
}
void __bpf_free_used_btfs(struct btf_mod_pair *used_btfs, u32 len);
static inline struct bpf_prog *bpf_prog_get_type(u32 ufd,
enum bpf_prog_type type)
{
return bpf_prog_get_type_dev(ufd, type, false);
}
void __bpf_free_used_maps(struct bpf_prog_aux *aux,
struct bpf_map **used_maps, u32 len);
bool bpf_prog_get_ok(struct bpf_prog *, enum bpf_prog_type *, bool);
int bpf_prog_offload_compile(struct bpf_prog *prog);
void bpf_prog_dev_bound_destroy(struct bpf_prog *prog);
int bpf_prog_offload_info_fill(struct bpf_prog_info *info,
struct bpf_prog *prog);
int bpf_map_offload_info_fill(struct bpf_map_info *info, struct bpf_map *map);
int bpf_map_offload_lookup_elem(struct bpf_map *map, void *key, void *value);
int bpf_map_offload_update_elem(struct bpf_map *map,
void *key, void *value, u64 flags);
int bpf_map_offload_delete_elem(struct bpf_map *map, void *key);
int bpf_map_offload_get_next_key(struct bpf_map *map,
void *key, void *next_key);
bool bpf_offload_prog_map_match(struct bpf_prog *prog, struct bpf_map *map);
struct bpf_offload_dev *
bpf_offload_dev_create(const struct bpf_prog_offload_ops *ops, void *priv);
void bpf_offload_dev_destroy(struct bpf_offload_dev *offdev);
void *bpf_offload_dev_priv(struct bpf_offload_dev *offdev);
int bpf_offload_dev_netdev_register(struct bpf_offload_dev *offdev,
struct net_device *netdev);
void bpf_offload_dev_netdev_unregister(struct bpf_offload_dev *offdev,
struct net_device *netdev);
bool bpf_offload_dev_match(struct bpf_prog *prog, struct net_device *netdev);
void unpriv_ebpf_notify(int new_state);
#if defined(CONFIG_NET) && defined(CONFIG_BPF_SYSCALL)
int bpf_dev_bound_kfunc_check(struct bpf_verifier_log *log,
struct bpf_prog_aux *prog_aux);
void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id);
int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr);
int bpf_prog_dev_bound_inherit(struct bpf_prog *new_prog, struct bpf_prog *old_prog);
void bpf_dev_bound_netdev_unregister(struct net_device *dev);
static inline bool bpf_prog_is_dev_bound(const struct bpf_prog_aux *aux)
{
return aux->dev_bound;
}
static inline bool bpf_prog_is_offloaded(const struct bpf_prog_aux *aux)
{
return aux->offload_requested;
}
bool bpf_prog_dev_bound_match(const struct bpf_prog *lhs, const struct bpf_prog *rhs);
static inline bool bpf_map_is_offloaded(struct bpf_map *map)
{
return unlikely(map->ops == &bpf_map_offload_ops);
}
struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr);
void bpf_map_offload_map_free(struct bpf_map *map);
u64 bpf_map_offload_map_mem_usage(const struct bpf_map *map);
int bpf_prog_test_run_syscall(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr);
int sock_map_get_from_fd(const union bpf_attr *attr, struct bpf_prog *prog);
int sock_map_prog_detach(const union bpf_attr *attr, enum bpf_prog_type ptype);
int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value, u64 flags);
int sock_map_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr);
int sock_map_link_create(const union bpf_attr *attr, struct bpf_prog *prog);
void sock_map_unhash(struct sock *sk);
void sock_map_destroy(struct sock *sk);
void sock_map_close(struct sock *sk, long timeout);
#else
static inline int bpf_dev_bound_kfunc_check(struct bpf_verifier_log *log,
struct bpf_prog_aux *prog_aux)
{
return -EOPNOTSUPP;
}
static inline void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog,
u32 func_id)
{
return NULL;
}
static inline int bpf_prog_dev_bound_init(struct bpf_prog *prog,
union bpf_attr *attr)
{
return -EOPNOTSUPP;
}
static inline int bpf_prog_dev_bound_inherit(struct bpf_prog *new_prog,
struct bpf_prog *old_prog)
{
return -EOPNOTSUPP;
}
static inline void bpf_dev_bound_netdev_unregister(struct net_device *dev)
{
}
static inline bool bpf_prog_is_dev_bound(const struct bpf_prog_aux *aux)
{
return false;
}
static inline bool bpf_prog_is_offloaded(struct bpf_prog_aux *aux)
{
return false;
}
static inline bool bpf_prog_dev_bound_match(const struct bpf_prog *lhs, const struct bpf_prog *rhs)
{
return false;
}
static inline bool bpf_map_is_offloaded(struct bpf_map *map)
{
return false;
}
static inline struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline void bpf_map_offload_map_free(struct bpf_map *map)
{
}
static inline u64 bpf_map_offload_map_mem_usage(const struct bpf_map *map)
{
return 0;
}
static inline int bpf_prog_test_run_syscall(struct bpf_prog *prog,
const union bpf_attr *kattr,
union bpf_attr __user *uattr)
{
return -ENOTSUPP;
}
#ifdef CONFIG_BPF_SYSCALL
static inline int sock_map_get_from_fd(const union bpf_attr *attr,
struct bpf_prog *prog)
{
return -EINVAL;
}
static inline int sock_map_prog_detach(const union bpf_attr *attr,
enum bpf_prog_type ptype)
{
return -EOPNOTSUPP;
}
static inline int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value,
u64 flags)
{
return -EOPNOTSUPP;
}
static inline int sock_map_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr)
{
return -EINVAL;
}
static inline int sock_map_link_create(const union bpf_attr *attr, struct bpf_prog *prog)
{
return -EOPNOTSUPP;
}
#endif /* CONFIG_BPF_SYSCALL */
#endif /* CONFIG_NET && CONFIG_BPF_SYSCALL */
static __always_inline void
bpf_prog_inc_misses_counters(const struct bpf_prog_array *array)
{
const struct bpf_prog_array_item *item;
struct bpf_prog *prog;
if (unlikely(!array))
return;
item = &array->items[0];
while ((prog = READ_ONCE(item->prog))) {
bpf_prog_inc_misses_counter(prog);
item++;
}
}
#if defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL)
void bpf_sk_reuseport_detach(struct sock *sk);
int bpf_fd_reuseport_array_lookup_elem(struct bpf_map *map, void *key,
void *value);
int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, void *key,
void *value, u64 map_flags);
#else
static inline void bpf_sk_reuseport_detach(struct sock *sk)
{
}
#ifdef CONFIG_BPF_SYSCALL
static inline int bpf_fd_reuseport_array_lookup_elem(struct bpf_map *map,
void *key, void *value)
{
return -EOPNOTSUPP;
}
static inline int bpf_fd_reuseport_array_update_elem(struct bpf_map *map,
void *key, void *value,
u64 map_flags)
{
return -EOPNOTSUPP;
}
#endif /* CONFIG_BPF_SYSCALL */
#endif /* defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL) */
#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL)
struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags);
struct bpf_key *bpf_lookup_system_key(u64 id);
void bpf_key_put(struct bpf_key *bkey);
int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring);
#else
static inline struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags)
{
return NULL;
}
static inline struct bpf_key *bpf_lookup_system_key(u64 id)
{
return NULL;
}
static inline void bpf_key_put(struct bpf_key *bkey)
{
}
static inline int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
return -EOPNOTSUPP;
}
#endif /* defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) */
/* verifier prototypes for helper functions called from eBPF programs */
extern const struct bpf_func_proto bpf_map_lookup_elem_proto;
extern const struct bpf_func_proto bpf_map_update_elem_proto;
extern const struct bpf_func_proto bpf_map_delete_elem_proto;
extern const struct bpf_func_proto bpf_map_push_elem_proto;
extern const struct bpf_func_proto bpf_map_pop_elem_proto;
extern const struct bpf_func_proto bpf_map_peek_elem_proto;
extern const struct bpf_func_proto bpf_map_lookup_percpu_elem_proto;
extern const struct bpf_func_proto bpf_get_prandom_u32_proto;
extern const struct bpf_func_proto bpf_get_smp_processor_id_proto;
extern const struct bpf_func_proto bpf_get_numa_node_id_proto;
extern const struct bpf_func_proto bpf_tail_call_proto;
extern const struct bpf_func_proto bpf_ktime_get_ns_proto;
extern const struct bpf_func_proto bpf_ktime_get_boot_ns_proto;
extern const struct bpf_func_proto bpf_ktime_get_tai_ns_proto;
extern const struct bpf_func_proto bpf_get_current_pid_tgid_proto;
extern const struct bpf_func_proto bpf_get_current_uid_gid_proto;
extern const struct bpf_func_proto bpf_get_current_comm_proto;
extern const struct bpf_func_proto bpf_get_stackid_proto;
extern const struct bpf_func_proto bpf_get_stack_proto;
extern const struct bpf_func_proto bpf_get_stack_sleepable_proto;
extern const struct bpf_func_proto bpf_get_task_stack_proto;
extern const struct bpf_func_proto bpf_get_task_stack_sleepable_proto;
extern const struct bpf_func_proto bpf_get_stackid_proto_pe;
extern const struct bpf_func_proto bpf_get_stack_proto_pe;
extern const struct bpf_func_proto bpf_sock_map_update_proto;
extern const struct bpf_func_proto bpf_sock_hash_update_proto;
extern const struct bpf_func_proto bpf_get_current_cgroup_id_proto;
extern const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto;
extern const struct bpf_func_proto bpf_get_cgroup_classid_curr_proto;
extern const struct bpf_func_proto bpf_current_task_under_cgroup_proto;
extern const struct bpf_func_proto bpf_msg_redirect_hash_proto;
extern const struct bpf_func_proto bpf_msg_redirect_map_proto;
extern const struct bpf_func_proto bpf_sk_redirect_hash_proto;
extern const struct bpf_func_proto bpf_sk_redirect_map_proto;
extern const struct bpf_func_proto bpf_spin_lock_proto;
extern const struct bpf_func_proto bpf_spin_unlock_proto;
extern const struct bpf_func_proto bpf_get_local_storage_proto;
extern const struct bpf_func_proto bpf_strtol_proto;
extern const struct bpf_func_proto bpf_strtoul_proto;
extern const struct bpf_func_proto bpf_tcp_sock_proto;
extern const struct bpf_func_proto bpf_jiffies64_proto;
extern const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto;
extern const struct bpf_func_proto bpf_event_output_data_proto;
extern const struct bpf_func_proto bpf_ringbuf_output_proto;
extern const struct bpf_func_proto bpf_ringbuf_reserve_proto;
extern const struct bpf_func_proto bpf_ringbuf_submit_proto;
extern const struct bpf_func_proto bpf_ringbuf_discard_proto;
extern const struct bpf_func_proto bpf_ringbuf_query_proto;
extern const struct bpf_func_proto bpf_ringbuf_reserve_dynptr_proto;
extern const struct bpf_func_proto bpf_ringbuf_submit_dynptr_proto;
extern const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto;
extern const struct bpf_func_proto bpf_skc_to_tcp6_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_tcp_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_tcp_timewait_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_tcp_request_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_udp6_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_unix_sock_proto;
extern const struct bpf_func_proto bpf_skc_to_mptcp_sock_proto;
extern const struct bpf_func_proto bpf_copy_from_user_proto;
extern const struct bpf_func_proto bpf_snprintf_btf_proto;
extern const struct bpf_func_proto bpf_snprintf_proto;
extern const struct bpf_func_proto bpf_per_cpu_ptr_proto;
extern const struct bpf_func_proto bpf_this_cpu_ptr_proto;
extern const struct bpf_func_proto bpf_ktime_get_coarse_ns_proto;
extern const struct bpf_func_proto bpf_sock_from_file_proto;
extern const struct bpf_func_proto bpf_get_socket_ptr_cookie_proto;
extern const struct bpf_func_proto bpf_task_storage_get_recur_proto;
extern const struct bpf_func_proto bpf_task_storage_get_proto;
extern const struct bpf_func_proto bpf_task_storage_delete_recur_proto;
extern const struct bpf_func_proto bpf_task_storage_delete_proto;
extern const struct bpf_func_proto bpf_for_each_map_elem_proto;
extern const struct bpf_func_proto bpf_btf_find_by_name_kind_proto;
extern const struct bpf_func_proto bpf_sk_setsockopt_proto;
extern const struct bpf_func_proto bpf_sk_getsockopt_proto;
extern const struct bpf_func_proto bpf_sk_setsockopt_nodelay_proto;
extern const struct bpf_func_proto bpf_unlocked_sk_setsockopt_proto;
extern const struct bpf_func_proto bpf_unlocked_sk_getsockopt_proto;
extern const struct bpf_func_proto bpf_find_vma_proto;
extern const struct bpf_func_proto bpf_loop_proto;
extern const struct bpf_func_proto bpf_copy_from_user_task_proto;
extern const struct bpf_func_proto bpf_set_retval_proto;
extern const struct bpf_func_proto bpf_get_retval_proto;
extern const struct bpf_func_proto bpf_user_ringbuf_drain_proto;
extern const struct bpf_func_proto bpf_cgrp_storage_get_proto;
extern const struct bpf_func_proto bpf_cgrp_storage_delete_proto;
const struct bpf_func_proto *tracing_prog_func_proto(
enum bpf_func_id func_id, const struct bpf_prog *prog);
/* Shared helpers among cBPF and eBPF. */
void bpf_user_rnd_init_once(void);
u64 bpf_user_rnd_u32(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
#if defined(CONFIG_NET)
bool bpf_sock_common_is_valid_access(int off, int size,
enum bpf_access_type type,
struct bpf_insn_access_aux *info);
bool bpf_sock_is_valid_access(int off, int size, enum bpf_access_type type,
struct bpf_insn_access_aux *info);
u32 bpf_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size);
int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags,
struct bpf_dynptr *ptr);
#else
static inline bool bpf_sock_common_is_valid_access(int off, int size,
enum bpf_access_type type,
struct bpf_insn_access_aux *info)
{
return false;
}
static inline bool bpf_sock_is_valid_access(int off, int size,
enum bpf_access_type type,
struct bpf_insn_access_aux *info)
{
return false;
}
static inline u32 bpf_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size)
{
return 0;
}
static inline int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags,
struct bpf_dynptr *ptr)
{
return -EOPNOTSUPP;
}
#endif
#ifdef CONFIG_INET
struct sk_reuseport_kern {
struct sk_buff *skb;
struct sock *sk;
struct sock *selected_sk;
struct sock *migrating_sk;
void *data_end;
u32 hash;
u32 reuseport_id;
bool bind_inany;
};
bool bpf_tcp_sock_is_valid_access(int off, int size, enum bpf_access_type type,
struct bpf_insn_access_aux *info);
u32 bpf_tcp_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size);
bool bpf_xdp_sock_is_valid_access(int off, int size, enum bpf_access_type type,
struct bpf_insn_access_aux *info);
u32 bpf_xdp_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size);
#else
static inline bool bpf_tcp_sock_is_valid_access(int off, int size,
enum bpf_access_type type,
struct bpf_insn_access_aux *info)
{
return false;
}
static inline u32 bpf_tcp_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size)
{
return 0;
}
static inline bool bpf_xdp_sock_is_valid_access(int off, int size,
enum bpf_access_type type,
struct bpf_insn_access_aux *info)
{
return false;
}
static inline u32 bpf_xdp_sock_convert_ctx_access(enum bpf_access_type type,
const struct bpf_insn *si,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size)
{
return 0;
}
#endif /* CONFIG_INET */
enum bpf_text_poke_type {
BPF_MOD_NOP,
BPF_MOD_CALL,
BPF_MOD_JUMP,
};
int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
enum bpf_text_poke_type new_t, void *old_addr,
void *new_addr);
void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke,
struct bpf_prog *new, struct bpf_prog *old);
void *bpf_arch_text_copy(void *dst, void *src, size_t len);
int bpf_arch_text_invalidate(void *dst, size_t len);
struct btf_id_set;
bool btf_id_set_contains(const struct btf_id_set *set, u32 id);
#define MAX_BPRINTF_VARARGS 12
#define MAX_BPRINTF_BUF 1024
/* Per-cpu temp buffers used by printf-like helpers to store the bprintf binary
* arguments representation.
*/
#define MAX_BPRINTF_BIN_ARGS 512
struct bpf_bprintf_buffers {
char bin_args[MAX_BPRINTF_BIN_ARGS];
char buf[MAX_BPRINTF_BUF];
};
struct bpf_bprintf_data {
u32 *bin_args;
char *buf;
bool get_bin_args;
bool get_buf;
};
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data);
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs);
void bpf_put_buffers(void);
void bpf_prog_stream_init(struct bpf_prog *prog);
void bpf_prog_stream_free(struct bpf_prog *prog);
int bpf_prog_stream_read(struct bpf_prog *prog, enum bpf_stream_id stream_id, void __user *buf, int len);
void bpf_stream_stage_init(struct bpf_stream_stage *ss);
void bpf_stream_stage_free(struct bpf_stream_stage *ss);
__printf(2, 3)
int bpf_stream_stage_printk(struct bpf_stream_stage *ss, const char *fmt, ...);
int bpf_stream_stage_commit(struct bpf_stream_stage *ss, struct bpf_prog *prog,
enum bpf_stream_id stream_id);
int bpf_stream_stage_dump_stack(struct bpf_stream_stage *ss);
#define bpf_stream_printk(ss, ...) bpf_stream_stage_printk(&ss, __VA_ARGS__)
#define bpf_stream_dump_stack(ss) bpf_stream_stage_dump_stack(&ss)
#define bpf_stream_stage(ss, prog, stream_id, expr) \
({ \
bpf_stream_stage_init(&ss); \
(expr); \
bpf_stream_stage_commit(&ss, prog, stream_id); \
bpf_stream_stage_free(&ss); \
})
#ifdef CONFIG_BPF_LSM
void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype);
void bpf_cgroup_atype_put(int cgroup_atype);
#else
static inline void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) {}
static inline void bpf_cgroup_atype_put(int cgroup_atype) {}
#endif /* CONFIG_BPF_LSM */
struct key;
#ifdef CONFIG_KEYS
struct bpf_key {
struct key *key;
bool has_ref;
};
#endif /* CONFIG_KEYS */
static inline bool type_is_alloc(u32 type)
{
return type & MEM_ALLOC;
}
static inline gfp_t bpf_memcg_flags(gfp_t flags)
{
if (memcg_bpf_enabled())
return flags | __GFP_ACCOUNT;
return flags;
}
static inline bool bpf_is_subprog(const struct bpf_prog *prog)
{
return prog->aux->func_idx != 0;
}
const struct bpf_line_info *bpf_find_linfo(const struct bpf_prog *prog, u32 insn_off);
void bpf_get_linfo_file_line(struct btf *btf, const struct bpf_line_info *linfo,
const char **filep, const char **linep, int *nump);
int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char **filep,
const char **linep, int *nump);
struct bpf_prog *bpf_prog_find_from_stack(void);
int bpf_insn_array_init(struct bpf_map *map, const struct bpf_prog *prog);
int bpf_insn_array_ready(struct bpf_map *map);
void bpf_insn_array_release(struct bpf_map *map);
void bpf_insn_array_adjust(struct bpf_map *map, u32 off, u32 len);
void bpf_insn_array_adjust_after_remove(struct bpf_map *map, u32 off, u32 len);
#ifdef CONFIG_BPF_SYSCALL
void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image);
#else
static inline void
bpf_prog_update_insn_ptrs(struct bpf_prog *prog, u32 *offsets, void *image)
{
}
#endif
static inline bool bpf_map_supports_cpu_flags(enum bpf_map_type map_type)
{
switch (map_type) {
case BPF_MAP_TYPE_PERCPU_ARRAY:
case BPF_MAP_TYPE_PERCPU_HASH:
case BPF_MAP_TYPE_LRU_PERCPU_HASH:
case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE:
return true;
default:
return false;
}
}
static inline int bpf_map_check_op_flags(struct bpf_map *map, u64 flags, u64 allowed_flags)
{
u32 cpu;
if ((u32)flags & ~allowed_flags)
return -EINVAL;
if ((flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK))
return -EINVAL;
if (!(flags & BPF_F_CPU) && flags >> 32)
return -EINVAL;
if (flags & (BPF_F_CPU | BPF_F_ALL_CPUS)) {
if (!bpf_map_supports_cpu_flags(map->map_type))
return -EINVAL;
if ((flags & BPF_F_CPU) && (flags & BPF_F_ALL_CPUS))
return -EINVAL;
cpu = flags >> 32;
if ((flags & BPF_F_CPU) && cpu >= num_possible_cpus())
return -ERANGE;
}
return 0;
}
#endif /* _LINUX_BPF_H */
]
|
| 544/6 |
2026/05/18 19:33 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 242733
output: 28
thoughts: 521
Thoughts: **Analyzing Potential BPF Vulnerabilities**
Okay, so I'm diving into this BPF code, and the first thing that jumps out is the potential buffer overflow in `kernel/bpf/stream.c`. Specifically, I see `MAX_BPRINTF_BUF` is set to 1024. Then, the code calls `bstr_printf` to format data into `data.buf` which is 1024 bytes. The critical part is where `bstr_printf`'s return value (`ret`) is used. If `ret` is 1024 or greater, `bpf_stream_push_str` is called, which in turn calls `memcpy`. Since `data.buf` is only 1024 bytes, and `memcpy`'s `len` argument is the potentially oversized `ret` value from `bstr_printf`, this is definitely a classic out-of-bounds write.
However, the user's focus was on `kernel/bpf/helpers.c`, so I need to pivot. My initial thought was to check if `bpf_stream_vprintk` was used in `helpers.c`. But that's defined in `stream.c`, though it's a kfunc and registered in `helpers.c`. That's a point to remember for cross-module dependencies.
The user's core question is about `snprintf` use in `helpers.c`. I've already confirmed that `snprintf` is only used within `bpf_bprintf_prepare`. But I need to be thorough. I've checked the common formatting functions – `snprintf`, `sprintf`, `vsnprintf`, `scnprintf`, `vscnprintf` – and I haven't seen anything else that poses a similar vulnerability.
Wait, I need to make sure I haven't missed any other formatting functions. I'm recalling `btf_type_snprintf_show`, and that's triggered from `____bpf_snprintf_btf`. I didn't recall this as being related initially. Now, I see `bpf_snprintf_btf_proto` in `helpers.c`, but I'm not seeing where `bpf_snprintf_btf` is defined. I'll search for `BPF_CALL.*snprintf_btf` to find the definition.
|
| 545/6 |
2026/05/18 19:33 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:BPF_CALL.*snprintf_btf]
Results: map[References:<nil> TruncatedOutput:false]
|
| 546/6 |
2026/05/18 19:33 |
llm |
codeexpert-compressor |
0m
Model:
Reply: The investigation into `kernel/bpf/helpers.c` regarding the improper use of `snprintf` and potential out-of-bounds pointer arithmetic is complete. The research confirms that the specific vulnerability pattern is localized to one function within this file, although the impact extends to other BPF helpers that rely on it.
### 1. Approaches and Results
* **Targeted Symbol Search:**
* The agent used `codesearch-find-references` to locate every instance of `snprintf` in `kernel/bpf/helpers.c`.
* **Result:** Only two instances were found, both within the function `bpf_bprintf_prepare`.
* **Variant and Wrapper Analysis:**
* Searched for common kernel formatting variants including `scnprintf`, `vsnprintf`, `sprintf`, `vscnprintf`, and `seq_printf`.
* **Result:** None of these functions are called directly within `kernel/bpf/helpers.c`.
* **Formatting Engine Investigation:**
* Identified `bstr_printf` as the primary engine for BPF string formatting. It is used in `____bpf_snprintf` (the implementation of the `bpf_snprintf` helper).
* **Result:** The usage in `____bpf_snprintf` (line 1084) is safe. It returns `err + 1` to the BPF program, but it does not use the return value to perform pointer arithmetic on a internal buffer, thus avoiding the overflow risk seen in the preparation phase.
* **Kfunc and Lateral Investigation:**
* Checked for newer kfuncs like `bpf_stream_vprintk` in `kernel/bpf/stream.c`.
* **Result:** `bpf_stream_vprintk` calls `bpf_bprintf_prepare`. This confirms that while the `snprintf` calls are only in `helpers.c`, the logic flaw affects multiple subsystems (tracing, streams, etc.) that use the shared preparation logic.
* **Exhaustive File Review:**
* The agent performed a full read of `kernel/bpf/helpers.c` to ensure no macros or complex wrappers (e.g., `BPF_CALL` macros) were hiding additional formatting calls.
* **Result:** Confirmed that other string-handling helpers (like `bpf_get_current_comm` or `bpf_copy_from_user_str`) use `strscpy_pad` or `strncpy_from_user`, which do not share the `snprintf` return value pitfall.
### 2. Current Hypotheses and Findings
**The Vulnerability Hypothesis:**
The vulnerability is strictly tied to the "argument preparation" logic. In BPF, certain complex types (like `%pB` for symbol names or IP addresses) cannot be formatted lazily by the final `bstr_printf` call because the required data might become unavailable or the context might change. Therefore, `bpf_bprintf_prepare` pre-formats these into a temporary buffer (`bin_args`).
**The Mechanism of Failure:**
The code assumes `snprintf` returns the number of bytes written. However, per C99/kernel semantics, `snprintf` returns the number of bytes that *would* have been written if the buffer were large enough. When truncation occurs:
1. `err` becomes greater than the remaining space (`tmp_buf_end - tmp_buf`).
2. `tmp_buf += (err + 1)` moves the pointer beyond `tmp_buf_end`.
3. Subsequent iterations of the loop perform writes to `tmp_buf`, resulting in an out-of-bounds write.
### 3. Critical File Paths and Code Snippets
* **File:** `kernel/bpf/helpers.c`
* **Function:** `bpf_bprintf_prepare`
* **Vulnerable Snippet 1 (Line 924) - Symbol formatting:**
```c
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1); // BUG: err can exceed remaining space
```
* **Vulnerable Snippet 2 (Line 965) - IP address formatting:**
```c
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1; // BUG: err can exceed remaining space
```
### 4. Reasoning Loops and Observations
* **Search for `snprintf_btf`:** The agent encountered a minor reasoning loop while searching for `bpf_snprintf_btf`. This helper is referenced in headers and the `bpf_base_func_proto` table, but its implementation is not in `helpers.c`. Further search revealed that BTF-based snprintf logic is often handled in `kernel/bpf/btf.c` via `btf_type_snprintf_show`.
* **Verification of `btf_type_snprintf_show`:** The agent checked this function and found it uses `vsnprintf` but manages a `len_left` counter correctly, preventing the specific pointer arithmetic bug found in `helpers.c`.
* **Conclusion:** There are **no other places** in `kernel/bpf/helpers.c` using `snprintf` in this manner. The issue is entirely contained within the `bpf_bprintf_prepare` function.
Tokens:
input: 236013
output: 1271
thoughts: 1042
|
| 547/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 3571
output: 23
thoughts: 377
Thoughts: **My Thorough Confirmation of `snprintf` Usage in `kernel/bpf/helpers.c`**
Okay, so the prompt is asking me about potential unchecked `snprintf` usage in `kernel/bpf/helpers.c`. The prior analysis seems pretty thorough, stating the investigation is complete and the vulnerability is localized to `bpf_bprintf_prepare`. It even specifies that only two instances were found, within that single function. But, I'm an expert, and I need to be *absolutely* certain. I can't just take that at face value. I need to independently verify these findings. This isn't just about speed; it's about accuracy and confirming my understanding of the codebase.
Therefore, despite the apparent completeness of the previous analysis, I'm going to conduct my own rigorous check.
First, I'll quickly list the files in `kernel/bpf/` just to ensure I'm even looking in the correct location and not missing something obvious – a quick sanity check never hurts.
Then, I'll perform the `snprintf` search within `kernel/bpf/helpers.c` myself to confirm the two instances mentioned are the only ones present.
Next, I'll widen the search and look for other formatting functions. `scnprintf`, `vsnprintf`, anything that might be used to format strings into buffers that could be missed during a simple `snprintf` grep. I have to consider the potential for indirect calls or variations on the standard function name.
Finally, I'll also double-check for any macros within `kernel/bpf/helpers.c` that might be abstracting `snprintf` or related formatting functions. Macros can be tricky and sometimes hide vulnerabilities. This is an essential step to ensure I haven't missed a hidden use of the function.
The previous summary seems to have covered all these steps, but I need to execute them myself for complete confidence in my conclusion. Ultimately, I expect the analysis to confirm the existing findings, localizing the potentially problematic usage of `snprintf` to the `bpf_bprintf_prepare` function, but I need to rule out all possibilities before I can definitively answer the prompt with confidence.
|
| 548/6 |
2026/05/18 19:34 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:[arena.c arraymap.c backtrack.c bloom_filter.c bpf_cgrp_storage.c bpf_inode_storage.c bpf_insn_array.c bpf_iter.c bpf_local_storage.c bpf_lru_list.c bpf_lru_list.h bpf_lsm.c bpf_lsm_proto.c bpf_struct_ops.c bpf_task_storage.c btf.c btf_iter.c btf_relocate.c cfg.c cgroup.c cgroup_iter.c check_btf.c const_fold.c core.c cpumap.c cpumask.c crypto.c devmap.c disasm.c disasm.h dispatcher.c dmabuf_iter.c fixups.c hashtab.c helpers.c inode.c kmem_cache_iter.c link_iter.c liveness.c local_storage.c log.c lpm_trie.c map_in_map.c map_in_map.h map_iter.c memalloc.c mmap_unlock_work.h mprog.c net_namespace.c offload.c percpu_freelist.c percpu_freelist.h prog_iter.c queue_stack_maps.c range_tree.c range_tree.h relo_core.c reuseport_array.c ringbuf.c rqspinlock.c rqspinlock.h stackmap.c states.c stream.c syscall.c sysfs_btf.c task_iter.c tcx.c tnum.c token.c trampoline.c verifier.c] Subdirs:[preload]]
|
| 549/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 5049
output: 35
thoughts: 26
Thoughts: Okay, here's my take on those initial thoughts:
**Dissecting the BPF Helper Implementation**
Alright, so I'm looking at `kernel/bpf/helpers.c`. My immediate thought is, "Right, this is where a lot of the core BPF helper functionality is defined." Given the nature of these sorts of files, I'm expecting to find a lot of essential, low-level functions that provide the interface between BPF programs and the kernel. Now, the goal is to pinpoint how string formatting is handled and I'll start by searching for `snprintf` in the file. Finding the usages will give me insights into how they're handling formatted output within the BPF context and if there are any restrictions or specific implementations I should be aware of.
|
| 550/6 |
2026/05/18 19:34 |
tool |
codesearch-find-references |
0m
Args: map[Name:snprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965]] TruncatedOutput:false]
|
| 551/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 5399
output: 35
thoughts: 62
Thoughts: Okay, here's my take on those code-diving thoughts.
**Finding and Analyzing String Formatting Function Usage in the BPF Kernel Helpers**
Alright, let's see, I'm digging into the BPF kernel helpers, specifically the string formatting side of things. First off, a quick `grep` for `snprintf` within `kernel/bpf/helpers.c` turns up precisely two hits, and they're both inside `bpf_bprintf_prepare`. That's a good starting point, but I need a broader view. I'm not going to assume that `snprintf` is the only function used here. I want a complete picture. Now, I should check to see if any of these are being used: `scnprintf`, `vsnprintf`, `sprintf`, and `vscnprintf`. Each of these function names provides a certain string formatting feature. I need to make sure that I understand the full scope of how string formatting is being handled in this section of code.
|
| 552/6 |
2026/05/18 19:34 |
tool |
codesearch-find-references |
0m
Args: map[Name:scnprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 553/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 5725
output: 36
thoughts: 14
Thoughts: Here's my take on the thought process:
**Avoiding `scnprintf` and Thinking About Alternatives**
Okay, so I need to format some output, but `scnprintf` isn't an option. That immediately sends a red flag up for me – it usually suggests I'm working with something where direct, safe input isn't guaranteed and I need to be *really* careful about buffer overflows and format string vulnerabilities. `scnprintf` is notorious for that. Fine, no big deal, there are usually alternatives. Let's see, what's the typical approach here? Ah, yes, `vsnprintf`. That's the go-to when you're dealing with variable argument lists and want more control. The `v` prefix always screams "variadic functions" which is usually what I need when dealing with the kind of flexible formatting I suspect I'm dealing with. So `vsnprintf` is the next avenue I'll explore. It'll allow me to handle the arguments safely and hopefully avoid the pitfalls of the approach I was initially considering. Now to see if it's the right fit for the task at hand.
|
| 554/6 |
2026/05/18 19:34 |
tool |
codesearch-find-references |
0m
Args: map[Name:vsnprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 555/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 6033
output: 34
thoughts: 13
Thoughts: Here's my take:
**Investigating String Formatting Options**
Okay, so I need to construct a formatted string here, and my initial instinct is to use the standard library. Let's see... I'm not going to bother with `vsnprintf` at this stage, it's probably overkill for this task. Let me switch gears to something more straightforward – `sprintf`. I'll explore that function first and see if it meets my needs. It's usually the simplest to use. Let's see if that'll work.
|
| 556/6 |
2026/05/18 19:34 |
tool |
codesearch-find-references |
0m
Args: map[Name:sprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 557/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 6213
output: 36
thoughts: 13
Thoughts: Here's a summary of my thought process:
**Thinking About String Formatting in C**
Okay, so I'm thinking about string formatting in C. The first thing that pops into my head is that `sprintf` is probably not the right tool for this particular job, for various reasons (likely buffer overflows and general safety concerns). Let's quickly shift gears and consider `vscnprintf`. I need to evaluate if that's the more appropriate function and the circumstances for its use.
|
| 558/6 |
2026/05/18 19:34 |
tool |
codesearch-find-references |
0m
Args: map[Name:vscnprintf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 559/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 6381
output: 37
thoughts: 17
Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience:
**Investigating `bstr_printf` Usage**
Alright, the last point brought up `bstr_printf`. My immediate thought is, let's drill down into its practical application within the codebase or the area we're discussing. I need to understand *how* it's being leveraged, and in what context, to properly assess its potential impact or relevance. That specific function call is a key data point; its surrounding usage offers significant insight into the problem at hand. I can't offer any useful analysis without that understanding.
|
| 560/6 |
2026/05/18 19:34 |
tool |
codesearch-find-references |
0m
Args: map[Name:bstr_printf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:____bpf_snprintf SourceFile:kernel/bpf/helpers.c SourceLine:1084]] TruncatedOutput:false]
|
| 561/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 6637
output: 43
thoughts: 23
Thoughts: Here's my analysis of the provided thought process, tailored for an expert audience:
**Diving into `bpf_snprintf` and `bstr_printf`**
Okay, so the thought process here immediately wants to understand the relationship between `____bpf_snprintf` and `bstr_printf`. I'm going straight for the implementation details. I'm assuming that since they are mentioned together, there's likely some significant code reuse or a specific interaction happening between them. I'm probably looking for how `bstr_printf` is being used as a building block *within* `____bpf_snprintf`. Is it being called directly, or is it part of a larger chain of functions? I expect to see `bstr_printf` providing core formatting capabilities, potentially dealing with the details of the BPF string buffer (`bstr`) management, and `____bpf_snprintf` providing the higher-level interface or a specialized formatting context that’s specific to the BPF environment. Perhaps it's also responsible for handling things like argument parsing, output length restrictions, or error checking specific to the BPF verifier. The naming convention suggests there might be other related snprintf functions that I should also be aware of, like for example maybe a `bpf_vsnprintf` function. Time to dig in!
|
| 562/6 |
2026/05/18 19:34 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:____bpf_snprintf]
Results: map[SourceCode:1064: BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
1065: const void *, args, u32, data_len)
1066: {
1067: struct bpf_bprintf_data data = {
1068: .get_bin_args = true,
1069: };
1070: int err, num_args;
1071:
1072: if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
1073: (data_len && !args))
1074: return -EINVAL;
1075: num_args = data_len / 8;
1076:
1077: /* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
1078: * can safely give an unbounded size.
1079: */
1080: err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
1081: if (err < 0)
1082: return err;
1083:
1084: err = bstr_printf(str, str_size, fmt, data.bin_args);
1085:
1086: bpf_bprintf_cleanup(&data);
1087:
1088: return err + 1;
1089: }
SourceFile:kernel/bpf/helpers.c]
|
| 563/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 7423
output: 44
thoughts: 237
Thoughts: **Analyzing String Formatting in BPF Helpers**
Okay, so I'm looking at how we handle string formatting within the BPF helper functions, specifically in `kernel/bpf/helpers.c`. I've just been thinking about `____bpf_snprintf`. It's calling `bstr_printf`, which behaves similarly to `vsnprintf`. The return value from `bstr_printf` is the number of characters *that would have been printed*. We're seeing an `err + 1` return from `____bpf_snprintf`, which makes sense because the documentation states that `bpf_snprintf` should return the number of bytes that *would* be written, including the null terminator. No pointer arithmetic with `err`, just a consistent return value when `str_size` is zero and `str` can be NULL. This is good.
But, the question I need to answer is whether there are any other instances in this file where similar string formatting is done with `snprintf` *without* proper return value checking.
I've already confirmed that `snprintf` itself is only used in `bpf_bprintf_prepare`. That's the only place it's even used, not `bpf_snprintf`, so let's go back and carefully examine those instances in `bpf_bprintf_prepare`. I need to ensure there aren't any potential issues related to improper error handling or buffer overflow risks there.
|
| 564/6 |
2026/05/18 19:34 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_prepare]
Results: map[SourceCode: 804: /*
805: * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
806: *
807: * Returns a negative value if fmt is an invalid format string or 0 otherwise.
808: *
809: * This can be used in two ways:
810: * - Format string verification only: when data->get_bin_args is false
811: * - Arguments preparation: in addition to the above verification, it writes in
812: * data->bin_args a binary representation of arguments usable by bstr_printf
813: * where pointers from BPF have been sanitized.
814: *
815: * In argument preparation mode, if 0 is returned, safe temporary buffers are
816: * allocated and bpf_bprintf_cleanup should be called to free them after use.
817: */
818: int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
819: u32 num_args, struct bpf_bprintf_data *data)
820: {
821: bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
822: char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
823: struct bpf_bprintf_buffers *buffers = NULL;
824: size_t sizeof_cur_arg, sizeof_cur_ip;
825: int err, i, num_spec = 0;
826: u64 cur_arg;
827: char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
828:
829: fmt_end = strnchr(fmt, fmt_size, 0);
830: if (!fmt_end)
831: return -EINVAL;
832: fmt_size = fmt_end - fmt;
833:
834: if (get_buffers && bpf_try_get_buffers(&buffers))
835: return -EBUSY;
836:
837: if (data->get_bin_args) {
838: if (num_args)
839: tmp_buf = buffers->bin_args;
840: tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
841: data->bin_args = (u32 *)tmp_buf;
842: }
843:
844: if (data->get_buf)
845: data->buf = buffers->buf;
846:
847: for (i = 0; i < fmt_size; i++) {
848: unsigned char c = fmt[i];
849:
850: /*
851: * Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
852: * through unchanged, while still rejecting ASCII control bytes.
853: */
854: if (isascii(c) && !isprint(c) && !isspace(c)) {
855: err = -EINVAL;
856: goto out;
857: }
858:
859: if (fmt[i] != '%')
860: continue;
861:
862: if (fmt[i + 1] == '%') {
863: i++;
864: continue;
865: }
866:
867: if (num_spec >= num_args) {
868: err = -EINVAL;
869: goto out;
870: }
871:
872: /* The string is zero-terminated so if fmt[i] != 0, we can
873: * always access fmt[i + 1], in the worst case it will be a 0
874: */
875: i++;
876: c = fmt[i];
877: /*
878: * The format parser below only understands ASCII conversion
879: * specifiers and modifiers, so reject non-ASCII after '%'.
880: */
881: if (!isascii(c)) {
882: err = -EINVAL;
883: goto out;
884: }
885:
886: /* skip optional "[0 +-][num]" width formatting field */
887: while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
888: fmt[i] == ' ')
889: i++;
890: if (fmt[i] >= '1' && fmt[i] <= '9') {
891: i++;
892: while (fmt[i] >= '0' && fmt[i] <= '9')
893: i++;
894: }
895:
896: if (fmt[i] == 'p') {
897: sizeof_cur_arg = sizeof(long);
898:
899: if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
900: ispunct(fmt[i + 1])) {
901: if (tmp_buf)
902: cur_arg = raw_args[num_spec];
903: goto nocopy_fmt;
904: }
905:
906: if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
907: fmt[i + 2] == 's') {
908: fmt_ptype = fmt[i + 1];
909: i += 2;
910: goto fmt_str;
911: }
912:
913: if (fmt[i + 1] == 'K' ||
914: fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
915: fmt[i + 1] == 'S') {
916: if (tmp_buf)
917: cur_arg = raw_args[num_spec];
918: i++;
919: goto nocopy_fmt;
920: }
921:
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
930:
931: i++;
932: num_spec++;
933: continue;
934: }
935:
936: /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
937: if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
938: (fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
939: err = -EINVAL;
940: goto out;
941: }
942:
943: i += 2;
944: if (!tmp_buf)
945: goto nocopy_fmt;
946:
947: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
948: if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
949: err = -ENOSPC;
950: goto out;
951: }
952:
953: unsafe_ptr = (char *)(long)raw_args[num_spec];
954: err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
955: sizeof_cur_ip);
956: if (err < 0)
957: memset(cur_ip, 0, sizeof_cur_ip);
958:
959: /* hack: bstr_printf expects IP addresses to be
960: * pre-formatted as strings, ironically, the easiest way
961: * to do that is to call snprintf.
962: */
963: ip_spec[2] = fmt[i - 1];
964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
969: num_spec++;
970:
971: continue;
972: } else if (fmt[i] == 's') {
973: fmt_ptype = fmt[i];
974: fmt_str:
975: if (fmt[i + 1] != 0 &&
976: !isspace(fmt[i + 1]) &&
977: !ispunct(fmt[i + 1])) {
978: err = -EINVAL;
979: goto out;
980: }
981:
982: if (!tmp_buf)
983: goto nocopy_fmt;
984:
985: if (tmp_buf_end == tmp_buf) {
986: err = -ENOSPC;
987: goto out;
988: }
989:
990: unsafe_ptr = (char *)(long)raw_args[num_spec];
991: err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
992: fmt_ptype,
993: tmp_buf_end - tmp_buf);
994: if (err < 0) {
995: tmp_buf[0] = '\0';
996: err = 1;
997: }
998:
999: tmp_buf += err;
1000: num_spec++;
1001:
1002: continue;
1003: } else if (fmt[i] == 'c') {
1004: if (!tmp_buf)
1005: goto nocopy_fmt;
1006:
1007: if (tmp_buf_end == tmp_buf) {
1008: err = -ENOSPC;
1009: goto out;
1010: }
1011:
1012: *tmp_buf = raw_args[num_spec];
1013: tmp_buf++;
1014: num_spec++;
1015:
1016: continue;
1017: }
1018:
1019: sizeof_cur_arg = sizeof(int);
1020:
1021: if (fmt[i] == 'l') {
1022: sizeof_cur_arg = sizeof(long);
1023: i++;
1024: }
1025: if (fmt[i] == 'l') {
1026: sizeof_cur_arg = sizeof(long long);
1027: i++;
1028: }
1029:
1030: if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
1031: fmt[i] != 'x' && fmt[i] != 'X') {
1032: err = -EINVAL;
1033: goto out;
1034: }
1035:
1036: if (tmp_buf)
1037: cur_arg = raw_args[num_spec];
1038: nocopy_fmt:
1039: if (tmp_buf) {
1040: tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
1041: if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
1042: err = -ENOSPC;
1043: goto out;
1044: }
1045:
1046: if (sizeof_cur_arg == 8) {
1047: *(u32 *)tmp_buf = *(u32 *)&cur_arg;
1048: *(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
1049: } else {
1050: *(u32 *)tmp_buf = (u32)(long)cur_arg;
1051: }
1052: tmp_buf += sizeof_cur_arg;
1053: }
1054: num_spec++;
1055: }
1056:
1057: err = 0;
1058: out:
1059: if (err)
1060: bpf_bprintf_cleanup(data);
1061: return err;
1062: }
SourceFile:kernel/bpf/helpers.c]
|
| 565/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 11855
output: 36
thoughts: 242
Thoughts: **Analyzing Potential Buffer Overflows in Kernel Code**
Okay, so I'm looking at this `kernel/bpf/helpers.c` file and immediately the `snprintf` calls jump out at me. Specifically, I see two instances that are making me a little uneasy. First, on line 924, there's `err = snprintf(tmp_buf, (tmp_buf_end - tmp_buf), "%pB", (void *)(long)raw_args[num_spec]);` which is followed immediately by `tmp_buf += (err + 1);`. And then on line 965, it's virtually the same: `err = snprintf(tmp_buf, tmp_buf_end - tmp_buf, ip_spec, &cur_ip);` and then `tmp_buf += err + 1;`.
The issue here is pretty clear: `snprintf` can return a value, `err`, that represents the *required* buffer size, which can be larger than the provided buffer size (`tmp_buf_end - tmp_buf`). So, if `err` is too large, the subsequent `tmp_buf += (err + 1);` will result in a buffer overflow. They're advancing `tmp_buf` without any bounds checking on the `err` value.
My next step is to perform a quick sweep of this file for any other potential vulnerabilities. I'm going to scan the source for any other instances of `snprintf`, `scnprintf`, `vsnprintf`, `sprintf`, or `vscnprintf`, since these are all known to be tricky. And for completeness, I should probably also check for `seq_printf`... although that's usually related to `seq_files` so maybe less likely. Still, it's worth checking.
|
| 566/6 |
2026/05/18 19:34 |
tool |
codesearch-find-references |
0m
Args: map[Name:seq_printf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 567/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 12540
output: 38
thoughts: 37
Thoughts: Okay, here's my summarized thought process, presented as if I were considering the code:
**Considering String Copy Functions**
Right, so we're talking about string copying, and I'm still mulling over those earlier suggestions in the notes. Specifically, what about the standard, albeit often maligned, `strscpy` or even `strncpy`? They came up in the context of `strscpy_pad` and `strncpy_from_user` in that last review. It's time to double-check those functions and see how they really stack up in this scenario. I need to make sure I've got the complete picture. Let me confirm their usage and applicability.
|
| 568/6 |
2026/05/18 19:34 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:strscpy_pad SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 569/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 12782
output: 36
thoughts: 24
Thoughts: Here's my thought process, as if I were considering this function:
**Investigating a Potential String Copy Function**
Okay, so I'm looking for a function called `strscpy_pad`, which, from the sound of it, is likely some sort of string copy function with padding capabilities. But wait a minute... I don't recall seeing that particular name in my standard library or any common third-party string manipulation libraries. That naming convention isn't immediately familiar. My instincts tell me it's not a standard function. Perhaps it's a custom-defined function specific to a project or library. Let me double-check my mental inventory of string functions, just in case. Could it be a typo? Maybe it was meant to be something very similar. Maybe the programmer was trying to use a padding capability with standard 'strscpy'. Okay let's try that...
|
| 570/6 |
2026/05/18 19:34 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:strscpy SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 571/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 13049
output: 27
thoughts: 17
Thoughts: Okay, here's my summary, in the first person, as requested:
**Suspicious Code Inspection**
That's odd. The behavior I'm seeing doesn't quite line up with my expectations. My first instinct, given the nature of the issue, is to suspect a potential buffer overflow or other memory corruption issue. Let me quickly examine the file content itself. My goal is to pinpoint if there are any obvious red flags, especially concerning string copying functions. I want to see if any `strcpy`, `strncpy`, `sprintf`, or similar functions are present, and how they're being used. Specifically, I'm looking for potential vulnerabilities related to unbounded or poorly managed string lengths in any part of the code. This seems like the logical first step in diagnosing this peculiar behavior.
|
| 572/6 |
2026/05/18 19:34 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/bpf/helpers.c]
Results: map[Entities:[map[Kind:function Name:____bpf_copy_from_user] map[Kind:function Name:____bpf_copy_from_user_task] map[Kind:function Name:____bpf_current_task_under_cgroup] map[Kind:function Name:____bpf_dynptr_data] map[Kind:function Name:____bpf_dynptr_from_mem] map[Kind:function Name:____bpf_dynptr_read] map[Kind:function Name:____bpf_dynptr_write] map[Kind:function Name:____bpf_event_output_data] map[Kind:function Name:____bpf_get_current_ancestor_cgroup_id] map[Kind:function Name:____bpf_get_current_cgroup_id] map[Kind:function Name:____bpf_get_current_comm] map[Kind:function Name:____bpf_get_current_pid_tgid] map[Kind:function Name:____bpf_get_current_uid_gid] map[Kind:function Name:____bpf_get_ns_current_pid_tgid] map[Kind:function Name:____bpf_get_numa_node_id] map[Kind:function Name:____bpf_get_smp_processor_id] map[Kind:function Name:____bpf_jiffies64] map[Kind:function Name:____bpf_kptr_xchg] map[Kind:function Name:____bpf_ktime_get_boot_ns] map[Kind:function Name:____bpf_ktime_get_coarse_ns] map[Kind:function Name:____bpf_ktime_get_ns] map[Kind:function Name:____bpf_ktime_get_tai_ns] map[Kind:function Name:____bpf_map_delete_elem] map[Kind:function Name:____bpf_map_lookup_elem] map[Kind:function Name:____bpf_map_lookup_percpu_elem] map[Kind:function Name:____bpf_map_peek_elem] map[Kind:function Name:____bpf_map_pop_elem] map[Kind:function Name:____bpf_map_push_elem] map[Kind:function Name:____bpf_map_update_elem] map[Kind:function Name:____bpf_per_cpu_ptr] map[Kind:function Name:____bpf_snprintf] map[Kind:function Name:____bpf_spin_lock] map[Kind:function Name:____bpf_spin_unlock] map[Kind:function Name:____bpf_strncmp] map[Kind:function Name:____bpf_strtol] map[Kind:function Name:____bpf_strtoul] map[Kind:function Name:____bpf_this_cpu_ptr] map[Kind:function Name:____bpf_timer_cancel] map[Kind:function Name:____bpf_timer_init] map[Kind:function Name:____bpf_timer_set_callback] map[Kind:function Name:____bpf_timer_start] map[Kind:function Name:__bpf_async_init] map[Kind:function Name:__bpf_async_set_callback] map[Kind:function Name:__bpf_dynptr_data] map[Kind:function Name:__bpf_dynptr_data_rw] map[Kind:function Name:__bpf_dynptr_is_rdonly] map[Kind:function Name:__bpf_dynptr_read] map[Kind:function Name:__bpf_dynptr_size] map[Kind:function Name:__bpf_dynptr_write] map[Kind:function Name:__bpf_list_add] map[Kind:function Name:__bpf_list_del] map[Kind:function Name:__bpf_obj_drop_impl] map[Kind:function Name:__bpf_rbtree_add] map[Kind:function Name:__bpf_spin_lock] map[Kind:function Name:__bpf_spin_lock_irqsave] map[Kind:function Name:__bpf_spin_unlock] map[Kind:function Name:__bpf_spin_unlock_irqrestore] map[Kind:function Name:__bpf_strncasecmp] map[Kind:function Name:__bpf_strnstr] map[Kind:function Name:__bpf_strtoll] map[Kind:function Name:__bpf_strtoull] map[Kind:function Name:__bpf_trap] map[Kind:function Name:bpf_async_cancel_and_free] map[Kind:function Name:bpf_async_cb_rcu_free] map[Kind:function Name:bpf_async_cb_rcu_tasks_trace_free] map[Kind:function Name:bpf_async_irq_worker] map[Kind:function Name:bpf_async_process_op] map[Kind:function Name:bpf_async_refcount_put] map[Kind:function Name:bpf_async_schedule_op] map[Kind:function Name:bpf_async_update_prog_callback] map[Kind:function Name:bpf_base_func_proto] map[Kind:function Name:bpf_bprintf_cleanup] map[Kind:function Name:bpf_bprintf_prepare] map[Kind:function Name:bpf_cast_to_kern_ctx] map[Kind:function Name:bpf_cgroup_acquire] map[Kind:function Name:bpf_cgroup_ancestor] map[Kind:function Name:bpf_cgroup_from_id] map[Kind:function Name:bpf_cgroup_release] map[Kind:function Name:bpf_cgroup_release_dtor] map[Kind:function Name:bpf_copy_from_user] map[Kind:function Name:bpf_copy_from_user_str] map[Kind:function Name:bpf_copy_from_user_task] map[Kind:function Name:bpf_copy_from_user_task_str] map[Kind:function Name:bpf_current_task_under_cgroup] map[Kind:function Name:bpf_dynptr_adjust] map[Kind:function Name:bpf_dynptr_advance_offset] map[Kind:function Name:bpf_dynptr_check_size] map[Kind:function Name:bpf_dynptr_clone] map[Kind:function Name:bpf_dynptr_copy] map[Kind:function Name:bpf_dynptr_data] map[Kind:function Name:bpf_dynptr_file_discard] map[Kind:function Name:bpf_dynptr_from_file] map[Kind:function Name:bpf_dynptr_from_file_sleepable] map[Kind:function Name:bpf_dynptr_from_mem] map[Kind:function Name:bpf_dynptr_get_type] map[Kind:function Name:bpf_dynptr_init] map[Kind:function Name:bpf_dynptr_is_null] map[Kind:function Name:bpf_dynptr_is_rdonly] map[Kind:function Name:bpf_dynptr_memset] map[Kind:function Name:bpf_dynptr_read] map[Kind:function Name:bpf_dynptr_set_null] map[Kind:function Name:bpf_dynptr_set_rdonly] map[Kind:function Name:bpf_dynptr_set_size] map[Kind:function Name:bpf_dynptr_set_type] map[Kind:function Name:bpf_dynptr_size] map[Kind:function Name:bpf_dynptr_slice] map[Kind:function Name:bpf_dynptr_slice_rdwr] map[Kind:function Name:bpf_dynptr_write] map[Kind:function Name:bpf_event_output_data] map[Kind:function Name:bpf_file_fetch_bytes] map[Kind:function Name:bpf_get_current_ancestor_cgroup_id] map[Kind:function Name:bpf_get_current_cgroup_id] map[Kind:function Name:bpf_get_current_comm] map[Kind:function Name:bpf_get_current_pid_tgid] map[Kind:function Name:bpf_get_current_uid_gid] map[Kind:function Name:bpf_get_ns_current_pid_tgid] map[Kind:function Name:bpf_get_numa_node_id] map[Kind:function Name:bpf_get_smp_processor_id] map[Kind:function Name:bpf_iter_bits_destroy] map[Kind:function Name:bpf_iter_bits_new] map[Kind:function Name:bpf_iter_bits_next] map[Kind:function Name:bpf_jiffies64] map[Kind:function Name:bpf_key_put] map[Kind:function Name:bpf_kptr_xchg] map[Kind:function Name:bpf_ktime_get_boot_ns] map[Kind:function Name:bpf_ktime_get_coarse_ns] map[Kind:function Name:bpf_ktime_get_ns] map[Kind:function Name:bpf_ktime_get_tai_ns] map[Kind:function Name:bpf_list_back] map[Kind:function Name:bpf_list_front] map[Kind:function Name:bpf_list_head_free] map[Kind:function Name:bpf_list_pop_back] map[Kind:function Name:bpf_list_pop_front] map[Kind:function Name:bpf_list_push_back] map[Kind:function Name:bpf_list_push_back_impl] map[Kind:function Name:bpf_list_push_front] map[Kind:function Name:bpf_list_push_front_impl] map[Kind:function Name:bpf_local_irq_restore] map[Kind:function Name:bpf_local_irq_save] map[Kind:function Name:bpf_lookup_system_key] map[Kind:function Name:bpf_lookup_user_key] map[Kind:function Name:bpf_map_delete_elem] map[Kind:function Name:bpf_map_free_internal_structs] map[Kind:function Name:bpf_map_lookup_elem] map[Kind:function Name:bpf_map_lookup_percpu_elem] map[Kind:function Name:bpf_map_peek_elem] map[Kind:function Name:bpf_map_pop_elem] map[Kind:function Name:bpf_map_push_elem] map[Kind:function Name:bpf_map_update_elem] map[Kind:function Name:bpf_obj_drop] map[Kind:function Name:bpf_obj_drop_impl] map[Kind:function Name:bpf_obj_new] map[Kind:function Name:bpf_obj_new_impl] map[Kind:function Name:bpf_per_cpu_ptr] map[Kind:function Name:bpf_percpu_obj_drop] map[Kind:function Name:bpf_percpu_obj_drop_impl] map[Kind:function Name:bpf_percpu_obj_new] map[Kind:function Name:bpf_percpu_obj_new_impl] map[Kind:function Name:bpf_preempt_disable] map[Kind:function Name:bpf_preempt_enable] map[Kind:function Name:bpf_put_buffers] map[Kind:function Name:bpf_rb_root_free] map[Kind:function Name:bpf_rbtree_add] map[Kind:function Name:bpf_rbtree_add_impl] map[Kind:function Name:bpf_rbtree_first] map[Kind:function Name:bpf_rbtree_left] map[Kind:function Name:bpf_rbtree_remove] map[Kind:function Name:bpf_rbtree_right] map[Kind:function Name:bpf_rbtree_root] map[Kind:function Name:bpf_rcu_read_lock] map[Kind:function Name:bpf_rcu_read_unlock] map[Kind:function Name:bpf_rdonly_cast] map[Kind:function Name:bpf_refcount_acquire] map[Kind:function Name:bpf_refcount_acquire_impl] map[Kind:function Name:bpf_snprintf] map[Kind:function Name:bpf_spin_lock] map[Kind:function Name:bpf_spin_unlock] map[Kind:function Name:bpf_stack_walker] map[Kind:function Name:bpf_strcasecmp] map[Kind:function Name:bpf_strcasestr] map[Kind:function Name:bpf_strchr] map[Kind:function Name:bpf_strchrnul] map[Kind:function Name:bpf_strcmp] map[Kind:function Name:bpf_strcspn] map[Kind:function Name:bpf_strlen] map[Kind:function Name:bpf_strncasecmp] map[Kind:function Name:bpf_strncasestr] map[Kind:function Name:bpf_strnchr] map[Kind:function Name:bpf_strncmp] map[Kind:function Name:bpf_strnlen] map[Kind:function Name:bpf_strnstr] map[Kind:function Name:bpf_strrchr] map[Kind:function Name:bpf_strspn] map[Kind:function Name:bpf_strstr] map[Kind:function Name:bpf_strtol] map[Kind:function Name:bpf_strtoul] map[Kind:function Name:bpf_task_acquire] map[Kind:function Name:bpf_task_from_pid] map[Kind:function Name:bpf_task_from_vpid] map[Kind:function Name:bpf_task_get_cgroup1] map[Kind:function Name:bpf_task_release] map[Kind:function Name:bpf_task_release_dtor] map[Kind:function Name:bpf_task_under_cgroup] map[Kind:function Name:bpf_task_work_acquire_ctx] map[Kind:function Name:bpf_task_work_callback] map[Kind:function Name:bpf_task_work_cancel] map[Kind:function Name:bpf_task_work_cancel_and_free] map[Kind:function Name:bpf_task_work_cancel_scheduled] map[Kind:function Name:bpf_task_work_ctx_put] map[Kind:function Name:bpf_task_work_ctx_reset] map[Kind:function Name:bpf_task_work_ctx_tryget] map[Kind:function Name:bpf_task_work_destroy] map[Kind:function Name:bpf_task_work_fetch_ctx] map[Kind:function Name:bpf_task_work_irq] map[Kind:function Name:bpf_task_work_schedule] map[Kind:function Name:bpf_task_work_schedule_resume] map[Kind:function Name:bpf_task_work_schedule_signal] map[Kind:function Name:bpf_this_cpu_ptr] map[Kind:function Name:bpf_throw] map[Kind:function Name:bpf_timer_cancel] map[Kind:function Name:bpf_timer_cancel_and_free] map[Kind:function Name:bpf_timer_cancel_async] map[Kind:function Name:bpf_timer_cb] map[Kind:function Name:bpf_timer_init] map[Kind:function Name:bpf_timer_set_callback] map[Kind:function Name:bpf_timer_start] map[Kind:function Name:bpf_trace_copy_string] map[Kind:function Name:bpf_try_get_buffers] map[Kind:function Name:bpf_verify_pkcs7_signature] map[Kind:function Name:bpf_wq_cancel_and_free] map[Kind:function Name:bpf_wq_init] map[Kind:function Name:bpf_wq_set_callback] map[Kind:function Name:bpf_wq_start] map[Kind:function Name:bpf_wq_work] map[Kind:function Name:copy_map_value_locked] map[Kind:function Name:defer_timer_wq_op] map[Kind:function Name:kfunc_init] map[Kind:function Name:make_file_dynptr] map[Kind:function Name:map_key_from_value] map[Kind:function Name:swap_ulong_in_u64] map[Kind:function Name:worker_for_call_rcu] map[Kind:struct Name:bpf_async_cb] map[Kind:struct Name:bpf_async_cmd] map[Kind:struct Name:bpf_async_kern] map[Kind:struct Name:bpf_dynptr_file_impl] map[Kind:struct Name:bpf_hrtimer] map[Kind:struct Name:bpf_iter_bits] map[Kind:struct Name:bpf_iter_bits_kern] map[Kind:struct Name:bpf_task_work_ctx] map[Kind:struct Name:bpf_task_work_kern] map[Kind:struct Name:bpf_throw_ctx] map[Kind:struct Name:bpf_work] map[Kind:global_variable Name:__UNIQUE_ID_addressable_bpf_base_func_proto_789] map[Kind:global_variable Name:__UNIQUE_ID_addressable_kfunc_init_832] map[Kind:global_variable Name:bpf_copy_from_user_proto] map[Kind:global_variable Name:bpf_copy_from_user_task_proto] map[Kind:global_variable Name:bpf_current_task_under_cgroup_proto] map[Kind:global_variable Name:bpf_dynptr_data_proto] map[Kind:global_variable Name:bpf_dynptr_from_mem_proto] map[Kind:global_variable Name:bpf_dynptr_read_proto] map[Kind:global_variable Name:bpf_dynptr_write_proto] map[Kind:global_variable Name:bpf_event_output_data_proto] map[Kind:global_variable Name:bpf_get_current_ancestor_cgroup_id_proto] map[Kind:global_variable Name:bpf_get_current_cgroup_id_proto] map[Kind:global_variable Name:bpf_get_current_comm_proto] map[Kind:global_variable Name:bpf_get_current_pid_tgid_proto] map[Kind:global_variable Name:bpf_get_current_uid_gid_proto] map[Kind:global_variable Name:bpf_get_ns_current_pid_tgid_proto] map[Kind:global_variable Name:bpf_get_numa_node_id_proto] map[Kind:global_variable Name:bpf_get_prandom_u32_proto] map[Kind:global_variable Name:bpf_get_raw_smp_processor_id_proto] map[Kind:global_variable Name:bpf_get_smp_processor_id_proto] map[Kind:global_variable Name:bpf_jiffies64_proto] map[Kind:global_variable Name:bpf_kptr_xchg_proto] map[Kind:global_variable Name:bpf_ktime_get_boot_ns_proto] map[Kind:global_variable Name:bpf_ktime_get_coarse_ns_proto] map[Kind:global_variable Name:bpf_ktime_get_ns_proto] map[Kind:global_variable Name:bpf_ktime_get_tai_ns_proto] map[Kind:global_variable Name:bpf_map_delete_elem_proto] map[Kind:global_variable Name:bpf_map_lookup_elem_proto] map[Kind:global_variable Name:bpf_map_lookup_percpu_elem_proto] map[Kind:global_variable Name:bpf_map_peek_elem_proto] map[Kind:global_variable Name:bpf_map_pop_elem_proto] map[Kind:global_variable Name:bpf_map_push_elem_proto] map[Kind:global_variable Name:bpf_map_update_elem_proto] map[Kind:global_variable Name:bpf_per_cpu_ptr_proto] map[Kind:global_variable Name:bpf_snprintf_proto] map[Kind:global_variable Name:bpf_spin_lock_proto] map[Kind:global_variable Name:bpf_spin_unlock_proto] map[Kind:global_variable Name:bpf_strncmp_proto] map[Kind:global_variable Name:bpf_strtol_proto] map[Kind:global_variable Name:bpf_strtoul_proto] map[Kind:global_variable Name:bpf_this_cpu_ptr_proto] map[Kind:global_variable Name:bpf_timer_cancel_proto] map[Kind:global_variable Name:bpf_timer_init_proto] map[Kind:global_variable Name:bpf_timer_set_callback_proto] map[Kind:global_variable Name:bpf_timer_start_proto] map[Kind:global_variable Name:common_btf_ids] map[Kind:global_variable Name:common_kfunc_set] map[Kind:global_variable Name:generic_btf_ids] map[Kind:global_variable Name:generic_kfunc_set] map[Kind:enum Name:bpf_async_op] map[Kind:enum Name:bpf_async_type] map[Kind:enum Name:bpf_task_work_state] map[Kind:typedef Name:bpf_task_work_callback_t] map[Kind:typedef Name:btf_bpf_copy_from_user] map[Kind:typedef Name:btf_bpf_copy_from_user_task] map[Kind:typedef Name:btf_bpf_current_task_under_cgroup] map[Kind:typedef Name:btf_bpf_dynptr_data] map[Kind:typedef Name:btf_bpf_dynptr_from_mem] map[Kind:typedef Name:btf_bpf_dynptr_read] map[Kind:typedef Name:btf_bpf_dynptr_write] map[Kind:typedef Name:btf_bpf_event_output_data] map[Kind:typedef Name:btf_bpf_get_current_ancestor_cgroup_id] map[Kind:typedef Name:btf_bpf_get_current_cgroup_id] map[Kind:typedef Name:btf_bpf_get_current_comm] map[Kind:typedef Name:btf_bpf_get_current_pid_tgid] map[Kind:typedef Name:btf_bpf_get_current_uid_gid] map[Kind:typedef Name:btf_bpf_get_ns_current_pid_tgid] map[Kind:typedef Name:btf_bpf_get_numa_node_id] map[Kind:typedef Name:btf_bpf_get_smp_processor_id] map[Kind:typedef Name:btf_bpf_jiffies64] map[Kind:typedef Name:btf_bpf_kptr_xchg] map[Kind:typedef Name:btf_bpf_ktime_get_boot_ns] map[Kind:typedef Name:btf_bpf_ktime_get_coarse_ns] map[Kind:typedef Name:btf_bpf_ktime_get_ns] map[Kind:typedef Name:btf_bpf_ktime_get_tai_ns] map[Kind:typedef Name:btf_bpf_map_delete_elem] map[Kind:typedef Name:btf_bpf_map_lookup_elem] map[Kind:typedef Name:btf_bpf_map_lookup_percpu_elem] map[Kind:typedef Name:btf_bpf_map_peek_elem] map[Kind:typedef Name:btf_bpf_map_pop_elem] map[Kind:typedef Name:btf_bpf_map_push_elem] map[Kind:typedef Name:btf_bpf_map_update_elem] map[Kind:typedef Name:btf_bpf_per_cpu_ptr] map[Kind:typedef Name:btf_bpf_snprintf] map[Kind:typedef Name:btf_bpf_spin_lock] map[Kind:typedef Name:btf_bpf_spin_unlock] map[Kind:typedef Name:btf_bpf_strncmp] map[Kind:typedef Name:btf_bpf_strtol] map[Kind:typedef Name:btf_bpf_strtoul] map[Kind:typedef Name:btf_bpf_this_cpu_ptr] map[Kind:typedef Name:btf_bpf_timer_cancel] map[Kind:typedef Name:btf_bpf_timer_init] map[Kind:typedef Name:btf_bpf_timer_set_callback] map[Kind:typedef Name:btf_bpf_timer_start]]]
|
| 573/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 20290
output: 46
thoughts: 15
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:
**Diving into `bpf_get_current_comm`**
Alright, let's zero in on `bpf_get_current_comm`. It's a fundamental BPF helper, and, as with all the "core" helpers, it warrants a close inspection. I'm thinking about its implications for tracing, profiling, and security monitoring, specifically. This is a helper I reach for all the time, but it’s always good to refresh my understanding of the edge cases and potential performance bottlenecks. I’m immediately considering things like the overhead of copying the comm string, any potential races, and how it interacts with different task states and signal handling within the kernel. Does it handle context switches cleanly? What's the impact on BPF program latency if called frequently? Does it introduce any noticeable overhead, especially when used inside an eBPF program, running in a tight loop? And of course, the ever-present security considerations: what are the potential vulnerabilities if a malicious user manipulates the process name? I need to think about how this helper operates in different kernel versions, ensuring that any assumptions I make are valid across different deployment scenarios. It's time to refresh myself on the source code, maybe step through its implementation with a debugger to be completely certain of the nuances. Gotta consider the bigger picture, too: how this relates to other helpers and what optimizations can be made at the BPF program level to use this effectively.
|
| 574/6 |
2026/05/18 19:34 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:____bpf_get_current_comm]
Results: map[SourceCode: 261: BPF_CALL_2(bpf_get_current_comm, char *, buf, u32, size)
262: {
263: struct task_struct *task = current;
264:
265: if (unlikely(!task))
266: goto err_clear;
267:
268: /* Verifier guarantees that size > 0 */
269: strscpy_pad(buf, task->comm, size);
270: return 0;
271: err_clear:
272: memset(buf, 0, size);
273: return -EINVAL;
274: }
SourceFile:kernel/bpf/helpers.c]
|
| 575/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 20900
output: 23
thoughts: 74
Thoughts: Okay, here's my summarized thought process, presented as if I'm the one having it:
**Investigating `strscpy_pad` Usage**
Right, so I'm seeing `strscpy_pad` used here. My initial search using `codesearch-find-references` didn't turn up anything. That's a bit odd. I need to figure out why. It could be a typo on my part, or perhaps the indexing mechanism I'm using isn't treating it as a standard "reference." Let me back off and take a slightly more brute-force approach. I'll search for the literal string "strscpy_pad" within this specific file to see where it occurs. That should give me the context I need to understand how it's being used, even if the reference search failed.
|
| 576/6 |
2026/05/18 19:34 |
tool |
read-file |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*/
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/bpf-cgroup.h>
#include <linux/cgroup.h>
#include <linux/rcupdate.h>
#include <linux/random.h>
#include <linux/smp.h>
#include <linux/topology.h>
#include <linux/ktime.h>
#include <linux/sched.h>
#include <linux/uidgid.h>
#include <linux/filter.h>
#include <linux/ctype.h>
#include <linux/jiffies.h>
#include <linux/pid_namespace.h>
#include <linux/poison.h>
#include <linux/proc_ns.h>
#include <linux/sched/task.h>
#include <linux/security.h>
#include <linux/btf_ids.h>
#include <linux/bpf_mem_alloc.h>
#include <linux/kasan.h>
#include <linux/bpf_verifier.h>
#include <linux/uaccess.h>
#include <linux/verification.h>
#include <linux/task_work.h>
#include <linux/irq_work.h>
#include <linux/buildid.h>
#include "../../lib/kstrtox.h"
/* If kernel subsystem is allowing eBPF programs to call this function,
* inside its own verifier_ops->get_func_proto() callback it should return
* bpf_map_lookup_elem_proto, so that verifier can properly check the arguments
*
* Different map implementations will rely on rcu in map methods
* lookup/update/delete, therefore eBPF programs must run under rcu lock
* if program is allowed to access maps, so check rcu_read_lock_held() or
* rcu_read_lock_trace_held() in all three functions.
*/
BPF_CALL_2(bpf_map_lookup_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_elem(map, key);
}
const struct bpf_func_proto bpf_map_lookup_elem_proto = {
.func = bpf_map_lookup_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_4(bpf_map_update_elem, struct bpf_map *, map, void *, key,
void *, value, u64, flags)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_update_elem(map, key, value, flags);
}
const struct bpf_func_proto bpf_map_update_elem_proto = {
.func = bpf_map_update_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_PTR_TO_MAP_VALUE,
.arg4_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_delete_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_delete_elem(map, key);
}
const struct bpf_func_proto bpf_map_delete_elem_proto = {
.func = bpf_map_delete_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_3(bpf_map_push_elem, struct bpf_map *, map, void *, value, u64, flags)
{
return map->ops->map_push_elem(map, value, flags);
}
const struct bpf_func_proto bpf_map_push_elem_proto = {
.func = bpf_map_push_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_pop_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_pop_elem(map, value);
}
const struct bpf_func_proto bpf_map_pop_elem_proto = {
.func = bpf_map_pop_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_2(bpf_map_peek_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_peek_elem(map, value);
}
const struct bpf_func_proto bpf_map_peek_elem_proto = {
.func = bpf_map_peek_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_3(bpf_map_lookup_percpu_elem, struct bpf_map *, map, void *, key, u32, cpu)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_percpu_elem(map, key, cpu);
}
const struct bpf_func_proto bpf_map_lookup_percpu_elem_proto = {
.func = bpf_map_lookup_percpu_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_ANYTHING,
};
const struct bpf_func_proto bpf_get_prandom_u32_proto = {
.func = bpf_user_rnd_u32,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_smp_processor_id)
{
return smp_processor_id();
}
const struct bpf_func_proto bpf_get_smp_processor_id_proto = {
.func = bpf_get_smp_processor_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.allow_fastcall = true,
};
BPF_CALL_0(bpf_get_numa_node_id)
{
return numa_node_id();
}
const struct bpf_func_proto bpf_get_numa_node_id_proto = {
.func = bpf_get_numa_node_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_ns)
{
/* NMI safe access to clock monotonic */
return ktime_get_mono_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_ns_proto = {
.func = bpf_ktime_get_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_boot_ns)
{
/* NMI safe access to clock boottime */
return ktime_get_boot_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_boot_ns_proto = {
.func = bpf_ktime_get_boot_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_coarse_ns)
{
return ktime_get_coarse_ns();
}
const struct bpf_func_proto bpf_ktime_get_coarse_ns_proto = {
.func = bpf_ktime_get_coarse_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_tai_ns)
{
/* NMI safe access to clock tai */
return ktime_get_tai_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_tai_ns_proto = {
.func = bpf_ktime_get_tai_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_pid_tgid)
{
struct task_struct *task = current;
if (unlikely(!task))
return -EINVAL;
return (u64) task->tgid << 32 | task->pid;
}
const struct bpf_func_proto bpf_get_current_pid_tgid_proto = {
.func = bpf_get_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_uid_gid)
{
struct task_struct *task = current;
kuid_t uid;
kgid_t gid;
if (unlikely(!task))
return -EINVAL;
current_uid_gid(&uid, &gid);
return (u64) from_kgid(&init_user_ns, gid) << 32 |
from_kuid(&init_user_ns, uid);
}
const struct bpf_func_proto bpf_get_current_uid_gid_proto = {
.func = bpf_get_current_uid_gid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_2(bpf_get_current_comm, char *, buf, u32, size)
{
struct task_struct *task = current;
if (unlikely(!task))
goto err_clear;
/* Verifier guarantees that size > 0 */
strscpy_pad(buf, task->comm, size);
return 0;
err_clear:
memset(buf, 0, size);
return -EINVAL;
}
const struct bpf_func_proto bpf_get_current_comm_proto = {
.func = bpf_get_current_comm,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE,
};
#if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK)
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
union {
__u32 val;
arch_spinlock_t lock;
} u = { .lock = __ARCH_SPIN_LOCK_UNLOCKED };
compiletime_assert(u.val == 0, "__ARCH_SPIN_LOCK_UNLOCKED not 0");
BUILD_BUG_ON(sizeof(*l) != sizeof(__u32));
BUILD_BUG_ON(sizeof(*lock) != sizeof(__u32));
preempt_disable();
arch_spin_lock(l);
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
arch_spin_unlock(l);
preempt_enable();
}
#else
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
BUILD_BUG_ON(sizeof(*l) != sizeof(*lock));
do {
atomic_cond_read_relaxed(l, !VAL);
} while (atomic_xchg(l, 1));
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
atomic_set_release(l, 0);
}
#endif
static DEFINE_PER_CPU(unsigned long, irqsave_flags);
static inline void __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
{
unsigned long flags;
local_irq_save(flags);
__bpf_spin_lock(lock);
__this_cpu_write(irqsave_flags, flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
{
__bpf_spin_lock_irqsave(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_lock_proto = {
.func = bpf_spin_lock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
{
unsigned long flags;
flags = __this_cpu_read(irqsave_flags);
__bpf_spin_unlock(lock);
local_irq_restore(flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
{
__bpf_spin_unlock_irqrestore(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_unlock_proto = {
.func = bpf_spin_unlock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
bool lock_src)
{
struct bpf_spin_lock *lock;
if (lock_src)
lock = src + map->record->spin_lock_off;
else
lock = dst + map->record->spin_lock_off;
preempt_disable();
__bpf_spin_lock_irqsave(lock);
copy_map_value(map, dst, src);
__bpf_spin_unlock_irqrestore(lock);
preempt_enable();
}
BPF_CALL_0(bpf_jiffies64)
{
return get_jiffies_64();
}
const struct bpf_func_proto bpf_jiffies64_proto = {
.func = bpf_jiffies64,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
#ifdef CONFIG_CGROUPS
BPF_CALL_0(bpf_get_current_cgroup_id)
{
struct cgroup *cgrp;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
cgrp_id = cgroup_id(cgrp);
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_cgroup_id_proto = {
.func = bpf_get_current_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_1(bpf_get_current_ancestor_cgroup_id, int, ancestor_level)
{
struct cgroup *cgrp;
struct cgroup *ancestor;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
ancestor = cgroup_ancestor(cgrp, ancestor_level);
cgrp_id = ancestor ? cgroup_id(ancestor) : 0;
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto = {
.func = bpf_get_current_ancestor_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
#endif /* CONFIG_CGROUPS */
#define BPF_STRTOX_BASE_MASK 0x1F
static int __bpf_strtoull(const char *buf, size_t buf_len, u64 flags,
unsigned long long *res, bool *is_negative)
{
unsigned int base = flags & BPF_STRTOX_BASE_MASK;
const char *cur_buf = buf;
size_t cur_len = buf_len;
unsigned int consumed;
size_t val_len;
char str[64];
if (!buf || !buf_len || !res || !is_negative)
return -EINVAL;
if (base != 0 && base != 8 && base != 10 && base != 16)
return -EINVAL;
if (flags & ~BPF_STRTOX_BASE_MASK)
return -EINVAL;
while (cur_buf < buf + buf_len && isspace(*cur_buf))
++cur_buf;
*is_negative = (cur_buf < buf + buf_len && *cur_buf == '-');
if (*is_negative)
++cur_buf;
consumed = cur_buf - buf;
cur_len -= consumed;
if (!cur_len)
return -EINVAL;
cur_len = min(cur_len, sizeof(str) - 1);
memcpy(str, cur_buf, cur_len);
str[cur_len] = '\0';
cur_buf = str;
cur_buf = _parse_integer_fixup_radix(cur_buf, &base);
val_len = _parse_integer(cur_buf, base, res);
if (val_len & KSTRTOX_OVERFLOW)
return -ERANGE;
if (val_len == 0)
return -EINVAL;
cur_buf += val_len;
consumed += cur_buf - str;
return consumed;
}
static int __bpf_strtoll(const char *buf, size_t buf_len, u64 flags,
long long *res)
{
unsigned long long _res;
bool is_negative;
int err;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative) {
if ((long long)-_res > 0)
return -ERANGE;
*res = -_res;
} else {
if ((long long)_res < 0)
return -ERANGE;
*res = _res;
}
return err;
}
BPF_CALL_4(bpf_strtol, const char *, buf, size_t, buf_len, u64, flags,
s64 *, res)
{
long long _res;
int err;
*res = 0;
err = __bpf_strtoll(buf, buf_len, flags, &_res);
if (err < 0)
return err;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtol_proto = {
.func = bpf_strtol,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(s64),
};
BPF_CALL_4(bpf_strtoul, const char *, buf, size_t, buf_len, u64, flags,
u64 *, res)
{
unsigned long long _res;
bool is_negative;
int err;
*res = 0;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative)
return -EINVAL;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtoul_proto = {
.func = bpf_strtoul,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(u64),
};
BPF_CALL_3(bpf_strncmp, const char *, s1, u32, s1_sz, const char *, s2)
{
return strncmp(s1, s2, s1_sz);
}
static const struct bpf_func_proto bpf_strncmp_proto = {
.func = bpf_strncmp,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_CONST_STR,
};
BPF_CALL_4(bpf_get_ns_current_pid_tgid, u64, dev, u64, ino,
struct bpf_pidns_info *, nsdata, u32, size)
{
struct task_struct *task = current;
struct pid_namespace *pidns;
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_pidns_info)))
goto clear;
if (unlikely((u64)(dev_t)dev != dev))
goto clear;
if (unlikely(!task))
goto clear;
pidns = task_active_pid_ns(task);
if (unlikely(!pidns)) {
err = -ENOENT;
goto clear;
}
if (!ns_match(&pidns->ns, (dev_t)dev, ino))
goto clear;
nsdata->pid = task_pid_nr_ns(task, pidns);
nsdata->tgid = task_tgid_nr_ns(task, pidns);
return 0;
clear:
memset((void *)nsdata, 0, (size_t) size);
return err;
}
const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = {
.func = bpf_get_ns_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
};
static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
.func = bpf_get_raw_cpu_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
}
const struct bpf_func_proto bpf_event_output_data_proto = {
.func = bpf_event_output_data,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
const void __user *, user_ptr)
{
int ret = copy_from_user(dst, user_ptr, size);
if (unlikely(ret)) {
memset(dst, 0, size);
ret = -EFAULT;
}
return ret;
}
const struct bpf_func_proto bpf_copy_from_user_proto = {
.func = bpf_copy_from_user,
.gpl_only = false,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_5(bpf_copy_from_user_task, void *, dst, u32, size,
const void __user *, user_ptr, struct task_struct *, tsk, u64, flags)
{
int ret;
/* flags is not used yet */
if (unlikely(flags))
return -EINVAL;
if (unlikely(!size))
return 0;
ret = access_process_vm(tsk, (unsigned long)user_ptr, dst, size, 0);
if (ret == size)
return 0;
memset(dst, 0, size);
/* Return -EFAULT for partial read */
return ret < 0 ? ret : -EFAULT;
}
const struct bpf_func_proto bpf_copy_from_user_task_proto = {
.func = bpf_copy_from_user_task,
.gpl_only = true,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_BTF_ID,
.arg4_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.arg5_type = ARG_ANYTHING
};
BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
{
if (cpu >= nr_cpu_ids)
return (unsigned long)NULL;
return (unsigned long)per_cpu_ptr((const void __percpu *)(const uintptr_t)ptr, cpu);
}
const struct bpf_func_proto bpf_per_cpu_ptr_proto = {
.func = bpf_per_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | PTR_MAYBE_NULL | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
.arg2_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_this_cpu_ptr, const void *, percpu_ptr)
{
return (unsigned long)this_cpu_ptr((const void __percpu *)(const uintptr_t)percpu_ptr);
}
const struct bpf_func_proto bpf_this_cpu_ptr_proto = {
.func = bpf_this_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
};
static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,
size_t bufsz)
{
void __user *user_ptr = (__force void __user *)unsafe_ptr;
buf[0] = 0;
switch (fmt_ptype) {
case 's':
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
if ((unsigned long)unsafe_ptr < TASK_SIZE)
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
fallthrough;
#endif
case 'k':
return strncpy_from_kernel_nofault(buf, unsafe_ptr, bufsz);
case 'u':
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
}
return -EINVAL;
}
/* Support executing three nested bprintf helper calls on a given CPU */
#define MAX_BPRINTF_NEST_LEVEL 3
static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs);
static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
{
int nest_level;
preempt_disable();
nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
return -EBUSY;
}
*bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
return 0;
}
void bpf_put_buffers(void)
{
if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
return;
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
}
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
{
if (!data->bin_args && !data->buf)
return;
bpf_put_buffers();
}
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)
{
if (map->map_type == BPF_MAP_TYPE_ARRAY) {
struct bpf_array *array = container_of(map, struct bpf_array, map);
*arr_idx = ((char *)value - array->value) / array->elem_size;
return arr_idx;
}
return (void *)value - round_up(map->key_size, 8);
}
enum bpf_async_type {
BPF_ASYNC_TYPE_TIMER = 0,
BPF_ASYNC_TYPE_WQ,
};
enum bpf_async_op {
BPF_ASYNC_START,
BPF_ASYNC_CANCEL
};
struct bpf_async_cmd {
struct llist_node node;
u64 nsec;
u32 mode;
enum bpf_async_op op;
};
struct bpf_async_cb {
struct bpf_map *map;
struct bpf_prog *prog;
void __rcu *callback_fn;
void *value;
struct rcu_head rcu;
u64 flags;
struct irq_work worker;
refcount_t refcnt;
enum bpf_async_type type;
struct llist_head async_cmds;
};
/* BPF map elements can contain 'struct bpf_timer'.
* Such map owns all of its BPF timers.
* 'struct bpf_timer' is allocated as part of map element allocation
* and it's zero initialized.
* That space is used to keep 'struct bpf_async_kern'.
* bpf_timer_init() allocates 'struct bpf_hrtimer', inits hrtimer, and
* remembers 'struct bpf_map *' pointer it's part of.
* bpf_timer_set_callback() increments prog refcnt and assign bpf callback_fn.
* bpf_timer_start() arms the timer.
* If user space reference to a map goes to zero at this point
* ops->map_release_uref callback is responsible for cancelling the timers,
* freeing their memory, and decrementing prog's refcnts.
* bpf_timer_cancel() cancels the timer and decrements prog's refcnt.
* Inner maps can contain bpf timers as well. ops->map_release_uref is
* freeing the timers when inner map is replaced or deleted by user space.
*/
struct bpf_hrtimer {
struct bpf_async_cb cb;
struct hrtimer timer;
atomic_t cancelling;
};
struct bpf_work {
struct bpf_async_cb cb;
struct work_struct work;
};
/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */
struct bpf_async_kern {
union {
struct bpf_async_cb *cb;
struct bpf_hrtimer *timer;
struct bpf_work *work;
};
} __attribute__((aligned(8)));
static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);
static void bpf_async_refcount_put(struct bpf_async_cb *cb);
static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer)
{
struct bpf_hrtimer *t = container_of(hrtimer, struct bpf_hrtimer, timer);
struct bpf_map *map = t->cb.map;
void *value = t->cb.value;
bpf_callback_t callback_fn;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_timer);
callback_fn = rcu_dereference_check(t->cb.callback_fn, rcu_read_lock_bh_held());
if (!callback_fn)
goto out;
/* bpf_timer_cb() runs in hrtimer_run_softirq. It doesn't migrate and
* cannot be preempted by another bpf_timer_cb() on the same cpu.
* Remember the timer this callback is servicing to prevent
* deadlock if callback_fn() calls bpf_timer_cancel() or
* bpf_map_delete_elem() on the same timer.
*/
this_cpu_write(hrtimer_running, t);
key = map_key_from_value(map, value, &idx);
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
/* The verifier checked that return value is zero. */
this_cpu_write(hrtimer_running, NULL);
out:
return HRTIMER_NORESTART;
}
static void bpf_wq_work(struct work_struct *work)
{
struct bpf_work *w = container_of(work, struct bpf_work, work);
struct bpf_async_cb *cb = &w->cb;
struct bpf_map *map = cb->map;
bpf_callback_t callback_fn;
void *value = cb->value;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_wq);
callback_fn = READ_ONCE(cb->callback_fn);
if (!callback_fn)
return;
key = map_key_from_value(map, value, &idx);
rcu_read_lock_trace();
migrate_disable();
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
migrate_enable();
rcu_read_unlock_trace();
}
static void bpf_async_cb_rcu_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
/*
* Drop the last reference to prog only after RCU GP, as set_callback()
* may race with cancel_and_free()
*/
if (cb->prog)
bpf_prog_put(cb->prog);
kfree_nolock(cb);
}
/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */
static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
bool retry = false;
/*
* bpf_async_cancel_and_free() tried to cancel timer/wq, but it
* could have raced with timer/wq_start. Now refcnt is zero and
* srcu/rcu GP completed. Cancel timer/wq again.
*/
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER:
if (hrtimer_try_to_cancel(&t->timer) < 0)
retry = true;
break;
case BPF_ASYNC_TYPE_WQ:
if (!cancel_work(&w->work) && work_busy(&w->work))
retry = true;
break;
}
if (retry) {
/*
* hrtimer or wq callback may still be running. It must be
* in rcu_tasks_trace or rcu CS, so wait for GP again.
* It won't retry forever, since refcnt zero prevents all
* operations on timer/wq.
*/
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
return;
}
/* RCU Tasks Trace grace period implies RCU grace period. */
bpf_async_cb_rcu_free(rcu);
}
static void worker_for_call_rcu(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
static void bpf_async_refcount_put(struct bpf_async_cb *cb)
{
if (!refcount_dec_and_test(&cb->refcnt))
return;
if (irqs_disabled()) {
cb->worker = IRQ_WORK_INIT(worker_for_call_rcu);
irq_work_queue(&cb->worker);
} else {
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async);
static void bpf_async_irq_worker(struct irq_work *work);
static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags,
enum bpf_async_type type)
{
struct bpf_async_cb *cb, *old_cb;
struct bpf_hrtimer *t;
struct bpf_work *w;
clockid_t clockid;
size_t size;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
size = sizeof(struct bpf_hrtimer);
break;
case BPF_ASYNC_TYPE_WQ:
size = sizeof(struct bpf_work);
break;
default:
return -EINVAL;
}
old_cb = READ_ONCE(async->cb);
if (old_cb)
return -EBUSY;
cb = bpf_map_kmalloc_nolock(map, size, 0, map->numa_node);
if (!cb)
return -ENOMEM;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
clockid = flags & (MAX_CLOCKS - 1);
t = (struct bpf_hrtimer *)cb;
atomic_set(&t->cancelling, 0);
hrtimer_setup(&t->timer, bpf_timer_cb, clockid, HRTIMER_MODE_REL_SOFT);
cb->value = (void *)async - map->record->timer_off;
break;
case BPF_ASYNC_TYPE_WQ:
w = (struct bpf_work *)cb;
INIT_WORK(&w->work, bpf_wq_work);
cb->value = (void *)async - map->record->wq_off;
break;
}
cb->map = map;
cb->prog = NULL;
cb->flags = flags;
cb->worker = IRQ_WORK_INIT(bpf_async_irq_worker);
init_llist_head(&cb->async_cmds);
refcount_set(&cb->refcnt, 1); /* map's reference */
cb->type = type;
rcu_assign_pointer(cb->callback_fn, NULL);
old_cb = cmpxchg(&async->cb, NULL, cb);
if (old_cb) {
/* Lost the race to initialize this bpf_async_kern, drop the allocated object */
kfree_nolock(cb);
return -EBUSY;
}
/* Guarantee the order between async->cb and map->usercnt. So
* when there are concurrent uref release and bpf timer init, either
* bpf_timer_cancel_and_free() called by uref release reads a no-NULL
* timer or atomic64_read() below returns a zero usercnt.
*/
smp_mb();
if (!atomic64_read(&map->usercnt)) {
/* maps with timers must be either held by user space
* or pinned in bpffs.
*/
bpf_async_cancel_and_free(async);
return -EPERM;
}
return 0;
}
BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
u64, flags)
{
clock_t clockid = flags & (MAX_CLOCKS - 1);
BUILD_BUG_ON(MAX_CLOCKS != 16);
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer));
if (flags >= MAX_CLOCKS ||
/* similar to timerfd except _ALARM variants are not supported */
(clockid != CLOCK_MONOTONIC &&
clockid != CLOCK_REALTIME &&
clockid != CLOCK_BOOTTIME))
return -EINVAL;
return __bpf_async_init(timer, map, flags, BPF_ASYNC_TYPE_TIMER);
}
static const struct bpf_func_proto bpf_timer_init_proto = {
.func = bpf_timer_init,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
static int bpf_async_update_prog_callback(struct bpf_async_cb *cb,
struct bpf_prog *prog,
void *callback_fn)
{
struct bpf_prog *prev;
/* Acquire a guard reference on prog to prevent it from being freed during the loop */
if (prog) {
prog = bpf_prog_inc_not_zero(prog);
if (IS_ERR(prog))
return PTR_ERR(prog);
}
do {
if (prog)
prog = bpf_prog_inc_not_zero(prog);
prev = xchg(&cb->prog, prog);
rcu_assign_pointer(cb->callback_fn, callback_fn);
/*
* Release previous prog, make sure that if other CPU is contending,
* to set bpf_prog, references are not leaked as each iteration acquires and
* releases one reference.
*/
if (prev)
bpf_prog_put(prev);
} while (READ_ONCE(cb->prog) != prog ||
(void __force *)READ_ONCE(cb->callback_fn) != callback_fn);
if (prog)
bpf_prog_put(prog);
return 0;
}
static DEFINE_PER_CPU(struct bpf_async_cb *, async_cb_running);
static int bpf_async_schedule_op(struct bpf_async_cb *cb, enum bpf_async_op op,
u64 nsec, u32 timer_mode)
{
/*
* Do not schedule another operation on this cpu if it's in irq_work
* callback that is processing async_cmds queue. Otherwise the following
* loop is possible:
* bpf_timer_start() -> bpf_async_schedule_op() -> irq_work_queue().
* irqrestore -> bpf_async_irq_worker() -> tracepoint -> bpf_timer_start().
*/
if (this_cpu_read(async_cb_running) == cb) {
bpf_async_refcount_put(cb);
return -EDEADLK;
}
struct bpf_async_cmd *cmd = kmalloc_nolock(sizeof(*cmd), 0, NUMA_NO_NODE);
if (!cmd) {
bpf_async_refcount_put(cb);
return -ENOMEM;
}
init_llist_node(&cmd->node);
cmd->nsec = nsec;
cmd->mode = timer_mode;
cmd->op = op;
if (llist_add(&cmd->node, &cb->async_cmds))
irq_work_queue(&cb->worker);
return 0;
}
static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback_fn,
struct bpf_prog *prog)
{
struct bpf_async_cb *cb;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
return bpf_async_update_prog_callback(cb, prog, callback_fn);
}
BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
struct bpf_prog_aux *, aux)
{
return __bpf_async_set_callback(timer, callback_fn, aux->prog);
}
static const struct bpf_func_proto bpf_timer_set_callback_proto = {
.func = bpf_timer_set_callback,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_PTR_TO_FUNC,
};
static bool defer_timer_wq_op(void)
{
return in_hardirq() || irqs_disabled();
}
BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags)
{
struct bpf_hrtimer *t;
u32 mode;
if (flags & ~(BPF_F_TIMER_ABS | BPF_F_TIMER_CPU_PIN))
return -EINVAL;
t = READ_ONCE(async->timer);
if (!t || !READ_ONCE(t->cb.prog))
return -EINVAL;
if (flags & BPF_F_TIMER_ABS)
mode = HRTIMER_MODE_ABS_SOFT;
else
mode = HRTIMER_MODE_REL_SOFT;
if (flags & BPF_F_TIMER_CPU_PIN)
mode |= HRTIMER_MODE_PINNED;
/*
* bpf_async_cancel_and_free() could have dropped refcnt to zero. In
* such case BPF progs are not allowed to arm the timer to prevent UAF.
*/
if (!refcount_inc_not_zero(&t->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
hrtimer_start(&t->timer, ns_to_ktime(nsecs), mode);
bpf_async_refcount_put(&t->cb);
return 0;
} else {
return bpf_async_schedule_op(&t->cb, BPF_ASYNC_START, nsecs, mode);
}
}
static const struct bpf_func_proto bpf_timer_start_proto = {
.func = bpf_timer_start,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async)
{
struct bpf_hrtimer *t, *cur_t;
bool inc = false;
int ret = 0;
if (defer_timer_wq_op())
return -EOPNOTSUPP;
t = READ_ONCE(async->timer);
if (!t)
return -EINVAL;
cur_t = this_cpu_read(hrtimer_running);
if (cur_t == t) {
/* If bpf callback_fn is trying to bpf_timer_cancel()
* its own timer the hrtimer_cancel() will deadlock
* since it waits for callback_fn to finish.
*/
return -EDEADLK;
}
/* Only account in-flight cancellations when invoked from a timer
* callback, since we want to avoid waiting only if other _callbacks_
* are waiting on us, to avoid introducing lockups. Non-callback paths
* are ok, since nobody would synchronously wait for their completion.
*/
if (!cur_t)
goto drop;
atomic_inc(&t->cancelling);
/* Need full barrier after relaxed atomic_inc */
smp_mb__after_atomic();
inc = true;
if (atomic_read(&cur_t->cancelling)) {
/* We're cancelling timer t, while some other timer callback is
* attempting to cancel us. In such a case, it might be possible
* that timer t belongs to the other callback, or some other
* callback waiting upon it (creating transitive dependencies
* upon us), and we will enter a deadlock if we continue
* cancelling and waiting for it synchronously, since it might
* do the same. Bail!
*/
atomic_dec(&t->cancelling);
return -EDEADLK;
}
drop:
bpf_async_update_prog_callback(&t->cb, NULL, NULL);
/* Cancel the timer and wait for associated callback to finish
* if it was running.
*/
ret = hrtimer_cancel(&t->timer);
if (inc)
atomic_dec(&t->cancelling);
return ret;
}
static const struct bpf_func_proto bpf_timer_cancel_proto = {
.func = bpf_timer_cancel,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
};
static void bpf_async_process_op(struct bpf_async_cb *cb, u32 op,
u64 timer_nsec, u32 timer_mode)
{
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER: {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
switch (op) {
case BPF_ASYNC_START:
hrtimer_start(&t->timer, ns_to_ktime(timer_nsec), timer_mode);
break;
case BPF_ASYNC_CANCEL:
hrtimer_try_to_cancel(&t->timer);
break;
}
break;
}
case BPF_ASYNC_TYPE_WQ: {
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
switch (op) {
case BPF_ASYNC_START:
schedule_work(&w->work);
break;
case BPF_ASYNC_CANCEL:
cancel_work(&w->work);
break;
}
break;
}
}
bpf_async_refcount_put(cb);
}
static void bpf_async_irq_worker(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
struct llist_node *pos, *n, *list;
list = llist_del_all(&cb->async_cmds);
if (!list)
return;
list = llist_reverse_order(list);
this_cpu_write(async_cb_running, cb);
llist_for_each_safe(pos, n, list) {
struct bpf_async_cmd *cmd;
cmd = container_of(pos, struct bpf_async_cmd, node);
bpf_async_process_op(cb, cmd->op, cmd->nsec, cmd->mode);
kfree_nolock(cmd);
}
this_cpu_write(async_cb_running, NULL);
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async)
{
struct bpf_async_cb *cb;
if (!READ_ONCE(async->cb))
return;
cb = xchg(&async->cb, NULL);
if (!cb)
return;
bpf_async_update_prog_callback(cb, NULL, NULL);
/*
* No refcount_inc_not_zero(&cb->refcnt) here. Dropping the last
* refcnt. Either synchronously or asynchronously in irq_work.
*/
if (!defer_timer_wq_op()) {
bpf_async_process_op(cb, BPF_ASYNC_CANCEL, 0, 0);
} else {
(void)bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
/*
* bpf_async_schedule_op() either enqueues allocated cmd into llist
* or fails with ENOMEM and drop the last refcnt.
* This is unlikely, but safe, since bpf_async_cb_rcu_tasks_trace_free()
* callback will do additional timer/wq_cancel due to races anyway.
*/
}
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_timer_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_wq_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
{
unsigned long *kptr = dst;
/* This helper may be inlined by verifier. */
return xchg(kptr, (unsigned long)ptr);
}
/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg()
* helper is determined dynamically by the verifier. Use BPF_PTR_POISON to
* denote type that verifier will determine.
*/
static const struct bpf_func_proto bpf_kptr_xchg_proto = {
.func = bpf_kptr_xchg,
.gpl_only = false,
.ret_type = RET_PTR_TO_BTF_ID_OR_NULL,
.ret_btf_id = BPF_PTR_POISON,
.arg1_type = ARG_KPTR_XCHG_DEST,
.arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL | OBJ_RELEASE,
.arg2_btf_id = BPF_PTR_POISON,
};
struct bpf_dynptr_file_impl {
struct freader freader;
/* 64 bit offset and size overriding 32 bit ones in bpf_dynptr_kern */
u64 offset;
u64 size;
};
/* Since the upper 8 bits of dynptr->size is reserved, the
* maximum supported size is 2^24 - 1.
*/
#define DYNPTR_MAX_SIZE ((1UL << 24) - 1)
#define DYNPTR_TYPE_SHIFT 28
#define DYNPTR_SIZE_MASK 0xFFFFFF
#define DYNPTR_RDONLY_BIT BIT(31)
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr)
{
return ptr->size & DYNPTR_RDONLY_BIT;
}
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
{
ptr->size |= DYNPTR_RDONLY_BIT;
}
static void bpf_dynptr_set_type(struct bpf_dynptr_kern *ptr, enum bpf_dynptr_type type)
{
ptr->size |= type << DYNPTR_TYPE_SHIFT;
}
static enum bpf_dynptr_type bpf_dynptr_get_type(const struct bpf_dynptr_kern *ptr)
{
return (ptr->size & ~(DYNPTR_RDONLY_BIT)) >> DYNPTR_TYPE_SHIFT;
}
u64 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
return df->size;
}
return ptr->size & DYNPTR_SIZE_MASK;
}
static void bpf_dynptr_advance_offset(struct bpf_dynptr_kern *ptr, u64 off)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->offset += off;
return;
}
ptr->offset += off;
}
static void bpf_dynptr_set_size(struct bpf_dynptr_kern *ptr, u64 new_size)
{
u32 metadata = ptr->size & ~DYNPTR_SIZE_MASK;
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->size = new_size;
return;
}
ptr->size = (u32)new_size | metadata;
}
int bpf_dynptr_check_size(u64 size)
{
return size > DYNPTR_MAX_SIZE ? -E2BIG : 0;
}
static int bpf_file_fetch_bytes(struct bpf_dynptr_file_impl *df, u64 offset, void *buf, u64 len)
{
const void *ptr;
if (!buf)
return -EINVAL;
df->freader.buf = buf;
df->freader.buf_sz = len;
ptr = freader_fetch(&df->freader, offset + df->offset, len);
if (!ptr)
return df->freader.err;
if (ptr != buf) /* Force copying into the buffer */
memcpy(buf, ptr, len);
return 0;
}
void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size)
{
ptr->data = data;
ptr->offset = offset;
ptr->size = size;
bpf_dynptr_set_type(ptr, type);
}
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)
{
memset(ptr, 0, sizeof(*ptr));
}
BPF_CALL_4(bpf_dynptr_from_mem, void *, data, u64, size, u64, flags, struct bpf_dynptr_kern *, ptr)
{
int err;
BTF_TYPE_EMIT(struct bpf_dynptr);
err = bpf_dynptr_check_size(size);
if (err)
goto error;
/* flags is currently unsupported */
if (flags) {
err = -EINVAL;
goto error;
}
bpf_dynptr_init(ptr, data, BPF_DYNPTR_TYPE_LOCAL, 0, size);
return 0;
error:
bpf_dynptr_set_null(ptr);
return err;
}
static const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
.func = bpf_dynptr_from_mem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL | MEM_UNINIT | MEM_WRITE,
};
static int __bpf_dynptr_read(void *dst, u64 len, const struct bpf_dynptr_kern *src,
u64 offset, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!src->data || flags)
return -EINVAL;
err = bpf_dynptr_check_off_len(src, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(src);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst, src->data + src->offset + offset, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_XDP:
return __bpf_xdp_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_SKB_META:
memmove(dst, bpf_skb_meta_pointer(src->data, src->offset + offset), len);
return 0;
case BPF_DYNPTR_TYPE_FILE:
return bpf_file_fetch_bytes(src->data, offset, dst, len);
default:
WARN_ONCE(true, "bpf_dynptr_read: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_read, void *, dst, u64, len, const struct bpf_dynptr_kern *, src,
u64, offset, u64, flags)
{
return __bpf_dynptr_read(dst, len, src, offset, flags);
}
static const struct bpf_func_proto bpf_dynptr_read_proto = {
.func = bpf_dynptr_read,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
int __bpf_dynptr_write(const struct bpf_dynptr_kern *dst, u64 offset, void *src,
u64 len, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!dst->data || __bpf_dynptr_is_rdonly(dst))
return -EINVAL;
err = bpf_dynptr_check_off_len(dst, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(dst);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
if (flags)
return -EINVAL;
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst->data + dst->offset + offset, src, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_store_bytes(dst->data, dst->offset + offset, src, len,
flags);
case BPF_DYNPTR_TYPE_XDP:
if (flags)
return -EINVAL;
return __bpf_xdp_store_bytes(dst->data, dst->offset + offset, src, len);
case BPF_DYNPTR_TYPE_SKB_META:
return __bpf_skb_meta_store_bytes(dst->data, dst->offset + offset, src,
len, flags);
default:
WARN_ONCE(true, "bpf_dynptr_write: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_write, const struct bpf_dynptr_kern *, dst, u64, offset, void *, src,
u64, len, u64, flags)
{
return __bpf_dynptr_write(dst, offset, src, len, flags);
}
static const struct bpf_func_proto bpf_dynptr_write_proto = {
.func = bpf_dynptr_write,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_dynptr_data, const struct bpf_dynptr_kern *, ptr, u64, offset, u64, len)
{
enum bpf_dynptr_type type;
int err;
if (!ptr->data)
return 0;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return 0;
if (__bpf_dynptr_is_rdonly(ptr))
return 0;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return (unsigned long)(ptr->data + ptr->offset + offset);
case BPF_DYNPTR_TYPE_SKB:
case BPF_DYNPTR_TYPE_XDP:
case BPF_DYNPTR_TYPE_SKB_META:
/* skb and xdp dynptrs should use bpf_dynptr_slice / bpf_dynptr_slice_rdwr */
return 0;
default:
WARN_ONCE(true, "bpf_dynptr_data: unknown dynptr type %d\n", type);
return 0;
}
}
static const struct bpf_func_proto bpf_dynptr_data_proto = {
.func = bpf_dynptr_data,
.gpl_only = false,
.ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
};
const struct bpf_func_proto bpf_get_current_task_proto __weak;
const struct bpf_func_proto bpf_get_current_task_btf_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_str_proto __weak;
const struct bpf_func_proto bpf_task_pt_regs_proto __weak;
const struct bpf_func_proto bpf_perf_event_read_proto __weak;
const struct bpf_func_proto bpf_send_signal_proto __weak;
const struct bpf_func_proto bpf_send_signal_thread_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_sleepable_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_proto __weak;
const struct bpf_func_proto bpf_get_branch_snapshot_proto __weak;
const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_map_lookup_elem:
return &bpf_map_lookup_elem_proto;
case BPF_FUNC_map_update_elem:
return &bpf_map_update_elem_proto;
case BPF_FUNC_map_delete_elem:
return &bpf_map_delete_elem_proto;
case BPF_FUNC_map_push_elem:
return &bpf_map_push_elem_proto;
case BPF_FUNC_map_pop_elem:
return &bpf_map_pop_elem_proto;
case BPF_FUNC_map_peek_elem:
return &bpf_map_peek_elem_proto;
case BPF_FUNC_map_lookup_percpu_elem:
return &bpf_map_lookup_percpu_elem_proto;
case BPF_FUNC_get_prandom_u32:
return &bpf_get_prandom_u32_proto;
case BPF_FUNC_get_smp_processor_id:
return &bpf_get_raw_smp_processor_id_proto;
case BPF_FUNC_get_numa_node_id:
return &bpf_get_numa_node_id_proto;
case BPF_FUNC_tail_call:
return &bpf_tail_call_proto;
case BPF_FUNC_ktime_get_ns:
return &bpf_ktime_get_ns_proto;
case BPF_FUNC_ktime_get_boot_ns:
return &bpf_ktime_get_boot_ns_proto;
case BPF_FUNC_ktime_get_tai_ns:
return &bpf_ktime_get_tai_ns_proto;
case BPF_FUNC_ringbuf_output:
return &bpf_ringbuf_output_proto;
case BPF_FUNC_ringbuf_reserve:
return &bpf_ringbuf_reserve_proto;
case BPF_FUNC_ringbuf_submit:
return &bpf_ringbuf_submit_proto;
case BPF_FUNC_ringbuf_discard:
return &bpf_ringbuf_discard_proto;
case BPF_FUNC_ringbuf_query:
return &bpf_ringbuf_query_proto;
case BPF_FUNC_strncmp:
return &bpf_strncmp_proto;
case BPF_FUNC_strtol:
return &bpf_strtol_proto;
case BPF_FUNC_strtoul:
return &bpf_strtoul_proto;
case BPF_FUNC_get_current_pid_tgid:
return &bpf_get_current_pid_tgid_proto;
case BPF_FUNC_get_ns_current_pid_tgid:
return &bpf_get_ns_current_pid_tgid_proto;
case BPF_FUNC_get_current_uid_gid:
return &bpf_get_current_uid_gid_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_BPF))
return NULL;
switch (func_id) {
case BPF_FUNC_spin_lock:
return &bpf_spin_lock_proto;
case BPF_FUNC_spin_unlock:
return &bpf_spin_unlock_proto;
case BPF_FUNC_jiffies64:
return &bpf_jiffies64_proto;
case BPF_FUNC_per_cpu_ptr:
return &bpf_per_cpu_ptr_proto;
case BPF_FUNC_this_cpu_ptr:
return &bpf_this_cpu_ptr_proto;
case BPF_FUNC_timer_init:
return &bpf_timer_init_proto;
case BPF_FUNC_timer_set_callback:
return &bpf_timer_set_callback_proto;
case BPF_FUNC_timer_start:
return &bpf_timer_start_proto;
case BPF_FUNC_timer_cancel:
return &bpf_timer_cancel_proto;
case BPF_FUNC_kptr_xchg:
return &bpf_kptr_xchg_proto;
case BPF_FUNC_for_each_map_elem:
return &bpf_for_each_map_elem_proto;
case BPF_FUNC_loop:
return &bpf_loop_proto;
case BPF_FUNC_user_ringbuf_drain:
return &bpf_user_ringbuf_drain_proto;
case BPF_FUNC_ringbuf_reserve_dynptr:
return &bpf_ringbuf_reserve_dynptr_proto;
case BPF_FUNC_ringbuf_submit_dynptr:
return &bpf_ringbuf_submit_dynptr_proto;
case BPF_FUNC_ringbuf_discard_dynptr:
return &bpf_ringbuf_discard_dynptr_proto;
case BPF_FUNC_dynptr_from_mem:
return &bpf_dynptr_from_mem_proto;
case BPF_FUNC_dynptr_read:
return &bpf_dynptr_read_proto;
case BPF_FUNC_dynptr_write:
return &bpf_dynptr_write_proto;
case BPF_FUNC_dynptr_data:
return &bpf_dynptr_data_proto;
#ifdef CONFIG_CGROUPS
case BPF_FUNC_cgrp_storage_get:
return &bpf_cgrp_storage_get_proto;
case BPF_FUNC_cgrp_storage_delete:
return &bpf_cgrp_storage_delete_proto;
case BPF_FUNC_get_current_cgroup_id:
return &bpf_get_current_cgroup_id_proto;
case BPF_FUNC_get_current_ancestor_cgroup_id:
return &bpf_get_current_ancestor_cgroup_id_proto;
case BPF_FUNC_current_task_under_cgroup:
return &bpf_current_task_under_cgroup_proto;
#endif
#ifdef CONFIG_CGROUP_NET_CLASSID
case BPF_FUNC_get_cgroup_classid:
return &bpf_get_cgroup_classid_curr_proto;
#endif
case BPF_FUNC_task_storage_get:
return &bpf_task_storage_get_proto;
case BPF_FUNC_task_storage_delete:
return &bpf_task_storage_delete_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_PERFMON))
return NULL;
switch (func_id) {
case BPF_FUNC_trace_printk:
return bpf_get_trace_printk_proto();
case BPF_FUNC_get_current_task:
return &bpf_get_current_task_proto;
case BPF_FUNC_get_current_task_btf:
return &bpf_get_current_task_btf_proto;
case BPF_FUNC_get_current_comm:
return &bpf_get_current_comm_proto;
case BPF_FUNC_probe_read_user:
return &bpf_probe_read_user_proto;
case BPF_FUNC_probe_read_kernel:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_proto;
case BPF_FUNC_probe_read_user_str:
return &bpf_probe_read_user_str_proto;
case BPF_FUNC_probe_read_kernel_str:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_str_proto;
case BPF_FUNC_copy_from_user:
return &bpf_copy_from_user_proto;
case BPF_FUNC_copy_from_user_task:
return &bpf_copy_from_user_task_proto;
case BPF_FUNC_snprintf_btf:
return &bpf_snprintf_btf_proto;
case BPF_FUNC_snprintf:
return &bpf_snprintf_proto;
case BPF_FUNC_task_pt_regs:
return &bpf_task_pt_regs_proto;
case BPF_FUNC_trace_vprintk:
return bpf_get_trace_vprintk_proto();
case BPF_FUNC_perf_event_read_value:
return bpf_get_perf_event_read_value_proto();
case BPF_FUNC_perf_event_read:
return &bpf_perf_event_read_proto;
case BPF_FUNC_send_signal:
return &bpf_send_signal_proto;
case BPF_FUNC_send_signal_thread:
return &bpf_send_signal_thread_proto;
case BPF_FUNC_get_task_stack:
return prog->sleepable ? &bpf_get_task_stack_sleepable_proto
: &bpf_get_task_stack_proto;
case BPF_FUNC_get_branch_snapshot:
return &bpf_get_branch_snapshot_proto;
case BPF_FUNC_find_vma:
return &bpf_find_vma_proto;
default:
return NULL;
}
}
EXPORT_SYMBOL_GPL(bpf_base_func_proto);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock)
{
struct list_head *head = list_head, *orig_head = list_head;
BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
/* Do the actual list draining outside the lock to not hold the lock for
* too long, and also prevent deadlocks if tracing programs end up
* executing on entry/exit of functions called inside the critical
* section, and end up doing map ops that call bpf_list_head_free for
* the same map value again.
*/
__bpf_spin_lock_irqsave(spin_lock);
if (!head->next || list_empty(head))
goto unlock;
head = head->next;
unlock:
INIT_LIST_HEAD(orig_head);
__bpf_spin_unlock_irqrestore(spin_lock);
while (head != orig_head) {
void *obj = head;
obj -= field->graph_root.node_offset;
head = head->next;
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
/* Like rbtree_postorder_for_each_entry_safe, but 'pos' and 'n' are
* 'rb_node *', so field name of rb_node within containing struct is not
* needed.
*
* Since bpf_rb_tree's node type has a corresponding struct btf_field with
* graph_root.node_offset, it's not necessary to know field name
* or type of node struct
*/
#define bpf_rbtree_postorder_for_each_entry_safe(pos, n, root) \
for (pos = rb_first_postorder(root); \
pos && ({ n = rb_next_postorder(pos); 1; }); \
pos = n)
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock)
{
struct rb_root_cached orig_root, *root = rb_root;
struct rb_node *pos, *n;
void *obj;
BUILD_BUG_ON(sizeof(struct rb_root_cached) > sizeof(struct bpf_rb_root));
BUILD_BUG_ON(__alignof__(struct rb_root_cached) > __alignof__(struct bpf_rb_root));
__bpf_spin_lock_irqsave(spin_lock);
orig_root = *root;
*root = RB_ROOT_CACHED;
__bpf_spin_unlock_irqrestore(spin_lock);
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
obj = pos;
obj -= field->graph_root.node_offset;
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
__bpf_kfunc_start_defs();
/**
* bpf_obj_new() - allocate an object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate an object of the type identified by @local_type_id__k and
* initialize its special fields. BPF programs can use
* bpf_core_type_id_local() to provide @local_type_id__k. The verifier
* rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
void *p;
p = bpf_mem_alloc(&bpf_global_ma, size);
if (!p)
return NULL;
if (meta)
bpf_obj_init(meta->record, p);
return p;
}
__bpf_kfunc void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_obj_new(local_type_id__k, meta__ign);
}
/**
* bpf_percpu_obj_new() - allocate a percpu object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate a percpu object of the type identified by @local_type_id__k. BPF
* programs can use bpf_core_type_id_local() to provide @local_type_id__k.
* The verifier rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated percpu object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_percpu_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
/* The verifier has ensured that meta must be NULL */
return bpf_mem_alloc(&bpf_global_percpu_ma, size);
}
__bpf_kfunc void *bpf_percpu_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_percpu_obj_new(local_type_id__k, meta__ign);
}
/* Must be called under migrate_disable(), as required by bpf_mem_free */
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu)
{
struct bpf_mem_alloc *ma;
if (rec && rec->refcount_off >= 0 &&
!refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) {
/* Object is refcounted and refcount_dec didn't result in 0
* refcount. Return without freeing the object
*/
return;
}
if (rec)
bpf_obj_free_fields(rec, p);
if (percpu)
ma = &bpf_global_percpu_ma;
else
ma = &bpf_global_ma;
bpf_mem_free_rcu(ma, p);
}
/**
* bpf_obj_drop() - drop a previously allocated object
* @p__alloc: object to free
* @meta: verifier-supplied struct metadata
*
* Destroy special fields in @p__alloc as needed and free the object. The
* verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
void *p = p__alloc;
__bpf_obj_drop_impl(p, meta ? meta->record : NULL, false);
}
__bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
{
return bpf_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_percpu_obj_drop() - drop a previously allocated percpu object
* @p__alloc: percpu object to free
* @meta: verifier-supplied struct metadata
*
* Free @p__alloc. The verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_percpu_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
/* The verifier has ensured that meta must be NULL */
bpf_mem_free_rcu(&bpf_global_percpu_ma, p__alloc);
}
__bpf_kfunc void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign)
{
bpf_percpu_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_refcount_acquire() - turn a local kptr into an owning reference
* @p__refcounted_kptr: non-owning local kptr
* @meta: verifier-supplied struct metadata
*
* Increment the refcount for @p__refcounted_kptr. The verifier rewrites
* @meta; BPF programs do not set it.
*
* Return: Owning reference to @p__refcounted_kptr, or %NULL on failure.
*/
__bpf_kfunc void *bpf_refcount_acquire(void *p__refcounted_kptr, struct btf_struct_meta *meta)
{
struct bpf_refcount *ref;
/* Could just cast directly to refcount_t *, but need some code using
* bpf_refcount type so that it is emitted in vmlinux BTF
*/
ref = (struct bpf_refcount *)(p__refcounted_kptr + meta->record->refcount_off);
if (!refcount_inc_not_zero((refcount_t *)ref))
return NULL;
/* Verifier strips KF_RET_NULL if input is owned ref, see is_kfunc_ret_null
* in verifier.c
*/
return (void *)p__refcounted_kptr;
}
__bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta__ign)
{
return bpf_refcount_acquire(p__refcounted_kptr, meta__ign);
}
static int __bpf_list_add(struct bpf_list_node_kern *node,
struct bpf_list_head *head,
bool tail, struct btf_record *rec, u64 off)
{
struct list_head *n = &node->list_head, *h = (void *)head;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
/* node->owner != NULL implies !list_empty(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
tail ? list_add_tail(n, h) : list_add(n, h);
WRITE_ONCE(node->owner, head);
return 0;
}
/**
* bpf_list_push_front() - add a node to the front of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the front of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_front(head, node, meta__ign, off);
}
/**
* bpf_list_push_back() - add a node to the back of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the back of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_back(head, node, meta__ign, off);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
{
struct list_head *n, *h = (void *)head;
struct bpf_list_node_kern *node;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
if (list_empty(h))
return NULL;
n = tail ? h->prev : h->next;
node = container_of(n, struct bpf_list_node_kern, list_head);
if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))
return NULL;
list_del_init(n);
WRITE_ONCE(node->owner, NULL);
return (struct bpf_list_node *)n;
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
{
return __bpf_list_del(head, false);
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
{
return __bpf_list_del(head, true);
}
__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->next;
}
__bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->prev;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
struct rb_root_cached *r = (struct rb_root_cached *)root;
struct rb_node *n = &node_internal->rb_node;
/* node_internal->owner != root implies either RB_EMPTY_NODE(n) or
* n is owned by some other tree. No need to check RB_EMPTY_NODE(n)
*/
if (READ_ONCE(node_internal->owner) != root)
return NULL;
rb_erase_cached(n, r);
RB_CLEAR_NODE(n);
WRITE_ONCE(node_internal->owner, NULL);
return (struct bpf_rb_node *)n;
}
/* Need to copy rbtree_add_cached's logic here because our 'less' is a BPF
* program
*/
static int __bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node_kern *node,
void *less, struct btf_record *rec, u64 off)
{
struct rb_node **link = &((struct rb_root_cached *)root)->rb_root.rb_node;
struct rb_node *parent = NULL, *n = &node->rb_node;
bpf_callback_t cb = (bpf_callback_t)less;
bool leftmost = true;
/* node->owner != NULL implies !RB_EMPTY_NODE(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
while (*link) {
parent = *link;
if (cb((uintptr_t)node, (uintptr_t)parent, 0, 0, 0)) {
link = &parent->rb_left;
} else {
link = &parent->rb_right;
leftmost = false;
}
}
rb_link_node(n, parent, link);
rb_insert_color_cached(n, (struct rb_root_cached *)root, leftmost);
WRITE_ONCE(node->owner, root);
return 0;
}
/**
* bpf_rbtree_add() - add a node to a BPF rbtree
* @root: tree root
* @node: node to insert
* @less: comparator used to order nodes
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node into @root using @less. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked in a tree.
*/
__bpf_kfunc int bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_rb_node_kern *n = (void *)node;
return __bpf_rbtree_add(root, n, (void *)less, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
void *meta__ign, u64 off)
{
return bpf_rbtree_add(root, node, less, meta__ign, off);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)rb_first_cached(r);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_root(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)r->rb_root.rb_node;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_left(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_left;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_right(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_right;
}
/**
* bpf_task_acquire - Acquire a reference to a task. A task acquired by this
* kfunc which is not stored in a map as a kptr, must be released by calling
* bpf_task_release().
* @p: The task on which a reference is being acquired.
*/
__bpf_kfunc struct task_struct *bpf_task_acquire(struct task_struct *p)
{
if (refcount_inc_not_zero(&p->rcu_users))
return p;
return NULL;
}
/**
* bpf_task_release - Release the reference acquired on a task.
* @p: The task on which a reference is being released.
*/
__bpf_kfunc void bpf_task_release(struct task_struct *p)
{
put_task_struct_rcu_user(p);
}
__bpf_kfunc void bpf_task_release_dtor(void *p)
{
put_task_struct_rcu_user(p);
}
CFI_NOSEAL(bpf_task_release_dtor);
#ifdef CONFIG_CGROUPS
/**
* bpf_cgroup_acquire - Acquire a reference to a cgroup. A cgroup acquired by
* this kfunc which is not stored in a map as a kptr, must be released by
* calling bpf_cgroup_release().
* @cgrp: The cgroup on which a reference is being acquired.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)
{
return cgroup_tryget(cgrp) ? cgrp : NULL;
}
/**
* bpf_cgroup_release - Release the reference acquired on a cgroup.
* If this kfunc is invoked in an RCU read region, the cgroup is guaranteed to
* not be freed until the current grace period has ended, even if its refcount
* drops to 0.
* @cgrp: The cgroup on which a reference is being released.
*/
__bpf_kfunc void bpf_cgroup_release(struct cgroup *cgrp)
{
cgroup_put(cgrp);
}
__bpf_kfunc void bpf_cgroup_release_dtor(void *cgrp)
{
cgroup_put(cgrp);
}
CFI_NOSEAL(bpf_cgroup_release_dtor);
/**
* bpf_cgroup_ancestor - Perform a lookup on an entry in a cgroup's ancestor
* array. A cgroup returned by this kfunc which is not subsequently stored in a
* map, must be released by calling bpf_cgroup_release().
* @cgrp: The cgroup for which we're performing a lookup.
* @level: The level of ancestor to look up.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)
{
struct cgroup *ancestor;
if (level > cgrp->level || level < 0)
return NULL;
/* cgrp's refcnt could be 0 here, but ancestors can still be accessed */
ancestor = cgrp->ancestors[level];
if (!cgroup_tryget(ancestor))
return NULL;
return ancestor;
}
/**
* bpf_cgroup_from_id - Find a cgroup from its ID. A cgroup returned by this
* kfunc which is not subsequently stored in a map, must be released by calling
* bpf_cgroup_release().
* @cgid: cgroup id.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_from_id(u64 cgid)
{
struct cgroup *cgrp;
cgrp = __cgroup_get_from_id(cgid);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
/**
* bpf_task_under_cgroup - wrap task_under_cgroup_hierarchy() as a kfunc, test
* task's membership of cgroup ancestry.
* @task: the task to be tested
* @ancestor: possible ancestor of @task's cgroup
*
* Tests whether @task's default cgroup hierarchy is a descendant of @ancestor.
* It follows all the same rules as cgroup_is_descendant, and only applies
* to the default hierarchy.
*/
__bpf_kfunc long bpf_task_under_cgroup(struct task_struct *task,
struct cgroup *ancestor)
{
long ret;
rcu_read_lock();
ret = task_under_cgroup_hierarchy(task, ancestor);
rcu_read_unlock();
return ret;
}
BPF_CALL_2(bpf_current_task_under_cgroup, struct bpf_map *, map, u32, idx)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
struct cgroup *cgrp;
if (unlikely(idx >= array->map.max_entries))
return -E2BIG;
cgrp = READ_ONCE(array->ptrs[idx]);
if (unlikely(!cgrp))
return -EAGAIN;
return task_under_cgroup_hierarchy(current, cgrp);
}
const struct bpf_func_proto bpf_current_task_under_cgroup_proto = {
.func = bpf_current_task_under_cgroup,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
};
/**
* bpf_task_get_cgroup1 - Acquires the associated cgroup of a task within a
* specific cgroup1 hierarchy. The cgroup1 hierarchy is identified by its
* hierarchy ID.
* @task: The target task
* @hierarchy_id: The ID of a cgroup1 hierarchy
*
* On success, the cgroup is returen. On failure, NULL is returned.
*/
__bpf_kfunc struct cgroup *
bpf_task_get_cgroup1(struct task_struct *task, int hierarchy_id)
{
struct cgroup *cgrp = task_get_cgroup1(task, hierarchy_id);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
#endif /* CONFIG_CGROUPS */
/**
* bpf_task_from_pid - Find a struct task_struct from its pid by looking it up
* in the root pid namespace idr. If a task is returned, it must either be
* stored in a map, or released with bpf_task_release().
* @pid: The pid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_pid_ns(pid, &init_pid_ns);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_task_from_vpid - Find a struct task_struct from its vpid by looking it up
* in the pid namespace of the current task. If a task is returned, it must
* either be stored in a map, or released with bpf_task_release().
* @vpid: The vpid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_vpid(vpid);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_dynptr_slice() - Obtain a read-only pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* If the intention is to write to the data slice, please use
* bpf_dynptr_slice_rdwr.
*
* The user must check that the returned pointer is not null before using it.
*
* Please note that in the case of skb and xdp dynptrs, bpf_dynptr_slice
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a read-only
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
enum bpf_dynptr_type type;
u64 len = buffer__szk;
int err;
if (!ptr->data)
return NULL;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return NULL;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return ptr->data + ptr->offset + offset;
case BPF_DYNPTR_TYPE_SKB:
if (buffer__nullable)
return skb_header_pointer(ptr->data, ptr->offset + offset, len, buffer__nullable);
else
return skb_pointer_if_linear(ptr->data, ptr->offset + offset, len);
case BPF_DYNPTR_TYPE_XDP:
{
void *xdp_ptr = bpf_xdp_pointer(ptr->data, ptr->offset + offset, len);
if (!IS_ERR_OR_NULL(xdp_ptr))
return xdp_ptr;
if (!buffer__nullable)
return NULL;
bpf_xdp_copy_buf(ptr->data, ptr->offset + offset, buffer__nullable, len, false);
return buffer__nullable;
}
case BPF_DYNPTR_TYPE_SKB_META:
return bpf_skb_meta_pointer(ptr->data, ptr->offset + offset);
case BPF_DYNPTR_TYPE_FILE:
err = bpf_file_fetch_bytes(ptr->data, offset, buffer__nullable, buffer__szk);
return err ? NULL : buffer__nullable;
default:
WARN_ONCE(true, "unknown dynptr type %d\n", type);
return NULL;
}
}
/**
* bpf_dynptr_slice_rdwr() - Obtain a writable pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* The returned pointer is writable and may point to either directly the dynptr
* data at the requested offset or to the buffer if unable to obtain a direct
* data pointer to (example: the requested slice is to the paged area of an skb
* packet). In the case where the returned pointer is to the buffer, the user
* is responsible for persisting writes through calling bpf_dynptr_write(). This
* usually looks something like this pattern:
*
* struct eth_hdr *eth = bpf_dynptr_slice_rdwr(&dynptr, 0, buffer, sizeof(buffer));
* if (!eth)
* return TC_ACT_SHOT;
*
* // mutate eth header //
*
* if (eth == buffer)
* bpf_dynptr_write(&ptr, 0, buffer, sizeof(buffer), 0);
*
* Please note that, as in the example above, the user must check that the
* returned pointer is not null before using it.
*
* Please also note that in the case of skb and xdp dynptrs, bpf_dynptr_slice_rdwr
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice_rdwr will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data || __bpf_dynptr_is_rdonly(ptr))
return NULL;
/* bpf_dynptr_slice_rdwr is the same logic as bpf_dynptr_slice.
*
* For skb-type dynptrs, it is safe to write into the returned pointer
* if the bpf program allows skb data writes. There are two possibilities
* that may occur when calling bpf_dynptr_slice_rdwr:
*
* 1) The requested slice is in the head of the skb. In this case, the
* returned pointer is directly to skb data, and if the skb is cloned, the
* verifier will have uncloned it (see bpf_unclone_prologue()) already.
* The pointer can be directly written into.
*
* 2) Some portion of the requested slice is in the paged buffer area.
* In this case, the requested data will be copied out into the buffer
* and the returned pointer will be a pointer to the buffer. The skb
* will not be pulled. To persist the write, the user will need to call
* bpf_dynptr_write(), which will pull the skb and commit the write.
*
* Similarly for xdp programs, if the requested slice is not across xdp
* fragments, then a direct pointer will be returned, otherwise the data
* will be copied out into the buffer and the user will need to call
* bpf_dynptr_write() to commit changes.
*/
return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
}
__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 size;
if (!ptr->data || start > end)
return -EINVAL;
size = __bpf_dynptr_size(ptr);
if (start > size || end > size)
return -ERANGE;
bpf_dynptr_advance_offset(ptr, start);
bpf_dynptr_set_size(ptr, end - start);
return 0;
}
__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
return !ptr->data;
}
__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return false;
return __bpf_dynptr_is_rdonly(ptr);
}
__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return -EINVAL;
return __bpf_dynptr_size(ptr);
}
__bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
struct bpf_dynptr *clone__uninit)
{
struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data) {
bpf_dynptr_set_null(clone);
return -EINVAL;
}
*clone = *ptr;
return 0;
}
/**
* bpf_dynptr_copy() - Copy data from one dynptr to another.
* @dst_ptr: Destination dynptr - where data should be copied to
* @dst_off: Offset into the destination dynptr
* @src_ptr: Source dynptr - where data should be copied from
* @src_off: Offset into the source dynptr
* @size: Length of the data to copy from source to destination
*
* Copies data from source dynptr to destination dynptr.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
{
struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
void *src_slice, *dst_slice;
char buf[256];
u64 off;
src_slice = bpf_dynptr_slice(src_ptr, src_off, NULL, size);
dst_slice = bpf_dynptr_slice_rdwr(dst_ptr, dst_off, NULL, size);
if (src_slice && dst_slice) {
memmove(dst_slice, src_slice, size);
return 0;
}
if (src_slice)
return __bpf_dynptr_write(dst, dst_off, src_slice, size, 0);
if (dst_slice)
return __bpf_dynptr_read(dst_slice, size, src, src_off, 0);
if (bpf_dynptr_check_off_len(dst, dst_off, size) ||
bpf_dynptr_check_off_len(src, src_off, size))
return -E2BIG;
off = 0;
while (off < size) {
u64 chunk_sz = min_t(u64, sizeof(buf), size - off);
int err;
err = __bpf_dynptr_read(buf, chunk_sz, src, src_off + off, 0);
if (err)
return err;
err = __bpf_dynptr_write(dst, dst_off + off, buf, chunk_sz, 0);
if (err)
return err;
off += chunk_sz;
}
return 0;
}
/**
* bpf_dynptr_memset() - Fill dynptr memory with a constant byte.
* @p: Destination dynptr - where data will be filled
* @offset: Offset into the dynptr to start filling from
* @size: Number of bytes to fill
* @val: Constant byte to fill the memory with
*
* Fills the @size bytes of the memory area pointed to by @p
* at @offset with the constant byte @val.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 chunk_sz, write_off;
char buf[256];
void* slice;
int err;
slice = bpf_dynptr_slice_rdwr(p, offset, NULL, size);
if (likely(slice)) {
memset(slice, val, size);
return 0;
}
if (__bpf_dynptr_is_rdonly(ptr))
return -EINVAL;
err = bpf_dynptr_check_off_len(ptr, offset, size);
if (err)
return err;
/* Non-linear data under the dynptr, write from a local buffer */
chunk_sz = min_t(u64, sizeof(buf), size);
memset(buf, val, chunk_sz);
for (write_off = 0; write_off < size; write_off += chunk_sz) {
chunk_sz = min_t(u64, sizeof(buf), size - write_off);
err = __bpf_dynptr_write(ptr, offset + write_off, buf, chunk_sz, 0);
if (err)
return err;
}
return 0;
}
__bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
{
return obj;
}
__bpf_kfunc void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k)
{
return (void *)obj__ign;
}
__bpf_kfunc void bpf_rcu_read_lock(void)
{
rcu_read_lock();
}
__bpf_kfunc void bpf_rcu_read_unlock(void)
{
rcu_read_unlock();
}
struct bpf_throw_ctx {
struct bpf_prog_aux *aux;
u64 sp;
u64 bp;
int cnt;
};
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
{
struct bpf_throw_ctx *ctx = cookie;
struct bpf_prog *prog;
/*
* The RCU read lock is held to safely traverse the latch tree, but we
* don't need its protection when accessing the prog, since it has an
* active stack frame on the current stack trace, and won't disappear.
*/
rcu_read_lock();
prog = bpf_prog_ksym_find(ip);
rcu_read_unlock();
if (!prog)
return !ctx->cnt;
ctx->cnt++;
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
ctx->sp = sp;
ctx->bp = bp;
return false;
}
__bpf_kfunc void bpf_throw(u64 cookie)
{
struct bpf_throw_ctx ctx = {};
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
WARN_ON_ONCE(!ctx.aux->exception_boundary);
WARN_ON_ONCE(!ctx.bp);
WARN_ON_ONCE(!ctx.cnt);
/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
* deeper stack depths than ctx.sp as we do not return from bpf_throw,
* which skips compiler generated instrumentation to do the same.
*/
kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_map *map = p__map;
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq));
if (flags)
return -EINVAL;
return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ);
}
__bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_work *w;
if (flags)
return -EINVAL;
w = READ_ONCE(async->work);
if (!w || !READ_ONCE(w->cb.prog))
return -EINVAL;
if (!refcount_inc_not_zero(&w->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
schedule_work(&w->work);
bpf_async_refcount_put(&w->cb);
return 0;
} else {
return bpf_async_schedule_op(&w->cb, BPF_ASYNC_START, 0, 0);
}
}
__bpf_kfunc int bpf_wq_set_callback(struct bpf_wq *wq,
int (callback_fn)(void *map, int *key, void *value),
unsigned int flags,
struct bpf_prog_aux *aux)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
if (flags)
return -EINVAL;
return __bpf_async_set_callback(async, callback_fn, aux->prog);
}
__bpf_kfunc void bpf_preempt_disable(void)
{
preempt_disable();
}
__bpf_kfunc void bpf_preempt_enable(void)
{
preempt_enable();
}
struct bpf_iter_bits {
__u64 __opaque[2];
} __aligned(8);
#define BITS_ITER_NR_WORDS_MAX 511
struct bpf_iter_bits_kern {
union {
__u64 *bits;
__u64 bits_copy;
};
int nr_bits;
int bit;
} __aligned(8);
/* On 64-bit hosts, unsigned long and u64 have the same size, so passing
* a u64 pointer and an unsigned long pointer to find_next_bit() will
* return the same result, as both point to the same 8-byte area.
*
* For 32-bit little-endian hosts, using a u64 pointer or unsigned long
* pointer also makes no difference. This is because the first iterated
* unsigned long is composed of bits 0-31 of the u64 and the second unsigned
* long is composed of bits 32-63 of the u64.
*
* However, for 32-bit big-endian hosts, this is not the case. The first
* iterated unsigned long will be bits 32-63 of the u64, so swap these two
* ulong values within the u64.
*/
static void swap_ulong_in_u64(u64 *bits, unsigned int nr)
{
#if (BITS_PER_LONG == 32) && defined(__BIG_ENDIAN)
unsigned int i;
for (i = 0; i < nr; i++)
bits[i] = (bits[i] >> 32) | ((u64)(u32)bits[i] << 32);
#endif
}
/**
* bpf_iter_bits_new() - Initialize a new bits iterator for a given memory area
* @it: The new bpf_iter_bits to be created
* @unsafe_ptr__ign: A pointer pointing to a memory area to be iterated over
* @nr_words: The size of the specified memory area, measured in 8-byte units.
* The maximum value of @nr_words is @BITS_ITER_NR_WORDS_MAX. This limit may be
* further reduced by the BPF memory allocator implementation.
*
* This function initializes a new bpf_iter_bits structure for iterating over
* a memory area which is specified by the @unsafe_ptr__ign and @nr_words. It
* copies the data of the memory area to the newly created bpf_iter_bits @it for
* subsequent iteration operations.
*
* On success, 0 is returned. On failure, ERR is returned.
*/
__bpf_kfunc int
bpf_iter_bits_new(struct bpf_iter_bits *it, const u64 *unsafe_ptr__ign, u32 nr_words)
{
struct bpf_iter_bits_kern *kit = (void *)it;
u32 nr_bytes = nr_words * sizeof(u64);
u32 nr_bits = BYTES_TO_BITS(nr_bytes);
int err;
BUILD_BUG_ON(sizeof(struct bpf_iter_bits_kern) != sizeof(struct bpf_iter_bits));
BUILD_BUG_ON(__alignof__(struct bpf_iter_bits_kern) !=
__alignof__(struct bpf_iter_bits));
kit->nr_bits = 0;
kit->bits_copy = 0;
kit->bit = -1;
if (!unsafe_ptr__ign || !nr_words)
return -EINVAL;
if (nr_words > BITS_ITER_NR_WORDS_MAX)
return -E2BIG;
/* Optimization for u64 mask */
if (nr_bits == 64) {
err = bpf_probe_read_kernel_common(&kit->bits_copy, nr_bytes, unsafe_ptr__ign);
if (err)
return -EFAULT;
swap_ulong_in_u64(&kit->bits_copy, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
if (bpf_mem_alloc_check_size(false, nr_bytes))
return -E2BIG;
/* Fallback to memalloc */
kit->bits = bpf_mem_alloc(&bpf_global_ma, nr_bytes);
if (!kit->bits)
return -ENOMEM;
err = bpf_probe_read_kernel_common(kit->bits, nr_bytes, unsafe_ptr__ign);
if (err) {
bpf_mem_free(&bpf_global_ma, kit->bits);
return err;
}
swap_ulong_in_u64(kit->bits, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
/**
* bpf_iter_bits_next() - Get the next bit in a bpf_iter_bits
* @it: The bpf_iter_bits to be checked
*
* This function returns a pointer to a number representing the value of the
* next bit in the bits.
*
* If there are no further bits available, it returns NULL.
*/
__bpf_kfunc int *bpf_iter_bits_next(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
int bit = kit->bit, nr_bits = kit->nr_bits;
const void *bits;
if (!nr_bits || bit >= nr_bits)
return NULL;
bits = nr_bits == 64 ? &kit->bits_copy : kit->bits;
bit = find_next_bit(bits, nr_bits, bit + 1);
if (bit >= nr_bits) {
kit->bit = bit;
return NULL;
}
kit->bit = bit;
return &kit->bit;
}
/**
* bpf_iter_bits_destroy() - Destroy a bpf_iter_bits
* @it: The bpf_iter_bits to be destroyed
*
* Destroy the resource associated with the bpf_iter_bits.
*/
__bpf_kfunc void bpf_iter_bits_destroy(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
if (kit->nr_bits <= 64)
return;
bpf_mem_free(&bpf_global_ma, kit->bits);
}
/**
* bpf_copy_from_user_str() - Copy a string from an unsafe user address
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address, in user space.
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL-terminated string from userspace to BPF space. If user string is
* too long this will still ensure zero termination in the dst buffer unless
* buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst to 0 on success and
* memset all of @dst on failure.
*/
__bpf_kfunc int bpf_copy_from_user_str(void *dst, u32 dst__sz, const void __user *unsafe_ptr__ign, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(!dst__sz))
return 0;
ret = strncpy_from_user(dst, unsafe_ptr__ign, dst__sz - 1);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst + ret, 0, dst__sz - ret);
else
((char *)dst)[ret] = '\0';
return ret + 1;
}
/**
* bpf_copy_from_user_task_str() - Copy a string from an task's address space
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address in the task's address space.
* @tsk: The task whose address space will be used
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL terminated string from a task's address space to @dst__sz
* buffer. If user string is too long this will still ensure zero termination
* in the @dst__sz buffer unless buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst__sz to 0 on success
* and memset all of @dst__sz on failure.
*
* Return: The number of copied bytes on success including the NUL terminator.
* A negative error code on failure.
*/
__bpf_kfunc int bpf_copy_from_user_task_str(void *dst, u32 dst__sz,
const void __user *unsafe_ptr__ign,
struct task_struct *tsk, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(dst__sz == 0))
return 0;
ret = copy_remote_vm_str(tsk, (unsigned long)unsafe_ptr__ign, dst, dst__sz, 0);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset(dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset(dst + ret, 0, dst__sz - ret);
return ret + 1;
}
/* Keep unsinged long in prototype so that kfunc is usable when emitted to
* vmlinux.h in BPF programs directly, but note that while in BPF prog, the
* unsigned long always points to 8-byte region on stack, the kernel may only
* read and write the 4-bytes on 32-bit.
*/
__bpf_kfunc void bpf_local_irq_save(unsigned long *flags__irq_flag)
{
local_irq_save(*flags__irq_flag);
}
__bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)
{
local_irq_restore(*flags__irq_flag);
}
__bpf_kfunc void __bpf_trap(void)
{
}
/*
* Kfuncs for string operations.
*
* Since strings are not necessarily %NUL-terminated, we cannot directly call
* in-kernel implementations. Instead, we open-code the implementations using
* __get_kernel_nofault instead of plain dereference to make them safe.
*/
static int __bpf_strncasecmp(const char *s1, const char *s2, bool ignore_case, size_t len)
{
char c1, c2;
int i;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < len && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c1, s1, char, err_out);
__get_kernel_nofault(&c2, s2, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 != c2)
return c1 < c2 ? -1 : 1;
if (c1 == '\0')
return 0;
s1++;
s2++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : 0;
err_out:
return -EFAULT;
}
/**
* bpf_strcmp - Compare two strings
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, false, XATTR_SIZE_MAX);
}
/**
* bpf_strcasecmp - Compare two strings, ignoring the case of the characters
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasecmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, XATTR_SIZE_MAX);
}
/*
* bpf_strncasecmp - Compare two length-limited strings, ignoring case
* @s1__ign: One string
* @s2__ign: Another string
* @len: The maximum number of characters to compare
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasecmp(const char *s1__ign, const char *s2__ign, size_t len)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, len);
}
/**
* bpf_strnchr - Find a character in a length limited string
* @s__ign: The string to be searched
* @count: The number of characters to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in the first @count characters of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnchr(const char *s__ign, size_t count, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
return i;
if (sc == '\0')
return -ENOENT;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : -ENOENT;
err_out:
return -EFAULT;
}
/**
* bpf_strchr - Find the first occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - The index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchr(const char *s__ign, char c)
{
return bpf_strnchr(s__ign, XATTR_SIZE_MAX, c);
}
/**
* bpf_strchrnul - Find and return a character in a string, or end of string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign or index of
* the null byte at the end of @s__ign when @c is not found
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchrnul(const char *s__ign, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == '\0' || sc == c)
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strrchr - Find the last occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the last occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strrchr(const char *s__ign, int c)
{
char sc;
int i, last = -ENOENT;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
last = i;
if (sc == '\0')
return last;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strnlen - Calculate the length of a length-limited string
* @s__ign: The string
* @count: The maximum number of characters to count
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnlen(const char *s__ign, size_t count)
{
char c;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c, s__ign, char, err_out);
if (c == '\0')
return i;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : i;
err_out:
return -EFAULT;
}
/**
* bpf_strlen - Calculate the length of a string
* @s__ign: The string
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strlen(const char *s__ign)
{
return bpf_strnlen(s__ign, XATTR_SIZE_MAX);
}
/**
* bpf_strspn - Calculate the length of the initial substring of @s__ign which
* only contains letters in @accept__ign
* @s__ign: The string to be searched
* @accept__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which only
* contains letters from @accept__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strspn(const char *s__ign, const char *accept__ign)
{
char cs, ca;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(accept__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&ca, accept__ign + j, char, err_out);
if (cs == ca || ca == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (ca == '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strcspn - Calculate the length of the initial substring of @s__ign which
* does not contain letters in @reject__ign
* @s__ign: The string to be searched
* @reject__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which does not
* contain letters from @reject__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcspn(const char *s__ign, const char *reject__ign)
{
char cs, cr;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(reject__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&cr, reject__ign + j, char, err_out);
if (cs == cr || cr == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (cr != '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
static int __bpf_strnstr(const char *s1, const char *s2, size_t len,
bool ignore_case)
{
char c1, c2;
int i, j;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
for (j = 0; i + j <= len && j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&c2, s2 + j, char, err_out);
if (c2 == '\0')
return i;
/*
* We allow reading an extra byte from s2 (note the
* `i + j <= len` above) to cover the case when s2 is
* a suffix of the first len chars of s1.
*/
if (i + j == len)
break;
__get_kernel_nofault(&c1, s1 + j, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 == '\0')
return -ENOENT;
if (c1 != c2)
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (i + j == len)
return -ENOENT;
s1++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strstr - Find the first substring in a string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strstr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, false);
}
/**
* bpf_strcasestr - Find the first substring in a string, ignoring the case of
* the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasestr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, true);
}
/**
* bpf_strnstr - Find the first substring in a length-limited string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strnstr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, false);
}
/**
* bpf_strncasestr - Find the first substring in a length-limited string,
* ignoring the case of the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasestr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, true);
}
#ifdef CONFIG_KEYS
/**
* bpf_lookup_user_key - lookup a key by its serial
* @serial: key handle serial number
* @flags: lookup-specific flags
*
* Search a key with a given *serial* and the provided *flags*.
* If found, increment the reference count of the key by one, and
* return it in the bpf_key structure.
*
* The bpf_key structure must be passed to bpf_key_put() when done
* with it, so that the key reference count is decremented and the
* bpf_key structure is freed.
*
* Permission checks are deferred to the time the key is used by
* one of the available key-specific kfuncs.
*
* Set *flags* with KEY_LOOKUP_CREATE, to attempt creating a requested
* special keyring (e.g. session keyring), if it doesn't yet exist.
* Set *flags* with KEY_LOOKUP_PARTIAL, to lookup a key without waiting
* for the key construction, and to retrieve uninstantiated keys (keys
* without data attached to them).
*
* Return: a bpf_key pointer with a valid key pointer if the key is found, a
* NULL pointer otherwise.
*/
__bpf_kfunc struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags)
{
key_ref_t key_ref;
struct bpf_key *bkey;
if (flags & ~KEY_LOOKUP_ALL)
return NULL;
/*
* Permission check is deferred until the key is used, as the
* intent of the caller is unknown here.
*/
key_ref = lookup_user_key(serial, flags, KEY_DEFER_PERM_CHECK);
if (IS_ERR(key_ref))
return NULL;
bkey = kmalloc_obj(*bkey);
if (!bkey) {
key_put(key_ref_to_ptr(key_ref));
return NULL;
}
bkey->key = key_ref_to_ptr(key_ref);
bkey->has_ref = true;
return bkey;
}
/**
* bpf_lookup_system_key - lookup a key by a system-defined ID
* @id: key ID
*
* Obtain a bpf_key structure with a key pointer set to the passed key ID.
* The key pointer is marked as invalid, to prevent bpf_key_put() from
* attempting to decrement the key reference count on that pointer. The key
* pointer set in such way is currently understood only by
* verify_pkcs7_signature().
*
* Set *id* to one of the values defined in include/linux/verification.h:
* 0 for the primary keyring (immutable keyring of system keys);
* VERIFY_USE_SECONDARY_KEYRING for both the primary and secondary keyring
* (where keys can be added only if they are vouched for by existing keys
* in those keyrings); VERIFY_USE_PLATFORM_KEYRING for the platform
* keyring (primarily used by the integrity subsystem to verify a kexec'ed
* kerned image and, possibly, the initramfs signature).
*
* Return: a bpf_key pointer with an invalid key pointer set from the
* pre-determined ID on success, a NULL pointer otherwise
*/
__bpf_kfunc struct bpf_key *bpf_lookup_system_key(u64 id)
{
struct bpf_key *bkey;
if (system_keyring_id_check(id) < 0)
return NULL;
bkey = kmalloc_obj(*bkey, GFP_ATOMIC);
if (!bkey)
return NULL;
bkey->key = (struct key *)(unsigned long)id;
bkey->has_ref = false;
return bkey;
}
/**
* bpf_key_put - decrement key reference count if key is valid and free bpf_key
* @bkey: bpf_key structure
*
* Decrement the reference count of the key inside *bkey*, if the pointer
* is valid, and free *bkey*.
*/
__bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
{
if (bkey->has_ref)
key_put(bkey->key);
kfree(bkey);
}
/**
* bpf_verify_pkcs7_signature - verify a PKCS#7 signature
* @data_p: data to verify
* @sig_p: signature of the data
* @trusted_keyring: keyring with keys trusted for signature verification
*
* Verify the PKCS#7 signature *sig_ptr* against the supplied *data_ptr*
* with keys in a keyring referenced by *trusted_keyring*.
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const void *data, *sig;
u32 data_len, sig_len;
int ret;
if (trusted_keyring->has_ref) {
/*
* Do the permission check deferred in bpf_lookup_user_key().
* See bpf_lookup_user_key() for more details.
*
* A call to key_task_permission() here would be redundant, as
* it is already done by keyring_search() called by
* find_asymmetric_key().
*/
ret = key_validate(trusted_keyring->key);
if (ret < 0)
return ret;
}
data_len = __bpf_dynptr_size(data_ptr);
data = __bpf_dynptr_data(data_ptr, data_len);
sig_len = __bpf_dynptr_size(sig_ptr);
sig = __bpf_dynptr_data(sig_ptr, sig_len);
return verify_pkcs7_signature(data, data_len, sig, sig_len,
trusted_keyring->key,
VERIFYING_BPF_SIGNATURE, NULL,
NULL);
#else
return -EOPNOTSUPP;
#endif /* CONFIG_SYSTEM_DATA_VERIFICATION */
}
#endif /* CONFIG_KEYS */
typedef int (*bpf_task_work_callback_t)(struct bpf_map *map, void *key, void *value);
enum bpf_task_work_state {
/* bpf_task_work is ready to be used */
BPF_TW_STANDBY = 0,
/* irq work scheduling in progress */
BPF_TW_PENDING,
/* task work scheduling in progress */
BPF_TW_SCHEDULING,
/* task work is scheduled successfully */
BPF_TW_SCHEDULED,
/* callback is running */
BPF_TW_RUNNING,
/* associated BPF map value is deleted */
BPF_TW_FREED,
};
struct bpf_task_work_ctx {
enum bpf_task_work_state state;
refcount_t refcnt;
struct callback_head work;
struct irq_work irq_work;
/* bpf_prog that schedules task work */
struct bpf_prog *prog;
/* task for which callback is scheduled */
struct task_struct *task;
/* the map and map value associated with this context */
struct bpf_map *map;
void *map_val;
enum task_work_notify_mode mode;
bpf_task_work_callback_t callback_fn;
struct rcu_head rcu;
} __aligned(8);
/* Actual type for struct bpf_task_work */
struct bpf_task_work_kern {
struct bpf_task_work_ctx *ctx;
};
static void bpf_task_work_ctx_reset(struct bpf_task_work_ctx *ctx)
{
if (ctx->prog) {
bpf_prog_put(ctx->prog);
ctx->prog = NULL;
}
if (ctx->task) {
bpf_task_release(ctx->task);
ctx->task = NULL;
}
}
static bool bpf_task_work_ctx_tryget(struct bpf_task_work_ctx *ctx)
{
return refcount_inc_not_zero(&ctx->refcnt);
}
static void bpf_task_work_destroy(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_ctx_reset(ctx);
kfree_rcu(ctx, rcu);
}
static void bpf_task_work_ctx_put(struct bpf_task_work_ctx *ctx)
{
if (!refcount_dec_and_test(&ctx->refcnt))
return;
if (irqs_disabled()) {
ctx->irq_work = IRQ_WORK_INIT(bpf_task_work_destroy);
irq_work_queue(&ctx->irq_work);
} else {
bpf_task_work_destroy(&ctx->irq_work);
}
}
static void bpf_task_work_cancel(struct bpf_task_work_ctx *ctx)
{
/*
* Scheduled task_work callback holds ctx ref, so if we successfully
* cancelled, we put that ref on callback's behalf. If we couldn't
* cancel, callback will inevitably run or has already completed
* running, and it would have taken care of its ctx ref itself.
*/
if (task_work_cancel(ctx->task, &ctx->work))
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_callback(struct callback_head *cb)
{
struct bpf_task_work_ctx *ctx = container_of(cb, struct bpf_task_work_ctx, work);
enum bpf_task_work_state state;
u32 idx;
void *key;
/* Read lock is needed to protect ctx and map key/value access */
guard(rcu_tasks_trace)();
/*
* This callback may start running before bpf_task_work_irq() switched to
* SCHEDULED state, so handle both transition variants SCHEDULING|SCHEDULED -> RUNNING.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_RUNNING);
if (state == BPF_TW_SCHEDULED)
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULED, BPF_TW_RUNNING);
if (state == BPF_TW_FREED) {
bpf_task_work_ctx_put(ctx);
return;
}
key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx);
migrate_disable();
ctx->callback_fn(ctx->map, key, ctx->map_val);
migrate_enable();
bpf_task_work_ctx_reset(ctx);
(void)cmpxchg(&ctx->state, BPF_TW_RUNNING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_irq(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
enum bpf_task_work_state state;
int err;
guard(rcu)();
if (cmpxchg(&ctx->state, BPF_TW_PENDING, BPF_TW_SCHEDULING) != BPF_TW_PENDING) {
bpf_task_work_ctx_put(ctx);
return;
}
err = task_work_add(ctx->task, &ctx->work, ctx->mode);
if (err) {
bpf_task_work_ctx_reset(ctx);
/*
* try to switch back to STANDBY for another task_work reuse, but we might have
* gone to FREED already, which is fine as we already cleaned up after ourselves
*/
(void)cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
return;
}
/*
* It's technically possible for just scheduled task_work callback to
* complete running by now, going SCHEDULING -> RUNNING and then
* dropping its ctx refcount. Instead of capturing an extra ref just
* to protect below ctx->state access, we rely on rcu_read_lock
* above to prevent kfree_rcu from freeing ctx before we return.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_SCHEDULED);
if (state == BPF_TW_FREED)
bpf_task_work_cancel(ctx); /* clean up if we switched into FREED state */
}
static struct bpf_task_work_ctx *bpf_task_work_fetch_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_kern *twk = (void *)tw;
struct bpf_task_work_ctx *ctx, *old_ctx;
ctx = READ_ONCE(twk->ctx);
if (ctx)
return ctx;
ctx = bpf_map_kmalloc_nolock(map, sizeof(*ctx), 0, NUMA_NO_NODE);
if (!ctx)
return ERR_PTR(-ENOMEM);
memset(ctx, 0, sizeof(*ctx));
refcount_set(&ctx->refcnt, 1); /* map's own ref */
ctx->state = BPF_TW_STANDBY;
old_ctx = cmpxchg(&twk->ctx, NULL, ctx);
if (old_ctx) {
/*
* tw->ctx is set by concurrent BPF program, release allocated
* memory and try to reuse already set context.
*/
kfree_nolock(ctx);
return old_ctx;
}
return ctx; /* Success */
}
static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_ctx *ctx;
/*
* Sleepable BPF programs hold rcu_read_lock_trace but not
* regular rcu_read_lock. Since kfree_rcu waits for regular
* RCU GP, the ctx can be freed while we're between reading
* the pointer and incrementing the refcount. Take regular
* rcu_read_lock to prevent kfree_rcu from freeing the ctx
* before we can tryget it.
*/
scoped_guard(rcu) {
ctx = bpf_task_work_fetch_ctx(tw, map);
if (IS_ERR(ctx))
return ctx;
/* try to get ref for task_work callback to hold */
if (!bpf_task_work_ctx_tryget(ctx))
return ERR_PTR(-EBUSY);
}
if (cmpxchg(&ctx->state, BPF_TW_STANDBY, BPF_TW_PENDING) != BPF_TW_STANDBY) {
/* lost acquiring race or map_release_uref() stole it from us, put ref and bail */
bpf_task_work_ctx_put(ctx);
return ERR_PTR(-EBUSY);
}
/*
* If no process or bpffs is holding a reference to the map, no new callbacks should be
* scheduled. This does not address any race or correctness issue, but rather is a policy
* choice: dropping user references should stop everything.
*/
if (!atomic64_read(&map->usercnt)) {
/* drop ref we just got for task_work callback itself */
bpf_task_work_ctx_put(ctx);
/* transfer map's ref into cancel_and_free() */
bpf_task_work_cancel_and_free(tw);
return ERR_PTR(-EBUSY);
}
return ctx;
}
static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw,
struct bpf_map *map, bpf_task_work_callback_t callback_fn,
struct bpf_prog_aux *aux, enum task_work_notify_mode mode)
{
struct bpf_prog *prog;
struct bpf_task_work_ctx *ctx;
int err;
BTF_TYPE_EMIT(struct bpf_task_work);
prog = bpf_prog_inc_not_zero(aux->prog);
if (IS_ERR(prog))
return -EBADF;
task = bpf_task_acquire(task);
if (!task) {
err = -EBADF;
goto release_prog;
}
ctx = bpf_task_work_acquire_ctx(tw, map);
if (IS_ERR(ctx)) {
err = PTR_ERR(ctx);
goto release_all;
}
ctx->task = task;
ctx->callback_fn = callback_fn;
ctx->prog = prog;
ctx->mode = mode;
ctx->map = map;
ctx->map_val = (void *)tw - map->record->task_work_off;
init_task_work(&ctx->work, bpf_task_work_callback);
init_irq_work(&ctx->irq_work, bpf_task_work_irq);
irq_work_queue(&ctx->irq_work);
return 0;
release_all:
bpf_task_release(task);
release_prog:
bpf_prog_put(prog);
return err;
}
/**
* bpf_task_work_schedule_signal - Schedule BPF callback using task_work_add with TWA_SIGNAL
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL);
}
/**
* bpf_task_work_schedule_resume - Schedule BPF callback using task_work_add with TWA_RESUME
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME);
}
static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
struct bpf_dynptr_kern *ptr)
{
struct bpf_dynptr_file_impl *state;
/* flags is currently unsupported */
if (flags) {
bpf_dynptr_set_null(ptr);
return -EINVAL;
}
state = kmalloc_nolock(sizeof(*state), 0, NUMA_NO_NODE);
if (!state) {
bpf_dynptr_set_null(ptr);
return -ENOMEM;
}
state->offset = 0;
state->size = U64_MAX; /* Don't restrict size, as file may change anyways */
freader_init_from_file(&state->freader, NULL, 0, file, may_sleep);
bpf_dynptr_init(ptr, state, BPF_DYNPTR_TYPE_FILE, 0, 0);
bpf_dynptr_set_rdonly(ptr);
return 0;
}
__bpf_kfunc int bpf_dynptr_from_file(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, false, (struct bpf_dynptr_kern *)ptr__uninit);
}
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, true, (struct bpf_dynptr_kern *)ptr__uninit);
}
__bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)dynptr;
struct bpf_dynptr_file_impl *df = ptr->data;
if (!df)
return 0;
freader_cleanup(&df->freader);
kfree_nolock(df);
bpf_dynptr_set_null(ptr);
return 0;
}
/**
* bpf_timer_cancel_async - try to deactivate a timer
* @timer: bpf_timer to stop
*
* Returns:
*
* * 0 when the timer was not active
* * 1 when the timer was active
* * -1 when the timer is currently executing the callback function and
* cannot be stopped
* * -ECANCELED when the timer will be cancelled asynchronously
* * -ENOMEM when out of memory
* * -EINVAL when the timer was not initialized
* * -ENOENT when this kfunc is racing with timer deletion
*/
__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer)
{
struct bpf_async_kern *async = (void *)timer;
struct bpf_async_cb *cb;
int ret;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
/*
* Unlike hrtimer_start() it's ok to synchronously call
* hrtimer_try_to_cancel() when refcnt reached zero, but deferring to
* irq_work is not, since irq callback may execute after RCU GP and
* cb could be freed at that time. Check for refcnt zero for
* consistency.
*/
if (!refcount_inc_not_zero(&cb->refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
ret = hrtimer_try_to_cancel(&t->timer);
bpf_async_refcount_put(cb);
return ret;
} else {
ret = bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
return ret ? ret : -ECANCELED;
}
}
__bpf_kfunc_end_defs();
static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_cancel(ctx); /* this might put task_work callback's ref */
bpf_task_work_ctx_put(ctx); /* and here we put map's own ref that was transferred to us */
}
void bpf_task_work_cancel_and_free(void *val)
{
struct bpf_task_work_kern *twk = val;
struct bpf_task_work_ctx *ctx;
enum bpf_task_work_state state;
ctx = xchg(&twk->ctx, NULL);
if (!ctx)
return;
state = xchg(&ctx->state, BPF_TW_FREED);
if (state == BPF_TW_SCHEDULED) {
/* run in irq_work to avoid locks in NMI */
init_irq_work(&ctx->irq_work, bpf_task_work_cancel_scheduled);
irq_work_queue(&ctx->irq_work);
return;
}
bpf_task_work_ctx_put(ctx); /* put bpf map's ref */
}
BTF_KFUNCS_START(generic_btf_ids)
#ifdef CONFIG_CRASH_DUMP
BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
#endif
BTF_ID_FLAGS(func, bpf_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_percpu_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_refcount_acquire, KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE | KF_RET_NULL | KF_RCU)
BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_front_impl)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_back_impl)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_rbtree_add_impl)
BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_cgroup_ancestor, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU)
BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
#endif
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_send_signal_task)
#endif
#ifdef CONFIG_KEYS
BTF_ID_FLAGS(func, bpf_lookup_user_key, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_lookup_system_key, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_key_put, KF_RELEASE)
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
BTF_ID_FLAGS(func, bpf_verify_pkcs7_signature, KF_SLEEPABLE)
#endif
#endif
#ifdef CONFIG_S390
BTF_ID_FLAGS(func, bpf_get_lowcore)
#endif
BTF_KFUNCS_END(generic_btf_ids)
static const struct btf_kfunc_id_set generic_kfunc_set = {
.owner = THIS_MODULE,
.set = &generic_btf_ids,
};
BTF_ID_LIST(generic_dtor_ids)
BTF_ID(struct, task_struct)
BTF_ID(func, bpf_task_release_dtor)
#ifdef CONFIG_CGROUPS
BTF_ID(struct, cgroup)
BTF_ID(func, bpf_cgroup_release_dtor)
#endif
BTF_KFUNCS_START(common_btf_ids)
BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rcu_read_lock)
BTF_ID_FLAGS(func, bpf_rcu_read_unlock)
BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_css_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_css_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_destroy, KF_ITER_DESTROY)
#endif
BTF_ID_FLAGS(func, bpf_iter_task_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_dynptr_adjust)
BTF_ID_FLAGS(func, bpf_dynptr_is_null)
BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
BTF_ID_FLAGS(func, bpf_dynptr_size)
BTF_ID_FLAGS(func, bpf_dynptr_clone)
BTF_ID_FLAGS(func, bpf_dynptr_copy)
BTF_ID_FLAGS(func, bpf_dynptr_memset)
#ifdef CONFIG_NET
BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
#endif
BTF_ID_FLAGS(func, bpf_wq_init)
BTF_ID_FLAGS(func, bpf_wq_set_callback, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_wq_start)
BTF_ID_FLAGS(func, bpf_preempt_disable)
BTF_ID_FLAGS(func, bpf_preempt_enable)
BTF_ID_FLAGS(func, bpf_iter_bits_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_bits_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_bits_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_copy_from_user_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_get_kmem_cache)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_local_irq_save)
BTF_ID_FLAGS(func, bpf_local_irq_restore)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr)
BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE)
#endif
#ifdef CONFIG_DMA_SHARED_BUFFER
BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
#endif
BTF_ID_FLAGS(func, __bpf_trap)
BTF_ID_FLAGS(func, bpf_strcmp);
BTF_ID_FLAGS(func, bpf_strcasecmp);
BTF_ID_FLAGS(func, bpf_strncasecmp);
BTF_ID_FLAGS(func, bpf_strchr);
BTF_ID_FLAGS(func, bpf_strchrnul);
BTF_ID_FLAGS(func, bpf_strnchr);
BTF_ID_FLAGS(func, bpf_strrchr);
BTF_ID_FLAGS(func, bpf_strlen);
BTF_ID_FLAGS(func, bpf_strnlen);
BTF_ID_FLAGS(func, bpf_strspn);
BTF_ID_FLAGS(func, bpf_strcspn);
BTF_ID_FLAGS(func, bpf_strstr);
BTF_ID_FLAGS(func, bpf_strcasestr);
BTF_ID_FLAGS(func, bpf_strnstr);
BTF_ID_FLAGS(func, bpf_strncasestr);
#if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS)
BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU)
#endif
BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_dynptr_from_file)
BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
BTF_ID_FLAGS(func, bpf_timer_cancel_async)
BTF_KFUNCS_END(common_btf_ids)
static const struct btf_kfunc_id_set common_kfunc_set = {
.owner = THIS_MODULE,
.set = &common_btf_ids,
};
static int __init kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc generic_dtors[] = {
{
.btf_id = generic_dtor_ids[0],
.kfunc_btf_id = generic_dtor_ids[1]
},
#ifdef CONFIG_CGROUPS
{
.btf_id = generic_dtor_ids[2],
.kfunc_btf_id = generic_dtor_ids[3]
},
#endif
};
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &generic_kfunc_set);
ret = ret ?: register_btf_id_dtor_kfuncs(generic_dtors,
ARRAY_SIZE(generic_dtors),
THIS_MODULE);
return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
}
late_initcall(kfunc_init);
/* Get a pointer to dynptr data up to len bytes for read only access. If
* the dynptr doesn't have continuous data up to len bytes, return NULL.
*/
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u64 len)
{
const struct bpf_dynptr *p = (struct bpf_dynptr *)ptr;
return bpf_dynptr_slice(p, 0, NULL, len);
}
/* Get a pointer to dynptr data up to len bytes for read write access. If
* the dynptr doesn't have continuous data up to len bytes, or the dynptr
* is read only, return NULL.
*/
void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u64 len)
{
if (__bpf_dynptr_is_rdonly(ptr))
return NULL;
return (void *)__bpf_dynptr_data(ptr, len);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *val)
{
if (btf_record_has_field(map->record, BPF_TIMER))
bpf_obj_free_timer(map->record, val);
if (btf_record_has_field(map->record, BPF_WORKQUEUE))
bpf_obj_free_workqueue(map->record, val);
if (btf_record_has_field(map->record, BPF_TASK_WORK))
bpf_obj_free_task_work(map->record, val);
}
]
|
| 577/6 |
2026/05/18 19:34 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 75734
output: 37
thoughts: 684
Thoughts: **Vulnerability Analysis of `bpf_bprintf_prepare`**
Okay, I've thoroughly reviewed the code. My primary concern, as suspected, lies within the `bpf_bprintf_prepare` function, specifically within `kernel/bpf/helpers.c`. I can confirm the only uses of `snprintf` in the file are located at lines 924 and 965, both within this function, as I'd expect given the description.
The issue stems from how `snprintf`'s return value, `err`, is handled when truncation occurs. `err` is the number of characters *written* (not necessarily the number of characters that *would* have been written). If `snprintf` truncates (meaning it attempts to write more than the provided buffer size), `err` will be at least as big as the buffer size. The code then increments `tmp_buf` by `err + 1`.
The critical flaw surfaces when `err` is greater than or equal to the remaining buffer space (`tmp_buf_end - tmp_buf`). In this scenario, `tmp_buf` will advance past the end of the allocated buffer. Subsequent writes using `tmp_buf` will then become out-of-bounds, opening the door for exploitation.
Let me be precise, for example: At line 991, the code calls `bpf_trace_copy_string`, passing `tmp_buf`, a potentially invalid `unsafe_ptr`, the format type `fmt_ptype`, and the remaining buffer size `tmp_buf_end - tmp_buf`. If `tmp_buf` is now beyond `tmp_buf_end`, then the subtraction results in a massive positive `size_t` value, leading to a substantial buffer overflow within `bpf_trace_copy_string`. This is a classic overflow vulnerability.
At line 1012, dereferencing `tmp_buf` directly leads to an out-of-bounds write. Line 1047, is another direct out-of-bounds write. Both of these are immediately exploitable.
My initial thought was to see if `bstr_printf` was anywhere else, which I did. Line 1084 uses `____bpf_snprintf`, which I should revisit and determine whether the same issue exists in that function. It is used at `bstr_printf(str, str_size, fmt, data.bin_args)`. It doesn't look like `bstr_printf` does the same buffer overrun checks and it looks like a different vulnerability entirely.
I should search for all uses of `bstr_printf` to make absolutely certain, but my initial assessment stands.
|
| 578/6 |
2026/05/18 19:35 |
tool |
codesearch-find-references |
0m
Args: map[Name:bstr_printf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:____bpf_snprintf SourceFile:kernel/bpf/helpers.c SourceLine:1084]] TruncatedOutput:false]
|
| 579/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 77106
output: 34
thoughts: 81
Thoughts: Okay, here's my thought process on this.
**Streamlining Format String Handling**
Alright, the core requirement is to minimize calls to `bstr_printf`. I remember we had some issues with that function before, so I'm trying to be mindful of its usage. First thought is, can I consolidate everything into a single call?
Now, the `vsnprintf` function comes to mind. I poked at it before, but the results were...unclear. I should re-examine that function, but before I dive back down that rabbit hole, I should ensure I haven't missed any other similar formatting functions floating around. Let me quickly broaden the net, thinking about related functions in the standard library. I should search for `vscnprintf`, `scnprintf`, `sprintf`, `vsprintf` as they could have potential application, especially when combined in a single call.
Okay, let's cast a wider net. I'm going to do a global search for "printf" throughout the file, just to be absolutely certain I'm not overlooking an existing, potentially more efficient, implementation. Better safe than sorry when optimizing this kind of stuff.
|
| 580/6 |
2026/05/18 19:35 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:printf SourceTreePrefix:kernel/bpf/helpers.c]
Results: map[References:<nil> TruncatedOutput:false]
|
| 581/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 77486
output: 32
thoughts: 41
Thoughts: Okay, here's my take on this thought process:
**Hunting Down the Source of a Printing Function**
Right, so `printf` itself isn't a defined entity, which is a bit of a curveball. That means the debugging journey is going to involve something else. So, I need to start thinking about the context. If it's not `printf`, and we are working in the context of the kernel and BPF, there's a strong chance we're looking at something like `snprintf` instead, because `printf` is potentially unsafe for the kernel. I should look into all the other occurrences of `snprintf` used in the `kernel/bpf/` directory. Gotta make sure nothing else is using this string formatting function.
|
| 582/6 |
2026/05/18 19:35 |
tool |
codesearch-find-references |
0m
Args: map[Name:snprintf SourceTreePrefix:kernel/bpf/]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__func_get_name SourceFile:kernel/bpf/disasm.c SourceLine:36] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__func_get_name SourceFile:kernel/bpf/disasm.c SourceLine:38] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:__func_imm_name SourceFile:kernel/bpf/disasm.c SourceLine:50] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:924] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:965] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_fmt_stack_mask SourceFile:kernel/bpf/backtrack.c SourceLine:239] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_prog_ksym_set_name SourceFile:kernel/bpf/core.c SourceLine:581] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_prog_ksym_set_name SourceFile:kernel/bpf/core.c SourceLine:589] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_prog_ksym_set_name SourceFile:kernel/bpf/core.c SourceLine:594] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_struct_ops_ksym_init SourceFile:kernel/bpf/bpf_struct_ops.c SourceLine:647] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_tramp_image_alloc SourceFile:kernel/bpf/trampoline.c SourceLine:592] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_check_iter_kfuncs SourceFile:kernel/bpf/btf.c SourceLine:8662] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_nested_type_is_trusted SourceFile:kernel/bpf/btf.c SourceLine:9592] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_show_name SourceFile:kernel/bpf/btf.c SourceLine:1215] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:btf_show_name SourceFile:kernel/bpf/btf.c SourceLine:1218] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:find_kfunc_impl_proto SourceFile:kernel/bpf/verifier.c SourceLine:2912] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:fmt_instance SourceFile:kernel/bpf/liveness.c SourceLine:419] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:fmt_reg_mask SourceFile:kernel/bpf/backtrack.c SourceLine:220] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:fmt_subprog SourceFile:kernel/bpf/liveness.c SourceLine:412] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:format_callchain SourceFile:kernel/bpf/states.c SourceLine:144] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:format_callchain SourceFile:kernel/bpf/states.c SourceLine:148] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:format_callchain SourceFile:kernel/bpf/states.c SourceLine:151] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:reg_type_str SourceFile:kernel/bpf/log.c SourceLine:435] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:reg_type_str SourceFile:kernel/bpf/log.c SourceLine:445] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:543] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:545] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:547] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:tnum_strn SourceFile:kernel/bpf/log.c SourceLine:549]] TruncatedOutput:false]
|
| 583/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 79204
output: 40
thoughts: 702
Thoughts: **Analyzing String Formatting in `kernel/bpf/helpers.c`**
Okay, so the prompt is zeroing in on `kernel/bpf/helpers.c`. My initial thought is that I should quickly check the source for uses of `snprintf` because it's the obvious one that comes to mind, and I find only two instances. But, that's not enough; I need to be more thorough here.
I need to expand my search. Let's see. I'll check for all standard C formatting functions. I'll scan for `scnprintf`, `vsnprintf`, `sprintf`, `vscnprintf`, and `seq_printf`. Yep, checked those. Nothing else. How about `bstr_printf`? Nope, not there either.
I need to think more broadly: are there *other* functions that format strings into a buffer within this file? Ah, what about `bpf_trace_copy_string`? Bingo! It's using `strncpy_from_user_nofault` and `strncpy_from_kernel_nofault`.
Now I need to understand how *those* functions work in relation to `bpf_trace_copy_string`. I see that `bpf_trace_copy_string` uses `strncpy_from_user_nofault` and `strncpy_from_kernel_nofault` depending on a `fmt_ptype` character. And the important thing is that `strncpy_from_kernel_nofault` (and `strncpy_from_user_nofault`) return the number of characters copied, *excluding* the NUL terminator, or a negative error value.
Looking at how it's used in `bpf_bprintf_prepare`, if the return from the copy is positive, it's the number of bytes written. Good. I see it's bounded by `tmp_buf_end - tmp_buf`, which looks like they are correctly handling the buffer size.
Now I need to be *absolutely* sure what these functions are actually returning. It would be easy to misread this. This is the critical piece, especially with `strncpy_from_kernel_nofault` (which might be an alias to something else in newer kernels, like `strscpy_from_kernel_nofault`). I'm going to double check the exact definition of `strncpy_from_kernel_nofault`.
|
| 584/6 |
2026/05/18 19:35 |
tool |
codesearch-definition-comment |
0m
Args: map[ContextFile:kernel/bpf/helpers.c Name:strncpy_from_kernel_nofault]
Results: map[Comment: Kind:function]
|
| 585/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 80478
output: 45
thoughts: 7
Thoughts: Okay, here's the summary, assuming I'm an expert in the field and the "source" is something relevant to my expertise:
**Checking the Foundation**
Alright, before jumping to any conclusions or forming any opinions, the first and most critical step is always verification. I need to go back to the source material. What is this based on? Is it a primary source? A secondary interpretation? How reliable is it, given the context? What methodology was used to obtain the data, if any? I need to understand the fundamental building blocks before I can even begin to think about what this *means*. Context is everything in my work, and the original provenance of an idea or claim is the very foundation upon which I build my analysis. Let's see what we're working with.
|
| 586/6 |
2026/05/18 19:35 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:strncpy_from_kernel_nofault]
Results: map[SourceCode: 86: long strncpy_from_kernel_nofault(char *dst, const void *unsafe_addr, long count)
87: {
88: const void *src = unsafe_addr;
89:
90: if (unlikely(count <= 0))
91: return 0;
92: if (!copy_from_kernel_nofault_allowed(unsafe_addr, count))
93: return -ERANGE;
94:
95: pagefault_disable();
96: do {
97: __get_kernel_nofault(dst, src, u8, Efault);
98: dst++;
99: src++;
100: } while (dst[-1] && src - unsafe_addr < count);
101: pagefault_enable();
102:
103: dst[-1] = '\0';
104: return src - unsafe_addr;
105: Efault:
106: pagefault_enable();
107: dst[0] = '\0';
108: return -EFAULT;
109: }
SourceFile:mm/maccess.c]
|
| 587/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 81039
output: 26
thoughts: 522
Thoughts: **Analyzing String Copying and Formatting in the BPF Context**
Okay, let's break this down. My initial observation is focused on how string copying behaves, specifically within the kernel's BPF environment. The `bpf_trace_copy_string` function, and by extension the `strncpy_from_kernel_nofault` function it uses under the hood, seem to operate reasonably, respecting the specified `count` parameter. The goal is to copy a string safely from kernel space, taking into account the potential for faults, which is a good safety measure.
If the loop in `strncpy_from_kernel_nofault` finishes when the number of copied bytes equals `count`, it means the copy completed successfully within the specified bounds, so it returns `count`. But there's a specific case when `count` is 1 that needs to be scrutinized. It looks like it forces the last byte copied to be a null terminator (`\0`), regardless, effectively making sure the buffer will always be null-terminated. This happens when the loop only runs once and then ensures the last byte written is null.
My primary concern, however, revolves around the `snprintf` function's behavior in this context, specifically in relation to the `bufsz` parameter. The documentation mentions that the return value of `snprintf` doesn't always reflect the buffer size properly, and this is where potential issues could arise. I'm verifying this by checking all calls to `snprintf` within `kernel/bpf/helpers.c`.
That said, I recall the discussion on `bpf_snprintf_btf`. This is a crucial element that I mustn't ignore! I need to ensure there are no surprises with the BTF (BPF Type Format) related `snprintf` logic. This is where the initial reasoning loop came in - I was looking for the implementation of `bpf_snprintf_btf` in the usual place, but found it wasn't there. So, I am checking where `bpf_snprintf_btf` is defined; it seems to be in a separate BTF handling module.
|
| 588/6 |
2026/05/18 19:35 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf]
Results: map[References:<nil> TruncatedOutput:false]
|
| 589/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 82056
output: 27
thoughts: 33
Thoughts: Here's my take on the thought process:
**Diving into that BPF Helper**
Okay, hold on a second. I'm looking at `bpf_snprintf_btf`. Right, it's a BPF helper, isn't it? So that immediately means its actual implementation, the function we'd look for in the low-level code, is going to be `____bpf_snprintf_btf`. That's the convention, right? The double underscores are a pretty standard indicator for BPF helpers' underlying functions.
|
| 590/6 |
2026/05/18 19:35 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:____bpf_snprintf_btf]
Results: map[References:<nil> TruncatedOutput:false]
|
| 591/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 82257
output: 28
thoughts: 89
Thoughts: Okay, let's dive back into this `bpf_base_func_proto` situation, specifically focusing on the `kernel/bpf/helpers.c` file. I need to refresh my understanding of how `bpf_snprintf_btf` is handled.
The code snippet shows a clear branching scenario within the `bpf_base_func_proto` function. It appears to be a dispatch table, handling various BPF functions based on their type, and I'm interested in line 1198 and 1199, where the `BPF_FUNC_snprintf_btf` case is declared. The function seems to be returning a pointer to `bpf_snprintf_btf_proto` when this specific case is encountered.
Now, the critical question, the one I need answered: Where exactly is `bpf_snprintf_btf_proto` defined? I need to trace back from this point. Is it a static definition within this file, or is it declared elsewhere, possibly in a header? Is it part of the core BPF infrastructure, or something specific to BTF handling? This is the crucial next step to understand the context.
|
| 592/6 |
2026/05/18 19:35 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf_proto]
Results: map[References:<nil> TruncatedOutput:false]
|
| 593/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 82649
output: 35
thoughts: 24
Thoughts: Okay, here's my summary, tailored for an expert audience, as if it's coming directly from my own thought process:
**Diving into the bpf_snprintf_btf_proto Codebase**
Alright, time to get to work. My initial strategy is clear: I need to understand how `bpf_snprintf_btf_proto` functions are being used throughout this whole project. Given the nature of BTF and the complexities of BPF, I suspect this function plays a critical role in some form of debugging, printing, or perhaps even in generating the human-readable representations of BPF programs. To do this comprehensively and avoid missing anything, I will start with a *very* broad search. I'm going to launch a search across the entire codebase for the literal string `"bpf_snprintf_btf_proto"`. This will include every file, including test files, documentation, and potentially generated code. Once I have the results, I can start filtering by context. This will give me a comprehensive understanding of where and how this function comes into play. It's a brute-force approach, but it is necessary to get a full view of the use-cases and prevent oversight.
|
| 594/6 |
2026/05/18 19:35 |
tool |
codesearch-find-references |
0m
Error:
Args: map[Name:bpf_snprintf_btf_proto SourceTreePrefix:]
Results: map[References:<nil> TruncatedOutput:false]
|
| 595/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 82984
output: 27
thoughts: 105
Thoughts: **Debugging a Reference Search**
This is perplexing. I was relying on `codesearch-find-references`, and it's letting me down. This isn't the expected behavior at all, and it's throwing off my flow. I usually go straight to that when I'm chasing something down.
Okay, I could fall back on `grep`, but since that's not directly available, I'll need to improvise. I could use `codesearch-dir-index` and read the files directly, which is a bit slower, or I could just trust the previous information that I'd gleaned. It's a trade-off: speed vs. verification.
Wait... I wonder if there's some sort of edge case I'm missing with `codesearch-find-references`. Can I use it to search for an empty string, just to see what the general results are? No, that's not going to fly.
Alright, let's get back to what I was trying to do. I need to get some more info on `btf_type_snprintf_show`. It was mentioned in that previous summary, so let's try a targeted search using that as the key.
|
| 596/6 |
2026/05/18 19:35 |
tool |
codesearch-find-references |
0m
Args: map[Name:btf_type_snprintf_show]
Results: map[References:<nil> TruncatedOutput:false]
|
| 597/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 83390
output: 28
thoughts: 35
Thoughts: Okay, let's see...
**Investigating a Missing Function**
Hold on a second, it seems that `btf_type_snprintf_show` is *also* missing? That's unexpected. I thought that function was defined in relation to the work I was looking at. Now I'm starting to think I need to dig deeper. Perhaps I should go directly to the source code to get a handle on what might be going on, instead of assuming something that may not be true. Time to check `kernel/bpf/btf.c` and see what that module has to say.
|
| 598/6 |
2026/05/18 19:35 |
tool |
codesearch-file-index |
0m
Args: map[SourceFile:kernel/bpf/btf.c]
Results: map[Entities:[map[Kind:function Name:____bpf_btf_find_by_name_kind] map[Kind:function Name:__btf_array_show] map[Kind:function Name:__btf_kfunc_is_allowed] map[Kind:function Name:__btf_name_by_offset] map[Kind:function Name:__btf_name_char_ok] map[Kind:function Name:__btf_new_fd] map[Kind:function Name:__btf_resolve_size] map[Kind:function Name:__btf_show_indent] map[Kind:function Name:__btf_show_obj_safe] map[Kind:function Name:__btf_struct_show] map[Kind:function Name:__btf_type_int_is_regular] map[Kind:function Name:__btf_verifier_log] map[Kind:function Name:__btf_verifier_log_type] map[Kind:function Name:__get_type_fmodel_flags] map[Kind:function Name:__get_type_size] map[Kind:function Name:__print_cand_cache] map[Kind:function Name:__register_bpf_struct_ops] map[Kind:function Name:__register_btf_kfunc_id_set] map[Kind:function Name:bpf_btf_find_by_name_kind] map[Kind:function Name:bpf_btf_show_fdinfo] map[Kind:function Name:bpf_core_add_cands] map[Kind:function Name:bpf_core_apply] map[Kind:function Name:bpf_core_essential_name_len] map[Kind:function Name:bpf_core_find_cands] map[Kind:function Name:bpf_core_is_flavor_sep] map[Kind:function Name:bpf_core_types_are_compat] map[Kind:function Name:bpf_core_types_match] map[Kind:function Name:bpf_find_btf_id] map[Kind:function Name:bpf_free_cands] map[Kind:function Name:bpf_free_cands_from_cache] map[Kind:function Name:bpf_prog_get_target_btf] map[Kind:function Name:bpf_prog_type_to_kfunc_hook] map[Kind:function Name:bpf_struct_ops_find] map[Kind:function Name:bpf_struct_ops_find_value] map[Kind:function Name:btf_add_struct_ops] map[Kind:function Name:btf_add_type] map[Kind:function Name:btf_alloc_id] map[Kind:function Name:btf_array_check_member] map[Kind:function Name:btf_array_check_meta] map[Kind:function Name:btf_array_log] map[Kind:function Name:btf_array_resolve] map[Kind:function Name:btf_array_show] map[Kind:function Name:btf_base_btf] map[Kind:function Name:btf_bitfield_show] map[Kind:function Name:btf_check_all_metas] map[Kind:function Name:btf_check_all_types] map[Kind:function Name:btf_check_and_fixup_fields] map[Kind:function Name:btf_check_dtor_kfuncs] map[Kind:function Name:btf_check_func_type_match] map[Kind:function Name:btf_check_iter_arg] map[Kind:function Name:btf_check_iter_kfuncs] map[Kind:function Name:btf_check_kfunc_protos] map[Kind:function Name:btf_check_meta] map[Kind:function Name:btf_check_sec_info] map[Kind:function Name:btf_check_sorted] map[Kind:function Name:btf_check_type_match] map[Kind:function Name:btf_check_type_tags] map[Kind:function Name:btf_ctx_access] map[Kind:function Name:btf_ctx_arg_idx] map[Kind:function Name:btf_ctx_arg_offset] map[Kind:function Name:btf_datasec_check_meta] map[Kind:function Name:btf_datasec_log] map[Kind:function Name:btf_datasec_resolve] map[Kind:function Name:btf_datasec_show] map[Kind:function Name:btf_decl_tag_check_meta] map[Kind:function Name:btf_decl_tag_log] map[Kind:function Name:btf_decl_tag_resolve] map[Kind:function Name:btf_df_check_kflag_member] map[Kind:function Name:btf_df_check_member] map[Kind:function Name:btf_df_resolve] map[Kind:function Name:btf_df_show] map[Kind:function Name:btf_distill_func_proto] map[Kind:function Name:btf_enum64_check_meta] map[Kind:function Name:btf_enum64_show] map[Kind:function Name:btf_enum_check_kflag_member] map[Kind:function Name:btf_enum_check_member] map[Kind:function Name:btf_enum_check_meta] map[Kind:function Name:btf_enum_log] map[Kind:function Name:btf_enum_show] map[Kind:function Name:btf_field_cmp] map[Kind:function Name:btf_find_by_name_kind] map[Kind:function Name:btf_find_by_name_kind_bsearch] map[Kind:function Name:btf_find_datasec_var] map[Kind:function Name:btf_find_decl_tag_value] map[Kind:function Name:btf_find_dtor_kfunc] map[Kind:function Name:btf_find_field] map[Kind:function Name:btf_find_field_one] map[Kind:function Name:btf_find_graph_root] map[Kind:function Name:btf_find_kptr] map[Kind:function Name:btf_find_nested_struct] map[Kind:function Name:btf_find_next_decl_tag] map[Kind:function Name:btf_find_struct] map[Kind:function Name:btf_find_struct_field] map[Kind:function Name:btf_find_struct_meta] map[Kind:function Name:btf_float_check_member] map[Kind:function Name:btf_float_check_meta] map[Kind:function Name:btf_float_log] map[Kind:function Name:btf_free] map[Kind:function Name:btf_free_dtor_kfunc_tab] map[Kind:function Name:btf_free_id] map[Kind:function Name:btf_free_kfunc_set_tab] map[Kind:function Name:btf_free_rcu] map[Kind:function Name:btf_free_struct_meta_tab] map[Kind:function Name:btf_free_struct_ops_tab] map[Kind:function Name:btf_func_check] map[Kind:function Name:btf_func_check_meta] map[Kind:function Name:btf_func_proto_check] map[Kind:function Name:btf_func_proto_check_meta] map[Kind:function Name:btf_func_proto_log] map[Kind:function Name:btf_func_resolve] map[Kind:function Name:btf_fwd_check_meta] map[Kind:function Name:btf_fwd_type_log] map[Kind:function Name:btf_generic_check_kflag_member] map[Kind:function Name:btf_get] map[Kind:function Name:btf_get_by_fd] map[Kind:function Name:btf_get_fd_by_id] map[Kind:function Name:btf_get_field_type] map[Kind:function Name:btf_get_info_by_fd] map[Kind:function Name:btf_get_module_btf] map[Kind:function Name:btf_get_name] map[Kind:function Name:btf_get_ptr_to_btf_id] map[Kind:function Name:btf_header] map[Kind:function Name:btf_int128_print] map[Kind:function Name:btf_int128_shift] map[Kind:function Name:btf_int_bits_show] map[Kind:function Name:btf_int_check_kflag_member] map[Kind:function Name:btf_int_check_member] map[Kind:function Name:btf_int_check_meta] map[Kind:function Name:btf_int_encoding_str] map[Kind:function Name:btf_int_log] map[Kind:function Name:btf_int_show] map[Kind:function Name:btf_is_dynptr_ptr] map[Kind:function Name:btf_is_kernel] map[Kind:function Name:btf_is_module] map[Kind:function Name:btf_is_prog_ctx_type] map[Kind:function Name:btf_is_projection_of] map[Kind:function Name:btf_is_vmlinux] map[Kind:function Name:btf_kfunc_flags] map[Kind:function Name:btf_kfunc_id_set_contains] map[Kind:function Name:btf_kfunc_is_allowed] map[Kind:function Name:btf_kfunc_is_modify_return] map[Kind:function Name:btf_member_is_reg_int] map[Kind:function Name:btf_modifier_check_kflag_member] map[Kind:function Name:btf_modifier_check_member] map[Kind:function Name:btf_modifier_resolve] map[Kind:function Name:btf_modifier_show] map[Kind:function Name:btf_name_by_offset] map[Kind:function Name:btf_name_offset_valid] map[Kind:function Name:btf_name_valid_identifier] map[Kind:function Name:btf_name_valid_section] map[Kind:function Name:btf_named_start_id] map[Kind:function Name:btf_nested_type_is_trusted] map[Kind:function Name:btf_new_fd] map[Kind:function Name:btf_nr_types] map[Kind:function Name:btf_obj_id] map[Kind:function Name:btf_param_match_suffix] map[Kind:function Name:btf_parse] map[Kind:function Name:btf_parse_base] map[Kind:function Name:btf_parse_fields] map[Kind:function Name:btf_parse_graph_root] map[Kind:function Name:btf_parse_hdr] map[Kind:function Name:btf_parse_kptr] map[Kind:function Name:btf_parse_layout_sec] map[Kind:function Name:btf_parse_list_head] map[Kind:function Name:btf_parse_rb_root] map[Kind:function Name:btf_parse_str_sec] map[Kind:function Name:btf_parse_struct_metas] map[Kind:function Name:btf_parse_type_sec] map[Kind:function Name:btf_parse_vmlinux] map[Kind:function Name:btf_populate_kfunc_set] map[Kind:function Name:btf_prepare_func_args] map[Kind:function Name:btf_ptr_check_member] map[Kind:function Name:btf_ptr_resolve] map[Kind:function Name:btf_ptr_show] map[Kind:function Name:btf_put] map[Kind:function Name:btf_ref_type_check_meta] map[Kind:function Name:btf_ref_type_log] map[Kind:function Name:btf_release] map[Kind:function Name:btf_relocate_id] map[Kind:function Name:btf_repeat_fields] map[Kind:function Name:btf_resolve] map[Kind:function Name:btf_resolve_size] map[Kind:function Name:btf_resolve_valid] map[Kind:function Name:btf_resolved_type_id] map[Kind:function Name:btf_resolved_type_size] map[Kind:function Name:btf_sec_info_cmp] map[Kind:function Name:btf_seq_show] map[Kind:function Name:btf_set_base_btf] map[Kind:function Name:btf_show] map[Kind:function Name:btf_show_delim] map[Kind:function Name:btf_show_end_aggr_type] map[Kind:function Name:btf_show_end_array_member] map[Kind:function Name:btf_show_end_array_type] map[Kind:function Name:btf_show_end_member] map[Kind:function Name:btf_show_end_struct_type] map[Kind:function Name:btf_show_end_type] map[Kind:function Name:btf_show_indent] map[Kind:function Name:btf_show_name] map[Kind:function Name:btf_show_newline] map[Kind:function Name:btf_show_obj_is_safe] map[Kind:function Name:btf_show_obj_safe] map[Kind:function Name:btf_show_obj_size_left] map[Kind:function Name:btf_show_start_aggr_type] map[Kind:function Name:btf_show_start_array_member] map[Kind:function Name:btf_show_start_array_type] map[Kind:function Name:btf_show_start_member] map[Kind:function Name:btf_show_start_struct_type] map[Kind:function Name:btf_show_start_type] map[Kind:function Name:btf_snprintf_show] map[Kind:function Name:btf_start_id] map[Kind:function Name:btf_str_by_offset] map[Kind:function Name:btf_struct_access] map[Kind:function Name:btf_struct_check_member] map[Kind:function Name:btf_struct_check_meta] map[Kind:function Name:btf_struct_ids_match] map[Kind:function Name:btf_struct_log] map[Kind:function Name:btf_struct_metas_free] map[Kind:function Name:btf_struct_resolve] map[Kind:function Name:btf_struct_show] map[Kind:function Name:btf_struct_walk] map[Kind:function Name:btf_translate_to_vmlinux] map[Kind:function Name:btf_try_get_module] map[Kind:function Name:btf_type_array] map[Kind:function Name:btf_type_by_id] map[Kind:function Name:btf_type_decl_tag] map[Kind:function Name:btf_type_enum] map[Kind:function Name:btf_type_enum64] map[Kind:function Name:btf_type_has_size] map[Kind:function Name:btf_type_id_resolve] map[Kind:function Name:btf_type_id_size] map[Kind:function Name:btf_type_ids_nocast_alias] map[Kind:function Name:btf_type_int] map[Kind:function Name:btf_type_int_is_regular] map[Kind:function Name:btf_type_is_datasec] map[Kind:function Name:btf_type_is_decl_tag] map[Kind:function Name:btf_type_is_decl_tag_target] map[Kind:function Name:btf_type_is_i32] map[Kind:function Name:btf_type_is_i64] map[Kind:function Name:btf_type_is_modifier] map[Kind:function Name:btf_type_is_primitive] map[Kind:function Name:btf_type_is_resolve_source_only] map[Kind:function Name:btf_type_is_void] map[Kind:function Name:btf_type_needs_resolve] map[Kind:function Name:btf_type_nosize] map[Kind:function Name:btf_type_nosize_or_null] map[Kind:function Name:btf_type_ops] map[Kind:function Name:btf_type_resolve_func_ptr] map[Kind:function Name:btf_type_resolve_ptr] map[Kind:function Name:btf_type_seq_show] map[Kind:function Name:btf_type_seq_show_flags] map[Kind:function Name:btf_type_show] map[Kind:function Name:btf_type_skip_modifiers] map[Kind:function Name:btf_type_skip_qualifiers] map[Kind:function Name:btf_type_snprintf_show] map[Kind:function Name:btf_type_str] map[Kind:function Name:btf_type_var] map[Kind:function Name:btf_types_are_same] map[Kind:function Name:btf_validate_prog_ctx_type] map[Kind:function Name:btf_var_check_meta] map[Kind:function Name:btf_var_log] map[Kind:function Name:btf_var_resolve] map[Kind:function Name:btf_var_show] map[Kind:function Name:btf_verifier_env_free] map[Kind:function Name:btf_verifier_log] map[Kind:function Name:btf_verifier_log_hdr] map[Kind:function Name:btf_verifier_log_member] map[Kind:function Name:btf_verifier_log_vsi] map[Kind:function Name:check_btf_kconfigs] map[Kind:function Name:check_cand_cache] map[Kind:function Name:env_resolve_init] map[Kind:function Name:env_stack_peak] map[Kind:function Name:env_stack_pop_resolved] map[Kind:function Name:env_stack_push] map[Kind:function Name:env_stack_set_next_member] map[Kind:function Name:env_type_is_resolve_sink] map[Kind:function Name:env_type_is_resolved] map[Kind:function Name:finalize_log] map[Kind:function Name:find_canonical_prog_ctx_type] map[Kind:function Name:find_kern_ctx_type_id] map[Kind:function Name:get_kern_ctx_btf_id] map[Kind:function Name:hash_cands] map[Kind:function Name:populate_cand_cache] map[Kind:function Name:print_cand_cache] map[Kind:function Name:prog_args_trusted] map[Kind:function Name:register_btf_fmodret_id_set] map[Kind:function Name:register_btf_id_dtor_kfuncs] map[Kind:function Name:register_btf_kfunc_id_set] map[Kind:function Name:sizeof_cands] map[Kind:struct Name:bpf_cand_cache] map[Kind:struct Name:bpf_ctx_convert] map[Kind:struct Name:bpf_raw_tp_null_args] map[Kind:struct Name:btf] map[Kind:struct Name:btf_field_info] map[Kind:struct Name:btf_id_dtor_kfunc_tab] map[Kind:struct Name:btf_kfunc_hook_filter] map[Kind:struct Name:btf_kfunc_set_tab] map[Kind:struct Name:btf_kind_operations] map[Kind:struct Name:btf_sec_info] map[Kind:struct Name:btf_show] map[Kind:struct Name:btf_show_snprintf] map[Kind:struct Name:btf_struct_ops_tab] map[Kind:struct Name:btf_verifier_env] map[Kind:struct Name:resolve_vertex] map[Kind:global_variable Name:__UNIQUE_ID_addressable___register_bpf_struct_ops_1014] map[Kind:global_variable Name:__UNIQUE_ID_addressable_bpf_find_btf_id_981] map[Kind:global_variable Name:__UNIQUE_ID_addressable_btf_ctx_access_999] map[Kind:global_variable Name:__UNIQUE_ID_addressable_btf_type_by_id_982] map[Kind:global_variable Name:__UNIQUE_ID_addressable_register_btf_fmodret_id_set_1011] map[Kind:global_variable Name:__UNIQUE_ID_addressable_register_btf_id_dtor_kfuncs_1013] map[Kind:global_variable Name:__UNIQUE_ID_addressable_register_btf_kfunc_id_set_1010] map[Kind:global_variable Name:alloc_obj_fields] map[Kind:global_variable Name:array_ops] map[Kind:global_variable Name:bpf_btf_find_by_name_kind_proto] map[Kind:global_variable Name:bpf_ctx_convert_map] map[Kind:global_variable Name:btf_fops] map[Kind:global_variable Name:btf_idr] map[Kind:global_variable Name:btf_idr_lock] map[Kind:global_variable Name:btf_kind_str] map[Kind:global_variable Name:btf_sec_info_offset] map[Kind:global_variable Name:cand_cache_mutex] map[Kind:global_variable Name:datasec_ops] map[Kind:global_variable Name:decl_tag_ops] map[Kind:global_variable Name:enum64_ops] map[Kind:global_variable Name:enum_ops] map[Kind:global_variable Name:float_ops] map[Kind:global_variable Name:func_ops] map[Kind:global_variable Name:func_proto_ops] map[Kind:global_variable Name:fwd_ops] map[Kind:global_variable Name:int_ops] map[Kind:global_variable Name:kind_ops] map[Kind:global_variable Name:modifier_ops] map[Kind:global_variable Name:ptr_ops] map[Kind:global_variable Name:raw_tp_null_args] map[Kind:global_variable Name:struct_ops] map[Kind:global_variable Name:var_ops] map[Kind:enum Name:bpf_struct_walk_result] map[Kind:enum Name:btf_arg_tag] map[Kind:enum Name:btf_kfunc_hook] map[Kind:enum Name:resolve_mode] map[Kind:enum Name:verifier_phase] map[Kind:enum Name:visit_state] map[Kind:typedef Name:btf_bpf_btf_find_by_name_kind]]]
|
| 599/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 90160
output: 47
thoughts: 33
Thoughts: Here's my summary, as though I'm thinking through this process:
**Diving into BTF Type Printing**
Okay, I've spotted `btf_type_snprintf_show` within `kernel/bpf/btf.c`. Right, that's where the kernel's BTF (BPF Type Format) information is being formatted for display, likely in a human-readable string. Let's dig into the implementation of that function and see what it's doing. I need to understand how these BTF types are being structured into a string representation. I'm especially interested in how it handles the different type kinds, nested structures, and any specific formatting details. This should give me a good handle on how the kernel exposes its type information.
|
| 600/6 |
2026/05/18 19:35 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/btf.c IncludeLines:true Name:btf_type_snprintf_show]
Results: map[SourceCode:8151: int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
8152: char *buf, int len, u64 flags)
8153: {
8154: struct btf_show_snprintf ssnprintf;
8155:
8156: ssnprintf.show.target = buf;
8157: ssnprintf.show.flags = flags;
8158: ssnprintf.show.showfn = btf_snprintf_show;
8159: ssnprintf.len_left = len;
8160: ssnprintf.len = 0;
8161:
8162: btf_type_show(btf, type_id, obj, (struct btf_show *)&ssnprintf);
8163:
8164: /* If we encountered an error, return it. */
8165: if (ssnprintf.show.state.status)
8166: return ssnprintf.show.state.status;
8167:
8168: /* Otherwise return length we would have written */
8169: return ssnprintf.len;
8170: }
SourceFile:kernel/bpf/btf.c]
|
| 601/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 90766
output: 45
thoughts: 10
Thoughts: Okay, here's my attempt at summarizing the thought process, tailored for an expert audience, and written in the first person:
**My Deep Dive into `btf_snprintf_show`**
Alright, let's break down this `btf_snprintf_show` function. My initial impression is that it's likely a utility function designed to format and display BTF (BPF Type Format) data in a human-readable string. Knowing BTF, the complexity is baked in, so let's dig a bit deeper into the presumed design goals:
Firstly, this function is undoubtedly handling a lot of internal data structure traversal and string building. I’m betting it's taking a BTF data structure as input, either directly or indirectly through a context pointer, and then recursively walking through it. The key challenge, as always with BTF, is to handle the various type descriptors: integers, enums, structs, unions, pointers, arrays, function prototypes, typedefs, etc. We all know how quickly this complexity balloons.
Secondly, and perhaps more importantly, the "snprintf" part tells me efficiency is a major concern. It's not just about getting the *right* output; it's about doing it *fast*. This likely implies pre-allocation of a buffer or smart use of a format string to avoid excessive memory allocation and copying. The potential for buffer overflows is definitely something I'd be watching out for, and I would be checking for any size parameters passed to the function, and whether these are properly checked within the formatting logic.
I am also considering what output context or parameters this function is receiving; whether it is a global context or specific data that is being passed in. It is critical to know what parameters, such as a buffer, are being passed to this function, and whether that is a constant, or has a lifecycle. This greatly impacts the type of analysis needed.
I'd be curious to see how the function handles nested structures. Does it use recursion, iteration, or a combination of both? Recursion can be elegant, but it can also lead to stack overflows with deeply nested types. Iteration might be more efficient in some cases, but potentially more complex to code. Another detail is what style of information is presented - such as whether the BTF data is printed with or without debug information (such as line numbers, comments, source file/line information), and whether there are options for concise or full output of the details.
Also, I'd immediately be looking for error handling. How does it deal with corrupted or invalid BTF data? Does it return an error code, print an error message, or potentially trigger some kind of panic or assertion? Robust error handling is crucial for any code interacting with potentially untrusted or malformed input.
Finally, I am thinking that this `snprintf` is a key component to a debugging or introspection tool. This would give me insights to look for other uses of this formatting function in the broader codebase, to understand how it is used. I am assuming other functions use the output generated by `btf_snprintf_show`, perhaps for logging, debugging output, or even for generating code (e.g. for generating C definitions from BTF). This would inform my next course of action.
|
| 602/6 |
2026/05/18 19:35 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/btf.c IncludeLines:true Name:btf_snprintf_show]
Results: map[SourceCode:8129: __printf(2, 0) static void btf_snprintf_show(struct btf_show *show, const char *fmt,
8130: va_list args)
8131: {
8132: struct btf_show_snprintf *ssnprintf = (struct btf_show_snprintf *)show;
8133: int len;
8134:
8135: len = vsnprintf(show->target, ssnprintf->len_left, fmt, args);
8136:
8137: if (len < 0) {
8138: ssnprintf->len_left = 0;
8139: ssnprintf->len = len;
8140: } else if (len >= ssnprintf->len_left) {
8141: /* no space, drive on to get length we would have written */
8142: ssnprintf->len_left = 0;
8143: ssnprintf->len += len;
8144: } else {
8145: ssnprintf->len_left -= len;
8146: ssnprintf->len += len;
8147: show->target += len;
8148: }
8149: }
SourceFile:kernel/bpf/btf.c]
|
| 603/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 91874
output: 23
thoughts: 408
Thoughts: **Analyzing `snprintf` Usage in `kernel/bpf/helpers.c`**
Okay, this looks safe; the logic correctly handles `len_left` and only increments `show->target` when the length is within bounds. Excellent.
Now, back to `kernel/bpf/helpers.c`. I need to be absolutely sure about `snprintf` usage within this file. I've already established that the only direct calls to `snprintf` are within `bpf_bprintf_prepare`.
Let's make sure, are there any *other* uses of `snprintf` in this file? No, I'm certain there aren't.
Alright, what about other functions that format strings into a buffer without proper return value checking? I've checked and I didn't find any.
Hold on a second... what about the `bpf_snprintf` helper function itself? Let me take a closer look:
```c
1064: BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
1065: const void *, args, u32, data_len)
...
1084: err = bstr_printf(str, str_size, fmt, data.bin_args);
...
1088: return err + 1;
```
Right, `bstr_printf` returns the number of bytes that *would* have been written. The helper then returns `err + 1`. This is by design for `bpf_snprintf` (it reports the size of the formatted string, including the null terminator). It doesn't use the return value, `err`, to advance an internal pointer in a way that could lead to an out-of-bounds write, so that is safe.
So, in summary, the answer to the question "Is there any other place in `kernel/bpf/helpers.c` where `snprintf` is used..." is a resounding NO.
Okay, just to be completely thorough, I'll search for `snprintf` in the file again, just to be certain. I need to be 100% sure I haven't missed a macro expansion.
|
| 604/6 |
2026/05/18 19:35 |
tool |
read-file |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*/
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/bpf-cgroup.h>
#include <linux/cgroup.h>
#include <linux/rcupdate.h>
#include <linux/random.h>
#include <linux/smp.h>
#include <linux/topology.h>
#include <linux/ktime.h>
#include <linux/sched.h>
#include <linux/uidgid.h>
#include <linux/filter.h>
#include <linux/ctype.h>
#include <linux/jiffies.h>
#include <linux/pid_namespace.h>
#include <linux/poison.h>
#include <linux/proc_ns.h>
#include <linux/sched/task.h>
#include <linux/security.h>
#include <linux/btf_ids.h>
#include <linux/bpf_mem_alloc.h>
#include <linux/kasan.h>
#include <linux/bpf_verifier.h>
#include <linux/uaccess.h>
#include <linux/verification.h>
#include <linux/task_work.h>
#include <linux/irq_work.h>
#include <linux/buildid.h>
#include "../../lib/kstrtox.h"
/* If kernel subsystem is allowing eBPF programs to call this function,
* inside its own verifier_ops->get_func_proto() callback it should return
* bpf_map_lookup_elem_proto, so that verifier can properly check the arguments
*
* Different map implementations will rely on rcu in map methods
* lookup/update/delete, therefore eBPF programs must run under rcu lock
* if program is allowed to access maps, so check rcu_read_lock_held() or
* rcu_read_lock_trace_held() in all three functions.
*/
BPF_CALL_2(bpf_map_lookup_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_elem(map, key);
}
const struct bpf_func_proto bpf_map_lookup_elem_proto = {
.func = bpf_map_lookup_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_4(bpf_map_update_elem, struct bpf_map *, map, void *, key,
void *, value, u64, flags)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_update_elem(map, key, value, flags);
}
const struct bpf_func_proto bpf_map_update_elem_proto = {
.func = bpf_map_update_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_PTR_TO_MAP_VALUE,
.arg4_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_delete_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_delete_elem(map, key);
}
const struct bpf_func_proto bpf_map_delete_elem_proto = {
.func = bpf_map_delete_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_3(bpf_map_push_elem, struct bpf_map *, map, void *, value, u64, flags)
{
return map->ops->map_push_elem(map, value, flags);
}
const struct bpf_func_proto bpf_map_push_elem_proto = {
.func = bpf_map_push_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_pop_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_pop_elem(map, value);
}
const struct bpf_func_proto bpf_map_pop_elem_proto = {
.func = bpf_map_pop_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_2(bpf_map_peek_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_peek_elem(map, value);
}
const struct bpf_func_proto bpf_map_peek_elem_proto = {
.func = bpf_map_peek_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_3(bpf_map_lookup_percpu_elem, struct bpf_map *, map, void *, key, u32, cpu)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_percpu_elem(map, key, cpu);
}
const struct bpf_func_proto bpf_map_lookup_percpu_elem_proto = {
.func = bpf_map_lookup_percpu_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_ANYTHING,
};
const struct bpf_func_proto bpf_get_prandom_u32_proto = {
.func = bpf_user_rnd_u32,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_smp_processor_id)
{
return smp_processor_id();
}
const struct bpf_func_proto bpf_get_smp_processor_id_proto = {
.func = bpf_get_smp_processor_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.allow_fastcall = true,
};
BPF_CALL_0(bpf_get_numa_node_id)
{
return numa_node_id();
}
const struct bpf_func_proto bpf_get_numa_node_id_proto = {
.func = bpf_get_numa_node_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_ns)
{
/* NMI safe access to clock monotonic */
return ktime_get_mono_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_ns_proto = {
.func = bpf_ktime_get_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_boot_ns)
{
/* NMI safe access to clock boottime */
return ktime_get_boot_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_boot_ns_proto = {
.func = bpf_ktime_get_boot_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_coarse_ns)
{
return ktime_get_coarse_ns();
}
const struct bpf_func_proto bpf_ktime_get_coarse_ns_proto = {
.func = bpf_ktime_get_coarse_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_tai_ns)
{
/* NMI safe access to clock tai */
return ktime_get_tai_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_tai_ns_proto = {
.func = bpf_ktime_get_tai_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_pid_tgid)
{
struct task_struct *task = current;
if (unlikely(!task))
return -EINVAL;
return (u64) task->tgid << 32 | task->pid;
}
const struct bpf_func_proto bpf_get_current_pid_tgid_proto = {
.func = bpf_get_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_uid_gid)
{
struct task_struct *task = current;
kuid_t uid;
kgid_t gid;
if (unlikely(!task))
return -EINVAL;
current_uid_gid(&uid, &gid);
return (u64) from_kgid(&init_user_ns, gid) << 32 |
from_kuid(&init_user_ns, uid);
}
const struct bpf_func_proto bpf_get_current_uid_gid_proto = {
.func = bpf_get_current_uid_gid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_2(bpf_get_current_comm, char *, buf, u32, size)
{
struct task_struct *task = current;
if (unlikely(!task))
goto err_clear;
/* Verifier guarantees that size > 0 */
strscpy_pad(buf, task->comm, size);
return 0;
err_clear:
memset(buf, 0, size);
return -EINVAL;
}
const struct bpf_func_proto bpf_get_current_comm_proto = {
.func = bpf_get_current_comm,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE,
};
#if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK)
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
union {
__u32 val;
arch_spinlock_t lock;
} u = { .lock = __ARCH_SPIN_LOCK_UNLOCKED };
compiletime_assert(u.val == 0, "__ARCH_SPIN_LOCK_UNLOCKED not 0");
BUILD_BUG_ON(sizeof(*l) != sizeof(__u32));
BUILD_BUG_ON(sizeof(*lock) != sizeof(__u32));
preempt_disable();
arch_spin_lock(l);
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
arch_spin_unlock(l);
preempt_enable();
}
#else
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
BUILD_BUG_ON(sizeof(*l) != sizeof(*lock));
do {
atomic_cond_read_relaxed(l, !VAL);
} while (atomic_xchg(l, 1));
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
atomic_set_release(l, 0);
}
#endif
static DEFINE_PER_CPU(unsigned long, irqsave_flags);
static inline void __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
{
unsigned long flags;
local_irq_save(flags);
__bpf_spin_lock(lock);
__this_cpu_write(irqsave_flags, flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
{
__bpf_spin_lock_irqsave(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_lock_proto = {
.func = bpf_spin_lock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
{
unsigned long flags;
flags = __this_cpu_read(irqsave_flags);
__bpf_spin_unlock(lock);
local_irq_restore(flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
{
__bpf_spin_unlock_irqrestore(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_unlock_proto = {
.func = bpf_spin_unlock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
bool lock_src)
{
struct bpf_spin_lock *lock;
if (lock_src)
lock = src + map->record->spin_lock_off;
else
lock = dst + map->record->spin_lock_off;
preempt_disable();
__bpf_spin_lock_irqsave(lock);
copy_map_value(map, dst, src);
__bpf_spin_unlock_irqrestore(lock);
preempt_enable();
}
BPF_CALL_0(bpf_jiffies64)
{
return get_jiffies_64();
}
const struct bpf_func_proto bpf_jiffies64_proto = {
.func = bpf_jiffies64,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
#ifdef CONFIG_CGROUPS
BPF_CALL_0(bpf_get_current_cgroup_id)
{
struct cgroup *cgrp;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
cgrp_id = cgroup_id(cgrp);
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_cgroup_id_proto = {
.func = bpf_get_current_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_1(bpf_get_current_ancestor_cgroup_id, int, ancestor_level)
{
struct cgroup *cgrp;
struct cgroup *ancestor;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
ancestor = cgroup_ancestor(cgrp, ancestor_level);
cgrp_id = ancestor ? cgroup_id(ancestor) : 0;
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto = {
.func = bpf_get_current_ancestor_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
#endif /* CONFIG_CGROUPS */
#define BPF_STRTOX_BASE_MASK 0x1F
static int __bpf_strtoull(const char *buf, size_t buf_len, u64 flags,
unsigned long long *res, bool *is_negative)
{
unsigned int base = flags & BPF_STRTOX_BASE_MASK;
const char *cur_buf = buf;
size_t cur_len = buf_len;
unsigned int consumed;
size_t val_len;
char str[64];
if (!buf || !buf_len || !res || !is_negative)
return -EINVAL;
if (base != 0 && base != 8 && base != 10 && base != 16)
return -EINVAL;
if (flags & ~BPF_STRTOX_BASE_MASK)
return -EINVAL;
while (cur_buf < buf + buf_len && isspace(*cur_buf))
++cur_buf;
*is_negative = (cur_buf < buf + buf_len && *cur_buf == '-');
if (*is_negative)
++cur_buf;
consumed = cur_buf - buf;
cur_len -= consumed;
if (!cur_len)
return -EINVAL;
cur_len = min(cur_len, sizeof(str) - 1);
memcpy(str, cur_buf, cur_len);
str[cur_len] = '\0';
cur_buf = str;
cur_buf = _parse_integer_fixup_radix(cur_buf, &base);
val_len = _parse_integer(cur_buf, base, res);
if (val_len & KSTRTOX_OVERFLOW)
return -ERANGE;
if (val_len == 0)
return -EINVAL;
cur_buf += val_len;
consumed += cur_buf - str;
return consumed;
}
static int __bpf_strtoll(const char *buf, size_t buf_len, u64 flags,
long long *res)
{
unsigned long long _res;
bool is_negative;
int err;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative) {
if ((long long)-_res > 0)
return -ERANGE;
*res = -_res;
} else {
if ((long long)_res < 0)
return -ERANGE;
*res = _res;
}
return err;
}
BPF_CALL_4(bpf_strtol, const char *, buf, size_t, buf_len, u64, flags,
s64 *, res)
{
long long _res;
int err;
*res = 0;
err = __bpf_strtoll(buf, buf_len, flags, &_res);
if (err < 0)
return err;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtol_proto = {
.func = bpf_strtol,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(s64),
};
BPF_CALL_4(bpf_strtoul, const char *, buf, size_t, buf_len, u64, flags,
u64 *, res)
{
unsigned long long _res;
bool is_negative;
int err;
*res = 0;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative)
return -EINVAL;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtoul_proto = {
.func = bpf_strtoul,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(u64),
};
BPF_CALL_3(bpf_strncmp, const char *, s1, u32, s1_sz, const char *, s2)
{
return strncmp(s1, s2, s1_sz);
}
static const struct bpf_func_proto bpf_strncmp_proto = {
.func = bpf_strncmp,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_CONST_STR,
};
BPF_CALL_4(bpf_get_ns_current_pid_tgid, u64, dev, u64, ino,
struct bpf_pidns_info *, nsdata, u32, size)
{
struct task_struct *task = current;
struct pid_namespace *pidns;
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_pidns_info)))
goto clear;
if (unlikely((u64)(dev_t)dev != dev))
goto clear;
if (unlikely(!task))
goto clear;
pidns = task_active_pid_ns(task);
if (unlikely(!pidns)) {
err = -ENOENT;
goto clear;
}
if (!ns_match(&pidns->ns, (dev_t)dev, ino))
goto clear;
nsdata->pid = task_pid_nr_ns(task, pidns);
nsdata->tgid = task_tgid_nr_ns(task, pidns);
return 0;
clear:
memset((void *)nsdata, 0, (size_t) size);
return err;
}
const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = {
.func = bpf_get_ns_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
};
static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
.func = bpf_get_raw_cpu_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
}
const struct bpf_func_proto bpf_event_output_data_proto = {
.func = bpf_event_output_data,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
const void __user *, user_ptr)
{
int ret = copy_from_user(dst, user_ptr, size);
if (unlikely(ret)) {
memset(dst, 0, size);
ret = -EFAULT;
}
return ret;
}
const struct bpf_func_proto bpf_copy_from_user_proto = {
.func = bpf_copy_from_user,
.gpl_only = false,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_5(bpf_copy_from_user_task, void *, dst, u32, size,
const void __user *, user_ptr, struct task_struct *, tsk, u64, flags)
{
int ret;
/* flags is not used yet */
if (unlikely(flags))
return -EINVAL;
if (unlikely(!size))
return 0;
ret = access_process_vm(tsk, (unsigned long)user_ptr, dst, size, 0);
if (ret == size)
return 0;
memset(dst, 0, size);
/* Return -EFAULT for partial read */
return ret < 0 ? ret : -EFAULT;
}
const struct bpf_func_proto bpf_copy_from_user_task_proto = {
.func = bpf_copy_from_user_task,
.gpl_only = true,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_BTF_ID,
.arg4_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.arg5_type = ARG_ANYTHING
};
BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
{
if (cpu >= nr_cpu_ids)
return (unsigned long)NULL;
return (unsigned long)per_cpu_ptr((const void __percpu *)(const uintptr_t)ptr, cpu);
}
const struct bpf_func_proto bpf_per_cpu_ptr_proto = {
.func = bpf_per_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | PTR_MAYBE_NULL | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
.arg2_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_this_cpu_ptr, const void *, percpu_ptr)
{
return (unsigned long)this_cpu_ptr((const void __percpu *)(const uintptr_t)percpu_ptr);
}
const struct bpf_func_proto bpf_this_cpu_ptr_proto = {
.func = bpf_this_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
};
static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,
size_t bufsz)
{
void __user *user_ptr = (__force void __user *)unsafe_ptr;
buf[0] = 0;
switch (fmt_ptype) {
case 's':
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
if ((unsigned long)unsafe_ptr < TASK_SIZE)
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
fallthrough;
#endif
case 'k':
return strncpy_from_kernel_nofault(buf, unsafe_ptr, bufsz);
case 'u':
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
}
return -EINVAL;
}
/* Support executing three nested bprintf helper calls on a given CPU */
#define MAX_BPRINTF_NEST_LEVEL 3
static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs);
static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
{
int nest_level;
preempt_disable();
nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
return -EBUSY;
}
*bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
return 0;
}
void bpf_put_buffers(void)
{
if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
return;
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
}
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
{
if (!data->bin_args && !data->buf)
return;
bpf_put_buffers();
}
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)
{
if (map->map_type == BPF_MAP_TYPE_ARRAY) {
struct bpf_array *array = container_of(map, struct bpf_array, map);
*arr_idx = ((char *)value - array->value) / array->elem_size;
return arr_idx;
}
return (void *)value - round_up(map->key_size, 8);
}
enum bpf_async_type {
BPF_ASYNC_TYPE_TIMER = 0,
BPF_ASYNC_TYPE_WQ,
};
enum bpf_async_op {
BPF_ASYNC_START,
BPF_ASYNC_CANCEL
};
struct bpf_async_cmd {
struct llist_node node;
u64 nsec;
u32 mode;
enum bpf_async_op op;
};
struct bpf_async_cb {
struct bpf_map *map;
struct bpf_prog *prog;
void __rcu *callback_fn;
void *value;
struct rcu_head rcu;
u64 flags;
struct irq_work worker;
refcount_t refcnt;
enum bpf_async_type type;
struct llist_head async_cmds;
};
/* BPF map elements can contain 'struct bpf_timer'.
* Such map owns all of its BPF timers.
* 'struct bpf_timer' is allocated as part of map element allocation
* and it's zero initialized.
* That space is used to keep 'struct bpf_async_kern'.
* bpf_timer_init() allocates 'struct bpf_hrtimer', inits hrtimer, and
* remembers 'struct bpf_map *' pointer it's part of.
* bpf_timer_set_callback() increments prog refcnt and assign bpf callback_fn.
* bpf_timer_start() arms the timer.
* If user space reference to a map goes to zero at this point
* ops->map_release_uref callback is responsible for cancelling the timers,
* freeing their memory, and decrementing prog's refcnts.
* bpf_timer_cancel() cancels the timer and decrements prog's refcnt.
* Inner maps can contain bpf timers as well. ops->map_release_uref is
* freeing the timers when inner map is replaced or deleted by user space.
*/
struct bpf_hrtimer {
struct bpf_async_cb cb;
struct hrtimer timer;
atomic_t cancelling;
};
struct bpf_work {
struct bpf_async_cb cb;
struct work_struct work;
};
/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */
struct bpf_async_kern {
union {
struct bpf_async_cb *cb;
struct bpf_hrtimer *timer;
struct bpf_work *work;
};
} __attribute__((aligned(8)));
static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);
static void bpf_async_refcount_put(struct bpf_async_cb *cb);
static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer)
{
struct bpf_hrtimer *t = container_of(hrtimer, struct bpf_hrtimer, timer);
struct bpf_map *map = t->cb.map;
void *value = t->cb.value;
bpf_callback_t callback_fn;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_timer);
callback_fn = rcu_dereference_check(t->cb.callback_fn, rcu_read_lock_bh_held());
if (!callback_fn)
goto out;
/* bpf_timer_cb() runs in hrtimer_run_softirq. It doesn't migrate and
* cannot be preempted by another bpf_timer_cb() on the same cpu.
* Remember the timer this callback is servicing to prevent
* deadlock if callback_fn() calls bpf_timer_cancel() or
* bpf_map_delete_elem() on the same timer.
*/
this_cpu_write(hrtimer_running, t);
key = map_key_from_value(map, value, &idx);
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
/* The verifier checked that return value is zero. */
this_cpu_write(hrtimer_running, NULL);
out:
return HRTIMER_NORESTART;
}
static void bpf_wq_work(struct work_struct *work)
{
struct bpf_work *w = container_of(work, struct bpf_work, work);
struct bpf_async_cb *cb = &w->cb;
struct bpf_map *map = cb->map;
bpf_callback_t callback_fn;
void *value = cb->value;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_wq);
callback_fn = READ_ONCE(cb->callback_fn);
if (!callback_fn)
return;
key = map_key_from_value(map, value, &idx);
rcu_read_lock_trace();
migrate_disable();
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
migrate_enable();
rcu_read_unlock_trace();
}
static void bpf_async_cb_rcu_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
/*
* Drop the last reference to prog only after RCU GP, as set_callback()
* may race with cancel_and_free()
*/
if (cb->prog)
bpf_prog_put(cb->prog);
kfree_nolock(cb);
}
/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */
static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
bool retry = false;
/*
* bpf_async_cancel_and_free() tried to cancel timer/wq, but it
* could have raced with timer/wq_start. Now refcnt is zero and
* srcu/rcu GP completed. Cancel timer/wq again.
*/
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER:
if (hrtimer_try_to_cancel(&t->timer) < 0)
retry = true;
break;
case BPF_ASYNC_TYPE_WQ:
if (!cancel_work(&w->work) && work_busy(&w->work))
retry = true;
break;
}
if (retry) {
/*
* hrtimer or wq callback may still be running. It must be
* in rcu_tasks_trace or rcu CS, so wait for GP again.
* It won't retry forever, since refcnt zero prevents all
* operations on timer/wq.
*/
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
return;
}
/* RCU Tasks Trace grace period implies RCU grace period. */
bpf_async_cb_rcu_free(rcu);
}
static void worker_for_call_rcu(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
static void bpf_async_refcount_put(struct bpf_async_cb *cb)
{
if (!refcount_dec_and_test(&cb->refcnt))
return;
if (irqs_disabled()) {
cb->worker = IRQ_WORK_INIT(worker_for_call_rcu);
irq_work_queue(&cb->worker);
} else {
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async);
static void bpf_async_irq_worker(struct irq_work *work);
static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags,
enum bpf_async_type type)
{
struct bpf_async_cb *cb, *old_cb;
struct bpf_hrtimer *t;
struct bpf_work *w;
clockid_t clockid;
size_t size;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
size = sizeof(struct bpf_hrtimer);
break;
case BPF_ASYNC_TYPE_WQ:
size = sizeof(struct bpf_work);
break;
default:
return -EINVAL;
}
old_cb = READ_ONCE(async->cb);
if (old_cb)
return -EBUSY;
cb = bpf_map_kmalloc_nolock(map, size, 0, map->numa_node);
if (!cb)
return -ENOMEM;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
clockid = flags & (MAX_CLOCKS - 1);
t = (struct bpf_hrtimer *)cb;
atomic_set(&t->cancelling, 0);
hrtimer_setup(&t->timer, bpf_timer_cb, clockid, HRTIMER_MODE_REL_SOFT);
cb->value = (void *)async - map->record->timer_off;
break;
case BPF_ASYNC_TYPE_WQ:
w = (struct bpf_work *)cb;
INIT_WORK(&w->work, bpf_wq_work);
cb->value = (void *)async - map->record->wq_off;
break;
}
cb->map = map;
cb->prog = NULL;
cb->flags = flags;
cb->worker = IRQ_WORK_INIT(bpf_async_irq_worker);
init_llist_head(&cb->async_cmds);
refcount_set(&cb->refcnt, 1); /* map's reference */
cb->type = type;
rcu_assign_pointer(cb->callback_fn, NULL);
old_cb = cmpxchg(&async->cb, NULL, cb);
if (old_cb) {
/* Lost the race to initialize this bpf_async_kern, drop the allocated object */
kfree_nolock(cb);
return -EBUSY;
}
/* Guarantee the order between async->cb and map->usercnt. So
* when there are concurrent uref release and bpf timer init, either
* bpf_timer_cancel_and_free() called by uref release reads a no-NULL
* timer or atomic64_read() below returns a zero usercnt.
*/
smp_mb();
if (!atomic64_read(&map->usercnt)) {
/* maps with timers must be either held by user space
* or pinned in bpffs.
*/
bpf_async_cancel_and_free(async);
return -EPERM;
}
return 0;
}
BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
u64, flags)
{
clock_t clockid = flags & (MAX_CLOCKS - 1);
BUILD_BUG_ON(MAX_CLOCKS != 16);
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer));
if (flags >= MAX_CLOCKS ||
/* similar to timerfd except _ALARM variants are not supported */
(clockid != CLOCK_MONOTONIC &&
clockid != CLOCK_REALTIME &&
clockid != CLOCK_BOOTTIME))
return -EINVAL;
return __bpf_async_init(timer, map, flags, BPF_ASYNC_TYPE_TIMER);
}
static const struct bpf_func_proto bpf_timer_init_proto = {
.func = bpf_timer_init,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
static int bpf_async_update_prog_callback(struct bpf_async_cb *cb,
struct bpf_prog *prog,
void *callback_fn)
{
struct bpf_prog *prev;
/* Acquire a guard reference on prog to prevent it from being freed during the loop */
if (prog) {
prog = bpf_prog_inc_not_zero(prog);
if (IS_ERR(prog))
return PTR_ERR(prog);
}
do {
if (prog)
prog = bpf_prog_inc_not_zero(prog);
prev = xchg(&cb->prog, prog);
rcu_assign_pointer(cb->callback_fn, callback_fn);
/*
* Release previous prog, make sure that if other CPU is contending,
* to set bpf_prog, references are not leaked as each iteration acquires and
* releases one reference.
*/
if (prev)
bpf_prog_put(prev);
} while (READ_ONCE(cb->prog) != prog ||
(void __force *)READ_ONCE(cb->callback_fn) != callback_fn);
if (prog)
bpf_prog_put(prog);
return 0;
}
static DEFINE_PER_CPU(struct bpf_async_cb *, async_cb_running);
static int bpf_async_schedule_op(struct bpf_async_cb *cb, enum bpf_async_op op,
u64 nsec, u32 timer_mode)
{
/*
* Do not schedule another operation on this cpu if it's in irq_work
* callback that is processing async_cmds queue. Otherwise the following
* loop is possible:
* bpf_timer_start() -> bpf_async_schedule_op() -> irq_work_queue().
* irqrestore -> bpf_async_irq_worker() -> tracepoint -> bpf_timer_start().
*/
if (this_cpu_read(async_cb_running) == cb) {
bpf_async_refcount_put(cb);
return -EDEADLK;
}
struct bpf_async_cmd *cmd = kmalloc_nolock(sizeof(*cmd), 0, NUMA_NO_NODE);
if (!cmd) {
bpf_async_refcount_put(cb);
return -ENOMEM;
}
init_llist_node(&cmd->node);
cmd->nsec = nsec;
cmd->mode = timer_mode;
cmd->op = op;
if (llist_add(&cmd->node, &cb->async_cmds))
irq_work_queue(&cb->worker);
return 0;
}
static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback_fn,
struct bpf_prog *prog)
{
struct bpf_async_cb *cb;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
return bpf_async_update_prog_callback(cb, prog, callback_fn);
}
BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
struct bpf_prog_aux *, aux)
{
return __bpf_async_set_callback(timer, callback_fn, aux->prog);
}
static const struct bpf_func_proto bpf_timer_set_callback_proto = {
.func = bpf_timer_set_callback,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_PTR_TO_FUNC,
};
static bool defer_timer_wq_op(void)
{
return in_hardirq() || irqs_disabled();
}
BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags)
{
struct bpf_hrtimer *t;
u32 mode;
if (flags & ~(BPF_F_TIMER_ABS | BPF_F_TIMER_CPU_PIN))
return -EINVAL;
t = READ_ONCE(async->timer);
if (!t || !READ_ONCE(t->cb.prog))
return -EINVAL;
if (flags & BPF_F_TIMER_ABS)
mode = HRTIMER_MODE_ABS_SOFT;
else
mode = HRTIMER_MODE_REL_SOFT;
if (flags & BPF_F_TIMER_CPU_PIN)
mode |= HRTIMER_MODE_PINNED;
/*
* bpf_async_cancel_and_free() could have dropped refcnt to zero. In
* such case BPF progs are not allowed to arm the timer to prevent UAF.
*/
if (!refcount_inc_not_zero(&t->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
hrtimer_start(&t->timer, ns_to_ktime(nsecs), mode);
bpf_async_refcount_put(&t->cb);
return 0;
} else {
return bpf_async_schedule_op(&t->cb, BPF_ASYNC_START, nsecs, mode);
}
}
static const struct bpf_func_proto bpf_timer_start_proto = {
.func = bpf_timer_start,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async)
{
struct bpf_hrtimer *t, *cur_t;
bool inc = false;
int ret = 0;
if (defer_timer_wq_op())
return -EOPNOTSUPP;
t = READ_ONCE(async->timer);
if (!t)
return -EINVAL;
cur_t = this_cpu_read(hrtimer_running);
if (cur_t == t) {
/* If bpf callback_fn is trying to bpf_timer_cancel()
* its own timer the hrtimer_cancel() will deadlock
* since it waits for callback_fn to finish.
*/
return -EDEADLK;
}
/* Only account in-flight cancellations when invoked from a timer
* callback, since we want to avoid waiting only if other _callbacks_
* are waiting on us, to avoid introducing lockups. Non-callback paths
* are ok, since nobody would synchronously wait for their completion.
*/
if (!cur_t)
goto drop;
atomic_inc(&t->cancelling);
/* Need full barrier after relaxed atomic_inc */
smp_mb__after_atomic();
inc = true;
if (atomic_read(&cur_t->cancelling)) {
/* We're cancelling timer t, while some other timer callback is
* attempting to cancel us. In such a case, it might be possible
* that timer t belongs to the other callback, or some other
* callback waiting upon it (creating transitive dependencies
* upon us), and we will enter a deadlock if we continue
* cancelling and waiting for it synchronously, since it might
* do the same. Bail!
*/
atomic_dec(&t->cancelling);
return -EDEADLK;
}
drop:
bpf_async_update_prog_callback(&t->cb, NULL, NULL);
/* Cancel the timer and wait for associated callback to finish
* if it was running.
*/
ret = hrtimer_cancel(&t->timer);
if (inc)
atomic_dec(&t->cancelling);
return ret;
}
static const struct bpf_func_proto bpf_timer_cancel_proto = {
.func = bpf_timer_cancel,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
};
static void bpf_async_process_op(struct bpf_async_cb *cb, u32 op,
u64 timer_nsec, u32 timer_mode)
{
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER: {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
switch (op) {
case BPF_ASYNC_START:
hrtimer_start(&t->timer, ns_to_ktime(timer_nsec), timer_mode);
break;
case BPF_ASYNC_CANCEL:
hrtimer_try_to_cancel(&t->timer);
break;
}
break;
}
case BPF_ASYNC_TYPE_WQ: {
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
switch (op) {
case BPF_ASYNC_START:
schedule_work(&w->work);
break;
case BPF_ASYNC_CANCEL:
cancel_work(&w->work);
break;
}
break;
}
}
bpf_async_refcount_put(cb);
}
static void bpf_async_irq_worker(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
struct llist_node *pos, *n, *list;
list = llist_del_all(&cb->async_cmds);
if (!list)
return;
list = llist_reverse_order(list);
this_cpu_write(async_cb_running, cb);
llist_for_each_safe(pos, n, list) {
struct bpf_async_cmd *cmd;
cmd = container_of(pos, struct bpf_async_cmd, node);
bpf_async_process_op(cb, cmd->op, cmd->nsec, cmd->mode);
kfree_nolock(cmd);
}
this_cpu_write(async_cb_running, NULL);
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async)
{
struct bpf_async_cb *cb;
if (!READ_ONCE(async->cb))
return;
cb = xchg(&async->cb, NULL);
if (!cb)
return;
bpf_async_update_prog_callback(cb, NULL, NULL);
/*
* No refcount_inc_not_zero(&cb->refcnt) here. Dropping the last
* refcnt. Either synchronously or asynchronously in irq_work.
*/
if (!defer_timer_wq_op()) {
bpf_async_process_op(cb, BPF_ASYNC_CANCEL, 0, 0);
} else {
(void)bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
/*
* bpf_async_schedule_op() either enqueues allocated cmd into llist
* or fails with ENOMEM and drop the last refcnt.
* This is unlikely, but safe, since bpf_async_cb_rcu_tasks_trace_free()
* callback will do additional timer/wq_cancel due to races anyway.
*/
}
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_timer_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_wq_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
{
unsigned long *kptr = dst;
/* This helper may be inlined by verifier. */
return xchg(kptr, (unsigned long)ptr);
}
/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg()
* helper is determined dynamically by the verifier. Use BPF_PTR_POISON to
* denote type that verifier will determine.
*/
static const struct bpf_func_proto bpf_kptr_xchg_proto = {
.func = bpf_kptr_xchg,
.gpl_only = false,
.ret_type = RET_PTR_TO_BTF_ID_OR_NULL,
.ret_btf_id = BPF_PTR_POISON,
.arg1_type = ARG_KPTR_XCHG_DEST,
.arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL | OBJ_RELEASE,
.arg2_btf_id = BPF_PTR_POISON,
};
struct bpf_dynptr_file_impl {
struct freader freader;
/* 64 bit offset and size overriding 32 bit ones in bpf_dynptr_kern */
u64 offset;
u64 size;
};
/* Since the upper 8 bits of dynptr->size is reserved, the
* maximum supported size is 2^24 - 1.
*/
#define DYNPTR_MAX_SIZE ((1UL << 24) - 1)
#define DYNPTR_TYPE_SHIFT 28
#define DYNPTR_SIZE_MASK 0xFFFFFF
#define DYNPTR_RDONLY_BIT BIT(31)
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr)
{
return ptr->size & DYNPTR_RDONLY_BIT;
}
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
{
ptr->size |= DYNPTR_RDONLY_BIT;
}
static void bpf_dynptr_set_type(struct bpf_dynptr_kern *ptr, enum bpf_dynptr_type type)
{
ptr->size |= type << DYNPTR_TYPE_SHIFT;
}
static enum bpf_dynptr_type bpf_dynptr_get_type(const struct bpf_dynptr_kern *ptr)
{
return (ptr->size & ~(DYNPTR_RDONLY_BIT)) >> DYNPTR_TYPE_SHIFT;
}
u64 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
return df->size;
}
return ptr->size & DYNPTR_SIZE_MASK;
}
static void bpf_dynptr_advance_offset(struct bpf_dynptr_kern *ptr, u64 off)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->offset += off;
return;
}
ptr->offset += off;
}
static void bpf_dynptr_set_size(struct bpf_dynptr_kern *ptr, u64 new_size)
{
u32 metadata = ptr->size & ~DYNPTR_SIZE_MASK;
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->size = new_size;
return;
}
ptr->size = (u32)new_size | metadata;
}
int bpf_dynptr_check_size(u64 size)
{
return size > DYNPTR_MAX_SIZE ? -E2BIG : 0;
}
static int bpf_file_fetch_bytes(struct bpf_dynptr_file_impl *df, u64 offset, void *buf, u64 len)
{
const void *ptr;
if (!buf)
return -EINVAL;
df->freader.buf = buf;
df->freader.buf_sz = len;
ptr = freader_fetch(&df->freader, offset + df->offset, len);
if (!ptr)
return df->freader.err;
if (ptr != buf) /* Force copying into the buffer */
memcpy(buf, ptr, len);
return 0;
}
void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size)
{
ptr->data = data;
ptr->offset = offset;
ptr->size = size;
bpf_dynptr_set_type(ptr, type);
}
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)
{
memset(ptr, 0, sizeof(*ptr));
}
BPF_CALL_4(bpf_dynptr_from_mem, void *, data, u64, size, u64, flags, struct bpf_dynptr_kern *, ptr)
{
int err;
BTF_TYPE_EMIT(struct bpf_dynptr);
err = bpf_dynptr_check_size(size);
if (err)
goto error;
/* flags is currently unsupported */
if (flags) {
err = -EINVAL;
goto error;
}
bpf_dynptr_init(ptr, data, BPF_DYNPTR_TYPE_LOCAL, 0, size);
return 0;
error:
bpf_dynptr_set_null(ptr);
return err;
}
static const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
.func = bpf_dynptr_from_mem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL | MEM_UNINIT | MEM_WRITE,
};
static int __bpf_dynptr_read(void *dst, u64 len, const struct bpf_dynptr_kern *src,
u64 offset, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!src->data || flags)
return -EINVAL;
err = bpf_dynptr_check_off_len(src, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(src);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst, src->data + src->offset + offset, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_XDP:
return __bpf_xdp_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_SKB_META:
memmove(dst, bpf_skb_meta_pointer(src->data, src->offset + offset), len);
return 0;
case BPF_DYNPTR_TYPE_FILE:
return bpf_file_fetch_bytes(src->data, offset, dst, len);
default:
WARN_ONCE(true, "bpf_dynptr_read: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_read, void *, dst, u64, len, const struct bpf_dynptr_kern *, src,
u64, offset, u64, flags)
{
return __bpf_dynptr_read(dst, len, src, offset, flags);
}
static const struct bpf_func_proto bpf_dynptr_read_proto = {
.func = bpf_dynptr_read,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
int __bpf_dynptr_write(const struct bpf_dynptr_kern *dst, u64 offset, void *src,
u64 len, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!dst->data || __bpf_dynptr_is_rdonly(dst))
return -EINVAL;
err = bpf_dynptr_check_off_len(dst, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(dst);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
if (flags)
return -EINVAL;
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst->data + dst->offset + offset, src, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_store_bytes(dst->data, dst->offset + offset, src, len,
flags);
case BPF_DYNPTR_TYPE_XDP:
if (flags)
return -EINVAL;
return __bpf_xdp_store_bytes(dst->data, dst->offset + offset, src, len);
case BPF_DYNPTR_TYPE_SKB_META:
return __bpf_skb_meta_store_bytes(dst->data, dst->offset + offset, src,
len, flags);
default:
WARN_ONCE(true, "bpf_dynptr_write: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_write, const struct bpf_dynptr_kern *, dst, u64, offset, void *, src,
u64, len, u64, flags)
{
return __bpf_dynptr_write(dst, offset, src, len, flags);
}
static const struct bpf_func_proto bpf_dynptr_write_proto = {
.func = bpf_dynptr_write,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_dynptr_data, const struct bpf_dynptr_kern *, ptr, u64, offset, u64, len)
{
enum bpf_dynptr_type type;
int err;
if (!ptr->data)
return 0;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return 0;
if (__bpf_dynptr_is_rdonly(ptr))
return 0;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return (unsigned long)(ptr->data + ptr->offset + offset);
case BPF_DYNPTR_TYPE_SKB:
case BPF_DYNPTR_TYPE_XDP:
case BPF_DYNPTR_TYPE_SKB_META:
/* skb and xdp dynptrs should use bpf_dynptr_slice / bpf_dynptr_slice_rdwr */
return 0;
default:
WARN_ONCE(true, "bpf_dynptr_data: unknown dynptr type %d\n", type);
return 0;
}
}
static const struct bpf_func_proto bpf_dynptr_data_proto = {
.func = bpf_dynptr_data,
.gpl_only = false,
.ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
};
const struct bpf_func_proto bpf_get_current_task_proto __weak;
const struct bpf_func_proto bpf_get_current_task_btf_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_str_proto __weak;
const struct bpf_func_proto bpf_task_pt_regs_proto __weak;
const struct bpf_func_proto bpf_perf_event_read_proto __weak;
const struct bpf_func_proto bpf_send_signal_proto __weak;
const struct bpf_func_proto bpf_send_signal_thread_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_sleepable_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_proto __weak;
const struct bpf_func_proto bpf_get_branch_snapshot_proto __weak;
const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_map_lookup_elem:
return &bpf_map_lookup_elem_proto;
case BPF_FUNC_map_update_elem:
return &bpf_map_update_elem_proto;
case BPF_FUNC_map_delete_elem:
return &bpf_map_delete_elem_proto;
case BPF_FUNC_map_push_elem:
return &bpf_map_push_elem_proto;
case BPF_FUNC_map_pop_elem:
return &bpf_map_pop_elem_proto;
case BPF_FUNC_map_peek_elem:
return &bpf_map_peek_elem_proto;
case BPF_FUNC_map_lookup_percpu_elem:
return &bpf_map_lookup_percpu_elem_proto;
case BPF_FUNC_get_prandom_u32:
return &bpf_get_prandom_u32_proto;
case BPF_FUNC_get_smp_processor_id:
return &bpf_get_raw_smp_processor_id_proto;
case BPF_FUNC_get_numa_node_id:
return &bpf_get_numa_node_id_proto;
case BPF_FUNC_tail_call:
return &bpf_tail_call_proto;
case BPF_FUNC_ktime_get_ns:
return &bpf_ktime_get_ns_proto;
case BPF_FUNC_ktime_get_boot_ns:
return &bpf_ktime_get_boot_ns_proto;
case BPF_FUNC_ktime_get_tai_ns:
return &bpf_ktime_get_tai_ns_proto;
case BPF_FUNC_ringbuf_output:
return &bpf_ringbuf_output_proto;
case BPF_FUNC_ringbuf_reserve:
return &bpf_ringbuf_reserve_proto;
case BPF_FUNC_ringbuf_submit:
return &bpf_ringbuf_submit_proto;
case BPF_FUNC_ringbuf_discard:
return &bpf_ringbuf_discard_proto;
case BPF_FUNC_ringbuf_query:
return &bpf_ringbuf_query_proto;
case BPF_FUNC_strncmp:
return &bpf_strncmp_proto;
case BPF_FUNC_strtol:
return &bpf_strtol_proto;
case BPF_FUNC_strtoul:
return &bpf_strtoul_proto;
case BPF_FUNC_get_current_pid_tgid:
return &bpf_get_current_pid_tgid_proto;
case BPF_FUNC_get_ns_current_pid_tgid:
return &bpf_get_ns_current_pid_tgid_proto;
case BPF_FUNC_get_current_uid_gid:
return &bpf_get_current_uid_gid_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_BPF))
return NULL;
switch (func_id) {
case BPF_FUNC_spin_lock:
return &bpf_spin_lock_proto;
case BPF_FUNC_spin_unlock:
return &bpf_spin_unlock_proto;
case BPF_FUNC_jiffies64:
return &bpf_jiffies64_proto;
case BPF_FUNC_per_cpu_ptr:
return &bpf_per_cpu_ptr_proto;
case BPF_FUNC_this_cpu_ptr:
return &bpf_this_cpu_ptr_proto;
case BPF_FUNC_timer_init:
return &bpf_timer_init_proto;
case BPF_FUNC_timer_set_callback:
return &bpf_timer_set_callback_proto;
case BPF_FUNC_timer_start:
return &bpf_timer_start_proto;
case BPF_FUNC_timer_cancel:
return &bpf_timer_cancel_proto;
case BPF_FUNC_kptr_xchg:
return &bpf_kptr_xchg_proto;
case BPF_FUNC_for_each_map_elem:
return &bpf_for_each_map_elem_proto;
case BPF_FUNC_loop:
return &bpf_loop_proto;
case BPF_FUNC_user_ringbuf_drain:
return &bpf_user_ringbuf_drain_proto;
case BPF_FUNC_ringbuf_reserve_dynptr:
return &bpf_ringbuf_reserve_dynptr_proto;
case BPF_FUNC_ringbuf_submit_dynptr:
return &bpf_ringbuf_submit_dynptr_proto;
case BPF_FUNC_ringbuf_discard_dynptr:
return &bpf_ringbuf_discard_dynptr_proto;
case BPF_FUNC_dynptr_from_mem:
return &bpf_dynptr_from_mem_proto;
case BPF_FUNC_dynptr_read:
return &bpf_dynptr_read_proto;
case BPF_FUNC_dynptr_write:
return &bpf_dynptr_write_proto;
case BPF_FUNC_dynptr_data:
return &bpf_dynptr_data_proto;
#ifdef CONFIG_CGROUPS
case BPF_FUNC_cgrp_storage_get:
return &bpf_cgrp_storage_get_proto;
case BPF_FUNC_cgrp_storage_delete:
return &bpf_cgrp_storage_delete_proto;
case BPF_FUNC_get_current_cgroup_id:
return &bpf_get_current_cgroup_id_proto;
case BPF_FUNC_get_current_ancestor_cgroup_id:
return &bpf_get_current_ancestor_cgroup_id_proto;
case BPF_FUNC_current_task_under_cgroup:
return &bpf_current_task_under_cgroup_proto;
#endif
#ifdef CONFIG_CGROUP_NET_CLASSID
case BPF_FUNC_get_cgroup_classid:
return &bpf_get_cgroup_classid_curr_proto;
#endif
case BPF_FUNC_task_storage_get:
return &bpf_task_storage_get_proto;
case BPF_FUNC_task_storage_delete:
return &bpf_task_storage_delete_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_PERFMON))
return NULL;
switch (func_id) {
case BPF_FUNC_trace_printk:
return bpf_get_trace_printk_proto();
case BPF_FUNC_get_current_task:
return &bpf_get_current_task_proto;
case BPF_FUNC_get_current_task_btf:
return &bpf_get_current_task_btf_proto;
case BPF_FUNC_get_current_comm:
return &bpf_get_current_comm_proto;
case BPF_FUNC_probe_read_user:
return &bpf_probe_read_user_proto;
case BPF_FUNC_probe_read_kernel:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_proto;
case BPF_FUNC_probe_read_user_str:
return &bpf_probe_read_user_str_proto;
case BPF_FUNC_probe_read_kernel_str:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_str_proto;
case BPF_FUNC_copy_from_user:
return &bpf_copy_from_user_proto;
case BPF_FUNC_copy_from_user_task:
return &bpf_copy_from_user_task_proto;
case BPF_FUNC_snprintf_btf:
return &bpf_snprintf_btf_proto;
case BPF_FUNC_snprintf:
return &bpf_snprintf_proto;
case BPF_FUNC_task_pt_regs:
return &bpf_task_pt_regs_proto;
case BPF_FUNC_trace_vprintk:
return bpf_get_trace_vprintk_proto();
case BPF_FUNC_perf_event_read_value:
return bpf_get_perf_event_read_value_proto();
case BPF_FUNC_perf_event_read:
return &bpf_perf_event_read_proto;
case BPF_FUNC_send_signal:
return &bpf_send_signal_proto;
case BPF_FUNC_send_signal_thread:
return &bpf_send_signal_thread_proto;
case BPF_FUNC_get_task_stack:
return prog->sleepable ? &bpf_get_task_stack_sleepable_proto
: &bpf_get_task_stack_proto;
case BPF_FUNC_get_branch_snapshot:
return &bpf_get_branch_snapshot_proto;
case BPF_FUNC_find_vma:
return &bpf_find_vma_proto;
default:
return NULL;
}
}
EXPORT_SYMBOL_GPL(bpf_base_func_proto);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock)
{
struct list_head *head = list_head, *orig_head = list_head;
BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
/* Do the actual list draining outside the lock to not hold the lock for
* too long, and also prevent deadlocks if tracing programs end up
* executing on entry/exit of functions called inside the critical
* section, and end up doing map ops that call bpf_list_head_free for
* the same map value again.
*/
__bpf_spin_lock_irqsave(spin_lock);
if (!head->next || list_empty(head))
goto unlock;
head = head->next;
unlock:
INIT_LIST_HEAD(orig_head);
__bpf_spin_unlock_irqrestore(spin_lock);
while (head != orig_head) {
void *obj = head;
obj -= field->graph_root.node_offset;
head = head->next;
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
/* Like rbtree_postorder_for_each_entry_safe, but 'pos' and 'n' are
* 'rb_node *', so field name of rb_node within containing struct is not
* needed.
*
* Since bpf_rb_tree's node type has a corresponding struct btf_field with
* graph_root.node_offset, it's not necessary to know field name
* or type of node struct
*/
#define bpf_rbtree_postorder_for_each_entry_safe(pos, n, root) \
for (pos = rb_first_postorder(root); \
pos && ({ n = rb_next_postorder(pos); 1; }); \
pos = n)
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock)
{
struct rb_root_cached orig_root, *root = rb_root;
struct rb_node *pos, *n;
void *obj;
BUILD_BUG_ON(sizeof(struct rb_root_cached) > sizeof(struct bpf_rb_root));
BUILD_BUG_ON(__alignof__(struct rb_root_cached) > __alignof__(struct bpf_rb_root));
__bpf_spin_lock_irqsave(spin_lock);
orig_root = *root;
*root = RB_ROOT_CACHED;
__bpf_spin_unlock_irqrestore(spin_lock);
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
obj = pos;
obj -= field->graph_root.node_offset;
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
__bpf_kfunc_start_defs();
/**
* bpf_obj_new() - allocate an object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate an object of the type identified by @local_type_id__k and
* initialize its special fields. BPF programs can use
* bpf_core_type_id_local() to provide @local_type_id__k. The verifier
* rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
void *p;
p = bpf_mem_alloc(&bpf_global_ma, size);
if (!p)
return NULL;
if (meta)
bpf_obj_init(meta->record, p);
return p;
}
__bpf_kfunc void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_obj_new(local_type_id__k, meta__ign);
}
/**
* bpf_percpu_obj_new() - allocate a percpu object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate a percpu object of the type identified by @local_type_id__k. BPF
* programs can use bpf_core_type_id_local() to provide @local_type_id__k.
* The verifier rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated percpu object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_percpu_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
/* The verifier has ensured that meta must be NULL */
return bpf_mem_alloc(&bpf_global_percpu_ma, size);
}
__bpf_kfunc void *bpf_percpu_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_percpu_obj_new(local_type_id__k, meta__ign);
}
/* Must be called under migrate_disable(), as required by bpf_mem_free */
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu)
{
struct bpf_mem_alloc *ma;
if (rec && rec->refcount_off >= 0 &&
!refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) {
/* Object is refcounted and refcount_dec didn't result in 0
* refcount. Return without freeing the object
*/
return;
}
if (rec)
bpf_obj_free_fields(rec, p);
if (percpu)
ma = &bpf_global_percpu_ma;
else
ma = &bpf_global_ma;
bpf_mem_free_rcu(ma, p);
}
/**
* bpf_obj_drop() - drop a previously allocated object
* @p__alloc: object to free
* @meta: verifier-supplied struct metadata
*
* Destroy special fields in @p__alloc as needed and free the object. The
* verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
void *p = p__alloc;
__bpf_obj_drop_impl(p, meta ? meta->record : NULL, false);
}
__bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
{
return bpf_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_percpu_obj_drop() - drop a previously allocated percpu object
* @p__alloc: percpu object to free
* @meta: verifier-supplied struct metadata
*
* Free @p__alloc. The verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_percpu_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
/* The verifier has ensured that meta must be NULL */
bpf_mem_free_rcu(&bpf_global_percpu_ma, p__alloc);
}
__bpf_kfunc void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign)
{
bpf_percpu_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_refcount_acquire() - turn a local kptr into an owning reference
* @p__refcounted_kptr: non-owning local kptr
* @meta: verifier-supplied struct metadata
*
* Increment the refcount for @p__refcounted_kptr. The verifier rewrites
* @meta; BPF programs do not set it.
*
* Return: Owning reference to @p__refcounted_kptr, or %NULL on failure.
*/
__bpf_kfunc void *bpf_refcount_acquire(void *p__refcounted_kptr, struct btf_struct_meta *meta)
{
struct bpf_refcount *ref;
/* Could just cast directly to refcount_t *, but need some code using
* bpf_refcount type so that it is emitted in vmlinux BTF
*/
ref = (struct bpf_refcount *)(p__refcounted_kptr + meta->record->refcount_off);
if (!refcount_inc_not_zero((refcount_t *)ref))
return NULL;
/* Verifier strips KF_RET_NULL if input is owned ref, see is_kfunc_ret_null
* in verifier.c
*/
return (void *)p__refcounted_kptr;
}
__bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta__ign)
{
return bpf_refcount_acquire(p__refcounted_kptr, meta__ign);
}
static int __bpf_list_add(struct bpf_list_node_kern *node,
struct bpf_list_head *head,
bool tail, struct btf_record *rec, u64 off)
{
struct list_head *n = &node->list_head, *h = (void *)head;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
/* node->owner != NULL implies !list_empty(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
tail ? list_add_tail(n, h) : list_add(n, h);
WRITE_ONCE(node->owner, head);
return 0;
}
/**
* bpf_list_push_front() - add a node to the front of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the front of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_front(head, node, meta__ign, off);
}
/**
* bpf_list_push_back() - add a node to the back of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the back of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_back(head, node, meta__ign, off);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
{
struct list_head *n, *h = (void *)head;
struct bpf_list_node_kern *node;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
if (list_empty(h))
return NULL;
n = tail ? h->prev : h->next;
node = container_of(n, struct bpf_list_node_kern, list_head);
if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))
return NULL;
list_del_init(n);
WRITE_ONCE(node->owner, NULL);
return (struct bpf_list_node *)n;
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
{
return __bpf_list_del(head, false);
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
{
return __bpf_list_del(head, true);
}
__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->next;
}
__bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->prev;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
struct rb_root_cached *r = (struct rb_root_cached *)root;
struct rb_node *n = &node_internal->rb_node;
/* node_internal->owner != root implies either RB_EMPTY_NODE(n) or
* n is owned by some other tree. No need to check RB_EMPTY_NODE(n)
*/
if (READ_ONCE(node_internal->owner) != root)
return NULL;
rb_erase_cached(n, r);
RB_CLEAR_NODE(n);
WRITE_ONCE(node_internal->owner, NULL);
return (struct bpf_rb_node *)n;
}
/* Need to copy rbtree_add_cached's logic here because our 'less' is a BPF
* program
*/
static int __bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node_kern *node,
void *less, struct btf_record *rec, u64 off)
{
struct rb_node **link = &((struct rb_root_cached *)root)->rb_root.rb_node;
struct rb_node *parent = NULL, *n = &node->rb_node;
bpf_callback_t cb = (bpf_callback_t)less;
bool leftmost = true;
/* node->owner != NULL implies !RB_EMPTY_NODE(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
while (*link) {
parent = *link;
if (cb((uintptr_t)node, (uintptr_t)parent, 0, 0, 0)) {
link = &parent->rb_left;
} else {
link = &parent->rb_right;
leftmost = false;
}
}
rb_link_node(n, parent, link);
rb_insert_color_cached(n, (struct rb_root_cached *)root, leftmost);
WRITE_ONCE(node->owner, root);
return 0;
}
/**
* bpf_rbtree_add() - add a node to a BPF rbtree
* @root: tree root
* @node: node to insert
* @less: comparator used to order nodes
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node into @root using @less. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked in a tree.
*/
__bpf_kfunc int bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_rb_node_kern *n = (void *)node;
return __bpf_rbtree_add(root, n, (void *)less, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
void *meta__ign, u64 off)
{
return bpf_rbtree_add(root, node, less, meta__ign, off);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)rb_first_cached(r);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_root(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)r->rb_root.rb_node;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_left(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_left;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_right(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_right;
}
/**
* bpf_task_acquire - Acquire a reference to a task. A task acquired by this
* kfunc which is not stored in a map as a kptr, must be released by calling
* bpf_task_release().
* @p: The task on which a reference is being acquired.
*/
__bpf_kfunc struct task_struct *bpf_task_acquire(struct task_struct *p)
{
if (refcount_inc_not_zero(&p->rcu_users))
return p;
return NULL;
}
/**
* bpf_task_release - Release the reference acquired on a task.
* @p: The task on which a reference is being released.
*/
__bpf_kfunc void bpf_task_release(struct task_struct *p)
{
put_task_struct_rcu_user(p);
}
__bpf_kfunc void bpf_task_release_dtor(void *p)
{
put_task_struct_rcu_user(p);
}
CFI_NOSEAL(bpf_task_release_dtor);
#ifdef CONFIG_CGROUPS
/**
* bpf_cgroup_acquire - Acquire a reference to a cgroup. A cgroup acquired by
* this kfunc which is not stored in a map as a kptr, must be released by
* calling bpf_cgroup_release().
* @cgrp: The cgroup on which a reference is being acquired.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)
{
return cgroup_tryget(cgrp) ? cgrp : NULL;
}
/**
* bpf_cgroup_release - Release the reference acquired on a cgroup.
* If this kfunc is invoked in an RCU read region, the cgroup is guaranteed to
* not be freed until the current grace period has ended, even if its refcount
* drops to 0.
* @cgrp: The cgroup on which a reference is being released.
*/
__bpf_kfunc void bpf_cgroup_release(struct cgroup *cgrp)
{
cgroup_put(cgrp);
}
__bpf_kfunc void bpf_cgroup_release_dtor(void *cgrp)
{
cgroup_put(cgrp);
}
CFI_NOSEAL(bpf_cgroup_release_dtor);
/**
* bpf_cgroup_ancestor - Perform a lookup on an entry in a cgroup's ancestor
* array. A cgroup returned by this kfunc which is not subsequently stored in a
* map, must be released by calling bpf_cgroup_release().
* @cgrp: The cgroup for which we're performing a lookup.
* @level: The level of ancestor to look up.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)
{
struct cgroup *ancestor;
if (level > cgrp->level || level < 0)
return NULL;
/* cgrp's refcnt could be 0 here, but ancestors can still be accessed */
ancestor = cgrp->ancestors[level];
if (!cgroup_tryget(ancestor))
return NULL;
return ancestor;
}
/**
* bpf_cgroup_from_id - Find a cgroup from its ID. A cgroup returned by this
* kfunc which is not subsequently stored in a map, must be released by calling
* bpf_cgroup_release().
* @cgid: cgroup id.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_from_id(u64 cgid)
{
struct cgroup *cgrp;
cgrp = __cgroup_get_from_id(cgid);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
/**
* bpf_task_under_cgroup - wrap task_under_cgroup_hierarchy() as a kfunc, test
* task's membership of cgroup ancestry.
* @task: the task to be tested
* @ancestor: possible ancestor of @task's cgroup
*
* Tests whether @task's default cgroup hierarchy is a descendant of @ancestor.
* It follows all the same rules as cgroup_is_descendant, and only applies
* to the default hierarchy.
*/
__bpf_kfunc long bpf_task_under_cgroup(struct task_struct *task,
struct cgroup *ancestor)
{
long ret;
rcu_read_lock();
ret = task_under_cgroup_hierarchy(task, ancestor);
rcu_read_unlock();
return ret;
}
BPF_CALL_2(bpf_current_task_under_cgroup, struct bpf_map *, map, u32, idx)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
struct cgroup *cgrp;
if (unlikely(idx >= array->map.max_entries))
return -E2BIG;
cgrp = READ_ONCE(array->ptrs[idx]);
if (unlikely(!cgrp))
return -EAGAIN;
return task_under_cgroup_hierarchy(current, cgrp);
}
const struct bpf_func_proto bpf_current_task_under_cgroup_proto = {
.func = bpf_current_task_under_cgroup,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
};
/**
* bpf_task_get_cgroup1 - Acquires the associated cgroup of a task within a
* specific cgroup1 hierarchy. The cgroup1 hierarchy is identified by its
* hierarchy ID.
* @task: The target task
* @hierarchy_id: The ID of a cgroup1 hierarchy
*
* On success, the cgroup is returen. On failure, NULL is returned.
*/
__bpf_kfunc struct cgroup *
bpf_task_get_cgroup1(struct task_struct *task, int hierarchy_id)
{
struct cgroup *cgrp = task_get_cgroup1(task, hierarchy_id);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
#endif /* CONFIG_CGROUPS */
/**
* bpf_task_from_pid - Find a struct task_struct from its pid by looking it up
* in the root pid namespace idr. If a task is returned, it must either be
* stored in a map, or released with bpf_task_release().
* @pid: The pid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_pid_ns(pid, &init_pid_ns);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_task_from_vpid - Find a struct task_struct from its vpid by looking it up
* in the pid namespace of the current task. If a task is returned, it must
* either be stored in a map, or released with bpf_task_release().
* @vpid: The vpid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_vpid(vpid);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_dynptr_slice() - Obtain a read-only pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* If the intention is to write to the data slice, please use
* bpf_dynptr_slice_rdwr.
*
* The user must check that the returned pointer is not null before using it.
*
* Please note that in the case of skb and xdp dynptrs, bpf_dynptr_slice
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a read-only
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
enum bpf_dynptr_type type;
u64 len = buffer__szk;
int err;
if (!ptr->data)
return NULL;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return NULL;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return ptr->data + ptr->offset + offset;
case BPF_DYNPTR_TYPE_SKB:
if (buffer__nullable)
return skb_header_pointer(ptr->data, ptr->offset + offset, len, buffer__nullable);
else
return skb_pointer_if_linear(ptr->data, ptr->offset + offset, len);
case BPF_DYNPTR_TYPE_XDP:
{
void *xdp_ptr = bpf_xdp_pointer(ptr->data, ptr->offset + offset, len);
if (!IS_ERR_OR_NULL(xdp_ptr))
return xdp_ptr;
if (!buffer__nullable)
return NULL;
bpf_xdp_copy_buf(ptr->data, ptr->offset + offset, buffer__nullable, len, false);
return buffer__nullable;
}
case BPF_DYNPTR_TYPE_SKB_META:
return bpf_skb_meta_pointer(ptr->data, ptr->offset + offset);
case BPF_DYNPTR_TYPE_FILE:
err = bpf_file_fetch_bytes(ptr->data, offset, buffer__nullable, buffer__szk);
return err ? NULL : buffer__nullable;
default:
WARN_ONCE(true, "unknown dynptr type %d\n", type);
return NULL;
}
}
/**
* bpf_dynptr_slice_rdwr() - Obtain a writable pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* The returned pointer is writable and may point to either directly the dynptr
* data at the requested offset or to the buffer if unable to obtain a direct
* data pointer to (example: the requested slice is to the paged area of an skb
* packet). In the case where the returned pointer is to the buffer, the user
* is responsible for persisting writes through calling bpf_dynptr_write(). This
* usually looks something like this pattern:
*
* struct eth_hdr *eth = bpf_dynptr_slice_rdwr(&dynptr, 0, buffer, sizeof(buffer));
* if (!eth)
* return TC_ACT_SHOT;
*
* // mutate eth header //
*
* if (eth == buffer)
* bpf_dynptr_write(&ptr, 0, buffer, sizeof(buffer), 0);
*
* Please note that, as in the example above, the user must check that the
* returned pointer is not null before using it.
*
* Please also note that in the case of skb and xdp dynptrs, bpf_dynptr_slice_rdwr
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice_rdwr will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data || __bpf_dynptr_is_rdonly(ptr))
return NULL;
/* bpf_dynptr_slice_rdwr is the same logic as bpf_dynptr_slice.
*
* For skb-type dynptrs, it is safe to write into the returned pointer
* if the bpf program allows skb data writes. There are two possibilities
* that may occur when calling bpf_dynptr_slice_rdwr:
*
* 1) The requested slice is in the head of the skb. In this case, the
* returned pointer is directly to skb data, and if the skb is cloned, the
* verifier will have uncloned it (see bpf_unclone_prologue()) already.
* The pointer can be directly written into.
*
* 2) Some portion of the requested slice is in the paged buffer area.
* In this case, the requested data will be copied out into the buffer
* and the returned pointer will be a pointer to the buffer. The skb
* will not be pulled. To persist the write, the user will need to call
* bpf_dynptr_write(), which will pull the skb and commit the write.
*
* Similarly for xdp programs, if the requested slice is not across xdp
* fragments, then a direct pointer will be returned, otherwise the data
* will be copied out into the buffer and the user will need to call
* bpf_dynptr_write() to commit changes.
*/
return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
}
__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 size;
if (!ptr->data || start > end)
return -EINVAL;
size = __bpf_dynptr_size(ptr);
if (start > size || end > size)
return -ERANGE;
bpf_dynptr_advance_offset(ptr, start);
bpf_dynptr_set_size(ptr, end - start);
return 0;
}
__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
return !ptr->data;
}
__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return false;
return __bpf_dynptr_is_rdonly(ptr);
}
__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return -EINVAL;
return __bpf_dynptr_size(ptr);
}
__bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
struct bpf_dynptr *clone__uninit)
{
struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data) {
bpf_dynptr_set_null(clone);
return -EINVAL;
}
*clone = *ptr;
return 0;
}
/**
* bpf_dynptr_copy() - Copy data from one dynptr to another.
* @dst_ptr: Destination dynptr - where data should be copied to
* @dst_off: Offset into the destination dynptr
* @src_ptr: Source dynptr - where data should be copied from
* @src_off: Offset into the source dynptr
* @size: Length of the data to copy from source to destination
*
* Copies data from source dynptr to destination dynptr.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
{
struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
void *src_slice, *dst_slice;
char buf[256];
u64 off;
src_slice = bpf_dynptr_slice(src_ptr, src_off, NULL, size);
dst_slice = bpf_dynptr_slice_rdwr(dst_ptr, dst_off, NULL, size);
if (src_slice && dst_slice) {
memmove(dst_slice, src_slice, size);
return 0;
}
if (src_slice)
return __bpf_dynptr_write(dst, dst_off, src_slice, size, 0);
if (dst_slice)
return __bpf_dynptr_read(dst_slice, size, src, src_off, 0);
if (bpf_dynptr_check_off_len(dst, dst_off, size) ||
bpf_dynptr_check_off_len(src, src_off, size))
return -E2BIG;
off = 0;
while (off < size) {
u64 chunk_sz = min_t(u64, sizeof(buf), size - off);
int err;
err = __bpf_dynptr_read(buf, chunk_sz, src, src_off + off, 0);
if (err)
return err;
err = __bpf_dynptr_write(dst, dst_off + off, buf, chunk_sz, 0);
if (err)
return err;
off += chunk_sz;
}
return 0;
}
/**
* bpf_dynptr_memset() - Fill dynptr memory with a constant byte.
* @p: Destination dynptr - where data will be filled
* @offset: Offset into the dynptr to start filling from
* @size: Number of bytes to fill
* @val: Constant byte to fill the memory with
*
* Fills the @size bytes of the memory area pointed to by @p
* at @offset with the constant byte @val.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 chunk_sz, write_off;
char buf[256];
void* slice;
int err;
slice = bpf_dynptr_slice_rdwr(p, offset, NULL, size);
if (likely(slice)) {
memset(slice, val, size);
return 0;
}
if (__bpf_dynptr_is_rdonly(ptr))
return -EINVAL;
err = bpf_dynptr_check_off_len(ptr, offset, size);
if (err)
return err;
/* Non-linear data under the dynptr, write from a local buffer */
chunk_sz = min_t(u64, sizeof(buf), size);
memset(buf, val, chunk_sz);
for (write_off = 0; write_off < size; write_off += chunk_sz) {
chunk_sz = min_t(u64, sizeof(buf), size - write_off);
err = __bpf_dynptr_write(ptr, offset + write_off, buf, chunk_sz, 0);
if (err)
return err;
}
return 0;
}
__bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
{
return obj;
}
__bpf_kfunc void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k)
{
return (void *)obj__ign;
}
__bpf_kfunc void bpf_rcu_read_lock(void)
{
rcu_read_lock();
}
__bpf_kfunc void bpf_rcu_read_unlock(void)
{
rcu_read_unlock();
}
struct bpf_throw_ctx {
struct bpf_prog_aux *aux;
u64 sp;
u64 bp;
int cnt;
};
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
{
struct bpf_throw_ctx *ctx = cookie;
struct bpf_prog *prog;
/*
* The RCU read lock is held to safely traverse the latch tree, but we
* don't need its protection when accessing the prog, since it has an
* active stack frame on the current stack trace, and won't disappear.
*/
rcu_read_lock();
prog = bpf_prog_ksym_find(ip);
rcu_read_unlock();
if (!prog)
return !ctx->cnt;
ctx->cnt++;
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
ctx->sp = sp;
ctx->bp = bp;
return false;
}
__bpf_kfunc void bpf_throw(u64 cookie)
{
struct bpf_throw_ctx ctx = {};
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
WARN_ON_ONCE(!ctx.aux->exception_boundary);
WARN_ON_ONCE(!ctx.bp);
WARN_ON_ONCE(!ctx.cnt);
/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
* deeper stack depths than ctx.sp as we do not return from bpf_throw,
* which skips compiler generated instrumentation to do the same.
*/
kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_map *map = p__map;
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq));
if (flags)
return -EINVAL;
return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ);
}
__bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_work *w;
if (flags)
return -EINVAL;
w = READ_ONCE(async->work);
if (!w || !READ_ONCE(w->cb.prog))
return -EINVAL;
if (!refcount_inc_not_zero(&w->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
schedule_work(&w->work);
bpf_async_refcount_put(&w->cb);
return 0;
} else {
return bpf_async_schedule_op(&w->cb, BPF_ASYNC_START, 0, 0);
}
}
__bpf_kfunc int bpf_wq_set_callback(struct bpf_wq *wq,
int (callback_fn)(void *map, int *key, void *value),
unsigned int flags,
struct bpf_prog_aux *aux)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
if (flags)
return -EINVAL;
return __bpf_async_set_callback(async, callback_fn, aux->prog);
}
__bpf_kfunc void bpf_preempt_disable(void)
{
preempt_disable();
}
__bpf_kfunc void bpf_preempt_enable(void)
{
preempt_enable();
}
struct bpf_iter_bits {
__u64 __opaque[2];
} __aligned(8);
#define BITS_ITER_NR_WORDS_MAX 511
struct bpf_iter_bits_kern {
union {
__u64 *bits;
__u64 bits_copy;
};
int nr_bits;
int bit;
} __aligned(8);
/* On 64-bit hosts, unsigned long and u64 have the same size, so passing
* a u64 pointer and an unsigned long pointer to find_next_bit() will
* return the same result, as both point to the same 8-byte area.
*
* For 32-bit little-endian hosts, using a u64 pointer or unsigned long
* pointer also makes no difference. This is because the first iterated
* unsigned long is composed of bits 0-31 of the u64 and the second unsigned
* long is composed of bits 32-63 of the u64.
*
* However, for 32-bit big-endian hosts, this is not the case. The first
* iterated unsigned long will be bits 32-63 of the u64, so swap these two
* ulong values within the u64.
*/
static void swap_ulong_in_u64(u64 *bits, unsigned int nr)
{
#if (BITS_PER_LONG == 32) && defined(__BIG_ENDIAN)
unsigned int i;
for (i = 0; i < nr; i++)
bits[i] = (bits[i] >> 32) | ((u64)(u32)bits[i] << 32);
#endif
}
/**
* bpf_iter_bits_new() - Initialize a new bits iterator for a given memory area
* @it: The new bpf_iter_bits to be created
* @unsafe_ptr__ign: A pointer pointing to a memory area to be iterated over
* @nr_words: The size of the specified memory area, measured in 8-byte units.
* The maximum value of @nr_words is @BITS_ITER_NR_WORDS_MAX. This limit may be
* further reduced by the BPF memory allocator implementation.
*
* This function initializes a new bpf_iter_bits structure for iterating over
* a memory area which is specified by the @unsafe_ptr__ign and @nr_words. It
* copies the data of the memory area to the newly created bpf_iter_bits @it for
* subsequent iteration operations.
*
* On success, 0 is returned. On failure, ERR is returned.
*/
__bpf_kfunc int
bpf_iter_bits_new(struct bpf_iter_bits *it, const u64 *unsafe_ptr__ign, u32 nr_words)
{
struct bpf_iter_bits_kern *kit = (void *)it;
u32 nr_bytes = nr_words * sizeof(u64);
u32 nr_bits = BYTES_TO_BITS(nr_bytes);
int err;
BUILD_BUG_ON(sizeof(struct bpf_iter_bits_kern) != sizeof(struct bpf_iter_bits));
BUILD_BUG_ON(__alignof__(struct bpf_iter_bits_kern) !=
__alignof__(struct bpf_iter_bits));
kit->nr_bits = 0;
kit->bits_copy = 0;
kit->bit = -1;
if (!unsafe_ptr__ign || !nr_words)
return -EINVAL;
if (nr_words > BITS_ITER_NR_WORDS_MAX)
return -E2BIG;
/* Optimization for u64 mask */
if (nr_bits == 64) {
err = bpf_probe_read_kernel_common(&kit->bits_copy, nr_bytes, unsafe_ptr__ign);
if (err)
return -EFAULT;
swap_ulong_in_u64(&kit->bits_copy, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
if (bpf_mem_alloc_check_size(false, nr_bytes))
return -E2BIG;
/* Fallback to memalloc */
kit->bits = bpf_mem_alloc(&bpf_global_ma, nr_bytes);
if (!kit->bits)
return -ENOMEM;
err = bpf_probe_read_kernel_common(kit->bits, nr_bytes, unsafe_ptr__ign);
if (err) {
bpf_mem_free(&bpf_global_ma, kit->bits);
return err;
}
swap_ulong_in_u64(kit->bits, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
/**
* bpf_iter_bits_next() - Get the next bit in a bpf_iter_bits
* @it: The bpf_iter_bits to be checked
*
* This function returns a pointer to a number representing the value of the
* next bit in the bits.
*
* If there are no further bits available, it returns NULL.
*/
__bpf_kfunc int *bpf_iter_bits_next(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
int bit = kit->bit, nr_bits = kit->nr_bits;
const void *bits;
if (!nr_bits || bit >= nr_bits)
return NULL;
bits = nr_bits == 64 ? &kit->bits_copy : kit->bits;
bit = find_next_bit(bits, nr_bits, bit + 1);
if (bit >= nr_bits) {
kit->bit = bit;
return NULL;
}
kit->bit = bit;
return &kit->bit;
}
/**
* bpf_iter_bits_destroy() - Destroy a bpf_iter_bits
* @it: The bpf_iter_bits to be destroyed
*
* Destroy the resource associated with the bpf_iter_bits.
*/
__bpf_kfunc void bpf_iter_bits_destroy(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
if (kit->nr_bits <= 64)
return;
bpf_mem_free(&bpf_global_ma, kit->bits);
}
/**
* bpf_copy_from_user_str() - Copy a string from an unsafe user address
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address, in user space.
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL-terminated string from userspace to BPF space. If user string is
* too long this will still ensure zero termination in the dst buffer unless
* buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst to 0 on success and
* memset all of @dst on failure.
*/
__bpf_kfunc int bpf_copy_from_user_str(void *dst, u32 dst__sz, const void __user *unsafe_ptr__ign, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(!dst__sz))
return 0;
ret = strncpy_from_user(dst, unsafe_ptr__ign, dst__sz - 1);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst + ret, 0, dst__sz - ret);
else
((char *)dst)[ret] = '\0';
return ret + 1;
}
/**
* bpf_copy_from_user_task_str() - Copy a string from an task's address space
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address in the task's address space.
* @tsk: The task whose address space will be used
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL terminated string from a task's address space to @dst__sz
* buffer. If user string is too long this will still ensure zero termination
* in the @dst__sz buffer unless buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst__sz to 0 on success
* and memset all of @dst__sz on failure.
*
* Return: The number of copied bytes on success including the NUL terminator.
* A negative error code on failure.
*/
__bpf_kfunc int bpf_copy_from_user_task_str(void *dst, u32 dst__sz,
const void __user *unsafe_ptr__ign,
struct task_struct *tsk, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(dst__sz == 0))
return 0;
ret = copy_remote_vm_str(tsk, (unsigned long)unsafe_ptr__ign, dst, dst__sz, 0);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset(dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset(dst + ret, 0, dst__sz - ret);
return ret + 1;
}
/* Keep unsinged long in prototype so that kfunc is usable when emitted to
* vmlinux.h in BPF programs directly, but note that while in BPF prog, the
* unsigned long always points to 8-byte region on stack, the kernel may only
* read and write the 4-bytes on 32-bit.
*/
__bpf_kfunc void bpf_local_irq_save(unsigned long *flags__irq_flag)
{
local_irq_save(*flags__irq_flag);
}
__bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)
{
local_irq_restore(*flags__irq_flag);
}
__bpf_kfunc void __bpf_trap(void)
{
}
/*
* Kfuncs for string operations.
*
* Since strings are not necessarily %NUL-terminated, we cannot directly call
* in-kernel implementations. Instead, we open-code the implementations using
* __get_kernel_nofault instead of plain dereference to make them safe.
*/
static int __bpf_strncasecmp(const char *s1, const char *s2, bool ignore_case, size_t len)
{
char c1, c2;
int i;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < len && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c1, s1, char, err_out);
__get_kernel_nofault(&c2, s2, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 != c2)
return c1 < c2 ? -1 : 1;
if (c1 == '\0')
return 0;
s1++;
s2++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : 0;
err_out:
return -EFAULT;
}
/**
* bpf_strcmp - Compare two strings
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, false, XATTR_SIZE_MAX);
}
/**
* bpf_strcasecmp - Compare two strings, ignoring the case of the characters
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasecmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, XATTR_SIZE_MAX);
}
/*
* bpf_strncasecmp - Compare two length-limited strings, ignoring case
* @s1__ign: One string
* @s2__ign: Another string
* @len: The maximum number of characters to compare
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasecmp(const char *s1__ign, const char *s2__ign, size_t len)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, len);
}
/**
* bpf_strnchr - Find a character in a length limited string
* @s__ign: The string to be searched
* @count: The number of characters to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in the first @count characters of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnchr(const char *s__ign, size_t count, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
return i;
if (sc == '\0')
return -ENOENT;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : -ENOENT;
err_out:
return -EFAULT;
}
/**
* bpf_strchr - Find the first occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - The index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchr(const char *s__ign, char c)
{
return bpf_strnchr(s__ign, XATTR_SIZE_MAX, c);
}
/**
* bpf_strchrnul - Find and return a character in a string, or end of string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign or index of
* the null byte at the end of @s__ign when @c is not found
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchrnul(const char *s__ign, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == '\0' || sc == c)
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strrchr - Find the last occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the last occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strrchr(const char *s__ign, int c)
{
char sc;
int i, last = -ENOENT;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
last = i;
if (sc == '\0')
return last;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strnlen - Calculate the length of a length-limited string
* @s__ign: The string
* @count: The maximum number of characters to count
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnlen(const char *s__ign, size_t count)
{
char c;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c, s__ign, char, err_out);
if (c == '\0')
return i;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : i;
err_out:
return -EFAULT;
}
/**
* bpf_strlen - Calculate the length of a string
* @s__ign: The string
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strlen(const char *s__ign)
{
return bpf_strnlen(s__ign, XATTR_SIZE_MAX);
}
/**
* bpf_strspn - Calculate the length of the initial substring of @s__ign which
* only contains letters in @accept__ign
* @s__ign: The string to be searched
* @accept__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which only
* contains letters from @accept__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strspn(const char *s__ign, const char *accept__ign)
{
char cs, ca;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(accept__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&ca, accept__ign + j, char, err_out);
if (cs == ca || ca == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (ca == '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strcspn - Calculate the length of the initial substring of @s__ign which
* does not contain letters in @reject__ign
* @s__ign: The string to be searched
* @reject__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which does not
* contain letters from @reject__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcspn(const char *s__ign, const char *reject__ign)
{
char cs, cr;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(reject__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&cr, reject__ign + j, char, err_out);
if (cs == cr || cr == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (cr != '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
static int __bpf_strnstr(const char *s1, const char *s2, size_t len,
bool ignore_case)
{
char c1, c2;
int i, j;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
for (j = 0; i + j <= len && j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&c2, s2 + j, char, err_out);
if (c2 == '\0')
return i;
/*
* We allow reading an extra byte from s2 (note the
* `i + j <= len` above) to cover the case when s2 is
* a suffix of the first len chars of s1.
*/
if (i + j == len)
break;
__get_kernel_nofault(&c1, s1 + j, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 == '\0')
return -ENOENT;
if (c1 != c2)
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (i + j == len)
return -ENOENT;
s1++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strstr - Find the first substring in a string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strstr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, false);
}
/**
* bpf_strcasestr - Find the first substring in a string, ignoring the case of
* the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasestr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, true);
}
/**
* bpf_strnstr - Find the first substring in a length-limited string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strnstr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, false);
}
/**
* bpf_strncasestr - Find the first substring in a length-limited string,
* ignoring the case of the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasestr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, true);
}
#ifdef CONFIG_KEYS
/**
* bpf_lookup_user_key - lookup a key by its serial
* @serial: key handle serial number
* @flags: lookup-specific flags
*
* Search a key with a given *serial* and the provided *flags*.
* If found, increment the reference count of the key by one, and
* return it in the bpf_key structure.
*
* The bpf_key structure must be passed to bpf_key_put() when done
* with it, so that the key reference count is decremented and the
* bpf_key structure is freed.
*
* Permission checks are deferred to the time the key is used by
* one of the available key-specific kfuncs.
*
* Set *flags* with KEY_LOOKUP_CREATE, to attempt creating a requested
* special keyring (e.g. session keyring), if it doesn't yet exist.
* Set *flags* with KEY_LOOKUP_PARTIAL, to lookup a key without waiting
* for the key construction, and to retrieve uninstantiated keys (keys
* without data attached to them).
*
* Return: a bpf_key pointer with a valid key pointer if the key is found, a
* NULL pointer otherwise.
*/
__bpf_kfunc struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags)
{
key_ref_t key_ref;
struct bpf_key *bkey;
if (flags & ~KEY_LOOKUP_ALL)
return NULL;
/*
* Permission check is deferred until the key is used, as the
* intent of the caller is unknown here.
*/
key_ref = lookup_user_key(serial, flags, KEY_DEFER_PERM_CHECK);
if (IS_ERR(key_ref))
return NULL;
bkey = kmalloc_obj(*bkey);
if (!bkey) {
key_put(key_ref_to_ptr(key_ref));
return NULL;
}
bkey->key = key_ref_to_ptr(key_ref);
bkey->has_ref = true;
return bkey;
}
/**
* bpf_lookup_system_key - lookup a key by a system-defined ID
* @id: key ID
*
* Obtain a bpf_key structure with a key pointer set to the passed key ID.
* The key pointer is marked as invalid, to prevent bpf_key_put() from
* attempting to decrement the key reference count on that pointer. The key
* pointer set in such way is currently understood only by
* verify_pkcs7_signature().
*
* Set *id* to one of the values defined in include/linux/verification.h:
* 0 for the primary keyring (immutable keyring of system keys);
* VERIFY_USE_SECONDARY_KEYRING for both the primary and secondary keyring
* (where keys can be added only if they are vouched for by existing keys
* in those keyrings); VERIFY_USE_PLATFORM_KEYRING for the platform
* keyring (primarily used by the integrity subsystem to verify a kexec'ed
* kerned image and, possibly, the initramfs signature).
*
* Return: a bpf_key pointer with an invalid key pointer set from the
* pre-determined ID on success, a NULL pointer otherwise
*/
__bpf_kfunc struct bpf_key *bpf_lookup_system_key(u64 id)
{
struct bpf_key *bkey;
if (system_keyring_id_check(id) < 0)
return NULL;
bkey = kmalloc_obj(*bkey, GFP_ATOMIC);
if (!bkey)
return NULL;
bkey->key = (struct key *)(unsigned long)id;
bkey->has_ref = false;
return bkey;
}
/**
* bpf_key_put - decrement key reference count if key is valid and free bpf_key
* @bkey: bpf_key structure
*
* Decrement the reference count of the key inside *bkey*, if the pointer
* is valid, and free *bkey*.
*/
__bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
{
if (bkey->has_ref)
key_put(bkey->key);
kfree(bkey);
}
/**
* bpf_verify_pkcs7_signature - verify a PKCS#7 signature
* @data_p: data to verify
* @sig_p: signature of the data
* @trusted_keyring: keyring with keys trusted for signature verification
*
* Verify the PKCS#7 signature *sig_ptr* against the supplied *data_ptr*
* with keys in a keyring referenced by *trusted_keyring*.
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const void *data, *sig;
u32 data_len, sig_len;
int ret;
if (trusted_keyring->has_ref) {
/*
* Do the permission check deferred in bpf_lookup_user_key().
* See bpf_lookup_user_key() for more details.
*
* A call to key_task_permission() here would be redundant, as
* it is already done by keyring_search() called by
* find_asymmetric_key().
*/
ret = key_validate(trusted_keyring->key);
if (ret < 0)
return ret;
}
data_len = __bpf_dynptr_size(data_ptr);
data = __bpf_dynptr_data(data_ptr, data_len);
sig_len = __bpf_dynptr_size(sig_ptr);
sig = __bpf_dynptr_data(sig_ptr, sig_len);
return verify_pkcs7_signature(data, data_len, sig, sig_len,
trusted_keyring->key,
VERIFYING_BPF_SIGNATURE, NULL,
NULL);
#else
return -EOPNOTSUPP;
#endif /* CONFIG_SYSTEM_DATA_VERIFICATION */
}
#endif /* CONFIG_KEYS */
typedef int (*bpf_task_work_callback_t)(struct bpf_map *map, void *key, void *value);
enum bpf_task_work_state {
/* bpf_task_work is ready to be used */
BPF_TW_STANDBY = 0,
/* irq work scheduling in progress */
BPF_TW_PENDING,
/* task work scheduling in progress */
BPF_TW_SCHEDULING,
/* task work is scheduled successfully */
BPF_TW_SCHEDULED,
/* callback is running */
BPF_TW_RUNNING,
/* associated BPF map value is deleted */
BPF_TW_FREED,
};
struct bpf_task_work_ctx {
enum bpf_task_work_state state;
refcount_t refcnt;
struct callback_head work;
struct irq_work irq_work;
/* bpf_prog that schedules task work */
struct bpf_prog *prog;
/* task for which callback is scheduled */
struct task_struct *task;
/* the map and map value associated with this context */
struct bpf_map *map;
void *map_val;
enum task_work_notify_mode mode;
bpf_task_work_callback_t callback_fn;
struct rcu_head rcu;
} __aligned(8);
/* Actual type for struct bpf_task_work */
struct bpf_task_work_kern {
struct bpf_task_work_ctx *ctx;
};
static void bpf_task_work_ctx_reset(struct bpf_task_work_ctx *ctx)
{
if (ctx->prog) {
bpf_prog_put(ctx->prog);
ctx->prog = NULL;
}
if (ctx->task) {
bpf_task_release(ctx->task);
ctx->task = NULL;
}
}
static bool bpf_task_work_ctx_tryget(struct bpf_task_work_ctx *ctx)
{
return refcount_inc_not_zero(&ctx->refcnt);
}
static void bpf_task_work_destroy(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_ctx_reset(ctx);
kfree_rcu(ctx, rcu);
}
static void bpf_task_work_ctx_put(struct bpf_task_work_ctx *ctx)
{
if (!refcount_dec_and_test(&ctx->refcnt))
return;
if (irqs_disabled()) {
ctx->irq_work = IRQ_WORK_INIT(bpf_task_work_destroy);
irq_work_queue(&ctx->irq_work);
} else {
bpf_task_work_destroy(&ctx->irq_work);
}
}
static void bpf_task_work_cancel(struct bpf_task_work_ctx *ctx)
{
/*
* Scheduled task_work callback holds ctx ref, so if we successfully
* cancelled, we put that ref on callback's behalf. If we couldn't
* cancel, callback will inevitably run or has already completed
* running, and it would have taken care of its ctx ref itself.
*/
if (task_work_cancel(ctx->task, &ctx->work))
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_callback(struct callback_head *cb)
{
struct bpf_task_work_ctx *ctx = container_of(cb, struct bpf_task_work_ctx, work);
enum bpf_task_work_state state;
u32 idx;
void *key;
/* Read lock is needed to protect ctx and map key/value access */
guard(rcu_tasks_trace)();
/*
* This callback may start running before bpf_task_work_irq() switched to
* SCHEDULED state, so handle both transition variants SCHEDULING|SCHEDULED -> RUNNING.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_RUNNING);
if (state == BPF_TW_SCHEDULED)
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULED, BPF_TW_RUNNING);
if (state == BPF_TW_FREED) {
bpf_task_work_ctx_put(ctx);
return;
}
key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx);
migrate_disable();
ctx->callback_fn(ctx->map, key, ctx->map_val);
migrate_enable();
bpf_task_work_ctx_reset(ctx);
(void)cmpxchg(&ctx->state, BPF_TW_RUNNING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_irq(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
enum bpf_task_work_state state;
int err;
guard(rcu)();
if (cmpxchg(&ctx->state, BPF_TW_PENDING, BPF_TW_SCHEDULING) != BPF_TW_PENDING) {
bpf_task_work_ctx_put(ctx);
return;
}
err = task_work_add(ctx->task, &ctx->work, ctx->mode);
if (err) {
bpf_task_work_ctx_reset(ctx);
/*
* try to switch back to STANDBY for another task_work reuse, but we might have
* gone to FREED already, which is fine as we already cleaned up after ourselves
*/
(void)cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
return;
}
/*
* It's technically possible for just scheduled task_work callback to
* complete running by now, going SCHEDULING -> RUNNING and then
* dropping its ctx refcount. Instead of capturing an extra ref just
* to protect below ctx->state access, we rely on rcu_read_lock
* above to prevent kfree_rcu from freeing ctx before we return.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_SCHEDULED);
if (state == BPF_TW_FREED)
bpf_task_work_cancel(ctx); /* clean up if we switched into FREED state */
}
static struct bpf_task_work_ctx *bpf_task_work_fetch_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_kern *twk = (void *)tw;
struct bpf_task_work_ctx *ctx, *old_ctx;
ctx = READ_ONCE(twk->ctx);
if (ctx)
return ctx;
ctx = bpf_map_kmalloc_nolock(map, sizeof(*ctx), 0, NUMA_NO_NODE);
if (!ctx)
return ERR_PTR(-ENOMEM);
memset(ctx, 0, sizeof(*ctx));
refcount_set(&ctx->refcnt, 1); /* map's own ref */
ctx->state = BPF_TW_STANDBY;
old_ctx = cmpxchg(&twk->ctx, NULL, ctx);
if (old_ctx) {
/*
* tw->ctx is set by concurrent BPF program, release allocated
* memory and try to reuse already set context.
*/
kfree_nolock(ctx);
return old_ctx;
}
return ctx; /* Success */
}
static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_ctx *ctx;
/*
* Sleepable BPF programs hold rcu_read_lock_trace but not
* regular rcu_read_lock. Since kfree_rcu waits for regular
* RCU GP, the ctx can be freed while we're between reading
* the pointer and incrementing the refcount. Take regular
* rcu_read_lock to prevent kfree_rcu from freeing the ctx
* before we can tryget it.
*/
scoped_guard(rcu) {
ctx = bpf_task_work_fetch_ctx(tw, map);
if (IS_ERR(ctx))
return ctx;
/* try to get ref for task_work callback to hold */
if (!bpf_task_work_ctx_tryget(ctx))
return ERR_PTR(-EBUSY);
}
if (cmpxchg(&ctx->state, BPF_TW_STANDBY, BPF_TW_PENDING) != BPF_TW_STANDBY) {
/* lost acquiring race or map_release_uref() stole it from us, put ref and bail */
bpf_task_work_ctx_put(ctx);
return ERR_PTR(-EBUSY);
}
/*
* If no process or bpffs is holding a reference to the map, no new callbacks should be
* scheduled. This does not address any race or correctness issue, but rather is a policy
* choice: dropping user references should stop everything.
*/
if (!atomic64_read(&map->usercnt)) {
/* drop ref we just got for task_work callback itself */
bpf_task_work_ctx_put(ctx);
/* transfer map's ref into cancel_and_free() */
bpf_task_work_cancel_and_free(tw);
return ERR_PTR(-EBUSY);
}
return ctx;
}
static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw,
struct bpf_map *map, bpf_task_work_callback_t callback_fn,
struct bpf_prog_aux *aux, enum task_work_notify_mode mode)
{
struct bpf_prog *prog;
struct bpf_task_work_ctx *ctx;
int err;
BTF_TYPE_EMIT(struct bpf_task_work);
prog = bpf_prog_inc_not_zero(aux->prog);
if (IS_ERR(prog))
return -EBADF;
task = bpf_task_acquire(task);
if (!task) {
err = -EBADF;
goto release_prog;
}
ctx = bpf_task_work_acquire_ctx(tw, map);
if (IS_ERR(ctx)) {
err = PTR_ERR(ctx);
goto release_all;
}
ctx->task = task;
ctx->callback_fn = callback_fn;
ctx->prog = prog;
ctx->mode = mode;
ctx->map = map;
ctx->map_val = (void *)tw - map->record->task_work_off;
init_task_work(&ctx->work, bpf_task_work_callback);
init_irq_work(&ctx->irq_work, bpf_task_work_irq);
irq_work_queue(&ctx->irq_work);
return 0;
release_all:
bpf_task_release(task);
release_prog:
bpf_prog_put(prog);
return err;
}
/**
* bpf_task_work_schedule_signal - Schedule BPF callback using task_work_add with TWA_SIGNAL
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL);
}
/**
* bpf_task_work_schedule_resume - Schedule BPF callback using task_work_add with TWA_RESUME
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME);
}
static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
struct bpf_dynptr_kern *ptr)
{
struct bpf_dynptr_file_impl *state;
/* flags is currently unsupported */
if (flags) {
bpf_dynptr_set_null(ptr);
return -EINVAL;
}
state = kmalloc_nolock(sizeof(*state), 0, NUMA_NO_NODE);
if (!state) {
bpf_dynptr_set_null(ptr);
return -ENOMEM;
}
state->offset = 0;
state->size = U64_MAX; /* Don't restrict size, as file may change anyways */
freader_init_from_file(&state->freader, NULL, 0, file, may_sleep);
bpf_dynptr_init(ptr, state, BPF_DYNPTR_TYPE_FILE, 0, 0);
bpf_dynptr_set_rdonly(ptr);
return 0;
}
__bpf_kfunc int bpf_dynptr_from_file(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, false, (struct bpf_dynptr_kern *)ptr__uninit);
}
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, true, (struct bpf_dynptr_kern *)ptr__uninit);
}
__bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)dynptr;
struct bpf_dynptr_file_impl *df = ptr->data;
if (!df)
return 0;
freader_cleanup(&df->freader);
kfree_nolock(df);
bpf_dynptr_set_null(ptr);
return 0;
}
/**
* bpf_timer_cancel_async - try to deactivate a timer
* @timer: bpf_timer to stop
*
* Returns:
*
* * 0 when the timer was not active
* * 1 when the timer was active
* * -1 when the timer is currently executing the callback function and
* cannot be stopped
* * -ECANCELED when the timer will be cancelled asynchronously
* * -ENOMEM when out of memory
* * -EINVAL when the timer was not initialized
* * -ENOENT when this kfunc is racing with timer deletion
*/
__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer)
{
struct bpf_async_kern *async = (void *)timer;
struct bpf_async_cb *cb;
int ret;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
/*
* Unlike hrtimer_start() it's ok to synchronously call
* hrtimer_try_to_cancel() when refcnt reached zero, but deferring to
* irq_work is not, since irq callback may execute after RCU GP and
* cb could be freed at that time. Check for refcnt zero for
* consistency.
*/
if (!refcount_inc_not_zero(&cb->refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
ret = hrtimer_try_to_cancel(&t->timer);
bpf_async_refcount_put(cb);
return ret;
} else {
ret = bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
return ret ? ret : -ECANCELED;
}
}
__bpf_kfunc_end_defs();
static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_cancel(ctx); /* this might put task_work callback's ref */
bpf_task_work_ctx_put(ctx); /* and here we put map's own ref that was transferred to us */
}
void bpf_task_work_cancel_and_free(void *val)
{
struct bpf_task_work_kern *twk = val;
struct bpf_task_work_ctx *ctx;
enum bpf_task_work_state state;
ctx = xchg(&twk->ctx, NULL);
if (!ctx)
return;
state = xchg(&ctx->state, BPF_TW_FREED);
if (state == BPF_TW_SCHEDULED) {
/* run in irq_work to avoid locks in NMI */
init_irq_work(&ctx->irq_work, bpf_task_work_cancel_scheduled);
irq_work_queue(&ctx->irq_work);
return;
}
bpf_task_work_ctx_put(ctx); /* put bpf map's ref */
}
BTF_KFUNCS_START(generic_btf_ids)
#ifdef CONFIG_CRASH_DUMP
BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
#endif
BTF_ID_FLAGS(func, bpf_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_percpu_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_refcount_acquire, KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE | KF_RET_NULL | KF_RCU)
BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_front_impl)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_back_impl)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_rbtree_add_impl)
BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_cgroup_ancestor, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU)
BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
#endif
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_send_signal_task)
#endif
#ifdef CONFIG_KEYS
BTF_ID_FLAGS(func, bpf_lookup_user_key, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_lookup_system_key, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_key_put, KF_RELEASE)
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
BTF_ID_FLAGS(func, bpf_verify_pkcs7_signature, KF_SLEEPABLE)
#endif
#endif
#ifdef CONFIG_S390
BTF_ID_FLAGS(func, bpf_get_lowcore)
#endif
BTF_KFUNCS_END(generic_btf_ids)
static const struct btf_kfunc_id_set generic_kfunc_set = {
.owner = THIS_MODULE,
.set = &generic_btf_ids,
};
BTF_ID_LIST(generic_dtor_ids)
BTF_ID(struct, task_struct)
BTF_ID(func, bpf_task_release_dtor)
#ifdef CONFIG_CGROUPS
BTF_ID(struct, cgroup)
BTF_ID(func, bpf_cgroup_release_dtor)
#endif
BTF_KFUNCS_START(common_btf_ids)
BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rcu_read_lock)
BTF_ID_FLAGS(func, bpf_rcu_read_unlock)
BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_css_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_css_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_destroy, KF_ITER_DESTROY)
#endif
BTF_ID_FLAGS(func, bpf_iter_task_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_dynptr_adjust)
BTF_ID_FLAGS(func, bpf_dynptr_is_null)
BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
BTF_ID_FLAGS(func, bpf_dynptr_size)
BTF_ID_FLAGS(func, bpf_dynptr_clone)
BTF_ID_FLAGS(func, bpf_dynptr_copy)
BTF_ID_FLAGS(func, bpf_dynptr_memset)
#ifdef CONFIG_NET
BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
#endif
BTF_ID_FLAGS(func, bpf_wq_init)
BTF_ID_FLAGS(func, bpf_wq_set_callback, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_wq_start)
BTF_ID_FLAGS(func, bpf_preempt_disable)
BTF_ID_FLAGS(func, bpf_preempt_enable)
BTF_ID_FLAGS(func, bpf_iter_bits_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_bits_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_bits_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_copy_from_user_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_get_kmem_cache)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_local_irq_save)
BTF_ID_FLAGS(func, bpf_local_irq_restore)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr)
BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE)
#endif
#ifdef CONFIG_DMA_SHARED_BUFFER
BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
#endif
BTF_ID_FLAGS(func, __bpf_trap)
BTF_ID_FLAGS(func, bpf_strcmp);
BTF_ID_FLAGS(func, bpf_strcasecmp);
BTF_ID_FLAGS(func, bpf_strncasecmp);
BTF_ID_FLAGS(func, bpf_strchr);
BTF_ID_FLAGS(func, bpf_strchrnul);
BTF_ID_FLAGS(func, bpf_strnchr);
BTF_ID_FLAGS(func, bpf_strrchr);
BTF_ID_FLAGS(func, bpf_strlen);
BTF_ID_FLAGS(func, bpf_strnlen);
BTF_ID_FLAGS(func, bpf_strspn);
BTF_ID_FLAGS(func, bpf_strcspn);
BTF_ID_FLAGS(func, bpf_strstr);
BTF_ID_FLAGS(func, bpf_strcasestr);
BTF_ID_FLAGS(func, bpf_strnstr);
BTF_ID_FLAGS(func, bpf_strncasestr);
#if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS)
BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU)
#endif
BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_dynptr_from_file)
BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
BTF_ID_FLAGS(func, bpf_timer_cancel_async)
BTF_KFUNCS_END(common_btf_ids)
static const struct btf_kfunc_id_set common_kfunc_set = {
.owner = THIS_MODULE,
.set = &common_btf_ids,
};
static int __init kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc generic_dtors[] = {
{
.btf_id = generic_dtor_ids[0],
.kfunc_btf_id = generic_dtor_ids[1]
},
#ifdef CONFIG_CGROUPS
{
.btf_id = generic_dtor_ids[2],
.kfunc_btf_id = generic_dtor_ids[3]
},
#endif
};
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &generic_kfunc_set);
ret = ret ?: register_btf_id_dtor_kfuncs(generic_dtors,
ARRAY_SIZE(generic_dtors),
THIS_MODULE);
return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
}
late_initcall(kfunc_init);
/* Get a pointer to dynptr data up to len bytes for read only access. If
* the dynptr doesn't have continuous data up to len bytes, return NULL.
*/
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u64 len)
{
const struct bpf_dynptr *p = (struct bpf_dynptr *)ptr;
return bpf_dynptr_slice(p, 0, NULL, len);
}
/* Get a pointer to dynptr data up to len bytes for read write access. If
* the dynptr doesn't have continuous data up to len bytes, or the dynptr
* is read only, return NULL.
*/
void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u64 len)
{
if (__bpf_dynptr_is_rdonly(ptr))
return NULL;
return (void *)__bpf_dynptr_data(ptr, len);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *val)
{
if (btf_record_has_field(map->record, BPF_TIMER))
bpf_obj_free_timer(map->record, val);
if (btf_record_has_field(map->record, BPF_WORKQUEUE))
bpf_obj_free_workqueue(map->record, val);
if (btf_record_has_field(map->record, BPF_TASK_WORK))
bpf_obj_free_task_work(map->record, val);
}
]
|
| 605/6 |
2026/05/18 19:35 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 147357
output: 760
thoughts: 305
Thoughts: **My Assessment of `kernel/bpf/helpers.c` and `snprintf` Usage**
Alright, let's break this down. I've just meticulously examined the entirety of `kernel/bpf/helpers.c` and specifically zoned in on the use of `snprintf`. My findings are pretty straightforward: this file isn't riddled with dangerous `snprintf` calls, but there are a couple of points that warrant attention.
Specifically, I see `snprintf` used *only* twice within the `bpf_bprintf_prepare` function, once at line 924 and again at 965. The critical piece here is the potential for truncation within the `snprintf` function calls. If `snprintf` were to truncate the output, the subsequent pointer arithmetic `tmp_buf += (err + 1);` or `tmp_buf += err + 1;` could lead to out-of-bounds access. The code relies on `err` (the return value of `snprintf`), which represents the number of characters *that would have* been written, *not* necessarily the number of characters *actually* written if truncation occurs.
Importantly, I've confirmed that there are no other instances of `snprintf` in this file, which is good. I checked for and didn't find any instances of the related, and potentially more dangerous, functions like `scnprintf`, `vsnprintf`, `sprintf`, `vscnprintf`, or `seq_printf`. I've also checked other related functions, to make sure there are no other issues, such as the `bstr_printf` call in `____bpf_snprintf`, which is safe and the calls to `strncpy_from_user_nofault` and `strncpy_from_kernel_nofault` in `bpf_trace_copy_string`, which are safe because they correctly manage the buffer size and return the actual number of bytes copied.
Based on this comprehensive review, I'm confident in identifying the two `snprintf` calls within `bpf_bprintf_prepare` as the only potential vulnerabilities.
|
| 606/4 |
2026/05/18 19:36 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 96039
output: 636
thoughts: 876
Thoughts: **Finalizing the Response**
Okay, the codeexpert's confirmation aligns perfectly with my own analysis – excellent. I'm now ready to formulate the definitive response.
The core of the fix is sound. I'm specifically addressing a few key issues:
1. First, I've corrected the `snprintf` return value handling for `%pB` and `%pI` formats. The implementation correctly limits the increment of `tmp_buf` to the available space using `limit`. This prevents buffer overflows which is critical.
2. Next, I've resolved an integer signedness bug in bounds checking. The logic now correctly checks available buffer space by ensuring that `tmp_buf + sizeof_cur_arg` is within the buffer boundaries.
3. Finally, I've hardened the equality checks for `%s` and `%c` formats to be `tmp_buf >= tmp_buf_end`, which is a more robust check.
From a software engineering perspective, I'm confident that this fix addresses the underlying root cause comprehensively.
I did a quick sanity check on the patch and it checks out. I confirmed that:
* The `%pB` handling is correct when `tmp_buf` is NULL, and when it isn't. The calculation of the size limit and the advancement of `tmp_buf` after `snprintf` are correct.
* The IP address formatting using `snprintf` is safe, even though there's a small hack.
* The pointer arithmetic within the bounds checking logic is safe because `tmp_buf` is guaranteed to be non-NULL.
* The `PTR_ALIGN` and bounds check logic in the final block are also correct.
The patch is solid. Time to write the response.
|
| 607/3 |
2026/05/18 19:36 |
action |
test-patch |
45m
Results: map[PatchDiff:diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2bb60200c..be70abbc4 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -921,11 +921,19 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
- err = snprintf(tmp_buf,
- (tmp_buf_end - tmp_buf),
- "%pB",
- (void *)(long)raw_args[num_spec]);
- tmp_buf += (err + 1);
+ size_t limit =
+ tmp_buf < tmp_buf_end ?
+ tmp_buf_end - tmp_buf :
+ 0;
+
+ err = snprintf(
+ tmp_buf, limit, "%pB",
+ (void *)(long)
+ raw_args[num_spec]);
+ if (err < 0)
+ err = 0;
+ tmp_buf +=
+ min_t(size_t, err + 1, limit);
}
i++;
@@ -945,7 +953,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
- if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
+ if (tmp_buf + sizeof_cur_ip > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -962,10 +970,17 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
- err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
- ip_spec, &cur_ip);
-
- tmp_buf += err + 1;
+ {
+ size_t limit = tmp_buf < tmp_buf_end ?
+ tmp_buf_end - tmp_buf :
+ 0;
+
+ err = snprintf(tmp_buf, limit, ip_spec,
+ &cur_ip);
+ if (err < 0)
+ err = 0;
+ tmp_buf += min_t(size_t, err + 1, limit);
+ }
num_spec++;
continue;
@@ -982,7 +997,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1004,7 +1019,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1038,7 +1053,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
- if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
TestError:watchdog: BUG: soft lockup - CPU#1 stuck for 46s! [swapper/1:0]
CPU#1 Utilization every 4000ms during lockup:
#1: 26% system, 3% softirq, 3% hardirq, 70% idle
#2: 48% system, 5% softirq, 3% hardirq, 46% idle
#3: 54% system, 4% softirq, 3% hardirq, 38% idle
#4: 0% system, 0% softirq, 79% hardirq, 1% idle
#5: 0% system, 0% softirq, 0% hardirq, 1% idle
Modules linked in:
irq event stamp: 4886784
hardirqs last enabled at (4886783): [<ffffffff81654d3a>] cpuidle_idle_call kernel/sched/idle.c:199 [inline]
hardirqs last enabled at (4886783): [<ffffffff81654d3a>] do_idle+0x36a/0x5e0 kernel/sched/idle.c:352
hardirqs last disabled at (4886784): [<ffffffff8b9eb11e>] sysvec_apic_timer_interrupt+0xe/0xb0 arch/x86/kernel/apic/apic.c:1061
softirqs last enabled at (4886760): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last enabled at (4886760): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last enabled at (4886760): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
softirqs last disabled at (4886221): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last disabled at (4886221): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last disabled at (4886221): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:native_irq_disable arch/x86/include/asm/irqflags.h:37 [inline]
RIP: 0010:arch_local_irq_disable arch/x86/include/asm/irqflags.h:114 [inline]
RIP: 0010:default_idle+0xf/0x20 arch/x86/kernel/process.c:768
Code: 4c 01 c7 4c 29 c2 e9 6e ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 13 7e 23 00 fb f4 <fa> e9 8b 8d 02 00 cc 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
RSP: 0018:ffffc9000014fe28 EFLAGS: 00000206
RAX: 00000000004a90ff RBX: ffffffff81654d3a RCX: 0000000080000001
RDX: 0000000000000001 RSI: ffffffff8e2b9503 RDI: ffffffff8c1c60a0
RBP: ffffc9000014ff20 R08: ffff888069532993 R09: 1ffff1100d2a6532
R10: dffffc0000000000 R11: ffffed100d2a6533 R12: 0000000000000001
R13: 1ffff11002c19000 R14: 0000000000000001 R15: 1ffff11002c19000
FS: 0000000000000000(0000) GS:ffff8880d5e54000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe1f92f1ef0 CR3: 000000002225a000 CR4: 00000000000006f0
Call Trace:
<TASK>
default_idle_call+0x70/0xb0 kernel/sched/idle.c:122
cpuidle_idle_call kernel/sched/idle.c:199 [inline]
do_idle+0x36a/0x5e0 kernel/sched/idle.c:352
cpu_startup_entry+0x43/0x60 kernel/sched/idle.c:451
start_secondary+0x101/0x110 arch/x86/kernel/smpboot.c:312
common_startup_64+0x12c/0x137
</TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 4676 Comm: dhcpcd Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:lookup_chain_cache kernel/locking/lockdep.c:3802 [inline]
RIP: 0010:lookup_chain_cache_add kernel/locking/lockdep.c:3821 [inline]
RIP: 0010:validate_chain kernel/locking/lockdep.c:3876 [inline]
RIP: 0010:__lock_acquire+0xc4b/0x2d80 kernel/locking/lockdep.c:5237
Code: 48 ff cb f7 c1 00 60 00 00 74 c4 e9 90 00 00 00 48 c7 c7 d0 88 99 8e 4c 89 fe e8 b0 9c 24 03 48 8b 54 24 18 eb c4 48 8b 48 18 <48> 39 e9 0f 84 b8 00 00 00 48 8b 40 08 48 85 c0 0f 84 cc fe ff ff
RSP: 0018:ffffc900000078d0 EFLAGS: 00000046
RAX: ffffffff93a979b0 RBX: 0000000000003f6d RCX: dbc3fa7544b72e03
RDX: 000000009b99b500 RSI: 00000000d40ffd9b RDI: ffff888023900000
RBP: dbc3fa7544b72e03 R08: ffff888069428c60 R09: ffffffff818f225b
R10: dffffc0000000000 R11: fffffbfff20c9f5f R12: 0000000000000001
R13: ffff888023900ba0 R14: ffff888023900000 R15: 0000000000000002
FS: 00007f4300fae780(0000) GS:ffff8880d5d54000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555576e3c490 CR3: 0000000019bd4000 CR4: 00000000000006f0
Call Trace:
<IRQ>
lock_acquire+0x106/0x350 kernel/locking/lockdep.c:5868
local_lock_acquire include/linux/local_lock_internal.h:46 [inline]
kcov_remote_stop+0x84/0x6c0 kernel/kcov.c:1030
ieee80211_rx_list+0x2b2a/0x3690 net/mac80211/rx.c:5641
ieee80211_rx_napi+0x1b1/0x3e0 net/mac80211/rx.c:5661
ieee80211_rx include/net/mac80211.h:5395 [inline]
ieee80211_handle_queued_frames+0xe8/0x1e0 net/mac80211/main.c:452
tasklet_action_common+0x2d8/0x4b0 kernel/softirq.c:938
handle_softirqs+0x22a/0x830 kernel/softirq.c:622
__do_softirq kernel/softirq.c:656 [inline]
invoke_softirq kernel/softirq.c:496 [inline]
__irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
irq_exit_rcu+0x9/0x30 kernel/softirq.c:752
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1061 [inline]
sysvec_apic_timer_interrupt+0x90/0xb0 arch/x86/kernel/apic/apic.c:1061
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:csd_lock_wait kernel/smp.c:342 [inline]
RIP: 0010:smp_call_function_many_cond+0xfcb/0x13b0 kernel/smp.c:892
Code: 79 45 8b 2e 44 89 ee 83 e6 01 31 ff e8 be c6 0b 00 41 83 e5 01 49 bd 00 00 00 00 00 fc ff df 75 07 e8 69 c2 0b 00 eb 37 f3 90 <43> 0f b6 04 2c 84 c0 75 10 41 f7 06 01 00 00 00 74 1e e8 4e c2 0b
RSP: 0018:ffffc900015675a0 EFLAGS: 00000293
RAX: ffffffff81834d62 RBX: ffff88806943ad08 RCX: ffff888023900000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc900015676d0 R08: ffffffff9064faf7 R09: 1ffffffff20c9f5e
R10: dffffc0000000000 R11: fffffbfff20c9f5f R12: 1ffff1100d2a7edd
R13: dffffc0000000000 R14: ffff88806953f6e8 R15: 0000000000000001
on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1057
native_flush_tlb_multi arch/x86/mm/tlb.c:-1 [inline]
flush_tlb_multi arch/x86/mm/tlb.c:1361 [inline]
flush_tlb_mm_range+0xa65/0x12f0 arch/x86/mm/tlb.c:1451
dup_mmap+0x179b/0x1d70 mm/mmap.c:1905
dup_mm kernel/fork.c:1534 [inline]
copy_mm+0x136/0x480 kernel/fork.c:1586
copy_process+0x1e96/0x4290 kernel/fork.c:2264
kernel_clone+0x283/0x870 kernel/fork.c:2721
__do_sys_clone kernel/fork.c:2862 [inline]
__se_sys_clone kernel/fork.c:2846 [inline]
__x64_sys_clone+0x1b6/0x230 kernel/fork.c:2846
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0x10a/0x4d0 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f4301082636
Code: 89 df e8 6d e8 f6 ff 45 31 c0 31 d2 31 f6 64 48 8b 04 25 10 00 00 00 bf 11 00 20 01 4c 8d 90 d0 02 00 00 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 52 89 c5 85 c0 75 31 64 48 8b 04 25 10 00 00
RSP: 002b:00007fff6569b460 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
RAX: ffffffffffffffda RBX: 00007fff6569b468 RCX: 00007f4301082636
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
RBP: 00007fff656bb9b0 R08: 0000000000000000 R09: 0000000000000001
R10: 00007f4300faea50 R11: 0000000000000246 R12: 00007fff6569b5a0
R13: 0000000000000000 R14: 0000000000000000 R15: 000055d88a403ac0
</TASK>
]
|
| 608/2 |
2026/05/18 20:21 |
iteration |
1 |
41m
|
| 609/3 |
2026/05/18 20:21 |
agent |
patch-generator |
8m
Model:
Instruction:
You are an experienced Linux kernel developer tasked with creating a fix for a kernel bug.
You will be given a crash report, and an initial explanation of the root cause done by another
kernel expert.
Use the codeeditor tool to do code edits.
Note: you will not see your changes when looking at the code using codesearch tools.
Use the patch-diff tool to review the modifications you applied.
Your final reply should contain explanation of what you did in the patch and why
(details not present in the initial explanation of the bug).
Your fix must not just prevent the given crash, but also be the best fix for the underlying
root cause from the software engineering point of view. There can be several ways to fix the
same bug. Consider alternatives, and pick the best one. For example, additional checks may be
added at different locations/functions, it's usually better to place them earlier in the
execution to avoid multiple checks at various locations later.
Frequently the same coding mistake is done in several locations in the source code.
Check if your fix should be extended/applied to similar cases around to fix other similar bugs.
But don't go too wide, don't try to fix problems kernel-wide, fix similar issues
in the same file only.
If you are changing post-conditions of a function, consider all callers of the functions,
and if they need to be updated to handle new post-conditions. For example, if you make
a function that previously never returned a NULL, return NULL, consider if callers
need to be updated to handle NULL return value.
Prefer calling several tools at the same time to save round-trips.
Prompt:
The crash that corresponds to the bug is:
watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/u9:4:1179]
CPU#0 Utilization every 4000ms during lockup:
#1: 100% system, 1% softirq, 1% hardirq, 0% idle
#2: 100% system, 1% softirq, 0% hardirq, 0% idle
#3: 100% system, 1% softirq, 1% hardirq, 0% idle
#4: 100% system, 1% softirq, 1% hardirq, 0% idle
#5: 100% system, 1% softirq, 1% hardirq, 0% idle
Modules linked in:
irq event stamp: 386488
hardirqs last enabled at (386487): [<ffffffff8100148a>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
hardirqs last disabled at (386488): [<ffffffff8b9eb11e>] sysvec_apic_timer_interrupt+0xe/0xb0 arch/x86/kernel/apic/apic.c:1061
softirqs last enabled at (386486): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
softirqs last disabled at (386409): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
CPU: 0 UID: 0 PID: 1179 Comm: kworker/u9:4 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:342 [inline]
RIP: 0010:smp_call_function_many_cond+0xfcb/0x13b0 kernel/smp.c:892
Code: 79 45 8b 2e 44 89 ee 83 e6 01 31 ff e8 be c6 0b 00 41 83 e5 01 49 bd 00 00 00 00 00 fc ff df 75 07 e8 69 c2 0b 00 eb 37 f3 90 <43> 0f b6 04 2c 84 c0 75 10 41 f7 06 01 00 00 00 74 1e e8 4e c2 0b
RSP: 0018:ffffc90005fcf720 EFLAGS: 00000293
RAX: ffffffff81834d62 RBX: ffff88806943ad08 RCX: ffff88801d361c00
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90005fcf860 R08: ffffffff908e5af7 R09: 1ffffffff211cb5e
R10: dffffc0000000000 R11: fffffbfff211cb5f R12: 1ffff1100d2a7edd
R13: dffffc0000000000 R14: ffff88806953f6e8 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff8880d5abe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555578641490 CR3: 000000000eaf4000 CR4: 00000000000006f0
Call Trace:
<TASK>
on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1057
on_each_cpu include/linux/smp.h:72 [inline]
smp_text_poke_sync_each_cpu arch/x86/kernel/alternative.c:2773 [inline]
smp_text_poke_batch_finish+0x5e6/0x1090 arch/x86/kernel/alternative.c:2983
arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
static_key_enable_cpuslocked+0x128/0x240 kernel/jump_label.c:210
static_key_enable+0x1a/0x20 kernel/jump_label.c:223
toggle_allocation_gate+0xab/0x290 mm/kfence/core.c:906
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xad1/0x1740 kernel/workqueue.c:3397
worker_thread+0xa59/0xfc0 kernel/workqueue.c:3478
kthread+0x388/0x470 kernel/kthread.c:436
ret_from_fork+0x437/0x9d0 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:__remove_hrtimer+0x8e/0x3e0 kernel/time/hrtimer.c:1145
Code: ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d
RSP: 0018:ffffc900001b8d90 EFLAGS: 00000006
RAX: ffffffff817d161e RBX: ffff888069527f40 RCX: 0000000000010000
RDX: ffff8880160c8000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: dffffc0000000000 R08: ffffffff94f48153 R09: 1ffffffff29e902a
R10: dffffc0000000000 R11: fffffbfff29e902b R12: 0000000000000000
R13: 0000000000000001 R14: ffff888024042300 R15: ffff888069528000
FS: 0000000000000000(0000) GS:ffff8880d5bbe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f83db077470 CR3: 000000001f132000 CR4: 00000000000006f0
Call Trace:
<IRQ>
__run_hrtimer kernel/time/hrtimer.c:1910 [inline]
__hrtimer_run_queues+0x299/0xa10 kernel/time/hrtimer.c:1994
hrtimer_interrupt+0x44b/0x950 kernel/time/hrtimer.c:2113
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1050 [inline]
__sysvec_apic_timer_interrupt+0x102/0x420 arch/x86/kernel/apic/apic.c:1067
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1061 [inline]
sysvec_apic_timer_interrupt+0x8b/0xb0 arch/x86/kernel/apic/apic.c:1061
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:native_irq_disable arch/x86/include/asm/irqflags.h:37 [inline]
RIP: 0010:arch_local_irq_disable arch/x86/include/asm/irqflags.h:114 [inline]
RIP: 0010:default_idle+0xf/0x20 arch/x86/kernel/process.c:768
Code: 4c 01 c7 4c 29 c2 e9 6e ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 13 a4 23 00 fb f4 <fa> e9 8b 8d 02 00 cc 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
RSP: 0018:ffffc9000014fe28 EFLAGS: 00000202
RAX: 00000000006a8f01 RBX: ffffffff81654d3a RCX: 0000000080000001
RDX: 0000000000000001 RSI: ffffffff8e3ef513 RDI: ffffffff8c21df20
RBP: ffffc9000014ff20 R08: ffff888069532993 R09: 1ffff1100d2a6532
R10: dffffc0000000000 R11: ffffed100d2a6533 R12: 0000000000000001
R13: 1ffff11002c19000 R14: 0000000000000001 R15: 1ffff11002c19000
default_idle_call+0x70/0xb0 kernel/sched/idle.c:122
cpuidle_idle_call kernel/sched/idle.c:199 [inline]
do_idle+0x36a/0x5e0 kernel/sched/idle.c:352
cpu_startup_entry+0x43/0x60 kernel/sched/idle.c:451
start_secondary+0x101/0x110 arch/x86/kernel/smpboot.c:312
common_startup_64+0x12c/0x137
</TASK>
The explanation of the root cause of the bug is:
The root cause of this crash is a massive out-of-bounds memory write (buffer overflow) in the BPF helper `bpf_snprintf` (and other `bprintf`-like helpers) due to integer signedness and unchecked return values in `bpf_bprintf_prepare()`.
This memory corruption overwrites adjacent kernel memory, which in this case included `hrtimer` structures and their `timerqueue` rbtree nodes. When CPU 1 later attempted to remove a corrupted timer via `__remove_hrtimer` -> `timerqueue_linked_del` -> `rb_erase_linked`, it got stuck in an infinite loop traversing a cyclic rbtree. CPU 0 then detected a soft lockup because it was waiting for CPU 1 to acknowledge an IPI (`csd_lock_wait`), but CPU 1 had interrupts disabled while processing the timer queue.
### Detailed Analysis
In `kernel/bpf/helpers.c`, the `bpf_bprintf_prepare()` function formats arguments into a temporary 512-byte buffer (`buffers->bin_args`). There are two distinct bugs in this function that allow `tmp_buf` to advance past `tmp_buf_end`, bypassing bounds checks.
#### Bug 1: Unchecked `snprintf` return value for `%pB`
When handling the `%pB` format specifier, the code does:
```c
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
```
`snprintf` returns the number of characters that *would* have been written if the buffer was large enough, excluding the null terminator. If the remaining buffer space (`tmp_buf_end - tmp_buf`) is smaller than the formatted string, `err` will be greater than or equal to the remaining space. Consequently, `tmp_buf += (err + 1)` will advance `tmp_buf` **past** `tmp_buf_end`.
#### Bug 2: Signedness bug in bounds checking
When handling standard format specifiers (like `%d`, `%x`), the code aligns `tmp_buf` and checks if there is enough space:
```c
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
```
Here, `tmp_buf_end - tmp_buf` is of type `ptrdiff_t` (signed), and `sizeof_cur_arg` is of type `size_t` (unsigned). In C, this comparison forces the signed `ptrdiff_t` to be cast to an unsigned `size_t`.
If `tmp_buf` has already been advanced past `tmp_buf_end` (e.g., via the `%pB` bug above, or by using `%c` to make `tmp_buf` unaligned such that `PTR_ALIGN` pushes it past `tmp_buf_end`), `tmp_buf_end - tmp_buf` becomes negative. When cast to `size_t`, this negative value becomes a huge positive number (e.g., `0xFFFFFFFFFFFFFFF0`).
The condition `HUGE_NUMBER < sizeof_cur_arg` evaluates to `false`, bypassing the `-ENOSPC` check. The code then writes to `tmp_buf`, corrupting memory out of bounds.
Furthermore, if a `%s` specifier is used after `tmp_buf` is out of bounds:
```c
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
```
The equality check `tmp_buf_end == tmp_buf` fails to catch that `tmp_buf > tmp_buf_end`. Then `bpf_trace_copy_string` is called with a negative `bufsz` (`tmp_buf_end - tmp_buf`), which is cast to a huge `size_t`. This results in copying an arbitrarily large string into kernel memory, causing massive corruption.
### Fix
To fix this vulnerability:
1. For `%pB`, `tmp_buf` should only be advanced by the actual number of bytes written, and it must not exceed `tmp_buf_end`.
2. The bounds checks must correctly handle the case where `tmp_buf >= tmp_buf_end`. A safe way is to cast `tmp_buf_end - tmp_buf` to a signed type and check if it's less than the required size, or simply check `if (tmp_buf + sizeof_cur_arg > tmp_buf_end)`.
Another developer tried to fix this bug, and come up with the following strategy for fixing:
I have analyzed the crash report and the root cause explanation, and implemented a comprehensive fix for the vulnerabilities in `bpf_bprintf_prepare()`.
### What I did in the patch:
1. **Fixed the `snprintf` return value handling for `%pB` and `%pI`:**
`snprintf` returns the number of characters that *would* have been written if the buffer was large enough. Blindly advancing `tmp_buf` by `err + 1` causes the pointer to exceed `tmp_buf_end` when truncation occurs. I introduced a `limit` variable to represent the remaining buffer space and used `min_t(size_t, err + 1, limit)` to advance `tmp_buf`. This guarantees that `tmp_buf` never exceeds `tmp_buf_end` under any circumstances.
2. **Fixed the integer signedness bug in bounds checking:**
The original bounds checks used `tmp_buf_end - tmp_buf < sizeof_cur_arg`. Because `sizeof_cur_arg` is an unsigned `size_t`, the signed `ptrdiff_t` result of the pointer subtraction was implicitly cast to unsigned. If `tmp_buf` ever exceeded `tmp_buf_end`, the negative difference became a huge positive number, bypassing the check. I rewrote these checks as `tmp_buf + sizeof_cur_arg > tmp_buf_end`. This uses safe pointer arithmetic and completely eliminates the signedness issue.
3. **Hardened the equality checks for `%s` and `%c`:**
The original code used `if (tmp_buf_end == tmp_buf)` to check for a full buffer. While the fixes above prevent `tmp_buf` from exceeding `tmp_buf_end`, it is a better software engineering practice to use `>=` for bounds checking. I updated these checks to `if (tmp_buf >= tmp_buf_end)` to make the code robust against any future modifications that might inadvertently advance `tmp_buf` out of bounds.
4. **Applied the fix consistently:**
The initial explanation only mentioned the `%pB` bug, but a careful review revealed that the exact same `snprintf` bug existed for IP address formatting (`%pI4`, `%pI6`, etc.). I applied the same `limit` and `min_t` fix to the IP address formatting block to ensure all similar vulnerabilities in the file are addressed.
### Why this is the best fix:
Instead of just patching the specific `%pB` case, this fix addresses the underlying architectural fragility. By changing the bounds checks to use pointer addition (`tmp_buf + size > tmp_buf_end`) and hardening the equality checks (`>=`), the function is now structurally immune to signedness bypasses. Furthermore, by strictly capping the pointer advancement using `min_t`, we ensure that `tmp_buf` remains a valid pointer within the allocated buffer at all times, preventing any cascading failures in subsequent loop iterations.
and the following patch:
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2bb60200c..be70abbc4 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -921,11 +921,19 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
- err = snprintf(tmp_buf,
- (tmp_buf_end - tmp_buf),
- "%pB",
- (void *)(long)raw_args[num_spec]);
- tmp_buf += (err + 1);
+ size_t limit =
+ tmp_buf < tmp_buf_end ?
+ tmp_buf_end - tmp_buf :
+ 0;
+
+ err = snprintf(
+ tmp_buf, limit, "%pB",
+ (void *)(long)
+ raw_args[num_spec]);
+ if (err < 0)
+ err = 0;
+ tmp_buf +=
+ min_t(size_t, err + 1, limit);
}
i++;
@@ -945,7 +953,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
- if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
+ if (tmp_buf + sizeof_cur_ip > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -962,10 +970,17 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
- err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
- ip_spec, &cur_ip);
-
- tmp_buf += err + 1;
+ {
+ size_t limit = tmp_buf < tmp_buf_end ?
+ tmp_buf_end - tmp_buf :
+ 0;
+
+ err = snprintf(tmp_buf, limit, ip_spec,
+ &cur_ip);
+ if (err < 0)
+ err = 0;
+ tmp_buf += min_t(size_t, err + 1, limit);
+ }
num_spec++;
continue;
@@ -982,7 +997,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1004,7 +1019,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1038,7 +1053,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
- if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
However, the patch testing failed with the following error:
watchdog: BUG: soft lockup - CPU#1 stuck for 46s! [swapper/1:0]
CPU#1 Utilization every 4000ms during lockup:
#1: 26% system, 3% softirq, 3% hardirq, 70% idle
#2: 48% system, 5% softirq, 3% hardirq, 46% idle
#3: 54% system, 4% softirq, 3% hardirq, 38% idle
#4: 0% system, 0% softirq, 79% hardirq, 1% idle
#5: 0% system, 0% softirq, 0% hardirq, 1% idle
Modules linked in:
irq event stamp: 4886784
hardirqs last enabled at (4886783): [<ffffffff81654d3a>] cpuidle_idle_call kernel/sched/idle.c:199 [inline]
hardirqs last enabled at (4886783): [<ffffffff81654d3a>] do_idle+0x36a/0x5e0 kernel/sched/idle.c:352
hardirqs last disabled at (4886784): [<ffffffff8b9eb11e>] sysvec_apic_timer_interrupt+0xe/0xb0 arch/x86/kernel/apic/apic.c:1061
softirqs last enabled at (4886760): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last enabled at (4886760): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last enabled at (4886760): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
softirqs last disabled at (4886221): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last disabled at (4886221): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last disabled at (4886221): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:native_irq_disable arch/x86/include/asm/irqflags.h:37 [inline]
RIP: 0010:arch_local_irq_disable arch/x86/include/asm/irqflags.h:114 [inline]
RIP: 0010:default_idle+0xf/0x20 arch/x86/kernel/process.c:768
Code: 4c 01 c7 4c 29 c2 e9 6e ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 13 7e 23 00 fb f4 <fa> e9 8b 8d 02 00 cc 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
RSP: 0018:ffffc9000014fe28 EFLAGS: 00000206
RAX: 00000000004a90ff RBX: ffffffff81654d3a RCX: 0000000080000001
RDX: 0000000000000001 RSI: ffffffff8e2b9503 RDI: ffffffff8c1c60a0
RBP: ffffc9000014ff20 R08: ffff888069532993 R09: 1ffff1100d2a6532
R10: dffffc0000000000 R11: ffffed100d2a6533 R12: 0000000000000001
R13: 1ffff11002c19000 R14: 0000000000000001 R15: 1ffff11002c19000
FS: 0000000000000000(0000) GS:ffff8880d5e54000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe1f92f1ef0 CR3: 000000002225a000 CR4: 00000000000006f0
Call Trace:
<TASK>
default_idle_call+0x70/0xb0 kernel/sched/idle.c:122
cpuidle_idle_call kernel/sched/idle.c:199 [inline]
do_idle+0x36a/0x5e0 kernel/sched/idle.c:352
cpu_startup_entry+0x43/0x60 kernel/sched/idle.c:451
start_secondary+0x101/0x110 arch/x86/kernel/smpboot.c:312
common_startup_64+0x12c/0x137
</TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 4676 Comm: dhcpcd Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:lookup_chain_cache kernel/locking/lockdep.c:3802 [inline]
RIP: 0010:lookup_chain_cache_add kernel/locking/lockdep.c:3821 [inline]
RIP: 0010:validate_chain kernel/locking/lockdep.c:3876 [inline]
RIP: 0010:__lock_acquire+0xc4b/0x2d80 kernel/locking/lockdep.c:5237
Code: 48 ff cb f7 c1 00 60 00 00 74 c4 e9 90 00 00 00 48 c7 c7 d0 88 99 8e 4c 89 fe e8 b0 9c 24 03 48 8b 54 24 18 eb c4 48 8b 48 18 <48> 39 e9 0f 84 b8 00 00 00 48 8b 40 08 48 85 c0 0f 84 cc fe ff ff
RSP: 0018:ffffc900000078d0 EFLAGS: 00000046
RAX: ffffffff93a979b0 RBX: 0000000000003f6d RCX: dbc3fa7544b72e03
RDX: 000000009b99b500 RSI: 00000000d40ffd9b RDI: ffff888023900000
RBP: dbc3fa7544b72e03 R08: ffff888069428c60 R09: ffffffff818f225b
R10: dffffc0000000000 R11: fffffbfff20c9f5f R12: 0000000000000001
R13: ffff888023900ba0 R14: ffff888023900000 R15: 0000000000000002
FS: 00007f4300fae780(0000) GS:ffff8880d5d54000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555576e3c490 CR3: 0000000019bd4000 CR4: 00000000000006f0
Call Trace:
<IRQ>
lock_acquire+0x106/0x350 kernel/locking/lockdep.c:5868
local_lock_acquire include/linux/local_lock_internal.h:46 [inline]
kcov_remote_stop+0x84/0x6c0 kernel/kcov.c:1030
ieee80211_rx_list+0x2b2a/0x3690 net/mac80211/rx.c:5641
ieee80211_rx_napi+0x1b1/0x3e0 net/mac80211/rx.c:5661
ieee80211_rx include/net/mac80211.h:5395 [inline]
ieee80211_handle_queued_frames+0xe8/0x1e0 net/mac80211/main.c:452
tasklet_action_common+0x2d8/0x4b0 kernel/softirq.c:938
handle_softirqs+0x22a/0x830 kernel/softirq.c:622
__do_softirq kernel/softirq.c:656 [inline]
invoke_softirq kernel/softirq.c:496 [inline]
__irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
irq_exit_rcu+0x9/0x30 kernel/softirq.c:752
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1061 [inline]
sysvec_apic_timer_interrupt+0x90/0xb0 arch/x86/kernel/apic/apic.c:1061
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:csd_lock_wait kernel/smp.c:342 [inline]
RIP: 0010:smp_call_function_many_cond+0xfcb/0x13b0 kernel/smp.c:892
Code: 79 45 8b 2e 44 89 ee 83 e6 01 31 ff e8 be c6 0b 00 41 83 e5 01 49 bd 00 00 00 00 00 fc ff df 75 07 e8 69 c2 0b 00 eb 37 f3 90 <43> 0f b6 04 2c 84 c0 75 10 41 f7 06 01 00 00 00 74 1e e8 4e c2 0b
RSP: 0018:ffffc900015675a0 EFLAGS: 00000293
RAX: ffffffff81834d62 RBX: ffff88806943ad08 RCX: ffff888023900000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc900015676d0 R08: ffffffff9064faf7 R09: 1ffffffff20c9f5e
R10: dffffc0000000000 R11: fffffbfff20c9f5f R12: 1ffff1100d2a7edd
R13: dffffc0000000000 R14: ffff88806953f6e8 R15: 0000000000000001
on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1057
native_flush_tlb_multi arch/x86/mm/tlb.c:-1 [inline]
flush_tlb_multi arch/x86/mm/tlb.c:1361 [inline]
flush_tlb_mm_range+0xa65/0x12f0 arch/x86/mm/tlb.c:1451
dup_mmap+0x179b/0x1d70 mm/mmap.c:1905
dup_mm kernel/fork.c:1534 [inline]
copy_mm+0x136/0x480 kernel/fork.c:1586
copy_process+0x1e96/0x4290 kernel/fork.c:2264
kernel_clone+0x283/0x870 kernel/fork.c:2721
__do_sys_clone kernel/fork.c:2862 [inline]
__se_sys_clone kernel/fork.c:2846 [inline]
__x64_sys_clone+0x1b6/0x230 kernel/fork.c:2846
do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
do_syscall_64+0x10a/0x4d0 arch/x86/entry/syscall_64.c:94
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f4301082636
Code: 89 df e8 6d e8 f6 ff 45 31 c0 31 d2 31 f6 64 48 8b 04 25 10 00 00 00 bf 11 00 20 01 4c 8d 90 d0 02 00 00 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 52 89 c5 85 c0 75 31 64 48 8b 04 25 10 00 00
RSP: 002b:00007fff6569b460 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
RAX: ffffffffffffffda RBX: 00007fff6569b468 RCX: 00007f4301082636
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
RBP: 00007fff656bb9b0 R08: 0000000000000000 R09: 0000000000000001
R10: 00007f4300faea50 R11: 0000000000000246 R12: 00007fff6569b5a0
R13: 0000000000000000 R14: 0000000000000000 R15: 000055d88a403ac0
</TASK>
If the error is fixable, and the fix patch is correct overall,
the create a new fixed patch based on the provided one with the errors fixed.
If the error points to a fundamental issue with the approach in the patch,
then create a new patch from scratch.
Note: in both cases the source tree does not contain the patch yet
(so if you want to create a new fixed patch, you need to recreate it
in its entirety from scratch using the codeeditor tool).
Reply: I have analyzed the crash report, the root cause explanation, and the failed patch.
### What was wrong with the previous patch:
The previous patch attempted to fix the out-of-bounds write by truncating the string using `min_t(size_t, err + 1, limit)`. However, it did not return `-ENOSPC` when the internal binary arguments buffer (`bin_args`) was full. This caused `bpf_bprintf_prepare()` to return success (`0`) even when the string was truncated or completely omitted (if `limit == 0`).
Because `bstr_printf()` expects a fully formed, null-terminated string in the binary buffer for `%pB` and `%pI4`/`%pI6`, returning success on a truncated/missing string caused `bstr_printf()` to read out of bounds. If `limit == 0`, `snprintf()` wrote nothing, leaving `tmp_buf` at `tmp_buf_end` without a null terminator. `bstr_printf()` then called `strlen()` on uninitialized memory past the end of the buffer, leading to the soft lockup observed in the failed test.
### What I did in the new patch:
1. **Properly handle `snprintf` truncation for `%pB` and `%pI4`/`%pI6`:**
Instead of silently truncating the string, the code now checks if `snprintf()` returns an error (`err < 0`) or if it indicates truncation (`err >= tmp_buf_end - tmp_buf`). If truncation occurs, it means the internal 512-byte `bin_args` buffer is full, and we correctly return `-ENOSPC`. This safely aborts the formatting process and propagates the error to the BPF program, preventing `bstr_printf()` from reading malformed data.
2. **Fixed the flawed bounds check for IP addresses:**
The original code had a check `if (tmp_buf_end - tmp_buf < sizeof_cur_ip)` before formatting IP addresses. `sizeof_cur_ip` is the size of the *binary* IP address (4 or 16 bytes), but `snprintf()` writes the *formatted string* (up to 40 bytes for IPv6). This allowed `snprintf()` to write past `tmp_buf_end`, causing the exact same buffer overflow as `%pB`. I removed this incorrect check and replaced it with the robust `snprintf()` return value check described above.
3. **Fixed the integer signedness bug in bounds checking:**
The original bounds checks used `tmp_buf_end - tmp_buf < sizeof_cur_arg`. Because `sizeof_cur_arg` is an unsigned `size_t`, the signed `ptrdiff_t` result of the pointer subtraction was implicitly cast to unsigned. If `tmp_buf` ever exceeded `tmp_buf_end`, the negative difference became a huge positive number, bypassing the check. I rewrote these checks as `tmp_buf + sizeof_cur_arg > tmp_buf_end`. This uses safe pointer arithmetic and completely eliminates the signedness issue.
4. **Hardened equality checks:**
I updated the buffer full checks for `%s` and `%c` from `if (tmp_buf_end == tmp_buf)` to `if (tmp_buf >= tmp_buf_end)` to make the code robust against any future modifications that might inadvertently advance `tmp_buf` out of bounds.
### Why this is the best fix:
This fix addresses the root cause of the buffer overflows without breaking the contract between `bpf_bprintf_prepare()` and `bstr_printf()`. By returning `-ENOSPC` when the internal buffer is full, we ensure that `bstr_printf()` only ever operates on perfectly formed, null-terminated strings, completely eliminating the risk of out-of-bounds reads or soft lockups. Furthermore, rewriting the bounds checks to use pointer addition structurally immunizes the function against signedness bypasses.
|
| 610/4 |
2026/05/18 20:21 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 15441
output: 44
thoughts: 447
Thoughts: **Analyzing a Soft Lockup Crash Rooted in `bpf_bprintf_prepare()`**
Okay, so we've got a soft lockup crash, and it looks like it's stemming from an out-of-bounds memory write within the `bpf_bprintf_prepare()` function. The provided patch attempts to address this. My initial thought is to really dig into what's being done in this function.
Let's dissect the patch. It seems to be using `min_t(size_t, err + 1, limit)` to advance `tmp_buf`. Wait a second… `err` is the return from `snprintf`! If `err` is the number of characters that *would have* been written (or that *were* written), then `err + 1` *includes* space for the null terminator. That's good, if that's what we want. The patch also handles the case where `err < 0` by setting `err = 0`. Right, that's expected.
Now, let's trace this logic further. Okay, so `tmp_buf += min_t(size_t, err + 1, limit);`... hmm... If `limit` is zero, we're advancing `tmp_buf` by zero, which is likely intended behavior.
But, hang on... let's re-think this... if `err` is the number of characters *written* and `limit` represents the *remaining space*, then `snprintf` will write at *most* `limit` bytes, including the null terminator. This is important to remember. If `limit` is zero, `snprintf` should return the number of characters *it would have* written, but it won't write anything.
Okay, let's step through this one more time: We're advancing `tmp_buf` by `min_t(size_t, err + 1, limit)`. If `err + 1` (the potential space including the null terminator) *is greater* than `limit`, then the advancement is limited to `limit`. Is *that* the correct behavior we want? That's what we need to figure out.
Let's go back to the original code in `bpf_bprintf_prepare`:
```c
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
```
Okay, the `'B'` specifier is what we're focused on, and there are other specifiers. We need to check those. Let's look at the `bpf_bprintf_prepare` function, and how it uses `kernel/bpf/helpers.c`.
|
| 611/4 |
2026/05/18 20:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_prepare]
Results: map[SourceCode: 804: /*
805: * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
806: *
807: * Returns a negative value if fmt is an invalid format string or 0 otherwise.
808: *
809: * This can be used in two ways:
810: * - Format string verification only: when data->get_bin_args is false
811: * - Arguments preparation: in addition to the above verification, it writes in
812: * data->bin_args a binary representation of arguments usable by bstr_printf
813: * where pointers from BPF have been sanitized.
814: *
815: * In argument preparation mode, if 0 is returned, safe temporary buffers are
816: * allocated and bpf_bprintf_cleanup should be called to free them after use.
817: */
818: int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
819: u32 num_args, struct bpf_bprintf_data *data)
820: {
821: bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
822: char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
823: struct bpf_bprintf_buffers *buffers = NULL;
824: size_t sizeof_cur_arg, sizeof_cur_ip;
825: int err, i, num_spec = 0;
826: u64 cur_arg;
827: char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
828:
829: fmt_end = strnchr(fmt, fmt_size, 0);
830: if (!fmt_end)
831: return -EINVAL;
832: fmt_size = fmt_end - fmt;
833:
834: if (get_buffers && bpf_try_get_buffers(&buffers))
835: return -EBUSY;
836:
837: if (data->get_bin_args) {
838: if (num_args)
839: tmp_buf = buffers->bin_args;
840: tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
841: data->bin_args = (u32 *)tmp_buf;
842: }
843:
844: if (data->get_buf)
845: data->buf = buffers->buf;
846:
847: for (i = 0; i < fmt_size; i++) {
848: unsigned char c = fmt[i];
849:
850: /*
851: * Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
852: * through unchanged, while still rejecting ASCII control bytes.
853: */
854: if (isascii(c) && !isprint(c) && !isspace(c)) {
855: err = -EINVAL;
856: goto out;
857: }
858:
859: if (fmt[i] != '%')
860: continue;
861:
862: if (fmt[i + 1] == '%') {
863: i++;
864: continue;
865: }
866:
867: if (num_spec >= num_args) {
868: err = -EINVAL;
869: goto out;
870: }
871:
872: /* The string is zero-terminated so if fmt[i] != 0, we can
873: * always access fmt[i + 1], in the worst case it will be a 0
874: */
875: i++;
876: c = fmt[i];
877: /*
878: * The format parser below only understands ASCII conversion
879: * specifiers and modifiers, so reject non-ASCII after '%'.
880: */
881: if (!isascii(c)) {
882: err = -EINVAL;
883: goto out;
884: }
885:
886: /* skip optional "[0 +-][num]" width formatting field */
887: while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
888: fmt[i] == ' ')
889: i++;
890: if (fmt[i] >= '1' && fmt[i] <= '9') {
891: i++;
892: while (fmt[i] >= '0' && fmt[i] <= '9')
893: i++;
894: }
895:
896: if (fmt[i] == 'p') {
897: sizeof_cur_arg = sizeof(long);
898:
899: if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
900: ispunct(fmt[i + 1])) {
901: if (tmp_buf)
902: cur_arg = raw_args[num_spec];
903: goto nocopy_fmt;
904: }
905:
906: if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
907: fmt[i + 2] == 's') {
908: fmt_ptype = fmt[i + 1];
909: i += 2;
910: goto fmt_str;
911: }
912:
913: if (fmt[i + 1] == 'K' ||
914: fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
915: fmt[i + 1] == 'S') {
916: if (tmp_buf)
917: cur_arg = raw_args[num_spec];
918: i++;
919: goto nocopy_fmt;
920: }
921:
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
930:
931: i++;
932: num_spec++;
933: continue;
934: }
935:
936: /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
937: if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
938: (fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
939: err = -EINVAL;
940: goto out;
941: }
942:
943: i += 2;
944: if (!tmp_buf)
945: goto nocopy_fmt;
946:
947: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
948: if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
949: err = -ENOSPC;
950: goto out;
951: }
952:
953: unsafe_ptr = (char *)(long)raw_args[num_spec];
954: err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
955: sizeof_cur_ip);
956: if (err < 0)
957: memset(cur_ip, 0, sizeof_cur_ip);
958:
959: /* hack: bstr_printf expects IP addresses to be
960: * pre-formatted as strings, ironically, the easiest way
961: * to do that is to call snprintf.
962: */
963: ip_spec[2] = fmt[i - 1];
964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
969: num_spec++;
970:
971: continue;
972: } else if (fmt[i] == 's') {
973: fmt_ptype = fmt[i];
974: fmt_str:
975: if (fmt[i + 1] != 0 &&
976: !isspace(fmt[i + 1]) &&
977: !ispunct(fmt[i + 1])) {
978: err = -EINVAL;
979: goto out;
980: }
981:
982: if (!tmp_buf)
983: goto nocopy_fmt;
984:
985: if (tmp_buf_end == tmp_buf) {
986: err = -ENOSPC;
987: goto out;
988: }
989:
990: unsafe_ptr = (char *)(long)raw_args[num_spec];
991: err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
992: fmt_ptype,
993: tmp_buf_end - tmp_buf);
994: if (err < 0) {
995: tmp_buf[0] = '\0';
996: err = 1;
997: }
998:
999: tmp_buf += err;
1000: num_spec++;
1001:
1002: continue;
1003: } else if (fmt[i] == 'c') {
1004: if (!tmp_buf)
1005: goto nocopy_fmt;
1006:
1007: if (tmp_buf_end == tmp_buf) {
1008: err = -ENOSPC;
1009: goto out;
1010: }
1011:
1012: *tmp_buf = raw_args[num_spec];
1013: tmp_buf++;
1014: num_spec++;
1015:
1016: continue;
1017: }
1018:
1019: sizeof_cur_arg = sizeof(int);
1020:
1021: if (fmt[i] == 'l') {
1022: sizeof_cur_arg = sizeof(long);
1023: i++;
1024: }
1025: if (fmt[i] == 'l') {
1026: sizeof_cur_arg = sizeof(long long);
1027: i++;
1028: }
1029:
1030: if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
1031: fmt[i] != 'x' && fmt[i] != 'X') {
1032: err = -EINVAL;
1033: goto out;
1034: }
1035:
1036: if (tmp_buf)
1037: cur_arg = raw_args[num_spec];
1038: nocopy_fmt:
1039: if (tmp_buf) {
1040: tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
1041: if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
1042: err = -ENOSPC;
1043: goto out;
1044: }
1045:
1046: if (sizeof_cur_arg == 8) {
1047: *(u32 *)tmp_buf = *(u32 *)&cur_arg;
1048: *(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
1049: } else {
1050: *(u32 *)tmp_buf = (u32)(long)cur_arg;
1051: }
1052: tmp_buf += sizeof_cur_arg;
1053: }
1054: num_spec++;
1055: }
1056:
1057: err = 0;
1058: out:
1059: if (err)
1060: bpf_bprintf_cleanup(data);
1061: return err;
1062: }
SourceFile:kernel/bpf/helpers.c]
|
| 612/4 |
2026/05/18 20:22 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 20438
output: 52
thoughts: 437
Thoughts: **Debugging a Buffer Overflow in bprintf**
Okay, so the patch failed, and I think I see why. The line `tmp_buf += min_t(size_t, err + 1, limit);` is probably the culprit. Let me think this through... If `limit` represents the remaining buffer space and `err` is the number of characters *that would* have been written, then `err + 1` is indeed the *total* space needed, including the null terminator. The crux of the problem seems to be when `err + 1 > limit`. That signifies `snprintf` truncated the output, writing `limit` bytes.
Wait, if `limit` is zero, `snprintf` doesn't write anything. Consequently, `tmp_buf` would advance by zero. However, if `limit` is zero, then `snprintf` *is* called with a size of zero! It *returns* the number of characters that would have been written. The original logic does indeed advance the buffer.
Now, going back to `err + 1 > limit`, it looks like `tmp_buf` is advanced by `limit`. Okay, that makes sense. But... if `tmp_buf` is advanced by `limit`, that means `tmp_buf` has now reached `tmp_buf_end`.
Hold on. In the *next* iteration of the loop, when we come across another format specifier, `tmp_buf` will be at `tmp_buf_end`. And then, *bam*, `tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));`. If `tmp_buf` is at `tmp_buf_end` and `tmp_buf_end` isn't aligned to `sizeof(u32)`, `PTR_ALIGN` will *advance* `tmp_buf` *past* the end of allocated memory. Bingo! That's the overflow!
Ah, I see it now. `tmp_buf_end` is `tmp_buf + MAX_BPRINTF_BIN_ARGS`. `MAX_BPRINTF_BIN_ARGS` is 512. Okay, so `tmp_buf` is the buffer `buffers->bin_args`. Is that an array of `u8` or `u32`? I had better check the `struct bpf_bprintf_buffers` definition to make sure of that.
|
| 613/4 |
2026/05/18 20:22 |
tool |
codesearch-find-references |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeSnippetLines:0 Name:bpf_bprintf_buffers SourceTreePrefix:]
Results: map[References:[map[ReferenceKind:uses ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:823] map[ReferenceKind:uses ReferencingEntityKind:function ReferencingEntityName:bpf_stream_elem_alloc SourceFile:kernel/bpf/stream.c SourceLine:20] map[ReferenceKind:uses ReferencingEntityKind:function ReferencingEntityName:bpf_stream_elem_alloc SourceFile:kernel/bpf/stream.c SourceLine:20] map[ReferenceKind:uses ReferencingEntityKind:function ReferencingEntityName:bpf_stream_stage_printk SourceFile:kernel/bpf/stream.c SourceLine:312] map[ReferenceKind:uses ReferencingEntityKind:function ReferencingEntityName:bpf_try_get_buffers SourceFile:kernel/bpf/helpers.c SourceLine:773]] TruncatedOutput:false]
|
| 614/4 |
2026/05/18 20:22 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 21726
output: 45
thoughts: 15
Thoughts: Okay, let's dive into this `bpf_try_get_buffers` function. I need to understand this better. It's obviously critical to buffer management within the eBPF framework, and I need a clear mental model of how it's implemented and where it fits in with the broader architecture. I want to consider things like:
* **How does it handle resource contention?** Is this a spinlock scenario? Atomic operations? How does it avoid race conditions when multiple eBPF programs or threads might be vying for available buffer space? Is this the right level of concurrency control for the performance expectations?
* **What's the allocation strategy?** Does it use pre-allocated pools, on-demand allocation with potentially costly `kmalloc` calls, or something more sophisticated? What are the implications for fragmentation and memory overhead?
* **What are the failure modes, and how are they handled?** If buffers are unavailable, does it return an error code, block, or drop data? How does that impact the overall system behavior and what error-handling mechanisms are in place? Does it implement backoff strategies?
* **Does it integrate with other memory management subsystems, like the page allocator or slab allocator?** If so, how do these components interact? What are the implications of those interactions for the overall performance and memory usage?
* **And of course, the big question**: How does the buffer retrieval strategy affect performance in real-world scenarios? What kinds of workloads does it scale well (or poorly) under? Does it handle bursty traffic? Does this influence any performance characteristics?
Basically, I want to dissect this function and understand its implications. This isn't just about reading the code; it's about building a mental picture of its role in the whole system's performance and stability. I need to know how it behaves and influences every other component.
|
| 615/4 |
2026/05/18 20:22 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_try_get_buffers]
Results: map[SourceCode: 773: int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
774: {
775: int nest_level;
776:
777: preempt_disable();
778: nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
779: if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
780: this_cpu_dec(bpf_bprintf_nest_level);
781: preempt_enable();
782: return -EBUSY;
783: }
784: *bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
785:
786: return 0;
787: }
SourceFile:kernel/bpf/helpers.c]
|
| 616/4 |
2026/05/18 20:22 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 22453
output: 44
thoughts: 14
Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience:
**Analyzing `bpf_bprintf_bufs`**
Alright, let's dive into `bpf_bprintf_bufs`. The name itself suggests a function related to formatted output within the BPF (Berkeley Packet Filter) environment. My initial thought is to quickly consider the potential performance implications. Given BPF's emphasis on efficiency, I immediately wonder about the overhead of string formatting within the kernel context. Is this implemented using a highly optimized internal buffer system to minimize copying and reduce the potential for context switches? Or is it more of a wrapper around existing `printf`-like functionality, perhaps with inherent penalties?
I need to quickly assess how it handles different data types; does it support the standard integers, strings, and potentially even structures? I am interested in looking at the alignment of these data types, too. Does it include any format string security mitigations, like checks to prevent format string vulnerabilities that are particularly nasty in kernel code? That's a critical consideration in BPF, given its privileged execution environment.
My next step is to consider the buffer management. Are we dealing with pre-allocated buffers, or is there dynamic allocation involved? Dynamic allocation is generally discouraged in BPF to avoid fragmentation and potential latency spikes. If it's pre-allocated, what's the typical buffer size and how is that configured? I wonder if there are mechanisms to detect buffer overflows or out-of-bounds writes during formatting. Protection against overflows is going to be especially important because that is easy to exploit.
Finally, I want to investigate the integration with existing BPF helpers. Does it rely on existing helper functions for memory management, or does it have its own internal set? How does it interact with the BPF verifier? Are there any limitations imposed by the verifier regarding the complexity or types of formatting operations allowed? This could be a significant bottleneck. I want to see how this function is integrated with other parts of the BPF ecosystem. Time to get my hands dirty and look at the implementation!
|
| 617/4 |
2026/05/18 20:22 |
tool |
codesearch-definition-source |
0m
Error:
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_bufs]
Results: map[SourceCode: SourceFile:]
|
| 618/4 |
2026/05/18 20:22 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 22962
output: 53
thoughts: 15
Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience:
**Hunting Down `bpf_bprintf_buffers`**
Okay, so I need to understand `bpf_bprintf_buffers`. My immediate thought is to track down its definition. Since this likely involves BPF, I'm probably looking at kernel headers or maybe some BPF helper documentation. I'll need the source to fully grasp what's going on. Knowing the definition is key; from there I can analyze how it’s being used and any potential implications. Let's get searching...
|
| 619/4 |
2026/05/18 20:22 |
tool |
codesearch-find-references |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeSnippetLines:0 Name:struct bpf_bprintf_buffers SourceTreePrefix:]
Results: map[References:[map[ReferenceKind:uses ReferencingEntityKind:function ReferencingEntityName:bpf_bprintf_prepare SourceFile:kernel/bpf/helpers.c SourceLine:823] map[ReferenceKind:uses ReferencingEntityKind:function ReferencingEntityName:bpf_stream_elem_alloc SourceFile:kernel/bpf/stream.c SourceLine:20] map[ReferenceKind:uses ReferencingEntityKind:function ReferencingEntityName:bpf_stream_elem_alloc SourceFile:kernel/bpf/stream.c SourceLine:20] map[ReferenceKind:uses ReferencingEntityKind:function ReferencingEntityName:bpf_stream_stage_printk SourceFile:kernel/bpf/stream.c SourceLine:312] map[ReferenceKind:uses ReferencingEntityKind:function ReferencingEntityName:bpf_try_get_buffers SourceFile:kernel/bpf/helpers.c SourceLine:773]] TruncatedOutput:false]
|
| 620/4 |
2026/05/18 20:22 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 23437
output: 23
thoughts: 18
Thoughts: Here's my thought process, summarized as you requested:
**Finding That Structure Definition**
Okay, I need to find the definition of `struct bpf_bprintf_buffers`. It seems like I'm trying to locate it within the code. Instead of manually sifting through the source, why am I bothering with this slow method? *grep*! That's the way to go, of course. I can use `grep` with the exact string `struct bpf_bprintf_buffers {` to pinpoint its declaration. This should be significantly faster and more accurate than any manual search. I'm assuming it's in a header somewhere, so this should work without too many false positives. Much more efficient. Let's see what `grep` turns up!
|
| 621/4 |
2026/05/18 20:22 |
tool |
grepper |
0m
Args: map[Expression:struct bpf_bprintf_buffers \{]
Results: map[Output:include/linux/bpf.h=3867=bool btf_id_set_contains(const struct btf_id_set *set, u32 id);
--
include/linux/bpf.h-3876-
include/linux/bpf.h:3877:struct bpf_bprintf_buffers {
include/linux/bpf.h-3878- char bin_args[MAX_BPRINTF_BIN_ARGS];
]
|
| 622/4 |
2026/05/18 20:22 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 23766
output: 37
thoughts: 4702
Thoughts: **Analyzing the `bpf_bprintf_prepare` Patch and Potential Soft Lockup**
Okay, so I've been wrestling with this patch and a related soft lockup issue. Initially, I was concerned about alignment issues with `tmp_buf` and how it interacts with the `char` array `bin_args`. The size `MAX_BPRINTF_BIN_ARGS` is 512, which is small. The potential for `tmp_buf` exceeding `tmp_buf_end` was a key worry, especially with `PTR_ALIGN`. However, the code's checks seem to prevent that. The original code had a buffer overflow with `%pI4` and `%pB` but the patch fixes this with `limit` and `min_t`.
The soft lockup, which turns out to be unrelated to the code changes, but worth noting, appears to be CPU 0 waiting on CPU 1, with CPU 1 stuck in `default_idle` with interrupts disabled, a pretty standard state for an idle CPU. This means I'm probably looking at the wrong part of the code to understand the crash.
I went back to the primary bug, a potential infinite loop in `bpf_bprintf_prepare`. Specifically, I scrutinized the code handling `%pB` and other format specifiers:
```c
err = snprintf(
tmp_buf, limit, "%pB",
(void *)(long)
raw_args[num_spec]);
if (err < 0)
err = 0;
tmp_buf +=
min_t(size_t, err + 1, limit);
```
The worry was, if `snprintf` returns an `err` such that `err + 1` is used, but `limit` is zero, we wouldn't advance `tmp_buf` and the loop might not terminate.
Then I realized that, for `%pB`, `snprintf` with a zero limit is safe. It doesn't write anything, but also doesn't cause problems and it is safe, so this can't be the core issue.
Then I realized the *original* code for `%pI4` had an issue:
```c
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf + sizeof_cur_ip > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
...
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
```
The check uses `sizeof_cur_ip`, but that is the size of the *binary* IP address, not the *formatted* string which can be 40 bytes. This meant the check would pass, but then `snprintf` would write out of bounds by up to 24 bytes and the `tmp_buf += err + 1` was broken. The same buffer overflow happened with `%pB` but the fix was to use `limit` and `min_t`.
But, if `err` is negative, `min_t(size_t, 1, limit)` is executed. This advances `tmp_buf` by 1. If `snprintf` fails and doesn't write a null terminator, then the next byte at `tmp_buf` will not be initialized, but we do advance by 1.
The crucial issue is the patch's changes and the `limit` usage. The original behavior for when the buffer is full has changed, if the BPF snprintf writes a large string that gets truncated due to reaching `tmp_buf_end` it used to advance `tmp_buf` (causing a buffer overflow). Now, when `tmp_buf` becomes `tmp_buf_end` on a subsequent call, `bpf_bprintf_prepare` returns `-ENOSPC`, which, I believe, is expected behavior. If a string is truncated, should we signal an error? In the older implementation, it simply overflowed the buffer.
Now, I need to understand whether `-ENOSPC` is the correct response for a truncated output from `snprintf`.
|
| 623/4 |
2026/05/18 20:23 |
tool |
codesearch-definition-comment |
0m
Args: map[ContextFile:kernel/bpf/helpers.c Name:bpf_snprintf]
Results: map[Comment: Kind:function]
|
| 624/4 |
2026/05/18 20:23 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 29467
output: 20
thoughts: 25
Thoughts: Okay, let's dive into this. Here's my thought process summarized in the first person:
**Investigating `bpf_snprintf` in the Kernel Headers**
Alright, I'm thinking I need to refresh my memory on `bpf_snprintf`. Where is that defined? Ah, yes, it should be in the user-space headers, specifically `include/uapi/linux/bpf.h`. I need to check its prototype and any associated documentation, especially any restrictions or gotchas related to its usage within BPF programs. I'm keen to understand things like maximum buffer sizes, format string limitations, and any security considerations. It's likely I'll be looking for nuances that could lead to vulnerabilities, performance bottlenecks, or limitations on the kinds of data I can efficiently log or expose via this function. I'll need to confirm that this is the best approach to formatted output within the BPF context for my current task. I'm also considering whether this function has undergone recent changes that impact its capabilities or security profile.
|
| 625/4 |
2026/05/18 20:23 |
tool |
grepper |
0m
Args: map[Expression:bpf_snprintf\(]
Results: map[Output:include/uapi/linux/bpf.h=1510=union bpf_attr {
--
include/uapi/linux/bpf.h-5246- *
include/uapi/linux/bpf.h:5247: * long bpf_snprintf(char *str, u32 str_size, const char *fmt, u64 *data, u32 data_len)
include/uapi/linux/bpf.h-5248- * Description
--
tools/include/uapi/linux/bpf.h=1510=union bpf_attr {
--
tools/include/uapi/linux/bpf.h-5246- *
tools/include/uapi/linux/bpf.h:5247: * long bpf_snprintf(char *str, u32 str_size, const char *fmt, u64 *data, u32 data_len)
tools/include/uapi/linux/bpf.h-5248- * Description
--
tools/lib/bpf/bpf_helpers.h=185=enum libbpf_tristate {
--
tools/lib/bpf/bpf_helpers.h-282- \
tools/lib/bpf/bpf_helpers.h:283: bpf_snprintf(out, out_size, ___fmt, \
tools/lib/bpf/bpf_helpers.h-284- ___param, sizeof(___param)); \
--
tools/perf/util/bpf_skel/kwork_trace.bpf.c=337=int report_workqueue_execute_end(struct trace_event_raw_workqueue_execute_end *ctx)
--
tools/perf/util/bpf_skel/kwork_trace.bpf.c-347- __builtin_memset(name, 0, sizeof(name));
tools/perf/util/bpf_skel/kwork_trace.bpf.c:348: bpf_snprintf(name, sizeof(name), "%ps", &func_addr, sizeof(func_addr));
tools/perf/util/bpf_skel/kwork_trace.bpf.c-349-
--
tools/perf/util/bpf_skel/kwork_trace.bpf.c=367=int latency_workqueue_execute_start(struct trace_event_raw_workqueue_execute_start *ctx)
--
tools/perf/util/bpf_skel/kwork_trace.bpf.c-377- __builtin_memset(name, 0, sizeof(name));
tools/perf/util/bpf_skel/kwork_trace.bpf.c:378: bpf_snprintf(name, sizeof(name), "%ps", &func_addr, sizeof(func_addr));
tools/perf/util/bpf_skel/kwork_trace.bpf.c-379-
--
tools/testing/selftests/bpf/progs/test_snprintf_single.c=11=int handler(const void *ctx)
--
tools/testing/selftests/bpf/progs/test_snprintf_single.c-14-
tools/testing/selftests/bpf/progs/test_snprintf_single.c:15: bpf_snprintf(NULL, 0, fmt, &arg, sizeof(arg));
tools/testing/selftests/bpf/progs/test_snprintf_single.c-16-
--
tools/testing/selftests/bpf/progs/verifier_ctx.c=617=int syscall_ctx_helper_zero_sized_read(void *ctx)
tools/testing/selftests/bpf/progs/verifier_ctx.c-618-{
tools/testing/selftests/bpf/progs/verifier_ctx.c:619: return bpf_snprintf(0, 0, ctx_snprintf_fmt, ctx, 0);
tools/testing/selftests/bpf/progs/verifier_ctx.c-620-}
--
tools/testing/selftests/bpf/progs/verifier_ctx.c=713=int syscall_ctx_kfunc_zero_sized(void *ctx)
--
tools/testing/selftests/bpf/progs/verifier_ctx.c-811- { \
tools/testing/selftests/bpf/progs/verifier_ctx.c:812: return bpf_snprintf(0, 0, ctx_snprintf_fmt, ctx, 0); \
tools/testing/selftests/bpf/progs/verifier_ctx.c-813- } \
--
tools/testing/selftests/bpf/progs/verifier_live_stack.c=1929=static __used __naked void fwd_parent_key_to_helper(void)
--
tools/testing/selftests/bpf/progs/verifier_live_stack.c-1949- * Regression for keeping later helper args after a whole-stack fallback
tools/testing/selftests/bpf/progs/verifier_live_stack.c:1950: * on an earlier local arg. The first bpf_snprintf() arg is a local
tools/testing/selftests/bpf/progs/verifier_live_stack.c-1951- * frame-derived pointer with offset-imprecise tracking (`fp1 ?`), which
]
|
| 626/4 |
2026/05/18 20:23 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 31103
output: 23
thoughts: 29
Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience:
**Diving into bpf.h for Context**
Okay, time to get granular. I need to understand this specific aspect of BPF, so let's start by looking directly at the source code. I'm going to pull up `include/uapi/linux/bpf.h` and focus on lines 5246 to 5260. That usually gives me a good starting point for understanding how a specific structure or definition works. I'm going to look for any macros or specific fields that might hold clues to the problem I am working on. Let's see what we can find. I am certain that this will quickly reveal any relevant elements.
|
| 627/4 |
2026/05/18 20:23 |
tool |
grepper |
0m
Args: map[Expression:bpf_snprintf\(char \*str]
Results: map[Output:include/uapi/linux/bpf.h=1510=union bpf_attr {
--
include/uapi/linux/bpf.h-5246- *
include/uapi/linux/bpf.h:5247: * long bpf_snprintf(char *str, u32 str_size, const char *fmt, u64 *data, u32 data_len)
include/uapi/linux/bpf.h-5248- * Description
--
tools/include/uapi/linux/bpf.h=1510=union bpf_attr {
--
tools/include/uapi/linux/bpf.h-5246- *
tools/include/uapi/linux/bpf.h:5247: * long bpf_snprintf(char *str, u32 str_size, const char *fmt, u64 *data, u32 data_len)
tools/include/uapi/linux/bpf.h-5248- * Description
]
|
| 628/4 |
2026/05/18 20:23 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 31580
output: 55
thoughts: 52
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Analyzing the BPF Header**
Alright, I need to dive into `include/uapi/linux/bpf.h`. Should I just pull it up with `read-file` or go straight for the more sophisticated `codesearch-definition-source`? Hmm, for a quick scan, `read-file` seems more efficient. Wait a second, considering the sheer volume and complexity of that header, a simple `grep` with context would actually be ideal for pinpointing exactly what I'm looking for right now. I don't need a full blown IDE access for this, a quick search will do the job.
|
| 629/4 |
2026/05/18 20:23 |
tool |
grepper |
0m
Error:
Args: map[Expression:long bpf_snprintf\(char \*str, u32 str_size, const char \*fmt, u64 \*data, u32 data_len\) -A 20]
Results:
|
| 630/4 |
2026/05/18 20:23 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 31851
output: 50
thoughts: 10
Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience:
**Precise Matching is Key**
Okay, so I'm looking at this `grep` command, and it's become immediately apparent that the expression needs to be *exact*. No room for fuzzy logic or assumptions here. I can't afford any false positives or negatives in this situation. A slight deviation in the pattern, a missed character, or an unescaped metacharacter, and the whole thing could fall apart. I'm going to have to make sure my regular expression is rock solid. Thinking specifically about *why* it needs to be precise will help me ensure that my expression is correct, and I will be able to make any necessary adjustments to improve its performance.
|
| 631/4 |
2026/05/18 20:23 |
tool |
grepper |
0m
Args: map[Expression:long bpf_snprintf\(char \*str, u32 str_size, const char \*fmt, u64 \*data, u32 data_len\)]
Results: map[Output:include/uapi/linux/bpf.h=1510=union bpf_attr {
--
include/uapi/linux/bpf.h-5246- *
include/uapi/linux/bpf.h:5247: * long bpf_snprintf(char *str, u32 str_size, const char *fmt, u64 *data, u32 data_len)
include/uapi/linux/bpf.h-5248- * Description
--
tools/include/uapi/linux/bpf.h=1510=union bpf_attr {
--
tools/include/uapi/linux/bpf.h-5246- *
tools/include/uapi/linux/bpf.h:5247: * long bpf_snprintf(char *str, u32 str_size, const char *fmt, u64 *data, u32 data_len)
tools/include/uapi/linux/bpf.h-5248- * Description
]
|
| 632/4 |
2026/05/18 20:23 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 32335
output: 50
thoughts: 29
Thoughts: Here's a first-person summary, tailored for an expert audience, as requested:
**Diving into the BPF Header**
Okay, time to get into the weeds. I'm going to take a look at the `include/uapi/linux/bpf.h` file, specifically lines 5247 through 5260. I know this section is crucial for understanding how the kernel interfaces with BPF programs. I'm expecting to see definitions related to... well, I need to jog my memory, but I have a strong hunch that these lines define the core structures or constants that govern... probably some aspect of map operations or perhaps how BPF programs interact with other kernel subsystems in a structured way. Let's see if my mental model is correct. I should expect a few critical enums or structs that have implications for my current work in... [insert relevant work area here based on user's expertise if available, otherwise assume system-level BPF program analysis]. I'll be looking for any changes or new features here that I may have missed during the last kernel update.
|
| 633/4 |
2026/05/18 20:23 |
tool |
grepper |
0m
Args: map[Expression:long bpf_snprintf\(char \*str, u32 str_size, const char \*fmt, u64 \*data, u32 data_len\)]
Results: map[Output:include/uapi/linux/bpf.h=1510=union bpf_attr {
--
include/uapi/linux/bpf.h-5246- *
include/uapi/linux/bpf.h:5247: * long bpf_snprintf(char *str, u32 str_size, const char *fmt, u64 *data, u32 data_len)
include/uapi/linux/bpf.h-5248- * Description
--
tools/include/uapi/linux/bpf.h=1510=union bpf_attr {
--
tools/include/uapi/linux/bpf.h-5246- *
tools/include/uapi/linux/bpf.h:5247: * long bpf_snprintf(char *str, u32 str_size, const char *fmt, u64 *data, u32 data_len)
tools/include/uapi/linux/bpf.h-5248- * Description
]
|
| 634/4 |
2026/05/18 20:23 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 32910
output: 42
thoughts: 90
Thoughts: Okay, here's my thought process on this task:
**Deconstructing the Problem**
Right, so I need to get information from a file, specifically related to something called `bpf_snprintf`. My initial instinct is to use `read-file` to grab the contents and then sift through the lines. Wait, hold on... if I had shell access, `sed` or `awk` would be a much cleaner solution for line extraction. But, alas, that's not an option here.
Hmm, could I use something like `codesearch-definition-comment`? No, that's not quite right; it's designed for function *definitions*, and I'm just looking for the occurrence of the string, which may be within a comment or code.
Alright, let's keep it simple. I'll stick with `read-file` and target the specific file, `kernel/bpf/helpers.c`. That should give me the context I need around where `bpf_snprintf` is used. Straightforward and avoids unnecessary complexities.
|
| 635/4 |
2026/05/18 20:23 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_snprintf]
Results: map[SourceCode:1064: BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
SourceFile:kernel/bpf/helpers.c]
|
| 636/4 |
2026/05/18 20:23 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 33336
output: 26
thoughts: 14
Thoughts: Okay, let's dive into this. Here's my take on those thoughts, as if they're my own:
**Analyzing the `bpf_snprintf` Function**
Right, so the first thing I need to do is get a complete picture. *I* need the full definition of the `bpf_snprintf` function. No point trying to grasp anything about it without the full context. What are its parameters? What's its return type? What headers does it rely on? What supporting functions does it call within its body? I need to understand its building blocks before I can even begin to assess its purpose or potential implications within the BPF ecosystem. This is foundational; without this, I'm just guessing, and that's not how I work. Let's see it in all its glory.
|
| 637/4 |
2026/05/18 20:24 |
tool |
grepper |
0m
Args: map[Expression:BPF_CALL_5\(bpf_snprintf]
Results: map[Output:kernel/bpf/helpers.c=818=int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
--
kernel/bpf/helpers.c-1063-
kernel/bpf/helpers.c:1064:BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
kernel/bpf/helpers.c-1065- const void *, args, u32, data_len)
--
kernel/trace/bpf_trace.c=976=static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
--
kernel/trace/bpf_trace.c-1005-
kernel/trace/bpf_trace.c:1006:BPF_CALL_5(bpf_snprintf_btf, char *, str, u32, str_size, struct btf_ptr *, ptr,
kernel/trace/bpf_trace.c-1007- u32, btf_ptr_size, u64, flags)
]
|
| 638/4 |
2026/05/18 20:24 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 33853
output: 45
thoughts: 27
Thoughts: Okay, let's dive into this. Here's my take on those lines of code:
**Analyzing BPF Helper Function Implementation: `kernel/bpf/helpers.c`**
Alright, I'm going to focus my attention on `kernel/bpf/helpers.c`, specifically lines 1064 to 1100. This area is likely to be a crucial component, and given the context I'm assuming it's related to some core BPF helper functions. I'll be looking for nuances in how data is handled, system calls are intercepted, or the overall design of a specific helper. I'll need to pay close attention to the data structures involved, any potential for concurrency issues, and of course, the general efficiency of the implementation. It's likely that I'll need to parse this with a specific task in mind (e.g. debugging, finding an optimization opportunity, etc.), so I'll keep that in the back of my mind as I move through it. I want to see how these helpers integrate with the larger BPF framework and what abstractions they provide to the eBPF programs. I know a careful read will yield deeper insights here.
|
| 639/4 |
2026/05/18 20:24 |
tool |
grepper |
0m
Args: map[Expression:BPF_CALL_5\(bpf_snprintf, char \*, str, u32, str_size, char \*, fmt,]
Results: map[Output:kernel/bpf/helpers.c=818=int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
--
kernel/bpf/helpers.c-1063-
kernel/bpf/helpers.c:1064:BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
kernel/bpf/helpers.c-1065- const void *, args, u32, data_len)
]
|
| 640/4 |
2026/05/18 20:24 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 34324
output: 42
thoughts: 34
Thoughts: Okay, here's my summary, presented as a direct thought process:
**Leveraging Tools for Efficient Code Exploration**
Hmm, I don't need to struggle through this manually. It just clicked – I can absolutely use `codeexpert` to quickly get the implementation details I'm after. Asking it directly for the `bpf_snprintf` implementation within `kernel/bpf/helpers.c` is the perfect way to sidestep a tedious manual search. That's precisely what I'll do. It's a much faster and more reliable approach, and I can start there.
|
| 641/4 |
2026/05/18 20:24 |
tool |
codeexpert |
0m
Args: map[Question:Show me the implementation of BPF_CALL_5(bpf_snprintf, ...) in kernel/bpf/helpers.c]
Results: map[Answer:The implementation of `BPF_CALL_5(bpf_snprintf, ...)` is located in `kernel/bpf/helpers.c`. This helper is used by BPF programs to format a string into a buffer, similar to the standard `snprintf` in C, but with specific safety checks and handling for BPF arguments.
Here is the implementation as found in the kernel source:
```c
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
```
### Key Components of the Implementation:
1. **Argument Validation**: It first ensures that `data_len` is a multiple of 8 (as BPF arguments are passed as 64-bit values) and does not exceed `MAX_BPRINTF_VARARGS`.
2. **`bpf_bprintf_prepare`**: This function parses the format string and the raw BPF arguments. It sanitizes pointers (e.g., for `%s` or `%p` specifiers) and prepares a binary argument buffer (`data.bin_args`) that `bstr_printf` can safely consume.
3. **`bstr_printf`**: This is a kernel-internal function (defined in `lib/vsprintf.c`) that performs the actual string formatting using the prepared binary arguments.
4. **`bpf_bprintf_cleanup`**: This cleans up any temporary buffers allocated during the preparation phase (e.g., buffers used to safely copy strings from kernel or user memory).
5. **Return Value**: It returns `err + 1`, where `err` is the number of characters that would have been written (excluding the null terminator), consistent with `snprintf` behavior but adjusted for BPF helper return expectations.
The associated metadata used by the BPF verifier is also defined in the same file:
```c
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
```]
|
| 642/5 |
2026/05/18 20:24 |
agent |
codeexpert |
0m
Model:
Instruction:
You are a capable Linux kernel developer tasked with researching complex questions
about kernel source code. You will be given a concrete question, and need to provide
a concrete answer.
Use tools extensively while researching the question. Don't make assumptions,
or rely on your previous knowledge about the kernel source code, use available tools
to access the actual source code.
Use all available sources of information:
- kernel source code
- documentation in the Documentation dir in the source tree
- git commits descriptions, git blame
Prefer calling several tools at the same time to save round-trips.
Prompt: Show me the implementation of BPF_CALL_5(bpf_snprintf, ...) in kernel/bpf/helpers.c
Reply: The implementation of `BPF_CALL_5(bpf_snprintf, ...)` is located in `kernel/bpf/helpers.c`. This helper is used by BPF programs to format a string into a buffer, similar to the standard `snprintf` in C, but with specific safety checks and handling for BPF arguments.
Here is the implementation as found in the kernel source:
```c
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
```
### Key Components of the Implementation:
1. **Argument Validation**: It first ensures that `data_len` is a multiple of 8 (as BPF arguments are passed as 64-bit values) and does not exceed `MAX_BPRINTF_VARARGS`.
2. **`bpf_bprintf_prepare`**: This function parses the format string and the raw BPF arguments. It sanitizes pointers (e.g., for `%s` or `%p` specifiers) and prepares a binary argument buffer (`data.bin_args`) that `bstr_printf` can safely consume.
3. **`bstr_printf`**: This is a kernel-internal function (defined in `lib/vsprintf.c`) that performs the actual string formatting using the prepared binary arguments.
4. **`bpf_bprintf_cleanup`**: This cleans up any temporary buffers allocated during the preparation phase (e.g., buffers used to safely copy strings from kernel or user memory).
5. **Return Value**: It returns `err + 1`, where `err` is the number of characters that would have been written (excluding the null terminator), consistent with `snprintf` behavior but adjusted for BPF helper return expectations.
The associated metadata used by the BPF verifier is also defined in the same file:
```c
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
```
|
| 643/6 |
2026/05/18 20:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 2280
output: 23
thoughts: 66
Thoughts: **My Investigation into bpf_snprintf Implementation**
Okay, so I'm curious about how `BPF_CALL_5(bpf_snprintf, ...)` is actually implemented. I need to get to the heart of this. My first step is to confirm the existence of `kernel/bpf/helpers.c`. That seems like the most logical place it would reside, given the context. I'll need to check if the file is where I expect it to be. After I've confirmed that, I can then dive into the file itself and hunt down the `bpf_snprintf` function definition. I'm assuming it's likely a BPF helper, so it should be there, wrapped inside the necessary BPF macro calls, but I need to see the actual code to be sure. It could be inline, using a macro to generate the code, but I highly doubt it. Let's start by confirming the file's presence.
|
| 644/6 |
2026/05/18 20:24 |
tool |
codesearch-dir-index |
0m
Args:
Results: map[Files:[arena.c arraymap.c backtrack.c bloom_filter.c bpf_cgrp_storage.c bpf_inode_storage.c bpf_insn_array.c bpf_iter.c bpf_local_storage.c bpf_lru_list.c bpf_lru_list.h bpf_lsm.c bpf_lsm_proto.c bpf_struct_ops.c bpf_task_storage.c btf.c btf_iter.c btf_relocate.c cfg.c cgroup.c cgroup_iter.c check_btf.c const_fold.c core.c cpumap.c cpumask.c crypto.c devmap.c disasm.c disasm.h dispatcher.c dmabuf_iter.c fixups.c hashtab.c helpers.c inode.c kmem_cache_iter.c link_iter.c liveness.c local_storage.c log.c lpm_trie.c map_in_map.c map_in_map.h map_iter.c memalloc.c mmap_unlock_work.h mprog.c net_namespace.c offload.c percpu_freelist.c percpu_freelist.h prog_iter.c queue_stack_maps.c range_tree.c range_tree.h relo_core.c reuseport_array.c ringbuf.c rqspinlock.c rqspinlock.h stackmap.c states.c stream.c syscall.c sysfs_btf.c task_iter.c tcx.c tnum.c token.c trampoline.c verifier.c] Subdirs:[preload]]
|
| 645/6 |
2026/05/18 20:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 3185
output: 42
thoughts: 28
Thoughts: Here's my thought process, summarized as you requested:
**Navigating the BPF Helper Landscape**
Okay, so I know I'm dealing with `kernel/bpf/helpers.c`. That's where I need to start. Now, I'm going to look specifically for `bpf_snprintf` within that file. I expect to find the implementation, or at least its declaration, if it's external. I’m hoping to understand its behavior and any potential limitations in that context. I need to get a handle on how `snprintf` is employed specifically within the BPF ecosystem. This will provide a solid foundation for evaluating its effectiveness and potential areas for optimization or modification.
|
| 646/6 |
2026/05/18 20:24 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_snprintf]
Results: map[SourceCode:1064: BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
SourceFile:kernel/bpf/helpers.c]
|
| 647/6 |
2026/05/18 20:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 3470
output: 23
thoughts: 83
Thoughts: **Investigating the Limited Output from `codesearch-definition-source`**
Okay, so `codesearch-definition-source` isn't behaving as expected. It's only giving me the first line of the definition, which is frustrating. I was hoping for the full entity, as it usually delivers. I need to get the whole thing. I could try `read-file` directly, which might be a more robust approach. Alternatively, I can tweak my call to `codesearch-definition-source` and expand the search scope, although that feels less efficient since it typically works as is. But wait...it *is* only returning a single line. Hmm. Okay, let's just go with `read-file` and try to read around line 1064 directly. That should give me the context I need.
|
| 648/6 |
2026/05/18 20:24 |
tool |
read-file |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com
*/
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/bpf-cgroup.h>
#include <linux/cgroup.h>
#include <linux/rcupdate.h>
#include <linux/random.h>
#include <linux/smp.h>
#include <linux/topology.h>
#include <linux/ktime.h>
#include <linux/sched.h>
#include <linux/uidgid.h>
#include <linux/filter.h>
#include <linux/ctype.h>
#include <linux/jiffies.h>
#include <linux/pid_namespace.h>
#include <linux/poison.h>
#include <linux/proc_ns.h>
#include <linux/sched/task.h>
#include <linux/security.h>
#include <linux/btf_ids.h>
#include <linux/bpf_mem_alloc.h>
#include <linux/kasan.h>
#include <linux/bpf_verifier.h>
#include <linux/uaccess.h>
#include <linux/verification.h>
#include <linux/task_work.h>
#include <linux/irq_work.h>
#include <linux/buildid.h>
#include "../../lib/kstrtox.h"
/* If kernel subsystem is allowing eBPF programs to call this function,
* inside its own verifier_ops->get_func_proto() callback it should return
* bpf_map_lookup_elem_proto, so that verifier can properly check the arguments
*
* Different map implementations will rely on rcu in map methods
* lookup/update/delete, therefore eBPF programs must run under rcu lock
* if program is allowed to access maps, so check rcu_read_lock_held() or
* rcu_read_lock_trace_held() in all three functions.
*/
BPF_CALL_2(bpf_map_lookup_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_elem(map, key);
}
const struct bpf_func_proto bpf_map_lookup_elem_proto = {
.func = bpf_map_lookup_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_4(bpf_map_update_elem, struct bpf_map *, map, void *, key,
void *, value, u64, flags)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_update_elem(map, key, value, flags);
}
const struct bpf_func_proto bpf_map_update_elem_proto = {
.func = bpf_map_update_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_PTR_TO_MAP_VALUE,
.arg4_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_delete_elem, struct bpf_map *, map, void *, key)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return map->ops->map_delete_elem(map, key);
}
const struct bpf_func_proto bpf_map_delete_elem_proto = {
.func = bpf_map_delete_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
};
BPF_CALL_3(bpf_map_push_elem, struct bpf_map *, map, void *, value, u64, flags)
{
return map->ops->map_push_elem(map, value, flags);
}
const struct bpf_func_proto bpf_map_push_elem_proto = {
.func = bpf_map_push_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_2(bpf_map_pop_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_pop_elem(map, value);
}
const struct bpf_func_proto bpf_map_pop_elem_proto = {
.func = bpf_map_pop_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_2(bpf_map_peek_elem, struct bpf_map *, map, void *, value)
{
return map->ops->map_peek_elem(map, value);
}
const struct bpf_func_proto bpf_map_peek_elem_proto = {
.func = bpf_map_peek_elem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_VALUE | MEM_UNINIT | MEM_WRITE,
};
BPF_CALL_3(bpf_map_lookup_percpu_elem, struct bpf_map *, map, void *, key, u32, cpu)
{
WARN_ON_ONCE(!bpf_rcu_lock_held());
return (unsigned long) map->ops->map_lookup_percpu_elem(map, key, cpu);
}
const struct bpf_func_proto bpf_map_lookup_percpu_elem_proto = {
.func = bpf_map_lookup_percpu_elem,
.gpl_only = false,
.pkt_access = true,
.ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_PTR_TO_MAP_KEY,
.arg3_type = ARG_ANYTHING,
};
const struct bpf_func_proto bpf_get_prandom_u32_proto = {
.func = bpf_user_rnd_u32,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_smp_processor_id)
{
return smp_processor_id();
}
const struct bpf_func_proto bpf_get_smp_processor_id_proto = {
.func = bpf_get_smp_processor_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.allow_fastcall = true,
};
BPF_CALL_0(bpf_get_numa_node_id)
{
return numa_node_id();
}
const struct bpf_func_proto bpf_get_numa_node_id_proto = {
.func = bpf_get_numa_node_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_ns)
{
/* NMI safe access to clock monotonic */
return ktime_get_mono_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_ns_proto = {
.func = bpf_ktime_get_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_boot_ns)
{
/* NMI safe access to clock boottime */
return ktime_get_boot_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_boot_ns_proto = {
.func = bpf_ktime_get_boot_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_coarse_ns)
{
return ktime_get_coarse_ns();
}
const struct bpf_func_proto bpf_ktime_get_coarse_ns_proto = {
.func = bpf_ktime_get_coarse_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_ktime_get_tai_ns)
{
/* NMI safe access to clock tai */
return ktime_get_tai_fast_ns();
}
const struct bpf_func_proto bpf_ktime_get_tai_ns_proto = {
.func = bpf_ktime_get_tai_ns,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_pid_tgid)
{
struct task_struct *task = current;
if (unlikely(!task))
return -EINVAL;
return (u64) task->tgid << 32 | task->pid;
}
const struct bpf_func_proto bpf_get_current_pid_tgid_proto = {
.func = bpf_get_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_0(bpf_get_current_uid_gid)
{
struct task_struct *task = current;
kuid_t uid;
kgid_t gid;
if (unlikely(!task))
return -EINVAL;
current_uid_gid(&uid, &gid);
return (u64) from_kgid(&init_user_ns, gid) << 32 |
from_kuid(&init_user_ns, uid);
}
const struct bpf_func_proto bpf_get_current_uid_gid_proto = {
.func = bpf_get_current_uid_gid,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_2(bpf_get_current_comm, char *, buf, u32, size)
{
struct task_struct *task = current;
if (unlikely(!task))
goto err_clear;
/* Verifier guarantees that size > 0 */
strscpy_pad(buf, task->comm, size);
return 0;
err_clear:
memset(buf, 0, size);
return -EINVAL;
}
const struct bpf_func_proto bpf_get_current_comm_proto = {
.func = bpf_get_current_comm,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE,
};
#if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK)
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
union {
__u32 val;
arch_spinlock_t lock;
} u = { .lock = __ARCH_SPIN_LOCK_UNLOCKED };
compiletime_assert(u.val == 0, "__ARCH_SPIN_LOCK_UNLOCKED not 0");
BUILD_BUG_ON(sizeof(*l) != sizeof(__u32));
BUILD_BUG_ON(sizeof(*lock) != sizeof(__u32));
preempt_disable();
arch_spin_lock(l);
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
arch_spinlock_t *l = (void *)lock;
arch_spin_unlock(l);
preempt_enable();
}
#else
static inline void __bpf_spin_lock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
BUILD_BUG_ON(sizeof(*l) != sizeof(*lock));
do {
atomic_cond_read_relaxed(l, !VAL);
} while (atomic_xchg(l, 1));
}
static inline void __bpf_spin_unlock(struct bpf_spin_lock *lock)
{
atomic_t *l = (void *)lock;
atomic_set_release(l, 0);
}
#endif
static DEFINE_PER_CPU(unsigned long, irqsave_flags);
static inline void __bpf_spin_lock_irqsave(struct bpf_spin_lock *lock)
{
unsigned long flags;
local_irq_save(flags);
__bpf_spin_lock(lock);
__this_cpu_write(irqsave_flags, flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_lock, struct bpf_spin_lock *, lock)
{
__bpf_spin_lock_irqsave(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_lock_proto = {
.func = bpf_spin_lock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
static inline void __bpf_spin_unlock_irqrestore(struct bpf_spin_lock *lock)
{
unsigned long flags;
flags = __this_cpu_read(irqsave_flags);
__bpf_spin_unlock(lock);
local_irq_restore(flags);
}
NOTRACE_BPF_CALL_1(bpf_spin_unlock, struct bpf_spin_lock *, lock)
{
__bpf_spin_unlock_irqrestore(lock);
return 0;
}
const struct bpf_func_proto bpf_spin_unlock_proto = {
.func = bpf_spin_unlock,
.gpl_only = false,
.ret_type = RET_VOID,
.arg1_type = ARG_PTR_TO_SPIN_LOCK,
.arg1_btf_id = BPF_PTR_POISON,
};
void copy_map_value_locked(struct bpf_map *map, void *dst, void *src,
bool lock_src)
{
struct bpf_spin_lock *lock;
if (lock_src)
lock = src + map->record->spin_lock_off;
else
lock = dst + map->record->spin_lock_off;
preempt_disable();
__bpf_spin_lock_irqsave(lock);
copy_map_value(map, dst, src);
__bpf_spin_unlock_irqrestore(lock);
preempt_enable();
}
BPF_CALL_0(bpf_jiffies64)
{
return get_jiffies_64();
}
const struct bpf_func_proto bpf_jiffies64_proto = {
.func = bpf_jiffies64,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
#ifdef CONFIG_CGROUPS
BPF_CALL_0(bpf_get_current_cgroup_id)
{
struct cgroup *cgrp;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
cgrp_id = cgroup_id(cgrp);
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_cgroup_id_proto = {
.func = bpf_get_current_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_1(bpf_get_current_ancestor_cgroup_id, int, ancestor_level)
{
struct cgroup *cgrp;
struct cgroup *ancestor;
u64 cgrp_id;
rcu_read_lock();
cgrp = task_dfl_cgroup(current);
ancestor = cgroup_ancestor(cgrp, ancestor_level);
cgrp_id = ancestor ? cgroup_id(ancestor) : 0;
rcu_read_unlock();
return cgrp_id;
}
const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto = {
.func = bpf_get_current_ancestor_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
#endif /* CONFIG_CGROUPS */
#define BPF_STRTOX_BASE_MASK 0x1F
static int __bpf_strtoull(const char *buf, size_t buf_len, u64 flags,
unsigned long long *res, bool *is_negative)
{
unsigned int base = flags & BPF_STRTOX_BASE_MASK;
const char *cur_buf = buf;
size_t cur_len = buf_len;
unsigned int consumed;
size_t val_len;
char str[64];
if (!buf || !buf_len || !res || !is_negative)
return -EINVAL;
if (base != 0 && base != 8 && base != 10 && base != 16)
return -EINVAL;
if (flags & ~BPF_STRTOX_BASE_MASK)
return -EINVAL;
while (cur_buf < buf + buf_len && isspace(*cur_buf))
++cur_buf;
*is_negative = (cur_buf < buf + buf_len && *cur_buf == '-');
if (*is_negative)
++cur_buf;
consumed = cur_buf - buf;
cur_len -= consumed;
if (!cur_len)
return -EINVAL;
cur_len = min(cur_len, sizeof(str) - 1);
memcpy(str, cur_buf, cur_len);
str[cur_len] = '\0';
cur_buf = str;
cur_buf = _parse_integer_fixup_radix(cur_buf, &base);
val_len = _parse_integer(cur_buf, base, res);
if (val_len & KSTRTOX_OVERFLOW)
return -ERANGE;
if (val_len == 0)
return -EINVAL;
cur_buf += val_len;
consumed += cur_buf - str;
return consumed;
}
static int __bpf_strtoll(const char *buf, size_t buf_len, u64 flags,
long long *res)
{
unsigned long long _res;
bool is_negative;
int err;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative) {
if ((long long)-_res > 0)
return -ERANGE;
*res = -_res;
} else {
if ((long long)_res < 0)
return -ERANGE;
*res = _res;
}
return err;
}
BPF_CALL_4(bpf_strtol, const char *, buf, size_t, buf_len, u64, flags,
s64 *, res)
{
long long _res;
int err;
*res = 0;
err = __bpf_strtoll(buf, buf_len, flags, &_res);
if (err < 0)
return err;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtol_proto = {
.func = bpf_strtol,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(s64),
};
BPF_CALL_4(bpf_strtoul, const char *, buf, size_t, buf_len, u64, flags,
u64 *, res)
{
unsigned long long _res;
bool is_negative;
int err;
*res = 0;
err = __bpf_strtoull(buf, buf_len, flags, &_res, &is_negative);
if (err < 0)
return err;
if (is_negative)
return -EINVAL;
*res = _res;
return err;
}
const struct bpf_func_proto bpf_strtoul_proto = {
.func = bpf_strtoul,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_FIXED_SIZE_MEM | MEM_UNINIT | MEM_WRITE | MEM_ALIGNED,
.arg4_size = sizeof(u64),
};
BPF_CALL_3(bpf_strncmp, const char *, s1, u32, s1_sz, const char *, s2)
{
return strncmp(s1, s2, s1_sz);
}
static const struct bpf_func_proto bpf_strncmp_proto = {
.func = bpf_strncmp,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg2_type = ARG_CONST_SIZE,
.arg3_type = ARG_PTR_TO_CONST_STR,
};
BPF_CALL_4(bpf_get_ns_current_pid_tgid, u64, dev, u64, ino,
struct bpf_pidns_info *, nsdata, u32, size)
{
struct task_struct *task = current;
struct pid_namespace *pidns;
int err = -EINVAL;
if (unlikely(size != sizeof(struct bpf_pidns_info)))
goto clear;
if (unlikely((u64)(dev_t)dev != dev))
goto clear;
if (unlikely(!task))
goto clear;
pidns = task_active_pid_ns(task);
if (unlikely(!pidns)) {
err = -ENOENT;
goto clear;
}
if (!ns_match(&pidns->ns, (dev_t)dev, ino))
goto clear;
nsdata->pid = task_pid_nr_ns(task, pidns);
nsdata->tgid = task_tgid_nr_ns(task, pidns);
return 0;
clear:
memset((void *)nsdata, 0, (size_t) size);
return err;
}
const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto = {
.func = bpf_get_ns_current_pid_tgid,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_UNINIT_MEM,
.arg4_type = ARG_CONST_SIZE,
};
static const struct bpf_func_proto bpf_get_raw_smp_processor_id_proto = {
.func = bpf_get_raw_cpu_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map,
u64, flags, void *, data, u64, size)
{
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
}
const struct bpf_func_proto bpf_event_output_data_proto = {
.func = bpf_event_output_data,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
const void __user *, user_ptr)
{
int ret = copy_from_user(dst, user_ptr, size);
if (unlikely(ret)) {
memset(dst, 0, size);
ret = -EFAULT;
}
return ret;
}
const struct bpf_func_proto bpf_copy_from_user_proto = {
.func = bpf_copy_from_user,
.gpl_only = false,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_5(bpf_copy_from_user_task, void *, dst, u32, size,
const void __user *, user_ptr, struct task_struct *, tsk, u64, flags)
{
int ret;
/* flags is not used yet */
if (unlikely(flags))
return -EINVAL;
if (unlikely(!size))
return 0;
ret = access_process_vm(tsk, (unsigned long)user_ptr, dst, size, 0);
if (ret == size)
return 0;
memset(dst, 0, size);
/* Return -EFAULT for partial read */
return ret < 0 ? ret : -EFAULT;
}
const struct bpf_func_proto bpf_copy_from_user_task_proto = {
.func = bpf_copy_from_user_task,
.gpl_only = true,
.might_sleep = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_BTF_ID,
.arg4_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK],
.arg5_type = ARG_ANYTHING
};
BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
{
if (cpu >= nr_cpu_ids)
return (unsigned long)NULL;
return (unsigned long)per_cpu_ptr((const void __percpu *)(const uintptr_t)ptr, cpu);
}
const struct bpf_func_proto bpf_per_cpu_ptr_proto = {
.func = bpf_per_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | PTR_MAYBE_NULL | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
.arg2_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_this_cpu_ptr, const void *, percpu_ptr)
{
return (unsigned long)this_cpu_ptr((const void __percpu *)(const uintptr_t)percpu_ptr);
}
const struct bpf_func_proto bpf_this_cpu_ptr_proto = {
.func = bpf_this_cpu_ptr,
.gpl_only = false,
.ret_type = RET_PTR_TO_MEM_OR_BTF_ID | MEM_RDONLY,
.arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
};
static int bpf_trace_copy_string(char *buf, void *unsafe_ptr, char fmt_ptype,
size_t bufsz)
{
void __user *user_ptr = (__force void __user *)unsafe_ptr;
buf[0] = 0;
switch (fmt_ptype) {
case 's':
#ifdef CONFIG_ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
if ((unsigned long)unsafe_ptr < TASK_SIZE)
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
fallthrough;
#endif
case 'k':
return strncpy_from_kernel_nofault(buf, unsafe_ptr, bufsz);
case 'u':
return strncpy_from_user_nofault(buf, user_ptr, bufsz);
}
return -EINVAL;
}
/* Support executing three nested bprintf helper calls on a given CPU */
#define MAX_BPRINTF_NEST_LEVEL 3
static DEFINE_PER_CPU(struct bpf_bprintf_buffers[MAX_BPRINTF_NEST_LEVEL], bpf_bprintf_bufs);
static DEFINE_PER_CPU(int, bpf_bprintf_nest_level);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs)
{
int nest_level;
preempt_disable();
nest_level = this_cpu_inc_return(bpf_bprintf_nest_level);
if (WARN_ON_ONCE(nest_level > MAX_BPRINTF_NEST_LEVEL)) {
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
return -EBUSY;
}
*bufs = this_cpu_ptr(&bpf_bprintf_bufs[nest_level - 1]);
return 0;
}
void bpf_put_buffers(void)
{
if (WARN_ON_ONCE(this_cpu_read(bpf_bprintf_nest_level) == 0))
return;
this_cpu_dec(bpf_bprintf_nest_level);
preempt_enable();
}
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
{
if (!data->bin_args && !data->buf)
return;
bpf_put_buffers();
}
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
static void *map_key_from_value(struct bpf_map *map, void *value, u32 *arr_idx)
{
if (map->map_type == BPF_MAP_TYPE_ARRAY) {
struct bpf_array *array = container_of(map, struct bpf_array, map);
*arr_idx = ((char *)value - array->value) / array->elem_size;
return arr_idx;
}
return (void *)value - round_up(map->key_size, 8);
}
enum bpf_async_type {
BPF_ASYNC_TYPE_TIMER = 0,
BPF_ASYNC_TYPE_WQ,
};
enum bpf_async_op {
BPF_ASYNC_START,
BPF_ASYNC_CANCEL
};
struct bpf_async_cmd {
struct llist_node node;
u64 nsec;
u32 mode;
enum bpf_async_op op;
};
struct bpf_async_cb {
struct bpf_map *map;
struct bpf_prog *prog;
void __rcu *callback_fn;
void *value;
struct rcu_head rcu;
u64 flags;
struct irq_work worker;
refcount_t refcnt;
enum bpf_async_type type;
struct llist_head async_cmds;
};
/* BPF map elements can contain 'struct bpf_timer'.
* Such map owns all of its BPF timers.
* 'struct bpf_timer' is allocated as part of map element allocation
* and it's zero initialized.
* That space is used to keep 'struct bpf_async_kern'.
* bpf_timer_init() allocates 'struct bpf_hrtimer', inits hrtimer, and
* remembers 'struct bpf_map *' pointer it's part of.
* bpf_timer_set_callback() increments prog refcnt and assign bpf callback_fn.
* bpf_timer_start() arms the timer.
* If user space reference to a map goes to zero at this point
* ops->map_release_uref callback is responsible for cancelling the timers,
* freeing their memory, and decrementing prog's refcnts.
* bpf_timer_cancel() cancels the timer and decrements prog's refcnt.
* Inner maps can contain bpf timers as well. ops->map_release_uref is
* freeing the timers when inner map is replaced or deleted by user space.
*/
struct bpf_hrtimer {
struct bpf_async_cb cb;
struct hrtimer timer;
atomic_t cancelling;
};
struct bpf_work {
struct bpf_async_cb cb;
struct work_struct work;
};
/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */
struct bpf_async_kern {
union {
struct bpf_async_cb *cb;
struct bpf_hrtimer *timer;
struct bpf_work *work;
};
} __attribute__((aligned(8)));
static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running);
static void bpf_async_refcount_put(struct bpf_async_cb *cb);
static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer)
{
struct bpf_hrtimer *t = container_of(hrtimer, struct bpf_hrtimer, timer);
struct bpf_map *map = t->cb.map;
void *value = t->cb.value;
bpf_callback_t callback_fn;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_timer);
callback_fn = rcu_dereference_check(t->cb.callback_fn, rcu_read_lock_bh_held());
if (!callback_fn)
goto out;
/* bpf_timer_cb() runs in hrtimer_run_softirq. It doesn't migrate and
* cannot be preempted by another bpf_timer_cb() on the same cpu.
* Remember the timer this callback is servicing to prevent
* deadlock if callback_fn() calls bpf_timer_cancel() or
* bpf_map_delete_elem() on the same timer.
*/
this_cpu_write(hrtimer_running, t);
key = map_key_from_value(map, value, &idx);
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
/* The verifier checked that return value is zero. */
this_cpu_write(hrtimer_running, NULL);
out:
return HRTIMER_NORESTART;
}
static void bpf_wq_work(struct work_struct *work)
{
struct bpf_work *w = container_of(work, struct bpf_work, work);
struct bpf_async_cb *cb = &w->cb;
struct bpf_map *map = cb->map;
bpf_callback_t callback_fn;
void *value = cb->value;
void *key;
u32 idx;
BTF_TYPE_EMIT(struct bpf_wq);
callback_fn = READ_ONCE(cb->callback_fn);
if (!callback_fn)
return;
key = map_key_from_value(map, value, &idx);
rcu_read_lock_trace();
migrate_disable();
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
migrate_enable();
rcu_read_unlock_trace();
}
static void bpf_async_cb_rcu_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
/*
* Drop the last reference to prog only after RCU GP, as set_callback()
* may race with cancel_and_free()
*/
if (cb->prog)
bpf_prog_put(cb->prog);
kfree_nolock(cb);
}
/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */
static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)
{
struct bpf_async_cb *cb = container_of(rcu, struct bpf_async_cb, rcu);
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
bool retry = false;
/*
* bpf_async_cancel_and_free() tried to cancel timer/wq, but it
* could have raced with timer/wq_start. Now refcnt is zero and
* srcu/rcu GP completed. Cancel timer/wq again.
*/
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER:
if (hrtimer_try_to_cancel(&t->timer) < 0)
retry = true;
break;
case BPF_ASYNC_TYPE_WQ:
if (!cancel_work(&w->work) && work_busy(&w->work))
retry = true;
break;
}
if (retry) {
/*
* hrtimer or wq callback may still be running. It must be
* in rcu_tasks_trace or rcu CS, so wait for GP again.
* It won't retry forever, since refcnt zero prevents all
* operations on timer/wq.
*/
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
return;
}
/* RCU Tasks Trace grace period implies RCU grace period. */
bpf_async_cb_rcu_free(rcu);
}
static void worker_for_call_rcu(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
static void bpf_async_refcount_put(struct bpf_async_cb *cb)
{
if (!refcount_dec_and_test(&cb->refcnt))
return;
if (irqs_disabled()) {
cb->worker = IRQ_WORK_INIT(worker_for_call_rcu);
irq_work_queue(&cb->worker);
} else {
call_rcu_tasks_trace(&cb->rcu, bpf_async_cb_rcu_tasks_trace_free);
}
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async);
static void bpf_async_irq_worker(struct irq_work *work);
static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags,
enum bpf_async_type type)
{
struct bpf_async_cb *cb, *old_cb;
struct bpf_hrtimer *t;
struct bpf_work *w;
clockid_t clockid;
size_t size;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
size = sizeof(struct bpf_hrtimer);
break;
case BPF_ASYNC_TYPE_WQ:
size = sizeof(struct bpf_work);
break;
default:
return -EINVAL;
}
old_cb = READ_ONCE(async->cb);
if (old_cb)
return -EBUSY;
cb = bpf_map_kmalloc_nolock(map, size, 0, map->numa_node);
if (!cb)
return -ENOMEM;
switch (type) {
case BPF_ASYNC_TYPE_TIMER:
clockid = flags & (MAX_CLOCKS - 1);
t = (struct bpf_hrtimer *)cb;
atomic_set(&t->cancelling, 0);
hrtimer_setup(&t->timer, bpf_timer_cb, clockid, HRTIMER_MODE_REL_SOFT);
cb->value = (void *)async - map->record->timer_off;
break;
case BPF_ASYNC_TYPE_WQ:
w = (struct bpf_work *)cb;
INIT_WORK(&w->work, bpf_wq_work);
cb->value = (void *)async - map->record->wq_off;
break;
}
cb->map = map;
cb->prog = NULL;
cb->flags = flags;
cb->worker = IRQ_WORK_INIT(bpf_async_irq_worker);
init_llist_head(&cb->async_cmds);
refcount_set(&cb->refcnt, 1); /* map's reference */
cb->type = type;
rcu_assign_pointer(cb->callback_fn, NULL);
old_cb = cmpxchg(&async->cb, NULL, cb);
if (old_cb) {
/* Lost the race to initialize this bpf_async_kern, drop the allocated object */
kfree_nolock(cb);
return -EBUSY;
}
/* Guarantee the order between async->cb and map->usercnt. So
* when there are concurrent uref release and bpf timer init, either
* bpf_timer_cancel_and_free() called by uref release reads a no-NULL
* timer or atomic64_read() below returns a zero usercnt.
*/
smp_mb();
if (!atomic64_read(&map->usercnt)) {
/* maps with timers must be either held by user space
* or pinned in bpffs.
*/
bpf_async_cancel_and_free(async);
return -EPERM;
}
return 0;
}
BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map,
u64, flags)
{
clock_t clockid = flags & (MAX_CLOCKS - 1);
BUILD_BUG_ON(MAX_CLOCKS != 16);
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer));
if (flags >= MAX_CLOCKS ||
/* similar to timerfd except _ALARM variants are not supported */
(clockid != CLOCK_MONOTONIC &&
clockid != CLOCK_REALTIME &&
clockid != CLOCK_BOOTTIME))
return -EINVAL;
return __bpf_async_init(timer, map, flags, BPF_ASYNC_TYPE_TIMER);
}
static const struct bpf_func_proto bpf_timer_init_proto = {
.func = bpf_timer_init,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_CONST_MAP_PTR,
.arg3_type = ARG_ANYTHING,
};
static int bpf_async_update_prog_callback(struct bpf_async_cb *cb,
struct bpf_prog *prog,
void *callback_fn)
{
struct bpf_prog *prev;
/* Acquire a guard reference on prog to prevent it from being freed during the loop */
if (prog) {
prog = bpf_prog_inc_not_zero(prog);
if (IS_ERR(prog))
return PTR_ERR(prog);
}
do {
if (prog)
prog = bpf_prog_inc_not_zero(prog);
prev = xchg(&cb->prog, prog);
rcu_assign_pointer(cb->callback_fn, callback_fn);
/*
* Release previous prog, make sure that if other CPU is contending,
* to set bpf_prog, references are not leaked as each iteration acquires and
* releases one reference.
*/
if (prev)
bpf_prog_put(prev);
} while (READ_ONCE(cb->prog) != prog ||
(void __force *)READ_ONCE(cb->callback_fn) != callback_fn);
if (prog)
bpf_prog_put(prog);
return 0;
}
static DEFINE_PER_CPU(struct bpf_async_cb *, async_cb_running);
static int bpf_async_schedule_op(struct bpf_async_cb *cb, enum bpf_async_op op,
u64 nsec, u32 timer_mode)
{
/*
* Do not schedule another operation on this cpu if it's in irq_work
* callback that is processing async_cmds queue. Otherwise the following
* loop is possible:
* bpf_timer_start() -> bpf_async_schedule_op() -> irq_work_queue().
* irqrestore -> bpf_async_irq_worker() -> tracepoint -> bpf_timer_start().
*/
if (this_cpu_read(async_cb_running) == cb) {
bpf_async_refcount_put(cb);
return -EDEADLK;
}
struct bpf_async_cmd *cmd = kmalloc_nolock(sizeof(*cmd), 0, NUMA_NO_NODE);
if (!cmd) {
bpf_async_refcount_put(cb);
return -ENOMEM;
}
init_llist_node(&cmd->node);
cmd->nsec = nsec;
cmd->mode = timer_mode;
cmd->op = op;
if (llist_add(&cmd->node, &cb->async_cmds))
irq_work_queue(&cb->worker);
return 0;
}
static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback_fn,
struct bpf_prog *prog)
{
struct bpf_async_cb *cb;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
return bpf_async_update_prog_callback(cb, prog, callback_fn);
}
BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn,
struct bpf_prog_aux *, aux)
{
return __bpf_async_set_callback(timer, callback_fn, aux->prog);
}
static const struct bpf_func_proto bpf_timer_set_callback_proto = {
.func = bpf_timer_set_callback,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_PTR_TO_FUNC,
};
static bool defer_timer_wq_op(void)
{
return in_hardirq() || irqs_disabled();
}
BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, async, u64, nsecs, u64, flags)
{
struct bpf_hrtimer *t;
u32 mode;
if (flags & ~(BPF_F_TIMER_ABS | BPF_F_TIMER_CPU_PIN))
return -EINVAL;
t = READ_ONCE(async->timer);
if (!t || !READ_ONCE(t->cb.prog))
return -EINVAL;
if (flags & BPF_F_TIMER_ABS)
mode = HRTIMER_MODE_ABS_SOFT;
else
mode = HRTIMER_MODE_REL_SOFT;
if (flags & BPF_F_TIMER_CPU_PIN)
mode |= HRTIMER_MODE_PINNED;
/*
* bpf_async_cancel_and_free() could have dropped refcnt to zero. In
* such case BPF progs are not allowed to arm the timer to prevent UAF.
*/
if (!refcount_inc_not_zero(&t->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
hrtimer_start(&t->timer, ns_to_ktime(nsecs), mode);
bpf_async_refcount_put(&t->cb);
return 0;
} else {
return bpf_async_schedule_op(&t->cb, BPF_ASYNC_START, nsecs, mode);
}
}
static const struct bpf_func_proto bpf_timer_start_proto = {
.func = bpf_timer_start,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_ANYTHING,
};
BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, async)
{
struct bpf_hrtimer *t, *cur_t;
bool inc = false;
int ret = 0;
if (defer_timer_wq_op())
return -EOPNOTSUPP;
t = READ_ONCE(async->timer);
if (!t)
return -EINVAL;
cur_t = this_cpu_read(hrtimer_running);
if (cur_t == t) {
/* If bpf callback_fn is trying to bpf_timer_cancel()
* its own timer the hrtimer_cancel() will deadlock
* since it waits for callback_fn to finish.
*/
return -EDEADLK;
}
/* Only account in-flight cancellations when invoked from a timer
* callback, since we want to avoid waiting only if other _callbacks_
* are waiting on us, to avoid introducing lockups. Non-callback paths
* are ok, since nobody would synchronously wait for their completion.
*/
if (!cur_t)
goto drop;
atomic_inc(&t->cancelling);
/* Need full barrier after relaxed atomic_inc */
smp_mb__after_atomic();
inc = true;
if (atomic_read(&cur_t->cancelling)) {
/* We're cancelling timer t, while some other timer callback is
* attempting to cancel us. In such a case, it might be possible
* that timer t belongs to the other callback, or some other
* callback waiting upon it (creating transitive dependencies
* upon us), and we will enter a deadlock if we continue
* cancelling and waiting for it synchronously, since it might
* do the same. Bail!
*/
atomic_dec(&t->cancelling);
return -EDEADLK;
}
drop:
bpf_async_update_prog_callback(&t->cb, NULL, NULL);
/* Cancel the timer and wait for associated callback to finish
* if it was running.
*/
ret = hrtimer_cancel(&t->timer);
if (inc)
atomic_dec(&t->cancelling);
return ret;
}
static const struct bpf_func_proto bpf_timer_cancel_proto = {
.func = bpf_timer_cancel,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_TIMER,
};
static void bpf_async_process_op(struct bpf_async_cb *cb, u32 op,
u64 timer_nsec, u32 timer_mode)
{
switch (cb->type) {
case BPF_ASYNC_TYPE_TIMER: {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
switch (op) {
case BPF_ASYNC_START:
hrtimer_start(&t->timer, ns_to_ktime(timer_nsec), timer_mode);
break;
case BPF_ASYNC_CANCEL:
hrtimer_try_to_cancel(&t->timer);
break;
}
break;
}
case BPF_ASYNC_TYPE_WQ: {
struct bpf_work *w = container_of(cb, struct bpf_work, cb);
switch (op) {
case BPF_ASYNC_START:
schedule_work(&w->work);
break;
case BPF_ASYNC_CANCEL:
cancel_work(&w->work);
break;
}
break;
}
}
bpf_async_refcount_put(cb);
}
static void bpf_async_irq_worker(struct irq_work *work)
{
struct bpf_async_cb *cb = container_of(work, struct bpf_async_cb, worker);
struct llist_node *pos, *n, *list;
list = llist_del_all(&cb->async_cmds);
if (!list)
return;
list = llist_reverse_order(list);
this_cpu_write(async_cb_running, cb);
llist_for_each_safe(pos, n, list) {
struct bpf_async_cmd *cmd;
cmd = container_of(pos, struct bpf_async_cmd, node);
bpf_async_process_op(cb, cmd->op, cmd->nsec, cmd->mode);
kfree_nolock(cmd);
}
this_cpu_write(async_cb_running, NULL);
}
static void bpf_async_cancel_and_free(struct bpf_async_kern *async)
{
struct bpf_async_cb *cb;
if (!READ_ONCE(async->cb))
return;
cb = xchg(&async->cb, NULL);
if (!cb)
return;
bpf_async_update_prog_callback(cb, NULL, NULL);
/*
* No refcount_inc_not_zero(&cb->refcnt) here. Dropping the last
* refcnt. Either synchronously or asynchronously in irq_work.
*/
if (!defer_timer_wq_op()) {
bpf_async_process_op(cb, BPF_ASYNC_CANCEL, 0, 0);
} else {
(void)bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
/*
* bpf_async_schedule_op() either enqueues allocated cmd into llist
* or fails with ENOMEM and drop the last refcnt.
* This is unlikely, but safe, since bpf_async_cb_rcu_tasks_trace_free()
* callback will do additional timer/wq_cancel due to races anyway.
*/
}
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_timer_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
/*
* This function is called by map_delete/update_elem for individual element and
* by ops->map_release_uref when the user space reference to a map reaches zero.
*/
void bpf_wq_cancel_and_free(void *val)
{
bpf_async_cancel_and_free(val);
}
BPF_CALL_2(bpf_kptr_xchg, void *, dst, void *, ptr)
{
unsigned long *kptr = dst;
/* This helper may be inlined by verifier. */
return xchg(kptr, (unsigned long)ptr);
}
/* Unlike other PTR_TO_BTF_ID helpers the btf_id in bpf_kptr_xchg()
* helper is determined dynamically by the verifier. Use BPF_PTR_POISON to
* denote type that verifier will determine.
*/
static const struct bpf_func_proto bpf_kptr_xchg_proto = {
.func = bpf_kptr_xchg,
.gpl_only = false,
.ret_type = RET_PTR_TO_BTF_ID_OR_NULL,
.ret_btf_id = BPF_PTR_POISON,
.arg1_type = ARG_KPTR_XCHG_DEST,
.arg2_type = ARG_PTR_TO_BTF_ID_OR_NULL | OBJ_RELEASE,
.arg2_btf_id = BPF_PTR_POISON,
};
struct bpf_dynptr_file_impl {
struct freader freader;
/* 64 bit offset and size overriding 32 bit ones in bpf_dynptr_kern */
u64 offset;
u64 size;
};
/* Since the upper 8 bits of dynptr->size is reserved, the
* maximum supported size is 2^24 - 1.
*/
#define DYNPTR_MAX_SIZE ((1UL << 24) - 1)
#define DYNPTR_TYPE_SHIFT 28
#define DYNPTR_SIZE_MASK 0xFFFFFF
#define DYNPTR_RDONLY_BIT BIT(31)
bool __bpf_dynptr_is_rdonly(const struct bpf_dynptr_kern *ptr)
{
return ptr->size & DYNPTR_RDONLY_BIT;
}
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr)
{
ptr->size |= DYNPTR_RDONLY_BIT;
}
static void bpf_dynptr_set_type(struct bpf_dynptr_kern *ptr, enum bpf_dynptr_type type)
{
ptr->size |= type << DYNPTR_TYPE_SHIFT;
}
static enum bpf_dynptr_type bpf_dynptr_get_type(const struct bpf_dynptr_kern *ptr)
{
return (ptr->size & ~(DYNPTR_RDONLY_BIT)) >> DYNPTR_TYPE_SHIFT;
}
u64 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
return df->size;
}
return ptr->size & DYNPTR_SIZE_MASK;
}
static void bpf_dynptr_advance_offset(struct bpf_dynptr_kern *ptr, u64 off)
{
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->offset += off;
return;
}
ptr->offset += off;
}
static void bpf_dynptr_set_size(struct bpf_dynptr_kern *ptr, u64 new_size)
{
u32 metadata = ptr->size & ~DYNPTR_SIZE_MASK;
if (bpf_dynptr_get_type(ptr) == BPF_DYNPTR_TYPE_FILE) {
struct bpf_dynptr_file_impl *df = ptr->data;
df->size = new_size;
return;
}
ptr->size = (u32)new_size | metadata;
}
int bpf_dynptr_check_size(u64 size)
{
return size > DYNPTR_MAX_SIZE ? -E2BIG : 0;
}
static int bpf_file_fetch_bytes(struct bpf_dynptr_file_impl *df, u64 offset, void *buf, u64 len)
{
const void *ptr;
if (!buf)
return -EINVAL;
df->freader.buf = buf;
df->freader.buf_sz = len;
ptr = freader_fetch(&df->freader, offset + df->offset, len);
if (!ptr)
return df->freader.err;
if (ptr != buf) /* Force copying into the buffer */
memcpy(buf, ptr, len);
return 0;
}
void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
enum bpf_dynptr_type type, u32 offset, u32 size)
{
ptr->data = data;
ptr->offset = offset;
ptr->size = size;
bpf_dynptr_set_type(ptr, type);
}
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)
{
memset(ptr, 0, sizeof(*ptr));
}
BPF_CALL_4(bpf_dynptr_from_mem, void *, data, u64, size, u64, flags, struct bpf_dynptr_kern *, ptr)
{
int err;
BTF_TYPE_EMIT(struct bpf_dynptr);
err = bpf_dynptr_check_size(size);
if (err)
goto error;
/* flags is currently unsupported */
if (flags) {
err = -EINVAL;
goto error;
}
bpf_dynptr_init(ptr, data, BPF_DYNPTR_TYPE_LOCAL, 0, size);
return 0;
error:
bpf_dynptr_set_null(ptr);
return err;
}
static const struct bpf_func_proto bpf_dynptr_from_mem_proto = {
.func = bpf_dynptr_from_mem,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_ANYTHING,
.arg4_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_LOCAL | MEM_UNINIT | MEM_WRITE,
};
static int __bpf_dynptr_read(void *dst, u64 len, const struct bpf_dynptr_kern *src,
u64 offset, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!src->data || flags)
return -EINVAL;
err = bpf_dynptr_check_off_len(src, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(src);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst, src->data + src->offset + offset, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_XDP:
return __bpf_xdp_load_bytes(src->data, src->offset + offset, dst, len);
case BPF_DYNPTR_TYPE_SKB_META:
memmove(dst, bpf_skb_meta_pointer(src->data, src->offset + offset), len);
return 0;
case BPF_DYNPTR_TYPE_FILE:
return bpf_file_fetch_bytes(src->data, offset, dst, len);
default:
WARN_ONCE(true, "bpf_dynptr_read: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_read, void *, dst, u64, len, const struct bpf_dynptr_kern *, src,
u64, offset, u64, flags)
{
return __bpf_dynptr_read(dst, len, src, offset, flags);
}
static const struct bpf_func_proto bpf_dynptr_read_proto = {
.func = bpf_dynptr_read,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
int __bpf_dynptr_write(const struct bpf_dynptr_kern *dst, u64 offset, void *src,
u64 len, u64 flags)
{
enum bpf_dynptr_type type;
int err;
if (!dst->data || __bpf_dynptr_is_rdonly(dst))
return -EINVAL;
err = bpf_dynptr_check_off_len(dst, offset, len);
if (err)
return err;
type = bpf_dynptr_get_type(dst);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
if (flags)
return -EINVAL;
/* Source and destination may possibly overlap, hence use memmove to
* copy the data. E.g. bpf_dynptr_from_mem may create two dynptr
* pointing to overlapping PTR_TO_MAP_VALUE regions.
*/
memmove(dst->data + dst->offset + offset, src, len);
return 0;
case BPF_DYNPTR_TYPE_SKB:
return __bpf_skb_store_bytes(dst->data, dst->offset + offset, src, len,
flags);
case BPF_DYNPTR_TYPE_XDP:
if (flags)
return -EINVAL;
return __bpf_xdp_store_bytes(dst->data, dst->offset + offset, src, len);
case BPF_DYNPTR_TYPE_SKB_META:
return __bpf_skb_meta_store_bytes(dst->data, dst->offset + offset, src,
len, flags);
default:
WARN_ONCE(true, "bpf_dynptr_write: unknown dynptr type %d\n", type);
return -EFAULT;
}
}
BPF_CALL_5(bpf_dynptr_write, const struct bpf_dynptr_kern *, dst, u64, offset, void *, src,
u64, len, u64, flags)
{
return __bpf_dynptr_write(dst, offset, src, len, flags);
}
static const struct bpf_func_proto bpf_dynptr_write_proto = {
.func = bpf_dynptr_write,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg4_type = ARG_CONST_SIZE_OR_ZERO,
.arg5_type = ARG_ANYTHING,
};
BPF_CALL_3(bpf_dynptr_data, const struct bpf_dynptr_kern *, ptr, u64, offset, u64, len)
{
enum bpf_dynptr_type type;
int err;
if (!ptr->data)
return 0;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return 0;
if (__bpf_dynptr_is_rdonly(ptr))
return 0;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return (unsigned long)(ptr->data + ptr->offset + offset);
case BPF_DYNPTR_TYPE_SKB:
case BPF_DYNPTR_TYPE_XDP:
case BPF_DYNPTR_TYPE_SKB_META:
/* skb and xdp dynptrs should use bpf_dynptr_slice / bpf_dynptr_slice_rdwr */
return 0;
default:
WARN_ONCE(true, "bpf_dynptr_data: unknown dynptr type %d\n", type);
return 0;
}
}
static const struct bpf_func_proto bpf_dynptr_data_proto = {
.func = bpf_dynptr_data,
.gpl_only = false,
.ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL,
.arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY,
.arg2_type = ARG_ANYTHING,
.arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO,
};
const struct bpf_func_proto bpf_get_current_task_proto __weak;
const struct bpf_func_proto bpf_get_current_task_btf_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_proto __weak;
const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_proto __weak;
const struct bpf_func_proto bpf_probe_read_kernel_str_proto __weak;
const struct bpf_func_proto bpf_task_pt_regs_proto __weak;
const struct bpf_func_proto bpf_perf_event_read_proto __weak;
const struct bpf_func_proto bpf_send_signal_proto __weak;
const struct bpf_func_proto bpf_send_signal_thread_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_sleepable_proto __weak;
const struct bpf_func_proto bpf_get_task_stack_proto __weak;
const struct bpf_func_proto bpf_get_branch_snapshot_proto __weak;
const struct bpf_func_proto *
bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_map_lookup_elem:
return &bpf_map_lookup_elem_proto;
case BPF_FUNC_map_update_elem:
return &bpf_map_update_elem_proto;
case BPF_FUNC_map_delete_elem:
return &bpf_map_delete_elem_proto;
case BPF_FUNC_map_push_elem:
return &bpf_map_push_elem_proto;
case BPF_FUNC_map_pop_elem:
return &bpf_map_pop_elem_proto;
case BPF_FUNC_map_peek_elem:
return &bpf_map_peek_elem_proto;
case BPF_FUNC_map_lookup_percpu_elem:
return &bpf_map_lookup_percpu_elem_proto;
case BPF_FUNC_get_prandom_u32:
return &bpf_get_prandom_u32_proto;
case BPF_FUNC_get_smp_processor_id:
return &bpf_get_raw_smp_processor_id_proto;
case BPF_FUNC_get_numa_node_id:
return &bpf_get_numa_node_id_proto;
case BPF_FUNC_tail_call:
return &bpf_tail_call_proto;
case BPF_FUNC_ktime_get_ns:
return &bpf_ktime_get_ns_proto;
case BPF_FUNC_ktime_get_boot_ns:
return &bpf_ktime_get_boot_ns_proto;
case BPF_FUNC_ktime_get_tai_ns:
return &bpf_ktime_get_tai_ns_proto;
case BPF_FUNC_ringbuf_output:
return &bpf_ringbuf_output_proto;
case BPF_FUNC_ringbuf_reserve:
return &bpf_ringbuf_reserve_proto;
case BPF_FUNC_ringbuf_submit:
return &bpf_ringbuf_submit_proto;
case BPF_FUNC_ringbuf_discard:
return &bpf_ringbuf_discard_proto;
case BPF_FUNC_ringbuf_query:
return &bpf_ringbuf_query_proto;
case BPF_FUNC_strncmp:
return &bpf_strncmp_proto;
case BPF_FUNC_strtol:
return &bpf_strtol_proto;
case BPF_FUNC_strtoul:
return &bpf_strtoul_proto;
case BPF_FUNC_get_current_pid_tgid:
return &bpf_get_current_pid_tgid_proto;
case BPF_FUNC_get_ns_current_pid_tgid:
return &bpf_get_ns_current_pid_tgid_proto;
case BPF_FUNC_get_current_uid_gid:
return &bpf_get_current_uid_gid_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_BPF))
return NULL;
switch (func_id) {
case BPF_FUNC_spin_lock:
return &bpf_spin_lock_proto;
case BPF_FUNC_spin_unlock:
return &bpf_spin_unlock_proto;
case BPF_FUNC_jiffies64:
return &bpf_jiffies64_proto;
case BPF_FUNC_per_cpu_ptr:
return &bpf_per_cpu_ptr_proto;
case BPF_FUNC_this_cpu_ptr:
return &bpf_this_cpu_ptr_proto;
case BPF_FUNC_timer_init:
return &bpf_timer_init_proto;
case BPF_FUNC_timer_set_callback:
return &bpf_timer_set_callback_proto;
case BPF_FUNC_timer_start:
return &bpf_timer_start_proto;
case BPF_FUNC_timer_cancel:
return &bpf_timer_cancel_proto;
case BPF_FUNC_kptr_xchg:
return &bpf_kptr_xchg_proto;
case BPF_FUNC_for_each_map_elem:
return &bpf_for_each_map_elem_proto;
case BPF_FUNC_loop:
return &bpf_loop_proto;
case BPF_FUNC_user_ringbuf_drain:
return &bpf_user_ringbuf_drain_proto;
case BPF_FUNC_ringbuf_reserve_dynptr:
return &bpf_ringbuf_reserve_dynptr_proto;
case BPF_FUNC_ringbuf_submit_dynptr:
return &bpf_ringbuf_submit_dynptr_proto;
case BPF_FUNC_ringbuf_discard_dynptr:
return &bpf_ringbuf_discard_dynptr_proto;
case BPF_FUNC_dynptr_from_mem:
return &bpf_dynptr_from_mem_proto;
case BPF_FUNC_dynptr_read:
return &bpf_dynptr_read_proto;
case BPF_FUNC_dynptr_write:
return &bpf_dynptr_write_proto;
case BPF_FUNC_dynptr_data:
return &bpf_dynptr_data_proto;
#ifdef CONFIG_CGROUPS
case BPF_FUNC_cgrp_storage_get:
return &bpf_cgrp_storage_get_proto;
case BPF_FUNC_cgrp_storage_delete:
return &bpf_cgrp_storage_delete_proto;
case BPF_FUNC_get_current_cgroup_id:
return &bpf_get_current_cgroup_id_proto;
case BPF_FUNC_get_current_ancestor_cgroup_id:
return &bpf_get_current_ancestor_cgroup_id_proto;
case BPF_FUNC_current_task_under_cgroup:
return &bpf_current_task_under_cgroup_proto;
#endif
#ifdef CONFIG_CGROUP_NET_CLASSID
case BPF_FUNC_get_cgroup_classid:
return &bpf_get_cgroup_classid_curr_proto;
#endif
case BPF_FUNC_task_storage_get:
return &bpf_task_storage_get_proto;
case BPF_FUNC_task_storage_delete:
return &bpf_task_storage_delete_proto;
default:
break;
}
if (!bpf_token_capable(prog->aux->token, CAP_PERFMON))
return NULL;
switch (func_id) {
case BPF_FUNC_trace_printk:
return bpf_get_trace_printk_proto();
case BPF_FUNC_get_current_task:
return &bpf_get_current_task_proto;
case BPF_FUNC_get_current_task_btf:
return &bpf_get_current_task_btf_proto;
case BPF_FUNC_get_current_comm:
return &bpf_get_current_comm_proto;
case BPF_FUNC_probe_read_user:
return &bpf_probe_read_user_proto;
case BPF_FUNC_probe_read_kernel:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_proto;
case BPF_FUNC_probe_read_user_str:
return &bpf_probe_read_user_str_proto;
case BPF_FUNC_probe_read_kernel_str:
return security_locked_down(LOCKDOWN_BPF_READ_KERNEL) < 0 ?
NULL : &bpf_probe_read_kernel_str_proto;
case BPF_FUNC_copy_from_user:
return &bpf_copy_from_user_proto;
case BPF_FUNC_copy_from_user_task:
return &bpf_copy_from_user_task_proto;
case BPF_FUNC_snprintf_btf:
return &bpf_snprintf_btf_proto;
case BPF_FUNC_snprintf:
return &bpf_snprintf_proto;
case BPF_FUNC_task_pt_regs:
return &bpf_task_pt_regs_proto;
case BPF_FUNC_trace_vprintk:
return bpf_get_trace_vprintk_proto();
case BPF_FUNC_perf_event_read_value:
return bpf_get_perf_event_read_value_proto();
case BPF_FUNC_perf_event_read:
return &bpf_perf_event_read_proto;
case BPF_FUNC_send_signal:
return &bpf_send_signal_proto;
case BPF_FUNC_send_signal_thread:
return &bpf_send_signal_thread_proto;
case BPF_FUNC_get_task_stack:
return prog->sleepable ? &bpf_get_task_stack_sleepable_proto
: &bpf_get_task_stack_proto;
case BPF_FUNC_get_branch_snapshot:
return &bpf_get_branch_snapshot_proto;
case BPF_FUNC_find_vma:
return &bpf_find_vma_proto;
default:
return NULL;
}
}
EXPORT_SYMBOL_GPL(bpf_base_func_proto);
void bpf_list_head_free(const struct btf_field *field, void *list_head,
struct bpf_spin_lock *spin_lock)
{
struct list_head *head = list_head, *orig_head = list_head;
BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head));
BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head));
/* Do the actual list draining outside the lock to not hold the lock for
* too long, and also prevent deadlocks if tracing programs end up
* executing on entry/exit of functions called inside the critical
* section, and end up doing map ops that call bpf_list_head_free for
* the same map value again.
*/
__bpf_spin_lock_irqsave(spin_lock);
if (!head->next || list_empty(head))
goto unlock;
head = head->next;
unlock:
INIT_LIST_HEAD(orig_head);
__bpf_spin_unlock_irqrestore(spin_lock);
while (head != orig_head) {
void *obj = head;
obj -= field->graph_root.node_offset;
head = head->next;
/* The contained type can also have resources, including a
* bpf_list_head which needs to be freed.
*/
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
/* Like rbtree_postorder_for_each_entry_safe, but 'pos' and 'n' are
* 'rb_node *', so field name of rb_node within containing struct is not
* needed.
*
* Since bpf_rb_tree's node type has a corresponding struct btf_field with
* graph_root.node_offset, it's not necessary to know field name
* or type of node struct
*/
#define bpf_rbtree_postorder_for_each_entry_safe(pos, n, root) \
for (pos = rb_first_postorder(root); \
pos && ({ n = rb_next_postorder(pos); 1; }); \
pos = n)
void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock)
{
struct rb_root_cached orig_root, *root = rb_root;
struct rb_node *pos, *n;
void *obj;
BUILD_BUG_ON(sizeof(struct rb_root_cached) > sizeof(struct bpf_rb_root));
BUILD_BUG_ON(__alignof__(struct rb_root_cached) > __alignof__(struct bpf_rb_root));
__bpf_spin_lock_irqsave(spin_lock);
orig_root = *root;
*root = RB_ROOT_CACHED;
__bpf_spin_unlock_irqrestore(spin_lock);
bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) {
obj = pos;
obj -= field->graph_root.node_offset;
__bpf_obj_drop_impl(obj, field->graph_root.value_rec, false);
}
}
__bpf_kfunc_start_defs();
/**
* bpf_obj_new() - allocate an object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate an object of the type identified by @local_type_id__k and
* initialize its special fields. BPF programs can use
* bpf_core_type_id_local() to provide @local_type_id__k. The verifier
* rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
void *p;
p = bpf_mem_alloc(&bpf_global_ma, size);
if (!p)
return NULL;
if (meta)
bpf_obj_init(meta->record, p);
return p;
}
__bpf_kfunc void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_obj_new(local_type_id__k, meta__ign);
}
/**
* bpf_percpu_obj_new() - allocate a percpu object described by program BTF
* @local_type_id__k: type ID in program BTF
* @meta: verifier-supplied struct metadata
*
* Allocate a percpu object of the type identified by @local_type_id__k. BPF
* programs can use bpf_core_type_id_local() to provide @local_type_id__k.
* The verifier rewrites @meta; BPF programs do not set it.
*
* Return: Pointer to the allocated percpu object, or %NULL on failure.
*/
__bpf_kfunc void *bpf_percpu_obj_new(u64 local_type_id__k, struct btf_struct_meta *meta)
{
u64 size = local_type_id__k;
/* The verifier has ensured that meta must be NULL */
return bpf_mem_alloc(&bpf_global_percpu_ma, size);
}
__bpf_kfunc void *bpf_percpu_obj_new_impl(u64 local_type_id__k, void *meta__ign)
{
return bpf_percpu_obj_new(local_type_id__k, meta__ign);
}
/* Must be called under migrate_disable(), as required by bpf_mem_free */
void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu)
{
struct bpf_mem_alloc *ma;
if (rec && rec->refcount_off >= 0 &&
!refcount_dec_and_test((refcount_t *)(p + rec->refcount_off))) {
/* Object is refcounted and refcount_dec didn't result in 0
* refcount. Return without freeing the object
*/
return;
}
if (rec)
bpf_obj_free_fields(rec, p);
if (percpu)
ma = &bpf_global_percpu_ma;
else
ma = &bpf_global_ma;
bpf_mem_free_rcu(ma, p);
}
/**
* bpf_obj_drop() - drop a previously allocated object
* @p__alloc: object to free
* @meta: verifier-supplied struct metadata
*
* Destroy special fields in @p__alloc as needed and free the object. The
* verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
void *p = p__alloc;
__bpf_obj_drop_impl(p, meta ? meta->record : NULL, false);
}
__bpf_kfunc void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)
{
return bpf_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_percpu_obj_drop() - drop a previously allocated percpu object
* @p__alloc: percpu object to free
* @meta: verifier-supplied struct metadata
*
* Free @p__alloc. The verifier rewrites @meta; BPF programs do not set it.
*/
__bpf_kfunc void bpf_percpu_obj_drop(void *p__alloc, struct btf_struct_meta *meta)
{
/* The verifier has ensured that meta must be NULL */
bpf_mem_free_rcu(&bpf_global_percpu_ma, p__alloc);
}
__bpf_kfunc void bpf_percpu_obj_drop_impl(void *p__alloc, void *meta__ign)
{
bpf_percpu_obj_drop(p__alloc, meta__ign);
}
/**
* bpf_refcount_acquire() - turn a local kptr into an owning reference
* @p__refcounted_kptr: non-owning local kptr
* @meta: verifier-supplied struct metadata
*
* Increment the refcount for @p__refcounted_kptr. The verifier rewrites
* @meta; BPF programs do not set it.
*
* Return: Owning reference to @p__refcounted_kptr, or %NULL on failure.
*/
__bpf_kfunc void *bpf_refcount_acquire(void *p__refcounted_kptr, struct btf_struct_meta *meta)
{
struct bpf_refcount *ref;
/* Could just cast directly to refcount_t *, but need some code using
* bpf_refcount type so that it is emitted in vmlinux BTF
*/
ref = (struct bpf_refcount *)(p__refcounted_kptr + meta->record->refcount_off);
if (!refcount_inc_not_zero((refcount_t *)ref))
return NULL;
/* Verifier strips KF_RET_NULL if input is owned ref, see is_kfunc_ret_null
* in verifier.c
*/
return (void *)p__refcounted_kptr;
}
__bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta__ign)
{
return bpf_refcount_acquire(p__refcounted_kptr, meta__ign);
}
static int __bpf_list_add(struct bpf_list_node_kern *node,
struct bpf_list_head *head,
bool tail, struct btf_record *rec, u64 off)
{
struct list_head *n = &node->list_head, *h = (void *)head;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
/* node->owner != NULL implies !list_empty(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
tail ? list_add_tail(n, h) : list_add(n, h);
WRITE_ONCE(node->owner, head);
return 0;
}
/**
* bpf_list_push_front() - add a node to the front of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the front of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_front(head, node, meta__ign, off);
}
/**
* bpf_list_push_back() - add a node to the back of a BPF linked list
* @head: list head
* @node: node to insert
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node at the back of @head. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked.
*/
__bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head,
struct bpf_list_node *node,
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_list_node_kern *n = (void *)node;
return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head,
struct bpf_list_node *node,
void *meta__ign, u64 off)
{
return bpf_list_push_back(head, node, meta__ign, off);
}
static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail)
{
struct list_head *n, *h = (void *)head;
struct bpf_list_node_kern *node;
/* If list_head was 0-initialized by map, bpf_obj_init_field wasn't
* called on its fields, so init here
*/
if (unlikely(!h->next))
INIT_LIST_HEAD(h);
if (list_empty(h))
return NULL;
n = tail ? h->prev : h->next;
node = container_of(n, struct bpf_list_node_kern, list_head);
if (WARN_ON_ONCE(READ_ONCE(node->owner) != head))
return NULL;
list_del_init(n);
WRITE_ONCE(node->owner, NULL);
return (struct bpf_list_node *)n;
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)
{
return __bpf_list_del(head, false);
}
__bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)
{
return __bpf_list_del(head, true);
}
__bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->next;
}
__bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head)
{
struct list_head *h = (struct list_head *)head;
if (list_empty(h) || unlikely(!h->next))
return NULL;
return (struct bpf_list_node *)h->prev;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
struct rb_root_cached *r = (struct rb_root_cached *)root;
struct rb_node *n = &node_internal->rb_node;
/* node_internal->owner != root implies either RB_EMPTY_NODE(n) or
* n is owned by some other tree. No need to check RB_EMPTY_NODE(n)
*/
if (READ_ONCE(node_internal->owner) != root)
return NULL;
rb_erase_cached(n, r);
RB_CLEAR_NODE(n);
WRITE_ONCE(node_internal->owner, NULL);
return (struct bpf_rb_node *)n;
}
/* Need to copy rbtree_add_cached's logic here because our 'less' is a BPF
* program
*/
static int __bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node_kern *node,
void *less, struct btf_record *rec, u64 off)
{
struct rb_node **link = &((struct rb_root_cached *)root)->rb_root.rb_node;
struct rb_node *parent = NULL, *n = &node->rb_node;
bpf_callback_t cb = (bpf_callback_t)less;
bool leftmost = true;
/* node->owner != NULL implies !RB_EMPTY_NODE(n), no need to separately
* check the latter
*/
if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) {
/* Only called from BPF prog, no need to migrate_disable */
__bpf_obj_drop_impl((void *)n - off, rec, false);
return -EINVAL;
}
while (*link) {
parent = *link;
if (cb((uintptr_t)node, (uintptr_t)parent, 0, 0, 0)) {
link = &parent->rb_left;
} else {
link = &parent->rb_right;
leftmost = false;
}
}
rb_link_node(n, parent, link);
rb_insert_color_cached(n, (struct rb_root_cached *)root, leftmost);
WRITE_ONCE(node->owner, root);
return 0;
}
/**
* bpf_rbtree_add() - add a node to a BPF rbtree
* @root: tree root
* @node: node to insert
* @less: comparator used to order nodes
* @meta: verifier-supplied struct metadata
* @off: verifier-supplied offset of @node within the containing object
*
* Insert @node into @root using @less. The verifier rewrites @meta and @off;
* BPF programs do not set them.
*
* Return: 0 on success, or %-EINVAL if @node is already linked in a tree.
*/
__bpf_kfunc int bpf_rbtree_add(struct bpf_rb_root *root,
struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
struct btf_struct_meta *meta,
u64 off)
{
struct bpf_rb_node_kern *n = (void *)node;
return __bpf_rbtree_add(root, n, (void *)less, meta ? meta->record : NULL, off);
}
__bpf_kfunc int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *node,
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b),
void *meta__ign, u64 off)
{
return bpf_rbtree_add(root, node, less, meta__ign, off);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)rb_first_cached(r);
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_root(struct bpf_rb_root *root)
{
struct rb_root_cached *r = (struct rb_root_cached *)root;
return (struct bpf_rb_node *)r->rb_root.rb_node;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_left(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_left;
}
__bpf_kfunc struct bpf_rb_node *bpf_rbtree_right(struct bpf_rb_root *root, struct bpf_rb_node *node)
{
struct bpf_rb_node_kern *node_internal = (struct bpf_rb_node_kern *)node;
if (READ_ONCE(node_internal->owner) != root)
return NULL;
return (struct bpf_rb_node *)node_internal->rb_node.rb_right;
}
/**
* bpf_task_acquire - Acquire a reference to a task. A task acquired by this
* kfunc which is not stored in a map as a kptr, must be released by calling
* bpf_task_release().
* @p: The task on which a reference is being acquired.
*/
__bpf_kfunc struct task_struct *bpf_task_acquire(struct task_struct *p)
{
if (refcount_inc_not_zero(&p->rcu_users))
return p;
return NULL;
}
/**
* bpf_task_release - Release the reference acquired on a task.
* @p: The task on which a reference is being released.
*/
__bpf_kfunc void bpf_task_release(struct task_struct *p)
{
put_task_struct_rcu_user(p);
}
__bpf_kfunc void bpf_task_release_dtor(void *p)
{
put_task_struct_rcu_user(p);
}
CFI_NOSEAL(bpf_task_release_dtor);
#ifdef CONFIG_CGROUPS
/**
* bpf_cgroup_acquire - Acquire a reference to a cgroup. A cgroup acquired by
* this kfunc which is not stored in a map as a kptr, must be released by
* calling bpf_cgroup_release().
* @cgrp: The cgroup on which a reference is being acquired.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)
{
return cgroup_tryget(cgrp) ? cgrp : NULL;
}
/**
* bpf_cgroup_release - Release the reference acquired on a cgroup.
* If this kfunc is invoked in an RCU read region, the cgroup is guaranteed to
* not be freed until the current grace period has ended, even if its refcount
* drops to 0.
* @cgrp: The cgroup on which a reference is being released.
*/
__bpf_kfunc void bpf_cgroup_release(struct cgroup *cgrp)
{
cgroup_put(cgrp);
}
__bpf_kfunc void bpf_cgroup_release_dtor(void *cgrp)
{
cgroup_put(cgrp);
}
CFI_NOSEAL(bpf_cgroup_release_dtor);
/**
* bpf_cgroup_ancestor - Perform a lookup on an entry in a cgroup's ancestor
* array. A cgroup returned by this kfunc which is not subsequently stored in a
* map, must be released by calling bpf_cgroup_release().
* @cgrp: The cgroup for which we're performing a lookup.
* @level: The level of ancestor to look up.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)
{
struct cgroup *ancestor;
if (level > cgrp->level || level < 0)
return NULL;
/* cgrp's refcnt could be 0 here, but ancestors can still be accessed */
ancestor = cgrp->ancestors[level];
if (!cgroup_tryget(ancestor))
return NULL;
return ancestor;
}
/**
* bpf_cgroup_from_id - Find a cgroup from its ID. A cgroup returned by this
* kfunc which is not subsequently stored in a map, must be released by calling
* bpf_cgroup_release().
* @cgid: cgroup id.
*/
__bpf_kfunc struct cgroup *bpf_cgroup_from_id(u64 cgid)
{
struct cgroup *cgrp;
cgrp = __cgroup_get_from_id(cgid);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
/**
* bpf_task_under_cgroup - wrap task_under_cgroup_hierarchy() as a kfunc, test
* task's membership of cgroup ancestry.
* @task: the task to be tested
* @ancestor: possible ancestor of @task's cgroup
*
* Tests whether @task's default cgroup hierarchy is a descendant of @ancestor.
* It follows all the same rules as cgroup_is_descendant, and only applies
* to the default hierarchy.
*/
__bpf_kfunc long bpf_task_under_cgroup(struct task_struct *task,
struct cgroup *ancestor)
{
long ret;
rcu_read_lock();
ret = task_under_cgroup_hierarchy(task, ancestor);
rcu_read_unlock();
return ret;
}
BPF_CALL_2(bpf_current_task_under_cgroup, struct bpf_map *, map, u32, idx)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
struct cgroup *cgrp;
if (unlikely(idx >= array->map.max_entries))
return -E2BIG;
cgrp = READ_ONCE(array->ptrs[idx]);
if (unlikely(!cgrp))
return -EAGAIN;
return task_under_cgroup_hierarchy(current, cgrp);
}
const struct bpf_func_proto bpf_current_task_under_cgroup_proto = {
.func = bpf_current_task_under_cgroup,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_CONST_MAP_PTR,
.arg2_type = ARG_ANYTHING,
};
/**
* bpf_task_get_cgroup1 - Acquires the associated cgroup of a task within a
* specific cgroup1 hierarchy. The cgroup1 hierarchy is identified by its
* hierarchy ID.
* @task: The target task
* @hierarchy_id: The ID of a cgroup1 hierarchy
*
* On success, the cgroup is returen. On failure, NULL is returned.
*/
__bpf_kfunc struct cgroup *
bpf_task_get_cgroup1(struct task_struct *task, int hierarchy_id)
{
struct cgroup *cgrp = task_get_cgroup1(task, hierarchy_id);
if (IS_ERR(cgrp))
return NULL;
return cgrp;
}
#endif /* CONFIG_CGROUPS */
/**
* bpf_task_from_pid - Find a struct task_struct from its pid by looking it up
* in the root pid namespace idr. If a task is returned, it must either be
* stored in a map, or released with bpf_task_release().
* @pid: The pid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_pid_ns(pid, &init_pid_ns);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_task_from_vpid - Find a struct task_struct from its vpid by looking it up
* in the pid namespace of the current task. If a task is returned, it must
* either be stored in a map, or released with bpf_task_release().
* @vpid: The vpid of the task being looked up.
*/
__bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid)
{
struct task_struct *p;
rcu_read_lock();
p = find_task_by_vpid(vpid);
if (p)
p = bpf_task_acquire(p);
rcu_read_unlock();
return p;
}
/**
* bpf_dynptr_slice() - Obtain a read-only pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* If the intention is to write to the data slice, please use
* bpf_dynptr_slice_rdwr.
*
* The user must check that the returned pointer is not null before using it.
*
* Please note that in the case of skb and xdp dynptrs, bpf_dynptr_slice
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a read-only
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
enum bpf_dynptr_type type;
u64 len = buffer__szk;
int err;
if (!ptr->data)
return NULL;
err = bpf_dynptr_check_off_len(ptr, offset, len);
if (err)
return NULL;
type = bpf_dynptr_get_type(ptr);
switch (type) {
case BPF_DYNPTR_TYPE_LOCAL:
case BPF_DYNPTR_TYPE_RINGBUF:
return ptr->data + ptr->offset + offset;
case BPF_DYNPTR_TYPE_SKB:
if (buffer__nullable)
return skb_header_pointer(ptr->data, ptr->offset + offset, len, buffer__nullable);
else
return skb_pointer_if_linear(ptr->data, ptr->offset + offset, len);
case BPF_DYNPTR_TYPE_XDP:
{
void *xdp_ptr = bpf_xdp_pointer(ptr->data, ptr->offset + offset, len);
if (!IS_ERR_OR_NULL(xdp_ptr))
return xdp_ptr;
if (!buffer__nullable)
return NULL;
bpf_xdp_copy_buf(ptr->data, ptr->offset + offset, buffer__nullable, len, false);
return buffer__nullable;
}
case BPF_DYNPTR_TYPE_SKB_META:
return bpf_skb_meta_pointer(ptr->data, ptr->offset + offset);
case BPF_DYNPTR_TYPE_FILE:
err = bpf_file_fetch_bytes(ptr->data, offset, buffer__nullable, buffer__szk);
return err ? NULL : buffer__nullable;
default:
WARN_ONCE(true, "unknown dynptr type %d\n", type);
return NULL;
}
}
/**
* bpf_dynptr_slice_rdwr() - Obtain a writable pointer to the dynptr data.
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__nullable: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
* length of the requested slice. This must be a constant.
*
* For non-skb and non-xdp type dynptrs, there is no difference between
* bpf_dynptr_slice and bpf_dynptr_data.
*
* If buffer__nullable is NULL, the call will fail if buffer_opt was needed.
*
* The returned pointer is writable and may point to either directly the dynptr
* data at the requested offset or to the buffer if unable to obtain a direct
* data pointer to (example: the requested slice is to the paged area of an skb
* packet). In the case where the returned pointer is to the buffer, the user
* is responsible for persisting writes through calling bpf_dynptr_write(). This
* usually looks something like this pattern:
*
* struct eth_hdr *eth = bpf_dynptr_slice_rdwr(&dynptr, 0, buffer, sizeof(buffer));
* if (!eth)
* return TC_ACT_SHOT;
*
* // mutate eth header //
*
* if (eth == buffer)
* bpf_dynptr_write(&ptr, 0, buffer, sizeof(buffer), 0);
*
* Please note that, as in the example above, the user must check that the
* returned pointer is not null before using it.
*
* Please also note that in the case of skb and xdp dynptrs, bpf_dynptr_slice_rdwr
* does not change the underlying packet data pointers, so a call to
* bpf_dynptr_slice_rdwr will not invalidate any ctx->data/data_end pointers in
* the bpf program.
*
* Return: NULL if the call failed (eg invalid dynptr), pointer to a
* data slice (can be either direct pointer to the data or a pointer to the user
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset,
void *buffer__nullable, u64 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data || __bpf_dynptr_is_rdonly(ptr))
return NULL;
/* bpf_dynptr_slice_rdwr is the same logic as bpf_dynptr_slice.
*
* For skb-type dynptrs, it is safe to write into the returned pointer
* if the bpf program allows skb data writes. There are two possibilities
* that may occur when calling bpf_dynptr_slice_rdwr:
*
* 1) The requested slice is in the head of the skb. In this case, the
* returned pointer is directly to skb data, and if the skb is cloned, the
* verifier will have uncloned it (see bpf_unclone_prologue()) already.
* The pointer can be directly written into.
*
* 2) Some portion of the requested slice is in the paged buffer area.
* In this case, the requested data will be copied out into the buffer
* and the returned pointer will be a pointer to the buffer. The skb
* will not be pulled. To persist the write, the user will need to call
* bpf_dynptr_write(), which will pull the skb and commit the write.
*
* Similarly for xdp programs, if the requested slice is not across xdp
* fragments, then a direct pointer will be returned, otherwise the data
* will be copied out into the buffer and the user will need to call
* bpf_dynptr_write() to commit changes.
*/
return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk);
}
__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 size;
if (!ptr->data || start > end)
return -EINVAL;
size = __bpf_dynptr_size(ptr);
if (start > size || end > size)
return -ERANGE;
bpf_dynptr_advance_offset(ptr, start);
bpf_dynptr_set_size(ptr, end - start);
return 0;
}
__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
return !ptr->data;
}
__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return false;
return __bpf_dynptr_is_rdonly(ptr);
}
__bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return -EINVAL;
return __bpf_dynptr_size(ptr);
}
__bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
struct bpf_dynptr *clone__uninit)
{
struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data) {
bpf_dynptr_set_null(clone);
return -EINVAL;
}
*clone = *ptr;
return 0;
}
/**
* bpf_dynptr_copy() - Copy data from one dynptr to another.
* @dst_ptr: Destination dynptr - where data should be copied to
* @dst_off: Offset into the destination dynptr
* @src_ptr: Source dynptr - where data should be copied from
* @src_off: Offset into the source dynptr
* @size: Length of the data to copy from source to destination
*
* Copies data from source dynptr to destination dynptr.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off,
struct bpf_dynptr *src_ptr, u64 src_off, u64 size)
{
struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr;
struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr;
void *src_slice, *dst_slice;
char buf[256];
u64 off;
src_slice = bpf_dynptr_slice(src_ptr, src_off, NULL, size);
dst_slice = bpf_dynptr_slice_rdwr(dst_ptr, dst_off, NULL, size);
if (src_slice && dst_slice) {
memmove(dst_slice, src_slice, size);
return 0;
}
if (src_slice)
return __bpf_dynptr_write(dst, dst_off, src_slice, size, 0);
if (dst_slice)
return __bpf_dynptr_read(dst_slice, size, src, src_off, 0);
if (bpf_dynptr_check_off_len(dst, dst_off, size) ||
bpf_dynptr_check_off_len(src, src_off, size))
return -E2BIG;
off = 0;
while (off < size) {
u64 chunk_sz = min_t(u64, sizeof(buf), size - off);
int err;
err = __bpf_dynptr_read(buf, chunk_sz, src, src_off + off, 0);
if (err)
return err;
err = __bpf_dynptr_write(dst, dst_off + off, buf, chunk_sz, 0);
if (err)
return err;
off += chunk_sz;
}
return 0;
}
/**
* bpf_dynptr_memset() - Fill dynptr memory with a constant byte.
* @p: Destination dynptr - where data will be filled
* @offset: Offset into the dynptr to start filling from
* @size: Number of bytes to fill
* @val: Constant byte to fill the memory with
*
* Fills the @size bytes of the memory area pointed to by @p
* at @offset with the constant byte @val.
* Returns 0 on success; negative error, otherwise.
*/
__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u64 chunk_sz, write_off;
char buf[256];
void* slice;
int err;
slice = bpf_dynptr_slice_rdwr(p, offset, NULL, size);
if (likely(slice)) {
memset(slice, val, size);
return 0;
}
if (__bpf_dynptr_is_rdonly(ptr))
return -EINVAL;
err = bpf_dynptr_check_off_len(ptr, offset, size);
if (err)
return err;
/* Non-linear data under the dynptr, write from a local buffer */
chunk_sz = min_t(u64, sizeof(buf), size);
memset(buf, val, chunk_sz);
for (write_off = 0; write_off < size; write_off += chunk_sz) {
chunk_sz = min_t(u64, sizeof(buf), size - write_off);
err = __bpf_dynptr_write(ptr, offset + write_off, buf, chunk_sz, 0);
if (err)
return err;
}
return 0;
}
__bpf_kfunc void *bpf_cast_to_kern_ctx(void *obj)
{
return obj;
}
__bpf_kfunc void *bpf_rdonly_cast(const void *obj__ign, u32 btf_id__k)
{
return (void *)obj__ign;
}
__bpf_kfunc void bpf_rcu_read_lock(void)
{
rcu_read_lock();
}
__bpf_kfunc void bpf_rcu_read_unlock(void)
{
rcu_read_unlock();
}
struct bpf_throw_ctx {
struct bpf_prog_aux *aux;
u64 sp;
u64 bp;
int cnt;
};
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
{
struct bpf_throw_ctx *ctx = cookie;
struct bpf_prog *prog;
/*
* The RCU read lock is held to safely traverse the latch tree, but we
* don't need its protection when accessing the prog, since it has an
* active stack frame on the current stack trace, and won't disappear.
*/
rcu_read_lock();
prog = bpf_prog_ksym_find(ip);
rcu_read_unlock();
if (!prog)
return !ctx->cnt;
ctx->cnt++;
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
ctx->sp = sp;
ctx->bp = bp;
return false;
}
__bpf_kfunc void bpf_throw(u64 cookie)
{
struct bpf_throw_ctx ctx = {};
arch_bpf_stack_walk(bpf_stack_walker, &ctx);
WARN_ON_ONCE(!ctx.aux);
if (ctx.aux)
WARN_ON_ONCE(!ctx.aux->exception_boundary);
WARN_ON_ONCE(!ctx.bp);
WARN_ON_ONCE(!ctx.cnt);
/* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
* deeper stack depths than ctx.sp as we do not return from bpf_throw,
* which skips compiler generated instrumentation to do the same.
*/
kasan_unpoison_task_stack_below((void *)(long)ctx.sp);
ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0);
WARN(1, "A call to BPF exception callback should never return\n");
}
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_map *map = p__map;
BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq));
BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq));
if (flags)
return -EINVAL;
return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ);
}
__bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
struct bpf_work *w;
if (flags)
return -EINVAL;
w = READ_ONCE(async->work);
if (!w || !READ_ONCE(w->cb.prog))
return -EINVAL;
if (!refcount_inc_not_zero(&w->cb.refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
schedule_work(&w->work);
bpf_async_refcount_put(&w->cb);
return 0;
} else {
return bpf_async_schedule_op(&w->cb, BPF_ASYNC_START, 0, 0);
}
}
__bpf_kfunc int bpf_wq_set_callback(struct bpf_wq *wq,
int (callback_fn)(void *map, int *key, void *value),
unsigned int flags,
struct bpf_prog_aux *aux)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
if (flags)
return -EINVAL;
return __bpf_async_set_callback(async, callback_fn, aux->prog);
}
__bpf_kfunc void bpf_preempt_disable(void)
{
preempt_disable();
}
__bpf_kfunc void bpf_preempt_enable(void)
{
preempt_enable();
}
struct bpf_iter_bits {
__u64 __opaque[2];
} __aligned(8);
#define BITS_ITER_NR_WORDS_MAX 511
struct bpf_iter_bits_kern {
union {
__u64 *bits;
__u64 bits_copy;
};
int nr_bits;
int bit;
} __aligned(8);
/* On 64-bit hosts, unsigned long and u64 have the same size, so passing
* a u64 pointer and an unsigned long pointer to find_next_bit() will
* return the same result, as both point to the same 8-byte area.
*
* For 32-bit little-endian hosts, using a u64 pointer or unsigned long
* pointer also makes no difference. This is because the first iterated
* unsigned long is composed of bits 0-31 of the u64 and the second unsigned
* long is composed of bits 32-63 of the u64.
*
* However, for 32-bit big-endian hosts, this is not the case. The first
* iterated unsigned long will be bits 32-63 of the u64, so swap these two
* ulong values within the u64.
*/
static void swap_ulong_in_u64(u64 *bits, unsigned int nr)
{
#if (BITS_PER_LONG == 32) && defined(__BIG_ENDIAN)
unsigned int i;
for (i = 0; i < nr; i++)
bits[i] = (bits[i] >> 32) | ((u64)(u32)bits[i] << 32);
#endif
}
/**
* bpf_iter_bits_new() - Initialize a new bits iterator for a given memory area
* @it: The new bpf_iter_bits to be created
* @unsafe_ptr__ign: A pointer pointing to a memory area to be iterated over
* @nr_words: The size of the specified memory area, measured in 8-byte units.
* The maximum value of @nr_words is @BITS_ITER_NR_WORDS_MAX. This limit may be
* further reduced by the BPF memory allocator implementation.
*
* This function initializes a new bpf_iter_bits structure for iterating over
* a memory area which is specified by the @unsafe_ptr__ign and @nr_words. It
* copies the data of the memory area to the newly created bpf_iter_bits @it for
* subsequent iteration operations.
*
* On success, 0 is returned. On failure, ERR is returned.
*/
__bpf_kfunc int
bpf_iter_bits_new(struct bpf_iter_bits *it, const u64 *unsafe_ptr__ign, u32 nr_words)
{
struct bpf_iter_bits_kern *kit = (void *)it;
u32 nr_bytes = nr_words * sizeof(u64);
u32 nr_bits = BYTES_TO_BITS(nr_bytes);
int err;
BUILD_BUG_ON(sizeof(struct bpf_iter_bits_kern) != sizeof(struct bpf_iter_bits));
BUILD_BUG_ON(__alignof__(struct bpf_iter_bits_kern) !=
__alignof__(struct bpf_iter_bits));
kit->nr_bits = 0;
kit->bits_copy = 0;
kit->bit = -1;
if (!unsafe_ptr__ign || !nr_words)
return -EINVAL;
if (nr_words > BITS_ITER_NR_WORDS_MAX)
return -E2BIG;
/* Optimization for u64 mask */
if (nr_bits == 64) {
err = bpf_probe_read_kernel_common(&kit->bits_copy, nr_bytes, unsafe_ptr__ign);
if (err)
return -EFAULT;
swap_ulong_in_u64(&kit->bits_copy, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
if (bpf_mem_alloc_check_size(false, nr_bytes))
return -E2BIG;
/* Fallback to memalloc */
kit->bits = bpf_mem_alloc(&bpf_global_ma, nr_bytes);
if (!kit->bits)
return -ENOMEM;
err = bpf_probe_read_kernel_common(kit->bits, nr_bytes, unsafe_ptr__ign);
if (err) {
bpf_mem_free(&bpf_global_ma, kit->bits);
return err;
}
swap_ulong_in_u64(kit->bits, nr_words);
kit->nr_bits = nr_bits;
return 0;
}
/**
* bpf_iter_bits_next() - Get the next bit in a bpf_iter_bits
* @it: The bpf_iter_bits to be checked
*
* This function returns a pointer to a number representing the value of the
* next bit in the bits.
*
* If there are no further bits available, it returns NULL.
*/
__bpf_kfunc int *bpf_iter_bits_next(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
int bit = kit->bit, nr_bits = kit->nr_bits;
const void *bits;
if (!nr_bits || bit >= nr_bits)
return NULL;
bits = nr_bits == 64 ? &kit->bits_copy : kit->bits;
bit = find_next_bit(bits, nr_bits, bit + 1);
if (bit >= nr_bits) {
kit->bit = bit;
return NULL;
}
kit->bit = bit;
return &kit->bit;
}
/**
* bpf_iter_bits_destroy() - Destroy a bpf_iter_bits
* @it: The bpf_iter_bits to be destroyed
*
* Destroy the resource associated with the bpf_iter_bits.
*/
__bpf_kfunc void bpf_iter_bits_destroy(struct bpf_iter_bits *it)
{
struct bpf_iter_bits_kern *kit = (void *)it;
if (kit->nr_bits <= 64)
return;
bpf_mem_free(&bpf_global_ma, kit->bits);
}
/**
* bpf_copy_from_user_str() - Copy a string from an unsafe user address
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address, in user space.
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL-terminated string from userspace to BPF space. If user string is
* too long this will still ensure zero termination in the dst buffer unless
* buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst to 0 on success and
* memset all of @dst on failure.
*/
__bpf_kfunc int bpf_copy_from_user_str(void *dst, u32 dst__sz, const void __user *unsafe_ptr__ign, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(!dst__sz))
return 0;
ret = strncpy_from_user(dst, unsafe_ptr__ign, dst__sz - 1);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset((char *)dst + ret, 0, dst__sz - ret);
else
((char *)dst)[ret] = '\0';
return ret + 1;
}
/**
* bpf_copy_from_user_task_str() - Copy a string from an task's address space
* @dst: Destination address, in kernel space. This buffer must be
* at least @dst__sz bytes long.
* @dst__sz: Maximum number of bytes to copy, includes the trailing NUL.
* @unsafe_ptr__ign: Source address in the task's address space.
* @tsk: The task whose address space will be used
* @flags: The only supported flag is BPF_F_PAD_ZEROS
*
* Copies a NUL terminated string from a task's address space to @dst__sz
* buffer. If user string is too long this will still ensure zero termination
* in the @dst__sz buffer unless buffer size is 0.
*
* If BPF_F_PAD_ZEROS flag is set, memset the tail of @dst__sz to 0 on success
* and memset all of @dst__sz on failure.
*
* Return: The number of copied bytes on success including the NUL terminator.
* A negative error code on failure.
*/
__bpf_kfunc int bpf_copy_from_user_task_str(void *dst, u32 dst__sz,
const void __user *unsafe_ptr__ign,
struct task_struct *tsk, u64 flags)
{
int ret;
if (unlikely(flags & ~BPF_F_PAD_ZEROS))
return -EINVAL;
if (unlikely(dst__sz == 0))
return 0;
ret = copy_remote_vm_str(tsk, (unsigned long)unsafe_ptr__ign, dst, dst__sz, 0);
if (ret < 0) {
if (flags & BPF_F_PAD_ZEROS)
memset(dst, 0, dst__sz);
return ret;
}
if (flags & BPF_F_PAD_ZEROS)
memset(dst + ret, 0, dst__sz - ret);
return ret + 1;
}
/* Keep unsinged long in prototype so that kfunc is usable when emitted to
* vmlinux.h in BPF programs directly, but note that while in BPF prog, the
* unsigned long always points to 8-byte region on stack, the kernel may only
* read and write the 4-bytes on 32-bit.
*/
__bpf_kfunc void bpf_local_irq_save(unsigned long *flags__irq_flag)
{
local_irq_save(*flags__irq_flag);
}
__bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)
{
local_irq_restore(*flags__irq_flag);
}
__bpf_kfunc void __bpf_trap(void)
{
}
/*
* Kfuncs for string operations.
*
* Since strings are not necessarily %NUL-terminated, we cannot directly call
* in-kernel implementations. Instead, we open-code the implementations using
* __get_kernel_nofault instead of plain dereference to make them safe.
*/
static int __bpf_strncasecmp(const char *s1, const char *s2, bool ignore_case, size_t len)
{
char c1, c2;
int i;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < len && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c1, s1, char, err_out);
__get_kernel_nofault(&c2, s2, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 != c2)
return c1 < c2 ? -1 : 1;
if (c1 == '\0')
return 0;
s1++;
s2++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : 0;
err_out:
return -EFAULT;
}
/**
* bpf_strcmp - Compare two strings
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, false, XATTR_SIZE_MAX);
}
/**
* bpf_strcasecmp - Compare two strings, ignoring the case of the characters
* @s1__ign: One string
* @s2__ign: Another string
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasecmp(const char *s1__ign, const char *s2__ign)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, XATTR_SIZE_MAX);
}
/*
* bpf_strncasecmp - Compare two length-limited strings, ignoring case
* @s1__ign: One string
* @s2__ign: Another string
* @len: The maximum number of characters to compare
*
* Return:
* * %0 - Strings are equal
* * %-1 - @s1__ign is smaller
* * %1 - @s2__ign is smaller
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of strings is too large
* * %-ERANGE - One of strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasecmp(const char *s1__ign, const char *s2__ign, size_t len)
{
return __bpf_strncasecmp(s1__ign, s2__ign, true, len);
}
/**
* bpf_strnchr - Find a character in a length limited string
* @s__ign: The string to be searched
* @count: The number of characters to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in the first @count characters of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnchr(const char *s__ign, size_t count, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
return i;
if (sc == '\0')
return -ENOENT;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : -ENOENT;
err_out:
return -EFAULT;
}
/**
* bpf_strchr - Find the first occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Note that the %NUL-terminator is considered part of the string, and can
* be searched for.
*
* Return:
* * >=0 - The index of the first occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchr(const char *s__ign, char c)
{
return bpf_strnchr(s__ign, XATTR_SIZE_MAX, c);
}
/**
* bpf_strchrnul - Find and return a character in a string, or end of string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the first occurrence of @c within @s__ign or index of
* the null byte at the end of @s__ign when @c is not found
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strchrnul(const char *s__ign, char c)
{
char sc;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == '\0' || sc == c)
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strrchr - Find the last occurrence of a character in a string
* @s__ign: The string to be searched
* @c: The character to search for
*
* Return:
* * >=0 - Index of the last occurrence of @c within @s__ign
* * %-ENOENT - @c not found in @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strrchr(const char *s__ign, int c)
{
char sc;
int i, last = -ENOENT;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&sc, s__ign, char, err_out);
if (sc == c)
last = i;
if (sc == '\0')
return last;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strnlen - Calculate the length of a length-limited string
* @s__ign: The string
* @count: The maximum number of characters to count
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strnlen(const char *s__ign, size_t count)
{
char c;
int i;
if (!copy_from_kernel_nofault_allowed(s__ign, 1))
return -ERANGE;
guard(pagefault)();
for (i = 0; i < count && i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&c, s__ign, char, err_out);
if (c == '\0')
return i;
s__ign++;
}
return i == XATTR_SIZE_MAX ? -E2BIG : i;
err_out:
return -EFAULT;
}
/**
* bpf_strlen - Calculate the length of a string
* @s__ign: The string
*
* Return:
* * >=0 - The length of @s__ign
* * %-EFAULT - Cannot read @s__ign
* * %-E2BIG - @s__ign is too large
* * %-ERANGE - @s__ign is outside of kernel address space
*/
__bpf_kfunc int bpf_strlen(const char *s__ign)
{
return bpf_strnlen(s__ign, XATTR_SIZE_MAX);
}
/**
* bpf_strspn - Calculate the length of the initial substring of @s__ign which
* only contains letters in @accept__ign
* @s__ign: The string to be searched
* @accept__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which only
* contains letters from @accept__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strspn(const char *s__ign, const char *accept__ign)
{
char cs, ca;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(accept__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&ca, accept__ign + j, char, err_out);
if (cs == ca || ca == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (ca == '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strcspn - Calculate the length of the initial substring of @s__ign which
* does not contain letters in @reject__ign
* @s__ign: The string to be searched
* @reject__ign: The string to search for
*
* Return:
* * >=0 - The length of the initial substring of @s__ign which does not
* contain letters from @reject__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcspn(const char *s__ign, const char *reject__ign)
{
char cs, cr;
int i, j;
if (!copy_from_kernel_nofault_allowed(s__ign, 1) ||
!copy_from_kernel_nofault_allowed(reject__ign, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
__get_kernel_nofault(&cs, s__ign, char, err_out);
if (cs == '\0')
return i;
for (j = 0; j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&cr, reject__ign + j, char, err_out);
if (cs == cr || cr == '\0')
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (cr != '\0')
return i;
s__ign++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
static int __bpf_strnstr(const char *s1, const char *s2, size_t len,
bool ignore_case)
{
char c1, c2;
int i, j;
if (!copy_from_kernel_nofault_allowed(s1, 1) ||
!copy_from_kernel_nofault_allowed(s2, 1)) {
return -ERANGE;
}
guard(pagefault)();
for (i = 0; i < XATTR_SIZE_MAX; i++) {
for (j = 0; i + j <= len && j < XATTR_SIZE_MAX; j++) {
__get_kernel_nofault(&c2, s2 + j, char, err_out);
if (c2 == '\0')
return i;
/*
* We allow reading an extra byte from s2 (note the
* `i + j <= len` above) to cover the case when s2 is
* a suffix of the first len chars of s1.
*/
if (i + j == len)
break;
__get_kernel_nofault(&c1, s1 + j, char, err_out);
if (ignore_case) {
c1 = tolower(c1);
c2 = tolower(c2);
}
if (c1 == '\0')
return -ENOENT;
if (c1 != c2)
break;
}
if (j == XATTR_SIZE_MAX)
return -E2BIG;
if (i + j == len)
return -ENOENT;
s1++;
}
return -E2BIG;
err_out:
return -EFAULT;
}
/**
* bpf_strstr - Find the first substring in a string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strstr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, false);
}
/**
* bpf_strcasestr - Find the first substring in a string, ignoring the case of
* the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within @s1__ign
* * %-ENOENT - @s2__ign is not a substring of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strcasestr(const char *s1__ign, const char *s2__ign)
{
return __bpf_strnstr(s1__ign, s2__ign, XATTR_SIZE_MAX, true);
}
/**
* bpf_strnstr - Find the first substring in a length-limited string
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strnstr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, false);
}
/**
* bpf_strncasestr - Find the first substring in a length-limited string,
* ignoring the case of the characters
* @s1__ign: The string to be searched
* @s2__ign: The string to search for
* @len: the maximum number of characters to search
*
* Return:
* * >=0 - Index of the first character of the first occurrence of @s2__ign
* within the first @len characters of @s1__ign
* * %-ENOENT - @s2__ign not found in the first @len characters of @s1__ign
* * %-EFAULT - Cannot read one of the strings
* * %-E2BIG - One of the strings is too large
* * %-ERANGE - One of the strings is outside of kernel address space
*/
__bpf_kfunc int bpf_strncasestr(const char *s1__ign, const char *s2__ign,
size_t len)
{
return __bpf_strnstr(s1__ign, s2__ign, len, true);
}
#ifdef CONFIG_KEYS
/**
* bpf_lookup_user_key - lookup a key by its serial
* @serial: key handle serial number
* @flags: lookup-specific flags
*
* Search a key with a given *serial* and the provided *flags*.
* If found, increment the reference count of the key by one, and
* return it in the bpf_key structure.
*
* The bpf_key structure must be passed to bpf_key_put() when done
* with it, so that the key reference count is decremented and the
* bpf_key structure is freed.
*
* Permission checks are deferred to the time the key is used by
* one of the available key-specific kfuncs.
*
* Set *flags* with KEY_LOOKUP_CREATE, to attempt creating a requested
* special keyring (e.g. session keyring), if it doesn't yet exist.
* Set *flags* with KEY_LOOKUP_PARTIAL, to lookup a key without waiting
* for the key construction, and to retrieve uninstantiated keys (keys
* without data attached to them).
*
* Return: a bpf_key pointer with a valid key pointer if the key is found, a
* NULL pointer otherwise.
*/
__bpf_kfunc struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags)
{
key_ref_t key_ref;
struct bpf_key *bkey;
if (flags & ~KEY_LOOKUP_ALL)
return NULL;
/*
* Permission check is deferred until the key is used, as the
* intent of the caller is unknown here.
*/
key_ref = lookup_user_key(serial, flags, KEY_DEFER_PERM_CHECK);
if (IS_ERR(key_ref))
return NULL;
bkey = kmalloc_obj(*bkey);
if (!bkey) {
key_put(key_ref_to_ptr(key_ref));
return NULL;
}
bkey->key = key_ref_to_ptr(key_ref);
bkey->has_ref = true;
return bkey;
}
/**
* bpf_lookup_system_key - lookup a key by a system-defined ID
* @id: key ID
*
* Obtain a bpf_key structure with a key pointer set to the passed key ID.
* The key pointer is marked as invalid, to prevent bpf_key_put() from
* attempting to decrement the key reference count on that pointer. The key
* pointer set in such way is currently understood only by
* verify_pkcs7_signature().
*
* Set *id* to one of the values defined in include/linux/verification.h:
* 0 for the primary keyring (immutable keyring of system keys);
* VERIFY_USE_SECONDARY_KEYRING for both the primary and secondary keyring
* (where keys can be added only if they are vouched for by existing keys
* in those keyrings); VERIFY_USE_PLATFORM_KEYRING for the platform
* keyring (primarily used by the integrity subsystem to verify a kexec'ed
* kerned image and, possibly, the initramfs signature).
*
* Return: a bpf_key pointer with an invalid key pointer set from the
* pre-determined ID on success, a NULL pointer otherwise
*/
__bpf_kfunc struct bpf_key *bpf_lookup_system_key(u64 id)
{
struct bpf_key *bkey;
if (system_keyring_id_check(id) < 0)
return NULL;
bkey = kmalloc_obj(*bkey, GFP_ATOMIC);
if (!bkey)
return NULL;
bkey->key = (struct key *)(unsigned long)id;
bkey->has_ref = false;
return bkey;
}
/**
* bpf_key_put - decrement key reference count if key is valid and free bpf_key
* @bkey: bpf_key structure
*
* Decrement the reference count of the key inside *bkey*, if the pointer
* is valid, and free *bkey*.
*/
__bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
{
if (bkey->has_ref)
key_put(bkey->key);
kfree(bkey);
}
/**
* bpf_verify_pkcs7_signature - verify a PKCS#7 signature
* @data_p: data to verify
* @sig_p: signature of the data
* @trusted_keyring: keyring with keys trusted for signature verification
*
* Verify the PKCS#7 signature *sig_ptr* against the supplied *data_ptr*
* with keys in a keyring referenced by *trusted_keyring*.
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const void *data, *sig;
u32 data_len, sig_len;
int ret;
if (trusted_keyring->has_ref) {
/*
* Do the permission check deferred in bpf_lookup_user_key().
* See bpf_lookup_user_key() for more details.
*
* A call to key_task_permission() here would be redundant, as
* it is already done by keyring_search() called by
* find_asymmetric_key().
*/
ret = key_validate(trusted_keyring->key);
if (ret < 0)
return ret;
}
data_len = __bpf_dynptr_size(data_ptr);
data = __bpf_dynptr_data(data_ptr, data_len);
sig_len = __bpf_dynptr_size(sig_ptr);
sig = __bpf_dynptr_data(sig_ptr, sig_len);
return verify_pkcs7_signature(data, data_len, sig, sig_len,
trusted_keyring->key,
VERIFYING_BPF_SIGNATURE, NULL,
NULL);
#else
return -EOPNOTSUPP;
#endif /* CONFIG_SYSTEM_DATA_VERIFICATION */
}
#endif /* CONFIG_KEYS */
typedef int (*bpf_task_work_callback_t)(struct bpf_map *map, void *key, void *value);
enum bpf_task_work_state {
/* bpf_task_work is ready to be used */
BPF_TW_STANDBY = 0,
/* irq work scheduling in progress */
BPF_TW_PENDING,
/* task work scheduling in progress */
BPF_TW_SCHEDULING,
/* task work is scheduled successfully */
BPF_TW_SCHEDULED,
/* callback is running */
BPF_TW_RUNNING,
/* associated BPF map value is deleted */
BPF_TW_FREED,
};
struct bpf_task_work_ctx {
enum bpf_task_work_state state;
refcount_t refcnt;
struct callback_head work;
struct irq_work irq_work;
/* bpf_prog that schedules task work */
struct bpf_prog *prog;
/* task for which callback is scheduled */
struct task_struct *task;
/* the map and map value associated with this context */
struct bpf_map *map;
void *map_val;
enum task_work_notify_mode mode;
bpf_task_work_callback_t callback_fn;
struct rcu_head rcu;
} __aligned(8);
/* Actual type for struct bpf_task_work */
struct bpf_task_work_kern {
struct bpf_task_work_ctx *ctx;
};
static void bpf_task_work_ctx_reset(struct bpf_task_work_ctx *ctx)
{
if (ctx->prog) {
bpf_prog_put(ctx->prog);
ctx->prog = NULL;
}
if (ctx->task) {
bpf_task_release(ctx->task);
ctx->task = NULL;
}
}
static bool bpf_task_work_ctx_tryget(struct bpf_task_work_ctx *ctx)
{
return refcount_inc_not_zero(&ctx->refcnt);
}
static void bpf_task_work_destroy(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_ctx_reset(ctx);
kfree_rcu(ctx, rcu);
}
static void bpf_task_work_ctx_put(struct bpf_task_work_ctx *ctx)
{
if (!refcount_dec_and_test(&ctx->refcnt))
return;
if (irqs_disabled()) {
ctx->irq_work = IRQ_WORK_INIT(bpf_task_work_destroy);
irq_work_queue(&ctx->irq_work);
} else {
bpf_task_work_destroy(&ctx->irq_work);
}
}
static void bpf_task_work_cancel(struct bpf_task_work_ctx *ctx)
{
/*
* Scheduled task_work callback holds ctx ref, so if we successfully
* cancelled, we put that ref on callback's behalf. If we couldn't
* cancel, callback will inevitably run or has already completed
* running, and it would have taken care of its ctx ref itself.
*/
if (task_work_cancel(ctx->task, &ctx->work))
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_callback(struct callback_head *cb)
{
struct bpf_task_work_ctx *ctx = container_of(cb, struct bpf_task_work_ctx, work);
enum bpf_task_work_state state;
u32 idx;
void *key;
/* Read lock is needed to protect ctx and map key/value access */
guard(rcu_tasks_trace)();
/*
* This callback may start running before bpf_task_work_irq() switched to
* SCHEDULED state, so handle both transition variants SCHEDULING|SCHEDULED -> RUNNING.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_RUNNING);
if (state == BPF_TW_SCHEDULED)
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULED, BPF_TW_RUNNING);
if (state == BPF_TW_FREED) {
bpf_task_work_ctx_put(ctx);
return;
}
key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx);
migrate_disable();
ctx->callback_fn(ctx->map, key, ctx->map_val);
migrate_enable();
bpf_task_work_ctx_reset(ctx);
(void)cmpxchg(&ctx->state, BPF_TW_RUNNING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
}
static void bpf_task_work_irq(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
enum bpf_task_work_state state;
int err;
guard(rcu)();
if (cmpxchg(&ctx->state, BPF_TW_PENDING, BPF_TW_SCHEDULING) != BPF_TW_PENDING) {
bpf_task_work_ctx_put(ctx);
return;
}
err = task_work_add(ctx->task, &ctx->work, ctx->mode);
if (err) {
bpf_task_work_ctx_reset(ctx);
/*
* try to switch back to STANDBY for another task_work reuse, but we might have
* gone to FREED already, which is fine as we already cleaned up after ourselves
*/
(void)cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_STANDBY);
bpf_task_work_ctx_put(ctx);
return;
}
/*
* It's technically possible for just scheduled task_work callback to
* complete running by now, going SCHEDULING -> RUNNING and then
* dropping its ctx refcount. Instead of capturing an extra ref just
* to protect below ctx->state access, we rely on rcu_read_lock
* above to prevent kfree_rcu from freeing ctx before we return.
*/
state = cmpxchg(&ctx->state, BPF_TW_SCHEDULING, BPF_TW_SCHEDULED);
if (state == BPF_TW_FREED)
bpf_task_work_cancel(ctx); /* clean up if we switched into FREED state */
}
static struct bpf_task_work_ctx *bpf_task_work_fetch_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_kern *twk = (void *)tw;
struct bpf_task_work_ctx *ctx, *old_ctx;
ctx = READ_ONCE(twk->ctx);
if (ctx)
return ctx;
ctx = bpf_map_kmalloc_nolock(map, sizeof(*ctx), 0, NUMA_NO_NODE);
if (!ctx)
return ERR_PTR(-ENOMEM);
memset(ctx, 0, sizeof(*ctx));
refcount_set(&ctx->refcnt, 1); /* map's own ref */
ctx->state = BPF_TW_STANDBY;
old_ctx = cmpxchg(&twk->ctx, NULL, ctx);
if (old_ctx) {
/*
* tw->ctx is set by concurrent BPF program, release allocated
* memory and try to reuse already set context.
*/
kfree_nolock(ctx);
return old_ctx;
}
return ctx; /* Success */
}
static struct bpf_task_work_ctx *bpf_task_work_acquire_ctx(struct bpf_task_work *tw,
struct bpf_map *map)
{
struct bpf_task_work_ctx *ctx;
/*
* Sleepable BPF programs hold rcu_read_lock_trace but not
* regular rcu_read_lock. Since kfree_rcu waits for regular
* RCU GP, the ctx can be freed while we're between reading
* the pointer and incrementing the refcount. Take regular
* rcu_read_lock to prevent kfree_rcu from freeing the ctx
* before we can tryget it.
*/
scoped_guard(rcu) {
ctx = bpf_task_work_fetch_ctx(tw, map);
if (IS_ERR(ctx))
return ctx;
/* try to get ref for task_work callback to hold */
if (!bpf_task_work_ctx_tryget(ctx))
return ERR_PTR(-EBUSY);
}
if (cmpxchg(&ctx->state, BPF_TW_STANDBY, BPF_TW_PENDING) != BPF_TW_STANDBY) {
/* lost acquiring race or map_release_uref() stole it from us, put ref and bail */
bpf_task_work_ctx_put(ctx);
return ERR_PTR(-EBUSY);
}
/*
* If no process or bpffs is holding a reference to the map, no new callbacks should be
* scheduled. This does not address any race or correctness issue, but rather is a policy
* choice: dropping user references should stop everything.
*/
if (!atomic64_read(&map->usercnt)) {
/* drop ref we just got for task_work callback itself */
bpf_task_work_ctx_put(ctx);
/* transfer map's ref into cancel_and_free() */
bpf_task_work_cancel_and_free(tw);
return ERR_PTR(-EBUSY);
}
return ctx;
}
static int bpf_task_work_schedule(struct task_struct *task, struct bpf_task_work *tw,
struct bpf_map *map, bpf_task_work_callback_t callback_fn,
struct bpf_prog_aux *aux, enum task_work_notify_mode mode)
{
struct bpf_prog *prog;
struct bpf_task_work_ctx *ctx;
int err;
BTF_TYPE_EMIT(struct bpf_task_work);
prog = bpf_prog_inc_not_zero(aux->prog);
if (IS_ERR(prog))
return -EBADF;
task = bpf_task_acquire(task);
if (!task) {
err = -EBADF;
goto release_prog;
}
ctx = bpf_task_work_acquire_ctx(tw, map);
if (IS_ERR(ctx)) {
err = PTR_ERR(ctx);
goto release_all;
}
ctx->task = task;
ctx->callback_fn = callback_fn;
ctx->prog = prog;
ctx->mode = mode;
ctx->map = map;
ctx->map_val = (void *)tw - map->record->task_work_off;
init_task_work(&ctx->work, bpf_task_work_callback);
init_irq_work(&ctx->irq_work, bpf_task_work_irq);
irq_work_queue(&ctx->irq_work);
return 0;
release_all:
bpf_task_release(task);
release_prog:
bpf_prog_put(prog);
return err;
}
/**
* bpf_task_work_schedule_signal - Schedule BPF callback using task_work_add with TWA_SIGNAL
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_signal(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_SIGNAL);
}
/**
* bpf_task_work_schedule_resume - Schedule BPF callback using task_work_add with TWA_RESUME
* mode
* @task: Task struct for which callback should be scheduled
* @tw: Pointer to struct bpf_task_work in BPF map value for internal bookkeeping
* @map__map: bpf_map that embeds struct bpf_task_work in the values
* @callback: pointer to BPF subprogram to call
* @aux: pointer to bpf_prog_aux of the caller BPF program, implicitly set by the verifier
*
* Return: 0 if task work has been scheduled successfully, negative error code otherwise
*/
__bpf_kfunc int bpf_task_work_schedule_resume(struct task_struct *task, struct bpf_task_work *tw,
void *map__map, bpf_task_work_callback_t callback,
struct bpf_prog_aux *aux)
{
return bpf_task_work_schedule(task, tw, map__map, callback, aux, TWA_RESUME);
}
static int make_file_dynptr(struct file *file, u32 flags, bool may_sleep,
struct bpf_dynptr_kern *ptr)
{
struct bpf_dynptr_file_impl *state;
/* flags is currently unsupported */
if (flags) {
bpf_dynptr_set_null(ptr);
return -EINVAL;
}
state = kmalloc_nolock(sizeof(*state), 0, NUMA_NO_NODE);
if (!state) {
bpf_dynptr_set_null(ptr);
return -ENOMEM;
}
state->offset = 0;
state->size = U64_MAX; /* Don't restrict size, as file may change anyways */
freader_init_from_file(&state->freader, NULL, 0, file, may_sleep);
bpf_dynptr_init(ptr, state, BPF_DYNPTR_TYPE_FILE, 0, 0);
bpf_dynptr_set_rdonly(ptr);
return 0;
}
__bpf_kfunc int bpf_dynptr_from_file(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, false, (struct bpf_dynptr_kern *)ptr__uninit);
}
int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, struct bpf_dynptr *ptr__uninit)
{
return make_file_dynptr(file, flags, true, (struct bpf_dynptr_kern *)ptr__uninit);
}
__bpf_kfunc int bpf_dynptr_file_discard(struct bpf_dynptr *dynptr)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)dynptr;
struct bpf_dynptr_file_impl *df = ptr->data;
if (!df)
return 0;
freader_cleanup(&df->freader);
kfree_nolock(df);
bpf_dynptr_set_null(ptr);
return 0;
}
/**
* bpf_timer_cancel_async - try to deactivate a timer
* @timer: bpf_timer to stop
*
* Returns:
*
* * 0 when the timer was not active
* * 1 when the timer was active
* * -1 when the timer is currently executing the callback function and
* cannot be stopped
* * -ECANCELED when the timer will be cancelled asynchronously
* * -ENOMEM when out of memory
* * -EINVAL when the timer was not initialized
* * -ENOENT when this kfunc is racing with timer deletion
*/
__bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer)
{
struct bpf_async_kern *async = (void *)timer;
struct bpf_async_cb *cb;
int ret;
cb = READ_ONCE(async->cb);
if (!cb)
return -EINVAL;
/*
* Unlike hrtimer_start() it's ok to synchronously call
* hrtimer_try_to_cancel() when refcnt reached zero, but deferring to
* irq_work is not, since irq callback may execute after RCU GP and
* cb could be freed at that time. Check for refcnt zero for
* consistency.
*/
if (!refcount_inc_not_zero(&cb->refcnt))
return -ENOENT;
if (!defer_timer_wq_op()) {
struct bpf_hrtimer *t = container_of(cb, struct bpf_hrtimer, cb);
ret = hrtimer_try_to_cancel(&t->timer);
bpf_async_refcount_put(cb);
return ret;
} else {
ret = bpf_async_schedule_op(cb, BPF_ASYNC_CANCEL, 0, 0);
return ret ? ret : -ECANCELED;
}
}
__bpf_kfunc_end_defs();
static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
{
struct bpf_task_work_ctx *ctx = container_of(irq_work, struct bpf_task_work_ctx, irq_work);
bpf_task_work_cancel(ctx); /* this might put task_work callback's ref */
bpf_task_work_ctx_put(ctx); /* and here we put map's own ref that was transferred to us */
}
void bpf_task_work_cancel_and_free(void *val)
{
struct bpf_task_work_kern *twk = val;
struct bpf_task_work_ctx *ctx;
enum bpf_task_work_state state;
ctx = xchg(&twk->ctx, NULL);
if (!ctx)
return;
state = xchg(&ctx->state, BPF_TW_FREED);
if (state == BPF_TW_SCHEDULED) {
/* run in irq_work to avoid locks in NMI */
init_irq_work(&ctx->irq_work, bpf_task_work_cancel_scheduled);
irq_work_queue(&ctx->irq_work);
return;
}
bpf_task_work_ctx_put(ctx); /* put bpf map's ref */
}
BTF_KFUNCS_START(generic_btf_ids)
#ifdef CONFIG_CRASH_DUMP
BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE)
#endif
BTF_ID_FLAGS(func, bpf_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_percpu_obj_new, KF_ACQUIRE | KF_RET_NULL | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_new_impl, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop, KF_RELEASE | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_percpu_obj_drop_impl, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_refcount_acquire, KF_ACQUIRE | KF_RET_NULL | KF_RCU | KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_refcount_acquire_impl, KF_ACQUIRE | KF_RET_NULL | KF_RCU)
BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_front_impl)
BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_list_push_back_impl)
BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_add, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_rbtree_add_impl)
BTF_ID_FLAGS(func, bpf_rbtree_first, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_root, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_left, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_rbtree_right, KF_RET_NULL)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_cgroup_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_release, KF_RELEASE)
BTF_ID_FLAGS(func, bpf_cgroup_ancestor, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU)
BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
#endif
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_send_signal_task)
#endif
#ifdef CONFIG_KEYS
BTF_ID_FLAGS(func, bpf_lookup_user_key, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_lookup_system_key, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_key_put, KF_RELEASE)
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
BTF_ID_FLAGS(func, bpf_verify_pkcs7_signature, KF_SLEEPABLE)
#endif
#endif
#ifdef CONFIG_S390
BTF_ID_FLAGS(func, bpf_get_lowcore)
#endif
BTF_KFUNCS_END(generic_btf_ids)
static const struct btf_kfunc_id_set generic_kfunc_set = {
.owner = THIS_MODULE,
.set = &generic_btf_ids,
};
BTF_ID_LIST(generic_dtor_ids)
BTF_ID(struct, task_struct)
BTF_ID(func, bpf_task_release_dtor)
#ifdef CONFIG_CGROUPS
BTF_ID(struct, cgroup)
BTF_ID(func, bpf_cgroup_release_dtor)
#endif
BTF_KFUNCS_START(common_btf_ids)
BTF_ID_FLAGS(func, bpf_cast_to_kern_ctx, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rdonly_cast, KF_FASTCALL)
BTF_ID_FLAGS(func, bpf_rcu_read_lock)
BTF_ID_FLAGS(func, bpf_rcu_read_unlock)
BTF_ID_FLAGS(func, bpf_dynptr_slice, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_dynptr_slice_rdwr, KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_num_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU)
BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY)
#ifdef CONFIG_CGROUPS
BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_iter_css_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_css_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_css_destroy, KF_ITER_DESTROY)
#endif
BTF_ID_FLAGS(func, bpf_iter_task_new, KF_ITER_NEW | KF_RCU_PROTECTED)
BTF_ID_FLAGS(func, bpf_iter_task_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_task_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_dynptr_adjust)
BTF_ID_FLAGS(func, bpf_dynptr_is_null)
BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly)
BTF_ID_FLAGS(func, bpf_dynptr_size)
BTF_ID_FLAGS(func, bpf_dynptr_clone)
BTF_ID_FLAGS(func, bpf_dynptr_copy)
BTF_ID_FLAGS(func, bpf_dynptr_memset)
#ifdef CONFIG_NET
BTF_ID_FLAGS(func, bpf_modify_return_test_tp)
#endif
BTF_ID_FLAGS(func, bpf_wq_init)
BTF_ID_FLAGS(func, bpf_wq_set_callback, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_wq_start)
BTF_ID_FLAGS(func, bpf_preempt_disable)
BTF_ID_FLAGS(func, bpf_preempt_enable)
BTF_ID_FLAGS(func, bpf_iter_bits_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_bits_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_bits_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_copy_from_user_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_get_kmem_cache)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_local_irq_save)
BTF_ID_FLAGS(func, bpf_local_irq_restore)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr)
BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr)
BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE)
#endif
#ifdef CONFIG_DMA_SHARED_BUFFER
BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE)
#endif
BTF_ID_FLAGS(func, __bpf_trap)
BTF_ID_FLAGS(func, bpf_strcmp);
BTF_ID_FLAGS(func, bpf_strcasecmp);
BTF_ID_FLAGS(func, bpf_strncasecmp);
BTF_ID_FLAGS(func, bpf_strchr);
BTF_ID_FLAGS(func, bpf_strchrnul);
BTF_ID_FLAGS(func, bpf_strnchr);
BTF_ID_FLAGS(func, bpf_strrchr);
BTF_ID_FLAGS(func, bpf_strlen);
BTF_ID_FLAGS(func, bpf_strnlen);
BTF_ID_FLAGS(func, bpf_strspn);
BTF_ID_FLAGS(func, bpf_strcspn);
BTF_ID_FLAGS(func, bpf_strstr);
BTF_ID_FLAGS(func, bpf_strcasestr);
BTF_ID_FLAGS(func, bpf_strnstr);
BTF_ID_FLAGS(func, bpf_strncasestr);
#if defined(CONFIG_BPF_LSM) && defined(CONFIG_CGROUPS)
BTF_ID_FLAGS(func, bpf_cgroup_read_xattr, KF_RCU)
#endif
BTF_ID_FLAGS(func, bpf_stream_vprintk, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_dynptr_from_file)
BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
BTF_ID_FLAGS(func, bpf_timer_cancel_async)
BTF_KFUNCS_END(common_btf_ids)
static const struct btf_kfunc_id_set common_kfunc_set = {
.owner = THIS_MODULE,
.set = &common_btf_ids,
};
static int __init kfunc_init(void)
{
int ret;
const struct btf_id_dtor_kfunc generic_dtors[] = {
{
.btf_id = generic_dtor_ids[0],
.kfunc_btf_id = generic_dtor_ids[1]
},
#ifdef CONFIG_CGROUPS
{
.btf_id = generic_dtor_ids[2],
.kfunc_btf_id = generic_dtor_ids[3]
},
#endif
};
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &generic_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SKB, &generic_kfunc_set);
ret = ret ?: register_btf_id_dtor_kfuncs(generic_dtors,
ARRAY_SIZE(generic_dtors),
THIS_MODULE);
return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set);
}
late_initcall(kfunc_init);
/* Get a pointer to dynptr data up to len bytes for read only access. If
* the dynptr doesn't have continuous data up to len bytes, return NULL.
*/
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u64 len)
{
const struct bpf_dynptr *p = (struct bpf_dynptr *)ptr;
return bpf_dynptr_slice(p, 0, NULL, len);
}
/* Get a pointer to dynptr data up to len bytes for read write access. If
* the dynptr doesn't have continuous data up to len bytes, or the dynptr
* is read only, return NULL.
*/
void *__bpf_dynptr_data_rw(const struct bpf_dynptr_kern *ptr, u64 len)
{
if (__bpf_dynptr_is_rdonly(ptr))
return NULL;
return (void *)__bpf_dynptr_data(ptr, len);
}
void bpf_map_free_internal_structs(struct bpf_map *map, void *val)
{
if (btf_record_has_field(map->record, BPF_TIMER))
bpf_obj_free_timer(map->record, val);
if (btf_record_has_field(map->record, BPF_WORKQUEUE))
bpf_obj_free_workqueue(map->record, val);
if (btf_record_has_field(map->record, BPF_TASK_WORK))
bpf_obj_free_task_work(map->record, val);
}
]
|
| 649/6 |
2026/05/18 20:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 58311
output: 810
thoughts: 607
Thoughts: **My Analysis of `bpf_snprintf` Implementation**
Okay, I've got the `kernel/bpf/helpers.c` content in front of me. The user specifically wants the implementation of `BPF_CALL_5(bpf_snprintf, ...)`, so let's get right to it. It starts at line 1064, and here's the code block:
```c
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, args, u32, data_len)
{
struct bpf_bprintf_data data = {
.get_bin_args = true,
};
int err, num_args;
if (data_len % 8 || data_len > MAX_BPRINTF_VARARGS * 8 ||
(data_len && !args))
return -EINVAL;
num_args = data_len / 8;
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
err = bpf_bprintf_prepare(fmt, UINT_MAX, args, num_args, &data);
if (err < 0)
return err;
err = bstr_printf(str, str_size, fmt, data.bin_args);
bpf_bprintf_cleanup(&data);
return err + 1;
}
```
Alright, so it's a `BPF_CALL_5` macro, meaning it's a BPF helper function that takes five arguments. The arguments are, in order: `str` (a character pointer for the output string), `str_size` (the size of that output buffer), `fmt` (the format string), `args` (a pointer to the variable arguments), and `data_len` (the size of the variable arguments data).
Inside, it initializes a `bpf_bprintf_data` struct and checks the validity of `data_len` and the `args` pointer. Specifically, it validates that `data_len` is a multiple of 8, within a maximum size, and that `args` is not `NULL` if `data_len` is not zero. A key point is the comment regarding `ARG_PTR_TO_CONST_STR` guaranteeing `fmt` is zero-terminated. This is crucial for security.
The core logic then calls `bpf_bprintf_prepare` to format the argument data, then `bstr_printf` to perform the actual formatted string write. Finally, it cleans up with `bpf_bprintf_cleanup` and returns the result (plus 1).
I should also include the `bpf_snprintf_proto` definition, which specifies the types and properties of the arguments. This is critical for BPF verifier.
```c
const struct bpf_func_proto bpf_snprintf_proto = {
.func = bpf_snprintf,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
.arg3_type = ARG_PTR_TO_CONST_STR,
.arg4_type = ARG_PTR_TO_MEM | PTR_MAYBE_NULL | MEM_RDONLY,
.arg5_type = ARG_CONST_SIZE_OR_ZERO,
};
```
This `bpf_snprintf_proto` is very useful when working on verifiers and BPF bytecode. It specifies that the function is GPL-only, returns an integer, and defines the type of each argument. For example, `arg1_type` is `ARG_PTR_TO_MEM_OR_NULL | MEM_WRITE`, indicating that the first argument (`str`) is a pointer to memory and is writeable, potentially NULL. The `arg3_type` of `ARG_PTR_TO_CONST_STR` is important, because this tells the verifier that the `fmt` string is safe to use.
Let's quickly check. Okay, both `bpf_bprintf_prepare` and `bstr_printf` are also in `kernel/bpf/helpers.c`, so I'll include them if asked.
|
| 650/4 |
2026/05/18 20:24 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 35348
output: 34
thoughts: 472
Thoughts: **Understanding the Error Handling in `bpf_snprintf`**
Okay, let's break down this `bpf_snprintf` process and where things can go wrong. I see that `bpf_snprintf` calls `bpf_bprintf_prepare` with `data.get_bin_args = true`. The critical thing here is that if `bpf_bprintf_prepare` returns `-ENOSPC`, `bpf_snprintf` *also* returns `-ENOSPC`. Now, the crucial question: is this expected behavior? Does `bpf_snprintf` *really* return `-ENOSPC` if the *internal* buffer, `bin_args`, is full?
Yes, that's it! The `bin_args` buffer has a hard limit of 512 bytes (`MAX_BPRINTF_BIN_ARGS`). If a BPF program passes arguments that, when processed, need *more* than 512 bytes within `bin_args`, then `bpf_bprintf_prepare` returns `-ENOSPC`. This **isn't** the same as the output string, `str`, being truncated! The output string `str` is formatted separately by `bstr_printf(str, str_size, fmt, data.bin_args)`. If `str_size` is too small, `bstr_printf` truncates the output, and returns the *would-be* written character count, and `bpf_snprintf` returns `err + 1`.
So, `-ENOSPC` from `bpf_bprintf_prepare` means the *internal* `bin_args` buffer is overflowing. That's a key distinction.
Now, a puzzle piece: the `%pB` specifier. Here, `snprintf` is used to format the argument into `bin_args`. Specifically:
```c
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
```
Wait, `%pB` uses `snprintf` to format to `tmp_buf` (which IS `bin_args` in this context), and *then* `bstr_printf` will come in to read it? That's unusual...and wait! Doesn't `bstr_printf` not know about `%pB`? It doesn't seem to...Let's go check the implementation in `lib/vsprintf.c` to be sure.
|
| 651/4 |
2026/05/18 20:24 |
tool |
codeexpert |
1m
Args: map[Question:How does bstr_printf handle %pB? Does it expect a pre-formatted string?]
Results: map[Answer:`bstr_printf` handles the `%pB` format specifier by expecting a **pre-formatted, null-terminated string** in the binary buffer (`bin_buf`).
### Detailed Explanation
`bstr_printf` is designed to be used in conjunction with `vbin_printf`. The process works in two stages:
1. **`vbin_printf` (Binary Packing Stage):**
When `vbin_printf` encounters a `%p` specifier, it decides whether to save the raw pointer or to format it immediately into a string. For most `%p` extensions (like `%pM` for MAC addresses, `%pI` for IP addresses, and `%pB` for backtrace symbols), it performs the formatting immediately.
In `lib/vsprintf.c`, the `vbin_printf` function has a switch statement for pointer formats:
```c
case FORMAT_STATE_PTR:
/* Dereferenced pointers must be done now */
switch (*fmt.str) {
/* Dereference of functions is still OK */
case 'S':
case 's':
case 'x':
case 'K':
case 'e':
save_arg(void *);
break;
default:
if (!isalnum(*fmt.str)) {
save_arg(void *);
break;
}
// For %pB, it falls through to here
str = pointer(fmt.str, str, end, va_arg(args, void *), spec);
if (str + 1 < end)
*str++ = '\0';
else
end[-1] = '\0'; /* Must be nul terminated */
}
```
Since `'B'` is alphanumeric and not in the "deferred" list (`S`, `s`, `x`, `K`, `e`), `vbin_printf` calls the `pointer()` function immediately, which resolves the address to a symbol string (using `sprint_backtrace`) and stores that string in the binary buffer, followed by a null terminator.
2. **`bstr_printf` (Text Formatting Stage):**
When `bstr_printf` later processes the binary buffer, it sees the `%pB` specifier and checks its own "deferred" list:
```c
case FORMAT_STATE_PTR: {
bool process = false;
int copy, len;
/* Non function dereferences were already done */
switch (*fmt.str) {
case 'S':
case 's':
case 'x':
case 'K':
case 'e':
process = true;
break;
default:
if (!isalnum(*fmt.str)) {
process = true;
break;
}
/* Pointer dereference was already processed */
if (str < end) {
len = copy = strlen(args);
if (copy > end - str)
copy = end - str;
memcpy(str, args, copy);
str += len;
args += len + 1;
}
}
if (process)
str = pointer(fmt.str, str, end, get_arg(void *), spec);
```
For `%pB`, `process` remains `false`. `bstr_printf` treats the current position in the binary buffer (`args`) as a string, calculates its length, and copies it directly to the output buffer.
### Conclusion
`bstr_printf` **does expect a pre-formatted string** for `%pB`. It does not receive the raw pointer; instead, it receives the result of the symbol lookup performed earlier by `vbin_printf`. This is different from `%pS` or `%ps`, which are deferred and resolved at the time `bstr_printf` is called.]
|
| 652/5 |
2026/05/18 20:24 |
agent |
codeexpert |
1m
Model:
Instruction:
You are a capable Linux kernel developer tasked with researching complex questions
about kernel source code. You will be given a concrete question, and need to provide
a concrete answer.
Use tools extensively while researching the question. Don't make assumptions,
or rely on your previous knowledge about the kernel source code, use available tools
to access the actual source code.
Use all available sources of information:
- kernel source code
- documentation in the Documentation dir in the source tree
- git commits descriptions, git blame
Prefer calling several tools at the same time to save round-trips.
Prompt: How does bstr_printf handle %pB? Does it expect a pre-formatted string?
Reply: `bstr_printf` handles the `%pB` format specifier by expecting a **pre-formatted, null-terminated string** in the binary buffer (`bin_buf`).
### Detailed Explanation
`bstr_printf` is designed to be used in conjunction with `vbin_printf`. The process works in two stages:
1. **`vbin_printf` (Binary Packing Stage):**
When `vbin_printf` encounters a `%p` specifier, it decides whether to save the raw pointer or to format it immediately into a string. For most `%p` extensions (like `%pM` for MAC addresses, `%pI` for IP addresses, and `%pB` for backtrace symbols), it performs the formatting immediately.
In `lib/vsprintf.c`, the `vbin_printf` function has a switch statement for pointer formats:
```c
case FORMAT_STATE_PTR:
/* Dereferenced pointers must be done now */
switch (*fmt.str) {
/* Dereference of functions is still OK */
case 'S':
case 's':
case 'x':
case 'K':
case 'e':
save_arg(void *);
break;
default:
if (!isalnum(*fmt.str)) {
save_arg(void *);
break;
}
// For %pB, it falls through to here
str = pointer(fmt.str, str, end, va_arg(args, void *), spec);
if (str + 1 < end)
*str++ = '\0';
else
end[-1] = '\0'; /* Must be nul terminated */
}
```
Since `'B'` is alphanumeric and not in the "deferred" list (`S`, `s`, `x`, `K`, `e`), `vbin_printf` calls the `pointer()` function immediately, which resolves the address to a symbol string (using `sprint_backtrace`) and stores that string in the binary buffer, followed by a null terminator.
2. **`bstr_printf` (Text Formatting Stage):**
When `bstr_printf` later processes the binary buffer, it sees the `%pB` specifier and checks its own "deferred" list:
```c
case FORMAT_STATE_PTR: {
bool process = false;
int copy, len;
/* Non function dereferences were already done */
switch (*fmt.str) {
case 'S':
case 's':
case 'x':
case 'K':
case 'e':
process = true;
break;
default:
if (!isalnum(*fmt.str)) {
process = true;
break;
}
/* Pointer dereference was already processed */
if (str < end) {
len = copy = strlen(args);
if (copy > end - str)
copy = end - str;
memcpy(str, args, copy);
str += len;
args += len + 1;
}
}
if (process)
str = pointer(fmt.str, str, end, get_arg(void *), spec);
```
For `%pB`, `process` remains `false`. `bstr_printf` treats the current position in the binary buffer (`args`) as a string, calculates its length, and copies it directly to the output buffer.
### Conclusion
`bstr_printf` **does expect a pre-formatted string** for `%pB`. It does not receive the raw pointer; instead, it receives the result of the symbol lookup performed earlier by `vbin_printf`. This is different from `%pS` or `%ps`, which are deferred and resolved at the time `bstr_printf` is called.
|
| 653/6 |
2026/05/18 20:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 2272
output: 22
thoughts: 53
Thoughts: Okay, here's my summary, taking on the persona you requested:
**Investigating `bstr_printf` and `%pB`**
Alright, let's dive into this. I'm curious about the internals of `bstr_printf` and, more specifically, how it's dealing with the `%pB` format specifier. My main question is: Does `bstr_printf` anticipate receiving a pre-formatted string when it encounters `%pB`, or does it perform some internal formatting itself? I need to understand this to ensure proper use of the function and avoid any unexpected behavior or potential bugs. Time to jump into the kernel source and start searching for `bstr_printf`. Hopefully, the implementation details are clear enough to shed light on this.
|
| 654/6 |
2026/05/18 20:24 |
tool |
codesearch-find-references |
0m
Args:
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:____bpf_snprintf SourceFile:kernel/bpf/helpers.c SourceLine:1084] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_stream_vprintk SourceFile:kernel/bpf/stream.c SourceLine:240] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:seq_bprintf SourceFile:fs/seq_file.c SourceLine:418] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:seq_buf_bprintf SourceFile:lib/seq_buf.c SourceLine:162]] TruncatedOutput:false]
|
| 655/6 |
2026/05/18 20:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 2732
output: 39
thoughts: 46
Thoughts: **Finding the Definition of `bstr_printf`**
Okay, so I'm looking at this code and I'm seeing `bstr_printf` popping up in a few key areas. My initial thought is, I need to understand this function, especially its implementation, before I go any further with my analysis. I'm going to leverage `codesearch-definition-source`. I'll use it to pinpoint exactly where this `bstr_printf` function is defined within the codebase. That's my first priority: locate the source and get a handle on what it's actually doing. Once I have the source code, I can review it in detail.
|
| 656/6 |
2026/05/18 20:24 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/vsprintf.c IncludeLines:true Name:bstr_printf]
Results: map[SourceCode:3260: /**
3261: * bstr_printf - Format a string from binary arguments and place it in a buffer
3262: * @buf: The buffer to place the result into
3263: * @size: The size of the buffer, including the trailing null space
3264: * @fmt_str: The format string to use
3265: * @bin_buf: Binary arguments for the format string
3266: *
3267: * This function like C99 vsnprintf, but the difference is that vsnprintf gets
3268: * arguments from stack, and bstr_printf gets arguments from @bin_buf which is
3269: * a binary buffer that generated by vbin_printf.
3270: *
3271: * The format follows C99 vsnprintf, but has some extensions:
3272: * see vsnprintf comment for details.
3273: *
3274: * The return value is the number of characters which would
3275: * be generated for the given input, excluding the trailing
3276: * '\0', as per ISO C99. If you want to have the exact
3277: * number of characters written into @buf as return value
3278: * (not including the trailing '\0'), use vscnprintf(). If the
3279: * return is greater than or equal to @size, the resulting
3280: * string is truncated.
3281: */
3282: int bstr_printf(char *buf, size_t size, const char *fmt_str, const u32 *bin_buf)
3283: {
3284: struct fmt fmt = {
3285: .str = fmt_str,
3286: .state = FORMAT_STATE_NONE,
3287: };
3288: struct printf_spec spec = {0};
3289: char *str, *end;
3290: const char *args = (const char *)bin_buf;
3291:
3292: if (WARN_ON_ONCE(size > INT_MAX))
3293: return 0;
3294:
3295: str = buf;
3296: end = buf + size;
3297:
3298: #define get_arg(type) \
3299: ({ \
3300: typeof(type) value; \
3301: if (sizeof(type) == 8) { \
3302: args = PTR_ALIGN(args, sizeof(u32)); \
3303: *(u32 *)&value = *(u32 *)args; \
3304: *((u32 *)&value + 1) = *(u32 *)(args + 4); \
3305: } else { \
3306: args = PTR_ALIGN(args, sizeof(type)); \
3307: value = *(typeof(type) *)args; \
3308: } \
3309: args += sizeof(type); \
3310: value; \
3311: })
3312:
3313: /* Make sure end is always >= buf */
3314: if (end < buf) {
3315: end = ((void *)-1);
3316: size = end - buf;
3317: }
3318:
3319: while (*fmt.str) {
3320: const char *old_fmt = fmt.str;
3321: unsigned long long num;
3322:
3323: fmt = format_decode(fmt, &spec);
3324: switch (fmt.state) {
3325: case FORMAT_STATE_NONE: {
3326: int read = fmt.str - old_fmt;
3327: if (str < end) {
3328: int copy = read;
3329: if (copy > end - str)
3330: copy = end - str;
3331: memcpy(str, old_fmt, copy);
3332: }
3333: str += read;
3334: continue;
3335: }
3336:
3337: case FORMAT_STATE_WIDTH:
3338: set_field_width(&spec, get_arg(int));
3339: continue;
3340:
3341: case FORMAT_STATE_PRECISION:
3342: set_precision(&spec, get_arg(int));
3343: continue;
3344:
3345: case FORMAT_STATE_CHAR: {
3346: char c;
3347:
3348: if (!(spec.flags & LEFT)) {
3349: while (--spec.field_width > 0) {
3350: if (str < end)
3351: *str = ' ';
3352: ++str;
3353: }
3354: }
3355: c = (unsigned char) get_arg(char);
3356: if (str < end)
3357: *str = c;
3358: ++str;
3359: while (--spec.field_width > 0) {
3360: if (str < end)
3361: *str = ' ';
3362: ++str;
3363: }
3364: continue;
3365: }
3366:
3367: case FORMAT_STATE_STR: {
3368: const char *str_arg = args;
3369: args += strlen(str_arg) + 1;
3370: str = string(str, end, (char *)str_arg, spec);
3371: continue;
3372: }
3373:
3374: case FORMAT_STATE_PTR: {
3375: bool process = false;
3376: int copy, len;
3377: /* Non function dereferences were already done */
3378: switch (*fmt.str) {
3379: case 'S':
3380: case 's':
3381: case 'x':
3382: case 'K':
3383: case 'e':
3384: process = true;
3385: break;
3386: default:
3387: if (!isalnum(*fmt.str)) {
3388: process = true;
3389: break;
3390: }
3391: /* Pointer dereference was already processed */
3392: if (str < end) {
3393: len = copy = strlen(args);
3394: if (copy > end - str)
3395: copy = end - str;
3396: memcpy(str, args, copy);
3397: str += len;
3398: args += len + 1;
3399: }
3400: }
3401: if (process)
3402: str = pointer(fmt.str, str, end, get_arg(void *), spec);
3403:
3404: while (isalnum(*fmt.str))
3405: fmt.str++;
3406: continue;
3407: }
3408:
3409: case FORMAT_STATE_PERCENT_CHAR:
3410: if (str < end)
3411: *str = '%';
3412: ++str;
3413: continue;
3414:
3415: case FORMAT_STATE_INVALID:
3416: goto out;
3417:
3418: case FORMAT_STATE_NUM:
3419: if (fmt.size > sizeof(int))
3420: num = get_arg(long long);
3421: else
3422: num = convert_num_spec(get_arg(int), fmt.size, spec);
3423: str = number(str, end, num, spec);
3424: continue;
3425: }
3426: } /* while(*fmt.str) */
3427:
3428: out:
3429: if (size > 0) {
3430: if (str < end)
3431: *str = '\0';
3432: else
3433: end[-1] = '\0';
3434: }
3435:
3436: #undef get_arg
3437:
3438: /* the trailing null byte doesn't count towards the total */
3439: return str - buf;
3440: }
SourceFile:lib/vsprintf.c]
|
| 657/6 |
2026/05/18 20:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 5409
output: 39
thoughts: 670
Thoughts: **Analyzing the `%p` Format Specifier in `bstr_printf`**
Okay, let's break down how `bstr_printf` handles the `%p` format specifier, specifically within the `FORMAT_STATE_PTR` case (lines 3374-3407). My initial thought is to trace the logic when a format like `%pB` is encountered.
I see the code first checks a few special cases: 'S', 's', 'x', 'K', and 'e'. If `*fmt.str` is one of those, `process` is set to `true`, and it moves to the `pointer()` function. However, with `%pB`, `*fmt.str` is 'B'. 'B' isn't in that list. It then hits the `default` case. Since 'B' is alphanumeric, `process` *remains* `false`.
Now here's where things get interesting. The code then comments, `/* Pointer dereference was already processed */`. The next section copies the content of `args` into `str`. This behavior seems odd, and it's essential to understand *why* the dereference might be considered pre-processed.
A key point is that `bstr_printf` is designed to work with `vbin_printf`. The comments indicate that while `vsnprintf` grabs arguments from the stack, `bstr_printf` receives them from a binary buffer generated by `vbin_printf`. This strongly suggests that `vbin_printf` is where the initial argument processing and dereferencing *actually* occur.
So, in the case of `%pB` in `bstr_printf`, the expectation is that `vbin_printf` has already handled the "pointer dereference." `bstr_printf` is just reading the processed result from the binary buffer and copying it to the output string. This confirms that the `%p` format specifier is part of a larger chain, with the prior function doing the heavy lifting. I should definitely check `vbin_printf` to see precisely how it interprets and transforms something like `%pB`.
|
| 658/6 |
2026/05/18 20:24 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/vsprintf.c IncludeLines:true Name:vbin_printf]
Results: map[SourceCode:3125: /**
3126: * vbin_printf - Parse a format string and place args' binary value in a buffer
3127: * @bin_buf: The buffer to place args' binary value
3128: * @size: The size of the buffer(by words(32bits), not characters)
3129: * @fmt_str: The format string to use
3130: * @args: Arguments for the format string
3131: *
3132: * The format follows C99 vsnprintf, except %n is ignored, and its argument
3133: * is skipped.
3134: *
3135: * The return value is the number of words(32bits) which would be generated for
3136: * the given input.
3137: *
3138: * NOTE:
3139: * If the return value is greater than @size, the resulting bin_buf is NOT
3140: * valid for bstr_printf().
3141: */
3142: int vbin_printf(u32 *bin_buf, size_t size, const char *fmt_str, va_list args)
3143: {
3144: struct fmt fmt = {
3145: .str = fmt_str,
3146: .state = FORMAT_STATE_NONE,
3147: };
3148: struct printf_spec spec = {0};
3149: char *str, *end;
3150: int width;
3151:
3152: str = (char *)bin_buf;
3153: end = (char *)(bin_buf + size);
3154:
3155: #define save_arg(type) \
3156: ({ \
3157: unsigned long long value; \
3158: if (sizeof(type) == 8) { \
3159: unsigned long long val8; \
3160: str = PTR_ALIGN(str, sizeof(u32)); \
3161: val8 = va_arg(args, unsigned long long); \
3162: if (str + sizeof(type) <= end) { \
3163: *(u32 *)str = *(u32 *)&val8; \
3164: *(u32 *)(str + 4) = *((u32 *)&val8 + 1); \
3165: } \
3166: value = val8; \
3167: } else { \
3168: unsigned int val4; \
3169: str = PTR_ALIGN(str, sizeof(type)); \
3170: val4 = va_arg(args, int); \
3171: if (str + sizeof(type) <= end) \
3172: *(typeof(type) *)str = (type)(long)val4; \
3173: value = (unsigned long long)val4; \
3174: } \
3175: str += sizeof(type); \
3176: value; \
3177: })
3178:
3179: while (*fmt.str) {
3180: fmt = format_decode(fmt, &spec);
3181:
3182: switch (fmt.state) {
3183: case FORMAT_STATE_NONE:
3184: case FORMAT_STATE_PERCENT_CHAR:
3185: break;
3186: case FORMAT_STATE_INVALID:
3187: goto out;
3188:
3189: case FORMAT_STATE_WIDTH:
3190: case FORMAT_STATE_PRECISION:
3191: width = (int)save_arg(int);
3192: /* Pointers may require the width */
3193: if (*fmt.str == 'p')
3194: set_field_width(&spec, width);
3195: break;
3196:
3197: case FORMAT_STATE_CHAR:
3198: save_arg(char);
3199: break;
3200:
3201: case FORMAT_STATE_STR: {
3202: const char *save_str = va_arg(args, char *);
3203: const char *err_msg;
3204: size_t len;
3205:
3206: err_msg = check_pointer_msg(save_str);
3207: if (err_msg)
3208: save_str = err_msg;
3209:
3210: len = strlen(save_str) + 1;
3211: if (str + len < end)
3212: memcpy(str, save_str, len);
3213: str += len;
3214: break;
3215: }
3216:
3217: case FORMAT_STATE_PTR:
3218: /* Dereferenced pointers must be done now */
3219: switch (*fmt.str) {
3220: /* Dereference of functions is still OK */
3221: case 'S':
3222: case 's':
3223: case 'x':
3224: case 'K':
3225: case 'e':
3226: save_arg(void *);
3227: break;
3228: default:
3229: if (!isalnum(*fmt.str)) {
3230: save_arg(void *);
3231: break;
3232: }
3233: str = pointer(fmt.str, str, end, va_arg(args, void *),
3234: spec);
3235: if (str + 1 < end)
3236: *str++ = '\0';
3237: else
3238: end[-1] = '\0'; /* Must be nul terminated */
3239: }
3240: /* skip all alphanumeric pointer suffixes */
3241: while (isalnum(*fmt.str))
3242: fmt.str++;
3243: break;
3244:
3245: case FORMAT_STATE_NUM:
3246: if (fmt.size > sizeof(int)) {
3247: save_arg(long long);
3248: } else {
3249: save_arg(int);
3250: }
3251: }
3252: }
3253:
3254: out:
3255: return (u32 *)(PTR_ALIGN(str, sizeof(u32))) - bin_buf;
3256: #undef save_arg
3257: }
SourceFile:lib/vsprintf.c]
|
| 659/6 |
2026/05/18 20:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 8472
output: 36
thoughts: 554
Thoughts: **Analyzing the vbin_printf Pointer Formatting**
Okay, so I'm looking at how `vbin_printf` handles pointers, specifically within the `FORMAT_STATE_PTR` case. The code dereferences pointers, and I'm focusing on how it deals with different pointer formats.
It checks the character pointed to by `fmt.str` to figure out the right formatting path. If the character is one of 'S', 's', 'x', 'K', or 'e', it uses `save_arg(void *)`. Otherwise, if the character is alphanumeric, and *not* in that "OK" list, it calls the `pointer()` function. This is where things get interesting. Since `%pB` gives us the character 'B', which is alphanumeric, and not in the "OK" list, it's going down the default path which calls `pointer()`.
The `pointer()` function is then responsible for the actual formatting of the pointer, based on the specifier. It seems that `vbin_printf` is pre-formatting the pointer into a string, and stores the result in a buffer. Finally, it null-terminates the string.
This implies that `%pB` is going to use `pointer()` to transform the pointer value somehow, and it's not simply passing it directly to a system call. I need to check what `%pB` actually *does* though. What is the specifier for 'B'? I should look up the documentation for `printk` formats to understand the expected behavior of `%pB`. I can also quickly peek at the implementation of the `pointer()` function, located in `lib/vsprintf.c`, to figure out what happens.
|
| 660/6 |
2026/05/18 20:24 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/vsprintf.c IncludeLines:true Name:pointer]
Results: map[SourceCode:2388: /*
2389: * Show a '%p' thing. A kernel extension is that the '%p' is followed
2390: * by an extra set of alphanumeric characters that are extended format
2391: * specifiers.
2392: *
2393: * Please update scripts/checkpatch.pl when adding/removing conversion
2394: * characters. (Search for "check for vsprintf extension").
2395: *
2396: * Right now we handle:
2397: *
2398: * - 'S' For symbolic direct pointers (or function descriptors) with offset
2399: * - 's' For symbolic direct pointers (or function descriptors) without offset
2400: * - '[Ss]R' as above with __builtin_extract_return_addr() translation
2401: * - 'S[R]b' as above with module build ID (for use in backtraces)
2402: * - '[Ff]' %pf and %pF were obsoleted and later removed in favor of
2403: * %ps and %pS. Be careful when re-using these specifiers.
2404: * - 'B' For backtraced symbolic direct pointers with offset
2405: * - 'Bb' as above with module build ID (for use in backtraces)
2406: * - 'R' For decoded struct resource, e.g., [mem 0x0-0x1f 64bit pref]
2407: * - 'r' For raw struct resource, e.g., [mem 0x0-0x1f flags 0x201]
2408: * - 'ra' For struct ranges, e.g., [range 0x0000000000000000 - 0x00000000000000ff]
2409: * - 'b[l]' For a bitmap, the number of bits is determined by the field
2410: * width which must be explicitly specified either as part of the
2411: * format string '%32b[l]' or through '%*b[l]', [l] selects
2412: * range-list format instead of hex format
2413: * - 'M' For a 6-byte MAC address, it prints the address in the
2414: * usual colon-separated hex notation
2415: * - 'm' For a 6-byte MAC address, it prints the hex address without colons
2416: * - 'MF' For a 6-byte MAC FDDI address, it prints the address
2417: * with a dash-separated hex notation
2418: * - '[mM]R' For a 6-byte MAC address, Reverse order (Bluetooth)
2419: * - 'I' [46] for IPv4/IPv6 addresses printed in the usual way
2420: * IPv4 uses dot-separated decimal without leading 0's (1.2.3.4)
2421: * IPv6 uses colon separated network-order 16 bit hex with leading 0's
2422: * [S][pfs]
2423: * Generic IPv4/IPv6 address (struct sockaddr *) that falls back to
2424: * [4] or [6] and is able to print port [p], flowinfo [f], scope [s]
2425: * - 'i' [46] for 'raw' IPv4/IPv6 addresses
2426: * IPv6 omits the colons (01020304...0f)
2427: * IPv4 uses dot-separated decimal with leading 0's (010.123.045.006)
2428: * [S][pfs]
2429: * Generic IPv4/IPv6 address (struct sockaddr *) that falls back to
2430: * [4] or [6] and is able to print port [p], flowinfo [f], scope [s]
2431: * - '[Ii][4S][hnbl]' IPv4 addresses in host, network, big or little endian order
2432: * - 'I[6S]c' for IPv6 addresses printed as specified by
2433: * https://tools.ietf.org/html/rfc5952
2434: * - 'E[achnops]' For an escaped buffer, where rules are defined by combination
2435: * of the following flags (see string_escape_mem() for the
2436: * details):
2437: * a - ESCAPE_ANY
2438: * c - ESCAPE_SPECIAL
2439: * h - ESCAPE_HEX
2440: * n - ESCAPE_NULL
2441: * o - ESCAPE_OCTAL
2442: * p - ESCAPE_NP
2443: * s - ESCAPE_SPACE
2444: * By default ESCAPE_ANY_NP is used.
2445: * - 'U' For a 16 byte UUID/GUID, it prints the UUID/GUID in the form
2446: * "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
2447: * Options for %pU are:
2448: * b big endian lower case hex (default)
2449: * B big endian UPPER case hex
2450: * l little endian lower case hex
2451: * L little endian UPPER case hex
2452: * big endian output byte order is:
2453: * [0][1][2][3]-[4][5]-[6][7]-[8][9]-[10][11][12][13][14][15]
2454: * little endian output byte order is:
2455: * [3][2][1][0]-[5][4]-[7][6]-[8][9]-[10][11][12][13][14][15]
2456: * - 'V' For a struct va_format which contains a format string * and va_list *,
2457: * call vsnprintf(->format, *->va_list).
2458: * Implements a "recursive vsnprintf".
2459: * Do not use this feature without some mechanism to verify the
2460: * correctness of the format string and va_list arguments.
2461: * - 'K' For a kernel pointer that should be hidden from unprivileged users.
2462: * Use only for procfs, sysfs and similar files, not printk(); please
2463: * read the documentation (path below) first.
2464: * - 'NF' For a netdev_features_t
2465: * - '4cc' V4L2 or DRM FourCC code, with endianness and raw numerical value.
2466: * - '4c[h[R]lb]' For generic FourCC code with raw numerical value. Both are
2467: * displayed in the big-endian format. This is the opposite of V4L2 or
2468: * DRM FourCCs.
2469: * The additional specifiers define what endianness is used to load
2470: * the stored bytes. The data might be interpreted using the host,
2471: * reversed host byte order, little-endian, or big-endian.
2472: * - 'h[CDN]' For a variable-length buffer, it prints it as a hex string with
2473: * a certain separator (' ' by default):
2474: * C colon
2475: * D dash
2476: * N no separator
2477: * The maximum supported length is 64 bytes of the input. Consider
2478: * to use print_hex_dump() for the larger input.
2479: * - 'a[pd]' For address types [p] phys_addr_t, [d] dma_addr_t and derivatives
2480: * (default assumed to be phys_addr_t, passed by reference)
2481: * - 'd[234]' For a dentry name (optionally 2-4 last components)
2482: * - 'D[234]' Same as 'd' but for a struct file
2483: * - 'g' For block_device name (gendisk + partition number)
2484: * - 't[RST][dt][r][s]' For time and date as represented by:
2485: * R struct rtc_time
2486: * S struct timespec64
2487: * T time64_t
2488: * - 'tSp' For time represented by struct timespec64 printed as <seconds>.<nanoseconds>
2489: * - 'C' For a clock, it prints the name (Common Clock Framework) or address
2490: * (legacy clock framework) of the clock
2491: * - 'G' For flags to be printed as a collection of symbolic strings that would
2492: * construct the specific value. Supported flags given by option:
2493: * p page flags (see struct page) given as pointer to unsigned long
2494: * g gfp flags (GFP_* and __GFP_*) given as pointer to gfp_t
2495: * v vma flags (VM_*) given as pointer to unsigned long
2496: * - 'OF[fnpPcCF]' For a device tree object
2497: * Without any optional arguments prints the full_name
2498: * f device node full_name
2499: * n device node name
2500: * p device node phandle
2501: * P device node path spec (name + @unit)
2502: * F device node flags
2503: * c major compatible string
2504: * C full compatible string
2505: * - 'fw[fP]' For a firmware node (struct fwnode_handle) pointer
2506: * Without an option prints the full name of the node
2507: * f full name
2508: * P node name, including a possible unit address
2509: * - 'x' For printing the address unmodified. Equivalent to "%lx".
2510: * Please read the documentation (path below) before using!
2511: * - '[ku]s' For a BPF/tracing related format specifier, e.g. used out of
2512: * bpf_trace_printk() where [ku] prefix specifies either kernel (k)
2513: * or user (u) memory to probe, and:
2514: * s a string, equivalent to "%s" on direct vsnprintf() use
2515: *
2516: * ** When making changes please also update:
2517: * Documentation/core-api/printk-formats.rst
2518: *
2519: * Note: The default behaviour (unadorned %p) is to hash the address,
2520: * rendering it useful as a unique identifier.
2521: *
2522: * There is also a '%pA' format specifier, but it is only intended to be used
2523: * from Rust code to format core::fmt::Arguments. Do *not* use it from C.
2524: * See rust/kernel/print.rs for details.
2525: */
2526: static noinline_for_stack
2527: char *pointer(const char *fmt, char *buf, char *end, void *ptr,
2528: struct printf_spec spec)
2529: {
2530: switch (*fmt) {
2531: case 'S':
2532: case 's':
2533: ptr = dereference_symbol_descriptor(ptr);
2534: fallthrough;
2535: case 'B':
2536: return symbol_string(buf, end, ptr, spec, fmt);
2537: case 'R':
2538: case 'r':
2539: return resource_or_range(fmt, buf, end, ptr, spec);
2540: case 'h':
2541: return hex_string(buf, end, ptr, spec, fmt);
2542: case 'b':
2543: switch (fmt[1]) {
2544: case 'l':
2545: return bitmap_list_string(buf, end, ptr, spec, fmt);
2546: default:
2547: return bitmap_string(buf, end, ptr, spec, fmt);
2548: }
2549: case 'M': /* Colon separated: 00:01:02:03:04:05 */
2550: case 'm': /* Contiguous: 000102030405 */
2551: /* [mM]F (FDDI) */
2552: /* [mM]R (Reverse order; Bluetooth) */
2553: return mac_address_string(buf, end, ptr, spec, fmt);
2554: case 'I': /* Formatted IP supported
2555: * 4: 1.2.3.4
2556: * 6: 0001:0203:...:0708
2557: * 6c: 1::708 or 1::1.2.3.4
2558: */
2559: case 'i': /* Contiguous:
2560: * 4: 001.002.003.004
2561: * 6: 000102...0f
2562: */
2563: return ip_addr_string(buf, end, ptr, spec, fmt);
2564: case 'E':
2565: return escaped_string(buf, end, ptr, spec, fmt);
2566: case 'U':
2567: return uuid_string(buf, end, ptr, spec, fmt);
2568: case 'V':
2569: return va_format(buf, end, ptr, spec);
2570: case 'K':
2571: return restricted_pointer(buf, end, ptr, spec);
2572: case 'N':
2573: return netdev_bits(buf, end, ptr, spec, fmt);
2574: case '4':
2575: return fourcc_string(buf, end, ptr, spec, fmt);
2576: case 'a':
2577: return address_val(buf, end, ptr, spec, fmt);
2578: case 'd':
2579: return dentry_name(buf, end, ptr, spec, fmt);
2580: case 't':
2581: return time_and_date(buf, end, ptr, spec, fmt);
2582: case 'C':
2583: return clock(buf, end, ptr, spec, fmt);
2584: case 'D':
2585: return file_dentry_name(buf, end, ptr, spec, fmt);
2586: #ifdef CONFIG_BLOCK
2587: case 'g':
2588: return bdev_name(buf, end, ptr, spec, fmt);
2589: #endif
2590:
2591: case 'G':
2592: return flags_string(buf, end, ptr, spec, fmt);
2593: case 'O':
2594: return device_node_string(buf, end, ptr, spec, fmt + 1);
2595: case 'f':
2596: return fwnode_string(buf, end, ptr, spec, fmt + 1);
2597: case 'A':
2598: if (!IS_ENABLED(CONFIG_RUST)) {
2599: WARN_ONCE(1, "Please remove %%pA from non-Rust code\n");
2600: return error_string(buf, end, "(%pA?)", spec);
2601: }
2602: return rust_fmt_argument(buf, end, ptr);
2603: case 'x':
2604: return pointer_string(buf, end, ptr, spec);
2605: case 'e':
2606: /* %pe with a non-ERR_PTR gets treated as plain %p */
2607: if (!IS_ERR(ptr))
2608: return default_pointer(buf, end, ptr, spec);
2609: return err_ptr(buf, end, ptr, spec);
2610: case 'u':
2611: case 'k':
2612: switch (fmt[1]) {
2613: case 's':
2614: return string(buf, end, ptr, spec);
2615: default:
2616: return error_string(buf, end, "(einval)", spec);
2617: }
2618: default:
2619: return default_pointer(buf, end, ptr, spec);
2620: }
2621: }
SourceFile:lib/vsprintf.c]
|
| 661/6 |
2026/05/18 20:24 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 14065
output: 21
thoughts: 2038
Thoughts: **Analyzing the `%pB` Format Specifier in `vbin_printf` and `bstr_printf`**
Okay, so I'm digging into how `%pB` is handled by `vbin_printf` and `bstr_printf`. I can see that `pointer()` function is at the heart of this. It's called by both functions, but the way they use it differs significantly.
In `vbin_printf`, when it encounters `%pB`, it goes through `pointer()` which then calls `symbol_string()`. This means that the symbol formatting is done *immediately* and the resulting string is stored in the binary buffer `bin_buf`. The code path is clear: The 'B' in the format string directs the control to execute `pointer()` at line 3233.
On the other hand, `bstr_printf` expects a pre-formatted string for `%pB`. It retrieves this formatted string from `bin_buf` at line 3391, which was already processed by `vbin_printf`.
This seems straightforward, but it raises questions. I'm used to `bstr_printf` deferring the formatting. In most cases, `vbin_printf` packs the arguments into a binary buffer, and `bstr_printf` later formats them. Extensions like `%pS` and `%ps` follow this deferred pattern. The key difference is that with `S` and `s` the program dereferences the symbol via `dereference_symbol_descriptor`, which is skipped for the `B` option.
Now I'm asking: Why is `%pB` treated differently? Why the immediate formatting?
Let's look at why `%pS` and `%ps` are "still OK". I look to line 3220, where 'S' and 's' are handled in a way that doesn't trigger the formatting in `vbin_printf`. This means that they are formatted later by `bstr_printf` at line 3402. So, for the `%pB` the code path skips this stage.
A key difference is that `%pB` is "backtraced symbolic direct pointers with offset", while `%pS` and `%ps` are "symbolic direct pointers (or function descriptors) with offset". This triggers a call to `dereference_symbol_descriptor()` function when formatting `pS` or `ps`, but not with the `B` option.
It seems the `vbin_printf` does the formatting right away for `%pB` *because* it *can* format it right away, while `%pS` may require context that is not available right away. This is because `%pB` might need special context that the `vbin_printf` function can use right away.
This makes me question if it was an oversight or if there's a specific reason. The most likely reason is the immediate formatting of `%pB` is because `vbin_printf` is often used in tracing where it wants to save the arguments as quickly as possible. Formatting a symbol by looking up the symbol name in kallsyms can be an expensive operation. So, perhaps the `B` option is different, where dereferencing of the symbol happens earlier, and the overhead is lower. The immediate formatting allows `vbin_printf` to be more efficient in capturing trace data.
I'm now going to use `git blame` to understand the code's evolution and see if any insights into these design decisions can be found.
|
| 662/6 |
2026/05/18 20:25 |
tool |
read-file |
0m
Args:
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/lib/vsprintf.c
*
* Copyright (C) 1991, 1992 Linus Torvalds
*/
/* vsprintf.c -- Lars Wirzenius & Linus Torvalds. */
/*
* Wirzenius wrote this portably, Torvalds fucked it up :-)
*/
/*
* Fri Jul 13 2001 Crutcher Dunnavant <crutcher+kernel@datastacks.com>
* - changed to provide snprintf and vsnprintf functions
* So Feb 1 16:51:32 CET 2004 Juergen Quade <quade@hsnr.de>
* - scnprintf and vscnprintf
*/
#include <linux/stdarg.h>
#include <linux/build_bug.h>
#include <linux/clk.h>
#include <linux/clk-provider.h>
#include <linux/errname.h>
#include <linux/module.h> /* for KSYM_SYMBOL_LEN */
#include <linux/types.h>
#include <linux/string.h>
#include <linux/ctype.h>
#include <linux/hex.h>
#include <linux/kernel.h>
#include <linux/kallsyms.h>
#include <linux/math64.h>
#include <linux/uaccess.h>
#include <linux/ioport.h>
#include <linux/dcache.h>
#include <linux/cred.h>
#include <linux/rtc.h>
#include <linux/sprintf.h>
#include <linux/time.h>
#include <linux/uuid.h>
#include <linux/of.h>
#include <net/addrconf.h>
#include <linux/siphash.h>
#include <linux/compiler.h>
#include <linux/property.h>
#include <linux/notifier.h>
#ifdef CONFIG_BLOCK
#include <linux/blkdev.h>
#endif
#include "../mm/internal.h" /* For the trace_print_flags arrays */
#include <asm/page.h> /* for PAGE_SIZE */
#include <asm/byteorder.h> /* cpu_to_le16 */
#include <linux/unaligned.h>
#include <linux/string_helpers.h>
#include "kstrtox.h"
/* Disable pointer hashing if requested */
bool no_hash_pointers __ro_after_init;
EXPORT_SYMBOL_GPL(no_hash_pointers);
/*
* Hashed pointers policy selected by "hash_pointers=..." boot param
*
* `auto` - Hashed pointers enabled unless disabled by slub_debug_enabled=true
* `always` - Hashed pointers enabled unconditionally
* `never` - Hashed pointers disabled unconditionally
*/
enum hash_pointers_policy {
HASH_PTR_AUTO = 0,
HASH_PTR_ALWAYS,
HASH_PTR_NEVER
};
static enum hash_pointers_policy hash_pointers_mode __initdata;
noinline
static unsigned long long simple_strntoull(const char *startp, char **endp, unsigned int base, size_t max_chars)
{
const char *cp;
unsigned long long result = 0ULL;
size_t prefix_chars;
unsigned int rv;
cp = _parse_integer_fixup_radix(startp, &base);
prefix_chars = cp - startp;
if (prefix_chars < max_chars) {
rv = _parse_integer_limit(cp, base, &result, max_chars - prefix_chars);
/* FIXME */
cp += (rv & ~KSTRTOX_OVERFLOW);
} else {
/* Field too short for prefix + digit, skip over without converting */
cp = startp + max_chars;
}
if (endp)
*endp = (char *)cp;
return result;
}
/**
* simple_strtoull - convert a string to an unsigned long long
* @cp: The start of the string
* @endp: A pointer to the end of the parsed string will be placed here
* @base: The number base to use
*
* This function has caveats. Please use kstrtoull instead.
*/
noinline
unsigned long long simple_strtoull(const char *cp, char **endp, unsigned int base)
{
return simple_strntoull(cp, endp, base, INT_MAX);
}
EXPORT_SYMBOL(simple_strtoull);
/**
* simple_strtoul - convert a string to an unsigned long
* @cp: The start of the string
* @endp: A pointer to the end of the parsed string will be placed here
* @base: The number base to use
*
* This function has caveats. Please use kstrtoul instead.
*/
unsigned long simple_strtoul(const char *cp, char **endp, unsigned int base)
{
return simple_strtoull(cp, endp, base);
}
EXPORT_SYMBOL(simple_strtoul);
unsigned long simple_strntoul(const char *cp, char **endp, unsigned int base,
size_t max_chars)
{
return simple_strntoull(cp, endp, base, max_chars);
}
EXPORT_SYMBOL(simple_strntoul);
/**
* simple_strtol - convert a string to a signed long
* @cp: The start of the string
* @endp: A pointer to the end of the parsed string will be placed here
* @base: The number base to use
*
* This function has caveats. Please use kstrtol instead.
*/
long simple_strtol(const char *cp, char **endp, unsigned int base)
{
if (*cp == '-')
return -simple_strtoul(cp + 1, endp, base);
return simple_strtoul(cp, endp, base);
}
EXPORT_SYMBOL(simple_strtol);
noinline
static long long simple_strntoll(const char *cp, char **endp, unsigned int base, size_t max_chars)
{
/*
* simple_strntoull() safely handles receiving max_chars==0 in the
* case cp[0] == '-' && max_chars == 1.
* If max_chars == 0 we can drop through and pass it to simple_strntoull()
* and the content of *cp is irrelevant.
*/
if (*cp == '-' && max_chars > 0)
return -simple_strntoull(cp + 1, endp, base, max_chars - 1);
return simple_strntoull(cp, endp, base, max_chars);
}
/**
* simple_strtoll - convert a string to a signed long long
* @cp: The start of the string
* @endp: A pointer to the end of the parsed string will be placed here
* @base: The number base to use
*
* This function has caveats. Please use kstrtoll instead.
*/
long long simple_strtoll(const char *cp, char **endp, unsigned int base)
{
return simple_strntoll(cp, endp, base, INT_MAX);
}
EXPORT_SYMBOL(simple_strtoll);
static inline int skip_atoi(const char **s)
{
int i = 0;
do {
i = i*10 + *((*s)++) - '0';
} while (isdigit(**s));
return i;
}
/*
* Decimal conversion is by far the most typical, and is used for
* /proc and /sys data. This directly impacts e.g. top performance
* with many processes running. We optimize it for speed by emitting
* two characters at a time, using a 200 byte lookup table. This
* roughly halves the number of multiplications compared to computing
* the digits one at a time. Implementation strongly inspired by the
* previous version, which in turn used ideas described at
* <http://www.cs.uiowa.edu/~jones/bcd/divide.html> (with permission
* from the author, Douglas W. Jones).
*
* It turns out there is precisely one 26 bit fixed-point
* approximation a of 64/100 for which x/100 == (x * (u64)a) >> 32
* holds for all x in [0, 10^8-1], namely a = 0x28f5c29. The actual
* range happens to be somewhat larger (x <= 1073741898), but that's
* irrelevant for our purpose.
*
* For dividing a number in the range [10^4, 10^6-1] by 100, we still
* need a 32x32->64 bit multiply, so we simply use the same constant.
*
* For dividing a number in the range [100, 10^4-1] by 100, there are
* several options. The simplest is (x * 0x147b) >> 19, which is valid
* for all x <= 43698.
*/
static const u16 decpair[100] = {
#define _(x) (__force u16) cpu_to_le16(((x % 10) | ((x / 10) << 8)) + 0x3030)
_( 0), _( 1), _( 2), _( 3), _( 4), _( 5), _( 6), _( 7), _( 8), _( 9),
_(10), _(11), _(12), _(13), _(14), _(15), _(16), _(17), _(18), _(19),
_(20), _(21), _(22), _(23), _(24), _(25), _(26), _(27), _(28), _(29),
_(30), _(31), _(32), _(33), _(34), _(35), _(36), _(37), _(38), _(39),
_(40), _(41), _(42), _(43), _(44), _(45), _(46), _(47), _(48), _(49),
_(50), _(51), _(52), _(53), _(54), _(55), _(56), _(57), _(58), _(59),
_(60), _(61), _(62), _(63), _(64), _(65), _(66), _(67), _(68), _(69),
_(70), _(71), _(72), _(73), _(74), _(75), _(76), _(77), _(78), _(79),
_(80), _(81), _(82), _(83), _(84), _(85), _(86), _(87), _(88), _(89),
_(90), _(91), _(92), _(93), _(94), _(95), _(96), _(97), _(98), _(99),
#undef _
};
/*
* This will print a single '0' even if r == 0, since we would
* immediately jump to out_r where two 0s would be written but only
* one of them accounted for in buf. This is needed by ip4_string
* below. All other callers pass a non-zero value of r.
*/
static noinline_for_stack
char *put_dec_trunc8(char *buf, unsigned r)
{
unsigned q;
/* 1 <= r < 10^8 */
if (r < 100)
goto out_r;
/* 100 <= r < 10^8 */
q = (r * (u64)0x28f5c29) >> 32;
*((u16 *)buf) = decpair[r - 100*q];
buf += 2;
/* 1 <= q < 10^6 */
if (q < 100)
goto out_q;
/* 100 <= q < 10^6 */
r = (q * (u64)0x28f5c29) >> 32;
*((u16 *)buf) = decpair[q - 100*r];
buf += 2;
/* 1 <= r < 10^4 */
if (r < 100)
goto out_r;
/* 100 <= r < 10^4 */
q = (r * 0x147b) >> 19;
*((u16 *)buf) = decpair[r - 100*q];
buf += 2;
out_q:
/* 1 <= q < 100 */
r = q;
out_r:
/* 1 <= r < 100 */
*((u16 *)buf) = decpair[r];
buf += r < 10 ? 1 : 2;
return buf;
}
#if BITS_PER_LONG == 64 && BITS_PER_LONG_LONG == 64
static noinline_for_stack
char *put_dec_full8(char *buf, unsigned r)
{
unsigned q;
/* 0 <= r < 10^8 */
q = (r * (u64)0x28f5c29) >> 32;
*((u16 *)buf) = decpair[r - 100*q];
buf += 2;
/* 0 <= q < 10^6 */
r = (q * (u64)0x28f5c29) >> 32;
*((u16 *)buf) = decpair[q - 100*r];
buf += 2;
/* 0 <= r < 10^4 */
q = (r * 0x147b) >> 19;
*((u16 *)buf) = decpair[r - 100*q];
buf += 2;
/* 0 <= q < 100 */
*((u16 *)buf) = decpair[q];
buf += 2;
return buf;
}
static noinline_for_stack
char *put_dec(char *buf, unsigned long long n)
{
if (n >= 100*1000*1000)
buf = put_dec_full8(buf, do_div(n, 100*1000*1000));
/* 1 <= n <= 1.6e11 */
if (n >= 100*1000*1000)
buf = put_dec_full8(buf, do_div(n, 100*1000*1000));
/* 1 <= n < 1e8 */
return put_dec_trunc8(buf, n);
}
#elif BITS_PER_LONG == 32 && BITS_PER_LONG_LONG == 64
static void
put_dec_full4(char *buf, unsigned r)
{
unsigned q;
/* 0 <= r < 10^4 */
q = (r * 0x147b) >> 19;
*((u16 *)buf) = decpair[r - 100*q];
buf += 2;
/* 0 <= q < 100 */
*((u16 *)buf) = decpair[q];
}
/*
* Call put_dec_full4 on x % 10000, return x / 10000.
* The approximation x/10000 == (x * 0x346DC5D7) >> 43
* holds for all x < 1,128,869,999. The largest value this
* helper will ever be asked to convert is 1,125,520,955.
* (second call in the put_dec code, assuming n is all-ones).
*/
static noinline_for_stack
unsigned put_dec_helper4(char *buf, unsigned x)
{
uint32_t q = (x * (uint64_t)0x346DC5D7) >> 43;
put_dec_full4(buf, x - q * 10000);
return q;
}
/* Based on code by Douglas W. Jones found at
* <http://www.cs.uiowa.edu/~jones/bcd/decimal.html#sixtyfour>
* (with permission from the author).
* Performs no 64-bit division and hence should be fast on 32-bit machines.
*/
static
char *put_dec(char *buf, unsigned long long n)
{
uint32_t d3, d2, d1, q, h;
if (n < 100*1000*1000)
return put_dec_trunc8(buf, n);
d1 = ((uint32_t)n >> 16); /* implicit "& 0xffff" */
h = (n >> 32);
d2 = (h ) & 0xffff;
d3 = (h >> 16); /* implicit "& 0xffff" */
/* n = 2^48 d3 + 2^32 d2 + 2^16 d1 + d0
= 281_4749_7671_0656 d3 + 42_9496_7296 d2 + 6_5536 d1 + d0 */
q = 656 * d3 + 7296 * d2 + 5536 * d1 + ((uint32_t)n & 0xffff);
q = put_dec_helper4(buf, q);
q += 7671 * d3 + 9496 * d2 + 6 * d1;
q = put_dec_helper4(buf+4, q);
q += 4749 * d3 + 42 * d2;
q = put_dec_helper4(buf+8, q);
q += 281 * d3;
buf += 12;
if (q)
buf = put_dec_trunc8(buf, q);
else while (buf[-1] == '0')
--buf;
return buf;
}
#endif
/*
* Convert passed number to decimal string.
* Returns the length of string. On buffer overflow, returns 0.
*
* If speed is not important, use snprintf(). It's easy to read the code.
*/
int num_to_str(char *buf, int size, unsigned long long num, unsigned int width)
{
/* put_dec requires 2-byte alignment of the buffer. */
char tmp[sizeof(num) * 3] __aligned(2);
int idx, len;
/* put_dec() may work incorrectly for num = 0 (generate "", not "0") */
if (num <= 9) {
tmp[0] = '0' + num;
len = 1;
} else {
len = put_dec(tmp, num) - tmp;
}
if (len > size || width > size)
return 0;
if (width > len) {
width = width - len;
for (idx = 0; idx < width; idx++)
buf[idx] = ' ';
} else {
width = 0;
}
for (idx = 0; idx < len; ++idx)
buf[idx + width] = tmp[len - idx - 1];
return len + width;
}
#define SIGN 1 /* unsigned/signed */
#define LEFT 2 /* left justified */
#define PLUS 4 /* show plus */
#define SPACE 8 /* space if plus */
#define ZEROPAD 16 /* pad with zero, must be 16 == '0' - ' ' */
#define SMALL 32 /* use lowercase in hex (must be 32 == 0x20) */
#define SPECIAL 64 /* prefix hex with "0x", octal with "0" */
static_assert(ZEROPAD == ('0' - ' '));
static_assert(SMALL == ('a' ^ 'A'));
enum format_state {
FORMAT_STATE_NONE, /* Just a string part */
FORMAT_STATE_NUM,
FORMAT_STATE_WIDTH,
FORMAT_STATE_PRECISION,
FORMAT_STATE_CHAR,
FORMAT_STATE_STR,
FORMAT_STATE_PTR,
FORMAT_STATE_PERCENT_CHAR,
FORMAT_STATE_INVALID,
};
struct printf_spec {
unsigned char flags; /* flags to number() */
unsigned char base; /* number base, 8, 10 or 16 only */
short precision; /* # of digits/chars */
int field_width; /* width of output field */
} __packed;
static_assert(sizeof(struct printf_spec) == 8);
#define FIELD_WIDTH_MAX ((1 << 23) - 1)
#define PRECISION_MAX ((1 << 15) - 1)
static noinline_for_stack
char *number(char *buf, char *end, unsigned long long num,
struct printf_spec spec)
{
/* put_dec requires 2-byte alignment of the buffer. */
char tmp[3 * sizeof(num)] __aligned(2);
char sign;
char locase;
int need_pfx = ((spec.flags & SPECIAL) && spec.base != 10);
int i;
bool is_zero = num == 0LL;
int field_width = spec.field_width;
int precision = spec.precision;
/* locase = 0 or 0x20. ORing digits or letters with 'locase'
* produces same digits or (maybe lowercased) letters */
locase = (spec.flags & SMALL);
if (spec.flags & LEFT)
spec.flags &= ~ZEROPAD;
sign = 0;
if (spec.flags & SIGN) {
if ((signed long long)num < 0) {
sign = '-';
num = -(signed long long)num;
field_width--;
} else if (spec.flags & PLUS) {
sign = '+';
field_width--;
} else if (spec.flags & SPACE) {
sign = ' ';
field_width--;
}
}
if (need_pfx) {
if (spec.base == 16)
field_width -= 2;
else if (!is_zero)
field_width--;
}
/* generate full string in tmp[], in reverse order */
i = 0;
if (num < spec.base)
tmp[i++] = hex_asc_upper[num] | locase;
else if (spec.base != 10) { /* 8 or 16 */
int mask = spec.base - 1;
int shift = 3;
if (spec.base == 16)
shift = 4;
do {
tmp[i++] = (hex_asc_upper[((unsigned char)num) & mask] | locase);
num >>= shift;
} while (num);
} else { /* base 10 */
i = put_dec(tmp, num) - tmp;
}
/* printing 100 using %2d gives "100", not "00" */
if (i > precision)
precision = i;
/* leading space padding */
field_width -= precision;
if (!(spec.flags & (ZEROPAD | LEFT))) {
while (--field_width >= 0) {
if (buf < end)
*buf = ' ';
++buf;
}
}
/* sign */
if (sign) {
if (buf < end)
*buf = sign;
++buf;
}
/* "0x" / "0" prefix */
if (need_pfx) {
if (spec.base == 16 || !is_zero) {
if (buf < end)
*buf = '0';
++buf;
}
if (spec.base == 16) {
if (buf < end)
*buf = ('X' | locase);
++buf;
}
}
/* zero or space padding */
if (!(spec.flags & LEFT)) {
char c = ' ' + (spec.flags & ZEROPAD);
while (--field_width >= 0) {
if (buf < end)
*buf = c;
++buf;
}
}
/* hmm even more zero padding? */
while (i <= --precision) {
if (buf < end)
*buf = '0';
++buf;
}
/* actual digits of result */
while (--i >= 0) {
if (buf < end)
*buf = tmp[i];
++buf;
}
/* trailing space padding */
while (--field_width >= 0) {
if (buf < end)
*buf = ' ';
++buf;
}
return buf;
}
#define special_hex_spec(size) \
(struct printf_spec) { \
.field_width = 2 + 2 * (size), /* 0x + hex */ \
.flags = SPECIAL | SMALL | ZEROPAD, \
.base = 16, \
.precision = -1, \
}
static noinline_for_stack
char *special_hex_number(char *buf, char *end, unsigned long long num, int size)
{
return number(buf, end, num, special_hex_spec(size));
}
static void move_right(char *buf, char *end, unsigned len, unsigned spaces)
{
size_t size;
if (buf >= end) /* nowhere to put anything */
return;
size = end - buf;
if (size <= spaces) {
memset(buf, ' ', size);
return;
}
if (len) {
if (len > size - spaces)
len = size - spaces;
memmove(buf + spaces, buf, len);
}
memset(buf, ' ', spaces);
}
/*
* Handle field width padding for a string.
* @buf: current buffer position
* @n: length of string
* @end: end of output buffer
* @spec: for field width and flags
* Returns: new buffer position after padding.
*/
static noinline_for_stack
char *widen_string(char *buf, int n, char *end, struct printf_spec spec)
{
unsigned spaces;
if (likely(n >= spec.field_width))
return buf;
/* we want to pad the sucker */
spaces = spec.field_width - n;
if (!(spec.flags & LEFT)) {
move_right(buf - n, end, n, spaces);
return buf + spaces;
}
while (spaces--) {
if (buf < end)
*buf = ' ';
++buf;
}
return buf;
}
/* Handle string from a well known address. */
static char *string_nocheck(char *buf, char *end, const char *s,
struct printf_spec spec)
{
int len = 0;
int lim = spec.precision;
while (lim--) {
char c = *s++;
if (!c)
break;
if (buf < end)
*buf = c;
++buf;
++len;
}
return widen_string(buf, len, end, spec);
}
static char *err_ptr(char *buf, char *end, void *ptr,
struct printf_spec spec)
{
int err = PTR_ERR(ptr);
const char *sym = errname(err);
if (sym)
return string_nocheck(buf, end, sym, spec);
/*
* Somebody passed ERR_PTR(-1234) or some other non-existing
* Efoo - or perhaps CONFIG_SYMBOLIC_ERRNAME=n. Fall back to
* printing it as its decimal representation.
*/
spec.flags |= SIGN;
spec.base = 10;
return number(buf, end, err, spec);
}
/* Be careful: error messages must fit into the given buffer. */
static char *error_string(char *buf, char *end, const char *s,
struct printf_spec spec)
{
/*
* Hard limit to avoid a completely insane messages. It actually
* works pretty well because most error messages are in
* the many pointer format modifiers.
*/
if (spec.precision == -1)
spec.precision = 2 * sizeof(void *);
return string_nocheck(buf, end, s, spec);
}
/*
* Do not call any complex external code here. Nested printk()/vsprintf()
* might cause infinite loops. Failures might break printk() and would
* be hard to debug.
*/
static const char *check_pointer_msg(const void *ptr)
{
if (!ptr)
return "(null)";
if ((unsigned long)ptr < PAGE_SIZE || IS_ERR_VALUE(ptr))
return "(efault)";
return NULL;
}
static int check_pointer(char **buf, char *end, const void *ptr,
struct printf_spec spec)
{
const char *err_msg;
err_msg = check_pointer_msg(ptr);
if (err_msg) {
*buf = error_string(*buf, end, err_msg, spec);
return -EFAULT;
}
return 0;
}
static noinline_for_stack
char *string(char *buf, char *end, const char *s,
struct printf_spec spec)
{
if (check_pointer(&buf, end, s, spec))
return buf;
return string_nocheck(buf, end, s, spec);
}
static char *pointer_string(char *buf, char *end,
const void *ptr,
struct printf_spec spec)
{
spec.base = 16;
spec.flags |= SMALL;
if (spec.field_width == -1) {
spec.field_width = 2 * sizeof(ptr);
spec.flags |= ZEROPAD;
}
return number(buf, end, (unsigned long int)ptr, spec);
}
/* Make pointers available for printing early in the boot sequence. */
static int debug_boot_weak_hash __ro_after_init;
static int __init debug_boot_weak_hash_enable(char *str)
{
debug_boot_weak_hash = 1;
pr_info("debug_boot_weak_hash enabled\n");
return 0;
}
early_param("debug_boot_weak_hash", debug_boot_weak_hash_enable);
static bool filled_random_ptr_key __read_mostly;
static siphash_key_t ptr_key __read_mostly;
static int fill_ptr_key(struct notifier_block *nb, unsigned long action, void *data)
{
get_random_bytes(&ptr_key, sizeof(ptr_key));
/* Pairs with smp_rmb() before reading ptr_key. */
smp_wmb();
WRITE_ONCE(filled_random_ptr_key, true);
return NOTIFY_DONE;
}
static int __init vsprintf_init_hashval(void)
{
static struct notifier_block fill_ptr_key_nb = { .notifier_call = fill_ptr_key };
execute_with_initialized_rng(&fill_ptr_key_nb);
return 0;
}
subsys_initcall(vsprintf_init_hashval)
/* Maps a pointer to a 32 bit unique identifier. */
static inline int __ptr_to_hashval(const void *ptr, unsigned long *hashval_out)
{
unsigned long hashval;
if (!READ_ONCE(filled_random_ptr_key))
return -EBUSY;
/* Pairs with smp_wmb() after writing ptr_key. */
smp_rmb();
#ifdef CONFIG_64BIT
hashval = (unsigned long)siphash_1u64((u64)ptr, &ptr_key);
/*
* Mask off the first 32 bits, this makes explicit that we have
* modified the address (and 32 bits is plenty for a unique ID).
*/
hashval = hashval & 0xffffffff;
#else
hashval = (unsigned long)siphash_1u32((u32)ptr, &ptr_key);
#endif
*hashval_out = hashval;
return 0;
}
int ptr_to_hashval(const void *ptr, unsigned long *hashval_out)
{
return __ptr_to_hashval(ptr, hashval_out);
}
static char *ptr_to_id(char *buf, char *end, const void *ptr,
struct printf_spec spec)
{
const char *str = sizeof(ptr) == 8 ? "(____ptrval____)" : "(ptrval)";
unsigned long hashval;
int ret;
/*
* Print the real pointer value for NULL and error pointers,
* as they are not actual addresses.
*/
if (IS_ERR_OR_NULL(ptr))
return pointer_string(buf, end, ptr, spec);
/* When debugging early boot use non-cryptographically secure hash. */
if (unlikely(debug_boot_weak_hash)) {
hashval = hash_long((unsigned long)ptr, 32);
return pointer_string(buf, end, (const void *)hashval, spec);
}
ret = __ptr_to_hashval(ptr, &hashval);
if (ret) {
spec.field_width = 2 * sizeof(ptr);
/* string length must be less than default_width */
return error_string(buf, end, str, spec);
}
return pointer_string(buf, end, (const void *)hashval, spec);
}
static char *default_pointer(char *buf, char *end, const void *ptr,
struct printf_spec spec)
{
/*
* default is to _not_ leak addresses, so hash before printing,
* unless no_hash_pointers is specified on the command line.
*/
if (unlikely(no_hash_pointers))
return pointer_string(buf, end, ptr, spec);
return ptr_to_id(buf, end, ptr, spec);
}
int kptr_restrict __read_mostly;
static noinline_for_stack
char *restricted_pointer(char *buf, char *end, const void *ptr,
struct printf_spec spec)
{
switch (kptr_restrict) {
case 0:
/* Handle as %p, hash and do _not_ leak addresses. */
return default_pointer(buf, end, ptr, spec);
case 1: {
const struct cred *cred;
/*
* kptr_restrict==1 cannot be used in IRQ context
* because its test for CAP_SYSLOG would be meaningless.
*/
if (in_hardirq() || in_serving_softirq() || in_nmi()) {
if (spec.field_width == -1)
spec.field_width = 2 * sizeof(ptr);
return error_string(buf, end, "pK-error", spec);
}
/*
* Only print the real pointer value if the current
* process has CAP_SYSLOG and is running with the
* same credentials it started with. This is because
* access to files is checked at open() time, but %pK
* checks permission at read() time. We don't want to
* leak pointer values if a binary opens a file using
* %pK and then elevates privileges before reading it.
*/
cred = current_cred();
if (!has_capability_noaudit(current, CAP_SYSLOG) ||
!uid_eq(cred->euid, cred->uid) ||
!gid_eq(cred->egid, cred->gid))
ptr = NULL;
break;
}
case 2:
default:
/* Always print 0's for %pK */
ptr = NULL;
break;
}
return pointer_string(buf, end, ptr, spec);
}
static noinline_for_stack
char *dentry_name(char *buf, char *end, const struct dentry *d, struct printf_spec spec,
const char *fmt)
{
const char *array[4], *s;
const struct dentry *p;
int depth;
int i, n;
switch (fmt[1]) {
case '2': case '3': case '4':
depth = fmt[1] - '0';
break;
default:
depth = 1;
}
rcu_read_lock();
for (i = 0; i < depth; i++, d = p) {
if (check_pointer(&buf, end, d, spec)) {
rcu_read_unlock();
return buf;
}
p = READ_ONCE(d->d_parent);
array[i] = READ_ONCE(d->d_name.name);
if (p == d) {
if (i)
array[i] = "";
i++;
break;
}
}
s = array[--i];
for (n = 0; n != spec.precision; n++, buf++) {
char c = *s++;
if (!c) {
if (!i)
break;
c = '/';
s = array[--i];
}
if (buf < end)
*buf = c;
}
rcu_read_unlock();
return widen_string(buf, n, end, spec);
}
static noinline_for_stack
char *file_dentry_name(char *buf, char *end, const struct file *f,
struct printf_spec spec, const char *fmt)
{
if (check_pointer(&buf, end, f, spec))
return buf;
return dentry_name(buf, end, f->f_path.dentry, spec, fmt);
}
#ifdef CONFIG_BLOCK
static noinline_for_stack
char *bdev_name(char *buf, char *end, struct block_device *bdev,
struct printf_spec spec, const char *fmt)
{
struct gendisk *hd;
if (check_pointer(&buf, end, bdev, spec))
return buf;
hd = bdev->bd_disk;
buf = string(buf, end, hd->disk_name, spec);
if (bdev_is_partition(bdev)) {
if (isdigit(hd->disk_name[strlen(hd->disk_name)-1])) {
if (buf < end)
*buf = 'p';
buf++;
}
buf = number(buf, end, bdev_partno(bdev), spec);
}
return buf;
}
#endif
static noinline_for_stack
char *symbol_string(char *buf, char *end, void *ptr,
struct printf_spec spec, const char *fmt)
{
unsigned long value;
#ifdef CONFIG_KALLSYMS
char sym[KSYM_SYMBOL_LEN];
#endif
if (fmt[1] == 'R')
ptr = __builtin_extract_return_addr(ptr);
value = (unsigned long)ptr;
#ifdef CONFIG_KALLSYMS
if (*fmt == 'B' && fmt[1] == 'b')
sprint_backtrace_build_id(sym, value);
else if (*fmt == 'B')
sprint_backtrace(sym, value);
else if (*fmt == 'S' && (fmt[1] == 'b' || (fmt[1] == 'R' && fmt[2] == 'b')))
sprint_symbol_build_id(sym, value);
else if (*fmt != 's')
sprint_symbol(sym, value);
else
sprint_symbol_no_offset(sym, value);
return string_nocheck(buf, end, sym, spec);
#else
return special_hex_number(buf, end, value, sizeof(void *));
#endif
}
static const struct printf_spec default_str_spec = {
.field_width = -1,
.precision = -1,
};
static const struct printf_spec default_flag_spec = {
.base = 16,
.precision = -1,
.flags = SPECIAL | SMALL,
};
static const struct printf_spec default_dec_spec = {
.base = 10,
.precision = -1,
};
static const struct printf_spec default_dec02_spec = {
.base = 10,
.field_width = 2,
.precision = -1,
.flags = ZEROPAD,
};
static const struct printf_spec default_dec04_spec = {
.base = 10,
.field_width = 4,
.precision = -1,
.flags = ZEROPAD,
};
static noinline_for_stack
char *hex_range(char *buf, char *end, u64 start_val, u64 end_val,
struct printf_spec spec)
{
buf = number(buf, end, start_val, spec);
if (start_val == end_val)
return buf;
if (buf < end)
*buf = '-';
++buf;
return number(buf, end, end_val, spec);
}
static noinline_for_stack
char *resource_string(char *buf, char *end, struct resource *res,
struct printf_spec spec, const char *fmt)
{
#ifndef IO_RSRC_PRINTK_SIZE
#define IO_RSRC_PRINTK_SIZE 6
#endif
#ifndef MEM_RSRC_PRINTK_SIZE
#define MEM_RSRC_PRINTK_SIZE 10
#endif
static const struct printf_spec io_spec = {
.base = 16,
.field_width = IO_RSRC_PRINTK_SIZE,
.precision = -1,
.flags = SPECIAL | SMALL | ZEROPAD,
};
static const struct printf_spec mem_spec = {
.base = 16,
.field_width = MEM_RSRC_PRINTK_SIZE,
.precision = -1,
.flags = SPECIAL | SMALL | ZEROPAD,
};
static const struct printf_spec bus_spec = {
.base = 16,
.field_width = 2,
.precision = -1,
.flags = SMALL | ZEROPAD,
};
static const struct printf_spec str_spec = {
.field_width = -1,
.precision = 10,
.flags = LEFT,
};
/* 32-bit res (sizeof==4): 10 chars in dec, 10 in hex ("0x" + 8)
* 64-bit res (sizeof==8): 20 chars in dec, 18 in hex ("0x" + 16) */
#define RSRC_BUF_SIZE ((2 * sizeof(resource_size_t)) + 4)
#define FLAG_BUF_SIZE (2 * sizeof(res->flags))
#define DECODED_BUF_SIZE sizeof("[mem - 64bit pref window disabled]")
#define RAW_BUF_SIZE sizeof("[mem - flags 0x]")
char sym[MAX(2*RSRC_BUF_SIZE + DECODED_BUF_SIZE,
2*RSRC_BUF_SIZE + FLAG_BUF_SIZE + RAW_BUF_SIZE)];
char *p = sym, *pend = sym + sizeof(sym);
bool decode = fmt[0] == 'R';
const struct printf_spec *specp;
if (check_pointer(&buf, end, res, spec))
return buf;
*p++ = '[';
if (res->flags & IORESOURCE_IO) {
p = string_nocheck(p, pend, "io ", str_spec);
specp = &io_spec;
} else if (res->flags & IORESOURCE_MEM) {
p = string_nocheck(p, pend, "mem ", str_spec);
specp = &mem_spec;
} else if (res->flags & IORESOURCE_IRQ) {
p = string_nocheck(p, pend, "irq ", str_spec);
specp = &default_dec_spec;
} else if (res->flags & IORESOURCE_DMA) {
p = string_nocheck(p, pend, "dma ", str_spec);
specp = &default_dec_spec;
} else if (res->flags & IORESOURCE_BUS) {
p = string_nocheck(p, pend, "bus ", str_spec);
specp = &bus_spec;
} else {
p = string_nocheck(p, pend, "??? ", str_spec);
specp = &mem_spec;
decode = false;
}
if (decode && res->flags & IORESOURCE_UNSET) {
p = string_nocheck(p, pend, "size ", str_spec);
p = number(p, pend, resource_size(res), *specp);
} else {
p = hex_range(p, pend, res->start, res->end, *specp);
}
if (decode) {
if (res->flags & IORESOURCE_MEM_64)
p = string_nocheck(p, pend, " 64bit", str_spec);
if (res->flags & IORESOURCE_PREFETCH)
p = string_nocheck(p, pend, " pref", str_spec);
if (res->flags & IORESOURCE_WINDOW)
p = string_nocheck(p, pend, " window", str_spec);
if (res->flags & IORESOURCE_DISABLED)
p = string_nocheck(p, pend, " disabled", str_spec);
} else {
p = string_nocheck(p, pend, " flags ", str_spec);
p = number(p, pend, res->flags, default_flag_spec);
}
*p++ = ']';
*p = '\0';
return string_nocheck(buf, end, sym, spec);
}
static noinline_for_stack
char *range_string(char *buf, char *end, const struct range *range,
struct printf_spec spec, const char *fmt)
{
char sym[sizeof("[range 0x0123456789abcdef-0x0123456789abcdef]")];
char *p = sym, *pend = sym + sizeof(sym);
if (check_pointer(&buf, end, range, spec))
return buf;
p = string_nocheck(p, pend, "[range ", default_str_spec);
p = hex_range(p, pend, range->start, range->end, special_hex_spec(sizeof(range->start)));
*p++ = ']';
*p = '\0';
return string_nocheck(buf, end, sym, spec);
}
static noinline_for_stack
char *hex_string(char *buf, char *end, u8 *addr, struct printf_spec spec,
const char *fmt)
{
int i, len = 1; /* if we pass '%ph[CDN]', field width remains
negative value, fallback to the default */
char separator;
if (spec.field_width == 0)
/* nothing to print */
return buf;
if (check_pointer(&buf, end, addr, spec))
return buf;
switch (fmt[1]) {
case 'C':
separator = ':';
break;
case 'D':
separator = '-';
break;
case 'N':
separator = 0;
break;
default:
separator = ' ';
break;
}
if (spec.field_width > 0)
len = min_t(int, spec.field_width, 64);
for (i = 0; i < len; ++i) {
if (buf < end)
*buf = hex_asc_hi(addr[i]);
++buf;
if (buf < end)
*buf = hex_asc_lo(addr[i]);
++buf;
if (separator && i != len - 1) {
if (buf < end)
*buf = separator;
++buf;
}
}
return buf;
}
static noinline_for_stack
char *bitmap_string(char *buf, char *end, const unsigned long *bitmap,
struct printf_spec spec, const char *fmt)
{
const int CHUNKSZ = 32;
int nr_bits = max_t(int, spec.field_width, 0);
int i, chunksz;
bool first = true;
if (check_pointer(&buf, end, bitmap, spec))
return buf;
/* reused to print numbers */
spec = (struct printf_spec){ .flags = SMALL | ZEROPAD, .base = 16 };
chunksz = nr_bits & (CHUNKSZ - 1);
if (chunksz == 0)
chunksz = CHUNKSZ;
i = ALIGN(nr_bits, CHUNKSZ) - CHUNKSZ;
for (; i >= 0; i -= CHUNKSZ) {
u32 chunkmask, val;
int word, bit;
chunkmask = ((1ULL << chunksz) - 1);
word = i / BITS_PER_LONG;
bit = i % BITS_PER_LONG;
val = (bitmap[word] >> bit) & chunkmask;
if (!first) {
if (buf < end)
*buf = ',';
buf++;
}
first = false;
spec.field_width = DIV_ROUND_UP(chunksz, 4);
buf = number(buf, end, val, spec);
chunksz = CHUNKSZ;
}
return buf;
}
static noinline_for_stack
char *bitmap_list_string(char *buf, char *end, const unsigned long *bitmap,
struct printf_spec spec, const char *fmt)
{
int nr_bits = max_t(int, spec.field_width, 0);
bool first = true;
int rbot, rtop;
if (check_pointer(&buf, end, bitmap, spec))
return buf;
for_each_set_bitrange(rbot, rtop, bitmap, nr_bits) {
if (!first) {
if (buf < end)
*buf = ',';
buf++;
}
first = false;
buf = number(buf, end, rbot, default_dec_spec);
if (rtop == rbot + 1)
continue;
if (buf < end)
*buf = '-';
buf = number(++buf, end, rtop - 1, default_dec_spec);
}
return buf;
}
static noinline_for_stack
char *mac_address_string(char *buf, char *end, u8 *addr,
struct printf_spec spec, const char *fmt)
{
char mac_addr[sizeof("xx:xx:xx:xx:xx:xx")];
char *p = mac_addr;
int i;
char separator;
bool reversed = false;
if (check_pointer(&buf, end, addr, spec))
return buf;
switch (fmt[1]) {
case 'F':
separator = '-';
break;
case 'R':
reversed = true;
fallthrough;
default:
separator = ':';
break;
}
for (i = 0; i < 6; i++) {
if (reversed)
p = hex_byte_pack(p, addr[5 - i]);
else
p = hex_byte_pack(p, addr[i]);
if (fmt[0] == 'M' && i != 5)
*p++ = separator;
}
*p = '\0';
return string_nocheck(buf, end, mac_addr, spec);
}
static noinline_for_stack
char *ip4_string(char *p, const u8 *addr, const char *fmt)
{
int i;
bool leading_zeros = (fmt[0] == 'i');
int index;
int step;
switch (fmt[2]) {
case 'h':
#ifdef __BIG_ENDIAN
index = 0;
step = 1;
#else
index = 3;
step = -1;
#endif
break;
case 'l':
index = 3;
step = -1;
break;
case 'n':
case 'b':
default:
index = 0;
step = 1;
break;
}
for (i = 0; i < 4; i++) {
char temp[4] __aligned(2); /* hold each IP quad in reverse order */
int digits = put_dec_trunc8(temp, addr[index]) - temp;
if (leading_zeros) {
if (digits < 3)
*p++ = '0';
if (digits < 2)
*p++ = '0';
}
/* reverse the digits in the quad */
while (digits--)
*p++ = temp[digits];
if (i < 3)
*p++ = '.';
index += step;
}
*p = '\0';
return p;
}
static noinline_for_stack
char *ip6_compressed_string(char *p, const char *addr)
{
int i, j, range;
unsigned char zerolength[8];
int longest = 1;
int colonpos = -1;
u16 word;
u8 hi, lo;
bool needcolon = false;
bool useIPv4;
struct in6_addr in6;
memcpy(&in6, addr, sizeof(struct in6_addr));
useIPv4 = ipv6_addr_v4mapped(&in6) || ipv6_addr_is_isatap(&in6);
memset(zerolength, 0, sizeof(zerolength));
if (useIPv4)
range = 6;
else
range = 8;
/* find position of longest 0 run */
for (i = 0; i < range; i++) {
for (j = i; j < range; j++) {
if (in6.s6_addr16[j] != 0)
break;
zerolength[i]++;
}
}
for (i = 0; i < range; i++) {
if (zerolength[i] > longest) {
longest = zerolength[i];
colonpos = i;
}
}
if (longest == 1) /* don't compress a single 0 */
colonpos = -1;
/* emit address */
for (i = 0; i < range; i++) {
if (i == colonpos) {
if (needcolon || i == 0)
*p++ = ':';
*p++ = ':';
needcolon = false;
i += longest - 1;
continue;
}
if (needcolon) {
*p++ = ':';
needcolon = false;
}
/* hex u16 without leading 0s */
word = ntohs(in6.s6_addr16[i]);
hi = word >> 8;
lo = word & 0xff;
if (hi) {
if (hi > 0x0f)
p = hex_byte_pack(p, hi);
else
*p++ = hex_asc_lo(hi);
p = hex_byte_pack(p, lo);
}
else if (lo > 0x0f)
p = hex_byte_pack(p, lo);
else
*p++ = hex_asc_lo(lo);
needcolon = true;
}
if (useIPv4) {
if (needcolon)
*p++ = ':';
p = ip4_string(p, &in6.s6_addr[12], "I4");
}
*p = '\0';
return p;
}
static noinline_for_stack
char *ip6_string(char *p, const char *addr, const char *fmt)
{
int i;
for (i = 0; i < 8; i++) {
p = hex_byte_pack(p, *addr++);
p = hex_byte_pack(p, *addr++);
if (fmt[0] == 'I' && i != 7)
*p++ = ':';
}
*p = '\0';
return p;
}
static noinline_for_stack
char *ip6_addr_string(char *buf, char *end, const u8 *addr,
struct printf_spec spec, const char *fmt)
{
char ip6_addr[sizeof("xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:255.255.255.255")];
if (fmt[0] == 'I' && fmt[2] == 'c')
ip6_compressed_string(ip6_addr, addr);
else
ip6_string(ip6_addr, addr, fmt);
return string_nocheck(buf, end, ip6_addr, spec);
}
static noinline_for_stack
char *ip4_addr_string(char *buf, char *end, const u8 *addr,
struct printf_spec spec, const char *fmt)
{
char ip4_addr[sizeof("255.255.255.255")];
ip4_string(ip4_addr, addr, fmt);
return string_nocheck(buf, end, ip4_addr, spec);
}
static noinline_for_stack
char *ip6_addr_string_sa(char *buf, char *end, const struct sockaddr_in6 *sa,
struct printf_spec spec, const char *fmt)
{
bool have_p = false, have_s = false, have_f = false, have_c = false;
char ip6_addr[sizeof("[xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:255.255.255.255]") +
sizeof(":12345") + sizeof("/123456789") +
sizeof("%1234567890")];
char *p = ip6_addr, *pend = ip6_addr + sizeof(ip6_addr);
const u8 *addr = (const u8 *) &sa->sin6_addr;
char fmt6[2] = { fmt[0], '6' };
u8 off = 0;
fmt++;
while (isalpha(*++fmt)) {
switch (*fmt) {
case 'p':
have_p = true;
break;
case 'f':
have_f = true;
break;
case 's':
have_s = true;
break;
case 'c':
have_c = true;
break;
}
}
if (have_p || have_s || have_f) {
*p = '[';
off = 1;
}
if (fmt6[0] == 'I' && have_c)
p = ip6_compressed_string(ip6_addr + off, addr);
else
p = ip6_string(ip6_addr + off, addr, fmt6);
if (have_p || have_s || have_f)
*p++ = ']';
if (have_p) {
*p++ = ':';
p = number(p, pend, ntohs(sa->sin6_port), spec);
}
if (have_f) {
*p++ = '/';
p = number(p, pend, ntohl(sa->sin6_flowinfo &
IPV6_FLOWINFO_MASK), spec);
}
if (have_s) {
*p++ = '%';
p = number(p, pend, sa->sin6_scope_id, spec);
}
*p = '\0';
return string_nocheck(buf, end, ip6_addr, spec);
}
static noinline_for_stack
char *ip4_addr_string_sa(char *buf, char *end, const struct sockaddr_in *sa,
struct printf_spec spec, const char *fmt)
{
bool have_p = false;
char *p, ip4_addr[sizeof("255.255.255.255") + sizeof(":12345")];
char *pend = ip4_addr + sizeof(ip4_addr);
const u8 *addr = (const u8 *) &sa->sin_addr.s_addr;
char fmt4[3] = { fmt[0], '4', 0 };
fmt++;
while (isalpha(*++fmt)) {
switch (*fmt) {
case 'p':
have_p = true;
break;
case 'h':
case 'l':
case 'n':
case 'b':
fmt4[2] = *fmt;
break;
}
}
p = ip4_string(ip4_addr, addr, fmt4);
if (have_p) {
*p++ = ':';
p = number(p, pend, ntohs(sa->sin_port), spec);
}
*p = '\0';
return string_nocheck(buf, end, ip4_addr, spec);
}
static noinline_for_stack
char *ip_addr_string(char *buf, char *end, const void *ptr,
struct printf_spec spec, const char *fmt)
{
char *err_fmt_msg;
if (check_pointer(&buf, end, ptr, spec))
return buf;
switch (fmt[1]) {
case '6':
return ip6_addr_string(buf, end, ptr, spec, fmt);
case '4':
return ip4_addr_string(buf, end, ptr, spec, fmt);
case 'S': {
const union {
struct sockaddr raw;
struct sockaddr_in v4;
struct sockaddr_in6 v6;
} *sa = ptr;
switch (sa->raw.sa_family) {
case AF_INET:
return ip4_addr_string_sa(buf, end, &sa->v4, spec, fmt);
case AF_INET6:
return ip6_addr_string_sa(buf, end, &sa->v6, spec, fmt);
default:
return error_string(buf, end, "(einval)", spec);
}}
}
err_fmt_msg = fmt[0] == 'i' ? "(%pi?)" : "(%pI?)";
return error_string(buf, end, err_fmt_msg, spec);
}
static noinline_for_stack
char *escaped_string(char *buf, char *end, u8 *addr, struct printf_spec spec,
const char *fmt)
{
bool found = true;
int count = 1;
unsigned int flags = 0;
int len;
if (spec.field_width == 0)
return buf; /* nothing to print */
if (check_pointer(&buf, end, addr, spec))
return buf;
do {
switch (fmt[count++]) {
case 'a':
flags |= ESCAPE_ANY;
break;
case 'c':
flags |= ESCAPE_SPECIAL;
break;
case 'h':
flags |= ESCAPE_HEX;
break;
case 'n':
flags |= ESCAPE_NULL;
break;
case 'o':
flags |= ESCAPE_OCTAL;
break;
case 'p':
flags |= ESCAPE_NP;
break;
case 's':
flags |= ESCAPE_SPACE;
break;
default:
found = false;
break;
}
} while (found);
if (!flags)
flags = ESCAPE_ANY_NP;
len = spec.field_width < 0 ? 1 : spec.field_width;
/*
* string_escape_mem() writes as many characters as it can to
* the given buffer, and returns the total size of the output
* had the buffer been big enough.
*/
buf += string_escape_mem(addr, len, buf, buf < end ? end - buf : 0, flags, NULL);
return buf;
}
__diag_push();
__diag_ignore(GCC, all, "-Wsuggest-attribute=format",
"Not a valid __printf() conversion candidate.");
static char *va_format(char *buf, char *end, struct va_format *va_fmt,
struct printf_spec spec)
{
va_list va;
if (check_pointer(&buf, end, va_fmt, spec))
return buf;
va_copy(va, *va_fmt->va);
buf += vsnprintf(buf, end > buf ? end - buf : 0, va_fmt->fmt, va);
va_end(va);
return buf;
}
__diag_pop();
static noinline_for_stack
char *uuid_string(char *buf, char *end, const u8 *addr,
struct printf_spec spec, const char *fmt)
{
char uuid[UUID_STRING_LEN + 1];
char *p = uuid;
int i;
const u8 *index = uuid_index;
bool uc = false;
if (check_pointer(&buf, end, addr, spec))
return buf;
switch (*(++fmt)) {
case 'L':
uc = true;
fallthrough;
case 'l':
index = guid_index;
break;
case 'B':
uc = true;
break;
}
for (i = 0; i < 16; i++) {
if (uc)
p = hex_byte_pack_upper(p, addr[index[i]]);
else
p = hex_byte_pack(p, addr[index[i]]);
switch (i) {
case 3:
case 5:
case 7:
case 9:
*p++ = '-';
break;
}
}
*p = 0;
return string_nocheck(buf, end, uuid, spec);
}
static noinline_for_stack
char *netdev_bits(char *buf, char *end, const void *addr,
struct printf_spec spec, const char *fmt)
{
unsigned long long num;
int size;
if (check_pointer(&buf, end, addr, spec))
return buf;
switch (fmt[1]) {
case 'F':
num = *(const netdev_features_t *)addr;
size = sizeof(netdev_features_t);
break;
default:
return error_string(buf, end, "(%pN?)", spec);
}
return special_hex_number(buf, end, num, size);
}
static noinline_for_stack
char *fourcc_string(char *buf, char *end, const u32 *fourcc,
struct printf_spec spec, const char *fmt)
{
char output[sizeof("0123 little-endian (0x01234567)")];
char *p = output;
unsigned int i;
bool pixel_fmt = false;
u32 orig, val;
if (fmt[1] != 'c')
return error_string(buf, end, "(%p4?)", spec);
if (check_pointer(&buf, end, fourcc, spec))
return buf;
orig = get_unaligned(fourcc);
switch (fmt[2]) {
case 'h':
if (fmt[3] == 'R')
orig = swab32(orig);
break;
case 'l':
orig = (__force u32)cpu_to_le32(orig);
break;
case 'b':
orig = (__force u32)cpu_to_be32(orig);
break;
case 'c':
/* Pixel formats are printed LSB-first */
pixel_fmt = true;
break;
default:
return error_string(buf, end, "(%p4?)", spec);
}
val = pixel_fmt ? swab32(orig & ~BIT(31)) : orig;
for (i = 0; i < sizeof(u32); i++) {
unsigned char c = val >> ((3 - i) * 8);
/* Print non-control ASCII characters as-is, dot otherwise */
*p++ = isascii(c) && isprint(c) ? c : '.';
}
if (pixel_fmt) {
*p++ = ' ';
strcpy(p, orig & BIT(31) ? "big-endian" : "little-endian");
p += strlen(p);
}
*p++ = ' ';
*p++ = '(';
p = special_hex_number(p, output + sizeof(output) - 2, orig, sizeof(u32));
*p++ = ')';
*p = '\0';
return string(buf, end, output, spec);
}
static noinline_for_stack
char *address_val(char *buf, char *end, const void *addr,
struct printf_spec spec, const char *fmt)
{
unsigned long long num;
int size;
if (check_pointer(&buf, end, addr, spec))
return buf;
switch (fmt[1]) {
case 'd':
num = *(const dma_addr_t *)addr;
size = sizeof(dma_addr_t);
break;
case 'p':
default:
num = *(const phys_addr_t *)addr;
size = sizeof(phys_addr_t);
break;
}
return special_hex_number(buf, end, num, size);
}
static noinline_for_stack
char *date_str(char *buf, char *end, const struct rtc_time *tm, bool r)
{
int year = tm->tm_year + (r ? 0 : 1900);
int mon = tm->tm_mon + (r ? 0 : 1);
buf = number(buf, end, year, default_dec04_spec);
if (buf < end)
*buf = '-';
buf++;
buf = number(buf, end, mon, default_dec02_spec);
if (buf < end)
*buf = '-';
buf++;
return number(buf, end, tm->tm_mday, default_dec02_spec);
}
static noinline_for_stack
char *time_str(char *buf, char *end, const struct rtc_time *tm, bool r)
{
buf = number(buf, end, tm->tm_hour, default_dec02_spec);
if (buf < end)
*buf = ':';
buf++;
buf = number(buf, end, tm->tm_min, default_dec02_spec);
if (buf < end)
*buf = ':';
buf++;
return number(buf, end, tm->tm_sec, default_dec02_spec);
}
static noinline_for_stack
char *rtc_str(char *buf, char *end, const struct rtc_time *tm,
struct printf_spec spec, const char *fmt)
{
bool have_t = true, have_d = true;
bool raw = false, iso8601_separator = true;
bool found = true;
int count = 2;
switch (fmt[count]) {
case 'd':
have_t = false;
count++;
break;
case 't':
have_d = false;
count++;
break;
}
do {
switch (fmt[count++]) {
case 'r':
raw = true;
break;
case 's':
iso8601_separator = false;
break;
default:
found = false;
break;
}
} while (found);
if (have_d)
buf = date_str(buf, end, tm, raw);
if (have_d && have_t) {
if (buf < end)
*buf = iso8601_separator ? 'T' : ' ';
buf++;
}
if (have_t)
buf = time_str(buf, end, tm, raw);
return buf;
}
static noinline_for_stack
char *time64_str(char *buf, char *end, const time64_t time,
struct printf_spec spec, const char *fmt)
{
struct rtc_time rtc_time;
struct tm tm;
time64_to_tm(time, 0, &tm);
rtc_time.tm_sec = tm.tm_sec;
rtc_time.tm_min = tm.tm_min;
rtc_time.tm_hour = tm.tm_hour;
rtc_time.tm_mday = tm.tm_mday;
rtc_time.tm_mon = tm.tm_mon;
rtc_time.tm_year = tm.tm_year;
rtc_time.tm_wday = tm.tm_wday;
rtc_time.tm_yday = tm.tm_yday;
rtc_time.tm_isdst = 0;
return rtc_str(buf, end, &rtc_time, spec, fmt);
}
static noinline_for_stack
char *timespec64_str(char *buf, char *end, const struct timespec64 *ts,
struct printf_spec spec, const char *fmt)
{
static const struct printf_spec default_dec09_spec = {
.base = 10,
.field_width = 9,
.precision = -1,
.flags = ZEROPAD,
};
if (fmt[2] == 'p')
buf = number(buf, end, ts->tv_sec, default_dec_spec);
else
buf = time64_str(buf, end, ts->tv_sec, spec, fmt);
if (buf < end)
*buf = '.';
buf++;
return number(buf, end, ts->tv_nsec, default_dec09_spec);
}
static noinline_for_stack
char *time_and_date(char *buf, char *end, void *ptr, struct printf_spec spec,
const char *fmt)
{
if (check_pointer(&buf, end, ptr, spec))
return buf;
switch (fmt[1]) {
case 'R':
return rtc_str(buf, end, (const struct rtc_time *)ptr, spec, fmt);
case 'S':
return timespec64_str(buf, end, (const struct timespec64 *)ptr, spec, fmt);
case 'T':
return time64_str(buf, end, *(const time64_t *)ptr, spec, fmt);
default:
return error_string(buf, end, "(%pt?)", spec);
}
}
static noinline_for_stack
char *clock(char *buf, char *end, struct clk *clk, struct printf_spec spec,
const char *fmt)
{
if (!IS_ENABLED(CONFIG_HAVE_CLK))
return error_string(buf, end, "(%pC?)", spec);
if (check_pointer(&buf, end, clk, spec))
return buf;
#ifdef CONFIG_COMMON_CLK
return string(buf, end, __clk_get_name(clk), spec);
#else
return ptr_to_id(buf, end, clk, spec);
#endif
}
static
char *format_flags(char *buf, char *end, unsigned long flags,
const struct trace_print_flags *names)
{
unsigned long mask;
for ( ; flags && names->name; names++) {
mask = names->mask;
if ((flags & mask) != mask)
continue;
buf = string(buf, end, names->name, default_str_spec);
flags &= ~mask;
if (flags) {
if (buf < end)
*buf = '|';
buf++;
}
}
if (flags)
buf = number(buf, end, flags, default_flag_spec);
return buf;
}
struct page_flags_fields {
int width;
int shift;
int mask;
const struct printf_spec *spec;
const char *name;
};
static const struct page_flags_fields pff[] = {
{SECTIONS_WIDTH, SECTIONS_PGSHIFT, SECTIONS_MASK,
&default_dec_spec, "section"},
{NODES_WIDTH, NODES_PGSHIFT, NODES_MASK,
&default_dec_spec, "node"},
{ZONES_WIDTH, ZONES_PGSHIFT, ZONES_MASK,
&default_dec_spec, "zone"},
{LAST_CPUPID_WIDTH, LAST_CPUPID_PGSHIFT, LAST_CPUPID_MASK,
&default_flag_spec, "lastcpupid"},
{KASAN_TAG_WIDTH, KASAN_TAG_PGSHIFT, KASAN_TAG_MASK,
&default_flag_spec, "kasantag"},
};
static
char *format_page_flags(char *buf, char *end, unsigned long flags)
{
unsigned long main_flags = flags & PAGEFLAGS_MASK;
bool append = false;
int i;
buf = number(buf, end, flags, default_flag_spec);
if (buf < end)
*buf = '(';
buf++;
/* Page flags from the main area. */
if (main_flags) {
buf = format_flags(buf, end, main_flags, pageflag_names);
append = true;
}
/* Page flags from the fields area */
for (i = 0; i < ARRAY_SIZE(pff); i++) {
/* Skip undefined fields. */
if (!pff[i].width)
continue;
/* Format: Flag Name + '=' (equals sign) + Number + '|' (separator) */
if (append) {
if (buf < end)
*buf = '|';
buf++;
}
buf = string(buf, end, pff[i].name, default_str_spec);
if (buf < end)
*buf = '=';
buf++;
buf = number(buf, end, (flags >> pff[i].shift) & pff[i].mask,
*pff[i].spec);
append = true;
}
if (buf < end)
*buf = ')';
buf++;
return buf;
}
static noinline_for_stack
char *flags_string(char *buf, char *end, void *flags_ptr,
struct printf_spec spec, const char *fmt)
{
unsigned long flags;
const struct trace_print_flags *names;
if (check_pointer(&buf, end, flags_ptr, spec))
return buf;
switch (fmt[1]) {
case 'p':
return format_page_flags(buf, end, *(unsigned long *)flags_ptr);
case 'v':
flags = *(unsigned long *)flags_ptr;
names = vmaflag_names;
break;
case 'g':
flags = (__force unsigned long)(*(gfp_t *)flags_ptr);
names = gfpflag_names;
break;
default:
return error_string(buf, end, "(%pG?)", spec);
}
return format_flags(buf, end, flags, names);
}
static noinline_for_stack
char *fwnode_full_name_string(struct fwnode_handle *fwnode, char *buf,
char *end)
{
int depth;
/* Loop starting from the root node to the current node. */
for (depth = fwnode_count_parents(fwnode); depth >= 0; depth--) {
/*
* Only get a reference for other nodes (i.e. parent nodes).
* fwnode refcount may be 0 here.
*/
struct fwnode_handle *__fwnode = depth ?
fwnode_get_nth_parent(fwnode, depth) : fwnode;
buf = string(buf, end, fwnode_get_name_prefix(__fwnode),
default_str_spec);
buf = string(buf, end, fwnode_get_name(__fwnode),
default_str_spec);
if (depth)
fwnode_handle_put(__fwnode);
}
return buf;
}
static noinline_for_stack
char *device_node_string(char *buf, char *end, struct device_node *dn,
struct printf_spec spec, const char *fmt)
{
char tbuf[sizeof("xxxx") + 1];
const char *p;
int ret;
char *buf_start = buf;
struct property *prop;
bool has_mult, pass;
struct printf_spec str_spec = spec;
str_spec.field_width = -1;
if (fmt[0] != 'F')
return error_string(buf, end, "(%pO?)", spec);
if (!IS_ENABLED(CONFIG_OF))
return error_string(buf, end, "(%pOF?)", spec);
if (check_pointer(&buf, end, dn, spec))
return buf;
/* simple case without anything any more format specifiers */
fmt++;
if (fmt[0] == '\0' || strcspn(fmt,"fnpPFcC") > 0)
fmt = "f";
for (pass = false; strspn(fmt,"fnpPFcC"); fmt++, pass = true) {
int precision;
if (pass) {
if (buf < end)
*buf = ':';
buf++;
}
switch (*fmt) {
case 'f': /* full_name */
buf = fwnode_full_name_string(of_fwnode_handle(dn), buf,
end);
break;
case 'n': /* name */
p = fwnode_get_name(of_fwnode_handle(dn));
precision = str_spec.precision;
str_spec.precision = strchrnul(p, '@') - p;
buf = string(buf, end, p, str_spec);
str_spec.precision = precision;
break;
case 'p': /* phandle */
buf = number(buf, end, (unsigned int)dn->phandle, default_dec_spec);
break;
case 'P': /* path-spec */
p = fwnode_get_name(of_fwnode_handle(dn));
if (!p[1])
p = "/";
buf = string(buf, end, p, str_spec);
break;
case 'F': /* flags */
tbuf[0] = of_node_check_flag(dn, OF_DYNAMIC) ? 'D' : '-';
tbuf[1] = of_node_check_flag(dn, OF_DETACHED) ? 'd' : '-';
tbuf[2] = of_node_check_flag(dn, OF_POPULATED) ? 'P' : '-';
tbuf[3] = of_node_check_flag(dn, OF_POPULATED_BUS) ? 'B' : '-';
tbuf[4] = 0;
buf = string_nocheck(buf, end, tbuf, str_spec);
break;
case 'c': /* major compatible string */
ret = of_property_read_string(dn, "compatible", &p);
if (!ret)
buf = string(buf, end, p, str_spec);
break;
case 'C': /* full compatible string */
has_mult = false;
of_property_for_each_string(dn, "compatible", prop, p) {
if (has_mult)
buf = string_nocheck(buf, end, ",", str_spec);
buf = string_nocheck(buf, end, "\"", str_spec);
buf = string(buf, end, p, str_spec);
buf = string_nocheck(buf, end, "\"", str_spec);
has_mult = true;
}
break;
default:
break;
}
}
return widen_string(buf, buf - buf_start, end, spec);
}
static noinline_for_stack
char *fwnode_string(char *buf, char *end, struct fwnode_handle *fwnode,
struct printf_spec spec, const char *fmt)
{
struct printf_spec str_spec = spec;
char *buf_start = buf;
str_spec.field_width = -1;
if (*fmt != 'w')
return error_string(buf, end, "(%pf?)", spec);
if (check_pointer(&buf, end, fwnode, spec))
return buf;
fmt++;
switch (*fmt) {
case 'P': /* name */
buf = string(buf, end, fwnode_get_name(fwnode), str_spec);
break;
case 'f': /* full_name */
default:
buf = fwnode_full_name_string(fwnode, buf, end);
break;
}
return widen_string(buf, buf - buf_start, end, spec);
}
static noinline_for_stack
char *resource_or_range(const char *fmt, char *buf, char *end, void *ptr,
struct printf_spec spec)
{
if (*fmt == 'r' && fmt[1] == 'a')
return range_string(buf, end, ptr, spec, fmt);
return resource_string(buf, end, ptr, spec, fmt);
}
void __init hash_pointers_finalize(bool slub_debug)
{
switch (hash_pointers_mode) {
case HASH_PTR_ALWAYS:
no_hash_pointers = false;
break;
case HASH_PTR_NEVER:
no_hash_pointers = true;
break;
case HASH_PTR_AUTO:
default:
no_hash_pointers = slub_debug;
break;
}
if (!no_hash_pointers)
return;
pr_warn("**********************************************************\n");
pr_warn("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n");
pr_warn("** **\n");
pr_warn("** This system shows unhashed kernel memory addresses **\n");
pr_warn("** via the console, logs, and other interfaces. This **\n");
pr_warn("** might reduce the security of your system. **\n");
pr_warn("** **\n");
pr_warn("** If you see this message and you are not debugging **\n");
pr_warn("** the kernel, report this immediately to your system **\n");
pr_warn("** administrator! **\n");
pr_warn("** **\n");
pr_warn("** Use hash_pointers=always to force this mode off **\n");
pr_warn("** **\n");
pr_warn("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n");
pr_warn("**********************************************************\n");
}
static int __init hash_pointers_mode_parse(char *str)
{
if (!str) {
pr_warn("Hash pointers mode empty; falling back to auto.\n");
hash_pointers_mode = HASH_PTR_AUTO;
} else if (strncmp(str, "auto", 4) == 0) {
pr_info("Hash pointers mode set to auto.\n");
hash_pointers_mode = HASH_PTR_AUTO;
} else if (strncmp(str, "never", 5) == 0) {
pr_info("Hash pointers mode set to never.\n");
hash_pointers_mode = HASH_PTR_NEVER;
} else if (strncmp(str, "always", 6) == 0) {
pr_info("Hash pointers mode set to always.\n");
hash_pointers_mode = HASH_PTR_ALWAYS;
} else {
pr_warn("Unknown hash_pointers mode '%s' specified; assuming auto.\n", str);
hash_pointers_mode = HASH_PTR_AUTO;
}
return 0;
}
early_param("hash_pointers", hash_pointers_mode_parse);
static int __init no_hash_pointers_enable(char *str)
{
return hash_pointers_mode_parse("never");
}
early_param("no_hash_pointers", no_hash_pointers_enable);
/*
* Show a '%p' thing. A kernel extension is that the '%p' is followed
* by an extra set of alphanumeric characters that are extended format
* specifiers.
*
* Please update scripts/checkpatch.pl when adding/removing conversion
* characters. (Search for "check for vsprintf extension").
*
* Right now we handle:
*
* - 'S' For symbolic direct pointers (or function descriptors) with offset
* - 's' For symbolic direct pointers (or function descriptors) without offset
* - '[Ss]R' as above with __builtin_extract_return_addr() translation
* - 'S[R]b' as above with module build ID (for use in backtraces)
* - '[Ff]' %pf and %pF were obsoleted and later removed in favor of
* %ps and %pS. Be careful when re-using these specifiers.
* - 'B' For backtraced symbolic direct pointers with offset
* - 'Bb' as above with module build ID (for use in backtraces)
* - 'R' For decoded struct resource, e.g., [mem 0x0-0x1f 64bit pref]
* - 'r' For raw struct resource, e.g., [mem 0x0-0x1f flags 0x201]
* - 'ra' For struct ranges, e.g., [range 0x0000000000000000 - 0x00000000000000ff]
* - 'b[l]' For a bitmap, the number of bits is determined by the field
* width which must be explicitly specified either as part of the
* format string '%32b[l]' or through '%*b[l]', [l] selects
* range-list format instead of hex format
* - 'M' For a 6-byte MAC address, it prints the address in the
* usual colon-separated hex notation
* - 'm' For a 6-byte MAC address, it prints the hex address without colons
* - 'MF' For a 6-byte MAC FDDI address, it prints the address
* with a dash-separated hex notation
* - '[mM]R' For a 6-byte MAC address, Reverse order (Bluetooth)
* - 'I' [46] for IPv4/IPv6 addresses printed in the usual way
* IPv4 uses dot-separated decimal without leading 0's (1.2.3.4)
* IPv6 uses colon separated network-order 16 bit hex with leading 0's
* [S][pfs]
* Generic IPv4/IPv6 address (struct sockaddr *) that falls back to
* [4] or [6] and is able to print port [p], flowinfo [f], scope [s]
* - 'i' [46] for 'raw' IPv4/IPv6 addresses
* IPv6 omits the colons (01020304...0f)
* IPv4 uses dot-separated decimal with leading 0's (010.123.045.006)
* [S][pfs]
* Generic IPv4/IPv6 address (struct sockaddr *) that falls back to
* [4] or [6] and is able to print port [p], flowinfo [f], scope [s]
* - '[Ii][4S][hnbl]' IPv4 addresses in host, network, big or little endian order
* - 'I[6S]c' for IPv6 addresses printed as specified by
* https://tools.ietf.org/html/rfc5952
* - 'E[achnops]' For an escaped buffer, where rules are defined by combination
* of the following flags (see string_escape_mem() for the
* details):
* a - ESCAPE_ANY
* c - ESCAPE_SPECIAL
* h - ESCAPE_HEX
* n - ESCAPE_NULL
* o - ESCAPE_OCTAL
* p - ESCAPE_NP
* s - ESCAPE_SPACE
* By default ESCAPE_ANY_NP is used.
* - 'U' For a 16 byte UUID/GUID, it prints the UUID/GUID in the form
* "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
* Options for %pU are:
* b big endian lower case hex (default)
* B big endian UPPER case hex
* l little endian lower case hex
* L little endian UPPER case hex
* big endian output byte order is:
* [0][1][2][3]-[4][5]-[6][7]-[8][9]-[10][11][12][13][14][15]
* little endian output byte order is:
* [3][2][1][0]-[5][4]-[7][6]-[8][9]-[10][11][12][13][14][15]
* - 'V' For a struct va_format which contains a format string * and va_list *,
* call vsnprintf(->format, *->va_list).
* Implements a "recursive vsnprintf".
* Do not use this feature without some mechanism to verify the
* correctness of the format string and va_list arguments.
* - 'K' For a kernel pointer that should be hidden from unprivileged users.
* Use only for procfs, sysfs and similar files, not printk(); please
* read the documentation (path below) first.
* - 'NF' For a netdev_features_t
* - '4cc' V4L2 or DRM FourCC code, with endianness and raw numerical value.
* - '4c[h[R]lb]' For generic FourCC code with raw numerical value. Both are
* displayed in the big-endian format. This is the opposite of V4L2 or
* DRM FourCCs.
* The additional specifiers define what endianness is used to load
* the stored bytes. The data might be interpreted using the host,
* reversed host byte order, little-endian, or big-endian.
* - 'h[CDN]' For a variable-length buffer, it prints it as a hex string with
* a certain separator (' ' by default):
* C colon
* D dash
* N no separator
* The maximum supported length is 64 bytes of the input. Consider
* to use print_hex_dump() for the larger input.
* - 'a[pd]' For address types [p] phys_addr_t, [d] dma_addr_t and derivatives
* (default assumed to be phys_addr_t, passed by reference)
* - 'd[234]' For a dentry name (optionally 2-4 last components)
* - 'D[234]' Same as 'd' but for a struct file
* - 'g' For block_device name (gendisk + partition number)
* - 't[RST][dt][r][s]' For time and date as represented by:
* R struct rtc_time
* S struct timespec64
* T time64_t
* - 'tSp' For time represented by struct timespec64 printed as <seconds>.<nanoseconds>
* - 'C' For a clock, it prints the name (Common Clock Framework) or address
* (legacy clock framework) of the clock
* - 'G' For flags to be printed as a collection of symbolic strings that would
* construct the specific value. Supported flags given by option:
* p page flags (see struct page) given as pointer to unsigned long
* g gfp flags (GFP_* and __GFP_*) given as pointer to gfp_t
* v vma flags (VM_*) given as pointer to unsigned long
* - 'OF[fnpPcCF]' For a device tree object
* Without any optional arguments prints the full_name
* f device node full_name
* n device node name
* p device node phandle
* P device node path spec (name + @unit)
* F device node flags
* c major compatible string
* C full compatible string
* - 'fw[fP]' For a firmware node (struct fwnode_handle) pointer
* Without an option prints the full name of the node
* f full name
* P node name, including a possible unit address
* - 'x' For printing the address unmodified. Equivalent to "%lx".
* Please read the documentation (path below) before using!
* - '[ku]s' For a BPF/tracing related format specifier, e.g. used out of
* bpf_trace_printk() where [ku] prefix specifies either kernel (k)
* or user (u) memory to probe, and:
* s a string, equivalent to "%s" on direct vsnprintf() use
*
* ** When making changes please also update:
* Documentation/core-api/printk-formats.rst
*
* Note: The default behaviour (unadorned %p) is to hash the address,
* rendering it useful as a unique identifier.
*
* There is also a '%pA' format specifier, but it is only intended to be used
* from Rust code to format core::fmt::Arguments. Do *not* use it from C.
* See rust/kernel/print.rs for details.
*/
static noinline_for_stack
char *pointer(const char *fmt, char *buf, char *end, void *ptr,
struct printf_spec spec)
{
switch (*fmt) {
case 'S':
case 's':
ptr = dereference_symbol_descriptor(ptr);
fallthrough;
case 'B':
return symbol_string(buf, end, ptr, spec, fmt);
case 'R':
case 'r':
return resource_or_range(fmt, buf, end, ptr, spec);
case 'h':
return hex_string(buf, end, ptr, spec, fmt);
case 'b':
switch (fmt[1]) {
case 'l':
return bitmap_list_string(buf, end, ptr, spec, fmt);
default:
return bitmap_string(buf, end, ptr, spec, fmt);
}
case 'M': /* Colon separated: 00:01:02:03:04:05 */
case 'm': /* Contiguous: 000102030405 */
/* [mM]F (FDDI) */
/* [mM]R (Reverse order; Bluetooth) */
return mac_address_string(buf, end, ptr, spec, fmt);
case 'I': /* Formatted IP supported
* 4: 1.2.3.4
* 6: 0001:0203:...:0708
* 6c: 1::708 or 1::1.2.3.4
*/
case 'i': /* Contiguous:
* 4: 001.002.003.004
* 6: 000102...0f
*/
return ip_addr_string(buf, end, ptr, spec, fmt);
case 'E':
return escaped_string(buf, end, ptr, spec, fmt);
case 'U':
return uuid_string(buf, end, ptr, spec, fmt);
case 'V':
return va_format(buf, end, ptr, spec);
case 'K':
return restricted_pointer(buf, end, ptr, spec);
case 'N':
return netdev_bits(buf, end, ptr, spec, fmt);
case '4':
return fourcc_string(buf, end, ptr, spec, fmt);
case 'a':
return address_val(buf, end, ptr, spec, fmt);
case 'd':
return dentry_name(buf, end, ptr, spec, fmt);
case 't':
return time_and_date(buf, end, ptr, spec, fmt);
case 'C':
return clock(buf, end, ptr, spec, fmt);
case 'D':
return file_dentry_name(buf, end, ptr, spec, fmt);
#ifdef CONFIG_BLOCK
case 'g':
return bdev_name(buf, end, ptr, spec, fmt);
#endif
case 'G':
return flags_string(buf, end, ptr, spec, fmt);
case 'O':
return device_node_string(buf, end, ptr, spec, fmt + 1);
case 'f':
return fwnode_string(buf, end, ptr, spec, fmt + 1);
case 'A':
if (!IS_ENABLED(CONFIG_RUST)) {
WARN_ONCE(1, "Please remove %%pA from non-Rust code\n");
return error_string(buf, end, "(%pA?)", spec);
}
return rust_fmt_argument(buf, end, ptr);
case 'x':
return pointer_string(buf, end, ptr, spec);
case 'e':
/* %pe with a non-ERR_PTR gets treated as plain %p */
if (!IS_ERR(ptr))
return default_pointer(buf, end, ptr, spec);
return err_ptr(buf, end, ptr, spec);
case 'u':
case 'k':
switch (fmt[1]) {
case 's':
return string(buf, end, ptr, spec);
default:
return error_string(buf, end, "(einval)", spec);
}
default:
return default_pointer(buf, end, ptr, spec);
}
}
struct fmt {
const char *str;
unsigned char state; // enum format_state
unsigned char size; // size of numbers
};
#define SPEC_CHAR(x, flag) [(x)-32] = flag
static unsigned char spec_flag(unsigned char c)
{
static const unsigned char spec_flag_array[] = {
SPEC_CHAR(' ', SPACE),
SPEC_CHAR('#', SPECIAL),
SPEC_CHAR('+', PLUS),
SPEC_CHAR('-', LEFT),
SPEC_CHAR('0', ZEROPAD),
};
c -= 32;
return (c < sizeof(spec_flag_array)) ? spec_flag_array[c] : 0;
}
/*
* Helper function to decode printf style format.
* Each call decode a token from the format and return the
* number of characters read (or likely the delta where it wants
* to go on the next call).
* The decoded token is returned through the parameters
*
* 'h', 'l', or 'L' for integer fields
* 'z' support added 23/7/1999 S.H.
* 'z' changed to 'Z' --davidm 1/25/99
* 'Z' changed to 'z' --adobriyan 2017-01-25
* 't' added for ptrdiff_t
*
* @fmt: the format string
* @type of the token returned
* @flags: various flags such as +, -, # tokens..
* @field_width: overwritten width
* @base: base of the number (octal, hex, ...)
* @precision: precision of a number
* @qualifier: qualifier of a number (long, size_t, ...)
*/
static noinline_for_stack
struct fmt format_decode(struct fmt fmt, struct printf_spec *spec)
{
const char *start = fmt.str;
char flag;
/* we finished early by reading the field width */
if (unlikely(fmt.state == FORMAT_STATE_WIDTH)) {
if (spec->field_width < 0) {
spec->field_width = -spec->field_width;
spec->flags |= LEFT;
}
fmt.state = FORMAT_STATE_NONE;
goto precision;
}
/* we finished early by reading the precision */
if (unlikely(fmt.state == FORMAT_STATE_PRECISION)) {
if (spec->precision < 0)
spec->precision = 0;
fmt.state = FORMAT_STATE_NONE;
goto qualifier;
}
/* By default */
fmt.state = FORMAT_STATE_NONE;
for (; *fmt.str ; fmt.str++) {
if (*fmt.str == '%')
break;
}
/* Return the current non-format string */
if (fmt.str != start || !*fmt.str)
return fmt;
/* Process flags. This also skips the first '%' */
spec->flags = 0;
do {
/* this also skips first '%' */
flag = spec_flag(*++fmt.str);
spec->flags |= flag;
} while (flag);
/* get field width */
spec->field_width = -1;
if (isdigit(*fmt.str))
spec->field_width = skip_atoi(&fmt.str);
else if (unlikely(*fmt.str == '*')) {
/* it's the next argument */
fmt.state = FORMAT_STATE_WIDTH;
fmt.str++;
return fmt;
}
precision:
/* get the precision */
spec->precision = -1;
if (unlikely(*fmt.str == '.')) {
fmt.str++;
if (isdigit(*fmt.str)) {
spec->precision = skip_atoi(&fmt.str);
if (spec->precision < 0)
spec->precision = 0;
} else if (*fmt.str == '*') {
/* it's the next argument */
fmt.state = FORMAT_STATE_PRECISION;
fmt.str++;
return fmt;
}
}
qualifier:
/* Set up default numeric format */
spec->base = 10;
fmt.state = FORMAT_STATE_NUM;
fmt.size = sizeof(int);
static const struct format_state {
unsigned char state;
unsigned char size;
unsigned char flags_or_double_size;
unsigned char base;
} lookup_state[256] = {
// Length
['l'] = { 0, sizeof(long), sizeof(long long) },
['L'] = { 0, sizeof(long long) },
['h'] = { 0, sizeof(short), sizeof(char) },
['H'] = { 0, sizeof(char) }, // Questionable historical
['z'] = { 0, sizeof(size_t) },
['t'] = { 0, sizeof(ptrdiff_t) },
// Non-numeric formats
['c'] = { FORMAT_STATE_CHAR },
['s'] = { FORMAT_STATE_STR },
['p'] = { FORMAT_STATE_PTR },
['%'] = { FORMAT_STATE_PERCENT_CHAR },
// Numerics
['o'] = { FORMAT_STATE_NUM, 0, 0, 8 },
['x'] = { FORMAT_STATE_NUM, 0, SMALL, 16 },
['X'] = { FORMAT_STATE_NUM, 0, 0, 16 },
['d'] = { FORMAT_STATE_NUM, 0, SIGN, 10 },
['i'] = { FORMAT_STATE_NUM, 0, SIGN, 10 },
['u'] = { FORMAT_STATE_NUM, 0, 0, 10, },
/*
* Since %n poses a greater security risk than
* utility, treat it as any other invalid or
* unsupported format specifier.
*/
};
const struct format_state *p = lookup_state + (u8)*fmt.str;
if (p->size) {
fmt.size = p->size;
if (p->flags_or_double_size && fmt.str[0] == fmt.str[1]) {
fmt.size = p->flags_or_double_size;
fmt.str++;
}
fmt.str++;
p = lookup_state + *fmt.str;
}
if (p->state) {
if (p->base)
spec->base = p->base;
spec->flags |= p->flags_or_double_size;
fmt.state = p->state;
fmt.str++;
return fmt;
}
WARN_ONCE(1, "Please remove unsupported %%%c in format string\n", *fmt.str);
fmt.state = FORMAT_STATE_INVALID;
return fmt;
}
static void
set_field_width(struct printf_spec *spec, int width)
{
spec->field_width = width;
if (WARN_ONCE(spec->field_width != width, "field width %d too large", width)) {
spec->field_width = clamp(width, -FIELD_WIDTH_MAX, FIELD_WIDTH_MAX);
}
}
static void
set_precision(struct printf_spec *spec, int prec)
{
spec->precision = prec;
if (WARN_ONCE(spec->precision != prec, "precision %d too large", prec)) {
spec->precision = clamp(prec, 0, PRECISION_MAX);
}
}
/*
* Turn a 1/2/4-byte value into a 64-bit one for printing: truncate
* as necessary and deal with signedness.
*
* 'size' is the size of the value in bytes.
*/
static unsigned long long convert_num_spec(unsigned int val, int size, struct printf_spec spec)
{
unsigned int shift = 32 - size*8;
val <<= shift;
if (!(spec.flags & SIGN))
return val >> shift;
return (int)val >> shift;
}
/**
* vsnprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt_str: The format string to use
* @args: Arguments for the format string
*
* This function generally follows C99 vsnprintf, but has some
* extensions and a few limitations:
*
* - ``%n`` is unsupported
* - ``%p*`` is handled by pointer()
*
* See pointer() or Documentation/core-api/printk-formats.rst for more
* extensive description.
*
* **Please update the documentation in both places when making changes**
*
* The return value is the number of characters which would
* be generated for the given input, excluding the trailing
* '\0', as per ISO C99. If you want to have the exact
* number of characters written into @buf as return value
* (not including the trailing '\0'), use vscnprintf(). If the
* return is greater than or equal to @size, the resulting
* string is truncated.
*
* If you're not already dealing with a va_list consider using snprintf().
*/
int vsnprintf(char *buf, size_t size, const char *fmt_str, va_list args)
{
char *str, *end;
struct printf_spec spec = {0};
struct fmt fmt = {
.str = fmt_str,
.state = FORMAT_STATE_NONE,
};
/* Reject out-of-range values early. Large positive sizes are
used for unknown buffer sizes. */
if (WARN_ON_ONCE(size > INT_MAX))
return 0;
str = buf;
end = buf + size;
/* Make sure end is always >= buf */
if (end < buf) {
end = ((void *)-1);
size = end - buf;
}
while (*fmt.str) {
const char *old_fmt = fmt.str;
fmt = format_decode(fmt, &spec);
switch (fmt.state) {
case FORMAT_STATE_NONE: {
int read = fmt.str - old_fmt;
if (str < end) {
int copy = read;
if (copy > end - str)
copy = end - str;
memcpy(str, old_fmt, copy);
}
str += read;
continue;
}
case FORMAT_STATE_NUM: {
unsigned long long num;
if (fmt.size > sizeof(int))
num = va_arg(args, long long);
else
num = convert_num_spec(va_arg(args, int), fmt.size, spec);
str = number(str, end, num, spec);
continue;
}
case FORMAT_STATE_WIDTH:
set_field_width(&spec, va_arg(args, int));
continue;
case FORMAT_STATE_PRECISION:
set_precision(&spec, va_arg(args, int));
continue;
case FORMAT_STATE_CHAR: {
char c;
if (!(spec.flags & LEFT)) {
while (--spec.field_width > 0) {
if (str < end)
*str = ' ';
++str;
}
}
c = (unsigned char) va_arg(args, int);
if (str < end)
*str = c;
++str;
while (--spec.field_width > 0) {
if (str < end)
*str = ' ';
++str;
}
continue;
}
case FORMAT_STATE_STR:
str = string(str, end, va_arg(args, char *), spec);
continue;
case FORMAT_STATE_PTR:
str = pointer(fmt.str, str, end, va_arg(args, void *),
spec);
while (isalnum(*fmt.str))
fmt.str++;
continue;
case FORMAT_STATE_PERCENT_CHAR:
if (str < end)
*str = '%';
++str;
continue;
default:
/*
* Presumably the arguments passed gcc's type
* checking, but there is no safe or sane way
* for us to continue parsing the format and
* fetching from the va_list; the remaining
* specifiers and arguments would be out of
* sync.
*/
goto out;
}
}
out:
if (size > 0) {
if (str < end)
*str = '\0';
else
end[-1] = '\0';
}
/* the trailing null byte doesn't count towards the total */
return str-buf;
}
EXPORT_SYMBOL(vsnprintf);
/**
* vscnprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt: The format string to use
* @args: Arguments for the format string
*
* The return value is the number of characters which have been written into
* the @buf not including the trailing '\0'. If @size is == 0 the function
* returns 0.
*
* If you're not already dealing with a va_list consider using scnprintf().
*
* See the vsnprintf() documentation for format string extensions over C99.
*/
int vscnprintf(char *buf, size_t size, const char *fmt, va_list args)
{
int i;
if (unlikely(!size))
return 0;
i = vsnprintf(buf, size, fmt, args);
if (likely(i < size))
return i;
return size - 1;
}
EXPORT_SYMBOL(vscnprintf);
/**
* snprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt: The format string to use
* @...: Arguments for the format string
*
* The return value is the number of characters which would be
* generated for the given input, excluding the trailing null,
* as per ISO C99. If the return is greater than or equal to
* @size, the resulting string is truncated.
*
* See the vsnprintf() documentation for format string extensions over C99.
*/
int snprintf(char *buf, size_t size, const char *fmt, ...)
{
va_list args;
int i;
va_start(args, fmt);
i = vsnprintf(buf, size, fmt, args);
va_end(args);
return i;
}
EXPORT_SYMBOL(snprintf);
/**
* scnprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt: The format string to use
* @...: Arguments for the format string
*
* The return value is the number of characters written into @buf not including
* the trailing '\0'. If @size is == 0 the function returns 0.
*/
int scnprintf(char *buf, size_t size, const char *fmt, ...)
{
va_list args;
int i;
va_start(args, fmt);
i = vscnprintf(buf, size, fmt, args);
va_end(args);
return i;
}
EXPORT_SYMBOL(scnprintf);
/**
* vsprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @fmt: The format string to use
* @args: Arguments for the format string
*
* The return value is the number of characters written into @buf not including
* the trailing '\0'. Use vsnprintf() or vscnprintf() in order to avoid
* buffer overflows.
*
* If you're not already dealing with a va_list consider using sprintf().
*
* See the vsnprintf() documentation for format string extensions over C99.
*/
int vsprintf(char *buf, const char *fmt, va_list args)
{
return vsnprintf(buf, INT_MAX, fmt, args);
}
EXPORT_SYMBOL(vsprintf);
/**
* sprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @fmt: The format string to use
* @...: Arguments for the format string
*
* The return value is the number of characters written into @buf not including
* the trailing '\0'. Use snprintf() or scnprintf() in order to avoid
* buffer overflows.
*
* See the vsnprintf() documentation for format string extensions over C99.
*/
int sprintf(char *buf, const char *fmt, ...)
{
va_list args;
int i;
va_start(args, fmt);
i = vsnprintf(buf, INT_MAX, fmt, args);
va_end(args);
return i;
}
EXPORT_SYMBOL(sprintf);
#ifdef CONFIG_BINARY_PRINTF
/*
* bprintf service:
* vbin_printf() - VA arguments to binary data
* bstr_printf() - Binary data to text string
*/
/**
* vbin_printf - Parse a format string and place args' binary value in a buffer
* @bin_buf: The buffer to place args' binary value
* @size: The size of the buffer(by words(32bits), not characters)
* @fmt_str: The format string to use
* @args: Arguments for the format string
*
* The format follows C99 vsnprintf, except %n is ignored, and its argument
* is skipped.
*
* The return value is the number of words(32bits) which would be generated for
* the given input.
*
* NOTE:
* If the return value is greater than @size, the resulting bin_buf is NOT
* valid for bstr_printf().
*/
int vbin_printf(u32 *bin_buf, size_t size, const char *fmt_str, va_list args)
{
struct fmt fmt = {
.str = fmt_str,
.state = FORMAT_STATE_NONE,
};
struct printf_spec spec = {0};
char *str, *end;
int width;
str = (char *)bin_buf;
end = (char *)(bin_buf + size);
#define save_arg(type) \
({ \
unsigned long long value; \
if (sizeof(type) == 8) { \
unsigned long long val8; \
str = PTR_ALIGN(str, sizeof(u32)); \
val8 = va_arg(args, unsigned long long); \
if (str + sizeof(type) <= end) { \
*(u32 *)str = *(u32 *)&val8; \
*(u32 *)(str + 4) = *((u32 *)&val8 + 1); \
} \
value = val8; \
} else { \
unsigned int val4; \
str = PTR_ALIGN(str, sizeof(type)); \
val4 = va_arg(args, int); \
if (str + sizeof(type) <= end) \
*(typeof(type) *)str = (type)(long)val4; \
value = (unsigned long long)val4; \
} \
str += sizeof(type); \
value; \
})
while (*fmt.str) {
fmt = format_decode(fmt, &spec);
switch (fmt.state) {
case FORMAT_STATE_NONE:
case FORMAT_STATE_PERCENT_CHAR:
break;
case FORMAT_STATE_INVALID:
goto out;
case FORMAT_STATE_WIDTH:
case FORMAT_STATE_PRECISION:
width = (int)save_arg(int);
/* Pointers may require the width */
if (*fmt.str == 'p')
set_field_width(&spec, width);
break;
case FORMAT_STATE_CHAR:
save_arg(char);
break;
case FORMAT_STATE_STR: {
const char *save_str = va_arg(args, char *);
const char *err_msg;
size_t len;
err_msg = check_pointer_msg(save_str);
if (err_msg)
save_str = err_msg;
len = strlen(save_str) + 1;
if (str + len < end)
memcpy(str, save_str, len);
str += len;
break;
}
case FORMAT_STATE_PTR:
/* Dereferenced pointers must be done now */
switch (*fmt.str) {
/* Dereference of functions is still OK */
case 'S':
case 's':
case 'x':
case 'K':
case 'e':
save_arg(void *);
break;
default:
if (!isalnum(*fmt.str)) {
save_arg(void *);
break;
}
str = pointer(fmt.str, str, end, va_arg(args, void *),
spec);
if (str + 1 < end)
*str++ = '\0';
else
end[-1] = '\0'; /* Must be nul terminated */
}
/* skip all alphanumeric pointer suffixes */
while (isalnum(*fmt.str))
fmt.str++;
break;
case FORMAT_STATE_NUM:
if (fmt.size > sizeof(int)) {
save_arg(long long);
} else {
save_arg(int);
}
}
}
out:
return (u32 *)(PTR_ALIGN(str, sizeof(u32))) - bin_buf;
#undef save_arg
}
EXPORT_SYMBOL_GPL(vbin_printf);
/**
* bstr_printf - Format a string from binary arguments and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt_str: The format string to use
* @bin_buf: Binary arguments for the format string
*
* This function like C99 vsnprintf, but the difference is that vsnprintf gets
* arguments from stack, and bstr_printf gets arguments from @bin_buf which is
* a binary buffer that generated by vbin_printf.
*
* The format follows C99 vsnprintf, but has some extensions:
* see vsnprintf comment for details.
*
* The return value is the number of characters which would
* be generated for the given input, excluding the trailing
* '\0', as per ISO C99. If you want to have the exact
* number of characters written into @buf as return value
* (not including the trailing '\0'), use vscnprintf(). If the
* return is greater than or equal to @size, the resulting
* string is truncated.
*/
int bstr_printf(char *buf, size_t size, const char *fmt_str, const u32 *bin_buf)
{
struct fmt fmt = {
.str = fmt_str,
.state = FORMAT_STATE_NONE,
};
struct printf_spec spec = {0};
char *str, *end;
const char *args = (const char *)bin_buf;
if (WARN_ON_ONCE(size > INT_MAX))
return 0;
str = buf;
end = buf + size;
#define get_arg(type) \
({ \
typeof(type) value; \
if (sizeof(type) == 8) { \
args = PTR_ALIGN(args, sizeof(u32)); \
*(u32 *)&value = *(u32 *)args; \
*((u32 *)&value + 1) = *(u32 *)(args + 4); \
} else { \
args = PTR_ALIGN(args, sizeof(type)); \
value = *(typeof(type) *)args; \
} \
args += sizeof(type); \
value; \
})
/* Make sure end is always >= buf */
if (end < buf) {
end = ((void *)-1);
size = end - buf;
}
while (*fmt.str) {
const char *old_fmt = fmt.str;
unsigned long long num;
fmt = format_decode(fmt, &spec);
switch (fmt.state) {
case FORMAT_STATE_NONE: {
int read = fmt.str - old_fmt;
if (str < end) {
int copy = read;
if (copy > end - str)
copy = end - str;
memcpy(str, old_fmt, copy);
}
str += read;
continue;
}
case FORMAT_STATE_WIDTH:
set_field_width(&spec, get_arg(int));
continue;
case FORMAT_STATE_PRECISION:
set_precision(&spec, get_arg(int));
continue;
case FORMAT_STATE_CHAR: {
char c;
if (!(spec.flags & LEFT)) {
while (--spec.field_width > 0) {
if (str < end)
*str = ' ';
++str;
}
}
c = (unsigned char) get_arg(char);
if (str < end)
*str = c;
++str;
while (--spec.field_width > 0) {
if (str < end)
*str = ' ';
++str;
}
continue;
}
case FORMAT_STATE_STR: {
const char *str_arg = args;
args += strlen(str_arg) + 1;
str = string(str, end, (char *)str_arg, spec);
continue;
}
case FORMAT_STATE_PTR: {
bool process = false;
int copy, len;
/* Non function dereferences were already done */
switch (*fmt.str) {
case 'S':
case 's':
case 'x':
case 'K':
case 'e':
process = true;
break;
default:
if (!isalnum(*fmt.str)) {
process = true;
break;
}
/* Pointer dereference was already processed */
if (str < end) {
len = copy = strlen(args);
if (copy > end - str)
copy = end - str;
memcpy(str, args, copy);
str += len;
args += len + 1;
}
}
if (process)
str = pointer(fmt.str, str, end, get_arg(void *), spec);
while (isalnum(*fmt.str))
fmt.str++;
continue;
}
case FORMAT_STATE_PERCENT_CHAR:
if (str < end)
*str = '%';
++str;
continue;
case FORMAT_STATE_INVALID:
goto out;
case FORMAT_STATE_NUM:
if (fmt.size > sizeof(int))
num = get_arg(long long);
else
num = convert_num_spec(get_arg(int), fmt.size, spec);
str = number(str, end, num, spec);
continue;
}
} /* while(*fmt.str) */
out:
if (size > 0) {
if (str < end)
*str = '\0';
else
end[-1] = '\0';
}
#undef get_arg
/* the trailing null byte doesn't count towards the total */
return str - buf;
}
EXPORT_SYMBOL_GPL(bstr_printf);
#endif /* CONFIG_BINARY_PRINTF */
/**
* vsscanf - Unformat a buffer into a list of arguments
* @buf: input buffer
* @fmt: format of buffer
* @args: arguments
*/
int vsscanf(const char *buf, const char *fmt, va_list args)
{
const char *str = buf;
char *next;
char digit;
int num = 0;
u8 qualifier;
unsigned int base;
union {
long long s;
unsigned long long u;
} val;
s16 field_width;
bool is_sign;
while (*fmt) {
/* skip any white space in format */
/* white space in format matches any amount of
* white space, including none, in the input.
*/
if (isspace(*fmt)) {
fmt = skip_spaces(++fmt);
str = skip_spaces(str);
}
/* anything that is not a conversion must match exactly */
if (*fmt != '%' && *fmt) {
if (*fmt++ != *str++)
break;
continue;
}
if (!*fmt)
break;
++fmt;
/* skip this conversion.
* advance both strings to next white space
*/
if (*fmt == '*') {
if (!*str)
break;
while (!isspace(*fmt) && *fmt != '%' && *fmt) {
/* '%*[' not yet supported, invalid format */
if (*fmt == '[')
return num;
fmt++;
}
while (!isspace(*str) && *str)
str++;
continue;
}
/* get field width */
field_width = -1;
if (isdigit(*fmt)) {
field_width = skip_atoi(&fmt);
if (field_width <= 0)
break;
}
/* get conversion qualifier */
qualifier = -1;
if (*fmt == 'h' || _tolower(*fmt) == 'l' ||
*fmt == 'z') {
qualifier = *fmt++;
if (unlikely(qualifier == *fmt)) {
if (qualifier == 'h') {
qualifier = 'H';
fmt++;
} else if (qualifier == 'l') {
qualifier = 'L';
fmt++;
}
}
}
if (!*fmt)
break;
if (*fmt == 'n') {
/* return number of characters read so far */
*va_arg(args, int *) = str - buf;
++fmt;
continue;
}
if (!*str)
break;
base = 10;
is_sign = false;
switch (*fmt++) {
case 'c':
{
char *s = (char *)va_arg(args, char*);
if (field_width == -1)
field_width = 1;
do {
*s++ = *str++;
} while (--field_width > 0 && *str);
num++;
}
continue;
case 's':
{
char *s = (char *)va_arg(args, char *);
if (field_width == -1)
field_width = SHRT_MAX;
/* first, skip leading white space in buffer */
str = skip_spaces(str);
/* now copy until next white space */
while (*str && !isspace(*str) && field_width--)
*s++ = *str++;
*s = '\0';
num++;
}
continue;
/*
* Warning: This implementation of the '[' conversion specifier
* deviates from its glibc counterpart in the following ways:
* (1) It does NOT support ranges i.e. '-' is NOT a special
* character
* (2) It cannot match the closing bracket ']' itself
* (3) A field width is required
* (4) '%*[' (discard matching input) is currently not supported
*
* Example usage:
* ret = sscanf("00:0a:95","%2[^:]:%2[^:]:%2[^:]",
* buf1, buf2, buf3);
* if (ret < 3)
* // etc..
*/
case '[':
{
char *s = (char *)va_arg(args, char *);
DECLARE_BITMAP(set, 256) = {0};
unsigned int len = 0;
bool negate = (*fmt == '^');
/* field width is required */
if (field_width == -1)
return num;
if (negate)
++fmt;
for ( ; *fmt && *fmt != ']'; ++fmt, ++len)
__set_bit((u8)*fmt, set);
/* no ']' or no character set found */
if (!*fmt || !len)
return num;
++fmt;
if (negate) {
bitmap_complement(set, set, 256);
/* exclude null '\0' byte */
__clear_bit(0, set);
}
/* match must be non-empty */
if (!test_bit((u8)*str, set))
return num;
while (test_bit((u8)*str, set) && field_width--)
*s++ = *str++;
*s = '\0';
++num;
}
continue;
case 'o':
base = 8;
break;
case 'x':
case 'X':
base = 16;
break;
case 'i':
base = 0;
fallthrough;
case 'd':
is_sign = true;
fallthrough;
case 'u':
break;
case '%':
/* looking for '%' in str */
if (*str++ != '%')
return num;
continue;
default:
/* invalid format; stop here */
return num;
}
/* have some sort of integer conversion.
* first, skip white space in buffer.
*/
str = skip_spaces(str);
digit = *str;
if (is_sign && digit == '-') {
if (field_width == 1)
break;
digit = *(str + 1);
}
if (!digit
|| (base == 16 && !isxdigit(digit))
|| (base == 10 && !isdigit(digit))
|| (base == 8 && !isodigit(digit))
|| (base == 0 && !isdigit(digit)))
break;
if (is_sign)
val.s = simple_strntoll(str, &next, base,
field_width >= 0 ? field_width : INT_MAX);
else
val.u = simple_strntoull(str, &next, base,
field_width >= 0 ? field_width : INT_MAX);
switch (qualifier) {
case 'H': /* that's 'hh' in format */
if (is_sign)
*va_arg(args, signed char *) = val.s;
else
*va_arg(args, unsigned char *) = val.u;
break;
case 'h':
if (is_sign)
*va_arg(args, short *) = val.s;
else
*va_arg(args, unsigned short *) = val.u;
break;
case 'l':
if (is_sign)
*va_arg(args, long *) = val.s;
else
*va_arg(args, unsigned long *) = val.u;
break;
case 'L':
if (is_sign)
*va_arg(args, long long *) = val.s;
else
*va_arg(args, unsigned long long *) = val.u;
break;
case 'z':
*va_arg(args, size_t *) = val.u;
break;
default:
if (is_sign)
*va_arg(args, int *) = val.s;
else
*va_arg(args, unsigned int *) = val.u;
break;
}
num++;
if (!next)
break;
str = next;
}
return num;
}
EXPORT_SYMBOL(vsscanf);
/**
* sscanf - Unformat a buffer into a list of arguments
* @buf: input buffer
* @fmt: formatting of buffer
* @...: resulting arguments
*/
int sscanf(const char *buf, const char *fmt, ...)
{
va_list args;
int i;
va_start(args, fmt);
i = vsscanf(buf, fmt, args);
va_end(args);
return i;
}
EXPORT_SYMBOL(sscanf);
]
|
| 663/6 |
2026/05/18 20:25 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 51894
output: 23
thoughts: 1955
Thoughts: **Analysis of `vbin_printf` and `bstr_printf` Handling of `%pB`**
Okay, I've dug into the code for `vbin_printf` and `bstr_printf` to understand how they handle the `%pB` format specifier, and I think I've got a grasp of the mechanics. My primary observation is that `bstr_printf` *expects* a pre-formatted string when it encounters `%pB`, which is generated by the earlier call to `vbin_printf`.
Let me break down the process. In `vbin_printf`, specifically within the `FORMAT_STATE_PTR` case, lines 3217-3239, when `%pB` is encountered (because the format string character is 'B' and not in the 'still OK' list), the code calls the `pointer()` function. This, in turn, calls `symbol_string()`, which performs the symbol lookup and formatting, placing the result directly into the `bin_buf` buffer and null-terminating it.
Now, in `bstr_printf` (lines 3282-3440), when it encounters `%pB`, it arrives at line 3391, skipping a section where the 'still OK' cases are handled. Instead of doing the formatting, it essentially treats the corresponding argument (`args`) as a string pointer (line 3393). It assumes this pointer holds the pre-formatted string. `bstr_printf` copies this string from the buffer (`args`) into its output buffer. It is the expected behavior, which is why it increments `args` by `len + 1` to skip over the now-copied null-terminated string.
The crux of the matter is that `vbin_printf` does the symbol lookup and formatting up front for `%pB`, unlike something like `%pS`, where it simply saves the pointer, deferring the formatting to the `bstr_printf`.
But this brings up the question of *why* this distinction exists between `%pB` and `%pS`. `%pB` is clearly intended for backtrace formatting. Looking deeper in the code within `symbol_string()` (lines 1003 onwards), there are format specifiers like `Bb` and `S`, but I noticed that `B` is used for backtraces. The logic is that if `vbin_printf` does the backtrace lookup, it's using the current state of kallsyms at the time of the `vbin_printf` call.
This design is likely linked to the intended usage of `vbin_printf` and `bstr_printf`. `vbin_printf` is often used in contexts where we want to minimize work, such as in NMIs or high-frequency tracepoints, while `bstr_printf` is called later, when reading the trace buffer. But by doing the formatting for `%pB` in `vbin_printf`, it seems we're contradicting the "minimize work" design principle.
Then I saw the comment: `/* Dereferenced pointers must be done now */`. Also, `/* Dereference of functions is still OK */`. The kernel developers made a decision to process data pointed to immediately if that data might be transient.
But even with this in mind, `%pB` refers to an instruction address, similar to `%pS`.
I'm starting to think that the reason behind this could be simply that `%pB` wasn't added with this design decision in mind. This distinction feels a bit... inconsistent, or perhaps a minor oversight during the development of this printf variant. I'll search for the commit that introduced `%pB` to see if it sheds any light on this.
|
| 664/6 |
2026/05/18 20:25 |
tool |
codesearch-find-references |
0m
Args: map[Name:sprint_backtrace]
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:symbol_string SourceFile:lib/vsprintf.c SourceLine:1007]] TruncatedOutput:false]
|
| 665/6 |
2026/05/18 20:25 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 54749
output: 42
thoughts: 988
Thoughts: **Analyzing the Handling of `%pB` in `vsprintf`**
Okay, I need to understand how `%pB` is handled within `lib/vsprintf.c`. My first instinct was to check the commit history to see when it was added and if the related functions were updated, but I can bypass that by just checking if it is explicitly handled within `vbin_printf`. Nope, it is not. Since it falls into the `default` case, it's immediately processed by the `pointer()` function.
Hmm, let me double-check `vbin_printf`'s `switch` statement for the "still OK" list. It includes 'S', 's', 'x', 'K', and 'e'. Wait, isn't `%px` just hex representation? Correct, `pointer_string` prints the address. It doesn't dereference, so deferring is safe. `%pK` depends on `kptr_restrict`, `%pe` calls `errname()`, and `%pS` and `%ps` both call `sprint_symbol`.
So, why isn't `%pB` there? `sprint_backtrace` is what `%pB` uses. That's very similar to `sprint_symbol`, so perhaps the original author just forgot to update the lists in `vbin_printf` and `bstr_printf` when `%pB` was introduced.
Let's confirm that by checking `bstr_printf` and `vbin_printf` again. Yep, both are missing 'B'. This means `%pB` is formatted immediately in `vbin_printf` and `bstr_printf` only copies the string, and this is happening because it falls into the `default` case.
Is this intentional? `%pB` is for "backtraced symbolic direct pointers with offset," according to the docs. If I examine `symbol_string`, it indeed calls `sprint_backtrace` when it encounters 'B'. And, `sprint_backtrace` is defined in `kernel/kallsyms.c`, that's where I need to look to really understand what it does.
|
| 666/6 |
2026/05/18 20:25 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/kallsyms.c IncludeLines:true Name:sprint_backtrace]
Results: map[SourceCode: 554: /**
555: * sprint_backtrace - Look up a backtrace symbol and return it in a text buffer
556: * @buffer: buffer to be stored
557: * @address: address to lookup
558: *
559: * This function is for stack backtrace and does the same thing as
560: * sprint_symbol() but with modified/decreased @address. If there is a
561: * tail-call to the function marked "noreturn", gcc optimized out code after
562: * the call so that the stack-saved return address could point outside of the
563: * caller. This function ensures that kallsyms will find the original caller
564: * by decreasing @address.
565: *
566: * This function returns the number of bytes stored in @buffer.
567: */
568: int sprint_backtrace(char *buffer, unsigned long address)
569: {
570: return __sprint_symbol(buffer, address, -1, 1, 0);
571: }
SourceFile:kernel/kallsyms.c]
|
| 667/6 |
2026/05/18 20:25 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 56572
output: 872
thoughts: 3009
Thoughts: **Analysis of `%pB` Handling in `bstr_printf`**
My understanding is that `bstr_printf` treats `%pB` formatted strings as pre-formatted strings, meaning it doesn't perform any formatting itself. Instead, it expects to receive a string that has already been processed and prepared by `vbin_printf`.
Here's my breakdown of why this is the case:
1. **`sprint_symbol` and `__sprint_symbol`**: The code for `%pB` uses `sprint_symbol`, which calls `__sprint_symbol`. The key difference between symbolic and backtraced addresses lies in the offset subtraction for return addresses.
2. **`%pB` is Not Deferred**: Unlike some other `%p` extensions (like `%pK` or `%pe`), the code path for `%pB` doesn't defer formatting. Formats in the list of `'S', 's', 'x', 'K', 'e'` are handled by saving the raw data.
3. **`bstr_printf`'s `default` Case**: The structure of `bstr_printf` indicates that it goes into the default case for alphanumeric arguments like `%pB`. In the `default` case, it copies a pre-formatted string directly from the binary buffer.
4. **`vbin_printf`'s Role**: `vbin_printf`, responsible for generating the binary buffer, handles `%pB` by calling `pointer()` which in turn calls `sprint_symbol()`, ultimately producing a string representation of the symbol.
5. **`bstr_printf` Just Copies**: `bstr_printf` merely copies this pre-formatted string (which is null-terminated) from the binary buffer to the output buffer, without performing any additional symbol lookup or formatting.
In essence, `bstr_printf` relies on the fact that `vbin_printf` has already done the heavy lifting of symbol resolution and formatting for `%pB`. This is why the binary buffer is constructed and why it expects the corresponding pre-formatted data. The handling is designed such that `bstr_printf` is decoupled from the details of symbol lookup; it just prints the pre-processed string. This design promotes efficiency because the time-consuming symbol resolution work is performed only once, during buffer generation by `vbin_printf`, and not repeatedly during the final printing.
|
| 668/4 |
2026/05/18 20:26 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 37296
output: 50
thoughts: 3662
Thoughts: **Analyzing the BPF Formatting Vulnerability**
Okay, so I've been digging into this `bpf_bprintf_prepare` function, and it's starting to make sense. Essentially, it's prepping arguments to be used by `bstr_printf`, specifically handling `%pB` (binary data) and `%pI4`/`%pI6` (IP addresses). It looks like `bpf_bprintf_prepare` uses `snprintf` to format these arguments into `bin_args`, which is then passed to `bstr_printf`. `snprintf` is key here: it can truncate strings if the provided buffer isn't large enough. The original thought was that if `snprintf` truncated a string, we might advance past the buffer, creating a buffer overflow. By using `limit` and `min_t` to manage the advance of `tmp_buf`, we can avoid the overflow.
The core of the problem appears to be null termination. `snprintf` guarantees null termination if `limit > 0`. However, the situation where `limit` is zero because `tmp_buf >= tmp_buf_end` is what is causing the problem. If `limit` is zero, `snprintf` writes nothing, and there's no null terminator. This is especially problematic if it's the *last* argument. The code *should* return `-ENOSPC` when `tmp_buf >= tmp_buf_end`, effectively signaling that we're out of space and avoiding calling `bstr_printf` entirely, because `bstr_printf` expects a null-terminated string and will otherwise read out-of-bounds, likely causing crashes or even soft lockups.
Now I'm looking into the patch itself. The patch aims to correct an issue where `tmp_buf` could overrun the end of `tmp_buf_end`. The patch introduces a `limit` variable, which uses `tmp_buf < tmp_buf_end ? tmp_buf_end - tmp_buf : 0`. The patch correctly checks for the case where `tmp_buf >= tmp_buf_end` and sets `limit` to 0. But then, if it is the last argument, `bstr_printf` is called with a non-null-terminated string that may cause a read out of bounds.
After careful analysis, my conclusion is that it is safe to return `-ENOSPC` when the buffer is full.
The fix I am proposing is that the code *must* return `-ENOSPC` in the situation where `tmp_buf >= tmp_buf_end` before calling `snprintf`.
The final concern is the `%pI4`/`%pI6` formatting. The issue is that the code doesn't guarantee enough space in `tmp_buf` to hold the complete IP address strings which, when truncated, might lead to an incomplete string, even if it is null terminated. This could lead to incorrect output and potentially security vulnerabilities.
|
| 669/4 |
2026/05/18 20:26 |
tool |
codeexpert |
1m
Args: map[Question:How does bstr_printf handle %pI4 and %pI6? Does it expect a pre-formatted string in bin_args, just like %pB?]
Results: map[Answer:In `bstr_printf`, the handling of `%pI4` and `%pI6` (as well as other alphanumeric pointer extensions like `%pM`, `%pU`, etc.) indeed expects a **pre-formatted string** in the binary arguments buffer (`bin_buf`). This is the same behavior as `%pB`.
### How it works:
The binary formatting process in the kernel is split into two parts:
1. **`vbin_printf`**: This function parses the format string and the `va_list` arguments, and produces a binary buffer (`bin_buf`).
2. **`bstr_printf`**: This function takes the format string and the `bin_buf` and produces the final formatted text.
#### 1. In `vbin_printf`:
When `vbin_printf` encounters a pointer format specifier (`%p`), it checks the extension character following the `p`.
* If the extension is one of `S`, `s`, `x`, `K`, or `e`, `vbin_printf` simply saves the raw pointer value into the binary buffer. These are "deferred" because they don't require immediate dereferencing of the pointer to capture the data, or because their resolution depends on the context of the reader (like `%pK`).
* For any other **alphanumeric** extension (like `I` in `%pI4`, or `B` in `%pB`), `vbin_printf` assumes that the pointer must be dereferenced immediately to capture the data (since the memory it points to might be transient).
* Consequently, `vbin_printf` calls the `pointer()` function immediately, which formats the data (e.g., the IP address or the backtrace symbol) into a null-terminated string and stores that string directly in the binary buffer.
#### 2. In `bstr_printf`:
When `bstr_printf` encounters the same `%p` specifier:
* It checks the extension character.
* If it is one of the deferred ones (`S`, `s`, `x`, `K`, `e`), it retrieves the raw pointer from the buffer and calls `pointer()` to format it.
* If it is any other alphanumeric character (like `I` or `B`), it recognizes that the "pointer dereference was already processed" by `vbin_printf`. It then treats the corresponding entry in the binary buffer as a null-terminated string and simply copies it to the output buffer.
### Summary of Specifiers:
* **Pre-formatted as strings in `bin_buf`**: `%pI4`, `%pI6`, `%pB`, `%pM`, `%pU`, `%pV`, `%pR`, etc. (any `%p` followed by an alphanumeric character not in the exception list).
* **Stored as raw pointers in `bin_buf`**: `%p` (plain), `%pS`, `%ps`, `%px`, `%pK`, `%pe`.
Thus, `bstr_printf` handles `%pI4` and `%pI6` by copying a string that was already generated by `vbin_printf`, exactly as it does for `%pB`.]
|
| 670/5 |
2026/05/18 20:26 |
agent |
codeexpert |
1m
Model:
Instruction:
You are a capable Linux kernel developer tasked with researching complex questions
about kernel source code. You will be given a concrete question, and need to provide
a concrete answer.
Use tools extensively while researching the question. Don't make assumptions,
or rely on your previous knowledge about the kernel source code, use available tools
to access the actual source code.
Use all available sources of information:
- kernel source code
- documentation in the Documentation dir in the source tree
- git commits descriptions, git blame
Prefer calling several tools at the same time to save round-trips.
Prompt: How does bstr_printf handle %pI4 and %pI6? Does it expect a pre-formatted string in bin_args, just like %pB?
Reply: In `bstr_printf`, the handling of `%pI4` and `%pI6` (as well as other alphanumeric pointer extensions like `%pM`, `%pU`, etc.) indeed expects a **pre-formatted string** in the binary arguments buffer (`bin_buf`). This is the same behavior as `%pB`.
### How it works:
The binary formatting process in the kernel is split into two parts:
1. **`vbin_printf`**: This function parses the format string and the `va_list` arguments, and produces a binary buffer (`bin_buf`).
2. **`bstr_printf`**: This function takes the format string and the `bin_buf` and produces the final formatted text.
#### 1. In `vbin_printf`:
When `vbin_printf` encounters a pointer format specifier (`%p`), it checks the extension character following the `p`.
* If the extension is one of `S`, `s`, `x`, `K`, or `e`, `vbin_printf` simply saves the raw pointer value into the binary buffer. These are "deferred" because they don't require immediate dereferencing of the pointer to capture the data, or because their resolution depends on the context of the reader (like `%pK`).
* For any other **alphanumeric** extension (like `I` in `%pI4`, or `B` in `%pB`), `vbin_printf` assumes that the pointer must be dereferenced immediately to capture the data (since the memory it points to might be transient).
* Consequently, `vbin_printf` calls the `pointer()` function immediately, which formats the data (e.g., the IP address or the backtrace symbol) into a null-terminated string and stores that string directly in the binary buffer.
#### 2. In `bstr_printf`:
When `bstr_printf` encounters the same `%p` specifier:
* It checks the extension character.
* If it is one of the deferred ones (`S`, `s`, `x`, `K`, `e`), it retrieves the raw pointer from the buffer and calls `pointer()` to format it.
* If it is any other alphanumeric character (like `I` or `B`), it recognizes that the "pointer dereference was already processed" by `vbin_printf`. It then treats the corresponding entry in the binary buffer as a null-terminated string and simply copies it to the output buffer.
### Summary of Specifiers:
* **Pre-formatted as strings in `bin_buf`**: `%pI4`, `%pI6`, `%pB`, `%pM`, `%pU`, `%pV`, `%pR`, etc. (any `%p` followed by an alphanumeric character not in the exception list).
* **Stored as raw pointers in `bin_buf`**: `%p` (plain), `%pS`, `%ps`, `%px`, `%pK`, `%pe`.
Thus, `bstr_printf` handles `%pI4` and `%pI6` by copying a string that was already generated by `vbin_printf`, exactly as it does for `%pB`.
|
| 671/6 |
2026/05/18 20:26 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 2288
output: 22
thoughts: 122
Thoughts: **Investigating `bstr_printf`'s IP Address Formatting**
Okay, so the question boils down to this: how does `bstr_printf` deal with the `%pI4` and `%pI6` format specifiers? Specifically, I need to understand if they expect a pre-formatted IP address string in the `bin_args`, much like how `%pB` expects a pre-formatted binary string.
To get a clear answer, my plan is simple, really. First, I need to locate the definition of the `bstr_printf` function. That's the starting point. Then, I'll dissect the code to see exactly how `%pI4` and `%pI6` are processed. Finally, I'll draw a direct comparison to how `%pB` is handled to nail down the similarities and differences in their expectations. This shouldn't take long; let's get to it!
|
| 672/6 |
2026/05/18 20:26 |
tool |
codesearch-find-references |
0m
Args:
Results: map[References:[map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:____bpf_snprintf SourceFile:kernel/bpf/helpers.c SourceLine:1084] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:bpf_stream_vprintk SourceFile:kernel/bpf/stream.c SourceLine:240] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:seq_bprintf SourceFile:fs/seq_file.c SourceLine:418] map[ReferenceKind:calls ReferencingEntityKind:function ReferencingEntityName:seq_buf_bprintf SourceFile:lib/seq_buf.c SourceLine:162]] TruncatedOutput:false]
|
| 673/6 |
2026/05/18 20:26 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 2852
output: 39
thoughts: 31
Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience:
**Hunting Down That `bstr_printf` Definition**
Alright, I've got a few leads, but I need the hard definition of `bstr_printf`. I'm not just looking for vague usage examples; I need the source code. Let's see... the most direct route seems to be using `codesearch-definition-source`. I'm hoping it'll cut through the noise and give me the precise declaration and implementation. No time for guesswork; let's get down to the brass tacks and figure out what we're really dealing with here.
|
| 674/6 |
2026/05/18 20:26 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/vsprintf.c IncludeLines:true Name:bstr_printf]
Results: map[SourceCode:3260: /**
3261: * bstr_printf - Format a string from binary arguments and place it in a buffer
3262: * @buf: The buffer to place the result into
3263: * @size: The size of the buffer, including the trailing null space
3264: * @fmt_str: The format string to use
3265: * @bin_buf: Binary arguments for the format string
3266: *
3267: * This function like C99 vsnprintf, but the difference is that vsnprintf gets
3268: * arguments from stack, and bstr_printf gets arguments from @bin_buf which is
3269: * a binary buffer that generated by vbin_printf.
3270: *
3271: * The format follows C99 vsnprintf, but has some extensions:
3272: * see vsnprintf comment for details.
3273: *
3274: * The return value is the number of characters which would
3275: * be generated for the given input, excluding the trailing
3276: * '\0', as per ISO C99. If you want to have the exact
3277: * number of characters written into @buf as return value
3278: * (not including the trailing '\0'), use vscnprintf(). If the
3279: * return is greater than or equal to @size, the resulting
3280: * string is truncated.
3281: */
3282: int bstr_printf(char *buf, size_t size, const char *fmt_str, const u32 *bin_buf)
3283: {
3284: struct fmt fmt = {
3285: .str = fmt_str,
3286: .state = FORMAT_STATE_NONE,
3287: };
3288: struct printf_spec spec = {0};
3289: char *str, *end;
3290: const char *args = (const char *)bin_buf;
3291:
3292: if (WARN_ON_ONCE(size > INT_MAX))
3293: return 0;
3294:
3295: str = buf;
3296: end = buf + size;
3297:
3298: #define get_arg(type) \
3299: ({ \
3300: typeof(type) value; \
3301: if (sizeof(type) == 8) { \
3302: args = PTR_ALIGN(args, sizeof(u32)); \
3303: *(u32 *)&value = *(u32 *)args; \
3304: *((u32 *)&value + 1) = *(u32 *)(args + 4); \
3305: } else { \
3306: args = PTR_ALIGN(args, sizeof(type)); \
3307: value = *(typeof(type) *)args; \
3308: } \
3309: args += sizeof(type); \
3310: value; \
3311: })
3312:
3313: /* Make sure end is always >= buf */
3314: if (end < buf) {
3315: end = ((void *)-1);
3316: size = end - buf;
3317: }
3318:
3319: while (*fmt.str) {
3320: const char *old_fmt = fmt.str;
3321: unsigned long long num;
3322:
3323: fmt = format_decode(fmt, &spec);
3324: switch (fmt.state) {
3325: case FORMAT_STATE_NONE: {
3326: int read = fmt.str - old_fmt;
3327: if (str < end) {
3328: int copy = read;
3329: if (copy > end - str)
3330: copy = end - str;
3331: memcpy(str, old_fmt, copy);
3332: }
3333: str += read;
3334: continue;
3335: }
3336:
3337: case FORMAT_STATE_WIDTH:
3338: set_field_width(&spec, get_arg(int));
3339: continue;
3340:
3341: case FORMAT_STATE_PRECISION:
3342: set_precision(&spec, get_arg(int));
3343: continue;
3344:
3345: case FORMAT_STATE_CHAR: {
3346: char c;
3347:
3348: if (!(spec.flags & LEFT)) {
3349: while (--spec.field_width > 0) {
3350: if (str < end)
3351: *str = ' ';
3352: ++str;
3353: }
3354: }
3355: c = (unsigned char) get_arg(char);
3356: if (str < end)
3357: *str = c;
3358: ++str;
3359: while (--spec.field_width > 0) {
3360: if (str < end)
3361: *str = ' ';
3362: ++str;
3363: }
3364: continue;
3365: }
3366:
3367: case FORMAT_STATE_STR: {
3368: const char *str_arg = args;
3369: args += strlen(str_arg) + 1;
3370: str = string(str, end, (char *)str_arg, spec);
3371: continue;
3372: }
3373:
3374: case FORMAT_STATE_PTR: {
3375: bool process = false;
3376: int copy, len;
3377: /* Non function dereferences were already done */
3378: switch (*fmt.str) {
3379: case 'S':
3380: case 's':
3381: case 'x':
3382: case 'K':
3383: case 'e':
3384: process = true;
3385: break;
3386: default:
3387: if (!isalnum(*fmt.str)) {
3388: process = true;
3389: break;
3390: }
3391: /* Pointer dereference was already processed */
3392: if (str < end) {
3393: len = copy = strlen(args);
3394: if (copy > end - str)
3395: copy = end - str;
3396: memcpy(str, args, copy);
3397: str += len;
3398: args += len + 1;
3399: }
3400: }
3401: if (process)
3402: str = pointer(fmt.str, str, end, get_arg(void *), spec);
3403:
3404: while (isalnum(*fmt.str))
3405: fmt.str++;
3406: continue;
3407: }
3408:
3409: case FORMAT_STATE_PERCENT_CHAR:
3410: if (str < end)
3411: *str = '%';
3412: ++str;
3413: continue;
3414:
3415: case FORMAT_STATE_INVALID:
3416: goto out;
3417:
3418: case FORMAT_STATE_NUM:
3419: if (fmt.size > sizeof(int))
3420: num = get_arg(long long);
3421: else
3422: num = convert_num_spec(get_arg(int), fmt.size, spec);
3423: str = number(str, end, num, spec);
3424: continue;
3425: }
3426: } /* while(*fmt.str) */
3427:
3428: out:
3429: if (size > 0) {
3430: if (str < end)
3431: *str = '\0';
3432: else
3433: end[-1] = '\0';
3434: }
3435:
3436: #undef get_arg
3437:
3438: /* the trailing null byte doesn't count towards the total */
3439: return str - buf;
3440: }
SourceFile:lib/vsprintf.c]
|
| 675/6 |
2026/05/18 20:26 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 5513
output: 39
thoughts: 585
Thoughts: **Analyzing Pointer Handling in `bstr_printf`**
Okay, let's break down this pointer handling logic in `bstr_printf` within the `FORMAT_STATE_PTR` case. It's clear that the code is dealing with pointers, but the details are a bit dense. The core of it seems to be in lines 3374-3407.
Essentially, the code checks the format specifier character (`*fmt.str`). There's a `switch` statement that appears to handle different pointer dereference scenarios. If the character is 'S', 's', 'x', 'K', or 'e', the `process` flag is set to true (lines 3379-3385). If not, and it *is* alphanumeric, then we're in the `default` case, as in the example of 'I'.
Now, here's where it gets interesting and potentially problematic. The comment on line 3391, "/* Pointer dereference was already processed */," implies some prior processing of the pointer. It then expects a string in `args`, which I think is `bin_buf` passed in from `vbin_printf`.
The line `str = pointer(fmt.str, str, end, get_arg(void *), spec);` (3402) suggests that the actual pointer formatting, presumably based on the specifier, is delegated to the `pointer()` function.
However, consider the case of `%pI4` or `%pI6`. The character 'I' falls into the `default` case. `isalnum('I')` evaluates to true, so it then goes to line 3391. The critical point is: *how* is the pointer dereference considered "already processed"? What does `vbin_printf` do to prepare this data? We need to verify that `bin_buf` is prepped correctly, otherwise we could have some serious issues. Let's shift our focus to `vbin_printf` to confirm the preparation of the binary buffer. It's designed to feed `bstr_printf`, so it holds the key.
|
| 676/6 |
2026/05/18 20:26 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/vsprintf.c IncludeLines:true Name:vbin_printf]
Results: map[SourceCode:3125: /**
3126: * vbin_printf - Parse a format string and place args' binary value in a buffer
3127: * @bin_buf: The buffer to place args' binary value
3128: * @size: The size of the buffer(by words(32bits), not characters)
3129: * @fmt_str: The format string to use
3130: * @args: Arguments for the format string
3131: *
3132: * The format follows C99 vsnprintf, except %n is ignored, and its argument
3133: * is skipped.
3134: *
3135: * The return value is the number of words(32bits) which would be generated for
3136: * the given input.
3137: *
3138: * NOTE:
3139: * If the return value is greater than @size, the resulting bin_buf is NOT
3140: * valid for bstr_printf().
3141: */
3142: int vbin_printf(u32 *bin_buf, size_t size, const char *fmt_str, va_list args)
3143: {
3144: struct fmt fmt = {
3145: .str = fmt_str,
3146: .state = FORMAT_STATE_NONE,
3147: };
3148: struct printf_spec spec = {0};
3149: char *str, *end;
3150: int width;
3151:
3152: str = (char *)bin_buf;
3153: end = (char *)(bin_buf + size);
3154:
3155: #define save_arg(type) \
3156: ({ \
3157: unsigned long long value; \
3158: if (sizeof(type) == 8) { \
3159: unsigned long long val8; \
3160: str = PTR_ALIGN(str, sizeof(u32)); \
3161: val8 = va_arg(args, unsigned long long); \
3162: if (str + sizeof(type) <= end) { \
3163: *(u32 *)str = *(u32 *)&val8; \
3164: *(u32 *)(str + 4) = *((u32 *)&val8 + 1); \
3165: } \
3166: value = val8; \
3167: } else { \
3168: unsigned int val4; \
3169: str = PTR_ALIGN(str, sizeof(type)); \
3170: val4 = va_arg(args, int); \
3171: if (str + sizeof(type) <= end) \
3172: *(typeof(type) *)str = (type)(long)val4; \
3173: value = (unsigned long long)val4; \
3174: } \
3175: str += sizeof(type); \
3176: value; \
3177: })
3178:
3179: while (*fmt.str) {
3180: fmt = format_decode(fmt, &spec);
3181:
3182: switch (fmt.state) {
3183: case FORMAT_STATE_NONE:
3184: case FORMAT_STATE_PERCENT_CHAR:
3185: break;
3186: case FORMAT_STATE_INVALID:
3187: goto out;
3188:
3189: case FORMAT_STATE_WIDTH:
3190: case FORMAT_STATE_PRECISION:
3191: width = (int)save_arg(int);
3192: /* Pointers may require the width */
3193: if (*fmt.str == 'p')
3194: set_field_width(&spec, width);
3195: break;
3196:
3197: case FORMAT_STATE_CHAR:
3198: save_arg(char);
3199: break;
3200:
3201: case FORMAT_STATE_STR: {
3202: const char *save_str = va_arg(args, char *);
3203: const char *err_msg;
3204: size_t len;
3205:
3206: err_msg = check_pointer_msg(save_str);
3207: if (err_msg)
3208: save_str = err_msg;
3209:
3210: len = strlen(save_str) + 1;
3211: if (str + len < end)
3212: memcpy(str, save_str, len);
3213: str += len;
3214: break;
3215: }
3216:
3217: case FORMAT_STATE_PTR:
3218: /* Dereferenced pointers must be done now */
3219: switch (*fmt.str) {
3220: /* Dereference of functions is still OK */
3221: case 'S':
3222: case 's':
3223: case 'x':
3224: case 'K':
3225: case 'e':
3226: save_arg(void *);
3227: break;
3228: default:
3229: if (!isalnum(*fmt.str)) {
3230: save_arg(void *);
3231: break;
3232: }
3233: str = pointer(fmt.str, str, end, va_arg(args, void *),
3234: spec);
3235: if (str + 1 < end)
3236: *str++ = '\0';
3237: else
3238: end[-1] = '\0'; /* Must be nul terminated */
3239: }
3240: /* skip all alphanumeric pointer suffixes */
3241: while (isalnum(*fmt.str))
3242: fmt.str++;
3243: break;
3244:
3245: case FORMAT_STATE_NUM:
3246: if (fmt.size > sizeof(int)) {
3247: save_arg(long long);
3248: } else {
3249: save_arg(int);
3250: }
3251: }
3252: }
3253:
3254: out:
3255: return (u32 *)(PTR_ALIGN(str, sizeof(u32))) - bin_buf;
3256: #undef save_arg
3257: }
SourceFile:lib/vsprintf.c]
|
| 677/6 |
2026/05/18 20:26 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 8494
output: 36
thoughts: 926
Thoughts: **Analyzing the Binary Formatting of Pointers in `vbin_printf` and `bstr_printf`**
Okay, let's break down how this works. Looking at `vbin_printf`, specifically within the `FORMAT_STATE_PTR` case, it seems the code handles dereferenced pointers in a nuanced way. I see that certain format specifiers, like `%S`, `%s`, `%x`, etc., are handled directly with `save_arg(void *)`. However, the interesting part is the `default` case. If the character following `%p` is alphanumeric *and* not explicitly handled, like with an 'I' in `%pI4`, then the `pointer()` function is called.
The `pointer()` function seems to be responsible for pre-formatting, specifically formatting the IP address into a string in the case of `%pI4` and `%pI6` and appending a null terminator. So, it appears that `vbin_printf` transforms these complex pointer formats into a string representation within the binary buffer.
Then `bstr_printf` comes into play. It expects this pre-formatted string. Looking at the `bstr_printf` code, I can see that it retrieves the pre-formatted string from the `args` (which is `bin_args` in this context) and copies it into the final output buffer.
Now, the user brought up `%pB`. Let's see how that fits. `%pB` is for backtrace symbols, right? Back in `vbin_printf`, I noticed that 'B' isn't explicitly listed in the list of handled characters, which are handled directly with `save_arg`. But 'B' is alphanumeric. That means that, just like the `%pI` formats, `%pB` should fall into the `default` case and get pre-formatted by `pointer()`.
I should probably confirm what `pointer()` does with `%pB` to be certain.
|
| 678/6 |
2026/05/18 20:26 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/vsprintf.c IncludeLines:true Name:pointer]
Results: map[SourceCode:2388: /*
2389: * Show a '%p' thing. A kernel extension is that the '%p' is followed
2390: * by an extra set of alphanumeric characters that are extended format
2391: * specifiers.
2392: *
2393: * Please update scripts/checkpatch.pl when adding/removing conversion
2394: * characters. (Search for "check for vsprintf extension").
2395: *
2396: * Right now we handle:
2397: *
2398: * - 'S' For symbolic direct pointers (or function descriptors) with offset
2399: * - 's' For symbolic direct pointers (or function descriptors) without offset
2400: * - '[Ss]R' as above with __builtin_extract_return_addr() translation
2401: * - 'S[R]b' as above with module build ID (for use in backtraces)
2402: * - '[Ff]' %pf and %pF were obsoleted and later removed in favor of
2403: * %ps and %pS. Be careful when re-using these specifiers.
2404: * - 'B' For backtraced symbolic direct pointers with offset
2405: * - 'Bb' as above with module build ID (for use in backtraces)
2406: * - 'R' For decoded struct resource, e.g., [mem 0x0-0x1f 64bit pref]
2407: * - 'r' For raw struct resource, e.g., [mem 0x0-0x1f flags 0x201]
2408: * - 'ra' For struct ranges, e.g., [range 0x0000000000000000 - 0x00000000000000ff]
2409: * - 'b[l]' For a bitmap, the number of bits is determined by the field
2410: * width which must be explicitly specified either as part of the
2411: * format string '%32b[l]' or through '%*b[l]', [l] selects
2412: * range-list format instead of hex format
2413: * - 'M' For a 6-byte MAC address, it prints the address in the
2414: * usual colon-separated hex notation
2415: * - 'm' For a 6-byte MAC address, it prints the hex address without colons
2416: * - 'MF' For a 6-byte MAC FDDI address, it prints the address
2417: * with a dash-separated hex notation
2418: * - '[mM]R' For a 6-byte MAC address, Reverse order (Bluetooth)
2419: * - 'I' [46] for IPv4/IPv6 addresses printed in the usual way
2420: * IPv4 uses dot-separated decimal without leading 0's (1.2.3.4)
2421: * IPv6 uses colon separated network-order 16 bit hex with leading 0's
2422: * [S][pfs]
2423: * Generic IPv4/IPv6 address (struct sockaddr *) that falls back to
2424: * [4] or [6] and is able to print port [p], flowinfo [f], scope [s]
2425: * - 'i' [46] for 'raw' IPv4/IPv6 addresses
2426: * IPv6 omits the colons (01020304...0f)
2427: * IPv4 uses dot-separated decimal with leading 0's (010.123.045.006)
2428: * [S][pfs]
2429: * Generic IPv4/IPv6 address (struct sockaddr *) that falls back to
2430: * [4] or [6] and is able to print port [p], flowinfo [f], scope [s]
2431: * - '[Ii][4S][hnbl]' IPv4 addresses in host, network, big or little endian order
2432: * - 'I[6S]c' for IPv6 addresses printed as specified by
2433: * https://tools.ietf.org/html/rfc5952
2434: * - 'E[achnops]' For an escaped buffer, where rules are defined by combination
2435: * of the following flags (see string_escape_mem() for the
2436: * details):
2437: * a - ESCAPE_ANY
2438: * c - ESCAPE_SPECIAL
2439: * h - ESCAPE_HEX
2440: * n - ESCAPE_NULL
2441: * o - ESCAPE_OCTAL
2442: * p - ESCAPE_NP
2443: * s - ESCAPE_SPACE
2444: * By default ESCAPE_ANY_NP is used.
2445: * - 'U' For a 16 byte UUID/GUID, it prints the UUID/GUID in the form
2446: * "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
2447: * Options for %pU are:
2448: * b big endian lower case hex (default)
2449: * B big endian UPPER case hex
2450: * l little endian lower case hex
2451: * L little endian UPPER case hex
2452: * big endian output byte order is:
2453: * [0][1][2][3]-[4][5]-[6][7]-[8][9]-[10][11][12][13][14][15]
2454: * little endian output byte order is:
2455: * [3][2][1][0]-[5][4]-[7][6]-[8][9]-[10][11][12][13][14][15]
2456: * - 'V' For a struct va_format which contains a format string * and va_list *,
2457: * call vsnprintf(->format, *->va_list).
2458: * Implements a "recursive vsnprintf".
2459: * Do not use this feature without some mechanism to verify the
2460: * correctness of the format string and va_list arguments.
2461: * - 'K' For a kernel pointer that should be hidden from unprivileged users.
2462: * Use only for procfs, sysfs and similar files, not printk(); please
2463: * read the documentation (path below) first.
2464: * - 'NF' For a netdev_features_t
2465: * - '4cc' V4L2 or DRM FourCC code, with endianness and raw numerical value.
2466: * - '4c[h[R]lb]' For generic FourCC code with raw numerical value. Both are
2467: * displayed in the big-endian format. This is the opposite of V4L2 or
2468: * DRM FourCCs.
2469: * The additional specifiers define what endianness is used to load
2470: * the stored bytes. The data might be interpreted using the host,
2471: * reversed host byte order, little-endian, or big-endian.
2472: * - 'h[CDN]' For a variable-length buffer, it prints it as a hex string with
2473: * a certain separator (' ' by default):
2474: * C colon
2475: * D dash
2476: * N no separator
2477: * The maximum supported length is 64 bytes of the input. Consider
2478: * to use print_hex_dump() for the larger input.
2479: * - 'a[pd]' For address types [p] phys_addr_t, [d] dma_addr_t and derivatives
2480: * (default assumed to be phys_addr_t, passed by reference)
2481: * - 'd[234]' For a dentry name (optionally 2-4 last components)
2482: * - 'D[234]' Same as 'd' but for a struct file
2483: * - 'g' For block_device name (gendisk + partition number)
2484: * - 't[RST][dt][r][s]' For time and date as represented by:
2485: * R struct rtc_time
2486: * S struct timespec64
2487: * T time64_t
2488: * - 'tSp' For time represented by struct timespec64 printed as <seconds>.<nanoseconds>
2489: * - 'C' For a clock, it prints the name (Common Clock Framework) or address
2490: * (legacy clock framework) of the clock
2491: * - 'G' For flags to be printed as a collection of symbolic strings that would
2492: * construct the specific value. Supported flags given by option:
2493: * p page flags (see struct page) given as pointer to unsigned long
2494: * g gfp flags (GFP_* and __GFP_*) given as pointer to gfp_t
2495: * v vma flags (VM_*) given as pointer to unsigned long
2496: * - 'OF[fnpPcCF]' For a device tree object
2497: * Without any optional arguments prints the full_name
2498: * f device node full_name
2499: * n device node name
2500: * p device node phandle
2501: * P device node path spec (name + @unit)
2502: * F device node flags
2503: * c major compatible string
2504: * C full compatible string
2505: * - 'fw[fP]' For a firmware node (struct fwnode_handle) pointer
2506: * Without an option prints the full name of the node
2507: * f full name
2508: * P node name, including a possible unit address
2509: * - 'x' For printing the address unmodified. Equivalent to "%lx".
2510: * Please read the documentation (path below) before using!
2511: * - '[ku]s' For a BPF/tracing related format specifier, e.g. used out of
2512: * bpf_trace_printk() where [ku] prefix specifies either kernel (k)
2513: * or user (u) memory to probe, and:
2514: * s a string, equivalent to "%s" on direct vsnprintf() use
2515: *
2516: * ** When making changes please also update:
2517: * Documentation/core-api/printk-formats.rst
2518: *
2519: * Note: The default behaviour (unadorned %p) is to hash the address,
2520: * rendering it useful as a unique identifier.
2521: *
2522: * There is also a '%pA' format specifier, but it is only intended to be used
2523: * from Rust code to format core::fmt::Arguments. Do *not* use it from C.
2524: * See rust/kernel/print.rs for details.
2525: */
2526: static noinline_for_stack
2527: char *pointer(const char *fmt, char *buf, char *end, void *ptr,
2528: struct printf_spec spec)
2529: {
2530: switch (*fmt) {
2531: case 'S':
2532: case 's':
2533: ptr = dereference_symbol_descriptor(ptr);
2534: fallthrough;
2535: case 'B':
2536: return symbol_string(buf, end, ptr, spec, fmt);
2537: case 'R':
2538: case 'r':
2539: return resource_or_range(fmt, buf, end, ptr, spec);
2540: case 'h':
2541: return hex_string(buf, end, ptr, spec, fmt);
2542: case 'b':
2543: switch (fmt[1]) {
2544: case 'l':
2545: return bitmap_list_string(buf, end, ptr, spec, fmt);
2546: default:
2547: return bitmap_string(buf, end, ptr, spec, fmt);
2548: }
2549: case 'M': /* Colon separated: 00:01:02:03:04:05 */
2550: case 'm': /* Contiguous: 000102030405 */
2551: /* [mM]F (FDDI) */
2552: /* [mM]R (Reverse order; Bluetooth) */
2553: return mac_address_string(buf, end, ptr, spec, fmt);
2554: case 'I': /* Formatted IP supported
2555: * 4: 1.2.3.4
2556: * 6: 0001:0203:...:0708
2557: * 6c: 1::708 or 1::1.2.3.4
2558: */
2559: case 'i': /* Contiguous:
2560: * 4: 001.002.003.004
2561: * 6: 000102...0f
2562: */
2563: return ip_addr_string(buf, end, ptr, spec, fmt);
2564: case 'E':
2565: return escaped_string(buf, end, ptr, spec, fmt);
2566: case 'U':
2567: return uuid_string(buf, end, ptr, spec, fmt);
2568: case 'V':
2569: return va_format(buf, end, ptr, spec);
2570: case 'K':
2571: return restricted_pointer(buf, end, ptr, spec);
2572: case 'N':
2573: return netdev_bits(buf, end, ptr, spec, fmt);
2574: case '4':
2575: return fourcc_string(buf, end, ptr, spec, fmt);
2576: case 'a':
2577: return address_val(buf, end, ptr, spec, fmt);
2578: case 'd':
2579: return dentry_name(buf, end, ptr, spec, fmt);
2580: case 't':
2581: return time_and_date(buf, end, ptr, spec, fmt);
2582: case 'C':
2583: return clock(buf, end, ptr, spec, fmt);
2584: case 'D':
2585: return file_dentry_name(buf, end, ptr, spec, fmt);
2586: #ifdef CONFIG_BLOCK
2587: case 'g':
2588: return bdev_name(buf, end, ptr, spec, fmt);
2589: #endif
2590:
2591: case 'G':
2592: return flags_string(buf, end, ptr, spec, fmt);
2593: case 'O':
2594: return device_node_string(buf, end, ptr, spec, fmt + 1);
2595: case 'f':
2596: return fwnode_string(buf, end, ptr, spec, fmt + 1);
2597: case 'A':
2598: if (!IS_ENABLED(CONFIG_RUST)) {
2599: WARN_ONCE(1, "Please remove %%pA from non-Rust code\n");
2600: return error_string(buf, end, "(%pA?)", spec);
2601: }
2602: return rust_fmt_argument(buf, end, ptr);
2603: case 'x':
2604: return pointer_string(buf, end, ptr, spec);
2605: case 'e':
2606: /* %pe with a non-ERR_PTR gets treated as plain %p */
2607: if (!IS_ERR(ptr))
2608: return default_pointer(buf, end, ptr, spec);
2609: return err_ptr(buf, end, ptr, spec);
2610: case 'u':
2611: case 'k':
2612: switch (fmt[1]) {
2613: case 's':
2614: return string(buf, end, ptr, spec);
2615: default:
2616: return error_string(buf, end, "(einval)", spec);
2617: }
2618: default:
2619: return default_pointer(buf, end, ptr, spec);
2620: }
2621: }
SourceFile:lib/vsprintf.c]
|
| 679/6 |
2026/05/18 20:26 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 14512
output: 40
thoughts: 5249
Thoughts: **Analysis of `vbin_printf` and `bstr_printf` Pointer Formatting**
Alright, let's break down this `vbin_printf`/`bstr_printf` formatting logic. The core of my understanding is this: `vbin_printf` is the recorder and `bstr_printf` is the renderer. `vbin_printf` prepares the data and stores it in a binary buffer (`bin_buf` or `bin_args`), while `bstr_printf` retrieves and formats it to the final output string. The crucial point is the handling of pointer-related format specifiers, especially when we are talking about *dereferenced* pointers.
The central question revolves around how `%pI4`, `%pI6`, and, in the user's inquiry, `%pB` (and presumably other alphanumeric pointer extensions) are handled. It seems the core logic is as follows: `vbin_printf` *pre-formats* these, writing the string representation directly into the binary buffer; then, `bstr_printf` simply copies this pre-formatted string.
Let me walk through the process. In `vbin_printf`, if a format specifier like `%pI4` is encountered, the code checks the character immediately following the `p`. If it's an alphanumeric character (e.g. 'I' for IP addresses, 'B' for backtrace symbols), the code calls the `pointer()` function. Inside `pointer()`, dedicated functions like `ip_addr_string` (for 'I') are responsible for *immediately* dereferencing the pointer and formatting the data into a string. The result is placed in the binary buffer.
Why is this immediate formatting happening? It's largely due to the nature of the data being pointed to. For an IP address (or for `pB`), the 4 bytes (or 16) are read, converted to a string, and written directly. We want to read it now, because these 4 bytes may not be valid later if they are stack variables or allocated with `kmalloc`.
Now, when `bstr_printf` later processes this format specifier, it finds a *pre-formatted* string in the binary buffer. The `bstr_printf` function then just copies the string from the buffer to the output.
There is a list of *exceptions*. This behavior *differs* for a few specific alphanumeric extensions: `%pS`, `%ps`, `%px`, `%pK`, and `%pe`. These are *deferred* formats. Why? In these cases, it's about symbol resolution or restricted pointers.
`%pS` and `%ps` are deferred because symbol resolution can be a computationally expensive process and sometimes it depends on the context of the running code (modules, etc.). This allows the code to store the raw pointer first, and resolve the symbol later in `bstr_printf`, potentially at a different time or in a different context.
`%pK` handles the credentials of the caller - so deferring to be processed in `bstr_printf` makes sure the correct credentials are used.
`%px` does not require dereferencing the pointer, and can be printed at any time.
With these deferred formats, `vbin_printf` just saves the raw pointer to the buffer, and when `bstr_printf` sees those, it calls `pointer()` which does symbol resolution and puts them in the string.
For `%pB`, and `%pI4` (and all other similar specifiers), however, immediate formatting is performed.
Therefore, to answer the user's question, `%pI4` and `%pI6` are *pre-formatted* in the binary buffer by `vbin_printf`, exactly like `%pB`. This is done because `vbin_printf` recognizes that these format specifiers inherently require dereferencing and *value* extraction, so the code extracts the information directly during the initial formatting process.
|
| 680/6 |
2026/05/18 20:27 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:lib/vsprintf.c IncludeLines:true Name:ip_addr_string]
Results: map[SourceCode:1617: static noinline_for_stack
1618: char *ip_addr_string(char *buf, char *end, const void *ptr,
1619: struct printf_spec spec, const char *fmt)
1620: {
1621: char *err_fmt_msg;
1622:
1623: if (check_pointer(&buf, end, ptr, spec))
1624: return buf;
1625:
1626: switch (fmt[1]) {
1627: case '6':
1628: return ip6_addr_string(buf, end, ptr, spec, fmt);
1629: case '4':
1630: return ip4_addr_string(buf, end, ptr, spec, fmt);
1631: case 'S': {
1632: const union {
1633: struct sockaddr raw;
1634: struct sockaddr_in v4;
1635: struct sockaddr_in6 v6;
1636: } *sa = ptr;
1637:
1638: switch (sa->raw.sa_family) {
1639: case AF_INET:
1640: return ip4_addr_string_sa(buf, end, &sa->v4, spec, fmt);
1641: case AF_INET6:
1642: return ip6_addr_string_sa(buf, end, &sa->v6, spec, fmt);
1643: default:
1644: return error_string(buf, end, "(einval)", spec);
1645: }}
1646: }
1647:
1648: err_fmt_msg = fmt[0] == 'i' ? "(%pi?)" : "(%pI?)";
1649: return error_string(buf, end, err_fmt_msg, spec);
1650: }
SourceFile:lib/vsprintf.c]
|
| 681/6 |
2026/05/18 20:27 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 21194
output: 21
thoughts: 1903
Thoughts: **Analysis of `%pI4`, `%pI6`, and Similar Formats with `vbin_printf` and `bstr_printf`**
Okay, let's break this down. My understanding is this: `vbin_printf` handles the initial formatting into a binary buffer, and `bstr_printf` copies from that buffer to the output string. The core of this hinges on how these formats interact.
1. **IP Address Formatting:** When `vbin_printf` encounters `%pI4` or `%pI6`, it knows it needs to format an IP address. It calls `pointer()`, which in turn calls `ip_addr_string()`. This function *directly* formats the IP (e.g., "127.0.0.1") into the binary buffer. Crucially, `vbin_printf` *always* adds a null terminator after the formatted string.
2. **`bstr_printf`'s Interpretation:** Now, `bstr_printf` sees `%pI4` or `%pI6`. It *doesn't* need to know about IP addresses specifically. It checks if the character after `p` is alphanumeric (which 'I' is). It *also* checks if the character is in the exclusion list, which it isn't. Seeing that the format specifier is alphanumeric, and is *not* in the exclusion list, it *assumes* the value in the binary buffer is already a null-terminated string, and it simply copies that string.
3. **The General Rule: Alphanumeric and Not Excluded:** The key here is the general rule. Any `%p<alphanumeric>` format, *except* those in the exclusion list (`{'S', 's', 'x', 'K', 'e'}`), will be pre-formatted into a string by `vbin_printf`.
4. **Confirmation and Details:** I double-checked the logic in `bstr_printf`. If the character is alphanumeric, it *always* expects a string (unless it is excluded). In `vbin_printf`, if the format character is not in the exclusion list, `pointer()` is called, and that in turn calls the correct functions, formats into a string and null-terminates.
5. **Handling of Other Formats:** I took a look at other similar cases, like `%pA` for Rust, and `%pV` for `struct va_format`. In both of those cases, those format specifiers are also handled like alphanumeric characters in the exclusion list, because those extensions are not in the list. Specifically, `%pV` utilizes `vsnprintf` internally within `va_format`, ensuring the output is a string.
6. **Buffer Truncation:** I also considered the edge case of the binary buffer being too small. If `pointer()` hits the end of the buffer while formatting, it truncates the string, but `vbin_printf` *still* ensures the buffer is null-terminated. This guarantees safety; `bstr_printf` will simply copy the (potentially truncated) null-terminated string.
7. **`%pB` Comparison:** The user specifically mentions "%pB". Checking `vbin_printf`, 'B' is not in the exclusion list. So, yes, it behaves exactly like `%pI4` and `%pI6`. It's pre-formatted into a string.
8. **Final Summary:** In essence, `bstr_printf` treats `%pI4` and `%pI6` (and `%pB` and others) as pre-formatted strings because `vbin_printf` formats them as strings. `vbin_printf` calls dedicated functions to put those specific formatting strings into the buffer. `bstr_printf` simply copies from the buffer. This avoids needing `bstr_printf` to handle IP address formatting itself. It is all handled in the formatting stages before the data is passed to `bstr_printf`.
So the answer is clear: `bstr_printf` expects `%pI4`, `%pI6`, and similar formats to be pre-formatted strings, because `vbin_printf` handles the actual formatting and null-termination. This behaviour is consistent with `%pB`. This is why the pointer needs to be dereferenced by the earlier code.
|
| 682/6 |
2026/05/18 20:27 |
tool |
read-file |
0m
Args:
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/lib/vsprintf.c
*
* Copyright (C) 1991, 1992 Linus Torvalds
*/
/* vsprintf.c -- Lars Wirzenius & Linus Torvalds. */
/*
* Wirzenius wrote this portably, Torvalds fucked it up :-)
*/
/*
* Fri Jul 13 2001 Crutcher Dunnavant <crutcher+kernel@datastacks.com>
* - changed to provide snprintf and vsnprintf functions
* So Feb 1 16:51:32 CET 2004 Juergen Quade <quade@hsnr.de>
* - scnprintf and vscnprintf
*/
#include <linux/stdarg.h>
#include <linux/build_bug.h>
#include <linux/clk.h>
#include <linux/clk-provider.h>
#include <linux/errname.h>
#include <linux/module.h> /* for KSYM_SYMBOL_LEN */
#include <linux/types.h>
#include <linux/string.h>
#include <linux/ctype.h>
#include <linux/hex.h>
#include <linux/kernel.h>
#include <linux/kallsyms.h>
#include <linux/math64.h>
#include <linux/uaccess.h>
#include <linux/ioport.h>
#include <linux/dcache.h>
#include <linux/cred.h>
#include <linux/rtc.h>
#include <linux/sprintf.h>
#include <linux/time.h>
#include <linux/uuid.h>
#include <linux/of.h>
#include <net/addrconf.h>
#include <linux/siphash.h>
#include <linux/compiler.h>
#include <linux/property.h>
#include <linux/notifier.h>
#ifdef CONFIG_BLOCK
#include <linux/blkdev.h>
#endif
#include "../mm/internal.h" /* For the trace_print_flags arrays */
#include <asm/page.h> /* for PAGE_SIZE */
#include <asm/byteorder.h> /* cpu_to_le16 */
#include <linux/unaligned.h>
#include <linux/string_helpers.h>
#include "kstrtox.h"
/* Disable pointer hashing if requested */
bool no_hash_pointers __ro_after_init;
EXPORT_SYMBOL_GPL(no_hash_pointers);
/*
* Hashed pointers policy selected by "hash_pointers=..." boot param
*
* `auto` - Hashed pointers enabled unless disabled by slub_debug_enabled=true
* `always` - Hashed pointers enabled unconditionally
* `never` - Hashed pointers disabled unconditionally
*/
enum hash_pointers_policy {
HASH_PTR_AUTO = 0,
HASH_PTR_ALWAYS,
HASH_PTR_NEVER
};
static enum hash_pointers_policy hash_pointers_mode __initdata;
noinline
static unsigned long long simple_strntoull(const char *startp, char **endp, unsigned int base, size_t max_chars)
{
const char *cp;
unsigned long long result = 0ULL;
size_t prefix_chars;
unsigned int rv;
cp = _parse_integer_fixup_radix(startp, &base);
prefix_chars = cp - startp;
if (prefix_chars < max_chars) {
rv = _parse_integer_limit(cp, base, &result, max_chars - prefix_chars);
/* FIXME */
cp += (rv & ~KSTRTOX_OVERFLOW);
} else {
/* Field too short for prefix + digit, skip over without converting */
cp = startp + max_chars;
}
if (endp)
*endp = (char *)cp;
return result;
}
/**
* simple_strtoull - convert a string to an unsigned long long
* @cp: The start of the string
* @endp: A pointer to the end of the parsed string will be placed here
* @base: The number base to use
*
* This function has caveats. Please use kstrtoull instead.
*/
noinline
unsigned long long simple_strtoull(const char *cp, char **endp, unsigned int base)
{
return simple_strntoull(cp, endp, base, INT_MAX);
}
EXPORT_SYMBOL(simple_strtoull);
/**
* simple_strtoul - convert a string to an unsigned long
* @cp: The start of the string
* @endp: A pointer to the end of the parsed string will be placed here
* @base: The number base to use
*
* This function has caveats. Please use kstrtoul instead.
*/
unsigned long simple_strtoul(const char *cp, char **endp, unsigned int base)
{
return simple_strtoull(cp, endp, base);
}
EXPORT_SYMBOL(simple_strtoul);
unsigned long simple_strntoul(const char *cp, char **endp, unsigned int base,
size_t max_chars)
{
return simple_strntoull(cp, endp, base, max_chars);
}
EXPORT_SYMBOL(simple_strntoul);
/**
* simple_strtol - convert a string to a signed long
* @cp: The start of the string
* @endp: A pointer to the end of the parsed string will be placed here
* @base: The number base to use
*
* This function has caveats. Please use kstrtol instead.
*/
long simple_strtol(const char *cp, char **endp, unsigned int base)
{
if (*cp == '-')
return -simple_strtoul(cp + 1, endp, base);
return simple_strtoul(cp, endp, base);
}
EXPORT_SYMBOL(simple_strtol);
noinline
static long long simple_strntoll(const char *cp, char **endp, unsigned int base, size_t max_chars)
{
/*
* simple_strntoull() safely handles receiving max_chars==0 in the
* case cp[0] == '-' && max_chars == 1.
* If max_chars == 0 we can drop through and pass it to simple_strntoull()
* and the content of *cp is irrelevant.
*/
if (*cp == '-' && max_chars > 0)
return -simple_strntoull(cp + 1, endp, base, max_chars - 1);
return simple_strntoull(cp, endp, base, max_chars);
}
/**
* simple_strtoll - convert a string to a signed long long
* @cp: The start of the string
* @endp: A pointer to the end of the parsed string will be placed here
* @base: The number base to use
*
* This function has caveats. Please use kstrtoll instead.
*/
long long simple_strtoll(const char *cp, char **endp, unsigned int base)
{
return simple_strntoll(cp, endp, base, INT_MAX);
}
EXPORT_SYMBOL(simple_strtoll);
static inline int skip_atoi(const char **s)
{
int i = 0;
do {
i = i*10 + *((*s)++) - '0';
} while (isdigit(**s));
return i;
}
/*
* Decimal conversion is by far the most typical, and is used for
* /proc and /sys data. This directly impacts e.g. top performance
* with many processes running. We optimize it for speed by emitting
* two characters at a time, using a 200 byte lookup table. This
* roughly halves the number of multiplications compared to computing
* the digits one at a time. Implementation strongly inspired by the
* previous version, which in turn used ideas described at
* <http://www.cs.uiowa.edu/~jones/bcd/divide.html> (with permission
* from the author, Douglas W. Jones).
*
* It turns out there is precisely one 26 bit fixed-point
* approximation a of 64/100 for which x/100 == (x * (u64)a) >> 32
* holds for all x in [0, 10^8-1], namely a = 0x28f5c29. The actual
* range happens to be somewhat larger (x <= 1073741898), but that's
* irrelevant for our purpose.
*
* For dividing a number in the range [10^4, 10^6-1] by 100, we still
* need a 32x32->64 bit multiply, so we simply use the same constant.
*
* For dividing a number in the range [100, 10^4-1] by 100, there are
* several options. The simplest is (x * 0x147b) >> 19, which is valid
* for all x <= 43698.
*/
static const u16 decpair[100] = {
#define _(x) (__force u16) cpu_to_le16(((x % 10) | ((x / 10) << 8)) + 0x3030)
_( 0), _( 1), _( 2), _( 3), _( 4), _( 5), _( 6), _( 7), _( 8), _( 9),
_(10), _(11), _(12), _(13), _(14), _(15), _(16), _(17), _(18), _(19),
_(20), _(21), _(22), _(23), _(24), _(25), _(26), _(27), _(28), _(29),
_(30), _(31), _(32), _(33), _(34), _(35), _(36), _(37), _(38), _(39),
_(40), _(41), _(42), _(43), _(44), _(45), _(46), _(47), _(48), _(49),
_(50), _(51), _(52), _(53), _(54), _(55), _(56), _(57), _(58), _(59),
_(60), _(61), _(62), _(63), _(64), _(65), _(66), _(67), _(68), _(69),
_(70), _(71), _(72), _(73), _(74), _(75), _(76), _(77), _(78), _(79),
_(80), _(81), _(82), _(83), _(84), _(85), _(86), _(87), _(88), _(89),
_(90), _(91), _(92), _(93), _(94), _(95), _(96), _(97), _(98), _(99),
#undef _
};
/*
* This will print a single '0' even if r == 0, since we would
* immediately jump to out_r where two 0s would be written but only
* one of them accounted for in buf. This is needed by ip4_string
* below. All other callers pass a non-zero value of r.
*/
static noinline_for_stack
char *put_dec_trunc8(char *buf, unsigned r)
{
unsigned q;
/* 1 <= r < 10^8 */
if (r < 100)
goto out_r;
/* 100 <= r < 10^8 */
q = (r * (u64)0x28f5c29) >> 32;
*((u16 *)buf) = decpair[r - 100*q];
buf += 2;
/* 1 <= q < 10^6 */
if (q < 100)
goto out_q;
/* 100 <= q < 10^6 */
r = (q * (u64)0x28f5c29) >> 32;
*((u16 *)buf) = decpair[q - 100*r];
buf += 2;
/* 1 <= r < 10^4 */
if (r < 100)
goto out_r;
/* 100 <= r < 10^4 */
q = (r * 0x147b) >> 19;
*((u16 *)buf) = decpair[r - 100*q];
buf += 2;
out_q:
/* 1 <= q < 100 */
r = q;
out_r:
/* 1 <= r < 100 */
*((u16 *)buf) = decpair[r];
buf += r < 10 ? 1 : 2;
return buf;
}
#if BITS_PER_LONG == 64 && BITS_PER_LONG_LONG == 64
static noinline_for_stack
char *put_dec_full8(char *buf, unsigned r)
{
unsigned q;
/* 0 <= r < 10^8 */
q = (r * (u64)0x28f5c29) >> 32;
*((u16 *)buf) = decpair[r - 100*q];
buf += 2;
/* 0 <= q < 10^6 */
r = (q * (u64)0x28f5c29) >> 32;
*((u16 *)buf) = decpair[q - 100*r];
buf += 2;
/* 0 <= r < 10^4 */
q = (r * 0x147b) >> 19;
*((u16 *)buf) = decpair[r - 100*q];
buf += 2;
/* 0 <= q < 100 */
*((u16 *)buf) = decpair[q];
buf += 2;
return buf;
}
static noinline_for_stack
char *put_dec(char *buf, unsigned long long n)
{
if (n >= 100*1000*1000)
buf = put_dec_full8(buf, do_div(n, 100*1000*1000));
/* 1 <= n <= 1.6e11 */
if (n >= 100*1000*1000)
buf = put_dec_full8(buf, do_div(n, 100*1000*1000));
/* 1 <= n < 1e8 */
return put_dec_trunc8(buf, n);
}
#elif BITS_PER_LONG == 32 && BITS_PER_LONG_LONG == 64
static void
put_dec_full4(char *buf, unsigned r)
{
unsigned q;
/* 0 <= r < 10^4 */
q = (r * 0x147b) >> 19;
*((u16 *)buf) = decpair[r - 100*q];
buf += 2;
/* 0 <= q < 100 */
*((u16 *)buf) = decpair[q];
}
/*
* Call put_dec_full4 on x % 10000, return x / 10000.
* The approximation x/10000 == (x * 0x346DC5D7) >> 43
* holds for all x < 1,128,869,999. The largest value this
* helper will ever be asked to convert is 1,125,520,955.
* (second call in the put_dec code, assuming n is all-ones).
*/
static noinline_for_stack
unsigned put_dec_helper4(char *buf, unsigned x)
{
uint32_t q = (x * (uint64_t)0x346DC5D7) >> 43;
put_dec_full4(buf, x - q * 10000);
return q;
}
/* Based on code by Douglas W. Jones found at
* <http://www.cs.uiowa.edu/~jones/bcd/decimal.html#sixtyfour>
* (with permission from the author).
* Performs no 64-bit division and hence should be fast on 32-bit machines.
*/
static
char *put_dec(char *buf, unsigned long long n)
{
uint32_t d3, d2, d1, q, h;
if (n < 100*1000*1000)
return put_dec_trunc8(buf, n);
d1 = ((uint32_t)n >> 16); /* implicit "& 0xffff" */
h = (n >> 32);
d2 = (h ) & 0xffff;
d3 = (h >> 16); /* implicit "& 0xffff" */
/* n = 2^48 d3 + 2^32 d2 + 2^16 d1 + d0
= 281_4749_7671_0656 d3 + 42_9496_7296 d2 + 6_5536 d1 + d0 */
q = 656 * d3 + 7296 * d2 + 5536 * d1 + ((uint32_t)n & 0xffff);
q = put_dec_helper4(buf, q);
q += 7671 * d3 + 9496 * d2 + 6 * d1;
q = put_dec_helper4(buf+4, q);
q += 4749 * d3 + 42 * d2;
q = put_dec_helper4(buf+8, q);
q += 281 * d3;
buf += 12;
if (q)
buf = put_dec_trunc8(buf, q);
else while (buf[-1] == '0')
--buf;
return buf;
}
#endif
/*
* Convert passed number to decimal string.
* Returns the length of string. On buffer overflow, returns 0.
*
* If speed is not important, use snprintf(). It's easy to read the code.
*/
int num_to_str(char *buf, int size, unsigned long long num, unsigned int width)
{
/* put_dec requires 2-byte alignment of the buffer. */
char tmp[sizeof(num) * 3] __aligned(2);
int idx, len;
/* put_dec() may work incorrectly for num = 0 (generate "", not "0") */
if (num <= 9) {
tmp[0] = '0' + num;
len = 1;
} else {
len = put_dec(tmp, num) - tmp;
}
if (len > size || width > size)
return 0;
if (width > len) {
width = width - len;
for (idx = 0; idx < width; idx++)
buf[idx] = ' ';
} else {
width = 0;
}
for (idx = 0; idx < len; ++idx)
buf[idx + width] = tmp[len - idx - 1];
return len + width;
}
#define SIGN 1 /* unsigned/signed */
#define LEFT 2 /* left justified */
#define PLUS 4 /* show plus */
#define SPACE 8 /* space if plus */
#define ZEROPAD 16 /* pad with zero, must be 16 == '0' - ' ' */
#define SMALL 32 /* use lowercase in hex (must be 32 == 0x20) */
#define SPECIAL 64 /* prefix hex with "0x", octal with "0" */
static_assert(ZEROPAD == ('0' - ' '));
static_assert(SMALL == ('a' ^ 'A'));
enum format_state {
FORMAT_STATE_NONE, /* Just a string part */
FORMAT_STATE_NUM,
FORMAT_STATE_WIDTH,
FORMAT_STATE_PRECISION,
FORMAT_STATE_CHAR,
FORMAT_STATE_STR,
FORMAT_STATE_PTR,
FORMAT_STATE_PERCENT_CHAR,
FORMAT_STATE_INVALID,
};
struct printf_spec {
unsigned char flags; /* flags to number() */
unsigned char base; /* number base, 8, 10 or 16 only */
short precision; /* # of digits/chars */
int field_width; /* width of output field */
} __packed;
static_assert(sizeof(struct printf_spec) == 8);
#define FIELD_WIDTH_MAX ((1 << 23) - 1)
#define PRECISION_MAX ((1 << 15) - 1)
static noinline_for_stack
char *number(char *buf, char *end, unsigned long long num,
struct printf_spec spec)
{
/* put_dec requires 2-byte alignment of the buffer. */
char tmp[3 * sizeof(num)] __aligned(2);
char sign;
char locase;
int need_pfx = ((spec.flags & SPECIAL) && spec.base != 10);
int i;
bool is_zero = num == 0LL;
int field_width = spec.field_width;
int precision = spec.precision;
/* locase = 0 or 0x20. ORing digits or letters with 'locase'
* produces same digits or (maybe lowercased) letters */
locase = (spec.flags & SMALL);
if (spec.flags & LEFT)
spec.flags &= ~ZEROPAD;
sign = 0;
if (spec.flags & SIGN) {
if ((signed long long)num < 0) {
sign = '-';
num = -(signed long long)num;
field_width--;
} else if (spec.flags & PLUS) {
sign = '+';
field_width--;
} else if (spec.flags & SPACE) {
sign = ' ';
field_width--;
}
}
if (need_pfx) {
if (spec.base == 16)
field_width -= 2;
else if (!is_zero)
field_width--;
}
/* generate full string in tmp[], in reverse order */
i = 0;
if (num < spec.base)
tmp[i++] = hex_asc_upper[num] | locase;
else if (spec.base != 10) { /* 8 or 16 */
int mask = spec.base - 1;
int shift = 3;
if (spec.base == 16)
shift = 4;
do {
tmp[i++] = (hex_asc_upper[((unsigned char)num) & mask] | locase);
num >>= shift;
} while (num);
} else { /* base 10 */
i = put_dec(tmp, num) - tmp;
}
/* printing 100 using %2d gives "100", not "00" */
if (i > precision)
precision = i;
/* leading space padding */
field_width -= precision;
if (!(spec.flags & (ZEROPAD | LEFT))) {
while (--field_width >= 0) {
if (buf < end)
*buf = ' ';
++buf;
}
}
/* sign */
if (sign) {
if (buf < end)
*buf = sign;
++buf;
}
/* "0x" / "0" prefix */
if (need_pfx) {
if (spec.base == 16 || !is_zero) {
if (buf < end)
*buf = '0';
++buf;
}
if (spec.base == 16) {
if (buf < end)
*buf = ('X' | locase);
++buf;
}
}
/* zero or space padding */
if (!(spec.flags & LEFT)) {
char c = ' ' + (spec.flags & ZEROPAD);
while (--field_width >= 0) {
if (buf < end)
*buf = c;
++buf;
}
}
/* hmm even more zero padding? */
while (i <= --precision) {
if (buf < end)
*buf = '0';
++buf;
}
/* actual digits of result */
while (--i >= 0) {
if (buf < end)
*buf = tmp[i];
++buf;
}
/* trailing space padding */
while (--field_width >= 0) {
if (buf < end)
*buf = ' ';
++buf;
}
return buf;
}
#define special_hex_spec(size) \
(struct printf_spec) { \
.field_width = 2 + 2 * (size), /* 0x + hex */ \
.flags = SPECIAL | SMALL | ZEROPAD, \
.base = 16, \
.precision = -1, \
}
static noinline_for_stack
char *special_hex_number(char *buf, char *end, unsigned long long num, int size)
{
return number(buf, end, num, special_hex_spec(size));
}
static void move_right(char *buf, char *end, unsigned len, unsigned spaces)
{
size_t size;
if (buf >= end) /* nowhere to put anything */
return;
size = end - buf;
if (size <= spaces) {
memset(buf, ' ', size);
return;
}
if (len) {
if (len > size - spaces)
len = size - spaces;
memmove(buf + spaces, buf, len);
}
memset(buf, ' ', spaces);
}
/*
* Handle field width padding for a string.
* @buf: current buffer position
* @n: length of string
* @end: end of output buffer
* @spec: for field width and flags
* Returns: new buffer position after padding.
*/
static noinline_for_stack
char *widen_string(char *buf, int n, char *end, struct printf_spec spec)
{
unsigned spaces;
if (likely(n >= spec.field_width))
return buf;
/* we want to pad the sucker */
spaces = spec.field_width - n;
if (!(spec.flags & LEFT)) {
move_right(buf - n, end, n, spaces);
return buf + spaces;
}
while (spaces--) {
if (buf < end)
*buf = ' ';
++buf;
}
return buf;
}
/* Handle string from a well known address. */
static char *string_nocheck(char *buf, char *end, const char *s,
struct printf_spec spec)
{
int len = 0;
int lim = spec.precision;
while (lim--) {
char c = *s++;
if (!c)
break;
if (buf < end)
*buf = c;
++buf;
++len;
}
return widen_string(buf, len, end, spec);
}
static char *err_ptr(char *buf, char *end, void *ptr,
struct printf_spec spec)
{
int err = PTR_ERR(ptr);
const char *sym = errname(err);
if (sym)
return string_nocheck(buf, end, sym, spec);
/*
* Somebody passed ERR_PTR(-1234) or some other non-existing
* Efoo - or perhaps CONFIG_SYMBOLIC_ERRNAME=n. Fall back to
* printing it as its decimal representation.
*/
spec.flags |= SIGN;
spec.base = 10;
return number(buf, end, err, spec);
}
/* Be careful: error messages must fit into the given buffer. */
static char *error_string(char *buf, char *end, const char *s,
struct printf_spec spec)
{
/*
* Hard limit to avoid a completely insane messages. It actually
* works pretty well because most error messages are in
* the many pointer format modifiers.
*/
if (spec.precision == -1)
spec.precision = 2 * sizeof(void *);
return string_nocheck(buf, end, s, spec);
}
/*
* Do not call any complex external code here. Nested printk()/vsprintf()
* might cause infinite loops. Failures might break printk() and would
* be hard to debug.
*/
static const char *check_pointer_msg(const void *ptr)
{
if (!ptr)
return "(null)";
if ((unsigned long)ptr < PAGE_SIZE || IS_ERR_VALUE(ptr))
return "(efault)";
return NULL;
}
static int check_pointer(char **buf, char *end, const void *ptr,
struct printf_spec spec)
{
const char *err_msg;
err_msg = check_pointer_msg(ptr);
if (err_msg) {
*buf = error_string(*buf, end, err_msg, spec);
return -EFAULT;
}
return 0;
}
static noinline_for_stack
char *string(char *buf, char *end, const char *s,
struct printf_spec spec)
{
if (check_pointer(&buf, end, s, spec))
return buf;
return string_nocheck(buf, end, s, spec);
}
static char *pointer_string(char *buf, char *end,
const void *ptr,
struct printf_spec spec)
{
spec.base = 16;
spec.flags |= SMALL;
if (spec.field_width == -1) {
spec.field_width = 2 * sizeof(ptr);
spec.flags |= ZEROPAD;
}
return number(buf, end, (unsigned long int)ptr, spec);
}
/* Make pointers available for printing early in the boot sequence. */
static int debug_boot_weak_hash __ro_after_init;
static int __init debug_boot_weak_hash_enable(char *str)
{
debug_boot_weak_hash = 1;
pr_info("debug_boot_weak_hash enabled\n");
return 0;
}
early_param("debug_boot_weak_hash", debug_boot_weak_hash_enable);
static bool filled_random_ptr_key __read_mostly;
static siphash_key_t ptr_key __read_mostly;
static int fill_ptr_key(struct notifier_block *nb, unsigned long action, void *data)
{
get_random_bytes(&ptr_key, sizeof(ptr_key));
/* Pairs with smp_rmb() before reading ptr_key. */
smp_wmb();
WRITE_ONCE(filled_random_ptr_key, true);
return NOTIFY_DONE;
}
static int __init vsprintf_init_hashval(void)
{
static struct notifier_block fill_ptr_key_nb = { .notifier_call = fill_ptr_key };
execute_with_initialized_rng(&fill_ptr_key_nb);
return 0;
}
subsys_initcall(vsprintf_init_hashval)
/* Maps a pointer to a 32 bit unique identifier. */
static inline int __ptr_to_hashval(const void *ptr, unsigned long *hashval_out)
{
unsigned long hashval;
if (!READ_ONCE(filled_random_ptr_key))
return -EBUSY;
/* Pairs with smp_wmb() after writing ptr_key. */
smp_rmb();
#ifdef CONFIG_64BIT
hashval = (unsigned long)siphash_1u64((u64)ptr, &ptr_key);
/*
* Mask off the first 32 bits, this makes explicit that we have
* modified the address (and 32 bits is plenty for a unique ID).
*/
hashval = hashval & 0xffffffff;
#else
hashval = (unsigned long)siphash_1u32((u32)ptr, &ptr_key);
#endif
*hashval_out = hashval;
return 0;
}
int ptr_to_hashval(const void *ptr, unsigned long *hashval_out)
{
return __ptr_to_hashval(ptr, hashval_out);
}
static char *ptr_to_id(char *buf, char *end, const void *ptr,
struct printf_spec spec)
{
const char *str = sizeof(ptr) == 8 ? "(____ptrval____)" : "(ptrval)";
unsigned long hashval;
int ret;
/*
* Print the real pointer value for NULL and error pointers,
* as they are not actual addresses.
*/
if (IS_ERR_OR_NULL(ptr))
return pointer_string(buf, end, ptr, spec);
/* When debugging early boot use non-cryptographically secure hash. */
if (unlikely(debug_boot_weak_hash)) {
hashval = hash_long((unsigned long)ptr, 32);
return pointer_string(buf, end, (const void *)hashval, spec);
}
ret = __ptr_to_hashval(ptr, &hashval);
if (ret) {
spec.field_width = 2 * sizeof(ptr);
/* string length must be less than default_width */
return error_string(buf, end, str, spec);
}
return pointer_string(buf, end, (const void *)hashval, spec);
}
static char *default_pointer(char *buf, char *end, const void *ptr,
struct printf_spec spec)
{
/*
* default is to _not_ leak addresses, so hash before printing,
* unless no_hash_pointers is specified on the command line.
*/
if (unlikely(no_hash_pointers))
return pointer_string(buf, end, ptr, spec);
return ptr_to_id(buf, end, ptr, spec);
}
int kptr_restrict __read_mostly;
static noinline_for_stack
char *restricted_pointer(char *buf, char *end, const void *ptr,
struct printf_spec spec)
{
switch (kptr_restrict) {
case 0:
/* Handle as %p, hash and do _not_ leak addresses. */
return default_pointer(buf, end, ptr, spec);
case 1: {
const struct cred *cred;
/*
* kptr_restrict==1 cannot be used in IRQ context
* because its test for CAP_SYSLOG would be meaningless.
*/
if (in_hardirq() || in_serving_softirq() || in_nmi()) {
if (spec.field_width == -1)
spec.field_width = 2 * sizeof(ptr);
return error_string(buf, end, "pK-error", spec);
}
/*
* Only print the real pointer value if the current
* process has CAP_SYSLOG and is running with the
* same credentials it started with. This is because
* access to files is checked at open() time, but %pK
* checks permission at read() time. We don't want to
* leak pointer values if a binary opens a file using
* %pK and then elevates privileges before reading it.
*/
cred = current_cred();
if (!has_capability_noaudit(current, CAP_SYSLOG) ||
!uid_eq(cred->euid, cred->uid) ||
!gid_eq(cred->egid, cred->gid))
ptr = NULL;
break;
}
case 2:
default:
/* Always print 0's for %pK */
ptr = NULL;
break;
}
return pointer_string(buf, end, ptr, spec);
}
static noinline_for_stack
char *dentry_name(char *buf, char *end, const struct dentry *d, struct printf_spec spec,
const char *fmt)
{
const char *array[4], *s;
const struct dentry *p;
int depth;
int i, n;
switch (fmt[1]) {
case '2': case '3': case '4':
depth = fmt[1] - '0';
break;
default:
depth = 1;
}
rcu_read_lock();
for (i = 0; i < depth; i++, d = p) {
if (check_pointer(&buf, end, d, spec)) {
rcu_read_unlock();
return buf;
}
p = READ_ONCE(d->d_parent);
array[i] = READ_ONCE(d->d_name.name);
if (p == d) {
if (i)
array[i] = "";
i++;
break;
}
}
s = array[--i];
for (n = 0; n != spec.precision; n++, buf++) {
char c = *s++;
if (!c) {
if (!i)
break;
c = '/';
s = array[--i];
}
if (buf < end)
*buf = c;
}
rcu_read_unlock();
return widen_string(buf, n, end, spec);
}
static noinline_for_stack
char *file_dentry_name(char *buf, char *end, const struct file *f,
struct printf_spec spec, const char *fmt)
{
if (check_pointer(&buf, end, f, spec))
return buf;
return dentry_name(buf, end, f->f_path.dentry, spec, fmt);
}
#ifdef CONFIG_BLOCK
static noinline_for_stack
char *bdev_name(char *buf, char *end, struct block_device *bdev,
struct printf_spec spec, const char *fmt)
{
struct gendisk *hd;
if (check_pointer(&buf, end, bdev, spec))
return buf;
hd = bdev->bd_disk;
buf = string(buf, end, hd->disk_name, spec);
if (bdev_is_partition(bdev)) {
if (isdigit(hd->disk_name[strlen(hd->disk_name)-1])) {
if (buf < end)
*buf = 'p';
buf++;
}
buf = number(buf, end, bdev_partno(bdev), spec);
}
return buf;
}
#endif
static noinline_for_stack
char *symbol_string(char *buf, char *end, void *ptr,
struct printf_spec spec, const char *fmt)
{
unsigned long value;
#ifdef CONFIG_KALLSYMS
char sym[KSYM_SYMBOL_LEN];
#endif
if (fmt[1] == 'R')
ptr = __builtin_extract_return_addr(ptr);
value = (unsigned long)ptr;
#ifdef CONFIG_KALLSYMS
if (*fmt == 'B' && fmt[1] == 'b')
sprint_backtrace_build_id(sym, value);
else if (*fmt == 'B')
sprint_backtrace(sym, value);
else if (*fmt == 'S' && (fmt[1] == 'b' || (fmt[1] == 'R' && fmt[2] == 'b')))
sprint_symbol_build_id(sym, value);
else if (*fmt != 's')
sprint_symbol(sym, value);
else
sprint_symbol_no_offset(sym, value);
return string_nocheck(buf, end, sym, spec);
#else
return special_hex_number(buf, end, value, sizeof(void *));
#endif
}
static const struct printf_spec default_str_spec = {
.field_width = -1,
.precision = -1,
};
static const struct printf_spec default_flag_spec = {
.base = 16,
.precision = -1,
.flags = SPECIAL | SMALL,
};
static const struct printf_spec default_dec_spec = {
.base = 10,
.precision = -1,
};
static const struct printf_spec default_dec02_spec = {
.base = 10,
.field_width = 2,
.precision = -1,
.flags = ZEROPAD,
};
static const struct printf_spec default_dec04_spec = {
.base = 10,
.field_width = 4,
.precision = -1,
.flags = ZEROPAD,
};
static noinline_for_stack
char *hex_range(char *buf, char *end, u64 start_val, u64 end_val,
struct printf_spec spec)
{
buf = number(buf, end, start_val, spec);
if (start_val == end_val)
return buf;
if (buf < end)
*buf = '-';
++buf;
return number(buf, end, end_val, spec);
}
static noinline_for_stack
char *resource_string(char *buf, char *end, struct resource *res,
struct printf_spec spec, const char *fmt)
{
#ifndef IO_RSRC_PRINTK_SIZE
#define IO_RSRC_PRINTK_SIZE 6
#endif
#ifndef MEM_RSRC_PRINTK_SIZE
#define MEM_RSRC_PRINTK_SIZE 10
#endif
static const struct printf_spec io_spec = {
.base = 16,
.field_width = IO_RSRC_PRINTK_SIZE,
.precision = -1,
.flags = SPECIAL | SMALL | ZEROPAD,
};
static const struct printf_spec mem_spec = {
.base = 16,
.field_width = MEM_RSRC_PRINTK_SIZE,
.precision = -1,
.flags = SPECIAL | SMALL | ZEROPAD,
};
static const struct printf_spec bus_spec = {
.base = 16,
.field_width = 2,
.precision = -1,
.flags = SMALL | ZEROPAD,
};
static const struct printf_spec str_spec = {
.field_width = -1,
.precision = 10,
.flags = LEFT,
};
/* 32-bit res (sizeof==4): 10 chars in dec, 10 in hex ("0x" + 8)
* 64-bit res (sizeof==8): 20 chars in dec, 18 in hex ("0x" + 16) */
#define RSRC_BUF_SIZE ((2 * sizeof(resource_size_t)) + 4)
#define FLAG_BUF_SIZE (2 * sizeof(res->flags))
#define DECODED_BUF_SIZE sizeof("[mem - 64bit pref window disabled]")
#define RAW_BUF_SIZE sizeof("[mem - flags 0x]")
char sym[MAX(2*RSRC_BUF_SIZE + DECODED_BUF_SIZE,
2*RSRC_BUF_SIZE + FLAG_BUF_SIZE + RAW_BUF_SIZE)];
char *p = sym, *pend = sym + sizeof(sym);
bool decode = fmt[0] == 'R';
const struct printf_spec *specp;
if (check_pointer(&buf, end, res, spec))
return buf;
*p++ = '[';
if (res->flags & IORESOURCE_IO) {
p = string_nocheck(p, pend, "io ", str_spec);
specp = &io_spec;
} else if (res->flags & IORESOURCE_MEM) {
p = string_nocheck(p, pend, "mem ", str_spec);
specp = &mem_spec;
} else if (res->flags & IORESOURCE_IRQ) {
p = string_nocheck(p, pend, "irq ", str_spec);
specp = &default_dec_spec;
} else if (res->flags & IORESOURCE_DMA) {
p = string_nocheck(p, pend, "dma ", str_spec);
specp = &default_dec_spec;
} else if (res->flags & IORESOURCE_BUS) {
p = string_nocheck(p, pend, "bus ", str_spec);
specp = &bus_spec;
} else {
p = string_nocheck(p, pend, "??? ", str_spec);
specp = &mem_spec;
decode = false;
}
if (decode && res->flags & IORESOURCE_UNSET) {
p = string_nocheck(p, pend, "size ", str_spec);
p = number(p, pend, resource_size(res), *specp);
} else {
p = hex_range(p, pend, res->start, res->end, *specp);
}
if (decode) {
if (res->flags & IORESOURCE_MEM_64)
p = string_nocheck(p, pend, " 64bit", str_spec);
if (res->flags & IORESOURCE_PREFETCH)
p = string_nocheck(p, pend, " pref", str_spec);
if (res->flags & IORESOURCE_WINDOW)
p = string_nocheck(p, pend, " window", str_spec);
if (res->flags & IORESOURCE_DISABLED)
p = string_nocheck(p, pend, " disabled", str_spec);
} else {
p = string_nocheck(p, pend, " flags ", str_spec);
p = number(p, pend, res->flags, default_flag_spec);
}
*p++ = ']';
*p = '\0';
return string_nocheck(buf, end, sym, spec);
}
static noinline_for_stack
char *range_string(char *buf, char *end, const struct range *range,
struct printf_spec spec, const char *fmt)
{
char sym[sizeof("[range 0x0123456789abcdef-0x0123456789abcdef]")];
char *p = sym, *pend = sym + sizeof(sym);
if (check_pointer(&buf, end, range, spec))
return buf;
p = string_nocheck(p, pend, "[range ", default_str_spec);
p = hex_range(p, pend, range->start, range->end, special_hex_spec(sizeof(range->start)));
*p++ = ']';
*p = '\0';
return string_nocheck(buf, end, sym, spec);
}
static noinline_for_stack
char *hex_string(char *buf, char *end, u8 *addr, struct printf_spec spec,
const char *fmt)
{
int i, len = 1; /* if we pass '%ph[CDN]', field width remains
negative value, fallback to the default */
char separator;
if (spec.field_width == 0)
/* nothing to print */
return buf;
if (check_pointer(&buf, end, addr, spec))
return buf;
switch (fmt[1]) {
case 'C':
separator = ':';
break;
case 'D':
separator = '-';
break;
case 'N':
separator = 0;
break;
default:
separator = ' ';
break;
}
if (spec.field_width > 0)
len = min_t(int, spec.field_width, 64);
for (i = 0; i < len; ++i) {
if (buf < end)
*buf = hex_asc_hi(addr[i]);
++buf;
if (buf < end)
*buf = hex_asc_lo(addr[i]);
++buf;
if (separator && i != len - 1) {
if (buf < end)
*buf = separator;
++buf;
}
}
return buf;
}
static noinline_for_stack
char *bitmap_string(char *buf, char *end, const unsigned long *bitmap,
struct printf_spec spec, const char *fmt)
{
const int CHUNKSZ = 32;
int nr_bits = max_t(int, spec.field_width, 0);
int i, chunksz;
bool first = true;
if (check_pointer(&buf, end, bitmap, spec))
return buf;
/* reused to print numbers */
spec = (struct printf_spec){ .flags = SMALL | ZEROPAD, .base = 16 };
chunksz = nr_bits & (CHUNKSZ - 1);
if (chunksz == 0)
chunksz = CHUNKSZ;
i = ALIGN(nr_bits, CHUNKSZ) - CHUNKSZ;
for (; i >= 0; i -= CHUNKSZ) {
u32 chunkmask, val;
int word, bit;
chunkmask = ((1ULL << chunksz) - 1);
word = i / BITS_PER_LONG;
bit = i % BITS_PER_LONG;
val = (bitmap[word] >> bit) & chunkmask;
if (!first) {
if (buf < end)
*buf = ',';
buf++;
}
first = false;
spec.field_width = DIV_ROUND_UP(chunksz, 4);
buf = number(buf, end, val, spec);
chunksz = CHUNKSZ;
}
return buf;
}
static noinline_for_stack
char *bitmap_list_string(char *buf, char *end, const unsigned long *bitmap,
struct printf_spec spec, const char *fmt)
{
int nr_bits = max_t(int, spec.field_width, 0);
bool first = true;
int rbot, rtop;
if (check_pointer(&buf, end, bitmap, spec))
return buf;
for_each_set_bitrange(rbot, rtop, bitmap, nr_bits) {
if (!first) {
if (buf < end)
*buf = ',';
buf++;
}
first = false;
buf = number(buf, end, rbot, default_dec_spec);
if (rtop == rbot + 1)
continue;
if (buf < end)
*buf = '-';
buf = number(++buf, end, rtop - 1, default_dec_spec);
}
return buf;
}
static noinline_for_stack
char *mac_address_string(char *buf, char *end, u8 *addr,
struct printf_spec spec, const char *fmt)
{
char mac_addr[sizeof("xx:xx:xx:xx:xx:xx")];
char *p = mac_addr;
int i;
char separator;
bool reversed = false;
if (check_pointer(&buf, end, addr, spec))
return buf;
switch (fmt[1]) {
case 'F':
separator = '-';
break;
case 'R':
reversed = true;
fallthrough;
default:
separator = ':';
break;
}
for (i = 0; i < 6; i++) {
if (reversed)
p = hex_byte_pack(p, addr[5 - i]);
else
p = hex_byte_pack(p, addr[i]);
if (fmt[0] == 'M' && i != 5)
*p++ = separator;
}
*p = '\0';
return string_nocheck(buf, end, mac_addr, spec);
}
static noinline_for_stack
char *ip4_string(char *p, const u8 *addr, const char *fmt)
{
int i;
bool leading_zeros = (fmt[0] == 'i');
int index;
int step;
switch (fmt[2]) {
case 'h':
#ifdef __BIG_ENDIAN
index = 0;
step = 1;
#else
index = 3;
step = -1;
#endif
break;
case 'l':
index = 3;
step = -1;
break;
case 'n':
case 'b':
default:
index = 0;
step = 1;
break;
}
for (i = 0; i < 4; i++) {
char temp[4] __aligned(2); /* hold each IP quad in reverse order */
int digits = put_dec_trunc8(temp, addr[index]) - temp;
if (leading_zeros) {
if (digits < 3)
*p++ = '0';
if (digits < 2)
*p++ = '0';
}
/* reverse the digits in the quad */
while (digits--)
*p++ = temp[digits];
if (i < 3)
*p++ = '.';
index += step;
}
*p = '\0';
return p;
}
static noinline_for_stack
char *ip6_compressed_string(char *p, const char *addr)
{
int i, j, range;
unsigned char zerolength[8];
int longest = 1;
int colonpos = -1;
u16 word;
u8 hi, lo;
bool needcolon = false;
bool useIPv4;
struct in6_addr in6;
memcpy(&in6, addr, sizeof(struct in6_addr));
useIPv4 = ipv6_addr_v4mapped(&in6) || ipv6_addr_is_isatap(&in6);
memset(zerolength, 0, sizeof(zerolength));
if (useIPv4)
range = 6;
else
range = 8;
/* find position of longest 0 run */
for (i = 0; i < range; i++) {
for (j = i; j < range; j++) {
if (in6.s6_addr16[j] != 0)
break;
zerolength[i]++;
}
}
for (i = 0; i < range; i++) {
if (zerolength[i] > longest) {
longest = zerolength[i];
colonpos = i;
}
}
if (longest == 1) /* don't compress a single 0 */
colonpos = -1;
/* emit address */
for (i = 0; i < range; i++) {
if (i == colonpos) {
if (needcolon || i == 0)
*p++ = ':';
*p++ = ':';
needcolon = false;
i += longest - 1;
continue;
}
if (needcolon) {
*p++ = ':';
needcolon = false;
}
/* hex u16 without leading 0s */
word = ntohs(in6.s6_addr16[i]);
hi = word >> 8;
lo = word & 0xff;
if (hi) {
if (hi > 0x0f)
p = hex_byte_pack(p, hi);
else
*p++ = hex_asc_lo(hi);
p = hex_byte_pack(p, lo);
}
else if (lo > 0x0f)
p = hex_byte_pack(p, lo);
else
*p++ = hex_asc_lo(lo);
needcolon = true;
}
if (useIPv4) {
if (needcolon)
*p++ = ':';
p = ip4_string(p, &in6.s6_addr[12], "I4");
}
*p = '\0';
return p;
}
static noinline_for_stack
char *ip6_string(char *p, const char *addr, const char *fmt)
{
int i;
for (i = 0; i < 8; i++) {
p = hex_byte_pack(p, *addr++);
p = hex_byte_pack(p, *addr++);
if (fmt[0] == 'I' && i != 7)
*p++ = ':';
}
*p = '\0';
return p;
}
static noinline_for_stack
char *ip6_addr_string(char *buf, char *end, const u8 *addr,
struct printf_spec spec, const char *fmt)
{
char ip6_addr[sizeof("xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:255.255.255.255")];
if (fmt[0] == 'I' && fmt[2] == 'c')
ip6_compressed_string(ip6_addr, addr);
else
ip6_string(ip6_addr, addr, fmt);
return string_nocheck(buf, end, ip6_addr, spec);
}
static noinline_for_stack
char *ip4_addr_string(char *buf, char *end, const u8 *addr,
struct printf_spec spec, const char *fmt)
{
char ip4_addr[sizeof("255.255.255.255")];
ip4_string(ip4_addr, addr, fmt);
return string_nocheck(buf, end, ip4_addr, spec);
}
static noinline_for_stack
char *ip6_addr_string_sa(char *buf, char *end, const struct sockaddr_in6 *sa,
struct printf_spec spec, const char *fmt)
{
bool have_p = false, have_s = false, have_f = false, have_c = false;
char ip6_addr[sizeof("[xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:255.255.255.255]") +
sizeof(":12345") + sizeof("/123456789") +
sizeof("%1234567890")];
char *p = ip6_addr, *pend = ip6_addr + sizeof(ip6_addr);
const u8 *addr = (const u8 *) &sa->sin6_addr;
char fmt6[2] = { fmt[0], '6' };
u8 off = 0;
fmt++;
while (isalpha(*++fmt)) {
switch (*fmt) {
case 'p':
have_p = true;
break;
case 'f':
have_f = true;
break;
case 's':
have_s = true;
break;
case 'c':
have_c = true;
break;
}
}
if (have_p || have_s || have_f) {
*p = '[';
off = 1;
}
if (fmt6[0] == 'I' && have_c)
p = ip6_compressed_string(ip6_addr + off, addr);
else
p = ip6_string(ip6_addr + off, addr, fmt6);
if (have_p || have_s || have_f)
*p++ = ']';
if (have_p) {
*p++ = ':';
p = number(p, pend, ntohs(sa->sin6_port), spec);
}
if (have_f) {
*p++ = '/';
p = number(p, pend, ntohl(sa->sin6_flowinfo &
IPV6_FLOWINFO_MASK), spec);
}
if (have_s) {
*p++ = '%';
p = number(p, pend, sa->sin6_scope_id, spec);
}
*p = '\0';
return string_nocheck(buf, end, ip6_addr, spec);
}
static noinline_for_stack
char *ip4_addr_string_sa(char *buf, char *end, const struct sockaddr_in *sa,
struct printf_spec spec, const char *fmt)
{
bool have_p = false;
char *p, ip4_addr[sizeof("255.255.255.255") + sizeof(":12345")];
char *pend = ip4_addr + sizeof(ip4_addr);
const u8 *addr = (const u8 *) &sa->sin_addr.s_addr;
char fmt4[3] = { fmt[0], '4', 0 };
fmt++;
while (isalpha(*++fmt)) {
switch (*fmt) {
case 'p':
have_p = true;
break;
case 'h':
case 'l':
case 'n':
case 'b':
fmt4[2] = *fmt;
break;
}
}
p = ip4_string(ip4_addr, addr, fmt4);
if (have_p) {
*p++ = ':';
p = number(p, pend, ntohs(sa->sin_port), spec);
}
*p = '\0';
return string_nocheck(buf, end, ip4_addr, spec);
}
static noinline_for_stack
char *ip_addr_string(char *buf, char *end, const void *ptr,
struct printf_spec spec, const char *fmt)
{
char *err_fmt_msg;
if (check_pointer(&buf, end, ptr, spec))
return buf;
switch (fmt[1]) {
case '6':
return ip6_addr_string(buf, end, ptr, spec, fmt);
case '4':
return ip4_addr_string(buf, end, ptr, spec, fmt);
case 'S': {
const union {
struct sockaddr raw;
struct sockaddr_in v4;
struct sockaddr_in6 v6;
} *sa = ptr;
switch (sa->raw.sa_family) {
case AF_INET:
return ip4_addr_string_sa(buf, end, &sa->v4, spec, fmt);
case AF_INET6:
return ip6_addr_string_sa(buf, end, &sa->v6, spec, fmt);
default:
return error_string(buf, end, "(einval)", spec);
}}
}
err_fmt_msg = fmt[0] == 'i' ? "(%pi?)" : "(%pI?)";
return error_string(buf, end, err_fmt_msg, spec);
}
static noinline_for_stack
char *escaped_string(char *buf, char *end, u8 *addr, struct printf_spec spec,
const char *fmt)
{
bool found = true;
int count = 1;
unsigned int flags = 0;
int len;
if (spec.field_width == 0)
return buf; /* nothing to print */
if (check_pointer(&buf, end, addr, spec))
return buf;
do {
switch (fmt[count++]) {
case 'a':
flags |= ESCAPE_ANY;
break;
case 'c':
flags |= ESCAPE_SPECIAL;
break;
case 'h':
flags |= ESCAPE_HEX;
break;
case 'n':
flags |= ESCAPE_NULL;
break;
case 'o':
flags |= ESCAPE_OCTAL;
break;
case 'p':
flags |= ESCAPE_NP;
break;
case 's':
flags |= ESCAPE_SPACE;
break;
default:
found = false;
break;
}
} while (found);
if (!flags)
flags = ESCAPE_ANY_NP;
len = spec.field_width < 0 ? 1 : spec.field_width;
/*
* string_escape_mem() writes as many characters as it can to
* the given buffer, and returns the total size of the output
* had the buffer been big enough.
*/
buf += string_escape_mem(addr, len, buf, buf < end ? end - buf : 0, flags, NULL);
return buf;
}
__diag_push();
__diag_ignore(GCC, all, "-Wsuggest-attribute=format",
"Not a valid __printf() conversion candidate.");
static char *va_format(char *buf, char *end, struct va_format *va_fmt,
struct printf_spec spec)
{
va_list va;
if (check_pointer(&buf, end, va_fmt, spec))
return buf;
va_copy(va, *va_fmt->va);
buf += vsnprintf(buf, end > buf ? end - buf : 0, va_fmt->fmt, va);
va_end(va);
return buf;
}
__diag_pop();
static noinline_for_stack
char *uuid_string(char *buf, char *end, const u8 *addr,
struct printf_spec spec, const char *fmt)
{
char uuid[UUID_STRING_LEN + 1];
char *p = uuid;
int i;
const u8 *index = uuid_index;
bool uc = false;
if (check_pointer(&buf, end, addr, spec))
return buf;
switch (*(++fmt)) {
case 'L':
uc = true;
fallthrough;
case 'l':
index = guid_index;
break;
case 'B':
uc = true;
break;
}
for (i = 0; i < 16; i++) {
if (uc)
p = hex_byte_pack_upper(p, addr[index[i]]);
else
p = hex_byte_pack(p, addr[index[i]]);
switch (i) {
case 3:
case 5:
case 7:
case 9:
*p++ = '-';
break;
}
}
*p = 0;
return string_nocheck(buf, end, uuid, spec);
}
static noinline_for_stack
char *netdev_bits(char *buf, char *end, const void *addr,
struct printf_spec spec, const char *fmt)
{
unsigned long long num;
int size;
if (check_pointer(&buf, end, addr, spec))
return buf;
switch (fmt[1]) {
case 'F':
num = *(const netdev_features_t *)addr;
size = sizeof(netdev_features_t);
break;
default:
return error_string(buf, end, "(%pN?)", spec);
}
return special_hex_number(buf, end, num, size);
}
static noinline_for_stack
char *fourcc_string(char *buf, char *end, const u32 *fourcc,
struct printf_spec spec, const char *fmt)
{
char output[sizeof("0123 little-endian (0x01234567)")];
char *p = output;
unsigned int i;
bool pixel_fmt = false;
u32 orig, val;
if (fmt[1] != 'c')
return error_string(buf, end, "(%p4?)", spec);
if (check_pointer(&buf, end, fourcc, spec))
return buf;
orig = get_unaligned(fourcc);
switch (fmt[2]) {
case 'h':
if (fmt[3] == 'R')
orig = swab32(orig);
break;
case 'l':
orig = (__force u32)cpu_to_le32(orig);
break;
case 'b':
orig = (__force u32)cpu_to_be32(orig);
break;
case 'c':
/* Pixel formats are printed LSB-first */
pixel_fmt = true;
break;
default:
return error_string(buf, end, "(%p4?)", spec);
}
val = pixel_fmt ? swab32(orig & ~BIT(31)) : orig;
for (i = 0; i < sizeof(u32); i++) {
unsigned char c = val >> ((3 - i) * 8);
/* Print non-control ASCII characters as-is, dot otherwise */
*p++ = isascii(c) && isprint(c) ? c : '.';
}
if (pixel_fmt) {
*p++ = ' ';
strcpy(p, orig & BIT(31) ? "big-endian" : "little-endian");
p += strlen(p);
}
*p++ = ' ';
*p++ = '(';
p = special_hex_number(p, output + sizeof(output) - 2, orig, sizeof(u32));
*p++ = ')';
*p = '\0';
return string(buf, end, output, spec);
}
static noinline_for_stack
char *address_val(char *buf, char *end, const void *addr,
struct printf_spec spec, const char *fmt)
{
unsigned long long num;
int size;
if (check_pointer(&buf, end, addr, spec))
return buf;
switch (fmt[1]) {
case 'd':
num = *(const dma_addr_t *)addr;
size = sizeof(dma_addr_t);
break;
case 'p':
default:
num = *(const phys_addr_t *)addr;
size = sizeof(phys_addr_t);
break;
}
return special_hex_number(buf, end, num, size);
}
static noinline_for_stack
char *date_str(char *buf, char *end, const struct rtc_time *tm, bool r)
{
int year = tm->tm_year + (r ? 0 : 1900);
int mon = tm->tm_mon + (r ? 0 : 1);
buf = number(buf, end, year, default_dec04_spec);
if (buf < end)
*buf = '-';
buf++;
buf = number(buf, end, mon, default_dec02_spec);
if (buf < end)
*buf = '-';
buf++;
return number(buf, end, tm->tm_mday, default_dec02_spec);
}
static noinline_for_stack
char *time_str(char *buf, char *end, const struct rtc_time *tm, bool r)
{
buf = number(buf, end, tm->tm_hour, default_dec02_spec);
if (buf < end)
*buf = ':';
buf++;
buf = number(buf, end, tm->tm_min, default_dec02_spec);
if (buf < end)
*buf = ':';
buf++;
return number(buf, end, tm->tm_sec, default_dec02_spec);
}
static noinline_for_stack
char *rtc_str(char *buf, char *end, const struct rtc_time *tm,
struct printf_spec spec, const char *fmt)
{
bool have_t = true, have_d = true;
bool raw = false, iso8601_separator = true;
bool found = true;
int count = 2;
switch (fmt[count]) {
case 'd':
have_t = false;
count++;
break;
case 't':
have_d = false;
count++;
break;
}
do {
switch (fmt[count++]) {
case 'r':
raw = true;
break;
case 's':
iso8601_separator = false;
break;
default:
found = false;
break;
}
} while (found);
if (have_d)
buf = date_str(buf, end, tm, raw);
if (have_d && have_t) {
if (buf < end)
*buf = iso8601_separator ? 'T' : ' ';
buf++;
}
if (have_t)
buf = time_str(buf, end, tm, raw);
return buf;
}
static noinline_for_stack
char *time64_str(char *buf, char *end, const time64_t time,
struct printf_spec spec, const char *fmt)
{
struct rtc_time rtc_time;
struct tm tm;
time64_to_tm(time, 0, &tm);
rtc_time.tm_sec = tm.tm_sec;
rtc_time.tm_min = tm.tm_min;
rtc_time.tm_hour = tm.tm_hour;
rtc_time.tm_mday = tm.tm_mday;
rtc_time.tm_mon = tm.tm_mon;
rtc_time.tm_year = tm.tm_year;
rtc_time.tm_wday = tm.tm_wday;
rtc_time.tm_yday = tm.tm_yday;
rtc_time.tm_isdst = 0;
return rtc_str(buf, end, &rtc_time, spec, fmt);
}
static noinline_for_stack
char *timespec64_str(char *buf, char *end, const struct timespec64 *ts,
struct printf_spec spec, const char *fmt)
{
static const struct printf_spec default_dec09_spec = {
.base = 10,
.field_width = 9,
.precision = -1,
.flags = ZEROPAD,
};
if (fmt[2] == 'p')
buf = number(buf, end, ts->tv_sec, default_dec_spec);
else
buf = time64_str(buf, end, ts->tv_sec, spec, fmt);
if (buf < end)
*buf = '.';
buf++;
return number(buf, end, ts->tv_nsec, default_dec09_spec);
}
static noinline_for_stack
char *time_and_date(char *buf, char *end, void *ptr, struct printf_spec spec,
const char *fmt)
{
if (check_pointer(&buf, end, ptr, spec))
return buf;
switch (fmt[1]) {
case 'R':
return rtc_str(buf, end, (const struct rtc_time *)ptr, spec, fmt);
case 'S':
return timespec64_str(buf, end, (const struct timespec64 *)ptr, spec, fmt);
case 'T':
return time64_str(buf, end, *(const time64_t *)ptr, spec, fmt);
default:
return error_string(buf, end, "(%pt?)", spec);
}
}
static noinline_for_stack
char *clock(char *buf, char *end, struct clk *clk, struct printf_spec spec,
const char *fmt)
{
if (!IS_ENABLED(CONFIG_HAVE_CLK))
return error_string(buf, end, "(%pC?)", spec);
if (check_pointer(&buf, end, clk, spec))
return buf;
#ifdef CONFIG_COMMON_CLK
return string(buf, end, __clk_get_name(clk), spec);
#else
return ptr_to_id(buf, end, clk, spec);
#endif
}
static
char *format_flags(char *buf, char *end, unsigned long flags,
const struct trace_print_flags *names)
{
unsigned long mask;
for ( ; flags && names->name; names++) {
mask = names->mask;
if ((flags & mask) != mask)
continue;
buf = string(buf, end, names->name, default_str_spec);
flags &= ~mask;
if (flags) {
if (buf < end)
*buf = '|';
buf++;
}
}
if (flags)
buf = number(buf, end, flags, default_flag_spec);
return buf;
}
struct page_flags_fields {
int width;
int shift;
int mask;
const struct printf_spec *spec;
const char *name;
};
static const struct page_flags_fields pff[] = {
{SECTIONS_WIDTH, SECTIONS_PGSHIFT, SECTIONS_MASK,
&default_dec_spec, "section"},
{NODES_WIDTH, NODES_PGSHIFT, NODES_MASK,
&default_dec_spec, "node"},
{ZONES_WIDTH, ZONES_PGSHIFT, ZONES_MASK,
&default_dec_spec, "zone"},
{LAST_CPUPID_WIDTH, LAST_CPUPID_PGSHIFT, LAST_CPUPID_MASK,
&default_flag_spec, "lastcpupid"},
{KASAN_TAG_WIDTH, KASAN_TAG_PGSHIFT, KASAN_TAG_MASK,
&default_flag_spec, "kasantag"},
};
static
char *format_page_flags(char *buf, char *end, unsigned long flags)
{
unsigned long main_flags = flags & PAGEFLAGS_MASK;
bool append = false;
int i;
buf = number(buf, end, flags, default_flag_spec);
if (buf < end)
*buf = '(';
buf++;
/* Page flags from the main area. */
if (main_flags) {
buf = format_flags(buf, end, main_flags, pageflag_names);
append = true;
}
/* Page flags from the fields area */
for (i = 0; i < ARRAY_SIZE(pff); i++) {
/* Skip undefined fields. */
if (!pff[i].width)
continue;
/* Format: Flag Name + '=' (equals sign) + Number + '|' (separator) */
if (append) {
if (buf < end)
*buf = '|';
buf++;
}
buf = string(buf, end, pff[i].name, default_str_spec);
if (buf < end)
*buf = '=';
buf++;
buf = number(buf, end, (flags >> pff[i].shift) & pff[i].mask,
*pff[i].spec);
append = true;
}
if (buf < end)
*buf = ')';
buf++;
return buf;
}
static noinline_for_stack
char *flags_string(char *buf, char *end, void *flags_ptr,
struct printf_spec spec, const char *fmt)
{
unsigned long flags;
const struct trace_print_flags *names;
if (check_pointer(&buf, end, flags_ptr, spec))
return buf;
switch (fmt[1]) {
case 'p':
return format_page_flags(buf, end, *(unsigned long *)flags_ptr);
case 'v':
flags = *(unsigned long *)flags_ptr;
names = vmaflag_names;
break;
case 'g':
flags = (__force unsigned long)(*(gfp_t *)flags_ptr);
names = gfpflag_names;
break;
default:
return error_string(buf, end, "(%pG?)", spec);
}
return format_flags(buf, end, flags, names);
}
static noinline_for_stack
char *fwnode_full_name_string(struct fwnode_handle *fwnode, char *buf,
char *end)
{
int depth;
/* Loop starting from the root node to the current node. */
for (depth = fwnode_count_parents(fwnode); depth >= 0; depth--) {
/*
* Only get a reference for other nodes (i.e. parent nodes).
* fwnode refcount may be 0 here.
*/
struct fwnode_handle *__fwnode = depth ?
fwnode_get_nth_parent(fwnode, depth) : fwnode;
buf = string(buf, end, fwnode_get_name_prefix(__fwnode),
default_str_spec);
buf = string(buf, end, fwnode_get_name(__fwnode),
default_str_spec);
if (depth)
fwnode_handle_put(__fwnode);
}
return buf;
}
static noinline_for_stack
char *device_node_string(char *buf, char *end, struct device_node *dn,
struct printf_spec spec, const char *fmt)
{
char tbuf[sizeof("xxxx") + 1];
const char *p;
int ret;
char *buf_start = buf;
struct property *prop;
bool has_mult, pass;
struct printf_spec str_spec = spec;
str_spec.field_width = -1;
if (fmt[0] != 'F')
return error_string(buf, end, "(%pO?)", spec);
if (!IS_ENABLED(CONFIG_OF))
return error_string(buf, end, "(%pOF?)", spec);
if (check_pointer(&buf, end, dn, spec))
return buf;
/* simple case without anything any more format specifiers */
fmt++;
if (fmt[0] == '\0' || strcspn(fmt,"fnpPFcC") > 0)
fmt = "f";
for (pass = false; strspn(fmt,"fnpPFcC"); fmt++, pass = true) {
int precision;
if (pass) {
if (buf < end)
*buf = ':';
buf++;
}
switch (*fmt) {
case 'f': /* full_name */
buf = fwnode_full_name_string(of_fwnode_handle(dn), buf,
end);
break;
case 'n': /* name */
p = fwnode_get_name(of_fwnode_handle(dn));
precision = str_spec.precision;
str_spec.precision = strchrnul(p, '@') - p;
buf = string(buf, end, p, str_spec);
str_spec.precision = precision;
break;
case 'p': /* phandle */
buf = number(buf, end, (unsigned int)dn->phandle, default_dec_spec);
break;
case 'P': /* path-spec */
p = fwnode_get_name(of_fwnode_handle(dn));
if (!p[1])
p = "/";
buf = string(buf, end, p, str_spec);
break;
case 'F': /* flags */
tbuf[0] = of_node_check_flag(dn, OF_DYNAMIC) ? 'D' : '-';
tbuf[1] = of_node_check_flag(dn, OF_DETACHED) ? 'd' : '-';
tbuf[2] = of_node_check_flag(dn, OF_POPULATED) ? 'P' : '-';
tbuf[3] = of_node_check_flag(dn, OF_POPULATED_BUS) ? 'B' : '-';
tbuf[4] = 0;
buf = string_nocheck(buf, end, tbuf, str_spec);
break;
case 'c': /* major compatible string */
ret = of_property_read_string(dn, "compatible", &p);
if (!ret)
buf = string(buf, end, p, str_spec);
break;
case 'C': /* full compatible string */
has_mult = false;
of_property_for_each_string(dn, "compatible", prop, p) {
if (has_mult)
buf = string_nocheck(buf, end, ",", str_spec);
buf = string_nocheck(buf, end, "\"", str_spec);
buf = string(buf, end, p, str_spec);
buf = string_nocheck(buf, end, "\"", str_spec);
has_mult = true;
}
break;
default:
break;
}
}
return widen_string(buf, buf - buf_start, end, spec);
}
static noinline_for_stack
char *fwnode_string(char *buf, char *end, struct fwnode_handle *fwnode,
struct printf_spec spec, const char *fmt)
{
struct printf_spec str_spec = spec;
char *buf_start = buf;
str_spec.field_width = -1;
if (*fmt != 'w')
return error_string(buf, end, "(%pf?)", spec);
if (check_pointer(&buf, end, fwnode, spec))
return buf;
fmt++;
switch (*fmt) {
case 'P': /* name */
buf = string(buf, end, fwnode_get_name(fwnode), str_spec);
break;
case 'f': /* full_name */
default:
buf = fwnode_full_name_string(fwnode, buf, end);
break;
}
return widen_string(buf, buf - buf_start, end, spec);
}
static noinline_for_stack
char *resource_or_range(const char *fmt, char *buf, char *end, void *ptr,
struct printf_spec spec)
{
if (*fmt == 'r' && fmt[1] == 'a')
return range_string(buf, end, ptr, spec, fmt);
return resource_string(buf, end, ptr, spec, fmt);
}
void __init hash_pointers_finalize(bool slub_debug)
{
switch (hash_pointers_mode) {
case HASH_PTR_ALWAYS:
no_hash_pointers = false;
break;
case HASH_PTR_NEVER:
no_hash_pointers = true;
break;
case HASH_PTR_AUTO:
default:
no_hash_pointers = slub_debug;
break;
}
if (!no_hash_pointers)
return;
pr_warn("**********************************************************\n");
pr_warn("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n");
pr_warn("** **\n");
pr_warn("** This system shows unhashed kernel memory addresses **\n");
pr_warn("** via the console, logs, and other interfaces. This **\n");
pr_warn("** might reduce the security of your system. **\n");
pr_warn("** **\n");
pr_warn("** If you see this message and you are not debugging **\n");
pr_warn("** the kernel, report this immediately to your system **\n");
pr_warn("** administrator! **\n");
pr_warn("** **\n");
pr_warn("** Use hash_pointers=always to force this mode off **\n");
pr_warn("** **\n");
pr_warn("** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **\n");
pr_warn("**********************************************************\n");
}
static int __init hash_pointers_mode_parse(char *str)
{
if (!str) {
pr_warn("Hash pointers mode empty; falling back to auto.\n");
hash_pointers_mode = HASH_PTR_AUTO;
} else if (strncmp(str, "auto", 4) == 0) {
pr_info("Hash pointers mode set to auto.\n");
hash_pointers_mode = HASH_PTR_AUTO;
} else if (strncmp(str, "never", 5) == 0) {
pr_info("Hash pointers mode set to never.\n");
hash_pointers_mode = HASH_PTR_NEVER;
} else if (strncmp(str, "always", 6) == 0) {
pr_info("Hash pointers mode set to always.\n");
hash_pointers_mode = HASH_PTR_ALWAYS;
} else {
pr_warn("Unknown hash_pointers mode '%s' specified; assuming auto.\n", str);
hash_pointers_mode = HASH_PTR_AUTO;
}
return 0;
}
early_param("hash_pointers", hash_pointers_mode_parse);
static int __init no_hash_pointers_enable(char *str)
{
return hash_pointers_mode_parse("never");
}
early_param("no_hash_pointers", no_hash_pointers_enable);
/*
* Show a '%p' thing. A kernel extension is that the '%p' is followed
* by an extra set of alphanumeric characters that are extended format
* specifiers.
*
* Please update scripts/checkpatch.pl when adding/removing conversion
* characters. (Search for "check for vsprintf extension").
*
* Right now we handle:
*
* - 'S' For symbolic direct pointers (or function descriptors) with offset
* - 's' For symbolic direct pointers (or function descriptors) without offset
* - '[Ss]R' as above with __builtin_extract_return_addr() translation
* - 'S[R]b' as above with module build ID (for use in backtraces)
* - '[Ff]' %pf and %pF were obsoleted and later removed in favor of
* %ps and %pS. Be careful when re-using these specifiers.
* - 'B' For backtraced symbolic direct pointers with offset
* - 'Bb' as above with module build ID (for use in backtraces)
* - 'R' For decoded struct resource, e.g., [mem 0x0-0x1f 64bit pref]
* - 'r' For raw struct resource, e.g., [mem 0x0-0x1f flags 0x201]
* - 'ra' For struct ranges, e.g., [range 0x0000000000000000 - 0x00000000000000ff]
* - 'b[l]' For a bitmap, the number of bits is determined by the field
* width which must be explicitly specified either as part of the
* format string '%32b[l]' or through '%*b[l]', [l] selects
* range-list format instead of hex format
* - 'M' For a 6-byte MAC address, it prints the address in the
* usual colon-separated hex notation
* - 'm' For a 6-byte MAC address, it prints the hex address without colons
* - 'MF' For a 6-byte MAC FDDI address, it prints the address
* with a dash-separated hex notation
* - '[mM]R' For a 6-byte MAC address, Reverse order (Bluetooth)
* - 'I' [46] for IPv4/IPv6 addresses printed in the usual way
* IPv4 uses dot-separated decimal without leading 0's (1.2.3.4)
* IPv6 uses colon separated network-order 16 bit hex with leading 0's
* [S][pfs]
* Generic IPv4/IPv6 address (struct sockaddr *) that falls back to
* [4] or [6] and is able to print port [p], flowinfo [f], scope [s]
* - 'i' [46] for 'raw' IPv4/IPv6 addresses
* IPv6 omits the colons (01020304...0f)
* IPv4 uses dot-separated decimal with leading 0's (010.123.045.006)
* [S][pfs]
* Generic IPv4/IPv6 address (struct sockaddr *) that falls back to
* [4] or [6] and is able to print port [p], flowinfo [f], scope [s]
* - '[Ii][4S][hnbl]' IPv4 addresses in host, network, big or little endian order
* - 'I[6S]c' for IPv6 addresses printed as specified by
* https://tools.ietf.org/html/rfc5952
* - 'E[achnops]' For an escaped buffer, where rules are defined by combination
* of the following flags (see string_escape_mem() for the
* details):
* a - ESCAPE_ANY
* c - ESCAPE_SPECIAL
* h - ESCAPE_HEX
* n - ESCAPE_NULL
* o - ESCAPE_OCTAL
* p - ESCAPE_NP
* s - ESCAPE_SPACE
* By default ESCAPE_ANY_NP is used.
* - 'U' For a 16 byte UUID/GUID, it prints the UUID/GUID in the form
* "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
* Options for %pU are:
* b big endian lower case hex (default)
* B big endian UPPER case hex
* l little endian lower case hex
* L little endian UPPER case hex
* big endian output byte order is:
* [0][1][2][3]-[4][5]-[6][7]-[8][9]-[10][11][12][13][14][15]
* little endian output byte order is:
* [3][2][1][0]-[5][4]-[7][6]-[8][9]-[10][11][12][13][14][15]
* - 'V' For a struct va_format which contains a format string * and va_list *,
* call vsnprintf(->format, *->va_list).
* Implements a "recursive vsnprintf".
* Do not use this feature without some mechanism to verify the
* correctness of the format string and va_list arguments.
* - 'K' For a kernel pointer that should be hidden from unprivileged users.
* Use only for procfs, sysfs and similar files, not printk(); please
* read the documentation (path below) first.
* - 'NF' For a netdev_features_t
* - '4cc' V4L2 or DRM FourCC code, with endianness and raw numerical value.
* - '4c[h[R]lb]' For generic FourCC code with raw numerical value. Both are
* displayed in the big-endian format. This is the opposite of V4L2 or
* DRM FourCCs.
* The additional specifiers define what endianness is used to load
* the stored bytes. The data might be interpreted using the host,
* reversed host byte order, little-endian, or big-endian.
* - 'h[CDN]' For a variable-length buffer, it prints it as a hex string with
* a certain separator (' ' by default):
* C colon
* D dash
* N no separator
* The maximum supported length is 64 bytes of the input. Consider
* to use print_hex_dump() for the larger input.
* - 'a[pd]' For address types [p] phys_addr_t, [d] dma_addr_t and derivatives
* (default assumed to be phys_addr_t, passed by reference)
* - 'd[234]' For a dentry name (optionally 2-4 last components)
* - 'D[234]' Same as 'd' but for a struct file
* - 'g' For block_device name (gendisk + partition number)
* - 't[RST][dt][r][s]' For time and date as represented by:
* R struct rtc_time
* S struct timespec64
* T time64_t
* - 'tSp' For time represented by struct timespec64 printed as <seconds>.<nanoseconds>
* - 'C' For a clock, it prints the name (Common Clock Framework) or address
* (legacy clock framework) of the clock
* - 'G' For flags to be printed as a collection of symbolic strings that would
* construct the specific value. Supported flags given by option:
* p page flags (see struct page) given as pointer to unsigned long
* g gfp flags (GFP_* and __GFP_*) given as pointer to gfp_t
* v vma flags (VM_*) given as pointer to unsigned long
* - 'OF[fnpPcCF]' For a device tree object
* Without any optional arguments prints the full_name
* f device node full_name
* n device node name
* p device node phandle
* P device node path spec (name + @unit)
* F device node flags
* c major compatible string
* C full compatible string
* - 'fw[fP]' For a firmware node (struct fwnode_handle) pointer
* Without an option prints the full name of the node
* f full name
* P node name, including a possible unit address
* - 'x' For printing the address unmodified. Equivalent to "%lx".
* Please read the documentation (path below) before using!
* - '[ku]s' For a BPF/tracing related format specifier, e.g. used out of
* bpf_trace_printk() where [ku] prefix specifies either kernel (k)
* or user (u) memory to probe, and:
* s a string, equivalent to "%s" on direct vsnprintf() use
*
* ** When making changes please also update:
* Documentation/core-api/printk-formats.rst
*
* Note: The default behaviour (unadorned %p) is to hash the address,
* rendering it useful as a unique identifier.
*
* There is also a '%pA' format specifier, but it is only intended to be used
* from Rust code to format core::fmt::Arguments. Do *not* use it from C.
* See rust/kernel/print.rs for details.
*/
static noinline_for_stack
char *pointer(const char *fmt, char *buf, char *end, void *ptr,
struct printf_spec spec)
{
switch (*fmt) {
case 'S':
case 's':
ptr = dereference_symbol_descriptor(ptr);
fallthrough;
case 'B':
return symbol_string(buf, end, ptr, spec, fmt);
case 'R':
case 'r':
return resource_or_range(fmt, buf, end, ptr, spec);
case 'h':
return hex_string(buf, end, ptr, spec, fmt);
case 'b':
switch (fmt[1]) {
case 'l':
return bitmap_list_string(buf, end, ptr, spec, fmt);
default:
return bitmap_string(buf, end, ptr, spec, fmt);
}
case 'M': /* Colon separated: 00:01:02:03:04:05 */
case 'm': /* Contiguous: 000102030405 */
/* [mM]F (FDDI) */
/* [mM]R (Reverse order; Bluetooth) */
return mac_address_string(buf, end, ptr, spec, fmt);
case 'I': /* Formatted IP supported
* 4: 1.2.3.4
* 6: 0001:0203:...:0708
* 6c: 1::708 or 1::1.2.3.4
*/
case 'i': /* Contiguous:
* 4: 001.002.003.004
* 6: 000102...0f
*/
return ip_addr_string(buf, end, ptr, spec, fmt);
case 'E':
return escaped_string(buf, end, ptr, spec, fmt);
case 'U':
return uuid_string(buf, end, ptr, spec, fmt);
case 'V':
return va_format(buf, end, ptr, spec);
case 'K':
return restricted_pointer(buf, end, ptr, spec);
case 'N':
return netdev_bits(buf, end, ptr, spec, fmt);
case '4':
return fourcc_string(buf, end, ptr, spec, fmt);
case 'a':
return address_val(buf, end, ptr, spec, fmt);
case 'd':
return dentry_name(buf, end, ptr, spec, fmt);
case 't':
return time_and_date(buf, end, ptr, spec, fmt);
case 'C':
return clock(buf, end, ptr, spec, fmt);
case 'D':
return file_dentry_name(buf, end, ptr, spec, fmt);
#ifdef CONFIG_BLOCK
case 'g':
return bdev_name(buf, end, ptr, spec, fmt);
#endif
case 'G':
return flags_string(buf, end, ptr, spec, fmt);
case 'O':
return device_node_string(buf, end, ptr, spec, fmt + 1);
case 'f':
return fwnode_string(buf, end, ptr, spec, fmt + 1);
case 'A':
if (!IS_ENABLED(CONFIG_RUST)) {
WARN_ONCE(1, "Please remove %%pA from non-Rust code\n");
return error_string(buf, end, "(%pA?)", spec);
}
return rust_fmt_argument(buf, end, ptr);
case 'x':
return pointer_string(buf, end, ptr, spec);
case 'e':
/* %pe with a non-ERR_PTR gets treated as plain %p */
if (!IS_ERR(ptr))
return default_pointer(buf, end, ptr, spec);
return err_ptr(buf, end, ptr, spec);
case 'u':
case 'k':
switch (fmt[1]) {
case 's':
return string(buf, end, ptr, spec);
default:
return error_string(buf, end, "(einval)", spec);
}
default:
return default_pointer(buf, end, ptr, spec);
}
}
struct fmt {
const char *str;
unsigned char state; // enum format_state
unsigned char size; // size of numbers
};
#define SPEC_CHAR(x, flag) [(x)-32] = flag
static unsigned char spec_flag(unsigned char c)
{
static const unsigned char spec_flag_array[] = {
SPEC_CHAR(' ', SPACE),
SPEC_CHAR('#', SPECIAL),
SPEC_CHAR('+', PLUS),
SPEC_CHAR('-', LEFT),
SPEC_CHAR('0', ZEROPAD),
};
c -= 32;
return (c < sizeof(spec_flag_array)) ? spec_flag_array[c] : 0;
}
/*
* Helper function to decode printf style format.
* Each call decode a token from the format and return the
* number of characters read (or likely the delta where it wants
* to go on the next call).
* The decoded token is returned through the parameters
*
* 'h', 'l', or 'L' for integer fields
* 'z' support added 23/7/1999 S.H.
* 'z' changed to 'Z' --davidm 1/25/99
* 'Z' changed to 'z' --adobriyan 2017-01-25
* 't' added for ptrdiff_t
*
* @fmt: the format string
* @type of the token returned
* @flags: various flags such as +, -, # tokens..
* @field_width: overwritten width
* @base: base of the number (octal, hex, ...)
* @precision: precision of a number
* @qualifier: qualifier of a number (long, size_t, ...)
*/
static noinline_for_stack
struct fmt format_decode(struct fmt fmt, struct printf_spec *spec)
{
const char *start = fmt.str;
char flag;
/* we finished early by reading the field width */
if (unlikely(fmt.state == FORMAT_STATE_WIDTH)) {
if (spec->field_width < 0) {
spec->field_width = -spec->field_width;
spec->flags |= LEFT;
}
fmt.state = FORMAT_STATE_NONE;
goto precision;
}
/* we finished early by reading the precision */
if (unlikely(fmt.state == FORMAT_STATE_PRECISION)) {
if (spec->precision < 0)
spec->precision = 0;
fmt.state = FORMAT_STATE_NONE;
goto qualifier;
}
/* By default */
fmt.state = FORMAT_STATE_NONE;
for (; *fmt.str ; fmt.str++) {
if (*fmt.str == '%')
break;
}
/* Return the current non-format string */
if (fmt.str != start || !*fmt.str)
return fmt;
/* Process flags. This also skips the first '%' */
spec->flags = 0;
do {
/* this also skips first '%' */
flag = spec_flag(*++fmt.str);
spec->flags |= flag;
} while (flag);
/* get field width */
spec->field_width = -1;
if (isdigit(*fmt.str))
spec->field_width = skip_atoi(&fmt.str);
else if (unlikely(*fmt.str == '*')) {
/* it's the next argument */
fmt.state = FORMAT_STATE_WIDTH;
fmt.str++;
return fmt;
}
precision:
/* get the precision */
spec->precision = -1;
if (unlikely(*fmt.str == '.')) {
fmt.str++;
if (isdigit(*fmt.str)) {
spec->precision = skip_atoi(&fmt.str);
if (spec->precision < 0)
spec->precision = 0;
} else if (*fmt.str == '*') {
/* it's the next argument */
fmt.state = FORMAT_STATE_PRECISION;
fmt.str++;
return fmt;
}
}
qualifier:
/* Set up default numeric format */
spec->base = 10;
fmt.state = FORMAT_STATE_NUM;
fmt.size = sizeof(int);
static const struct format_state {
unsigned char state;
unsigned char size;
unsigned char flags_or_double_size;
unsigned char base;
} lookup_state[256] = {
// Length
['l'] = { 0, sizeof(long), sizeof(long long) },
['L'] = { 0, sizeof(long long) },
['h'] = { 0, sizeof(short), sizeof(char) },
['H'] = { 0, sizeof(char) }, // Questionable historical
['z'] = { 0, sizeof(size_t) },
['t'] = { 0, sizeof(ptrdiff_t) },
// Non-numeric formats
['c'] = { FORMAT_STATE_CHAR },
['s'] = { FORMAT_STATE_STR },
['p'] = { FORMAT_STATE_PTR },
['%'] = { FORMAT_STATE_PERCENT_CHAR },
// Numerics
['o'] = { FORMAT_STATE_NUM, 0, 0, 8 },
['x'] = { FORMAT_STATE_NUM, 0, SMALL, 16 },
['X'] = { FORMAT_STATE_NUM, 0, 0, 16 },
['d'] = { FORMAT_STATE_NUM, 0, SIGN, 10 },
['i'] = { FORMAT_STATE_NUM, 0, SIGN, 10 },
['u'] = { FORMAT_STATE_NUM, 0, 0, 10, },
/*
* Since %n poses a greater security risk than
* utility, treat it as any other invalid or
* unsupported format specifier.
*/
};
const struct format_state *p = lookup_state + (u8)*fmt.str;
if (p->size) {
fmt.size = p->size;
if (p->flags_or_double_size && fmt.str[0] == fmt.str[1]) {
fmt.size = p->flags_or_double_size;
fmt.str++;
}
fmt.str++;
p = lookup_state + *fmt.str;
}
if (p->state) {
if (p->base)
spec->base = p->base;
spec->flags |= p->flags_or_double_size;
fmt.state = p->state;
fmt.str++;
return fmt;
}
WARN_ONCE(1, "Please remove unsupported %%%c in format string\n", *fmt.str);
fmt.state = FORMAT_STATE_INVALID;
return fmt;
}
static void
set_field_width(struct printf_spec *spec, int width)
{
spec->field_width = width;
if (WARN_ONCE(spec->field_width != width, "field width %d too large", width)) {
spec->field_width = clamp(width, -FIELD_WIDTH_MAX, FIELD_WIDTH_MAX);
}
}
static void
set_precision(struct printf_spec *spec, int prec)
{
spec->precision = prec;
if (WARN_ONCE(spec->precision != prec, "precision %d too large", prec)) {
spec->precision = clamp(prec, 0, PRECISION_MAX);
}
}
/*
* Turn a 1/2/4-byte value into a 64-bit one for printing: truncate
* as necessary and deal with signedness.
*
* 'size' is the size of the value in bytes.
*/
static unsigned long long convert_num_spec(unsigned int val, int size, struct printf_spec spec)
{
unsigned int shift = 32 - size*8;
val <<= shift;
if (!(spec.flags & SIGN))
return val >> shift;
return (int)val >> shift;
}
/**
* vsnprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt_str: The format string to use
* @args: Arguments for the format string
*
* This function generally follows C99 vsnprintf, but has some
* extensions and a few limitations:
*
* - ``%n`` is unsupported
* - ``%p*`` is handled by pointer()
*
* See pointer() or Documentation/core-api/printk-formats.rst for more
* extensive description.
*
* **Please update the documentation in both places when making changes**
*
* The return value is the number of characters which would
* be generated for the given input, excluding the trailing
* '\0', as per ISO C99. If you want to have the exact
* number of characters written into @buf as return value
* (not including the trailing '\0'), use vscnprintf(). If the
* return is greater than or equal to @size, the resulting
* string is truncated.
*
* If you're not already dealing with a va_list consider using snprintf().
*/
int vsnprintf(char *buf, size_t size, const char *fmt_str, va_list args)
{
char *str, *end;
struct printf_spec spec = {0};
struct fmt fmt = {
.str = fmt_str,
.state = FORMAT_STATE_NONE,
};
/* Reject out-of-range values early. Large positive sizes are
used for unknown buffer sizes. */
if (WARN_ON_ONCE(size > INT_MAX))
return 0;
str = buf;
end = buf + size;
/* Make sure end is always >= buf */
if (end < buf) {
end = ((void *)-1);
size = end - buf;
}
while (*fmt.str) {
const char *old_fmt = fmt.str;
fmt = format_decode(fmt, &spec);
switch (fmt.state) {
case FORMAT_STATE_NONE: {
int read = fmt.str - old_fmt;
if (str < end) {
int copy = read;
if (copy > end - str)
copy = end - str;
memcpy(str, old_fmt, copy);
}
str += read;
continue;
}
case FORMAT_STATE_NUM: {
unsigned long long num;
if (fmt.size > sizeof(int))
num = va_arg(args, long long);
else
num = convert_num_spec(va_arg(args, int), fmt.size, spec);
str = number(str, end, num, spec);
continue;
}
case FORMAT_STATE_WIDTH:
set_field_width(&spec, va_arg(args, int));
continue;
case FORMAT_STATE_PRECISION:
set_precision(&spec, va_arg(args, int));
continue;
case FORMAT_STATE_CHAR: {
char c;
if (!(spec.flags & LEFT)) {
while (--spec.field_width > 0) {
if (str < end)
*str = ' ';
++str;
}
}
c = (unsigned char) va_arg(args, int);
if (str < end)
*str = c;
++str;
while (--spec.field_width > 0) {
if (str < end)
*str = ' ';
++str;
}
continue;
}
case FORMAT_STATE_STR:
str = string(str, end, va_arg(args, char *), spec);
continue;
case FORMAT_STATE_PTR:
str = pointer(fmt.str, str, end, va_arg(args, void *),
spec);
while (isalnum(*fmt.str))
fmt.str++;
continue;
case FORMAT_STATE_PERCENT_CHAR:
if (str < end)
*str = '%';
++str;
continue;
default:
/*
* Presumably the arguments passed gcc's type
* checking, but there is no safe or sane way
* for us to continue parsing the format and
* fetching from the va_list; the remaining
* specifiers and arguments would be out of
* sync.
*/
goto out;
}
}
out:
if (size > 0) {
if (str < end)
*str = '\0';
else
end[-1] = '\0';
}
/* the trailing null byte doesn't count towards the total */
return str-buf;
}
EXPORT_SYMBOL(vsnprintf);
/**
* vscnprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt: The format string to use
* @args: Arguments for the format string
*
* The return value is the number of characters which have been written into
* the @buf not including the trailing '\0'. If @size is == 0 the function
* returns 0.
*
* If you're not already dealing with a va_list consider using scnprintf().
*
* See the vsnprintf() documentation for format string extensions over C99.
*/
int vscnprintf(char *buf, size_t size, const char *fmt, va_list args)
{
int i;
if (unlikely(!size))
return 0;
i = vsnprintf(buf, size, fmt, args);
if (likely(i < size))
return i;
return size - 1;
}
EXPORT_SYMBOL(vscnprintf);
/**
* snprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt: The format string to use
* @...: Arguments for the format string
*
* The return value is the number of characters which would be
* generated for the given input, excluding the trailing null,
* as per ISO C99. If the return is greater than or equal to
* @size, the resulting string is truncated.
*
* See the vsnprintf() documentation for format string extensions over C99.
*/
int snprintf(char *buf, size_t size, const char *fmt, ...)
{
va_list args;
int i;
va_start(args, fmt);
i = vsnprintf(buf, size, fmt, args);
va_end(args);
return i;
}
EXPORT_SYMBOL(snprintf);
/**
* scnprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt: The format string to use
* @...: Arguments for the format string
*
* The return value is the number of characters written into @buf not including
* the trailing '\0'. If @size is == 0 the function returns 0.
*/
int scnprintf(char *buf, size_t size, const char *fmt, ...)
{
va_list args;
int i;
va_start(args, fmt);
i = vscnprintf(buf, size, fmt, args);
va_end(args);
return i;
}
EXPORT_SYMBOL(scnprintf);
/**
* vsprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @fmt: The format string to use
* @args: Arguments for the format string
*
* The return value is the number of characters written into @buf not including
* the trailing '\0'. Use vsnprintf() or vscnprintf() in order to avoid
* buffer overflows.
*
* If you're not already dealing with a va_list consider using sprintf().
*
* See the vsnprintf() documentation for format string extensions over C99.
*/
int vsprintf(char *buf, const char *fmt, va_list args)
{
return vsnprintf(buf, INT_MAX, fmt, args);
}
EXPORT_SYMBOL(vsprintf);
/**
* sprintf - Format a string and place it in a buffer
* @buf: The buffer to place the result into
* @fmt: The format string to use
* @...: Arguments for the format string
*
* The return value is the number of characters written into @buf not including
* the trailing '\0'. Use snprintf() or scnprintf() in order to avoid
* buffer overflows.
*
* See the vsnprintf() documentation for format string extensions over C99.
*/
int sprintf(char *buf, const char *fmt, ...)
{
va_list args;
int i;
va_start(args, fmt);
i = vsnprintf(buf, INT_MAX, fmt, args);
va_end(args);
return i;
}
EXPORT_SYMBOL(sprintf);
#ifdef CONFIG_BINARY_PRINTF
/*
* bprintf service:
* vbin_printf() - VA arguments to binary data
* bstr_printf() - Binary data to text string
*/
/**
* vbin_printf - Parse a format string and place args' binary value in a buffer
* @bin_buf: The buffer to place args' binary value
* @size: The size of the buffer(by words(32bits), not characters)
* @fmt_str: The format string to use
* @args: Arguments for the format string
*
* The format follows C99 vsnprintf, except %n is ignored, and its argument
* is skipped.
*
* The return value is the number of words(32bits) which would be generated for
* the given input.
*
* NOTE:
* If the return value is greater than @size, the resulting bin_buf is NOT
* valid for bstr_printf().
*/
int vbin_printf(u32 *bin_buf, size_t size, const char *fmt_str, va_list args)
{
struct fmt fmt = {
.str = fmt_str,
.state = FORMAT_STATE_NONE,
};
struct printf_spec spec = {0};
char *str, *end;
int width;
str = (char *)bin_buf;
end = (char *)(bin_buf + size);
#define save_arg(type) \
({ \
unsigned long long value; \
if (sizeof(type) == 8) { \
unsigned long long val8; \
str = PTR_ALIGN(str, sizeof(u32)); \
val8 = va_arg(args, unsigned long long); \
if (str + sizeof(type) <= end) { \
*(u32 *)str = *(u32 *)&val8; \
*(u32 *)(str + 4) = *((u32 *)&val8 + 1); \
} \
value = val8; \
} else { \
unsigned int val4; \
str = PTR_ALIGN(str, sizeof(type)); \
val4 = va_arg(args, int); \
if (str + sizeof(type) <= end) \
*(typeof(type) *)str = (type)(long)val4; \
value = (unsigned long long)val4; \
} \
str += sizeof(type); \
value; \
})
while (*fmt.str) {
fmt = format_decode(fmt, &spec);
switch (fmt.state) {
case FORMAT_STATE_NONE:
case FORMAT_STATE_PERCENT_CHAR:
break;
case FORMAT_STATE_INVALID:
goto out;
case FORMAT_STATE_WIDTH:
case FORMAT_STATE_PRECISION:
width = (int)save_arg(int);
/* Pointers may require the width */
if (*fmt.str == 'p')
set_field_width(&spec, width);
break;
case FORMAT_STATE_CHAR:
save_arg(char);
break;
case FORMAT_STATE_STR: {
const char *save_str = va_arg(args, char *);
const char *err_msg;
size_t len;
err_msg = check_pointer_msg(save_str);
if (err_msg)
save_str = err_msg;
len = strlen(save_str) + 1;
if (str + len < end)
memcpy(str, save_str, len);
str += len;
break;
}
case FORMAT_STATE_PTR:
/* Dereferenced pointers must be done now */
switch (*fmt.str) {
/* Dereference of functions is still OK */
case 'S':
case 's':
case 'x':
case 'K':
case 'e':
save_arg(void *);
break;
default:
if (!isalnum(*fmt.str)) {
save_arg(void *);
break;
}
str = pointer(fmt.str, str, end, va_arg(args, void *),
spec);
if (str + 1 < end)
*str++ = '\0';
else
end[-1] = '\0'; /* Must be nul terminated */
}
/* skip all alphanumeric pointer suffixes */
while (isalnum(*fmt.str))
fmt.str++;
break;
case FORMAT_STATE_NUM:
if (fmt.size > sizeof(int)) {
save_arg(long long);
} else {
save_arg(int);
}
}
}
out:
return (u32 *)(PTR_ALIGN(str, sizeof(u32))) - bin_buf;
#undef save_arg
}
EXPORT_SYMBOL_GPL(vbin_printf);
/**
* bstr_printf - Format a string from binary arguments and place it in a buffer
* @buf: The buffer to place the result into
* @size: The size of the buffer, including the trailing null space
* @fmt_str: The format string to use
* @bin_buf: Binary arguments for the format string
*
* This function like C99 vsnprintf, but the difference is that vsnprintf gets
* arguments from stack, and bstr_printf gets arguments from @bin_buf which is
* a binary buffer that generated by vbin_printf.
*
* The format follows C99 vsnprintf, but has some extensions:
* see vsnprintf comment for details.
*
* The return value is the number of characters which would
* be generated for the given input, excluding the trailing
* '\0', as per ISO C99. If you want to have the exact
* number of characters written into @buf as return value
* (not including the trailing '\0'), use vscnprintf(). If the
* return is greater than or equal to @size, the resulting
* string is truncated.
*/
int bstr_printf(char *buf, size_t size, const char *fmt_str, const u32 *bin_buf)
{
struct fmt fmt = {
.str = fmt_str,
.state = FORMAT_STATE_NONE,
};
struct printf_spec spec = {0};
char *str, *end;
const char *args = (const char *)bin_buf;
if (WARN_ON_ONCE(size > INT_MAX))
return 0;
str = buf;
end = buf + size;
#define get_arg(type) \
({ \
typeof(type) value; \
if (sizeof(type) == 8) { \
args = PTR_ALIGN(args, sizeof(u32)); \
*(u32 *)&value = *(u32 *)args; \
*((u32 *)&value + 1) = *(u32 *)(args + 4); \
} else { \
args = PTR_ALIGN(args, sizeof(type)); \
value = *(typeof(type) *)args; \
} \
args += sizeof(type); \
value; \
})
/* Make sure end is always >= buf */
if (end < buf) {
end = ((void *)-1);
size = end - buf;
}
while (*fmt.str) {
const char *old_fmt = fmt.str;
unsigned long long num;
fmt = format_decode(fmt, &spec);
switch (fmt.state) {
case FORMAT_STATE_NONE: {
int read = fmt.str - old_fmt;
if (str < end) {
int copy = read;
if (copy > end - str)
copy = end - str;
memcpy(str, old_fmt, copy);
}
str += read;
continue;
}
case FORMAT_STATE_WIDTH:
set_field_width(&spec, get_arg(int));
continue;
case FORMAT_STATE_PRECISION:
set_precision(&spec, get_arg(int));
continue;
case FORMAT_STATE_CHAR: {
char c;
if (!(spec.flags & LEFT)) {
while (--spec.field_width > 0) {
if (str < end)
*str = ' ';
++str;
}
}
c = (unsigned char) get_arg(char);
if (str < end)
*str = c;
++str;
while (--spec.field_width > 0) {
if (str < end)
*str = ' ';
++str;
}
continue;
}
case FORMAT_STATE_STR: {
const char *str_arg = args;
args += strlen(str_arg) + 1;
str = string(str, end, (char *)str_arg, spec);
continue;
}
case FORMAT_STATE_PTR: {
bool process = false;
int copy, len;
/* Non function dereferences were already done */
switch (*fmt.str) {
case 'S':
case 's':
case 'x':
case 'K':
case 'e':
process = true;
break;
default:
if (!isalnum(*fmt.str)) {
process = true;
break;
}
/* Pointer dereference was already processed */
if (str < end) {
len = copy = strlen(args);
if (copy > end - str)
copy = end - str;
memcpy(str, args, copy);
str += len;
args += len + 1;
}
}
if (process)
str = pointer(fmt.str, str, end, get_arg(void *), spec);
while (isalnum(*fmt.str))
fmt.str++;
continue;
}
case FORMAT_STATE_PERCENT_CHAR:
if (str < end)
*str = '%';
++str;
continue;
case FORMAT_STATE_INVALID:
goto out;
case FORMAT_STATE_NUM:
if (fmt.size > sizeof(int))
num = get_arg(long long);
else
num = convert_num_spec(get_arg(int), fmt.size, spec);
str = number(str, end, num, spec);
continue;
}
} /* while(*fmt.str) */
out:
if (size > 0) {
if (str < end)
*str = '\0';
else
end[-1] = '\0';
}
#undef get_arg
/* the trailing null byte doesn't count towards the total */
return str - buf;
}
EXPORT_SYMBOL_GPL(bstr_printf);
#endif /* CONFIG_BINARY_PRINTF */
/**
* vsscanf - Unformat a buffer into a list of arguments
* @buf: input buffer
* @fmt: format of buffer
* @args: arguments
*/
int vsscanf(const char *buf, const char *fmt, va_list args)
{
const char *str = buf;
char *next;
char digit;
int num = 0;
u8 qualifier;
unsigned int base;
union {
long long s;
unsigned long long u;
} val;
s16 field_width;
bool is_sign;
while (*fmt) {
/* skip any white space in format */
/* white space in format matches any amount of
* white space, including none, in the input.
*/
if (isspace(*fmt)) {
fmt = skip_spaces(++fmt);
str = skip_spaces(str);
}
/* anything that is not a conversion must match exactly */
if (*fmt != '%' && *fmt) {
if (*fmt++ != *str++)
break;
continue;
}
if (!*fmt)
break;
++fmt;
/* skip this conversion.
* advance both strings to next white space
*/
if (*fmt == '*') {
if (!*str)
break;
while (!isspace(*fmt) && *fmt != '%' && *fmt) {
/* '%*[' not yet supported, invalid format */
if (*fmt == '[')
return num;
fmt++;
}
while (!isspace(*str) && *str)
str++;
continue;
}
/* get field width */
field_width = -1;
if (isdigit(*fmt)) {
field_width = skip_atoi(&fmt);
if (field_width <= 0)
break;
}
/* get conversion qualifier */
qualifier = -1;
if (*fmt == 'h' || _tolower(*fmt) == 'l' ||
*fmt == 'z') {
qualifier = *fmt++;
if (unlikely(qualifier == *fmt)) {
if (qualifier == 'h') {
qualifier = 'H';
fmt++;
} else if (qualifier == 'l') {
qualifier = 'L';
fmt++;
}
}
}
if (!*fmt)
break;
if (*fmt == 'n') {
/* return number of characters read so far */
*va_arg(args, int *) = str - buf;
++fmt;
continue;
}
if (!*str)
break;
base = 10;
is_sign = false;
switch (*fmt++) {
case 'c':
{
char *s = (char *)va_arg(args, char*);
if (field_width == -1)
field_width = 1;
do {
*s++ = *str++;
} while (--field_width > 0 && *str);
num++;
}
continue;
case 's':
{
char *s = (char *)va_arg(args, char *);
if (field_width == -1)
field_width = SHRT_MAX;
/* first, skip leading white space in buffer */
str = skip_spaces(str);
/* now copy until next white space */
while (*str && !isspace(*str) && field_width--)
*s++ = *str++;
*s = '\0';
num++;
}
continue;
/*
* Warning: This implementation of the '[' conversion specifier
* deviates from its glibc counterpart in the following ways:
* (1) It does NOT support ranges i.e. '-' is NOT a special
* character
* (2) It cannot match the closing bracket ']' itself
* (3) A field width is required
* (4) '%*[' (discard matching input) is currently not supported
*
* Example usage:
* ret = sscanf("00:0a:95","%2[^:]:%2[^:]:%2[^:]",
* buf1, buf2, buf3);
* if (ret < 3)
* // etc..
*/
case '[':
{
char *s = (char *)va_arg(args, char *);
DECLARE_BITMAP(set, 256) = {0};
unsigned int len = 0;
bool negate = (*fmt == '^');
/* field width is required */
if (field_width == -1)
return num;
if (negate)
++fmt;
for ( ; *fmt && *fmt != ']'; ++fmt, ++len)
__set_bit((u8)*fmt, set);
/* no ']' or no character set found */
if (!*fmt || !len)
return num;
++fmt;
if (negate) {
bitmap_complement(set, set, 256);
/* exclude null '\0' byte */
__clear_bit(0, set);
}
/* match must be non-empty */
if (!test_bit((u8)*str, set))
return num;
while (test_bit((u8)*str, set) && field_width--)
*s++ = *str++;
*s = '\0';
++num;
}
continue;
case 'o':
base = 8;
break;
case 'x':
case 'X':
base = 16;
break;
case 'i':
base = 0;
fallthrough;
case 'd':
is_sign = true;
fallthrough;
case 'u':
break;
case '%':
/* looking for '%' in str */
if (*str++ != '%')
return num;
continue;
default:
/* invalid format; stop here */
return num;
}
/* have some sort of integer conversion.
* first, skip white space in buffer.
*/
str = skip_spaces(str);
digit = *str;
if (is_sign && digit == '-') {
if (field_width == 1)
break;
digit = *(str + 1);
}
if (!digit
|| (base == 16 && !isxdigit(digit))
|| (base == 10 && !isdigit(digit))
|| (base == 8 && !isodigit(digit))
|| (base == 0 && !isdigit(digit)))
break;
if (is_sign)
val.s = simple_strntoll(str, &next, base,
field_width >= 0 ? field_width : INT_MAX);
else
val.u = simple_strntoull(str, &next, base,
field_width >= 0 ? field_width : INT_MAX);
switch (qualifier) {
case 'H': /* that's 'hh' in format */
if (is_sign)
*va_arg(args, signed char *) = val.s;
else
*va_arg(args, unsigned char *) = val.u;
break;
case 'h':
if (is_sign)
*va_arg(args, short *) = val.s;
else
*va_arg(args, unsigned short *) = val.u;
break;
case 'l':
if (is_sign)
*va_arg(args, long *) = val.s;
else
*va_arg(args, unsigned long *) = val.u;
break;
case 'L':
if (is_sign)
*va_arg(args, long long *) = val.s;
else
*va_arg(args, unsigned long long *) = val.u;
break;
case 'z':
*va_arg(args, size_t *) = val.u;
break;
default:
if (is_sign)
*va_arg(args, int *) = val.s;
else
*va_arg(args, unsigned int *) = val.u;
break;
}
num++;
if (!next)
break;
str = next;
}
return num;
}
EXPORT_SYMBOL(vsscanf);
/**
* sscanf - Unformat a buffer into a list of arguments
* @buf: input buffer
* @fmt: formatting of buffer
* @...: resulting arguments
*/
int sscanf(const char *buf, const char *fmt, ...)
{
va_list args;
int i;
va_start(args, fmt);
i = vsscanf(buf, fmt, args);
va_end(args);
return i;
}
EXPORT_SYMBOL(sscanf);
]
|
| 683/6 |
2026/05/18 20:27 |
llm |
codeexpert |
0m
Model:
Tokens:
input: 59057
output: 690
thoughts: 899
Thoughts: **Code Review and Pointer Formatting Logic**
Okay, I've just finished going over the `vbin_printf` and `bstr_printf` code, paying close attention to how they handle pointer formatting, specifically with these alphanumeric extensions. The core logic in the `FORMAT_STATE_PTR` case is, fortunately, consistent between the two functions.
Looking at `vbin_printf`, I see the `default` case handles the `%pI4`, `%pI6`, and `%pB` formats, essentially any alphanumeric format besides the explicitly listed exceptions like `%pS`, `%ps`, etc. It calls the `pointer` function to do the formatting into a string. `bstr_printf` does something similar but copies rather than formats them.
The key observation is that in both functions, formats like `%pI4` and `%pI6` will fall into the `default` case. With `vbin_printf`, they get *formatted* into a string. With `bstr_printf`, that means the *caller must have pre-formatted* the string. The latter then just copies them.
So, yes, it's clear: `bstr_printf` expects a pre-formatted string for `%pI4`, `%pI6`, and by extension, `%pB`. This is exactly the same as how the exception pointer formats are handled.
Just to be absolutely certain, I re-ran the logic through my head for other edge cases. What about `%pI` on its own? Well, `isalnum('I')` is true so that would be pre-formatted too. `%pi4`? Still pre-formatted. All the alphanumeric variants, it seems, are handled in the same manner. This also makes sense and aligns with the expected behavior of these formatted strings.
The conclusion is definite. I have a clear understanding of the logic here.
|
| 684/4 |
2026/05/18 20:27 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 42347
output: 44
thoughts: 7471
Thoughts: **Comprehensive Analysis of `bpf_bprintf_prepare` Vulnerabilities and Robust Fixes**
My thought process began by examining the implications of truncating the IP address string, initially considering it as a bug. I quickly realized this truncation could lead to a buffer overflow in `bin_args` if the formatted IP address string exceeded the allocated space. The original check, `if (tmp_buf + sizeof_cur_ip > tmp_buf_end)`, was fundamentally flawed because it only accounted for the binary IP address size (4 or 16 bytes) and not the *formatted* string size (which could be up to 40 bytes for %pI6). This led to a 24-byte overflow. Truncation, while preventing the overflow, presented a worse BPF program behavior as an outcome.
The solution was to perform a proper check for sufficient space *before* formatting and returning `-ENOSPC` when there is not enough room, aligning the behavior to the intended result of the BPF. Using `snprintf` and checking for truncation by comparing the returned value to the available space allowed an early and safe determination of whether the buffer was full, which is exactly the correct error to return.
I analyzed `%pB` and then `%pI4` and `%pI6` to confirm that the approach worked as well for those as well. I considered that `snprintf` could fail, and it should return `-EINVAL` if that happened. The original check for `%pI4` and `%pI6` using `sizeof_cur_ip` was incorrect because it checked the space for the binary IP address, not the formatted IP address, which led to a buffer overflow. This was resolved by removing the incorrect check and instead checking the return value from `snprintf`. The dry run behavior of `bpf_bprintf_prepare` (when `tmp_buf` is `NULL`) to validate the format string remained unchanged, meaning a check of the space available was done only when the buffer was allocated.
I identified the crucial issue related to the alignment and bounds check, a signedness bug. The original code's bounds check using `tmp_buf_end - tmp_buf < sizeof_cur_arg` was vulnerable because the signed `ptrdiff_t` (`tmp_buf_end - tmp_buf`) could become negative if `tmp_buf` advanced past `tmp_buf_end`, and then get cast to `size_t`, which would be huge. This caused an out-of-bounds write. I noted the alignment with `PTR_ALIGN` would cause that to happen.
The fix, returning `-ENOSPC` when `snprintf` had truncated, would fix any potential for `tmp_buf` to ever advance past `tmp_buf_end`. I reviewed how `tmp_buf` got advanced: `%s` and `%c` only advanced by the correct size. I confirmed that the checks for the other standard arguments would work as intended. I verified that, with the corrected approach of always checking and returning `-ENOSPC` if a string can't fit, `tmp_buf` would never advance past `tmp_buf_end`. Then, I added pointer addition to the bounds checks: `if (tmp_buf + sizeof_cur_arg > tmp_buf_end)` to directly avoid the signedness issue and ensure all cases were fully safe, not just a fix for *one* particular case. This prevented all problems caused by `tmp_buf` advancing past `tmp_buf_end`, which was the cause of the lockdep crash.
I reviewed the patch that failed to apply this fix, and confirmed the bounds checks the failed patch proposed were correct, but the handling of `%pB` and `%pI4` in that patch led to truncation, which didn't fix the underlying problem of `bstr_printf` getting the wrong length and reading out of bounds. The only problem with that patch was it's handling of the `snprintf` return. The correct fix involved returning `-ENOSPC` directly. Additionally, I added checks for `if (tmp_buf >= tmp_buf_end)` before any `snprintf` calls to provide an extra defense in depth, guaranteeing a clean and robust solution. The final implementations of `%pB` and `%pI4` were thus guaranteed safe and prevented all potential out-of-bounds writes.
|
| 685/4 |
2026/05/18 20:28 |
tool |
codesearch-definition-source |
0m
Args: map[ContextFile:kernel/bpf/helpers.c IncludeLines:true Name:bpf_bprintf_prepare]
Results: map[SourceCode: 804: /*
805: * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
806: *
807: * Returns a negative value if fmt is an invalid format string or 0 otherwise.
808: *
809: * This can be used in two ways:
810: * - Format string verification only: when data->get_bin_args is false
811: * - Arguments preparation: in addition to the above verification, it writes in
812: * data->bin_args a binary representation of arguments usable by bstr_printf
813: * where pointers from BPF have been sanitized.
814: *
815: * In argument preparation mode, if 0 is returned, safe temporary buffers are
816: * allocated and bpf_bprintf_cleanup should be called to free them after use.
817: */
818: int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
819: u32 num_args, struct bpf_bprintf_data *data)
820: {
821: bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
822: char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
823: struct bpf_bprintf_buffers *buffers = NULL;
824: size_t sizeof_cur_arg, sizeof_cur_ip;
825: int err, i, num_spec = 0;
826: u64 cur_arg;
827: char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
828:
829: fmt_end = strnchr(fmt, fmt_size, 0);
830: if (!fmt_end)
831: return -EINVAL;
832: fmt_size = fmt_end - fmt;
833:
834: if (get_buffers && bpf_try_get_buffers(&buffers))
835: return -EBUSY;
836:
837: if (data->get_bin_args) {
838: if (num_args)
839: tmp_buf = buffers->bin_args;
840: tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
841: data->bin_args = (u32 *)tmp_buf;
842: }
843:
844: if (data->get_buf)
845: data->buf = buffers->buf;
846:
847: for (i = 0; i < fmt_size; i++) {
848: unsigned char c = fmt[i];
849:
850: /*
851: * Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
852: * through unchanged, while still rejecting ASCII control bytes.
853: */
854: if (isascii(c) && !isprint(c) && !isspace(c)) {
855: err = -EINVAL;
856: goto out;
857: }
858:
859: if (fmt[i] != '%')
860: continue;
861:
862: if (fmt[i + 1] == '%') {
863: i++;
864: continue;
865: }
866:
867: if (num_spec >= num_args) {
868: err = -EINVAL;
869: goto out;
870: }
871:
872: /* The string is zero-terminated so if fmt[i] != 0, we can
873: * always access fmt[i + 1], in the worst case it will be a 0
874: */
875: i++;
876: c = fmt[i];
877: /*
878: * The format parser below only understands ASCII conversion
879: * specifiers and modifiers, so reject non-ASCII after '%'.
880: */
881: if (!isascii(c)) {
882: err = -EINVAL;
883: goto out;
884: }
885:
886: /* skip optional "[0 +-][num]" width formatting field */
887: while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
888: fmt[i] == ' ')
889: i++;
890: if (fmt[i] >= '1' && fmt[i] <= '9') {
891: i++;
892: while (fmt[i] >= '0' && fmt[i] <= '9')
893: i++;
894: }
895:
896: if (fmt[i] == 'p') {
897: sizeof_cur_arg = sizeof(long);
898:
899: if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
900: ispunct(fmt[i + 1])) {
901: if (tmp_buf)
902: cur_arg = raw_args[num_spec];
903: goto nocopy_fmt;
904: }
905:
906: if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
907: fmt[i + 2] == 's') {
908: fmt_ptype = fmt[i + 1];
909: i += 2;
910: goto fmt_str;
911: }
912:
913: if (fmt[i + 1] == 'K' ||
914: fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
915: fmt[i + 1] == 'S') {
916: if (tmp_buf)
917: cur_arg = raw_args[num_spec];
918: i++;
919: goto nocopy_fmt;
920: }
921:
922: if (fmt[i + 1] == 'B') {
923: if (tmp_buf) {
924: err = snprintf(tmp_buf,
925: (tmp_buf_end - tmp_buf),
926: "%pB",
927: (void *)(long)raw_args[num_spec]);
928: tmp_buf += (err + 1);
929: }
930:
931: i++;
932: num_spec++;
933: continue;
934: }
935:
936: /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
937: if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
938: (fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
939: err = -EINVAL;
940: goto out;
941: }
942:
943: i += 2;
944: if (!tmp_buf)
945: goto nocopy_fmt;
946:
947: sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
948: if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
949: err = -ENOSPC;
950: goto out;
951: }
952:
953: unsafe_ptr = (char *)(long)raw_args[num_spec];
954: err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
955: sizeof_cur_ip);
956: if (err < 0)
957: memset(cur_ip, 0, sizeof_cur_ip);
958:
959: /* hack: bstr_printf expects IP addresses to be
960: * pre-formatted as strings, ironically, the easiest way
961: * to do that is to call snprintf.
962: */
963: ip_spec[2] = fmt[i - 1];
964: ip_spec[3] = fmt[i];
965: err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
966: ip_spec, &cur_ip);
967:
968: tmp_buf += err + 1;
969: num_spec++;
970:
971: continue;
972: } else if (fmt[i] == 's') {
973: fmt_ptype = fmt[i];
974: fmt_str:
975: if (fmt[i + 1] != 0 &&
976: !isspace(fmt[i + 1]) &&
977: !ispunct(fmt[i + 1])) {
978: err = -EINVAL;
979: goto out;
980: }
981:
982: if (!tmp_buf)
983: goto nocopy_fmt;
984:
985: if (tmp_buf_end == tmp_buf) {
986: err = -ENOSPC;
987: goto out;
988: }
989:
990: unsafe_ptr = (char *)(long)raw_args[num_spec];
991: err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
992: fmt_ptype,
993: tmp_buf_end - tmp_buf);
994: if (err < 0) {
995: tmp_buf[0] = '\0';
996: err = 1;
997: }
998:
999: tmp_buf += err;
1000: num_spec++;
1001:
1002: continue;
1003: } else if (fmt[i] == 'c') {
1004: if (!tmp_buf)
1005: goto nocopy_fmt;
1006:
1007: if (tmp_buf_end == tmp_buf) {
1008: err = -ENOSPC;
1009: goto out;
1010: }
1011:
1012: *tmp_buf = raw_args[num_spec];
1013: tmp_buf++;
1014: num_spec++;
1015:
1016: continue;
1017: }
1018:
1019: sizeof_cur_arg = sizeof(int);
1020:
1021: if (fmt[i] == 'l') {
1022: sizeof_cur_arg = sizeof(long);
1023: i++;
1024: }
1025: if (fmt[i] == 'l') {
1026: sizeof_cur_arg = sizeof(long long);
1027: i++;
1028: }
1029:
1030: if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
1031: fmt[i] != 'x' && fmt[i] != 'X') {
1032: err = -EINVAL;
1033: goto out;
1034: }
1035:
1036: if (tmp_buf)
1037: cur_arg = raw_args[num_spec];
1038: nocopy_fmt:
1039: if (tmp_buf) {
1040: tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
1041: if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
1042: err = -ENOSPC;
1043: goto out;
1044: }
1045:
1046: if (sizeof_cur_arg == 8) {
1047: *(u32 *)tmp_buf = *(u32 *)&cur_arg;
1048: *(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
1049: } else {
1050: *(u32 *)tmp_buf = (u32)(long)cur_arg;
1051: }
1052: tmp_buf += sizeof_cur_arg;
1053: }
1054: num_spec++;
1055: }
1056:
1057: err = 0;
1058: out:
1059: if (err)
1060: bpf_bprintf_cleanup(data);
1061: return err;
1062: }
SourceFile:kernel/bpf/helpers.c]
|
| 686/4 |
2026/05/18 20:28 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 54633
output: 337
thoughts: 22
Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:
**Applying the Fix**
Alright, the issue's been identified, and I'm going to leverage `codeeditor` – that's the right tool for this job. First things first, the core problem seems to be rooted in how we're handling `%pB`. I need to zero in on that specific section of the code and dissect the current implementation. I'll need to figure out precisely where the logic is failing, trace the execution flow, and understand *why* `%pB` is causing problems. From there, I can determine the most efficient and robust replacement strategy. This will involve understanding the intent of that placeholder and designing a fix that doesn't introduce any regressions or unintended side effects in the rest of the application. I expect a careful replacement will be needed.
|
| 687/4 |
2026/05/18 20:29 |
tool |
codeeditor |
0m
Args: map[CurrentCode: if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
} NewCode: if (fmt[i + 1] == 'B') {
if (tmp_buf) {
if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
if (err < 0) {
err = -EINVAL;
goto out;
}
if (err >= tmp_buf_end - tmp_buf) {
err = -ENOSPC;
goto out;
}
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
} SourceFile:kernel/bpf/helpers.c]
Results:
|
| 688/4 |
2026/05/18 20:29 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 55188
output: 855
thoughts: 16
Thoughts: Okay, here's the summary, tailored for an expert audience, in the first person, reflecting the given thought process:
**Refactoring the `%pI` Handling**
Alright, the next thing on my plate is to tackle the `%pI4` and `%pI6` handling. We’ve known this needed attention for a while. I need to make sure this is done correctly, thoroughly, and consistently. I'm thinking about the various places this formatting might be used across the codebase and how the changes will ripple through. It is not something to be left till the end. I need to replace the old handling. I'll need to consider performance implications here, especially if these format specifiers are frequently used. Any optimization I can implement now will pay dividends later. I'll also need to carefully examine any existing test cases, and probably add some more, to ensure that the replacements don't introduce any regressions or unexpected behavior.
|
| 689/4 |
2026/05/18 20:29 |
tool |
codeeditor |
0m
Args: map[CurrentCode: /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
tmp_buf += err + 1;
num_spec++;
continue; NewCode: /* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
if (err < 0) {
err = -EINVAL;
goto out;
}
if (err >= tmp_buf_end - tmp_buf) {
err = -ENOSPC;
goto out;
}
tmp_buf += err + 1;
num_spec++;
continue; SourceFile:kernel/bpf/helpers.c]
Results:
|
| 690/4 |
2026/05/18 20:29 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 56263
output: 167
thoughts: 8
Thoughts: Okay, here's the summary, assuming I'm a seasoned developer thinking through this task:
**Refactoring the `%s` Handling**
Alright, let's tackle this `%s` issue. It's time to dig into the old code and see how we're dealing with string formatting, specifically that `%s` placeholder. I'm already anticipating some of the common pitfalls here. Is it robust? Does it handle potential errors gracefully? Is it thread-safe? The legacy systems often used `sprintf` type functions, which can lead to vulnerabilities if we're not careful about input sanitization and buffer overflows.
My first thought is, is this the only place we have this type of formatting? We probably need to do a full search to see where it exists. If it is only in one place, that makes things much easier. If this is pervasive, then we are looking at a more comprehensive overhaul. Then, I need to evaluate what this `%s` is actually *doing*. Is it just for basic string insertion, or are there any more complex scenarios, such as dynamically constructing SQL queries or generating HTML? Those are two completely different beasts.
I'm thinking about the different options for replacement. Is this a custom implementation, or are we using something like `format!` in Rust, or `f-strings` in Python? Maybe even a template engine? The ideal solution will be both safe and efficient. I need to be sure that the string replacement method will prevent any format string vulnerabilities. And efficiency here will mean both memory utilization and CPU usage. I would prefer something fast so that this doesn't impact performance.
I will need to consider the level of effort required to make a change. If this is a very small change, it should be simple. If it's a large change, I will want to be sure to have a good suite of tests, so that any bugs are easily detected. I'm going to need to replace `%s` handling with a more modern and, crucially, a safer approach. I'll need to assess the scope, identify the potential risks, and then plan the best path forward.
|
| 691/4 |
2026/05/18 20:29 |
tool |
codeeditor |
0m
Args: map[CurrentCode: if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec]; NewCode: if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec]; SourceFile:kernel/bpf/helpers.c]
Results:
|
| 692/4 |
2026/05/18 20:29 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 56889
output: 189
thoughts: 8
Thoughts: Okay, here's the summary, assuming I'm the expert working on this:
**Refactoring the `%c` Handling**
Alright, time to tackle the `%c` format specifier. We've known for a while that the current implementation isn't ideal. It likely has some edge cases we're not handling correctly, potential performance bottlenecks, or perhaps it's just unnecessarily complex compared to more elegant solutions. The goal here is to make sure it functions robustly, efficiently, and cleanly integrates with the rest of the format string parsing engine. We need to be meticulous to avoid regressions; this is a place where small oversights cause big problems. Let’s get to it.
|
| 693/4 |
2026/05/18 20:29 |
tool |
codeeditor |
0m
Args: map[CurrentCode: } else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec]; NewCode: } else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec]; SourceFile:kernel/bpf/helpers.c]
Results:
|
| 694/4 |
2026/05/18 20:29 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 57238
output: 209
thoughts: 7
Thoughts: Okay, here's the summary, assuming I'm a seasoned developer thinking about argument handling:
**Refactoring Our Argument Handling**
Alright, time to tackle this again. That current approach to argument handling is becoming unwieldy. We need a more robust and cleaner solution. It’s got to be more flexible, easier to maintain, and ideally, more intuitive to use. I'm thinking about replacing the standard, run-of-the-mill argument parsing we've got in place. There's got to be a better way than what we are doing, and it's time to find it. This rewrite will be a game changer.
|
| 695/4 |
2026/05/18 20:29 |
tool |
codeeditor |
0m
Args: map[CurrentCode: nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) { NewCode: nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) { SourceFile:kernel/bpf/helpers.c]
Results:
|
| 696/4 |
2026/05/18 20:29 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 57598
output: 23
thoughts: 7
Thoughts: Okay, I'm ready to delve into this diff. Let's get to it.
**Analyzing the Changes**
Alright, let's dive in. First, I need to understand the context of this change. What's the overall objective here? What are we trying to accomplish with this modification? I need to review this with a critical eye, considering potential implications for performance, security, and maintainability.
I'm particularly interested in any code modifications, but I should also keep an eye out for changes to any configuration files or external dependencies. I'll need to assess if the solution is aligned with the best practices and coding standards. I'll pay close attention to any edge cases that the implementation might have missed, or any opportunities for improvements. This needs to be thoroughly vetted.
|
| 697/4 |
2026/05/18 20:29 |
tool |
patch-diff |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Output:diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2bb60200c..ff4f42a7e 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -804,259 +804,280 @@ void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
+ err = -ENOSPC;
+ goto out;
+ }
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
- sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
- if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
+ sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
+
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += err + 1;
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
- } else if (fmt[i] == 'c') {
- if (!tmp_buf)
- goto nocopy_fmt;
+ } else if (fmt[i] == 'c') {
+ if (!tmp_buf)
+ goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
- err = -ENOSPC;
- goto out;
- }
+ if (tmp_buf >= tmp_buf_end) {
+ err = -ENOSPC;
+ goto out;
+ }
- *tmp_buf = raw_args[num_spec];
+ *tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
-nocopy_fmt:
+ nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
- if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
]
|
| 698/4 |
2026/05/18 20:29 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 60977
output: 227
thoughts: 26
Thoughts: Here's a summary of my immediate thought process:
**Fixing a Formatting Blunder**
Okay, hold on a second. I think I made a small but potentially significant formatting error here. Looking back, the indentation for `%c` and `nocopy_fmt` isn't quite right. That could cause downstream issues, or at least make things less readable, and in this context, readability *is* a major point. I need to correct that indentation right away to maintain the integrity of this section. Let's get that sorted out immediately.
|
| 699/4 |
2026/05/18 20:29 |
tool |
codeeditor |
0m
Args: map[CurrentCode: } else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
} NewCode: } else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
} SourceFile:kernel/bpf/helpers.c]
Results:
|
| 700/4 |
2026/05/18 20:29 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 61353
output: 190
thoughts: 7
Thoughts: Okay, here's my attempt at summarizing the thought process, tailored for an expert audience, and written in the first person:
**Reflecting on `nocopy_fmt` Implementation**
Right, so I've been mulling over this `nocopy_fmt` concept, and the more I consider it, the more I appreciate the core idea. We're talking about a significant performance optimization, particularly in scenarios involving string formatting or manipulation, where we can potentially bypass costly memory allocations and copies. My initial intuition was correct: the gains will be most noticeable in high-throughput, latency-sensitive applications where frequent string creation is a bottleneck.
The beauty, as I see it, lies in its potential to integrate seamlessly with existing formatting libraries or systems. I'm wondering about the specific design choices. Are we leaning towards a compile-time solution using techniques like string interning or constant folding, or are we aiming for a more runtime-oriented, dynamic approach? The trade-offs are pretty clear: compile-time offers maximum performance but limits flexibility, while runtime might give greater adaptability at the cost of some overhead.
From my perspective, the challenge is minimizing the potential for misuse. It's too easy to create an issue by making assumptions about string ownership or data lifetime that are violated, especially within a multithreaded environment. If the implementation doesn't provide the level of safety the experts demand, then this is an incomplete solution. I'm thinking strong compile-time checks and perhaps even introducing a new data type might be required to ensure that such instances are handled.
I need to dig deeper into the specifics of the proposed implementation; the devil, as always, is in the details. Looking at the interface, I'm already imagining the possibilities for supporting custom formatters and integrating this into existing templating engines. But the first thing will be to look for potential corner cases and threading issues, to guarantee that we're adding a tool that is really helpful, rather than one that will just add to our burden. It's a promising concept, absolutely, but a rigorous, detailed review is paramount.
|
| 701/4 |
2026/05/18 20:29 |
tool |
codeeditor |
0m
Args: map[CurrentCode: if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf + sizeof_cur_arg > tmp_buf_end) { NewCode: if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf + sizeof_cur_arg > tmp_buf_end) { SourceFile:kernel/bpf/helpers.c]
Results:
|
| 702/4 |
2026/05/18 20:29 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 61994
output: 23
thoughts: 8
Thoughts: Okay, I'm ready to dive into that diff. Let's get this done.
|
| 703/4 |
2026/05/18 20:29 |
tool |
patch-diff |
0m
Args: map[File:kernel/bpf/helpers.c]
Results: map[Output:diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2bb60200c..0e3d3bddd 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -804,259 +804,280 @@ void bpf_bprintf_cleanup(struct bpf_bprintf_data *data)
/*
* bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
* - Format string verification only: when data->get_bin_args is false
* - Arguments preparation: in addition to the above verification, it writes in
* data->bin_args a binary representation of arguments usable by bstr_printf
* where pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
* allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
u32 num_args, struct bpf_bprintf_data *data)
{
bool get_buffers = (data->get_bin_args && num_args) || data->get_buf;
char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
struct bpf_bprintf_buffers *buffers = NULL;
size_t sizeof_cur_arg, sizeof_cur_ip;
int err, i, num_spec = 0;
u64 cur_arg;
char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
if (get_buffers && bpf_try_get_buffers(&buffers))
return -EBUSY;
if (data->get_bin_args) {
if (num_args)
tmp_buf = buffers->bin_args;
tmp_buf_end = tmp_buf + MAX_BPRINTF_BIN_ARGS;
data->bin_args = (u32 *)tmp_buf;
}
if (data->get_buf)
data->buf = buffers->buf;
for (i = 0; i < fmt_size; i++) {
unsigned char c = fmt[i];
/*
* Permit bytes >= 0x80 in plain text so UTF-8 literals can pass
* through unchanged, while still rejecting ASCII control bytes.
*/
if (isascii(c) && !isprint(c) && !isspace(c)) {
err = -EINVAL;
goto out;
}
if (fmt[i] != '%')
continue;
if (fmt[i + 1] == '%') {
i++;
continue;
}
if (num_spec >= num_args) {
err = -EINVAL;
goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
* always access fmt[i + 1], in the worst case it will be a 0
*/
i++;
c = fmt[i];
/*
* The format parser below only understands ASCII conversion
* specifiers and modifiers, so reject non-ASCII after '%'.
*/
if (!isascii(c)) {
err = -EINVAL;
goto out;
}
/* skip optional "[0 +-][num]" width formatting field */
while (fmt[i] == '0' || fmt[i] == '+' || fmt[i] == '-' ||
fmt[i] == ' ')
i++;
if (fmt[i] >= '1' && fmt[i] <= '9') {
i++;
while (fmt[i] >= '0' && fmt[i] <= '9')
i++;
}
if (fmt[i] == 'p') {
sizeof_cur_arg = sizeof(long);
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1])) {
if (tmp_buf)
cur_arg = raw_args[num_spec];
goto nocopy_fmt;
}
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
fmt_ptype = fmt[i + 1];
i += 2;
goto fmt_str;
}
if (fmt[i + 1] == 'K' ||
fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
fmt[i + 1] == 'S') {
if (tmp_buf)
cur_arg = raw_args[num_spec];
i++;
goto nocopy_fmt;
}
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
+ err = -ENOSPC;
+ goto out;
+ }
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
goto out;
}
i += 2;
if (!tmp_buf)
goto nocopy_fmt;
- sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
- if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
+ sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
+
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
if (err < 0)
memset(cur_ip, 0, sizeof_cur_ip);
/* hack: bstr_printf expects IP addresses to be
* pre-formatted as strings, ironically, the easiest way
* to do that is to call snprintf.
*/
ip_spec[2] = fmt[i - 1];
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += err + 1;
num_spec++;
continue;
} else if (fmt[i] == 's') {
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
goto out;
}
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
tmp_buf += err;
num_spec++;
continue;
} else if (fmt[i] == 'c') {
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
*tmp_buf = raw_args[num_spec];
tmp_buf++;
num_spec++;
continue;
}
sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
goto out;
}
if (tmp_buf)
cur_arg = raw_args[num_spec];
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
- if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
if (sizeof_cur_arg == 8) {
*(u32 *)tmp_buf = *(u32 *)&cur_arg;
*(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
} else {
*(u32 *)tmp_buf = (u32)(long)cur_arg;
}
tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
out:
if (err)
bpf_bprintf_cleanup(data);
return err;
}
]
|
| 704/4 |
2026/05/18 20:29 |
llm |
patch-generator |
0m
Model:
Tokens:
input: 65149
output: 864
thoughts: 469
Thoughts: **Finalizing the Explanation**
Alright, this diff is looking good. Time to solidify the reasoning behind these changes. My thinking is, the prior patch attempt, while well-intentioned, fell short. The approach of truncating the string with `min_t(size_t, err + 1, limit)` was supposed to solve the out-of-bounds write issue, but it missed a critical piece: the `-ENOSPC` error signaling when the `bin_args` buffer was full. This oversight allowed `bpf_bprintf_prepare()` to falsely return success, even when the string was truncated or empty. That meant `bstr_printf()` was then called with potentially corrupt data, leading to the dreaded out-of-bounds reads and those nasty soft lockups, because `strlen()` was running wild.
The correct fix, as I see it now, is to ensure `-ENOSPC` gets returned whenever `snprintf()` indicates truncation occurred (i.e., when `err >= tmp_buf_end - tmp_buf`). This cleanly signals the BPF program that its internal buffer is too small, and the formatting operation is aborted safely. This is the crux of the problem.
Beyond that core issue, I noticed some flaws in the `%pI4` and `%pI6` handling. The previous bounds check (`tmp_buf_end - tmp_buf < sizeof_cur_ip`) was off. It was checking against the *binary* IP size, but `snprintf()` actually writes the *formatted* string, which can be far larger (up to 40 bytes for IPv6). This allowed `snprintf()` to write past the buffer end, causing a similar overflow as `%pB`. Removing the incorrect check and instead relying on `snprintf()`'s return value fixes this.
Finally, there was a signedness bug creeping in the bounds checks. The existing subtraction (`tmp_buf_end - tmp_buf < sizeof_cur_arg`) wasn't entirely reliable. Rewriting these as pointer additions (`tmp_buf + sizeof_cur_arg > tmp_buf_end`) and hardening the equality checks for `%s` and `%c` to use `>=` solves this. This approach is more robust because it ensures that even if `tmp_buf` somehow overruns `tmp_buf_end`, the checks won't be bypassed due to signedness quirks with large pointer differences. With these updates, the code is solid and safe.
|
| 705/3 |
2026/05/18 20:30 |
action |
test-patch |
33m
Results: map[PatchDiff:diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2bb60200c..0e3d3bddd 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -921,10 +921,22 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
+ err = -ENOSPC;
+ goto out;
+ }
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += (err + 1);
}
@@ -944,12 +956,13 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
- if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
+ sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
+
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
@@ -964,6 +977,14 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += err + 1;
num_spec++;
@@ -982,7 +1003,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1004,7 +1025,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1038,7 +1059,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
- if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
TestError:]
|
| 706/1 |
2026/05/18 21:03 |
agent |
fixes-finder |
1m
Model:
Results: map[FixesHash:48cac3f4a96ddf08df8e53809ed066de0dc93915]
Instruction:
You are an experienced Linux kernel developer tasked with identifying the commit
that introduced the bug being fixed. Identifying the correct buggy commit is crucial
for proper kernel maintenance (backporting to stable trees, etc.).
Your investigation strategy:
1. Examine the patch that fixes the bug. Use git tools (like git-log or git-blame)
to trace the history of the lines or functions modified by the patch.
2. Analyze the stack trace in the crash report. Identify the key files and functions
involved in the crash and investigate their history to see when the problematic
logic was introduced.
3. Compare the bug explanation with the commit history to find the point where
the described logic error first appeared.
A bug is typically introduced when a piece of code is first written, or when
a refactoring changed its logic in a way that introduced the bug.
Trace the history of relevant symbols or find when specific code patterns were introduced/removed.
You must provide exactly one bug-introducing commit hash.
If you are unable to confidently determine the bug-introducing commit after investigation,
return an empty string rather than guessing.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt:
The crash is:
watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/u9:4:1179]
CPU#0 Utilization every 4000ms during lockup:
#1: 100% system, 1% softirq, 1% hardirq, 0% idle
#2: 100% system, 1% softirq, 0% hardirq, 0% idle
#3: 100% system, 1% softirq, 1% hardirq, 0% idle
#4: 100% system, 1% softirq, 1% hardirq, 0% idle
#5: 100% system, 1% softirq, 1% hardirq, 0% idle
Modules linked in:
irq event stamp: 386488
hardirqs last enabled at (386487): [<ffffffff8100148a>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
hardirqs last disabled at (386488): [<ffffffff8b9eb11e>] sysvec_apic_timer_interrupt+0xe/0xb0 arch/x86/kernel/apic/apic.c:1061
softirqs last enabled at (386486): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
softirqs last disabled at (386409): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
CPU: 0 UID: 0 PID: 1179 Comm: kworker/u9:4 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:342 [inline]
RIP: 0010:smp_call_function_many_cond+0xfcb/0x13b0 kernel/smp.c:892
Code: 79 45 8b 2e 44 89 ee 83 e6 01 31 ff e8 be c6 0b 00 41 83 e5 01 49 bd 00 00 00 00 00 fc ff df 75 07 e8 69 c2 0b 00 eb 37 f3 90 <43> 0f b6 04 2c 84 c0 75 10 41 f7 06 01 00 00 00 74 1e e8 4e c2 0b
RSP: 0018:ffffc90005fcf720 EFLAGS: 00000293
RAX: ffffffff81834d62 RBX: ffff88806943ad08 RCX: ffff88801d361c00
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90005fcf860 R08: ffffffff908e5af7 R09: 1ffffffff211cb5e
R10: dffffc0000000000 R11: fffffbfff211cb5f R12: 1ffff1100d2a7edd
R13: dffffc0000000000 R14: ffff88806953f6e8 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff8880d5abe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555578641490 CR3: 000000000eaf4000 CR4: 00000000000006f0
Call Trace:
<TASK>
on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1057
on_each_cpu include/linux/smp.h:72 [inline]
smp_text_poke_sync_each_cpu arch/x86/kernel/alternative.c:2773 [inline]
smp_text_poke_batch_finish+0x5e6/0x1090 arch/x86/kernel/alternative.c:2983
arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
static_key_enable_cpuslocked+0x128/0x240 kernel/jump_label.c:210
static_key_enable+0x1a/0x20 kernel/jump_label.c:223
toggle_allocation_gate+0xab/0x290 mm/kfence/core.c:906
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xad1/0x1740 kernel/workqueue.c:3397
worker_thread+0xa59/0xfc0 kernel/workqueue.c:3478
kthread+0x388/0x470 kernel/kthread.c:436
ret_from_fork+0x437/0x9d0 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:__remove_hrtimer+0x8e/0x3e0 kernel/time/hrtimer.c:1145
Code: ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d
RSP: 0018:ffffc900001b8d90 EFLAGS: 00000006
RAX: ffffffff817d161e RBX: ffff888069527f40 RCX: 0000000000010000
RDX: ffff8880160c8000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: dffffc0000000000 R08: ffffffff94f48153 R09: 1ffffffff29e902a
R10: dffffc0000000000 R11: fffffbfff29e902b R12: 0000000000000000
R13: 0000000000000001 R14: ffff888024042300 R15: ffff888069528000
FS: 0000000000000000(0000) GS:ffff8880d5bbe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f83db077470 CR3: 000000001f132000 CR4: 00000000000006f0
Call Trace:
<IRQ>
__run_hrtimer kernel/time/hrtimer.c:1910 [inline]
__hrtimer_run_queues+0x299/0xa10 kernel/time/hrtimer.c:1994
hrtimer_interrupt+0x44b/0x950 kernel/time/hrtimer.c:2113
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1050 [inline]
__sysvec_apic_timer_interrupt+0x102/0x420 arch/x86/kernel/apic/apic.c:1067
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1061 [inline]
sysvec_apic_timer_interrupt+0x8b/0xb0 arch/x86/kernel/apic/apic.c:1061
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:native_irq_disable arch/x86/include/asm/irqflags.h:37 [inline]
RIP: 0010:arch_local_irq_disable arch/x86/include/asm/irqflags.h:114 [inline]
RIP: 0010:default_idle+0xf/0x20 arch/x86/kernel/process.c:768
Code: 4c 01 c7 4c 29 c2 e9 6e ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 13 a4 23 00 fb f4 <fa> e9 8b 8d 02 00 cc 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
RSP: 0018:ffffc9000014fe28 EFLAGS: 00000202
RAX: 00000000006a8f01 RBX: ffffffff81654d3a RCX: 0000000080000001
RDX: 0000000000000001 RSI: ffffffff8e3ef513 RDI: ffffffff8c21df20
RBP: ffffc9000014ff20 R08: ffff888069532993 R09: 1ffff1100d2a6532
R10: dffffc0000000000 R11: ffffed100d2a6533 R12: 0000000000000001
R13: 1ffff11002c19000 R14: 0000000000000001 R15: 1ffff11002c19000
default_idle_call+0x70/0xb0 kernel/sched/idle.c:122
cpuidle_idle_call kernel/sched/idle.c:199 [inline]
do_idle+0x36a/0x5e0 kernel/sched/idle.c:352
cpu_startup_entry+0x43/0x60 kernel/sched/idle.c:451
start_secondary+0x101/0x110 arch/x86/kernel/smpboot.c:312
common_startup_64+0x12c/0x137
</TASK>
The explanation of the root cause is:
The root cause of this crash is a massive out-of-bounds memory write (buffer overflow) in the BPF helper `bpf_snprintf` (and other `bprintf`-like helpers) due to integer signedness and unchecked return values in `bpf_bprintf_prepare()`.
This memory corruption overwrites adjacent kernel memory, which in this case included `hrtimer` structures and their `timerqueue` rbtree nodes. When CPU 1 later attempted to remove a corrupted timer via `__remove_hrtimer` -> `timerqueue_linked_del` -> `rb_erase_linked`, it got stuck in an infinite loop traversing a cyclic rbtree. CPU 0 then detected a soft lockup because it was waiting for CPU 1 to acknowledge an IPI (`csd_lock_wait`), but CPU 1 had interrupts disabled while processing the timer queue.
### Detailed Analysis
In `kernel/bpf/helpers.c`, the `bpf_bprintf_prepare()` function formats arguments into a temporary 512-byte buffer (`buffers->bin_args`). There are two distinct bugs in this function that allow `tmp_buf` to advance past `tmp_buf_end`, bypassing bounds checks.
#### Bug 1: Unchecked `snprintf` return value for `%pB`
When handling the `%pB` format specifier, the code does:
```c
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
```
`snprintf` returns the number of characters that *would* have been written if the buffer was large enough, excluding the null terminator. If the remaining buffer space (`tmp_buf_end - tmp_buf`) is smaller than the formatted string, `err` will be greater than or equal to the remaining space. Consequently, `tmp_buf += (err + 1)` will advance `tmp_buf` **past** `tmp_buf_end`.
#### Bug 2: Signedness bug in bounds checking
When handling standard format specifiers (like `%d`, `%x`), the code aligns `tmp_buf` and checks if there is enough space:
```c
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
```
Here, `tmp_buf_end - tmp_buf` is of type `ptrdiff_t` (signed), and `sizeof_cur_arg` is of type `size_t` (unsigned). In C, this comparison forces the signed `ptrdiff_t` to be cast to an unsigned `size_t`.
If `tmp_buf` has already been advanced past `tmp_buf_end` (e.g., via the `%pB` bug above, or by using `%c` to make `tmp_buf` unaligned such that `PTR_ALIGN` pushes it past `tmp_buf_end`), `tmp_buf_end - tmp_buf` becomes negative. When cast to `size_t`, this negative value becomes a huge positive number (e.g., `0xFFFFFFFFFFFFFFF0`).
The condition `HUGE_NUMBER < sizeof_cur_arg` evaluates to `false`, bypassing the `-ENOSPC` check. The code then writes to `tmp_buf`, corrupting memory out of bounds.
Furthermore, if a `%s` specifier is used after `tmp_buf` is out of bounds:
```c
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
```
The equality check `tmp_buf_end == tmp_buf` fails to catch that `tmp_buf > tmp_buf_end`. Then `bpf_trace_copy_string` is called with a negative `bufsz` (`tmp_buf_end - tmp_buf`), which is cast to a huge `size_t`. This results in copying an arbitrarily large string into kernel memory, causing massive corruption.
### Fix
To fix this vulnerability:
1. For `%pB`, `tmp_buf` should only be advanced by the actual number of bytes written, and it must not exceed `tmp_buf_end`.
2. The bounds checks must correctly handle the case where `tmp_buf >= tmp_buf_end`. A safe way is to cast `tmp_buf_end - tmp_buf` to a signed type and check if it's less than the required size, or simply check `if (tmp_buf + sizeof_cur_arg > tmp_buf_end)`.
The patch that fixes the bug is:
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2bb60200c..0e3d3bddd 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -921,10 +921,22 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
+ err = -ENOSPC;
+ goto out;
+ }
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += (err + 1);
}
@@ -944,12 +956,13 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
- if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
+ sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
+
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
@@ -964,6 +977,14 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += err + 1;
num_spec++;
@@ -982,7 +1003,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1004,7 +1025,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1038,7 +1059,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
- if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
Search for the commit(s) that introduced this bug.
|
| 707/2 |
2026/05/18 21:03 |
llm |
fixes-finder |
0m
Model:
Tokens:
input: 9755
output: 37
thoughts: 104
Thoughts: **Pinpointing the Bug's Genesis**
Okay, so we've got a patch that fixes a nasty signedness and unchecked return value issue in `bpf_bprintf_prepare()`. I need to track down the commit that actually *introduced* this bug. The file in question is `kernel/bpf/helpers.c`, and the specific function is, as mentioned, `bpf_bprintf_prepare()`. The problem seems to be linked to how `%pB` and potentially other format specifiers interact with integer signedness and, crucially, how we handle the return value of `snprintf`. I'm going to start by using `git log`. I'll focus the search on the history of that particular function within `kernel/bpf/helpers.c`. That should give me the commit history, and with a bit of code archaeology, I should be able to pinpoint the culprit. Let's see what `git-log` reveals.
|
| 708/2 |
2026/05/18 21:03 |
tool |
git-log |
0m
Args: map[SourcePath:kernel/bpf/helpers.c SymbolName:bpf_bprintf_prepare]
Results: map[Output:b960430ea886 bpf: allow UTF-8 literals in bpf_bprintf_prepare()
beb1097ec8bd Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf after rc6
0426729f46cd bpf: Refactor bprintf buffer support
f8242745871f bpf: Reject %p% format string in bprintf-like helpers
e2bb9e01d589 bpf: Remove trace_printk_lock
f19a4050455a bpf: Do cleanup in bpf_bprintf_cleanup only when needed
78aa1cc94043 bpf: Add struct for bin_args arg in bpf_bprintf_prepare
3478cfcfcddf bpf: Support "%c" in bpf_bprintf_prepare().
8afcc19fbf08 bpf: Clarify a bpf_bprintf_prepare macro
48cac3f4a96d bpf: Implement formatted output helpers with bstr_printf
]
|
| 709/2 |
2026/05/18 21:03 |
llm |
fixes-finder |
0m
Model:
Tokens:
input: 10377
output: 25
thoughts: 184
Thoughts: **Analyzing the BPF Formatting Patch**
Okay, so I'm diving into this commit, `48cac3f4a96d bpf: Implement formatted output helpers with bstr_printf`. My immediate thought is that this is where `bpf_bprintf_prepare` was introduced, so it's the logical starting point for understanding the current formatting implementation. I need to get a full understanding of the changes made, so I'm going to also pull up `0426729f46cd bpf: Refactor bprintf buffer support` and `3478cfcfcddf bpf: Support "%c" in bpf_bprintf_prepare()` to see the larger picture. I'm especially interested in these two patches since they seem to refine the buffer support.
I see that this initial patch tackles several crucial areas – fixing the `%pB` and various IP address formatting issues (`%pi4`/`%pi6`/`%pI4`/`%pI6`), along with the `%s` handling and a critical check around `tmp_buf_end - tmp_buf < sizeof_cur_arg`. I really want to understand the details of the changes to these, as they're foundational for robust format string handling in BPF.
Now, let's look at the actual diff for `48cac3f4a96d`. That's where the real insights will be.
|
| 710/2 |
2026/05/18 21:04 |
tool |
git-show |
0m
Args:
Results: map[Output:commit 48cac3f4a96ddf08df8e53809ed066de0dc93915
Author: Florent Revest <revest@chromium.org>
Date: Tue Apr 27 19:43:13 2021 +0200
bpf: Implement formatted output helpers with bstr_printf
BPF has three formatted output helpers: bpf_trace_printk, bpf_seq_printf
and bpf_snprintf. Their signatures specify that all arguments are
provided from the BPF world as u64s (in an array or as registers). All
of these helpers are currently implemented by calling functions such as
snprintf() whose signatures take a variable number of arguments, then
placed in a va_list by the compiler to call vsnprintf().
"d9c9e4db bpf: Factorize bpf_trace_printk and bpf_seq_printf" introduced
a bpf_printf_prepare function that fills an array of u64 sanitized
arguments with an array of "modifiers" which indicate what the "real"
size of each argument should be (given by the format specifier). The
BPF_CAST_FMT_ARG macro consumes these arrays and casts each argument to
its real size. However, the C promotion rules implicitely cast them all
back to u64s. Therefore, the arguments given to snprintf are u64s and
the va_list constructed by the compiler will use 64 bits for each
argument. On 64 bit machines, this happens to work well because 32 bit
arguments in va_lists need to occupy 64 bits anyway, but on 32 bit
architectures this breaks the layout of the va_list expected by the
called function and mangles values.
In "88a5c690b6 bpf: fix bpf_trace_printk on 32 bit archs", this problem
had been solved for bpf_trace_printk only with a "horrid workaround"
that emitted multiple calls to trace_printk where each call had
different argument types and generated different va_list layouts. One of
the call would be dynamically chosen at runtime. This was ok with the 3
arguments that bpf_trace_printk takes but bpf_seq_printf and
bpf_snprintf accept up to 12 arguments. Because this approach scales
code exponentially, it is not a viable option anymore.
Because the promotion rules are part of the language and because the
construction of a va_list is an arch-specific ABI, it's best to just
avoid variadic arguments and va_lists altogether. Thankfully the
kernel's snprintf() has an alternative in the form of bstr_printf() that
accepts arguments in a "binary buffer representation". These binary
buffers are currently created by vbin_printf and used in the tracing
subsystem to split the cost of printing into two parts: a fast one that
only dereferences and remembers values, and a slower one, called later,
that does the pretty-printing.
This patch refactors bpf_printf_prepare to construct binary buffers of
arguments consumable by bstr_printf() instead of arrays of arguments and
modifiers. This gets rid of BPF_CAST_FMT_ARG and greatly simplifies the
bpf_printf_prepare usage but there are a few gotchas that change how
bpf_printf_prepare needs to do things.
Currently, bpf_printf_prepare uses a per cpu temporary buffer as a
generic storage for strings and IP addresses. With this refactoring, the
temporary buffers now holds all the arguments in a structured binary
format.
To comply with the format expected by bstr_printf, certain format
specifiers also need to be pre-formatted: %pB and %pi6/%pi4/%pI4/%pI6.
Because vsnprintf subroutines for these specifiers are hard to expose,
we pre-format these arguments with calls to snprintf().
Reported-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
Signed-off-by: Florent Revest <revest@chromium.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20210427174313.860948-3-revest@chromium.org
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index ad4bcf1cadbb..b33f199c4cc2 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2081,24 +2081,8 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,
struct btf_id_set;
bool btf_id_set_contains(const struct btf_id_set *set, u32 id);
-enum bpf_printf_mod_type {
- BPF_PRINTF_INT,
- BPF_PRINTF_LONG,
- BPF_PRINTF_LONG_LONG,
-};
-
-/* Workaround for getting va_list handling working with different argument type
- * combinations generically for 32 and 64 bit archs.
- */
-#define BPF_CAST_FMT_ARG(arg_nb, args, mod) \
- (mod[arg_nb] == BPF_PRINTF_LONG_LONG || \
- (mod[arg_nb] == BPF_PRINTF_LONG && __BITS_PER_LONG == 64) \
- ? (u64)args[arg_nb] \
- : (u32)args[arg_nb])
-
-int bpf_printf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
- u64 *final_args, enum bpf_printf_mod_type *mod,
- u32 num_args);
-void bpf_printf_cleanup(void);
+int bpf_bprintf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
+ u32 **bin_buf, u32 num_args);
+void bpf_bprintf_cleanup(void);
#endif /* _LINUX_BPF_H */
diff --git a/init/Kconfig b/init/Kconfig
index 5deae45b8d81..0d82a1f838cc 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1708,6 +1708,7 @@ config BPF_SYSCALL
select BPF
select IRQ_WORK
select TASKS_TRACE_RCU
+ select BINARY_PRINTF
select NET_SOCK_MSG if INET
default n
help
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 85b26ca5aacd..544773970dbc 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -707,9 +707,6 @@ static int try_get_fmt_tmp_buf(char **tmp_buf)
struct bpf_printf_buf *bufs;
int used;
- if (*tmp_buf)
- return 0;
-
preempt_disable();
used = this_cpu_inc_return(bpf_printf_buf_used);
if (WARN_ON_ONCE(used > 1)) {
@@ -723,7 +720,7 @@ static int try_get_fmt_tmp_buf(char **tmp_buf)
return 0;
}
-void bpf_printf_cleanup(void)
+void bpf_bprintf_cleanup(void)
{
if (this_cpu_read(bpf_printf_buf_used)) {
this_cpu_dec(bpf_printf_buf_used);
@@ -732,43 +729,45 @@ void bpf_printf_cleanup(void)
}
/*
- * bpf_parse_fmt_str - Generic pass on format strings for printf-like helpers
+ * bpf_bprintf_prepare - Generic pass on format strings for bprintf-like helpers
*
* Returns a negative value if fmt is an invalid format string or 0 otherwise.
*
* This can be used in two ways:
- * - Format string verification only: when final_args and mod are NULL
+ * - Format string verification only: when bin_args is NULL
* - Arguments preparation: in addition to the above verification, it writes in
- * final_args a copy of raw_args where pointers from BPF have been sanitized
- * into pointers safe to use by snprintf. This also writes in the mod array
- * the size requirement of each argument, usable by BPF_CAST_FMT_ARG for ex.
+ * bin_args a binary representation of arguments usable by bstr_printf where
+ * pointers from BPF have been sanitized.
*
* In argument preparation mode, if 0 is returned, safe temporary buffers are
- * allocated and bpf_printf_cleanup should be called to free them after use.
+ * allocated and bpf_bprintf_cleanup should be called to free them after use.
*/
-int bpf_printf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
- u64 *final_args, enum bpf_printf_mod_type *mod,
- u32 num_args)
+int bpf_bprintf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
+ u32 **bin_args, u32 num_args)
{
- char *unsafe_ptr = NULL, *tmp_buf = NULL, *fmt_end;
- size_t tmp_buf_len = MAX_PRINTF_BUF_LEN;
- int err, i, num_spec = 0, copy_size;
- enum bpf_printf_mod_type cur_mod;
+ char *unsafe_ptr = NULL, *tmp_buf = NULL, *tmp_buf_end, *fmt_end;
+ size_t sizeof_cur_arg, sizeof_cur_ip;
+ int err, i, num_spec = 0;
u64 cur_arg;
- char fmt_ptype;
-
- if (!!final_args != !!mod)
- return -EINVAL;
+ char fmt_ptype, cur_ip[16], ip_spec[] = "%pXX";
fmt_end = strnchr(fmt, fmt_size, 0);
if (!fmt_end)
return -EINVAL;
fmt_size = fmt_end - fmt;
+ if (bin_args) {
+ if (num_args && try_get_fmt_tmp_buf(&tmp_buf))
+ return -EBUSY;
+
+ tmp_buf_end = tmp_buf + MAX_PRINTF_BUF_LEN;
+ *bin_args = (u32 *)tmp_buf;
+ }
+
for (i = 0; i < fmt_size; i++) {
if ((!isprint(fmt[i]) && !isspace(fmt[i])) || !isascii(fmt[i])) {
err = -EINVAL;
- goto cleanup;
+ goto out;
}
if (fmt[i] != '%')
@@ -781,7 +780,7 @@ int bpf_printf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
if (num_spec >= num_args) {
err = -EINVAL;
- goto cleanup;
+ goto out;
}
/* The string is zero-terminated so if fmt[i] != 0, we can
@@ -800,7 +799,7 @@ int bpf_printf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
}
if (fmt[i] == 'p') {
- cur_mod = BPF_PRINTF_LONG;
+ sizeof_cur_arg = sizeof(long);
if ((fmt[i + 1] == 'k' || fmt[i + 1] == 'u') &&
fmt[i + 2] == 's') {
@@ -811,117 +810,140 @@ int bpf_printf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
if (fmt[i + 1] == 0 || isspace(fmt[i + 1]) ||
ispunct(fmt[i + 1]) || fmt[i + 1] == 'K' ||
- fmt[i + 1] == 'x' || fmt[i + 1] == 'B' ||
- fmt[i + 1] == 's' || fmt[i + 1] == 'S') {
+ fmt[i + 1] == 'x' || fmt[i + 1] == 's' ||
+ fmt[i + 1] == 'S') {
/* just kernel pointers */
- if (final_args)
+ if (tmp_buf)
cur_arg = raw_args[num_spec];
- goto fmt_next;
+ i++;
+ goto nocopy_fmt;
+ }
+
+ if (fmt[i + 1] == 'B') {
+ if (tmp_buf) {
+ err = snprintf(tmp_buf,
+ (tmp_buf_end - tmp_buf),
+ "%pB",
+ (void *)(long)raw_args[num_spec]);
+ tmp_buf += (err + 1);
+ }
+
+ i++;
+ num_spec++;
+ continue;
}
/* only support "%pI4", "%pi4", "%pI6" and "%pi6". */
if ((fmt[i + 1] != 'i' && fmt[i + 1] != 'I') ||
(fmt[i + 2] != '4' && fmt[i + 2] != '6')) {
err = -EINVAL;
- goto cleanup;
+ goto out;
}
i += 2;
- if (!final_args)
- goto fmt_next;
+ if (!tmp_buf)
+ goto nocopy_fmt;
- if (try_get_fmt_tmp_buf(&tmp_buf)) {
- err = -EBUSY;
- goto out;
- }
-
- copy_size = (fmt[i + 2] == '4') ? 4 : 16;
- if (tmp_buf_len < copy_size) {
+ sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
+ if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
err = -ENOSPC;
- goto cleanup;
+ goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
- err = copy_from_kernel_nofault(tmp_buf, unsafe_ptr,
- copy_size);
+ err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
+ sizeof_cur_ip);
if (err < 0)
- memset(tmp_buf, 0, copy_size);
- cur_arg = (u64)(long)tmp_buf;
- tmp_buf += copy_size;
- tmp_buf_len -= copy_size;
+ memset(cur_ip, 0, sizeof_cur_ip);
+
+ /* hack: bstr_printf expects IP addresses to be
+ * pre-formatted as strings, ironically, the easiest way
+ * to do that is to call snprintf.
+ */
+ ip_spec[2] = fmt[i - 1];
+ ip_spec[3] = fmt[i];
+ err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
+ ip_spec, &cur_ip);
- goto fmt_next;
+ tmp_buf += err + 1;
+ num_spec++;
+
+ continue;
} else if (fmt[i] == 's') {
- cur_mod = BPF_PRINTF_LONG;
fmt_ptype = fmt[i];
fmt_str:
if (fmt[i + 1] != 0 &&
!isspace(fmt[i + 1]) &&
!ispunct(fmt[i + 1])) {
err = -EINVAL;
- goto cleanup;
- }
-
- if (!final_args)
- goto fmt_next;
-
- if (try_get_fmt_tmp_buf(&tmp_buf)) {
- err = -EBUSY;
goto out;
}
- if (!tmp_buf_len) {
+ if (!tmp_buf)
+ goto nocopy_fmt;
+
+ if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
- goto cleanup;
+ goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
- fmt_ptype, tmp_buf_len);
+ fmt_ptype,
+ tmp_buf_end - tmp_buf);
if (err < 0) {
tmp_buf[0] = '\0';
err = 1;
}
- cur_arg = (u64)(long)tmp_buf;
tmp_buf += err;
- tmp_buf_len -= err;
+ num_spec++;
- goto fmt_next;
+ continue;
}
- cur_mod = BPF_PRINTF_INT;
+ sizeof_cur_arg = sizeof(int);
if (fmt[i] == 'l') {
- cur_mod = BPF_PRINTF_LONG;
+ sizeof_cur_arg = sizeof(long);
i++;
}
if (fmt[i] == 'l') {
- cur_mod = BPF_PRINTF_LONG_LONG;
+ sizeof_cur_arg = sizeof(long long);
i++;
}
if (fmt[i] != 'i' && fmt[i] != 'd' && fmt[i] != 'u' &&
fmt[i] != 'x' && fmt[i] != 'X') {
err = -EINVAL;
- goto cleanup;
+ goto out;
}
- if (final_args)
+ if (tmp_buf)
cur_arg = raw_args[num_spec];
-fmt_next:
- if (final_args) {
- mod[num_spec] = cur_mod;
- final_args[num_spec] = cur_arg;
+nocopy_fmt:
+ if (tmp_buf) {
+ tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
+ if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ err = -ENOSPC;
+ goto out;
+ }
+
+ if (sizeof_cur_arg == 8) {
+ *(u32 *)tmp_buf = *(u32 *)&cur_arg;
+ *(u32 *)(tmp_buf + 4) = *((u32 *)&cur_arg + 1);
+ } else {
+ *(u32 *)tmp_buf = (u32)(long)cur_arg;
+ }
+ tmp_buf += sizeof_cur_arg;
}
num_spec++;
}
err = 0;
-cleanup:
- if (err)
- bpf_printf_cleanup();
out:
+ if (err)
+ bpf_bprintf_cleanup();
return err;
}
@@ -930,9 +952,8 @@ int bpf_printf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
const void *, data, u32, data_len)
{
- enum bpf_printf_mod_type mod[MAX_SNPRINTF_VARARGS];
- u64 args[MAX_SNPRINTF_VARARGS];
int err, num_args;
+ u32 *bin_args;
if (data_len % 8 || data_len > MAX_SNPRINTF_VARARGS * 8 ||
(data_len && !data))
@@ -942,22 +963,13 @@ BPF_CALL_5(bpf_snprintf, char *, str, u32, str_size, char *, fmt,
/* ARG_PTR_TO_CONST_STR guarantees that fmt is zero-terminated so we
* can safely give an unbounded size.
*/
- err = bpf_printf_prepare(fmt, UINT_MAX, data, args, mod, num_args);
+ err = bpf_bprintf_prepare(fmt, UINT_MAX, data, &bin_args, num_args);
if (err < 0)
return err;
- /* Maximumly we can have MAX_SNPRINTF_VARARGS parameters, just give
- * all of them to snprintf().
- */
- err = snprintf(str, str_size, fmt, BPF_CAST_FMT_ARG(0, args, mod),
- BPF_CAST_FMT_ARG(1, args, mod), BPF_CAST_FMT_ARG(2, args, mod),
- BPF_CAST_FMT_ARG(3, args, mod), BPF_CAST_FMT_ARG(4, args, mod),
- BPF_CAST_FMT_ARG(5, args, mod), BPF_CAST_FMT_ARG(6, args, mod),
- BPF_CAST_FMT_ARG(7, args, mod), BPF_CAST_FMT_ARG(8, args, mod),
- BPF_CAST_FMT_ARG(9, args, mod), BPF_CAST_FMT_ARG(10, args, mod),
- BPF_CAST_FMT_ARG(11, args, mod));
-
- bpf_printf_cleanup();
+ err = bstr_printf(str, str_size, fmt, bin_args);
+
+ bpf_bprintf_cleanup();
return err + 1;
}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 9145f88b2a0a..8fd552c16763 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -5946,7 +5946,7 @@ static int check_bpf_snprintf_call(struct bpf_verifier_env *env,
/* We are also guaranteed that fmt+fmt_map_off is NULL terminated, we
* can focus on validating the format specifiers.
*/
- err = bpf_printf_prepare(fmt, UINT_MAX, NULL, NULL, NULL, num_args);
+ err = bpf_bprintf_prepare(fmt, UINT_MAX, NULL, NULL, num_args);
if (err < 0)
verbose(env, "Invalid format string\n");
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 0e67d12a8f40..d2d7cf6cfe83 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -381,27 +381,23 @@ BPF_CALL_5(bpf_trace_printk, char *, fmt, u32, fmt_size, u64, arg1,
u64, arg2, u64, arg3)
{
u64 args[MAX_TRACE_PRINTK_VARARGS] = { arg1, arg2, arg3 };
- enum bpf_printf_mod_type mod[MAX_TRACE_PRINTK_VARARGS];
+ u32 *bin_args;
static char buf[BPF_TRACE_PRINTK_SIZE];
unsigned long flags;
int ret;
- ret = bpf_printf_prepare(fmt, fmt_size, args, args, mod,
- MAX_TRACE_PRINTK_VARARGS);
+ ret = bpf_bprintf_prepare(fmt, fmt_size, args, &bin_args,
+ MAX_TRACE_PRINTK_VARARGS);
if (ret < 0)
return ret;
raw_spin_lock_irqsave(&trace_printk_lock, flags);
- ret = snprintf(buf, sizeof(buf), fmt, BPF_CAST_FMT_ARG(0, args, mod),
- BPF_CAST_FMT_ARG(1, args, mod), BPF_CAST_FMT_ARG(2, args, mod));
- /* snprintf() will not append null for zero-length strings */
- if (ret == 0)
- buf[0] = '\0';
+ ret = bstr_printf(buf, sizeof(buf), fmt, bin_args);
trace_bpf_trace_printk(buf);
raw_spin_unlock_irqrestore(&trace_printk_lock, flags);
- bpf_printf_cleanup();
+ bpf_bprintf_cleanup();
return ret;
}
@@ -435,31 +431,21 @@ const struct bpf_func_proto *bpf_get_trace_printk_proto(void)
BPF_CALL_5(bpf_seq_printf, struct seq_file *, m, char *, fmt, u32, fmt_size,
const void *, data, u32, data_len)
{
- enum bpf_printf_mod_type mod[MAX_SEQ_PRINTF_VARARGS];
- u64 args[MAX_SEQ_PRINTF_VARARGS];
int err, num_args;
+ u32 *bin_args;
if (data_len & 7 || data_len > MAX_SEQ_PRINTF_VARARGS * 8 ||
(data_len && !data))
return -EINVAL;
num_args = data_len / 8;
- err = bpf_printf_prepare(fmt, fmt_size, data, args, mod, num_args);
+ err = bpf_bprintf_prepare(fmt, fmt_size, data, &bin_args, num_args);
if (err < 0)
return err;
- /* Maximumly we can have MAX_SEQ_PRINTF_VARARGS parameter, just give
- * all of them to seq_printf().
- */
- seq_printf(m, fmt, BPF_CAST_FMT_ARG(0, args, mod),
- BPF_CAST_FMT_ARG(1, args, mod), BPF_CAST_FMT_ARG(2, args, mod),
- BPF_CAST_FMT_ARG(3, args, mod), BPF_CAST_FMT_ARG(4, args, mod),
- BPF_CAST_FMT_ARG(5, args, mod), BPF_CAST_FMT_ARG(6, args, mod),
- BPF_CAST_FMT_ARG(7, args, mod), BPF_CAST_FMT_ARG(8, args, mod),
- BPF_CAST_FMT_ARG(9, args, mod), BPF_CAST_FMT_ARG(10, args, mod),
- BPF_CAST_FMT_ARG(11, args, mod));
-
- bpf_printf_cleanup();
+ seq_bprintf(m, fmt, bin_args);
+
+ bpf_bprintf_cleanup();
return seq_has_overflowed(m) ? -EOVERFLOW : 0;
}
]
|
| 711/2 |
2026/05/18 21:04 |
llm |
fixes-finder |
0m
Model:
Tokens:
input: 18206
output: 50
thoughts: 1140
Thoughts: **Analysis of Commit 48cac3f4a96ddf08df8e53809ed066de0dc93915 and its Bug Introductions**
Right, let me break this down. My initial hunch was correct: the culprit is commit `48cac3f4a96ddf08df8e53809ed066de0dc93915`. I needed to confirm that there wasn't an earlier, pre-existing issue in the codebase before this change.
My review is triggered by the crash report, which strongly suggests problems tied to `bpf_bprintf_prepare`. My thought process went like this: first, identify the key changes introduced by the commit. I see the new handling of `%pB`, the introduction of `%pi4`, the way `%s` is now handled with `bpf_trace_copy_string`, and the length check `tmp_buf_end - tmp_buf < sizeof_cur_arg`.
The first thing I needed to do was a deeper dive into commit `48cac3f4a96ddf08df8e53809ed066de0dc93915`. Specifically, I homed in on the `snprintf` use for `%pB` and how `tmp_buf` is incremented. I noted the explicit inclusion of the code related to `%pB`, with the `snprintf` call, and the crucial `tmp_buf += (err + 1);` increment, as that's immediately concerning. The commit *directly* introduces the `snprintf` handling of `%pB` which is a key change.
Next, I looked at the `%s` implementation: the check for `tmp_buf_end == tmp_buf` *before* calling `bpf_trace_copy_string` jumped out immediately, indicating a potential buffer overflow condition, which matched Bug 1 in the crash report.
Then, there was the alignment check `tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));` and size check `tmp_buf_end - tmp_buf < sizeof_cur_arg`, that also directly related to Bug 2 in the crash report.
To be absolutely sure, I went back a step and considered the *origin* of `bpf_bprintf_prepare`. It was essentially a refactoring and rename of `bpf_printf_prepare`. I needed to eliminate the possibility of a pre-existing issue. So, I checked the older code's behavior *before* this refactor. Before the commit, `%pB` was treated merely as a kernel pointer, without any `snprintf` call, and a different length check implementation existed. Therefore, I've confirmed that the bugs are new and were introduced within this commit.
Thus, my analysis definitively concludes that commit `48cac3f4a96ddf08df8e53809ed066de0dc93915` is the source of the bugs described in the crash report, introducing the flaws in the `%pB`, `%s`, and `tmp_buf_end - tmp_buf` handling.
|
| 712/2 |
2026/05/18 21:04 |
tool |
set-results |
0m
Args: map[FixesHash:48cac3f4a96ddf08df8e53809ed066de0dc93915]
Results: map[FixesHash:48cac3f4a96ddf08df8e53809ed066de0dc93915]
|
| 713/1 |
2026/05/18 21:04 |
action |
format-fixes |
0m
Results: map[Fixes:map[Hash:48cac3f4a96ddf08df8e53809ed066de0dc93915 Title:bpf: Implement formatted output helpers with bstr_printf]]
|
| 714/1 |
2026/05/18 21:04 |
action |
get-maintainers |
0m
Results: map[Recipients:[map[Email:andrii@kernel.org Name:Andrii Nakryiko To:false] map[Email:ast@kernel.org Name:Alexei Starovoitov To:false] map[Email:bpf@vger.kernel.org Name:open list:BPF [GENERAL] (Safe Dynamic Programs and Tools) To:false] map[Email:daniel@iogearbox.net Name:Daniel Borkmann To:false] map[Email:eddyz87@gmail.com Name:Eduard Zingerman To:false] map[Email:jolsa@kernel.org Name:Jiri Olsa To:false] map[Email:linux-kernel@vger.kernel.org Name: To:true] map[Email:martin.lau@linux.dev Name:Martin KaFai Lau To:false] map[Email:memxor@gmail.com Name:Kumar Kartikeya Dwivedi To:false] map[Email:song@kernel.org Name:Song Liu To:false] map[Email:yonghong.song@linux.dev Name:Yonghong Song To:false]]]
|
| 715/1 |
2026/05/18 21:04 |
action |
get-recent-commits |
0m
Results: map[RecentCommits:bpf: allow UTF-8 literals in bpf_bprintf_prepare()
bpf: Retire rcu_trace_implies_rcu_gp()
bpf: Migrate dynptr file to kmalloc_nolock
bpf: Migrate bpf_task_work to kmalloc_nolock
bpf: Support struct btf_struct_meta via KF_IMPLICIT_ARGS
s390: Introduce bpf_get_lowcore() kfunc
Convert 'alloc_obj' family to use the new default GFP_KERNEL argument
treewide: Replace kmalloc with kmalloc_obj for non-scalar types
bpf: Remove task local storage percpu counter
bpf: Prevent reentrance into call_rcu_tasks_trace()
bpf: Reset prog callback in bpf_async_cancel_and_free()
bpf: Check for running wq callback when freeing bpf_async_cb
bpf: Add a recursion check to prevent loops in bpf_timer
bpf: Tighten conditions when timer/wq can be called synchronously
bpf: Introduce bpf_timer_cancel_async() kfunc
bpf: Enable bpf_timer and bpf_wq in any context
bpf: Add bpf_stream_print_stack stack dumping kfunc
bpf: add bpf_strncasecmp kfunc
bpf: Simplify bpf_timer_cancel()
bpf: Introduce lock-free bpf_async_update_prog_callback()
]
|
| 716/1 |
2026/05/18 21:04 |
agent |
description-generator |
0m
Model:
Instruction:
You are an experienced Linux kernel developer tasked with writing a commit description for
a kernel bug fixing commit. The description should start with a one-line summary,
and then include description of the bug being fixed, and how it's fixed by the provided patch.
Your final reply should contain only the text of the commit description.
Phrase the one-line summary so that it is not longer than 72 characters.
IMPORTANT: Do not wrap lines manually (e.g., at 80 characters); we will reformat the text
automatically, so keep paragraphs as single lines without newlines.
Prefer calling several tools at the same time to save round-trips.
Prompt:
The crash that corresponds to the bug is:
watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/u9:4:1179]
CPU#0 Utilization every 4000ms during lockup:
#1: 100% system, 1% softirq, 1% hardirq, 0% idle
#2: 100% system, 1% softirq, 0% hardirq, 0% idle
#3: 100% system, 1% softirq, 1% hardirq, 0% idle
#4: 100% system, 1% softirq, 1% hardirq, 0% idle
#5: 100% system, 1% softirq, 1% hardirq, 0% idle
Modules linked in:
irq event stamp: 386488
hardirqs last enabled at (386487): [<ffffffff8100148a>] asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
hardirqs last disabled at (386488): [<ffffffff8b9eb11e>] sysvec_apic_timer_interrupt+0xe/0xb0 arch/x86/kernel/apic/apic.c:1061
softirqs last enabled at (386486): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last enabled at (386486): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
softirqs last disabled at (386409): [<ffffffff8154d59a>] __do_softirq kernel/softirq.c:656 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] invoke_softirq kernel/softirq.c:496 [inline]
softirqs last disabled at (386409): [<ffffffff8154d59a>] __irq_exit_rcu+0xca/0x210 kernel/softirq.c:735
CPU: 0 UID: 0 PID: 1179 Comm: kworker/u9:4 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:csd_lock_wait kernel/smp.c:342 [inline]
RIP: 0010:smp_call_function_many_cond+0xfcb/0x13b0 kernel/smp.c:892
Code: 79 45 8b 2e 44 89 ee 83 e6 01 31 ff e8 be c6 0b 00 41 83 e5 01 49 bd 00 00 00 00 00 fc ff df 75 07 e8 69 c2 0b 00 eb 37 f3 90 <43> 0f b6 04 2c 84 c0 75 10 41 f7 06 01 00 00 00 74 1e e8 4e c2 0b
RSP: 0018:ffffc90005fcf720 EFLAGS: 00000293
RAX: ffffffff81834d62 RBX: ffff88806943ad08 RCX: ffff88801d361c00
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc90005fcf860 R08: ffffffff908e5af7 R09: 1ffffffff211cb5e
R10: dffffc0000000000 R11: fffffbfff211cb5f R12: 1ffff1100d2a7edd
R13: dffffc0000000000 R14: ffff88806953f6e8 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff8880d5abe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555578641490 CR3: 000000000eaf4000 CR4: 00000000000006f0
Call Trace:
<TASK>
on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1057
on_each_cpu include/linux/smp.h:72 [inline]
smp_text_poke_sync_each_cpu arch/x86/kernel/alternative.c:2773 [inline]
smp_text_poke_batch_finish+0x5e6/0x1090 arch/x86/kernel/alternative.c:2983
arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
static_key_enable_cpuslocked+0x128/0x240 kernel/jump_label.c:210
static_key_enable+0x1a/0x20 kernel/jump_label.c:223
toggle_allocation_gate+0xab/0x290 mm/kfence/core.c:906
process_one_work kernel/workqueue.c:3314 [inline]
process_scheduled_works+0xad1/0x1740 kernel/workqueue.c:3397
worker_thread+0xa59/0xfc0 kernel/workqueue.c:3478
kthread+0x388/0x470 kernel/kthread.c:436
ret_from_fork+0x437/0x9d0 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted syzkaller #1 PREEMPT
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
RIP: 0010:__remove_hrtimer+0x8e/0x3e0 kernel/time/hrtimer.c:1145
Code: ee e8 f6 fd 11 00 45 85 ed 48 89 5c 24 08 74 2a 48 8d 7b 18 be ff ff ff ff e8 ce a2 21 0a 41 89 c5 31 ff 89 c6 e8 d2 fd 11 00 <45> 85 ed 0f 84 33 02 00 00 e8 84 f9 11 00 eb 05 e8 7d f9 11 00 4d
RSP: 0018:ffffc900001b8d90 EFLAGS: 00000006
RAX: ffffffff817d161e RBX: ffff888069527f40 RCX: 0000000000010000
RDX: ffff8880160c8000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: dffffc0000000000 R08: ffffffff94f48153 R09: 1ffffffff29e902a
R10: dffffc0000000000 R11: fffffbfff29e902b R12: 0000000000000000
R13: 0000000000000001 R14: ffff888024042300 R15: ffff888069528000
FS: 0000000000000000(0000) GS:ffff8880d5bbe000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f83db077470 CR3: 000000001f132000 CR4: 00000000000006f0
Call Trace:
<IRQ>
__run_hrtimer kernel/time/hrtimer.c:1910 [inline]
__hrtimer_run_queues+0x299/0xa10 kernel/time/hrtimer.c:1994
hrtimer_interrupt+0x44b/0x950 kernel/time/hrtimer.c:2113
local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1050 [inline]
__sysvec_apic_timer_interrupt+0x102/0x420 arch/x86/kernel/apic/apic.c:1067
instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1061 [inline]
sysvec_apic_timer_interrupt+0x8b/0xb0 arch/x86/kernel/apic/apic.c:1061
</IRQ>
<TASK>
asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
RIP: 0010:native_irq_disable arch/x86/include/asm/irqflags.h:37 [inline]
RIP: 0010:arch_local_irq_disable arch/x86/include/asm/irqflags.h:114 [inline]
RIP: 0010:default_idle+0xf/0x20 arch/x86/kernel/process.c:768
Code: 4c 01 c7 4c 29 c2 e9 6e ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 13 a4 23 00 fb f4 <fa> e9 8b 8d 02 00 cc 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
RSP: 0018:ffffc9000014fe28 EFLAGS: 00000202
RAX: 00000000006a8f01 RBX: ffffffff81654d3a RCX: 0000000080000001
RDX: 0000000000000001 RSI: ffffffff8e3ef513 RDI: ffffffff8c21df20
RBP: ffffc9000014ff20 R08: ffff888069532993 R09: 1ffff1100d2a6532
R10: dffffc0000000000 R11: ffffed100d2a6533 R12: 0000000000000001
R13: 1ffff11002c19000 R14: 0000000000000001 R15: 1ffff11002c19000
default_idle_call+0x70/0xb0 kernel/sched/idle.c:122
cpuidle_idle_call kernel/sched/idle.c:199 [inline]
do_idle+0x36a/0x5e0 kernel/sched/idle.c:352
cpu_startup_entry+0x43/0x60 kernel/sched/idle.c:451
start_secondary+0x101/0x110 arch/x86/kernel/smpboot.c:312
common_startup_64+0x12c/0x137
</TASK>
The explanation of the root cause of the bug is:
The root cause of this crash is a massive out-of-bounds memory write (buffer overflow) in the BPF helper `bpf_snprintf` (and other `bprintf`-like helpers) due to integer signedness and unchecked return values in `bpf_bprintf_prepare()`.
This memory corruption overwrites adjacent kernel memory, which in this case included `hrtimer` structures and their `timerqueue` rbtree nodes. When CPU 1 later attempted to remove a corrupted timer via `__remove_hrtimer` -> `timerqueue_linked_del` -> `rb_erase_linked`, it got stuck in an infinite loop traversing a cyclic rbtree. CPU 0 then detected a soft lockup because it was waiting for CPU 1 to acknowledge an IPI (`csd_lock_wait`), but CPU 1 had interrupts disabled while processing the timer queue.
### Detailed Analysis
In `kernel/bpf/helpers.c`, the `bpf_bprintf_prepare()` function formats arguments into a temporary 512-byte buffer (`buffers->bin_args`). There are two distinct bugs in this function that allow `tmp_buf` to advance past `tmp_buf_end`, bypassing bounds checks.
#### Bug 1: Unchecked `snprintf` return value for `%pB`
When handling the `%pB` format specifier, the code does:
```c
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
tmp_buf += (err + 1);
}
i++;
num_spec++;
continue;
}
```
`snprintf` returns the number of characters that *would* have been written if the buffer was large enough, excluding the null terminator. If the remaining buffer space (`tmp_buf_end - tmp_buf`) is smaller than the formatted string, `err` will be greater than or equal to the remaining space. Consequently, `tmp_buf += (err + 1)` will advance `tmp_buf` **past** `tmp_buf_end`.
#### Bug 2: Signedness bug in bounds checking
When handling standard format specifiers (like `%d`, `%x`), the code aligns `tmp_buf` and checks if there is enough space:
```c
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
err = -ENOSPC;
goto out;
}
```
Here, `tmp_buf_end - tmp_buf` is of type `ptrdiff_t` (signed), and `sizeof_cur_arg` is of type `size_t` (unsigned). In C, this comparison forces the signed `ptrdiff_t` to be cast to an unsigned `size_t`.
If `tmp_buf` has already been advanced past `tmp_buf_end` (e.g., via the `%pB` bug above, or by using `%c` to make `tmp_buf` unaligned such that `PTR_ALIGN` pushes it past `tmp_buf_end`), `tmp_buf_end - tmp_buf` becomes negative. When cast to `size_t`, this negative value becomes a huge positive number (e.g., `0xFFFFFFFFFFFFFFF0`).
The condition `HUGE_NUMBER < sizeof_cur_arg` evaluates to `false`, bypassing the `-ENOSPC` check. The code then writes to `tmp_buf`, corrupting memory out of bounds.
Furthermore, if a `%s` specifier is used after `tmp_buf` is out of bounds:
```c
if (tmp_buf_end == tmp_buf) {
err = -ENOSPC;
goto out;
}
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = bpf_trace_copy_string(tmp_buf, unsafe_ptr,
fmt_ptype,
tmp_buf_end - tmp_buf);
```
The equality check `tmp_buf_end == tmp_buf` fails to catch that `tmp_buf > tmp_buf_end`. Then `bpf_trace_copy_string` is called with a negative `bufsz` (`tmp_buf_end - tmp_buf`), which is cast to a huge `size_t`. This results in copying an arbitrarily large string into kernel memory, causing massive corruption.
### Fix
To fix this vulnerability:
1. For `%pB`, `tmp_buf` should only be advanced by the actual number of bytes written, and it must not exceed `tmp_buf_end`.
2. The bounds checks must correctly handle the case where `tmp_buf >= tmp_buf_end`. A safe way is to cast `tmp_buf_end - tmp_buf` to a signed type and check if it's less than the required size, or simply check `if (tmp_buf + sizeof_cur_arg > tmp_buf_end)`.
The diff of the bug fix is:
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 2bb60200c..0e3d3bddd 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -921,10 +921,22 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (fmt[i + 1] == 'B') {
if (tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
+ err = -ENOSPC;
+ goto out;
+ }
err = snprintf(tmp_buf,
(tmp_buf_end - tmp_buf),
"%pB",
(void *)(long)raw_args[num_spec]);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += (err + 1);
}
@@ -944,12 +956,13 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
- if (tmp_buf_end - tmp_buf < sizeof_cur_ip) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
+ sizeof_cur_ip = (fmt[i] == '4') ? 4 : 16;
+
unsafe_ptr = (char *)(long)raw_args[num_spec];
err = copy_from_kernel_nofault(cur_ip, unsafe_ptr,
sizeof_cur_ip);
@@ -964,6 +977,14 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
ip_spec[3] = fmt[i];
err = snprintf(tmp_buf, tmp_buf_end - tmp_buf,
ip_spec, &cur_ip);
+ if (err < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+ if (err >= tmp_buf_end - tmp_buf) {
+ err = -ENOSPC;
+ goto out;
+ }
tmp_buf += err + 1;
num_spec++;
@@ -982,7 +1003,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1004,7 +1025,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
if (!tmp_buf)
goto nocopy_fmt;
- if (tmp_buf_end == tmp_buf) {
+ if (tmp_buf >= tmp_buf_end) {
err = -ENOSPC;
goto out;
}
@@ -1038,7 +1059,7 @@ int bpf_bprintf_prepare(const char *fmt, u32 fmt_size, const u64 *raw_args,
nocopy_fmt:
if (tmp_buf) {
tmp_buf = PTR_ALIGN(tmp_buf, sizeof(u32));
- if (tmp_buf_end - tmp_buf < sizeof_cur_arg) {
+ if (tmp_buf + sizeof_cur_arg > tmp_buf_end) {
err = -ENOSPC;
goto out;
}
Additional description of the patch:
I have analyzed the crash report, the root cause explanation, and the failed patch.
### What was wrong with the previous patch:
The previous patch attempted to fix the out-of-bounds write by truncating the string using `min_t(size_t, err + 1, limit)`. However, it did not return `-ENOSPC` when the internal binary arguments buffer (`bin_args`) was full. This caused `bpf_bprintf_prepare()` to return success (`0`) even when the string was truncated or completely omitted (if `limit == 0`).
Because `bstr_printf()` expects a fully formed, null-terminated string in the binary buffer for `%pB` and `%pI4`/`%pI6`, returning success on a truncated/missing string caused `bstr_printf()` to read out of bounds. If `limit == 0`, `snprintf()` wrote nothing, leaving `tmp_buf` at `tmp_buf_end` without a null terminator. `bstr_printf()` then called `strlen()` on uninitialized memory past the end of the buffer, leading to the soft lockup observed in the failed test.
### What I did in the new patch:
1. **Properly handle `snprintf` truncation for `%pB` and `%pI4`/`%pI6`:**
Instead of silently truncating the string, the code now checks if `snprintf()` returns an error (`err < 0`) or if it indicates truncation (`err >= tmp_buf_end - tmp_buf`). If truncation occurs, it means the internal 512-byte `bin_args` buffer is full, and we correctly return `-ENOSPC`. This safely aborts the formatting process and propagates the error to the BPF program, preventing `bstr_printf()` from reading malformed data.
2. **Fixed the flawed bounds check for IP addresses:**
The original code had a check `if (tmp_buf_end - tmp_buf < sizeof_cur_ip)` before formatting IP addresses. `sizeof_cur_ip` is the size of the *binary* IP address (4 or 16 bytes), but `snprintf()` writes the *formatted string* (up to 40 bytes for IPv6). This allowed `snprintf()` to write past `tmp_buf_end`, causing the exact same buffer overflow as `%pB`. I removed this incorrect check and replaced it with the robust `snprintf()` return value check described above.
3. **Fixed the integer signedness bug in bounds checking:**
The original bounds checks used `tmp_buf_end - tmp_buf < sizeof_cur_arg`. Because `sizeof_cur_arg` is an unsigned `size_t`, the signed `ptrdiff_t` result of the pointer subtraction was implicitly cast to unsigned. If `tmp_buf` ever exceeded `tmp_buf_end`, the negative difference became a huge positive number, bypassing the check. I rewrote these checks as `tmp_buf + sizeof_cur_arg > tmp_buf_end`. This uses safe pointer arithmetic and completely eliminates the signedness issue.
4. **Hardened equality checks:**
I updated the buffer full checks for `%s` and `%c` from `if (tmp_buf_end == tmp_buf)` to `if (tmp_buf >= tmp_buf_end)` to make the code robust against any future modifications that might inadvertently advance `tmp_buf` out of bounds.
### Why this is the best fix:
This fix addresses the root cause of the buffer overflows without breaking the contract between `bpf_bprintf_prepare()` and `bstr_printf()`. By returning `-ENOSPC` when the internal buffer is full, we ensure that `bstr_printf()` only ever operates on perfectly formed, null-terminated strings, completely eliminating the risk of out-of-bounds reads or soft lockups. Furthermore, rewriting the bounds checks to use pointer addition structurally immunizes the function against signedness bypasses.
Here are summaries of recent commits that touched the same files.
Format the summary line consistently with these, look how prefixes
are specified, letter capitalization, style, etc.
bpf: allow UTF-8 literals in bpf_bprintf_prepare()
bpf: Retire rcu_trace_implies_rcu_gp()
bpf: Migrate dynptr file to kmalloc_nolock
bpf: Migrate bpf_task_work to kmalloc_nolock
bpf: Support struct btf_struct_meta via KF_IMPLICIT_ARGS
s390: Introduce bpf_get_lowcore() kfunc
Convert 'alloc_obj' family to use the new default GFP_KERNEL argument
treewide: Replace kmalloc with kmalloc_obj for non-scalar types
bpf: Remove task local storage percpu counter
bpf: Prevent reentrance into call_rcu_tasks_trace()
bpf: Reset prog callback in bpf_async_cancel_and_free()
bpf: Check for running wq callback when freeing bpf_async_cb
bpf: Add a recursion check to prevent loops in bpf_timer
bpf: Tighten conditions when timer/wq can be called synchronously
bpf: Introduce bpf_timer_cancel_async() kfunc
bpf: Enable bpf_timer and bpf_wq in any context
bpf: Add bpf_stream_print_stack stack dumping kfunc
bpf: add bpf_strncasecmp kfunc
bpf: Simplify bpf_timer_cancel()
bpf: Introduce lock-free bpf_async_update_prog_callback()
Reply: bpf: Fix out-of-bounds write in bpf_bprintf_prepare()
The `bpf_bprintf_prepare()` function formats arguments into a temporary 512-byte buffer. There are two distinct bugs in this function that allow `tmp_buf` to advance past `tmp_buf_end`, bypassing bounds checks and causing out-of-bounds memory writes.
First, when handling the `%pB` and `%pI4`/`%pI6` format specifiers, `snprintf()` is used to format the string. `snprintf()` returns the number of characters that would have been written if the buffer was large enough. If the remaining buffer space is smaller than the formatted string, the return value will be greater than or equal to the remaining space. The code previously advanced `tmp_buf` by `err + 1`, which could push `tmp_buf` past `tmp_buf_end`.
Second, when handling standard format specifiers, the code aligns `tmp_buf` and checks if there is enough space using `tmp_buf_end - tmp_buf < sizeof_cur_arg`. Since `sizeof_cur_arg` is an unsigned `size_t`, the signed `ptrdiff_t` result of the pointer subtraction is implicitly cast to unsigned. If `tmp_buf` has already been advanced past `tmp_buf_end`, the negative difference becomes a huge positive number, bypassing the check and allowing further out-of-bounds writes.
Fix these issues by checking the return value of `snprintf()` for `%pB` and `%pI4`/`%pI6`. If it indicates truncation or an error, return `-ENOSPC` or `-EINVAL` respectively, ensuring the buffer is not overrun and `bstr_printf()` does not read malformed data. Additionally, rewrite the bounds checks to use pointer addition (`tmp_buf + sizeof_cur_arg > tmp_buf_end`), which eliminates the integer signedness bypass, and harden equality checks for buffer fullness from `tmp_buf_end == tmp_buf` to `tmp_buf >= tmp_buf_end`.
This prevents memory corruption that can lead to soft lockups and other crashes.
|
| 717/2 |
2026/05/18 21:04 |
llm |
description-generator |
0m
Model:
Tokens:
input: 10738
output: 471
thoughts: 1134
Thoughts: **My Analysis of the `bpf_bprintf_prepare()` Bug Fix**
Okay, so I've been looking at this patch for `bpf_bprintf_prepare()` and the issue is clear: we've got an out-of-bounds write vulnerability. The function itself is responsible for formatting arguments into a temporary 512-byte buffer. The problem? There are two separate bugs that allow `tmp_buf` to increment past `tmp_buf_end`, circumventing the buffer boundary checks. It's pretty nasty because this can lead to memory corruption, soft lockups, and other crashes.
First, when dealing with `%pB` and `%pI4`/`%pI6` format specifiers, we're using `snprintf()`. The thing is, `snprintf()` returns the total number of bytes that *would* have been written, if the buffer were large enough. If the buffer doesn't have enough space for the output string, the return value is greater than or equal to the space remaining. The original code advanced `tmp_buf` by `err + 1`, and this could potentially push `tmp_buf` beyond the buffer boundary, resulting in an out-of-bounds write.
The second bug surfaces when using standard format specifiers. The code aligns `tmp_buf` and then checks space availability using `tmp_buf_end - tmp_buf < sizeof_cur_arg`. Now, `sizeof_cur_arg` is an unsigned `size_t`, but the result of the pointer subtraction is a signed `ptrdiff_t`. This signed subtraction gets implicitly cast to unsigned. If `tmp_buf` has already advanced past `tmp_buf_end`, the subtraction results in a huge positive number, effectively bypassing the bounds check and allowing further out-of-bounds writes. This is a signedness issue.
To fix this, I'm proposing the following:
First, I'm going to check the return value of `snprintf()` when processing `%pB` and `%pI4`/`%pI6`. If it indicates truncation or an error, I'll return `-ENOSPC` or `-EINVAL` respectively. This ensures the buffer is not overrun and that `bstr_printf()` doesn't end up reading malformed data. That's a key fix to ensure correct operation.
Second, I need to rewrite the bounds checks to use pointer addition: `tmp_buf + sizeof_cur_arg > tmp_buf_end`. This eliminates the integer signedness bypass.
Finally, I'm going to harden the equality checks for buffer fullness, changing `tmp_buf_end == tmp_buf` to `tmp_buf >= tmp_buf_end`. This is a small but important hardening step.
These combined changes should prevent the memory corruption, leading to a much more robust and safer implementation. The patch is essential to prevent system instability.
|
| 718/1 |
2026/05/18 21:04 |
action |
format-patch-description |
0m
Results: map[PatchDescription:bpf: Fix out-of-bounds write in bpf_bprintf_prepare()
The `bpf_bprintf_prepare()` function formats arguments into a temporary
512-byte buffer. There are two distinct bugs in this function that allow
`tmp_buf` to advance past `tmp_buf_end`, bypassing bounds checks and
causing out-of-bounds memory writes.
First, when handling the `%pB` and `%pI4`/`%pI6` format specifiers,
`snprintf()` is used to format the string. `snprintf()` returns the
number of characters that would have been written if the buffer was
large enough. If the remaining buffer space is smaller than the
formatted string, the return value will be greater than or equal to the
remaining space. The code previously advanced `tmp_buf` by `err + 1`,
which could push `tmp_buf` past `tmp_buf_end`.
Second, when handling standard format specifiers, the code aligns
`tmp_buf` and checks if there is enough space using `tmp_buf_end -
tmp_buf < sizeof_cur_arg`. Since `sizeof_cur_arg` is an unsigned
`size_t`, the signed `ptrdiff_t` result of the pointer subtraction is
implicitly cast to unsigned. If `tmp_buf` has already been advanced past
`tmp_buf_end`, the negative difference becomes a huge positive number,
bypassing the check and allowing further out-of-bounds writes.
Fix these issues by checking the return value of `snprintf()` for `%pB`
and `%pI4`/`%pI6`. If it indicates truncation or an error, return
`-ENOSPC` or `-EINVAL` respectively, ensuring the buffer is not overrun
and `bstr_printf()` does not read malformed data. Additionally, rewrite
the bounds checks to use pointer addition (`tmp_buf + sizeof_cur_arg >
tmp_buf_end`), which eliminates the integer signedness bypass, and
harden equality checks for buffer fullness from `tmp_buf_end == tmp_buf`
to `tmp_buf >= tmp_buf_end`.
This prevents memory corruption that can lead to soft lockups and other
crashes.]
|
| 719/1 |
2026/05/18 21:04 |
action |
empty-tags |
0m
Results: map[AckedBy:<nil> ReportedBy:<nil> ReviewedBy:<nil> TestedBy:<nil>]
|