| 0/0 |
2026/05/17 00:43 |
flow |
assessment-security |
57m
Results: map[DenialOfService:true Explanation:Based on the analysis of the bug report and the kernel source code, here is the
assessment of the vulnerability:
### Bug Explanation
The bug is a **Use-After-Free (UAF)** in the TCP request socket timer handling,
specifically within `reqsk_timer_handler()` in
`net/ipv4/inet_connection_sock.c`.
When a TCP listener socket is closed but has pending incomplete connections
(request sockets), the kernel attempts to migrate these request sockets to
another listener socket in the same `SO_REUSEPORT` group.
During this migration:
1. The original request socket (`oreq`) is cloned into a new request socket
(`nreq`).
2. A timer is scheduled for the new request socket via
`mod_timer(&req->rsk_timer, ...)` (where `req` points to `nreq`).
3. The code attempts to replace `oreq` with `nreq` in the established hash table
by calling `inet_ehash_insert()`.
If `inet_ehash_insert()` fails—which can happen if another CPU concurrently
receives the final ACK from the client and removes `oreq` from the hash
table—the code enters an error path to clean up `nreq`.
It calls `__inet_csk_reqsk_queue_drop(sk_listener, nreq, true)`. The `true`
argument is passed for the `from_timer` parameter, which instructs the function
**not** to call `timer_delete_sync()`. The developer mistakenly assumed that the
timer being dropped was the currently running timer (which is
`oreq->rsk_timer`), but the timer that needs to be deleted is the newly
scheduled `nreq->rsk_timer`.
Because `timer_delete_sync()` is skipped, `nreq->rsk_timer` is left pending. The
error path then immediately frees `nreq` by calling `__reqsk_free(nreq)`. When
the pending timer eventually fires, it executes `reqsk_timer_handler()` on the
freed `nreq` object. The warning in the bug report (`refcount_t: underflow;
use-after-free.`) is triggered when the timer handler finishes and calls
`reqsk_put()` on the already freed memory.
### Security Impact
* **Exploitable:** **Yes.** The UAF involves `struct request_sock`, which
contains function pointers (e.g., `rsk_ops`) and pointers to other kernel
structures. An attacker who successfully reallocates the freed memory with
controlled data can hijack the kernel control flow to achieve privilege
escalation or arbitrary code execution.
* **Denial of Service:** **Yes.** The Use-After-Free leads to memory corruption,
kernel warnings, and ultimately kernel panics.
* **Accessible From Unprivileged Processes / User Namespaces:** **Yes.** An
unprivileged local user can easily trigger this by creating TCP sockets with
`SO_REUSEPORT`, initiating connections to them, and closing the listeners to
force the migration race condition.
* **Ethernet Network / Remote Trigger:** **Yes.** This vulnerability can be
triggered **remotely** against a server running a service that utilizes
`SO_REUSEPORT` (such as Nginx or HAProxy). If the service reloads its
configuration (closing old sockets and opening new ones), a remote attacker can
send a SYN packet to create a request socket, and then precisely time the final
ACK packet to win the race against the timer handler's `inet_ehash_insert()`,
triggering the UAF. Exploitable:true FilesystemTrigger:false NetworkTrigger:true PeripheralTrigger:false RemoteTrigger:true Unprivileged:true UserNamespace:true VMGuestTrigger:false VMHostTrigger:false]
|
| 1/1 |
2026/05/17 00:43 |
action |
syz-repro-to-c-repro |
0m
Results:
|
| 2/1 |
2026/05/17 00:43 |
action |
kernel-checkouter |
1m
Results: map[KernelSrc:/app/workdir/cache/src/a7966833164a791a1f7c3009b83c0bfc0425cdbf]
|
| 3/1 |
2026/05/17 00:45 |
action |
kernel-builder |
29m
Results: map[KernelObj:/app/workdir/cache/build/21437f87c14bc26dd41c81fab3d488a1d20698a4]
|
| 4/1 |
2026/05/17 01:15 |
action |
codesearch-prepare |
20m
Results: map[Index:codesearch-index]
|
| 5/1 |
2026/05/17 01:36 |
agent |
expert |
5m
Model:
Results: map[DenialOfService:true Exploitable:true FilesystemTrigger:false NetworkTrigger:true PeripheralTrigger:false RemoteTrigger:true Unprivileged:true UserNamespace:true VMGuestTrigger:false VMHostTrigger:false]
Instruction:
You are an experienced Linux kernel security engineer. Your task is to analyze given kernel bug report
and determine its security impact based on the following dimensions.
Use the provided tools to examine the source code, check for capability checks (e.g., capable(), ns_capable()),
and understand the nature of the bug. Analyze the given kernel build and configuration.
You can check the kernel config by grepping ".config" file; you can check kernel cmdline by greeping
".config" file for "CONFIG_CMDLINE=". Assume sysctl parameters have default values.
But analyze for the corresponding production build w/o debugging tools enabled (like KASAN, KMSAN, UBSAN).
Don't make assumptions; verify them with source code access. Try different strategies when analyzing the bug:
- think of ways in which the vulnerable code is unreachable
- or the other way around: try to come up with different ideas of how an unprivileged user can reach the bug
If still unsure err on the side of the bug being non-exploitable/not-accessible.
In the final reply, provide a reasoning for your assessment.
Analysis dimensions:
* Exploitable:
Determine if the bug can result in memory corruption or elevated privileges.
Memory safety issues are almost always exploitable (KASAN or UBSAN reports for use-after-free, out-of-bounds;
refcounting issues, corrupted lists, etc). When kernel is crashing on a completly wild pointer access
(e.g. user-space address, or non-canonical address, but not on NULL or address corresponding to KASAN shadow
for NULL address), including both data accesses and control tranfers, that's also usually implies possibility
of exploitation. Such reports usually say "unable to handle kernel paging request".
Uses of uninitialized values detected by KMSAN may be exploitable b/c attacker frequently can affect uninit
values with spraying techniques. However, for these exploitabability depends on how exactly the uninit value
is used in the code, and what it affects.
Think of what happens after the bug is triggered. Some bugs cause kernel panic and halt execution,
they are harder to exploit. For example, BUG reports halts the kernel. However, WARNING reports don't halt
execution in production builds. Debug bug detection tools (like KASAN, KMSAN, KCSAN, UBSAN) are also not enabled
in production builds, so attacker can freely exploit these bugs w/o being detected by these tools.
If you see an integer overflow, think how the overflowed value used later (if it's used as allocation size,
or an array index). If you see an out-of-bounds read, think if it's followed by an out-of-bounds write as well.
Some KCSAN data-races may be exploitable by skilled attackers as well. Think what data structures got corrupted
as the result of data races and how. However, note that kernel has lots of "benign" data races that don't lead
to any runtime misbehavior at all.
* Denial Of Service:
Determine if the bug can result in denial-of-service. Most bugs can, since they cause system crash,
hangs, deadlocks, or resource leaks. This is mostly applicable to WARNING bugs that won't cause system crash
in production. For these think what will be consequences of the violation of the kernel assumptions flagged
by the WARNING. In some cases the unexpected condition is also properly handled by the normal control flow
(e.g. with "if (WARN_ON(...))"), these won't cause denial-of-service. If the condition is not handled,
then it may or may not cause denial-of-service.
* Accessible From Unprivileged Processes:
Determine if the bug can be reached from a typical (non-root) user process that does NOT have any special capabilities
(like CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_NET_RAW, CAP_PERFMON) or access to device nodes restricted to root.
Assume that unprivileged_bpf_disabled=1, that is eBPF loading is not accessible. However, cBPF (classical BPF)
is still accessible to non-root processes.
Assume that user namespaces are not accessible, that is, the process cannot get the mentioned capabilities even
within a new user namespace (checked by ns_capable() function in the kernel sources).
* Accessible From User Namespaces:
Determine if the bug can be reached within a user-namespace where the process has all capabilities
(including CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_NET_RAW, CAP_PERFMON). Such capabilities are checked with ns_capable()
function in the kernel sources.
* VM Guest Trigger:
Determine if the bug can be triggered from the context of a typical KVM guest (e.g., set up by a QEMU VMM).
Consider accesses to standard Linux host paravirtualized features (virtio-blk, virtio-net, etc.),
and handling of VM exits in the KVM code.
* VM Host Trigger in The Confidetial Computing Context:
Determine if the bug can be triggered in a confidential computing guest kernel from the context of a KVM host.
Consider access to standard Linux guest paravirtualized features (virtio-blk, virtio-net, etc.).
* Ethernet Network Trigger:
Determine if the bug can be triggered by processing ingress network Ethernet traffic, either directly (network stack)
or via drivers exposed to network data.
* Other Remote Trigger:
Determine if the bug can be triggered by processing remote traffic other than Ethernet (Wifi, Bluetooth, NFC, etc).
* Peripheral Trigger:
Determine if the bug can be triggered via an untrusted peripheral device that can be physically plugged
into a system, such as a USB device or a niche hardware driver handling external hardware inputs.
This is particularly important for mobile and desktop environments where users can plug in unknown devices.
* Malicious Filesystem Trigger:
Determine if the bug can be triggered by the kernel mounting and parsing a malicious filesystem image.
This is highly critical for Desktop and Mobile environments where external media or downloaded images
might be auto-mounted.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt:
The kernel bug report is:
------------[ cut here ]------------
refcount_t: underflow; use-after-free.
WARNING: lib/refcount.c:28 at refcount_warn_saturate+0xb2/0x110 lib/refcount.c:28, CPU#0: ktimers/0/16
Modules linked in:
CPU: 0 UID: 0 PID: 16 Comm: ktimers/0 Not tainted syzkaller #0 PREEMPT_{RT,(full)}
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/18/2026
RIP: 0010:refcount_warn_saturate+0xb2/0x110 lib/refcount.c:28
Code: 64 e3 f1 0a 67 48 0f b9 3a eb 4a e8 f8 2b 23 fd 48 8d 3d 61 e3 f1 0a 67 48 0f b9 3a eb 37 e8 e5 2b 23 fd 48 8d 3d 5e e3 f1 0a <67> 48 0f b9 3a eb 24 e8 d2 2b 23 fd 48 8d 3d 5b e3 f1 0a 67 48 0f
RSP: 0018:ffffc90000157a28 EFLAGS: 00010246
RAX: ffffffff84a13cdb RBX: 0000000000000003 RCX: ffff88801cee0000
RDX: 0000000000000100 RSI: 0000000000000000 RDI: ffffffff8f932040
RBP: ffffc90000157b30 R08: ffff88801cee0000 R09: 0000000000000005
R10: 0000000000000100 R11: 0000000000000004 R12: 0000000000000000
R13: 00000000ffffd579 R14: ffff88805efae820 R15: ffff88805efae9b8
FS: 0000000000000000(0000) GS:ffff88812617d000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000200000404030 CR3: 0000000040e8c000 CR4: 00000000003526f0
Call Trace:
<TASK>
call_timer_fn+0x192/0x5e0 kernel/time/timer.c:1748
expire_timers kernel/time/timer.c:1799 [inline]
__run_timers kernel/time/timer.c:2374 [inline]
__run_timer_base+0x6a3/0x9f0 kernel/time/timer.c:2386
run_timer_base kernel/time/timer.c:2395 [inline]
run_timer_softirq+0xb7/0x170 kernel/time/timer.c:2405
handle_softirqs+0x1de/0x6d0 kernel/softirq.c:622
__do_softirq kernel/softirq.c:656 [inline]
run_ktimerd+0x69/0x100 kernel/softirq.c:1151
smpboot_thread_fn+0x541/0xa50 kernel/smpboot.c:160
kthread+0x388/0x470 kernel/kthread.c:436
ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
</TASK>
----------------
Code disassembly (best guess):
0: 64 e3 f1 fs jrcxz 0xfffffff4
3: 0a 67 48 or 0x48(%rdi),%ah
6: 0f b9 3a ud1 (%rdx),%edi
9: eb 4a jmp 0x55
b: e8 f8 2b 23 fd call 0xfd232c08
10: 48 8d 3d 61 e3 f1 0a lea 0xaf1e361(%rip),%rdi # 0xaf1e378
17: 67 48 0f b9 3a ud1 (%edx),%rdi
1c: eb 37 jmp 0x55
1e: e8 e5 2b 23 fd call 0xfd232c08
23: 48 8d 3d 5e e3 f1 0a lea 0xaf1e35e(%rip),%rdi # 0xaf1e388
* 2a: 67 48 0f b9 3a ud1 (%edx),%rdi <-- trapping instruction
2f: eb 24 jmp 0x55
31: e8 d2 2b 23 fd call 0xfd232c08
36: 48 8d 3d 5b e3 f1 0a lea 0xaf1e35b(%rip),%rdi # 0xaf1e398
3d: 67 addr32
3e: 48 rex.W
3f: 0f .byte 0xf
Reply: Based on the analysis of the bug report and the kernel source code, here is the assessment of the vulnerability:
### Bug Explanation
The bug is a **Use-After-Free (UAF)** in the TCP request socket timer handling, specifically within `reqsk_timer_handler()` in `net/ipv4/inet_connection_sock.c`.
When a TCP listener socket is closed but has pending incomplete connections (request sockets), the kernel attempts to migrate these request sockets to another listener socket in the same `SO_REUSEPORT` group.
During this migration:
1. The original request socket (`oreq`) is cloned into a new request socket (`nreq`).
2. A timer is scheduled for the new request socket via `mod_timer(&req->rsk_timer, ...)` (where `req` points to `nreq`).
3. The code attempts to replace `oreq` with `nreq` in the established hash table by calling `inet_ehash_insert()`.
If `inet_ehash_insert()` fails—which can happen if another CPU concurrently receives the final ACK from the client and removes `oreq` from the hash table—the code enters an error path to clean up `nreq`.
It calls `__inet_csk_reqsk_queue_drop(sk_listener, nreq, true)`. The `true` argument is passed for the `from_timer` parameter, which instructs the function **not** to call `timer_delete_sync()`. The developer mistakenly assumed that the timer being dropped was the currently running timer (which is `oreq->rsk_timer`), but the timer that needs to be deleted is the newly scheduled `nreq->rsk_timer`.
Because `timer_delete_sync()` is skipped, `nreq->rsk_timer` is left pending. The error path then immediately frees `nreq` by calling `__reqsk_free(nreq)`. When the pending timer eventually fires, it executes `reqsk_timer_handler()` on the freed `nreq` object. The warning in the bug report (`refcount_t: underflow; use-after-free.`) is triggered when the timer handler finishes and calls `reqsk_put()` on the already freed memory.
### Security Impact
* **Exploitable:** **Yes.** The UAF involves `struct request_sock`, which contains function pointers (e.g., `rsk_ops`) and pointers to other kernel structures. An attacker who successfully reallocates the freed memory with controlled data can hijack the kernel control flow to achieve privilege escalation or arbitrary code execution.
* **Denial of Service:** **Yes.** The Use-After-Free leads to memory corruption, kernel warnings, and ultimately kernel panics.
* **Accessible From Unprivileged Processes / User Namespaces:** **Yes.** An unprivileged local user can easily trigger this by creating TCP sockets with `SO_REUSEPORT`, initiating connections to them, and closing the listeners to force the migration race condition.
* **Ethernet Network / Remote Trigger:** **Yes.** This vulnerability can be triggered **remotely** against a server running a service that utilizes `SO_REUSEPORT` (such as Nginx or HAProxy). If the service reloads its configuration (closing old sockets and opening new ones), a remote attacker can send a SYN packet to create a request socket, and then precisely time the final ACK packet to win the race against the timer handler's `inet_ehash_insert()`, triggering the UAF.
|
| 6/2 |
2026/05/17 01:36 |
llm |
expert |
0m
Model:
Tokens:
input: 6054
output: 20
thoughts: 2733
Thoughts: **Bug Analysis: refcount_t Underflow in Timer Callback**
Okay, so we've got a `refcount_t` underflow, use-after-free situation, and it's triggered within a timer callback. The key here is that the stack trace points to `call_timer_fn`, but the function that *actually* decrements the refcount and causes the warning isn't directly in the trace. That's immediately suspicious. My initial thought is it's an inlined function, which explains the missing entry.
I start by checking the code at the reported line. Right, `refcount_warn_saturate` is the culprit. This function's definitely not inlined. The warning suggests it's being called from `refcount_dec_and_test` or something similar, and *that's* inline. But `refcount_dec_and_test` itself doesn't cause a direct tail call. My intuition says there is an *inline* function that causes the underflow warning, being invoked from the timer callback.
The trace suggests it is either:
1. That the timer callback is inlined, or
2. Tail call optimization.
Since the trace is truncated and the code runs under `PREEMPT_RT`, unwinding might have been difficult. Let's dig deeper: I need to figure out *which* timer callback is being run. I start tracing how timer callbacks are called. The disassembly of `call_timer_fn` shows `fn(timer)` is being executed. The thing is, `fn` is a function pointer.
The crucial piece of the puzzle is that the warning *originates* from `refcount_warn_saturate`, but the *caller* isn't in the trace. This strongly suggests a function that *ends* with a call to `refcount_dec`. If the function simply calls `refcount_dec`, *that* could be tail-called.
Now, I'm thinking, is `fn` a BPF program, or even a module function? The "Modules linked in" section is empty, so that's a dead end. Since `fn` is missing from the trace, it means `fn` jumps to `refcount_warn_saturate` directly.
`fn` is likely a timer callback that *ends* with a refcount decrement (`refcount_dec`). If so, then the compiler optimizes the entire function.
I need to find those timer callbacks. So, I will focus on finding timer callbacks that have `refcount_dec` or similar functions at their end.
|
| 7/2 |
2026/05/17 01:36 |
tool |
grepper |
0m
Args: map[Expression:timer_setup.*,.*]
Results: map[Output:
Full output is too long, showing 500 out of 6338 lines.
Use more precise expression if possible.
[Documentation/core-api/local_ops.rst=136=Here is a sample module which implements a basic per cpu counter using
--
Documentation/core-api/local_ops.rst-185- /* initialize the timer that will increment the counter */
Documentation/core-api/local_ops.rst:186: timer_setup(&test_timer, do_test_timer, 0);
Documentation/core-api/local_ops.rst-187- mod_timer(&test_timer, jiffies + 1);
--
Documentation/translations/zh_CN/core-api/local_ops.rst=51=UP之间没有不同的行为，在你的架构的 ``local.h`` 中包括 ``asm-generic/local.h``
--
Documentation/translations/zh_CN/core-api/local_ops.rst-179- /* initialize the timer that will increment the counter */
Documentation/translations/zh_CN/core-api/local_ops.rst:180: timer_setup(&test_timer, do_test_timer, 0);
Documentation/translations/zh_CN/core-api/local_ops.rst-181- mod_timer(&test_timer, jiffies + 1);
--
arch/alpha/kernel/srmcons.c=198=srmcons_init(void)
--
arch/alpha/kernel/srmcons.c-202-
arch/alpha/kernel/srmcons.c:203: timer_setup(&srmcons_singleton.timer, srmcons_receive_chars, 0);
arch/alpha/kernel/srmcons.c-204-
--
arch/arm/mach-footbridge/dc21285.c=294=void __init dc21285_preinit(void)
--
arch/arm/mach-footbridge/dc21285.c-325-
arch/arm/mach-footbridge/dc21285.c:326: timer_setup(&serr_timer, dc21285_enable_error, 0);
arch/arm/mach-footbridge/dc21285.c:327: timer_setup(&perr_timer, dc21285_enable_error, 0);
arch/arm/mach-footbridge/dc21285.c-328-
--
arch/arm/mach-imx/mmdc.c=473=static int imx_mmdc_perf_init(struct platform_device *pdev, void __iomem *mmdc_base,
--
arch/arm/mach-imx/mmdc.c-511-
arch/arm/mach-imx/mmdc.c:512: hrtimer_setup(&pmu_mmdc->hrtimer, mmdc_pmu_timer_handler, CLOCK_MONOTONIC,
arch/arm/mach-imx/mmdc.c-513- HRTIMER_MODE_REL);
--
arch/arm/mach-pxa/sharpsl_pm.c=822=static int sharpsl_pm_probe(struct platform_device *pdev)
--
arch/arm/mach-pxa/sharpsl_pm.c-833-
arch/arm/mach-pxa/sharpsl_pm.c:834: timer_setup(&sharpsl_pm.ac_timer, sharpsl_ac_timer, 0);
arch/arm/mach-pxa/sharpsl_pm.c-835-
arch/arm/mach-pxa/sharpsl_pm.c:836: timer_setup(&sharpsl_pm.chrg_full_timer, sharpsl_chrg_full_timer, 0);
arch/arm/mach-pxa/sharpsl_pm.c-837-
--
arch/arm/mm/cache-l2x0-pmu.c=503=static __init int l2x0_pmu_init(void)
--
arch/arm/mm/cache-l2x0-pmu.c-541- l2x0_pmu_poll_period = ms_to_ktime(1000);
arch/arm/mm/cache-l2x0-pmu.c:542: hrtimer_setup(&l2x0_pmu_hrtimer, l2x0_pmu_poll, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
arch/arm/mm/cache-l2x0-pmu.c-543-
--
arch/arm64/kvm/arch_timer.c=1075=static void timer_context_init(struct kvm_vcpu *vcpu, int timerid)
--
arch/arm64/kvm/arch_timer.c-1090-
arch/arm64/kvm/arch_timer.c:1091: hrtimer_setup(&ctxt->hrtimer, kvm_hrtimer_expire, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD);
arch/arm64/kvm/arch_timer.c-1092-
--
arch/arm64/kvm/arch_timer.c=1105=void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu)
--
arch/arm64/kvm/arch_timer.c-1118-
arch/arm64/kvm/arch_timer.c:1119: hrtimer_setup(&timer->bg_timer, kvm_bg_timer_expire, CLOCK_MONOTONIC,
arch/arm64/kvm/arch_timer.c-1120- HRTIMER_MODE_ABS_HARD);
--
arch/loongarch/kvm/vcpu.c=1530=int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
--
arch/loongarch/kvm/vcpu.c-1537-
arch/loongarch/kvm/vcpu.c:1538: hrtimer_setup(&vcpu->arch.swtimer, kvm_swtimer_wakeup, CLOCK_MONOTONIC,
arch/loongarch/kvm/vcpu.c-1539- HRTIMER_MODE_ABS_PINNED_HARD);
--
arch/mips/kvm/mips.c=278=int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
--
arch/mips/kvm/mips.c-290-
arch/mips/kvm/mips.c:291: hrtimer_setup(&vcpu->arch.comparecount_timer, kvm_mips_comparecount_wakeup, CLOCK_MONOTONIC,
arch/mips/kvm/mips.c-292- HRTIMER_MODE_REL);
--
arch/mips/sgi-ip22/ip22-reset.c=120=static inline void power_button(void)
--
arch/mips/sgi-ip22/ip22-reset.c-134-
arch/mips/sgi-ip22/ip22-reset.c:135: timer_setup(&power_timer, power_timeout, 0);
arch/mips/sgi-ip22/ip22-reset.c-136- power_timer.expires = jiffies + POWERDOWN_TIMEOUT * HZ;
--
arch/mips/sgi-ip22/ip22-reset.c=140=static irqreturn_t panel_int(int irq, void *dev_id)
--
arch/mips/sgi-ip22/ip22-reset.c-149- disable_irq_nosync(SGI_PANEL_IRQ);
arch/mips/sgi-ip22/ip22-reset.c:150: timer_setup(&debounce_timer, debounce, 0);
arch/mips/sgi-ip22/ip22-reset.c-151- debounce_timer.expires = jiffies + 5;
--
arch/mips/sgi-ip22/ip22-reset.c=183=static int __init reboot_setup(void)
--
arch/mips/sgi-ip22/ip22-reset.c-196-
arch/mips/sgi-ip22/ip22-reset.c:197: timer_setup(&blink_timer, blink_timeout, 0);
arch/mips/sgi-ip22/ip22-reset.c-198- atomic_notifier_chain_register(&panic_notifier_list, &panic_block);
--
arch/mips/sgi-ip32/ip32-reset.c=95=void ip32_prepare_poweroff(void)
--
arch/mips/sgi-ip32/ip32-reset.c-108-
arch/mips/sgi-ip32/ip32-reset.c:109: timer_setup(&power_timer, power_timeout, 0);
arch/mips/sgi-ip32/ip32-reset.c-110- power_timer.expires = jiffies + POWERDOWN_TIMEOUT * HZ;
--
arch/mips/sgi-ip32/ip32-reset.c=137=static __init int ip32_reboot_setup(void)
--
arch/mips/sgi-ip32/ip32-reset.c-148-
arch/mips/sgi-ip32/ip32-reset.c:149: timer_setup(&blink_timer, blink_timeout, 0);
arch/mips/sgi-ip32/ip32-reset.c-150- atomic_notifier_chain_register(&panic_notifier_list, &panic_block);
--
arch/powerpc/kernel/watchdog.c=476=static void start_watchdog(void *arg)
--
arch/powerpc/kernel/watchdog.c-502-
arch/powerpc/kernel/watchdog.c:503: hrtimer_setup(hrtimer, watchdog_timer_fn, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
arch/powerpc/kernel/watchdog.c-504- hrtimer_start(hrtimer, ms_to_ktime(wd_timer_period_ms),
--
arch/powerpc/kvm/booke.c=1428=int kvmppc_subarch_vcpu_init(struct kvm_vcpu *vcpu)
--
arch/powerpc/kvm/booke.c-1431- spin_lock_init(&vcpu->arch.wdt_lock);
arch/powerpc/kvm/booke.c:1432: timer_setup(&vcpu->arch.wdt_timer, kvmppc_watchdog_func, 0);
arch/powerpc/kvm/booke.c-1433-
--
arch/powerpc/kvm/powerpc.c=756=int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
--
arch/powerpc/kvm/powerpc.c-759-
arch/powerpc/kvm/powerpc.c:760: hrtimer_setup(&vcpu->arch.dec_timer, kvmppc_decrementer_wakeup, CLOCK_REALTIME,
arch/powerpc/kvm/powerpc.c-761- HRTIMER_MODE_ABS);
--
arch/powerpc/mm/book3s64/hash_utils.c=1309=static void __init htab_initialize(void)
--
arch/powerpc/mm/book3s64/hash_utils.c-1344-
arch/powerpc/mm/book3s64/hash_utils.c:1345: timer_setup(&stress_hpt_timer, stress_hpt_timer_fn, 0);
arch/powerpc/mm/book3s64/hash_utils.c-1346- stress_hpt_timer.expires = jiffies + msecs_to_jiffies(10);
--
arch/powerpc/perf/vpa-dtl.c=350=static int vpa_dtl_event_init(struct perf_event *event)
--
arch/powerpc/perf/vpa-dtl.c-411-
arch/powerpc/perf/vpa-dtl.c:412: hrtimer_setup(&hwc->hrtimer, vpa_dtl_hrtimer_handle, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
arch/powerpc/perf/vpa-dtl.c-413-
--
arch/powerpc/platforms/cell/spufs/sched.c=1080=int __init spu_sched_init(void)
--
arch/powerpc/platforms/cell/spufs/sched.c-1094-
arch/powerpc/platforms/cell/spufs/sched.c:1095: timer_setup(&spusched_timer, spusched_wake, 0);
arch/powerpc/platforms/cell/spufs/sched.c:1096: timer_setup(&spuloadavg_timer, spuloadavg_wake, 0);
arch/powerpc/platforms/cell/spufs/sched.c-1097-
--
arch/powerpc/platforms/powermac/low_i2c.c=486=static struct pmac_i2c_host_kw *__init kw_i2c_host_init(struct device_node *np)
--
arch/powerpc/platforms/powermac/low_i2c.c-512- spin_lock_init(&host->lock);
arch/powerpc/platforms/powermac/low_i2c.c:513: timer_setup(&host->timeout_timer, kw_i2c_timeout, 0);
arch/powerpc/platforms/powermac/low_i2c.c-514-
--
arch/riscv/kvm/vcpu_timer.c=246=int kvm_riscv_vcpu_timer_init(struct kvm_vcpu *vcpu)
--
arch/riscv/kvm/vcpu_timer.c-258- t->sstc_enabled = true;
arch/riscv/kvm/vcpu_timer.c:259: hrtimer_setup(&t->hrt, kvm_riscv_vcpu_vstimer_expired, CLOCK_MONOTONIC,
arch/riscv/kvm/vcpu_timer.c-260- HRTIMER_MODE_REL);
--
arch/riscv/kvm/vcpu_timer.c-263- t->sstc_enabled = false;
arch/riscv/kvm/vcpu_timer.c:264: hrtimer_setup(&t->hrt, kvm_riscv_vcpu_hrtimer_expired, CLOCK_MONOTONIC,
arch/riscv/kvm/vcpu_timer.c-265- HRTIMER_MODE_REL);
--
arch/s390/kernel/lgr.c=175=static int __init lgr_init(void)
--
arch/s390/kernel/lgr.c-182- debug_event(lgr_dbf, 1, &lgr_info_last, sizeof(lgr_info_last));
arch/s390/kernel/lgr.c:183: timer_setup(&lgr_timer, lgr_timer_fn, TIMER_DEFERRABLE);
arch/s390/kernel/lgr.c-184- lgr_timer_set();
--
arch/s390/kernel/time.c=442=static int __init stp_init(void)
--
arch/s390/kernel/time.c-445- return 0;
arch/s390/kernel/time.c:446: timer_setup(&stp_timer, stp_timeout, 0);
arch/s390/kernel/time.c-447- time_init_wq();
--
arch/s390/kernel/topology.c=677=static int __init topology_init(void)
--
arch/s390/kernel/topology.c-681-
arch/s390/kernel/topology.c:682: timer_setup(&topology_timer, topology_timer_fn, TIMER_DEFERRABLE);
arch/s390/kernel/topology.c-683- if (cpu_has_topology())
--
arch/s390/kvm/interrupt.c=3149=void kvm_s390_gisa_init(struct kvm *kvm)
--
arch/s390/kvm/interrupt.c-3158- gi->expires = 50 * 1000; /* 50 usec */
arch/s390/kvm/interrupt.c:3159: hrtimer_setup(&gi->timer, gisa_vcpu_kicker, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
arch/s390/kvm/interrupt.c-3160- memset(gi->origin, 0, sizeof(struct kvm_s390_gisa));
--
arch/s390/kvm/kvm-s390.c=3611=static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
--
arch/s390/kvm/kvm-s390.c-3677- }
arch/s390/kvm/kvm-s390.c:3678: hrtimer_setup(&vcpu->arch.ckc_timer, kvm_s390_idle_wakeup, CLOCK_MONOTONIC,
arch/s390/kvm/kvm-s390.c-3679- HRTIMER_MODE_REL);
--
arch/s390/lib/test_unwind.c=389=static int test_unwind_irq(struct unwindme *u)
--
arch/s390/lib/test_unwind.c-392- init_completion(&u->task_ready);
arch/s390/lib/test_unwind.c:393: timer_setup(&unwind_timer, unwindme_timer_fn, 0);
arch/s390/lib/test_unwind.c-394- mod_timer(&unwind_timer, jiffies + 1);
--
arch/sh/drivers/heartbeat.c=74=static int heartbeat_drv_probe(struct platform_device *pdev)
--
arch/sh/drivers/heartbeat.c-132-
arch/sh/drivers/heartbeat.c:133: timer_setup(&hd->timer, heartbeat_timer, 0);
arch/sh/drivers/heartbeat.c-134- platform_set_drvdata(pdev, hd);
--
arch/sh/drivers/pci/common.c=107=void pcibios_enable_timers(struct pci_channel *hose)
--
arch/sh/drivers/pci/common.c-109- if (hose->err_irq) {
arch/sh/drivers/pci/common.c:110: timer_setup(&hose->err_timer, pcibios_enable_err, 0);
arch/sh/drivers/pci/common.c-111- }
--
arch/sh/drivers/pci/common.c-113- if (hose->serr_irq) {
arch/sh/drivers/pci/common.c:114: timer_setup(&hose->serr_timer, pcibios_enable_serr, 0);
arch/sh/drivers/pci/common.c-115- }
--
arch/sh/drivers/push-switch.c=43=static int switch_drv_probe(struct platform_device *pdev)
--
arch/sh/drivers/push-switch.c-77- INIT_WORK(&psw->work, switch_work_handler);
arch/sh/drivers/push-switch.c:78: timer_setup(&psw->debounce, switch_timer, 0);
arch/sh/drivers/push-switch.c-79-
--
arch/sparc/kernel/led.c=119=static int __init led_init(void)
arch/sparc/kernel/led.c-120-{
arch/sparc/kernel/led.c:121: timer_setup(&led_blink_timer, led_blink, 0);
arch/sparc/kernel/led.c-122-
--
arch/sparc/kernel/viohs.c=812=int vio_driver_init(struct vio_driver_state *vio, struct vio_dev *vdev,
--
arch/sparc/kernel/viohs.c-855-
arch/sparc/kernel/viohs.c:856: timer_setup(&vio->timer, vio_port_timer, 0);
arch/sparc/kernel/viohs.c-857-
--
arch/um/drivers/vector_kern.c=1574=static void vector_eth_configure(
--
arch/um/drivers/vector_kern.c-1639-
arch/um/drivers/vector_kern.c:1640: timer_setup(&vp->tl, vector_timer_expire, 0);
arch/um/drivers/vector_kern.c-1641-
--
arch/x86/events/amd/uncore.c=130=static void amd_uncore_init_hrtimer(struct amd_uncore_ctx *ctx)
arch/x86/events/amd/uncore.c-131-{
arch/x86/events/amd/uncore.c:132: hrtimer_setup(&ctx->hrtimer, amd_uncore_hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD);
arch/x86/events/amd/uncore.c-133-}
--
arch/x86/events/intel/uncore.c=342=static void uncore_pmu_init_hrtimer(struct intel_uncore_box *box)
arch/x86/events/intel/uncore.c-343-{
arch/x86/events/intel/uncore.c:344: hrtimer_setup(&box->hrtimer, uncore_pmu_hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD);
arch/x86/events/intel/uncore.c-345-}
--
arch/x86/events/rapl.c=274=static void rapl_hrtimer_init(struct rapl_pmu *rapl_pmu)
--
arch/x86/events/rapl.c-277-
arch/x86/events/rapl.c:278: hrtimer_setup(hr, rapl_hrtimer_handle, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
arch/x86/events/rapl.c-279-}
--
arch/x86/kernel/cpu/mce/core.c=2109=static void __mcheck_cpu_setup_timer(void)
--
arch/x86/kernel/cpu/mce/core.c-2112-
arch/x86/kernel/cpu/mce/core.c:2113: timer_setup(t, mce_timer_fn, TIMER_PINNED);
arch/x86/kernel/cpu/mce/core.c-2114-}
--
arch/x86/kernel/cpu/mce/core.c=2116=static void __mcheck_cpu_init_timer(void)
--
arch/x86/kernel/cpu/mce/core.c-2119-
arch/x86/kernel/cpu/mce/core.c:2120: timer_setup(t, mce_timer_fn, TIMER_PINNED);
arch/x86/kernel/cpu/mce/core.c-2121- mce_start_timer(t);
--
arch/x86/kernel/tsc_sync.c=110=static int __init start_sync_check_timer(void)
--
arch/x86/kernel/tsc_sync.c-114-
arch/x86/kernel/tsc_sync.c:115: timer_setup(&tsc_sync_check_timer, tsc_sync_check_timer_fn, 0);
arch/x86/kernel/tsc_sync.c-116- tsc_sync_check_timer.expires = jiffies + SYNC_CHECK_INTERVAL;
--
arch/x86/kvm/hyperv.c=955=static void stimer_init(struct kvm_vcpu_hv_stimer *stimer, int timer_index)
--
arch/x86/kvm/hyperv.c-958- stimer->index = timer_index;
arch/x86/kvm/hyperv.c:959: hrtimer_setup(&stimer->timer, stimer_timer_callback, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
arch/x86/kvm/hyperv.c-960- stimer_prepare_msg(stimer);
--
arch/x86/kvm/i8254.c=735=struct kvm_pit *kvm_create_pit(struct kvm *kvm, u32 flags)
--
arch/x86/kvm/i8254.c-761- pit_state = &pit->pit_state;
arch/x86/kvm/i8254.c:762: hrtimer_setup(&pit_state->timer, pit_timer_fn, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
arch/x86/kvm/i8254.c-763-
--
arch/x86/kvm/lapic.c=3055=int kvm_create_lapic(struct kvm_vcpu *vcpu)
--
arch/x86/kvm/lapic.c-3082-
arch/x86/kvm/lapic.c:3083: hrtimer_setup(&apic->lapic_timer.timer, apic_timer_fn, CLOCK_MONOTONIC,
arch/x86/kvm/lapic.c-3084- HRTIMER_MODE_ABS_HARD);
--
arch/x86/kvm/vmx/nested.c=5394=static int enter_vmx_operation(struct kvm_vcpu *vcpu)
--
arch/x86/kvm/vmx/nested.c-5414-
arch/x86/kvm/vmx/nested.c:5415: hrtimer_setup(&vmx->nested.preemption_timer, vmx_preemption_timer_fn, CLOCK_MONOTONIC,
arch/x86/kvm/vmx/nested.c-5416- HRTIMER_MODE_ABS_PINNED);
--
arch/x86/kvm/xen.c=2297=void kvm_xen_init_vcpu(struct kvm_vcpu *vcpu)
--
arch/x86/kvm/xen.c-2301-
arch/x86/kvm/xen.c:2302: timer_setup(&vcpu->arch.xen.poll_timer, cancel_evtchn_poll, 0);
arch/x86/kvm/xen.c:2303: hrtimer_setup(&vcpu->arch.xen.timer, xen_timer_callback, CLOCK_MONOTONIC,
arch/x86/kvm/xen.c-2304- HRTIMER_MODE_ABS_HARD);
--
arch/xtensa/platforms/iss/network.c=349=static int iss_net_open(struct net_device *dev)
--
arch/xtensa/platforms/iss/network.c-366-
arch/xtensa/platforms/iss/network.c:367: timer_setup(&lp->timer, iss_net_timer, 0);
arch/xtensa/platforms/iss/network.c-368- lp->timer_val = ISS_NET_TIMER_VALUE;
--
arch/xtensa/platforms/iss/network.c=479=static void iss_net_configure(int index, char *init)
--
arch/xtensa/platforms/iss/network.c-548-
arch/xtensa/platforms/iss/network.c:549: timer_setup(&lp->tl, iss_net_user_timer_expire, 0);
arch/xtensa/platforms/iss/network.c-550-
--
block/bfq-iosched.c=7195=static int bfq_init_queue(struct request_queue *q, struct elevator_queue *eq)
--
block/bfq-iosched.c-7271-
block/bfq-iosched.c:7272: hrtimer_setup(&bfqd->idle_slice_timer, bfq_idle_slice_timer, CLOCK_MONOTONIC,
block/bfq-iosched.c-7273- HRTIMER_MODE_REL);
--
block/blk-core.c=393=struct request_queue *blk_alloc_queue(struct queue_limits *lim, int node_id)
--
block/blk-core.c-425-
block/blk-core.c:426: timer_setup(&q->timeout, blk_rq_timed_out_timer, 0);
block/blk-core.c-427- INIT_WORK(&q->timeout_work, blk_timeout_work);
--
block/blk-iocost.c=2891=static int blk_iocost_init(struct gendisk *disk)
--
block/blk-iocost.c-2916- spin_lock_init(&ioc->lock);
block/blk-iocost.c:2917: timer_setup(&ioc->timer, ioc_timer_fn, 0);
block/blk-iocost.c-2918- INIT_LIST_HEAD(&ioc->active_iocgs);
--
block/blk-iocost.c=2993=static void ioc_pd_init(struct blkg_policy_data *pd)
--
block/blk-iocost.c-3014- init_waitqueue_head(&iocg->waitq);
block/blk-iocost.c:3015: hrtimer_setup(&iocg->waitq_timer, iocg_waitq_timer_fn, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
block/blk-iocost.c-3016-
--
block/blk-iolatency.c=758=static int blk_iolatency_init(struct gendisk *disk)
--
block/blk-iolatency.c-774-
block/blk-iolatency.c:775: timer_setup(&blkiolat->timer, blkiolatency_timer_fn, 0);
block/blk-iolatency.c-776- INIT_WORK(&blkiolat->enable_work, blkiolatency_enable_work_fn);
--
block/blk-stat.c=100=blk_stat_alloc_callback(void (*timer_fn)(struct blk_stat_callback *),
--
block/blk-stat.c-126- cb->buckets = buckets;
block/blk-stat.c:127: timer_setup(&cb->timer, blk_stat_timer_fn, 0);
block/blk-stat.c-128-
--
block/blk-throttle.c=253=static void throtl_service_queue_init(struct throtl_service_queue *sq)
--
block/blk-throttle.c-257- sq->pending_tree = RB_ROOT_CACHED;
block/blk-throttle.c:258: timer_setup(&sq->pending_timer, throtl_pending_timer_fn, 0);
block/blk-throttle.c-259-}
--
block/kyber-iosched.c=350=static struct kyber_queue_data *kyber_queue_data_alloc(struct request_queue *q)
--
block/kyber-iosched.c-367-
block/kyber-iosched.c:368: timer_setup(&kqd->timer, kyber_timer_fn, 0);
block/kyber-iosched.c-369-
--
drivers/accel/qaic/qaic_timesync.c=181=static int qaic_timesync_probe(struct mhi_device *mhi_dev, const struct mhi_device_id *id)
--
drivers/accel/qaic/qaic_timesync.c-211- mqtsdev->qtimer_addr = qdev->bar_mhi + QTIMER_REG_OFFSET;
drivers/accel/qaic/qaic_timesync.c:212: timer_setup(timer, qaic_timesync_timer, 0);
drivers/accel/qaic/qaic_timesync.c-213- timer->expires = jiffies + msecs_to_jiffies(timesync_delay_ms);
--
drivers/acpi/apei/ghes.c=1663=static int ghes_probe(struct platform_device *ghes_dev)
--
drivers/acpi/apei/ghes.c-1730- case ACPI_HEST_NOTIFY_POLLED:
drivers/acpi/apei/ghes.c:1731: timer_setup(&ghes->timer, ghes_poll_func, 0);
drivers/acpi/apei/ghes.c-1732- ghes_add_timer(ghes);
--
drivers/ata/libahci.c=1080=static void ahci_init_sw_activity(struct ata_link *link)
--
drivers/ata/libahci.c-1088- emp->link = link;
drivers/ata/libahci.c:1089: timer_setup(&emp->timer, ahci_sw_activity_blink, 0);
drivers/ata/libahci.c-1090-
--
drivers/ata/libata-core.c=5646=struct ata_port *ata_port_alloc(struct ata_host *host)
--
drivers/ata/libata-core.c-5672- init_completion(&ap->park_req_pending);
drivers/ata/libata-core.c:5673: timer_setup(&ap->fastdrain_timer, ata_eh_fastdrain_timerfn,
drivers/ata/libata-core.c-5674- TIMER_DEFERRABLE);
--
drivers/ata/pata_octeon_cf.c=802=static int octeon_cf_probe(struct platform_device *pdev)
--
drivers/ata/pata_octeon_cf.c-937- /* True IDE mode needs a timer to poll for not-busy. */
drivers/ata/pata_octeon_cf.c:938: hrtimer_setup(&cf_port->delayed_finish, octeon_cf_delayed_finish, CLOCK_MONOTONIC,
drivers/ata/pata_octeon_cf.c-939- HRTIMER_MODE_REL);
--
drivers/auxdisplay/line-display.c=438=int linedisp_attach(struct linedisp *linedisp, struct device *dev,
--
drivers/auxdisplay/line-display.c-457- /* initialise a timer for scrolling the message */
drivers/auxdisplay/line-display.c:458: timer_setup(&linedisp->timer, linedisp_scroll, 0);
drivers/auxdisplay/line-display.c-459-
--
drivers/auxdisplay/line-display.c=523=int linedisp_register(struct linedisp *linedisp, struct device *parent,
--
drivers/auxdisplay/line-display.c-553- /* initialise a timer for scrolling the message */
drivers/auxdisplay/line-display.c:554: timer_setup(&linedisp->timer, linedisp_scroll, 0);
drivers/auxdisplay/line-display.c-555-
--
drivers/auxdisplay/panel.c=1356=static void init_scan_timer(void)
--
drivers/auxdisplay/panel.c-1360-
drivers/auxdisplay/panel.c:1361: timer_setup(&scan_timer, panel_scan_timer, 0);
drivers/auxdisplay/panel.c-1362- scan_timer.expires = jiffies + INPUT_POLL_TIME;
--
drivers/base/power/main.c=569=static void dpm_watchdog_set(struct dpm_watchdog *wd, struct device *dev)
--
drivers/base/power/main.c-576-
drivers/base/power/main.c:577: timer_setup_on_stack(timer, dpm_watchdog_handler, 0);
drivers/base/power/main.c-578- /* use same timeout value for both suspend and resume */
--
drivers/base/power/runtime.c=1837=void pm_runtime_init(struct device *dev)
--
drivers/base/power/runtime.c-1858- dev->power.timer_expires = 0;
drivers/base/power/runtime.c:1859: hrtimer_setup(&dev->power.suspend_timer, pm_suspend_timer_fn, CLOCK_MONOTONIC,
drivers/base/power/runtime.c-1860- HRTIMER_MODE_ABS);
--
drivers/base/power/wakeup.c=165=static void wakeup_source_add(struct wakeup_source *ws)
--
drivers/base/power/wakeup.c-172- spin_lock_init(&ws->lock);
drivers/base/power/wakeup.c:173: timer_setup(&ws->timer, pm_wakeup_timer_fn, 0);
drivers/base/power/wakeup.c-174- ws->active = false;
--
drivers/block/amiflop.c=1869=static int __init amiga_floppy_probe(struct platform_device *pdev)
--
drivers/block/amiflop.c-1898- /* initialize variables */
drivers/block/amiflop.c:1899: timer_setup(&motor_on_timer, motor_on_callback, 0);
drivers/block/amiflop.c-1900- motor_on_timer.expires = 0;
drivers/block/amiflop.c-1901- for (i = 0; i < FD_MAX_UNITS; i++) {
drivers/block/amiflop.c:1902: timer_setup(&motor_off_timer[i], fd_motor_off, 0);
drivers/block/amiflop.c-1903- motor_off_timer[i].expires = 0;
drivers/block/amiflop.c:1904: timer_setup(&flush_track_timer[i], flush_track_callback, 0);
drivers/block/amiflop.c-1905- flush_track_timer[i].expires = 0;
--
drivers/block/amiflop.c-1909-
drivers/block/amiflop.c:1910: timer_setup(&post_write_timer, post_write_callback, 0);
drivers/block/amiflop.c-1911- post_write_timer.expires = 0;
--
drivers/block/aoe/aoedev.c=451=aoedev_by_aoeaddr(ulong maj, int min, int do_alloc)
--
drivers/block/aoe/aoedev.c-487- skb_queue_head_init(&d->skbpool);
drivers/block/aoe/aoedev.c:488: timer_setup(&d->timer, dummy_timer, 0);
drivers/block/aoe/aoedev.c-489- d->timer.expires = jiffies + HZ;
--
drivers/block/aoe/aoemain.c=43=aoe_init(void)
--
drivers/block/aoe/aoemain.c-72-
drivers/block/aoe/aoemain.c:73: timer_setup(&timer, discover_timer, 0);
drivers/block/aoe/aoemain.c-74- discover_timer(&timer);
--
drivers/block/drbd/drbd_main.c=1936=void drbd_init_set_defaults(struct drbd_device *device)
--
drivers/block/drbd/drbd_main.c-1977-
drivers/block/drbd/drbd_main.c:1978: timer_setup(&device->resync_timer, resync_timer_fn, 0);
drivers/block/drbd/drbd_main.c:1979: timer_setup(&device->md_sync_timer, md_sync_timer_fn, 0);
drivers/block/drbd/drbd_main.c:1980: timer_setup(&device->start_resync_timer, start_resync_timer_fn, 0);
drivers/block/drbd/drbd_main.c:1981: timer_setup(&device->request_timer, request_timer_fn, 0);
drivers/block/drbd/drbd_main.c-1982-
--
drivers/block/floppy.c=4567=static int __init do_floppy_init(void)
--
drivers/block/floppy.c-4601-
drivers/block/floppy.c:4602: timer_setup(&motor_off_timer[drive], motor_off_callback, 0);
drivers/block/floppy.c-4603- }
--
drivers/block/loop.c=1999=static int loop_add(int i)
--
drivers/block/loop.c-2016- INIT_LIST_HEAD(&lo->idle_worker_list);
drivers/block/loop.c:2017: timer_setup(&lo->timer, loop_free_idle_workers_timer, TIMER_DEFERRABLE);
drivers/block/loop.c-2018- WRITE_ONCE(lo->lo_state, Lo_unbound);
--
drivers/block/null_blk/main.c=1484=static void nullb_setup_bwtimer(struct nullb *nullb)
--
drivers/block/null_blk/main.c-1487-
drivers/block/null_blk/main.c:1488: hrtimer_setup(&nullb->bw_timer, nullb_bwtimer_fn, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
drivers/block/null_blk/main.c-1489- atomic_long_set(&nullb->cur_bytes, mb_per_tick(nullb->dev->mbps));
--
drivers/block/null_blk/main.c=1652=static blk_status_t null_queue_rq(struct blk_mq_hw_ctx *hctx,
--
drivers/block/null_blk/main.c-1664- if (!is_poll && nq->dev->irqmode == NULL_IRQ_TIMER) {
drivers/block/null_blk/main.c:1665: hrtimer_setup(&cmd->timer, null_cmd_timer_expired, CLOCK_MONOTONIC,
drivers/block/null_blk/main.c-1666- HRTIMER_MODE_REL);
--
drivers/block/swim3.c=1102=static int swim3_add_device(struct macio_dev *mdev, int index)
--
drivers/block/swim3.c-1171-
drivers/block/swim3.c:1172: timer_setup(&fs->timeout, NULL, 0);
drivers/block/swim3.c-1173-
--
drivers/bluetooth/bluecard_cs.c=680=static int bluecard_open(struct bluecard_info *info)
--
drivers/bluetooth/bluecard_cs.c-687-
drivers/bluetooth/bluecard_cs.c:688: timer_setup(&info->timer, bluecard_activity_led_timeout, 0);
drivers/bluetooth/bluecard_cs.c-689-
--
drivers/bluetooth/btnxpuart.c=504=static int ps_setup(struct hci_dev *hdev)
--
drivers/bluetooth/btnxpuart.c-559- mutex_init(&psdata->ps_lock);
drivers/bluetooth/btnxpuart.c:560: timer_setup(&psdata->ps_timer, ps_timeout_func, 0);
drivers/bluetooth/btnxpuart.c-561-
--
drivers/bluetooth/hci_bcsp.c=713=static int bcsp_open(struct hci_uart *hu)
--
drivers/bluetooth/hci_bcsp.c-728-
drivers/bluetooth/hci_bcsp.c:729: timer_setup(&bcsp->tbcsp, bcsp_timed_event, 0);
]
]
|
| 8/2 |
2026/05/17 01:36 |
llm |
expert |
0m
Model:
Tokens:
input: 20985
output: 21
thoughts: 1196
Thoughts: **Analyzing a `refcount_t` Underflow and Use-After-Free**
Okay, so I see a `timer_setup` is used all over the place. My initial thought is to check if `refcount_dec` or `sock_put` are used within timer handlers. `sock_put` is networking-related, so let's start with `refcount_dec` in `net/`, `kernel/`, and `lib/`. The bug report describes a `refcount_t: underflow; use-after-free` warning. This warrants closer inspection. Specifically, I'll search for `sock_put` in timer handlers. `sk_timer` is a common timer, so it's a good place to start. Let's see if any timers set up using `timer_setup` in `net/` call `sock_put`.
The hint about a tail call to `refcount_dec` is intriguing because the instruction before `call refcount_warn_saturate` should be the decrement. Then, `refcount_warn_saturate` is called from `refcount_dec_and_test` or `refcount_dec`. I'll examine the disassembly from the bug report closely.
The provided disassembly is actually *inside* `refcount_warn_saturate`! The line with `ud1` is the `WARN` macro on x86, which confirms that the trap occurs within `refcount_warn_saturate`. The call trace starts from `refcount_warn_saturate`, but why isn't the caller present? Perhaps it's an inline function. If the caller's caller is missing, maybe there was a tail call to `refcount_warn_saturate`?
But wait! `refcount_warn_saturate` is *not* tail-called from `refcount_dec_and_test`, because the latter returns a boolean. And `refcount_dec` returns void, not a boolean.
```c
static inline void refcount_dec(refcount_t *r)
{
if (unlikely(atomic_dec_and_test(&r->refs)))
refcount_warn_saturate(r, REFCOUNT_DEC_UAF);
}
```
Okay, so if `atomic_dec_and_test` returns true, it calls `refcount_warn_saturate`. I note that `REFCOUNT_DEC_UAF` is *not* the same as `REFCOUNT_SUB_UAF`. The warning is specific: "refcount_t: underflow; use-after-free."
Let's consult `lib/refcount.c`:
```c
void refcount_warn_saturate(refcount_t *r, enum refcount_saturation_type t)
{
refcount_set(r, REFCOUNT_SATURATED);
switch (t) {
case REFCOUNT_ADD_NOT_ZERO_OVF:
case REFCOUNT_ADD_OVF:
case REFCOUNT_ADD_UAF:
WARN(1, "refcount_t: addition on 0; use-after-free.\n");
break;
case REFCOUNT_SUB_UAF:
WARN(1, "refcount_t: underflow; use-after-free.\n");
break;
case REFCOUNT_DEC_LEAK:
WARN(1, "refcount_t: decrement hit 0; leaking memory.\n");
break;
default:
WARN(1, "refcount_t: unknown saturation event!?\n");
}
}
```
So `t` must be `REFCOUNT_SUB_UAF`. Now I need to identify which inline function passes `REFCOUNT_SUB_UAF`. I'll look at `include/linux/refcount.h`.
|
| 9/2 |
2026/05/17 01:36 |
tool |
grepper |
0m
Args: map[Expression:REFCOUNT_SUB_UAF]
Results: map[Output:drivers/of/unittest.c=3632=static void __init of_unittest_lifecycle(void)
--
drivers/of/unittest.c-3688- * __refcount_sub_and_test() calls:
drivers/of/unittest.c:3689: * refcount_warn_saturate(r, REFCOUNT_SUB_UAF)
drivers/of/unittest.c-3690- *
--
include/linux/refcount.h=117=enum refcount_saturation_type {
--
include/linux/refcount.h-120- REFCOUNT_ADD_UAF,
include/linux/refcount.h:121: REFCOUNT_SUB_UAF,
include/linux/refcount.h-122- REFCOUNT_DEC_LEAK,
--
include/linux/refcount.h=387=bool __refcount_sub_and_test(int i, refcount_t *r, int *oldp)
--
include/linux/refcount.h-399- if (unlikely(old <= 0 || old - i < 0))
include/linux/refcount.h:400: refcount_warn_saturate(r, REFCOUNT_SUB_UAF);
include/linux/refcount.h-401-
--
lib/refcount.c=13=void refcount_warn_saturate(refcount_t *r, enum refcount_saturation_type t)
--
lib/refcount.c-26- break;
lib/refcount.c:27: case REFCOUNT_SUB_UAF:
lib/refcount.c-28- REFCOUNT_WARN("underflow; use-after-free");
]
|
| 10/2 |
2026/05/17 01:36 |
llm |
expert |
0m
Model:
Tokens:
input: 23511
output: 23
thoughts: 44
Thoughts: **Analyzing the `REFCOUNT_SUB_UAF` Macro and its Usage**
Okay, so I'm looking at this code and I see `REFCOUNT_SUB_UAF` being passed into `__refcount_sub_and_test`. My first instinct is to trace back the usage, figure out the context where this specific macro is being invoked. That usually tells the story of the potential security concerns. Let's see who's actually calling `__refcount_sub_and_test`. Knowing the callers will help me understand the purpose of `REFCOUNT_SUB_UAF`, and whether it's correctly preventing a use-after-free vulnerability, or potentially contributing to one. Time to dig in!
|
| 11/2 |
2026/05/17 01:36 |
tool |
grepper |
0m
Args: map[Expression:__refcount_sub_and_test]
Results: map[Output:drivers/of/unittest.c=3632=static void __init of_unittest_lifecycle(void)
--
drivers/of/unittest.c-3687- * expect stack trace for subsequent of_node_put():
drivers/of/unittest.c:3688: * __refcount_sub_and_test() calls:
drivers/of/unittest.c-3689- * refcount_warn_saturate(r, REFCOUNT_SUB_UAF)
--
include/linux/refcount.h=386=static inline __must_check
include/linux/refcount.h:387:bool __refcount_sub_and_test(int i, refcount_t *r, int *oldp)
include/linux/refcount.h-388-{
--
include/linux/refcount.h=425=static inline __must_check bool refcount_sub_and_test(int i, refcount_t *r)
include/linux/refcount.h-426-{
include/linux/refcount.h:427: return __refcount_sub_and_test(i, r, NULL);
include/linux/refcount.h-428-}
--
include/linux/refcount.h=430=static inline __must_check bool __refcount_dec_and_test(refcount_t *r, int *oldp)
include/linux/refcount.h-431-{
include/linux/refcount.h:432: return __refcount_sub_and_test(1, r, oldp);
include/linux/refcount.h-433-}
--
net/core/sock.c=2671=void sock_wfree(struct sk_buff *skb)
--
net/core/sock.c-2681- rcu_read_lock();
net/core/sock.c:2682: free = __refcount_sub_and_test(len, &sk->sk_wmem_alloc,
net/core/sock.c-2683- &old);
]
|
| 12/2 |
2026/05/17 01:36 |
llm |
expert |
0m
Model:
Tokens:
input: 24259
output: 20
thoughts: 135
Thoughts: **Unraveling the Refcount Logic**
Okay, hold on a second. I'm tracing through this refcount logic and something's clicked. It looks like `__refcount_dec_and_test` ultimately calls `__refcount_sub_and_test(1, r, oldp)`. But, I see that `refcount_sub_and_test` itself then calls `__refcount_sub_and_test(i, r, NULL)`. So, if I understand this correctly, `REFCOUNT_SUB_UAF` isn't *just* triggered by `refcount_dec_and_test`--it could also be triggered through `refcount_dec`!
Now let me retrace that last part, *refcount_dec* calls `__refcount_dec(r, &r->refs)`. I need to solidify that understanding. Okay, let me go in and check `__refcount_dec`
|
| 13/2 |
2026/05/17 01:37 |
tool |
grepper |
0m
Args: map[Expression:__refcount_dec\(]
Results: map[Output:include/linux/refcount.h=448=static inline __must_check bool refcount_dec_and_test(refcount_t *r)
--
include/linux/refcount.h-452-
include/linux/refcount.h:453:static inline void __refcount_dec(refcount_t *r, int *oldp)
include/linux/refcount.h-454-{
--
include/linux/refcount.h=474=static inline void refcount_dec(refcount_t *r)
include/linux/refcount.h-475-{
include/linux/refcount.h:476: __refcount_dec(r, NULL);
include/linux/refcount.h-477-}
]
|
| 14/2 |
2026/05/17 01:37 |
llm |
expert |
0m
Model:
Tokens:
input: 24819
output: 23
thoughts: 23
Thoughts: Here's my take on those thoughts:
**Analyzing `__refcount_dec`**
Okay, let's dive into the core of reference counting. My focus is immediately drawn to `__refcount_dec` within `include/linux/refcount.h`. I'm going to want to understand its implementation and how it interacts with the broader kernel reference counting mechanisms. Specifically, I'm thinking about the low-level aspects, the atomicity guarantees provided, and how it handles potential underflows. I need to refresh my memory, but I will be digging into this specific function, to understand it at the deepest level. This is where the magic happens, or can fail spectacularly if handled improperly.
|
| 15/2 |
2026/05/17 01:37 |
tool |
read-file |
0m
Args: map[File:include/linux/refcount.h]
Results: map[Contents:/* SPDX-License-Identifier: GPL-2.0 */
/*
* Variant of atomic_t specialized for reference counts.
*
* The interface matches the atomic_t interface (to aid in porting) but only
* provides the few functions one should use for reference counting.
*
* Saturation semantics
* ====================
*
* refcount_t differs from atomic_t in that the counter saturates at
* REFCOUNT_SATURATED and will not move once there. This avoids wrapping the
* counter and causing 'spurious' use-after-free issues. In order to avoid the
* cost associated with introducing cmpxchg() loops into all of the saturating
* operations, we temporarily allow the counter to take on an unchecked value
* and then explicitly set it to REFCOUNT_SATURATED on detecting that underflow
* or overflow has occurred. Although this is racy when multiple threads
* access the refcount concurrently, by placing REFCOUNT_SATURATED roughly
* equidistant from 0 and INT_MAX we minimise the scope for error:
*
* INT_MAX REFCOUNT_SATURATED UINT_MAX
* 0 (0x7fff_ffff) (0xc000_0000) (0xffff_ffff)
* +--------------------------------+----------------+----------------+
* <---------- bad value! ---------->
*
* (in a signed view of the world, the "bad value" range corresponds to
* a negative counter value).
*
* As an example, consider a refcount_inc() operation that causes the counter
* to overflow:
*
* int old = atomic_fetch_add_relaxed(r);
* // old is INT_MAX, refcount now INT_MIN (0x8000_0000)
* if (old < 0)
* atomic_set(r, REFCOUNT_SATURATED);
*
* If another thread also performs a refcount_inc() operation between the two
* atomic operations, then the count will continue to edge closer to 0. If it
* reaches a value of 1 before /any/ of the threads reset it to the saturated
* value, then a concurrent refcount_dec_and_test() may erroneously free the
* underlying object.
* Linux limits the maximum number of tasks to PID_MAX_LIMIT, which is currently
* 0x400000 (and can't easily be raised in the future beyond FUTEX_TID_MASK).
* With the current PID limit, if no batched refcounting operations are used and
* the attacker can't repeatedly trigger kernel oopses in the middle of refcount
* operations, this makes it impossible for a saturated refcount to leave the
* saturation range, even if it is possible for multiple uses of the same
* refcount to nest in the context of a single task:
*
* (UINT_MAX+1-REFCOUNT_SATURATED) / PID_MAX_LIMIT =
* 0x40000000 / 0x400000 = 0x100 = 256
*
* If hundreds of references are added/removed with a single refcounting
* operation, it may potentially be possible to leave the saturation range; but
* given the precise timing details involved with the round-robin scheduling of
* each thread manipulating the refcount and the need to hit the race multiple
* times in succession, there doesn't appear to be a practical avenue of attack
* even if using refcount_add() operations with larger increments.
*
* Memory ordering
* ===============
*
* Memory ordering rules are slightly relaxed wrt regular atomic_t functions
* and provide only what is strictly required for refcounts.
*
* The increments are fully relaxed; these will not provide ordering. The
* rationale is that whatever is used to obtain the object we're increasing the
* reference count on will provide the ordering. For locked data structures,
* its the lock acquire, for RCU/lockless data structures its the dependent
* load.
*
* Do note that inc_not_zero() provides a control dependency which will order
* future stores against the inc, this ensures we'll never modify the object
* if we did not in fact acquire a reference.
*
* The decrements will provide release order, such that all the prior loads and
* stores will be issued before, it also provides a control dependency, which
* will order us against the subsequent free().
*
* The control dependency is against the load of the cmpxchg (ll/sc) that
* succeeded. This means the stores aren't fully ordered, but this is fine
* because the 1->0 transition indicates no concurrency.
*
* Note that the allocator is responsible for ordering things between free()
* and alloc().
*
* The decrements dec_and_test() and sub_and_test() also provide acquire
* ordering on success.
*
* refcount_{add|inc}_not_zero_acquire() and refcount_set_release() provide
* acquire and release ordering for cases when the memory occupied by the
* object might be reused to store another object. This is important for the
* cases where secondary validation is required to detect such reuse, e.g.
* SLAB_TYPESAFE_BY_RCU. The secondary validation checks have to happen after
* the refcount is taken, hence acquire order is necessary. Similarly, when the
* object is initialized, all stores to its attributes should be visible before
* the refcount is set, otherwise a stale attribute value might be used by
* another task which succeeds in taking a refcount to the new object.
*/
#ifndef _LINUX_REFCOUNT_H
#define _LINUX_REFCOUNT_H
#include <linux/atomic.h>
#include <linux/bug.h>
#include <linux/compiler.h>
#include <linux/limits.h>
#include <linux/refcount_types.h>
#include <linux/spinlock_types.h>
struct mutex;
#define REFCOUNT_INIT(n) { .refs = ATOMIC_INIT(n), }
#define REFCOUNT_MAX INT_MAX
#define REFCOUNT_SATURATED (INT_MIN / 2)
enum refcount_saturation_type {
REFCOUNT_ADD_NOT_ZERO_OVF,
REFCOUNT_ADD_OVF,
REFCOUNT_ADD_UAF,
REFCOUNT_SUB_UAF,
REFCOUNT_DEC_LEAK,
};
void refcount_warn_saturate(refcount_t *r, enum refcount_saturation_type t);
/**
* refcount_set - set a refcount's value
* @r: the refcount
* @n: value to which the refcount will be set
*/
static inline void refcount_set(refcount_t *r, int n)
{
atomic_set(&r->refs, n);
}
/**
* refcount_set_release - set a refcount's value with release ordering
* @r: the refcount
* @n: value to which the refcount will be set
*
* This function should be used when memory occupied by the object might be
* reused to store another object -- consider SLAB_TYPESAFE_BY_RCU.
*
* Provides release memory ordering which will order previous memory operations
* against this store. This ensures all updates to this object are visible
* once the refcount is set and stale values from the object previously
* occupying this memory are overwritten with new ones.
*
* This function should be called only after new object is fully initialized.
* After this call the object should be considered visible to other tasks even
* if it was not yet added into an object collection normally used to discover
* it. This is because other tasks might have discovered the object previously
* occupying the same memory and after memory reuse they can succeed in taking
* refcount to the new object and start using it.
*/
static inline void refcount_set_release(refcount_t *r, int n)
{
atomic_set_release(&r->refs, n);
}
/**
* refcount_read - get a refcount's value
* @r: the refcount
*
* Return: the refcount's value
*/
static inline unsigned int refcount_read(const refcount_t *r)
{
return atomic_read(&r->refs);
}
static inline __must_check
bool __refcount_add_not_zero(int i, refcount_t *r, int *oldp)
{
int old = refcount_read(r);
do {
if (!old)
break;
} while (!atomic_try_cmpxchg_relaxed(&r->refs, &old, old + i));
if (oldp)
*oldp = old;
if (unlikely(old < 0 || old + i < 0))
refcount_warn_saturate(r, REFCOUNT_ADD_NOT_ZERO_OVF);
return old;
}
/**
* refcount_add_not_zero - add a value to a refcount unless it is 0
* @i: the value to add to the refcount
* @r: the refcount
*
* Will saturate at REFCOUNT_SATURATED and WARN.
*
* Provides no memory ordering, it is assumed the caller has guaranteed the
* object memory to be stable (RCU, etc.). It does provide a control dependency
* and thereby orders future stores. See the comment on top.
*
* Use of this function is not recommended for the normal reference counting
* use case in which references are taken and released one at a time. In these
* cases, refcount_inc(), or one of its variants, should instead be used to
* increment a reference count.
*
* Return: false if the passed refcount is 0, true otherwise
*/
static inline __must_check bool refcount_add_not_zero(int i, refcount_t *r)
{
return __refcount_add_not_zero(i, r, NULL);
}
static inline __must_check
bool __refcount_add_not_zero_limited_acquire(int i, refcount_t *r, int *oldp,
int limit)
{
int old = refcount_read(r);
do {
if (!old)
break;
if (i > limit - old) {
if (oldp)
*oldp = old;
return false;
}
} while (!atomic_try_cmpxchg_acquire(&r->refs, &old, old + i));
if (oldp)
*oldp = old;
if (unlikely(old < 0 || old + i < 0))
refcount_warn_saturate(r, REFCOUNT_ADD_NOT_ZERO_OVF);
return old;
}
static inline __must_check bool
__refcount_inc_not_zero_limited_acquire(refcount_t *r, int *oldp, int limit)
{
return __refcount_add_not_zero_limited_acquire(1, r, oldp, limit);
}
static inline __must_check
bool __refcount_add_not_zero_acquire(int i, refcount_t *r, int *oldp)
{
return __refcount_add_not_zero_limited_acquire(i, r, oldp, INT_MAX);
}
/**
* refcount_add_not_zero_acquire - add a value to a refcount with acquire ordering unless it is 0
*
* @i: the value to add to the refcount
* @r: the refcount
*
* Will saturate at REFCOUNT_SATURATED and WARN.
*
* This function should be used when memory occupied by the object might be
* reused to store another object -- consider SLAB_TYPESAFE_BY_RCU.
*
* Provides acquire memory ordering on success, it is assumed the caller has
* guaranteed the object memory to be stable (RCU, etc.). It does provide a
* control dependency and thereby orders future stores. See the comment on top.
*
* Use of this function is not recommended for the normal reference counting
* use case in which references are taken and released one at a time. In these
* cases, refcount_inc_not_zero_acquire() should instead be used to increment a
* reference count.
*
* Return: false if the passed refcount is 0, true otherwise
*/
static inline __must_check bool refcount_add_not_zero_acquire(int i, refcount_t *r)
{
return __refcount_add_not_zero_acquire(i, r, NULL);
}
static inline
void __refcount_add(int i, refcount_t *r, int *oldp)
{
int old = atomic_fetch_add_relaxed(i, &r->refs);
if (oldp)
*oldp = old;
if (unlikely(!old))
refcount_warn_saturate(r, REFCOUNT_ADD_UAF);
else if (unlikely(old < 0 || old + i < 0))
refcount_warn_saturate(r, REFCOUNT_ADD_OVF);
}
/**
* refcount_add - add a value to a refcount
* @i: the value to add to the refcount
* @r: the refcount
*
* Similar to atomic_add(), but will saturate at REFCOUNT_SATURATED and WARN.
*
* Provides no memory ordering, it is assumed the caller has guaranteed the
* object memory to be stable (RCU, etc.). It does provide a control dependency
* and thereby orders future stores. See the comment on top.
*
* Use of this function is not recommended for the normal reference counting
* use case in which references are taken and released one at a time. In these
* cases, refcount_inc(), or one of its variants, should instead be used to
* increment a reference count.
*/
static inline void refcount_add(int i, refcount_t *r)
{
__refcount_add(i, r, NULL);
}
static inline __must_check bool __refcount_inc_not_zero(refcount_t *r, int *oldp)
{
return __refcount_add_not_zero(1, r, oldp);
}
/**
* refcount_inc_not_zero - increment a refcount unless it is 0
* @r: the refcount to increment
*
* Similar to atomic_inc_not_zero(), but will saturate at REFCOUNT_SATURATED
* and WARN.
*
* Provides no memory ordering, it is assumed the caller has guaranteed the
* object memory to be stable (RCU, etc.). It does provide a control dependency
* and thereby orders future stores. See the comment on top.
*
* Return: true if the increment was successful, false otherwise
*/
static inline __must_check bool refcount_inc_not_zero(refcount_t *r)
{
return __refcount_inc_not_zero(r, NULL);
}
static inline __must_check bool __refcount_inc_not_zero_acquire(refcount_t *r, int *oldp)
{
return __refcount_add_not_zero_acquire(1, r, oldp);
}
/**
* refcount_inc_not_zero_acquire - increment a refcount with acquire ordering unless it is 0
* @r: the refcount to increment
*
* Similar to refcount_inc_not_zero(), but provides acquire memory ordering on
* success.
*
* This function should be used when memory occupied by the object might be
* reused to store another object -- consider SLAB_TYPESAFE_BY_RCU.
*
* Provides acquire memory ordering on success, it is assumed the caller has
* guaranteed the object memory to be stable (RCU, etc.). It does provide a
* control dependency and thereby orders future stores. See the comment on top.
*
* Return: true if the increment was successful, false otherwise
*/
static inline __must_check bool refcount_inc_not_zero_acquire(refcount_t *r)
{
return __refcount_inc_not_zero_acquire(r, NULL);
}
static inline void __refcount_inc(refcount_t *r, int *oldp)
{
__refcount_add(1, r, oldp);
}
/**
* refcount_inc - increment a refcount
* @r: the refcount to increment
*
* Similar to atomic_inc(), but will saturate at REFCOUNT_SATURATED and WARN.
*
* Provides no memory ordering, it is assumed the caller already has a
* reference on the object.
*
* Will WARN if the refcount is 0, as this represents a possible use-after-free
* condition.
*/
static inline void refcount_inc(refcount_t *r)
{
__refcount_inc(r, NULL);
}
static inline __must_check
bool __refcount_sub_and_test(int i, refcount_t *r, int *oldp)
{
int old = atomic_fetch_sub_release(i, &r->refs);
if (oldp)
*oldp = old;
if (old > 0 && old == i) {
smp_acquire__after_ctrl_dep();
return true;
}
if (unlikely(old <= 0 || old - i < 0))
refcount_warn_saturate(r, REFCOUNT_SUB_UAF);
return false;
}
/**
* refcount_sub_and_test - subtract from a refcount and test if it is 0
* @i: amount to subtract from the refcount
* @r: the refcount
*
* Similar to atomic_dec_and_test(), but it will WARN, return false and
* ultimately leak on underflow and will fail to decrement when saturated
* at REFCOUNT_SATURATED.
*
* Provides release memory ordering, such that prior loads and stores are done
* before, and provides an acquire ordering on success such that free()
* must come after.
*
* Use of this function is not recommended for the normal reference counting
* use case in which references are taken and released one at a time. In these
* cases, refcount_dec(), or one of its variants, should instead be used to
* decrement a reference count.
*
* Return: true if the resulting refcount is 0, false otherwise
*/
static inline __must_check bool refcount_sub_and_test(int i, refcount_t *r)
{
return __refcount_sub_and_test(i, r, NULL);
}
static inline __must_check bool __refcount_dec_and_test(refcount_t *r, int *oldp)
{
return __refcount_sub_and_test(1, r, oldp);
}
/**
* refcount_dec_and_test - decrement a refcount and test if it is 0
* @r: the refcount
*
* Similar to atomic_dec_and_test(), it will WARN on underflow and fail to
* decrement when saturated at REFCOUNT_SATURATED.
*
* Provides release memory ordering, such that prior loads and stores are done
* before, and provides an acquire ordering on success such that free()
* must come after.
*
* Return: true if the resulting refcount is 0, false otherwise
*/
static inline __must_check bool refcount_dec_and_test(refcount_t *r)
{
return __refcount_dec_and_test(r, NULL);
}
static inline void __refcount_dec(refcount_t *r, int *oldp)
{
int old = atomic_fetch_sub_release(1, &r->refs);
if (oldp)
*oldp = old;
if (unlikely(old <= 1))
refcount_warn_saturate(r, REFCOUNT_DEC_LEAK);
}
/**
* refcount_dec - decrement a refcount
* @r: the refcount
*
* Similar to atomic_dec(), it will WARN on underflow and fail to decrement
* when saturated at REFCOUNT_SATURATED.
*
* Provides release memory ordering, such that prior loads and stores are done
* before.
*/
static inline void refcount_dec(refcount_t *r)
{
__refcount_dec(r, NULL);
}
extern __must_check bool refcount_dec_if_one(refcount_t *r);
extern __must_check bool refcount_dec_not_one(refcount_t *r);
extern __must_check bool refcount_dec_and_mutex_lock(refcount_t *r, struct mutex *lock) __cond_acquires(true, lock);
extern __must_check bool refcount_dec_and_lock(refcount_t *r, spinlock_t *lock) __cond_acquires(true, lock);
extern __must_check bool refcount_dec_and_lock_irqsave(refcount_t *r,
spinlock_t *lock,
unsigned long *flags) __cond_acquires(true, lock);
#endif /* _LINUX_REFCOUNT_H */
]
|
| 16/2 |
2026/05/17 01:37 |
llm |
expert |
0m
Model:
Tokens:
input: 30052
output: 19
thoughts: 973
Thoughts: **Refcount Underflow and Potential Tail Calls in Timer Callbacks**
Okay, let's break this down. My initial thought was about `__refcount_dec` and its call to `refcount_warn_saturate` with `REFCOUNT_DEC_LEAK`. Then I followed the call chain: `REFCOUNT_SUB_UAF` is exclusively passed from `__refcount_sub_and_test`. That's important. `__refcount_sub_and_test` is called by `refcount_sub_and_test` and `__refcount_dec_and_test`, the latter ultimately called by `refcount_dec_and_test`. I realized that if `__refcount_sub_and_test` returns true, the refcount hit zero. If it returns false, it *might* have underflowed, triggering `refcount_warn_saturate` with `REFCOUNT_SUB_UAF`.
Then the crucial point hit me: `__refcount_sub_and_test` is an *inline* function. That's a huge clue! If it's inlined into a timer callback, and the timer callback ends with a call to `refcount_warn_saturate`, the compiler might be able to optimize that into a tail call, right?
Looking at the code for `__refcount_sub_and_test`, I confirmed my suspicion:
```c
if (unlikely(old <= 0 || old - i < 0))
refcount_warn_saturate(r, REFCOUNT_SUB_UAF);
return false;
```
If `refcount_warn_saturate` gets called, it *has* to return `false` afterwards. This means it isn't, itself, a tail call because it has to return! However, if the caller *ignores* the return value, the compiler *can* optimize it into a tail call!
Imagine a timer callback that looks something like this:
```c
void my_timer_callback(struct timer_list *t)
{
...
refcount_dec_and_test(&obj->ref);
}
```
If `refcount_dec_and_test` is inlined, and the compiler sees:
```c
if (unlikely(old <= 0 || old - 1 < 0)) {
refcount_warn_saturate(r, REFCOUNT_SUB_UAF);
return false;
}
return false;
```
...because both branches return `false` (or the return value is ignored), the compiler can tail-call `refcount_warn_saturate`. Boom! A potential vulnerability. The timer callback ends with `refcount_dec_and_test(...)` or `refcount_sub_and_test(...)` and ignores the result.
Now, what if the caller *uses* the return value, for instance:
```c
if (refcount_dec_and_test(&obj->ref)) {
free(obj);
}
```
If there's an underflow, `refcount_dec_and_test` returns `false`, and the code *doesn't* free the object. The `false` means we have to return from the callback in *that* underflow branch. This is the crucial point! The code now looks like this:
```c
if (unlikely(old <= 0 || old - 1 < 0)) {
refcount_warn_saturate(r, REFCOUNT_SUB_UAF);
goto return_false;
}
```
Because of the early return in the underflow branch, the compiler can still translate this into a tail call!
I've got to check for timer callbacks that do `if (refcount_dec_and_test(...)) free(...);` at the end of the function. This is a super common pattern. Think "put" functions. `sock_put` does exactly this:
```c
void sock_put(struct sock *sk)
{
if (refcount_dec_and_test(&sk->sk_refcnt))
sk_free(sk);
}
```
If `sock_put` gets inlined at the end of a timer callback, or `sock_put` *itself* is tail-called from the timer callback... there could be issues! Also, `sock_put` is NOT an inline function in some configurations, but it *is* in others?! This needs further investigation. I must check `sock_put`.
|
| 17/2 |
2026/05/17 01:37 |
tool |
grepper |
0m
Args: map[Expression:void sock_put\(]
Results: map[Output:include/net/sock.h=1979=void sock_init_data(struct socket *sock, struct sock *sk);
--
include/net/sock.h-2006-/* Ungrab socket and destroy it, if it was the last reference. */
include/net/sock.h:2007:static inline void sock_put(struct sock *sk)
include/net/sock.h-2008-{
]
|
| 18/2 |
2026/05/17 01:37 |
llm |
expert |
0m
Model:
Tokens:
input: 32173
output: 21
thoughts: 428
Thoughts: **Analyzing a Suspected Timer Callback Bug**
Okay, so `sock_put` is defined as an inline function. That's immediately interesting because it means the compiler *could* optimize a timer callback that ends with `sock_put(sk)` by doing a tail call to `refcount_warn_saturate` if there is an underflow! That could explain a nasty bug. The same goes for `reqsk_put`, which is also inline.
My first thought is to find all the timer callbacks that end with `sock_put`, `reqsk_put`, or anything similar. The goal is to look at how these timers are set up. My initial instinct is to grep for `timer_setup` with the callback calling `sock_put` or `reqsk_put`. However, it's simpler to directly search for `sock_put` inside timer callbacks. I can look in net-related source files.
The bug report states "refcount_t: underflow; use-after-free." and mentions CPU, UID, PID and other important details. The key bit is that it happens after a `tail-call` - if `sock_put` is tail-called, the timer callback is missing.
I need to figure out exactly how to find this bug.
Since I don't have internet access, I can't search syzbot logs for that specific error in the `call_timer_fn` context, so I will have to look in the code. I'll need to use my experience. I'll need to consider recent bugs or common timer callbacks within the networking stack. Maybe something related to `sk_timer`? I'll check `net/core/sock.c` and other common networking areas. I'm confident I can pinpoint the issue with the right approach.
|
| 19/2 |
2026/05/17 01:37 |
tool |
grepper |
0m
Args: map[Expression:timer_setup.*sk_timer]
Results: map[Output:drivers/usb/gadget/function/f_ncm.c=1432=static int ncm_bind(struct usb_configuration *c, struct usb_function *f)
--
drivers/usb/gadget/function/f_ncm.c-1558-
drivers/usb/gadget/function/f_ncm.c:1559: hrtimer_setup(&ncm->task_timer, ncm_tx_timeout, CLOCK_MONOTONIC, HRTIMER_MODE_REL_SOFT);
drivers/usb/gadget/function/f_ncm.c-1560-
--
kernel/sched/deadline.c=1305=static void init_dl_task_timer(struct sched_dl_entity *dl_se)
--
kernel/sched/deadline.c-1308-
kernel/sched/deadline.c:1309: hrtimer_setup(timer, dl_task_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD);
kernel/sched/deadline.c-1310-}
--
kernel/sched/deadline.c=2015=static void init_dl_inactive_task_timer(struct sched_dl_entity *dl_se)
--
kernel/sched/deadline.c-2018-
kernel/sched/deadline.c:2019: hrtimer_setup(timer, inactive_task_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD);
kernel/sched/deadline.c-2020-}
--
net/appletalk/ddp.c=181=static inline void atalk_destroy_socket(struct sock *sk)
--
net/appletalk/ddp.c-186- if (sk_has_allocations(sk)) {
net/appletalk/ddp.c:187: timer_setup(&sk->sk_timer, atalk_destroy_timer, 0);
net/appletalk/ddp.c-188- sk->sk_timer.expires = jiffies + SOCK_DESTROY_TIME;
--
net/core/sock.c=3699=void sock_init_data_uid(struct socket *sock, struct sock *sk, kuid_t uid)
--
net/core/sock.c-3703-
net/core/sock.c:3704: timer_setup(&sk->sk_timer, NULL, 0);
net/core/sock.c-3705-
--
net/ipv4/inet_connection_sock.c=1024=static void reqsk_timer_handler(struct timer_list *t)
--
net/ipv4/inet_connection_sock.c-1050- refcount_set(&nreq->rsk_refcnt, 2 + 1);
net/ipv4/inet_connection_sock.c:1051: timer_setup(&nreq->rsk_timer, reqsk_timer_handler, TIMER_PINNED);
net/ipv4/inet_connection_sock.c-1052- reqsk_queue_migrated(&inet_csk(nsk)->icsk_accept_queue, req);
--
net/ipv4/inet_connection_sock.c=1141=static bool reqsk_queue_hash_req(struct request_sock *req)
--
net/ipv4/inet_connection_sock.c-1149- req->timeout = tcp_timeout_init((struct sock *)req);
net/ipv4/inet_connection_sock.c:1150: timer_setup(&req->rsk_timer, reqsk_timer_handler, TIMER_PINNED);
net/ipv4/inet_connection_sock.c-1151- mod_timer(&req->rsk_timer, jiffies + req->timeout);
--
net/tipc/socket.c=458=static int tipc_sk_create(struct net *net, struct socket *sock,
--
net/tipc/socket.c-514- msg_set_origport(msg, tsk->portid);
net/tipc/socket.c:515: timer_setup(&sk->sk_timer, tipc_sk_timeout, 0);
net/tipc/socket.c-516- sk->sk_shutdown = 0;
]
|
| 20/2 |
2026/05/17 01:37 |
llm |
expert |
0m
Model:
Tokens:
input: 34083
output: 27
thoughts: 28
Thoughts: Okay, let's dive into this. Here's how I'd approach that kernel code analysis, as though these are my internal thoughts:
**Analyzing `reqsk_timer_handler` in `net/ipv4/inet_connection_sock.c`**
Alright, time to crack open `net/ipv4/inet_connection_sock.c`. I need to understand how TCP connection requests are timed out. Specifically, I'm focusing on `reqsk_timer_handler`. This function is likely crucial for handling the retransmission of SYN-ACK packets, the management of the backlog queue, and potentially even garbage collection of partially established connections. I need to figure out what triggers this timer and what actions it performs. I'll need to look at how this handler interacts with the `request_sock` data structure, to identify which sockets are affected by the timer. I expect to see checks on various states, such as the SYN-ACK retransmission count, backoff algorithms being implemented, and possibly some connection cleanup logic. Let me look for where this function is called, what arguments are passed in, and what the associated data structures look like. This should be a good starting point for a deeper dive.
|
| 21/2 |
2026/05/17 01:37 |
tool |
read-file |
0m
Args: map[File:net/ipv4/inet_connection_sock.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-or-later
/*
* INET An implementation of the TCP/IP protocol suite for the LINUX
* operating system. INET is implemented using the BSD Socket
* interface as the means of communication with the user level.
*
* Support for INET connection oriented protocols.
*
* Authors: See the TCP sources
*/
#include <linux/module.h>
#include <linux/jhash.h>
#include <net/inet_connection_sock.h>
#include <net/inet_hashtables.h>
#include <net/inet_timewait_sock.h>
#include <net/ip.h>
#include <net/route.h>
#include <net/tcp_states.h>
#include <net/xfrm.h>
#include <net/tcp.h>
#include <net/tcp_ecn.h>
#include <net/sock_reuseport.h>
#include <net/addrconf.h>
#if IS_ENABLED(CONFIG_IPV6)
/* match_sk*_wildcard == true: IPV6_ADDR_ANY equals to any IPv6 addresses
* if IPv6 only, and any IPv4 addresses
* if not IPv6 only
* match_sk*_wildcard == false: addresses must be exactly the same, i.e.
* IPV6_ADDR_ANY only equals to IPV6_ADDR_ANY,
* and 0.0.0.0 equals to 0.0.0.0 only
*/
static bool ipv6_rcv_saddr_equal(const struct in6_addr *sk1_rcv_saddr6,
const struct in6_addr *sk2_rcv_saddr6,
__be32 sk1_rcv_saddr, __be32 sk2_rcv_saddr,
bool sk1_ipv6only, bool sk2_ipv6only,
bool match_sk1_wildcard,
bool match_sk2_wildcard)
{
int addr_type = ipv6_addr_type(sk1_rcv_saddr6);
int addr_type2 = sk2_rcv_saddr6 ? ipv6_addr_type(sk2_rcv_saddr6) : IPV6_ADDR_MAPPED;
/* if both are mapped, treat as IPv4 */
if (addr_type == IPV6_ADDR_MAPPED && addr_type2 == IPV6_ADDR_MAPPED) {
if (!sk2_ipv6only) {
if (sk1_rcv_saddr == sk2_rcv_saddr)
return true;
return (match_sk1_wildcard && !sk1_rcv_saddr) ||
(match_sk2_wildcard && !sk2_rcv_saddr);
}
return false;
}
if (addr_type == IPV6_ADDR_ANY && addr_type2 == IPV6_ADDR_ANY)
return true;
if (addr_type2 == IPV6_ADDR_ANY && match_sk2_wildcard &&
!(sk2_ipv6only && addr_type == IPV6_ADDR_MAPPED))
return true;
if (addr_type == IPV6_ADDR_ANY && match_sk1_wildcard &&
!(sk1_ipv6only && addr_type2 == IPV6_ADDR_MAPPED))
return true;
if (sk2_rcv_saddr6 &&
ipv6_addr_equal(sk1_rcv_saddr6, sk2_rcv_saddr6))
return true;
return false;
}
#endif
/* match_sk*_wildcard == true: 0.0.0.0 equals to any IPv4 addresses
* match_sk*_wildcard == false: addresses must be exactly the same, i.e.
* 0.0.0.0 only equals to 0.0.0.0
*/
static bool ipv4_rcv_saddr_equal(__be32 sk1_rcv_saddr, __be32 sk2_rcv_saddr,
bool sk2_ipv6only, bool match_sk1_wildcard,
bool match_sk2_wildcard)
{
if (!sk2_ipv6only) {
if (sk1_rcv_saddr == sk2_rcv_saddr)
return true;
return (match_sk1_wildcard && !sk1_rcv_saddr) ||
(match_sk2_wildcard && !sk2_rcv_saddr);
}
return false;
}
bool inet_rcv_saddr_equal(const struct sock *sk, const struct sock *sk2,
bool match_wildcard)
{
#if IS_ENABLED(CONFIG_IPV6)
if (sk->sk_family == AF_INET6)
return ipv6_rcv_saddr_equal(&sk->sk_v6_rcv_saddr,
inet6_rcv_saddr(sk2),
sk->sk_rcv_saddr,
sk2->sk_rcv_saddr,
ipv6_only_sock(sk),
ipv6_only_sock(sk2),
match_wildcard,
match_wildcard);
#endif
return ipv4_rcv_saddr_equal(sk->sk_rcv_saddr, sk2->sk_rcv_saddr,
ipv6_only_sock(sk2), match_wildcard,
match_wildcard);
}
bool inet_rcv_saddr_any(const struct sock *sk)
{
#if IS_ENABLED(CONFIG_IPV6)
if (sk->sk_family == AF_INET6)
return ipv6_addr_any(&sk->sk_v6_rcv_saddr);
#endif
return !sk->sk_rcv_saddr;
}
/**
* inet_sk_get_local_port_range - fetch ephemeral ports range
* @sk: socket
* @low: pointer to low port
* @high: pointer to high port
*
* Fetch netns port range (/proc/sys/net/ipv4/ip_local_port_range)
* Range can be overridden if socket got IP_LOCAL_PORT_RANGE option.
* Returns true if IP_LOCAL_PORT_RANGE was set on this socket.
*/
bool inet_sk_get_local_port_range(const struct sock *sk, int *low, int *high)
{
int lo, hi, sk_lo, sk_hi;
bool local_range = false;
u32 sk_range;
inet_get_local_port_range(sock_net(sk), &lo, &hi);
sk_range = READ_ONCE(inet_sk(sk)->local_port_range);
if (unlikely(sk_range)) {
sk_lo = sk_range & 0xffff;
sk_hi = sk_range >> 16;
if (lo <= sk_lo && sk_lo <= hi)
lo = sk_lo;
if (lo <= sk_hi && sk_hi <= hi)
hi = sk_hi;
local_range = true;
}
*low = lo;
*high = hi;
return local_range;
}
EXPORT_SYMBOL(inet_sk_get_local_port_range);
static bool inet_bind_conflict(const struct sock *sk, struct sock *sk2,
kuid_t uid, bool relax,
bool reuseport_cb_ok, bool reuseport_ok)
{
int bound_dev_if2;
if (sk == sk2)
return false;
bound_dev_if2 = READ_ONCE(sk2->sk_bound_dev_if);
if (!sk->sk_bound_dev_if || !bound_dev_if2 ||
sk->sk_bound_dev_if == bound_dev_if2) {
if (sk->sk_reuse && sk2->sk_reuse &&
sk2->sk_state != TCP_LISTEN) {
if (!relax || (!reuseport_ok && sk->sk_reuseport &&
sk2->sk_reuseport && reuseport_cb_ok &&
(sk2->sk_state == TCP_TIME_WAIT ||
uid_eq(uid, sk_uid(sk2)))))
return true;
} else if (!reuseport_ok || !sk->sk_reuseport ||
!sk2->sk_reuseport || !reuseport_cb_ok ||
(sk2->sk_state != TCP_TIME_WAIT &&
!uid_eq(uid, sk_uid(sk2)))) {
return true;
}
}
return false;
}
static bool __inet_bhash2_conflict(const struct sock *sk, struct sock *sk2,
kuid_t uid, bool relax,
bool reuseport_cb_ok, bool reuseport_ok)
{
if (ipv6_only_sock(sk2)) {
if (sk->sk_family == AF_INET)
return false;
#if IS_ENABLED(CONFIG_IPV6)
if (ipv6_addr_v4mapped(&sk->sk_v6_rcv_saddr))
return false;
#endif
}
return inet_bind_conflict(sk, sk2, uid, relax,
reuseport_cb_ok, reuseport_ok);
}
static bool inet_bhash2_conflict(const struct sock *sk,
const struct inet_bind2_bucket *tb2,
kuid_t uid,
bool relax, bool reuseport_cb_ok,
bool reuseport_ok)
{
struct sock *sk2;
sk_for_each_bound(sk2, &tb2->owners) {
if (__inet_bhash2_conflict(sk, sk2, uid, relax,
reuseport_cb_ok, reuseport_ok))
return true;
}
return false;
}
#define sk_for_each_bound_bhash(__sk, __tb2, __tb) \
hlist_for_each_entry(__tb2, &(__tb)->bhash2, bhash_node) \
sk_for_each_bound((__sk), &(__tb2)->owners)
/* This should be called only when the tb and tb2 hashbuckets' locks are held */
static int inet_csk_bind_conflict(const struct sock *sk,
const struct inet_bind_bucket *tb,
const struct inet_bind2_bucket *tb2, /* may be null */
bool relax, bool reuseport_ok)
{
struct sock_reuseport *reuseport_cb;
kuid_t uid = sk_uid(sk);
bool reuseport_cb_ok;
struct sock *sk2;
rcu_read_lock();
reuseport_cb = rcu_dereference(sk->sk_reuseport_cb);
/* paired with WRITE_ONCE() in __reuseport_(add|detach)_closed_sock */
reuseport_cb_ok = !reuseport_cb || READ_ONCE(reuseport_cb->num_closed_socks);
rcu_read_unlock();
/* Conflicts with an existing IPV6_ADDR_ANY (if ipv6) or INADDR_ANY (if
* ipv4) should have been checked already. We need to do these two
* checks separately because their spinlocks have to be acquired/released
* independently of each other, to prevent possible deadlocks
*/
if (inet_use_hash2_on_bind(sk))
return tb2 && inet_bhash2_conflict(sk, tb2, uid, relax,
reuseport_cb_ok, reuseport_ok);
/* Unlike other sk lookup places we do not check
* for sk_net here, since _all_ the socks listed
* in tb->owners and tb2->owners list belong
* to the same net - the one this bucket belongs to.
*/
sk_for_each_bound_bhash(sk2, tb2, tb) {
if (!inet_bind_conflict(sk, sk2, uid, relax, reuseport_cb_ok, reuseport_ok))
continue;
if (inet_rcv_saddr_equal(sk, sk2, true))
return true;
}
return false;
}
/* Determine if there is a bind conflict with an existing IPV6_ADDR_ANY (if ipv6) or
* INADDR_ANY (if ipv4) socket.
*
* Caller must hold bhash hashbucket lock with local bh disabled, to protect
* against concurrent binds on the port for addr any
*/
static bool inet_bhash2_addr_any_conflict(const struct sock *sk, int port, int l3mdev,
bool relax, bool reuseport_ok)
{
const struct net *net = sock_net(sk);
struct sock_reuseport *reuseport_cb;
struct inet_bind_hashbucket *head2;
struct inet_bind2_bucket *tb2;
kuid_t uid = sk_uid(sk);
bool conflict = false;
bool reuseport_cb_ok;
rcu_read_lock();
reuseport_cb = rcu_dereference(sk->sk_reuseport_cb);
/* paired with WRITE_ONCE() in __reuseport_(add|detach)_closed_sock */
reuseport_cb_ok = !reuseport_cb || READ_ONCE(reuseport_cb->num_closed_socks);
rcu_read_unlock();
head2 = inet_bhash2_addr_any_hashbucket(sk, net, port);
spin_lock(&head2->lock);
inet_bind_bucket_for_each(tb2, &head2->chain) {
if (!inet_bind2_bucket_match_addr_any(tb2, net, port, l3mdev, sk))
continue;
if (!inet_bhash2_conflict(sk, tb2, uid, relax, reuseport_cb_ok, reuseport_ok))
continue;
conflict = true;
break;
}
spin_unlock(&head2->lock);
return conflict;
}
/*
* Find an open port number for the socket. Returns with the
* inet_bind_hashbucket locks held if successful.
*/
static struct inet_bind_hashbucket *
inet_csk_find_open_port(const struct sock *sk, struct inet_bind_bucket **tb_ret,
struct inet_bind2_bucket **tb2_ret,
struct inet_bind_hashbucket **head2_ret, int *port_ret)
{
struct inet_hashinfo *hinfo = tcp_get_hashinfo(sk);
int i, low, high, attempt_half, port, l3mdev;
struct inet_bind_hashbucket *head, *head2;
struct net *net = sock_net(sk);
struct inet_bind2_bucket *tb2;
struct inet_bind_bucket *tb;
u32 remaining, offset;
bool relax = false;
l3mdev = inet_sk_bound_l3mdev(sk);
ports_exhausted:
attempt_half = (sk->sk_reuse == SK_CAN_REUSE) ? 1 : 0;
other_half_scan:
inet_sk_get_local_port_range(sk, &low, &high);
high++; /* [32768, 60999] -> [32768, 61000[ */
if (high - low < 4)
attempt_half = 0;
if (attempt_half) {
int half = low + (((high - low) >> 2) << 1);
if (attempt_half == 1)
high = half;
else
low = half;
}
remaining = high - low;
if (likely(remaining > 1))
remaining &= ~1U;
offset = get_random_u32_below(remaining);
/* __inet_hash_connect() favors ports having @low parity
* We do the opposite to not pollute connect() users.
*/
offset |= 1U;
other_parity_scan:
port = low + offset;
for (i = 0; i < remaining; i += 2, port += 2) {
if (unlikely(port >= high))
port -= remaining;
if (inet_is_local_reserved_port(net, port))
continue;
head = &hinfo->bhash[inet_bhashfn(net, port,
hinfo->bhash_size)];
spin_lock_bh(&head->lock);
if (inet_use_hash2_on_bind(sk)) {
if (inet_bhash2_addr_any_conflict(sk, port, l3mdev, relax, false))
goto next_port;
}
head2 = inet_bhashfn_portaddr(hinfo, sk, net, port);
spin_lock(&head2->lock);
tb2 = inet_bind2_bucket_find(head2, net, port, l3mdev, sk);
inet_bind_bucket_for_each(tb, &head->chain)
if (inet_bind_bucket_match(tb, net, port, l3mdev)) {
if (!inet_csk_bind_conflict(sk, tb, tb2,
relax, false))
goto success;
spin_unlock(&head2->lock);
goto next_port;
}
tb = NULL;
goto success;
next_port:
spin_unlock_bh(&head->lock);
cond_resched();
}
offset--;
if (!(offset & 1))
goto other_parity_scan;
if (attempt_half == 1) {
/* OK we now try the upper half of the range */
attempt_half = 2;
goto other_half_scan;
}
if (READ_ONCE(net->ipv4.sysctl_ip_autobind_reuse) && !relax) {
/* We still have a chance to connect to different destinations */
relax = true;
goto ports_exhausted;
}
return NULL;
success:
*port_ret = port;
*tb_ret = tb;
*tb2_ret = tb2;
*head2_ret = head2;
return head;
}
static inline int sk_reuseport_match(struct inet_bind_bucket *tb,
const struct sock *sk)
{
if (tb->fastreuseport <= 0)
return 0;
if (!sk->sk_reuseport)
return 0;
if (rcu_access_pointer(sk->sk_reuseport_cb))
return 0;
if (!uid_eq(tb->fastuid, sk_uid(sk)))
return 0;
/* We only need to check the rcv_saddr if this tb was once marked
* without fastreuseport and then was reset, as we can only know that
* the fast_*rcv_saddr doesn't have any conflicts with the socks on the
* owners list.
*/
if (tb->fastreuseport == FASTREUSEPORT_ANY)
return 1;
#if IS_ENABLED(CONFIG_IPV6)
if (tb->fast_sk_family == AF_INET6)
return ipv6_rcv_saddr_equal(&tb->fast_v6_rcv_saddr,
inet6_rcv_saddr(sk),
tb->fast_rcv_saddr,
sk->sk_rcv_saddr,
tb->fast_ipv6_only,
ipv6_only_sock(sk), true, false);
#endif
return ipv4_rcv_saddr_equal(tb->fast_rcv_saddr, sk->sk_rcv_saddr,
ipv6_only_sock(sk), true, false);
}
void inet_csk_update_fastreuse(const struct sock *sk,
struct inet_bind_bucket *tb,
struct inet_bind2_bucket *tb2)
{
bool reuse = sk->sk_reuse && sk->sk_state != TCP_LISTEN;
if (hlist_empty(&tb->bhash2)) {
tb->fastreuse = reuse;
if (sk->sk_reuseport) {
tb->fastreuseport = FASTREUSEPORT_ANY;
tb->fastuid = sk_uid(sk);
tb->fast_rcv_saddr = sk->sk_rcv_saddr;
tb->fast_ipv6_only = ipv6_only_sock(sk);
tb->fast_sk_family = sk->sk_family;
#if IS_ENABLED(CONFIG_IPV6)
tb->fast_v6_rcv_saddr = sk->sk_v6_rcv_saddr;
#endif
} else {
tb->fastreuseport = 0;
}
} else {
if (!reuse)
tb->fastreuse = 0;
if (sk->sk_reuseport) {
/* We didn't match or we don't have fastreuseport set on
* the tb, but we have sk_reuseport set on this socket
* and we know that there are no bind conflicts with
* this socket in this tb, so reset our tb's reuseport
* settings so that any subsequent sockets that match
* our current socket will be put on the fast path.
*
* If we reset we need to set FASTREUSEPORT_STRICT so we
* do extra checking for all subsequent sk_reuseport
* socks.
*/
if (!sk_reuseport_match(tb, sk)) {
tb->fastreuseport = FASTREUSEPORT_STRICT;
tb->fastuid = sk_uid(sk);
tb->fast_rcv_saddr = sk->sk_rcv_saddr;
tb->fast_ipv6_only = ipv6_only_sock(sk);
tb->fast_sk_family = sk->sk_family;
#if IS_ENABLED(CONFIG_IPV6)
tb->fast_v6_rcv_saddr = sk->sk_v6_rcv_saddr;
#endif
}
} else {
tb->fastreuseport = 0;
}
}
tb2->fastreuse = tb->fastreuse;
tb2->fastreuseport = tb->fastreuseport;
}
/* Obtain a reference to a local port for the given sock,
* if snum is zero it means select any available local port.
* We try to allocate an odd port (and leave even ports for connect())
*/
int inet_csk_get_port(struct sock *sk, unsigned short snum)
{
bool reuse = sk->sk_reuse && sk->sk_state != TCP_LISTEN;
bool found_port = false, check_bind_conflict = true;
bool bhash_created = false, bhash2_created = false;
struct inet_hashinfo *hinfo = tcp_get_hashinfo(sk);
int ret = -EADDRINUSE, port = snum, l3mdev;
struct inet_bind_hashbucket *head, *head2;
struct inet_bind2_bucket *tb2 = NULL;
struct inet_bind_bucket *tb = NULL;
bool head2_lock_acquired = false;
struct net *net = sock_net(sk);
l3mdev = inet_sk_bound_l3mdev(sk);
if (!port) {
head = inet_csk_find_open_port(sk, &tb, &tb2, &head2, &port);
if (!head)
return ret;
head2_lock_acquired = true;
if (tb && tb2)
goto success;
found_port = true;
} else {
head = &hinfo->bhash[inet_bhashfn(net, port,
hinfo->bhash_size)];
spin_lock_bh(&head->lock);
inet_bind_bucket_for_each(tb, &head->chain)
if (inet_bind_bucket_match(tb, net, port, l3mdev))
break;
}
if (!tb) {
tb = inet_bind_bucket_create(hinfo->bind_bucket_cachep, net,
head, port, l3mdev);
if (!tb)
goto fail_unlock;
bhash_created = true;
}
if (!found_port) {
if (!hlist_empty(&tb->bhash2)) {
if (sk->sk_reuse == SK_FORCE_REUSE ||
(tb->fastreuse > 0 && reuse) ||
sk_reuseport_match(tb, sk))
check_bind_conflict = false;
}
if (check_bind_conflict && inet_use_hash2_on_bind(sk)) {
if (inet_bhash2_addr_any_conflict(sk, port, l3mdev, true, true))
goto fail_unlock;
}
head2 = inet_bhashfn_portaddr(hinfo, sk, net, port);
spin_lock(&head2->lock);
head2_lock_acquired = true;
tb2 = inet_bind2_bucket_find(head2, net, port, l3mdev, sk);
}
if (!tb2) {
tb2 = inet_bind2_bucket_create(hinfo->bind2_bucket_cachep,
net, head2, tb, sk);
if (!tb2)
goto fail_unlock;
bhash2_created = true;
}
if (!found_port && check_bind_conflict) {
if (inet_csk_bind_conflict(sk, tb, tb2, true, true))
goto fail_unlock;
}
success:
inet_csk_update_fastreuse(sk, tb, tb2);
if (!inet_csk(sk)->icsk_bind_hash)
inet_bind_hash(sk, tb, tb2, port);
WARN_ON(inet_csk(sk)->icsk_bind_hash != tb);
WARN_ON(inet_csk(sk)->icsk_bind2_hash != tb2);
ret = 0;
fail_unlock:
if (ret) {
if (bhash2_created)
inet_bind2_bucket_destroy(hinfo->bind2_bucket_cachep, tb2);
if (bhash_created)
inet_bind_bucket_destroy(tb);
}
if (head2_lock_acquired)
spin_unlock(&head2->lock);
spin_unlock_bh(&head->lock);
return ret;
}
EXPORT_SYMBOL_GPL(inet_csk_get_port);
/*
* Wait for an incoming connection, avoid race conditions. This must be called
* with the socket locked.
*/
static int inet_csk_wait_for_connect(struct sock *sk, long timeo)
{
struct inet_connection_sock *icsk = inet_csk(sk);
DEFINE_WAIT(wait);
int err;
/*
* True wake-one mechanism for incoming connections: only
* one process gets woken up, not the 'whole herd'.
* Since we do not 'race & poll' for established sockets
* anymore, the common case will execute the loop only once.
*
* Subtle issue: "add_wait_queue_exclusive()" will be added
* after any current non-exclusive waiters, and we know that
* it will always _stay_ after any new non-exclusive waiters
* because all non-exclusive waiters are added at the
* beginning of the wait-queue. As such, it's ok to "drop"
* our exclusiveness temporarily when we get woken up without
* having to remove and re-insert us on the wait queue.
*/
for (;;) {
prepare_to_wait_exclusive(sk_sleep(sk), &wait,
TASK_INTERRUPTIBLE);
release_sock(sk);
if (reqsk_queue_empty(&icsk->icsk_accept_queue))
timeo = schedule_timeout(timeo);
sched_annotate_sleep();
lock_sock(sk);
err = 0;
if (!reqsk_queue_empty(&icsk->icsk_accept_queue))
break;
err = -EINVAL;
if (sk->sk_state != TCP_LISTEN)
break;
err = sock_intr_errno(timeo);
if (signal_pending(current))
break;
err = -EAGAIN;
if (!timeo)
break;
}
finish_wait(sk_sleep(sk), &wait);
return err;
}
/*
* This will accept the next outstanding connection.
*/
struct sock *inet_csk_accept(struct sock *sk, struct proto_accept_arg *arg)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct request_sock_queue *queue = &icsk->icsk_accept_queue;
struct request_sock *req;
struct sock *newsk;
int error;
lock_sock(sk);
/* We need to make sure that this socket is listening,
* and that it has something pending.
*/
error = -EINVAL;
if (sk->sk_state != TCP_LISTEN)
goto out_err;
/* Find already established connection */
if (reqsk_queue_empty(queue)) {
long timeo = sock_rcvtimeo(sk, arg->flags & O_NONBLOCK);
/* If this is a non blocking socket don't sleep */
error = -EAGAIN;
if (!timeo)
goto out_err;
error = inet_csk_wait_for_connect(sk, timeo);
if (error)
goto out_err;
}
req = reqsk_queue_remove(queue, sk);
arg->is_empty = reqsk_queue_empty(queue);
newsk = req->sk;
if (sk->sk_protocol == IPPROTO_TCP &&
tcp_rsk(req)->tfo_listener) {
spin_lock_bh(&queue->fastopenq.lock);
if (tcp_rsk(req)->tfo_listener) {
/* We are still waiting for the final ACK from 3WHS
* so can't free req now. Instead, we set req->sk to
* NULL to signify that the child socket is taken
* so reqsk_fastopen_remove() will free the req
* when 3WHS finishes (or is aborted).
*/
req->sk = NULL;
req = NULL;
}
spin_unlock_bh(&queue->fastopenq.lock);
}
release_sock(sk);
if (req)
reqsk_put(req);
inet_init_csk_locks(newsk);
return newsk;
out_err:
release_sock(sk);
arg->err = error;
return NULL;
}
/*
* Using different timers for retransmit, delayed acks and probes
* We may wish use just one timer maintaining a list of expire jiffies
* to optimize.
*/
void inet_csk_init_xmit_timers(struct sock *sk,
void (*retransmit_handler)(struct timer_list *t),
void (*delack_handler)(struct timer_list *t),
void (*keepalive_handler)(struct timer_list *t))
{
struct inet_connection_sock *icsk = inet_csk(sk);
timer_setup(&sk->tcp_retransmit_timer, retransmit_handler, 0);
timer_setup(&icsk->icsk_delack_timer, delack_handler, 0);
timer_setup(&icsk->icsk_keepalive_timer, keepalive_handler, 0);
icsk->icsk_pending = icsk->icsk_ack.pending = 0;
}
void inet_csk_clear_xmit_timers(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
smp_store_release(&icsk->icsk_pending, 0);
smp_store_release(&icsk->icsk_ack.pending, 0);
sk_stop_timer(sk, &sk->tcp_retransmit_timer);
sk_stop_timer(sk, &icsk->icsk_delack_timer);
sk_stop_timer(sk, &icsk->icsk_keepalive_timer);
}
void inet_csk_clear_xmit_timers_sync(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
/* ongoing timer handlers need to acquire socket lock. */
sock_not_owned_by_me(sk);
smp_store_release(&icsk->icsk_pending, 0);
smp_store_release(&icsk->icsk_ack.pending, 0);
sk_stop_timer_sync(sk, &sk->tcp_retransmit_timer);
sk_stop_timer_sync(sk, &icsk->icsk_delack_timer);
sk_stop_timer_sync(sk, &icsk->icsk_keepalive_timer);
}
struct dst_entry *inet_csk_route_req(const struct sock *sk,
struct flowi4 *fl4,
const struct request_sock *req)
{
const struct inet_request_sock *ireq = inet_rsk(req);
struct net *net = read_pnet(&ireq->ireq_net);
struct ip_options_rcu *opt;
struct rtable *rt;
rcu_read_lock();
opt = rcu_dereference(ireq->ireq_opt);
flowi4_init_output(fl4, ireq->ir_iif, ireq->ir_mark,
ip_sock_rt_tos(sk), ip_sock_rt_scope(sk),
sk->sk_protocol, inet_sk_flowi_flags(sk),
(opt && opt->opt.srr) ? opt->opt.faddr : ireq->ir_rmt_addr,
ireq->ir_loc_addr, ireq->ir_rmt_port,
htons(ireq->ir_num), sk_uid(sk));
security_req_classify_flow(req, flowi4_to_flowi_common(fl4));
rt = ip_route_output_flow(net, fl4, sk);
if (IS_ERR(rt))
goto no_route;
if (opt && opt->opt.is_strictroute && rt->rt_uses_gateway)
goto route_err;
rcu_read_unlock();
return &rt->dst;
route_err:
ip_rt_put(rt);
no_route:
rcu_read_unlock();
__IP_INC_STATS(net, IPSTATS_MIB_OUTNOROUTES);
return NULL;
}
struct dst_entry *inet_csk_route_child_sock(const struct sock *sk,
struct sock *newsk,
const struct request_sock *req)
{
const struct inet_request_sock *ireq = inet_rsk(req);
struct net *net = read_pnet(&ireq->ireq_net);
struct inet_sock *newinet = inet_sk(newsk);
struct ip_options_rcu *opt;
struct flowi4 *fl4;
struct rtable *rt;
opt = rcu_dereference(ireq->ireq_opt);
fl4 = &newinet->cork.fl.u.ip4;
flowi4_init_output(fl4, ireq->ir_iif, ireq->ir_mark,
ip_sock_rt_tos(sk), ip_sock_rt_scope(sk),
sk->sk_protocol, inet_sk_flowi_flags(sk),
(opt && opt->opt.srr) ? opt->opt.faddr : ireq->ir_rmt_addr,
ireq->ir_loc_addr, ireq->ir_rmt_port,
htons(ireq->ir_num), sk_uid(sk));
security_req_classify_flow(req, flowi4_to_flowi_common(fl4));
rt = ip_route_output_flow(net, fl4, sk);
if (IS_ERR(rt))
goto no_route;
if (opt && opt->opt.is_strictroute && rt->rt_uses_gateway)
goto route_err;
return &rt->dst;
route_err:
ip_rt_put(rt);
no_route:
__IP_INC_STATS(net, IPSTATS_MIB_OUTNOROUTES);
return NULL;
}
EXPORT_SYMBOL_GPL(inet_csk_route_child_sock);
/* Decide when to expire the request and when to resend SYN-ACK */
static void syn_ack_recalc(struct request_sock *req,
const int max_syn_ack_retries,
const u8 rskq_defer_accept,
int *expire, int *resend)
{
if (!rskq_defer_accept) {
*expire = req->num_timeout >= max_syn_ack_retries;
*resend = 1;
return;
}
*expire = req->num_timeout >= max_syn_ack_retries &&
(!inet_rsk(req)->acked || req->num_timeout >= rskq_defer_accept);
/* Do not resend while waiting for data after ACK,
* start to resend on end of deferring period to give
* last chance for data or ACK to create established socket.
*/
*resend = !inet_rsk(req)->acked ||
req->num_timeout >= rskq_defer_accept - 1;
}
static struct request_sock *
reqsk_alloc_noprof(const struct request_sock_ops *ops, struct sock *sk_listener,
bool attach_listener)
{
struct request_sock *req;
req = kmem_cache_alloc_noprof(ops->slab, GFP_ATOMIC | __GFP_NOWARN);
if (!req)
return NULL;
req->rsk_listener = NULL;
if (attach_listener) {
if (unlikely(!refcount_inc_not_zero(&sk_listener->sk_refcnt))) {
kmem_cache_free(ops->slab, req);
return NULL;
}
req->rsk_listener = sk_listener;
}
req->rsk_ops = ops;
req_to_sk(req)->sk_prot = sk_listener->sk_prot;
sk_node_init(&req_to_sk(req)->sk_node);
sk_tx_queue_clear(req_to_sk(req));
req->saved_syn = NULL;
req->syncookie = 0;
req->num_timeout = 0;
req->num_retrans = 0;
req->sk = NULL;
refcount_set(&req->rsk_refcnt, 0);
return req;
}
#define reqsk_alloc(...) alloc_hooks(reqsk_alloc_noprof(__VA_ARGS__))
struct request_sock *inet_reqsk_alloc(const struct request_sock_ops *ops,
struct sock *sk_listener,
bool attach_listener)
{
struct request_sock *req = reqsk_alloc(ops, sk_listener,
attach_listener);
if (req) {
struct inet_request_sock *ireq = inet_rsk(req);
ireq->ireq_opt = NULL;
#if IS_ENABLED(CONFIG_IPV6)
ireq->pktopts = NULL;
#endif
atomic64_set(&ireq->ir_cookie, 0);
ireq->ireq_state = TCP_NEW_SYN_RECV;
write_pnet(&ireq->ireq_net, sock_net(sk_listener));
ireq->ireq_family = sk_listener->sk_family;
}
return req;
}
EXPORT_SYMBOL(inet_reqsk_alloc);
void __reqsk_free(struct request_sock *req)
{
req->rsk_ops->destructor(req);
if (req->rsk_listener)
sock_put(req->rsk_listener);
kfree(req->saved_syn);
kmem_cache_free(req->rsk_ops->slab, req);
}
EXPORT_SYMBOL_GPL(__reqsk_free);
static struct request_sock *inet_reqsk_clone(struct request_sock *req,
struct sock *sk)
{
struct sock *req_sk, *nreq_sk;
struct request_sock *nreq;
nreq = kmem_cache_alloc(req->rsk_ops->slab, GFP_ATOMIC | __GFP_NOWARN);
if (!nreq) {
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMIGRATEREQFAILURE);
/* paired with refcount_inc_not_zero() in reuseport_migrate_sock() */
sock_put(sk);
return NULL;
}
req_sk = req_to_sk(req);
nreq_sk = req_to_sk(nreq);
memcpy(nreq_sk, req_sk,
offsetof(struct sock, sk_dontcopy_begin));
unsafe_memcpy(&nreq_sk->sk_dontcopy_end, &req_sk->sk_dontcopy_end,
req->rsk_ops->obj_size - offsetof(struct sock, sk_dontcopy_end),
/* alloc is larger than struct, see above */);
sk_node_init(&nreq_sk->sk_node);
nreq_sk->sk_tx_queue_mapping = req_sk->sk_tx_queue_mapping;
#ifdef CONFIG_SOCK_RX_QUEUE_MAPPING
nreq_sk->sk_rx_queue_mapping = req_sk->sk_rx_queue_mapping;
#endif
nreq_sk->sk_incoming_cpu = req_sk->sk_incoming_cpu;
nreq->rsk_listener = sk;
/* We need not acquire fastopenq->lock
* because the child socket is locked in inet_csk_listen_stop().
*/
if (sk->sk_protocol == IPPROTO_TCP && tcp_rsk(nreq)->tfo_listener)
rcu_assign_pointer(tcp_sk(nreq->sk)->fastopen_rsk, nreq);
return nreq;
}
static void reqsk_queue_migrated(struct request_sock_queue *queue,
const struct request_sock *req)
{
if (req->num_timeout == 0)
atomic_inc(&queue->young);
atomic_inc(&queue->qlen);
}
static void reqsk_migrate_reset(struct request_sock *req)
{
req->saved_syn = NULL;
#if IS_ENABLED(CONFIG_IPV6)
inet_rsk(req)->ipv6_opt = NULL;
inet_rsk(req)->pktopts = NULL;
#else
inet_rsk(req)->ireq_opt = NULL;
#endif
}
/* return true if req was found in the ehash table */
static bool reqsk_queue_unlink(struct request_sock *req)
{
struct sock *sk = req_to_sk(req);
bool found = false;
if (sk_hashed(sk)) {
struct inet_hashinfo *hashinfo = tcp_get_hashinfo(sk);
spinlock_t *lock;
lock = inet_ehash_lockp(hashinfo, req->rsk_hash);
spin_lock(lock);
found = __sk_nulls_del_node_init_rcu(sk);
spin_unlock(lock);
}
return found;
}
static bool __inet_csk_reqsk_queue_drop(struct sock *sk,
struct request_sock *req,
bool from_timer)
{
bool unlinked = reqsk_queue_unlink(req);
if (!from_timer && timer_delete_sync(&req->rsk_timer))
reqsk_put(req);
if (unlinked) {
reqsk_queue_removed(&inet_csk(sk)->icsk_accept_queue, req);
reqsk_put(req);
}
return unlinked;
}
bool inet_csk_reqsk_queue_drop(struct sock *sk, struct request_sock *req)
{
return __inet_csk_reqsk_queue_drop(sk, req, false);
}
void inet_csk_reqsk_queue_drop_and_put(struct sock *sk, struct request_sock *req)
{
inet_csk_reqsk_queue_drop(sk, req);
reqsk_put(req);
}
static void reqsk_timer_handler(struct timer_list *t)
{
struct request_sock *req = timer_container_of(req, t, rsk_timer);
struct request_sock *nreq = NULL, *oreq = req;
struct sock *sk_listener = req->rsk_listener;
struct inet_connection_sock *icsk;
struct request_sock_queue *queue;
struct net *net;
int max_syn_ack_retries, qlen, expire = 0, resend = 0;
if (inet_sk_state_load(sk_listener) != TCP_LISTEN) {
struct sock *nsk;
nsk = reuseport_migrate_sock(sk_listener, req_to_sk(req), NULL);
if (!nsk)
goto drop;
nreq = inet_reqsk_clone(req, nsk);
if (!nreq)
goto drop;
/* The new timer for the cloned req can decrease the 2
* by calling inet_csk_reqsk_queue_drop_and_put(), so
* hold another count to prevent use-after-free and
* call reqsk_put() just before return.
*/
refcount_set(&nreq->rsk_refcnt, 2 + 1);
timer_setup(&nreq->rsk_timer, reqsk_timer_handler, TIMER_PINNED);
reqsk_queue_migrated(&inet_csk(nsk)->icsk_accept_queue, req);
req = nreq;
sk_listener = nsk;
}
icsk = inet_csk(sk_listener);
net = sock_net(sk_listener);
max_syn_ack_retries = READ_ONCE(icsk->icsk_syn_retries) ? :
READ_ONCE(net->ipv4.sysctl_tcp_synack_retries);
/* Normally all the openreqs are young and become mature
* (i.e. converted to established socket) for first timeout.
* If synack was not acknowledged for 1 second, it means
* one of the following things: synack was lost, ack was lost,
* rtt is high or nobody planned to ack (i.e. synflood).
* When server is a bit loaded, queue is populated with old
* open requests, reducing effective size of queue.
* When server is well loaded, queue size reduces to zero
* after several minutes of work. It is not synflood,
* it is normal operation. The solution is pruning
* too old entries overriding normal timeout, when
* situation becomes dangerous.
*
* Essentially, we reserve half of room for young
* embrions; and abort old ones without pity, if old
* ones are about to clog our table.
*/
queue = &icsk->icsk_accept_queue;
qlen = reqsk_queue_len(queue);
if ((qlen << 1) > max(8U, READ_ONCE(sk_listener->sk_max_ack_backlog))) {
int young = reqsk_queue_len_young(queue) << 1;
while (max_syn_ack_retries > 2) {
if (qlen < young)
break;
max_syn_ack_retries--;
young <<= 1;
}
}
syn_ack_recalc(req, max_syn_ack_retries, READ_ONCE(queue->rskq_defer_accept),
&expire, &resend);
tcp_syn_ack_timeout(req);
if (!expire &&
(!resend ||
!tcp_rtx_synack(sk_listener, req) ||
inet_rsk(req)->acked)) {
if (req->num_retrans > 1 && tcp_rsk(req)->accecn_ok)
tcp_rsk(req)->accecn_fail_mode |= TCP_ACCECN_ACE_FAIL_SEND;
if (req->num_timeout++ == 0)
atomic_dec(&queue->young);
mod_timer(&req->rsk_timer, jiffies + tcp_reqsk_timeout(req));
if (!nreq)
return;
if (!inet_ehash_insert(req_to_sk(nreq), req_to_sk(oreq), NULL)) {
/* delete timer */
__inet_csk_reqsk_queue_drop(sk_listener, nreq, true);
goto no_ownership;
}
__NET_INC_STATS(net, LINUX_MIB_TCPMIGRATEREQSUCCESS);
reqsk_migrate_reset(oreq);
reqsk_queue_removed(&inet_csk(oreq->rsk_listener)->icsk_accept_queue, oreq);
reqsk_put(oreq);
reqsk_put(nreq);
return;
}
/* Even if we can clone the req, we may need not retransmit any more
* SYN+ACKs (nreq->num_timeout > max_syn_ack_retries, etc), or another
* CPU may win the "own_req" race so that inet_ehash_insert() fails.
*/
if (nreq) {
__NET_INC_STATS(net, LINUX_MIB_TCPMIGRATEREQFAILURE);
no_ownership:
reqsk_migrate_reset(nreq);
reqsk_queue_removed(queue, nreq);
__reqsk_free(nreq);
}
drop:
__inet_csk_reqsk_queue_drop(sk_listener, oreq, true);
reqsk_put(oreq);
}
static bool reqsk_queue_hash_req(struct request_sock *req)
{
bool found_dup_sk = false;
if (!inet_ehash_insert(req_to_sk(req), NULL, &found_dup_sk))
return false;
/* The timer needs to be setup after a successful insertion. */
req->timeout = tcp_timeout_init((struct sock *)req);
timer_setup(&req->rsk_timer, reqsk_timer_handler, TIMER_PINNED);
mod_timer(&req->rsk_timer, jiffies + req->timeout);
/* before letting lookups find us, make sure all req fields
* are committed to memory and refcnt initialized.
*/
smp_wmb();
refcount_set(&req->rsk_refcnt, 2 + 1);
return true;
}
bool inet_csk_reqsk_queue_hash_add(struct sock *sk, struct request_sock *req)
{
if (!reqsk_queue_hash_req(req))
return false;
inet_csk_reqsk_queue_added(sk);
return true;
}
static void inet_clone_ulp(const struct request_sock *req, struct sock *newsk,
const gfp_t priority)
{
struct inet_connection_sock *icsk = inet_csk(newsk);
if (!icsk->icsk_ulp_ops)
return;
icsk->icsk_ulp_ops->clone(req, newsk, priority);
}
/**
* inet_csk_clone_lock - clone an inet socket, and lock its clone
* @sk: the socket to clone
* @req: request_sock
* @priority: for allocation (%GFP_KERNEL, %GFP_ATOMIC, etc)
*
* Caller must unlock socket even in error path (bh_unlock_sock(newsk))
*/
struct sock *inet_csk_clone_lock(const struct sock *sk,
const struct request_sock *req,
const gfp_t priority)
{
struct sock *newsk = sk_clone_lock(sk, priority);
struct inet_connection_sock *newicsk;
const struct inet_request_sock *ireq;
struct inet_sock *newinet;
if (!newsk)
return NULL;
newicsk = inet_csk(newsk);
newinet = inet_sk(newsk);
ireq = inet_rsk(req);
newicsk->icsk_bind_hash = NULL;
newicsk->icsk_bind2_hash = NULL;
newinet->inet_dport = ireq->ir_rmt_port;
newinet->inet_num = ireq->ir_num;
newinet->inet_sport = htons(ireq->ir_num);
newsk->sk_bound_dev_if = ireq->ir_iif;
newsk->sk_daddr = ireq->ir_rmt_addr;
newsk->sk_rcv_saddr = ireq->ir_loc_addr;
newinet->inet_saddr = ireq->ir_loc_addr;
#if IS_ENABLED(CONFIG_IPV6)
newsk->sk_v6_daddr = ireq->ir_v6_rmt_addr;
newsk->sk_v6_rcv_saddr = ireq->ir_v6_loc_addr;
#endif
/* listeners have SOCK_RCU_FREE, not the children */
sock_reset_flag(newsk, SOCK_RCU_FREE);
inet_sk(newsk)->mc_list = NULL;
newsk->sk_mark = inet_rsk(req)->ir_mark;
atomic64_set(&newsk->sk_cookie,
atomic64_read(&inet_rsk(req)->ir_cookie));
newicsk->icsk_retransmits = 0;
newicsk->icsk_backoff = 0;
newicsk->icsk_probes_out = 0;
newicsk->icsk_probes_tstamp = 0;
/* Deinitialize accept_queue to trap illegal accesses. */
memset(&newicsk->icsk_accept_queue, 0,
sizeof(newicsk->icsk_accept_queue));
inet_sk_set_state(newsk, TCP_SYN_RECV);
inet_clone_ulp(req, newsk, priority);
security_inet_csk_clone(newsk, req);
return newsk;
}
/*
* At this point, there should be no process reference to this
* socket, and thus no user references at all. Therefore we
* can assume the socket waitqueue is inactive and nobody will
* try to jump onto it.
*/
void inet_csk_destroy_sock(struct sock *sk)
{
WARN_ON(sk->sk_state != TCP_CLOSE);
WARN_ON(!sock_flag(sk, SOCK_DEAD));
/* It cannot be in hash table! */
WARN_ON(!sk_unhashed(sk));
/* If it has not 0 inet_sk(sk)->inet_num, it must be bound */
WARN_ON(inet_sk(sk)->inet_num && !inet_csk(sk)->icsk_bind_hash);
sk->sk_prot->destroy(sk);
sk_stream_kill_queues(sk);
xfrm_sk_free_policy(sk);
tcp_orphan_count_dec();
sock_put(sk);
}
EXPORT_SYMBOL(inet_csk_destroy_sock);
void inet_csk_prepare_for_destroy_sock(struct sock *sk)
{
/* The below has to be done to allow calling inet_csk_destroy_sock */
sock_set_flag(sk, SOCK_DEAD);
tcp_orphan_count_inc();
}
/* This function allows to force a closure of a socket after the call to
* tcp_create_openreq_child().
*/
void inet_csk_prepare_forced_close(struct sock *sk)
__releases(&sk->sk_lock.slock)
{
/* sk_clone_lock locked the socket and set refcnt to 2 */
bh_unlock_sock(sk);
sock_put(sk);
inet_csk_prepare_for_destroy_sock(sk);
inet_sk(sk)->inet_num = 0;
}
EXPORT_SYMBOL(inet_csk_prepare_forced_close);
static int inet_ulp_can_listen(const struct sock *sk)
{
const struct inet_connection_sock *icsk = inet_csk(sk);
if (icsk->icsk_ulp_ops && !icsk->icsk_ulp_ops->clone)
return -EINVAL;
return 0;
}
static void reqsk_queue_alloc(struct request_sock_queue *queue)
{
queue->fastopenq.rskq_rst_head = NULL;
queue->fastopenq.rskq_rst_tail = NULL;
queue->fastopenq.qlen = 0;
queue->rskq_accept_head = NULL;
}
int inet_csk_listen_start(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct inet_sock *inet = inet_sk(sk);
int err;
err = inet_ulp_can_listen(sk);
if (unlikely(err))
return err;
reqsk_queue_alloc(&icsk->icsk_accept_queue);
sk->sk_ack_backlog = 0;
inet_csk_delack_init(sk);
/* There is race window here: we announce ourselves listening,
* but this transition is still not validated by get_port().
* It is OK, because this socket enters to hash table only
* after validation is complete.
*/
inet_sk_state_store(sk, TCP_LISTEN);
err = sk->sk_prot->get_port(sk, inet->inet_num);
if (!err) {
inet->inet_sport = htons(inet->inet_num);
sk_dst_reset(sk);
err = sk->sk_prot->hash(sk);
if (likely(!err))
return 0;
}
inet_sk_set_state(sk, TCP_CLOSE);
return err;
}
static void inet_child_forget(struct sock *sk, struct request_sock *req,
struct sock *child)
{
sk->sk_prot->disconnect(child, O_NONBLOCK);
sock_orphan(child);
tcp_orphan_count_inc();
if (sk->sk_protocol == IPPROTO_TCP && tcp_rsk(req)->tfo_listener) {
BUG_ON(rcu_access_pointer(tcp_sk(child)->fastopen_rsk) != req);
BUG_ON(sk != req->rsk_listener);
/* Paranoid, to prevent race condition if
* an inbound pkt destined for child is
* blocked by sock lock in tcp_v4_rcv().
* Also to satisfy an assertion in
* tcp_v4_destroy_sock().
*/
RCU_INIT_POINTER(tcp_sk(child)->fastopen_rsk, NULL);
}
inet_csk_destroy_sock(child);
}
struct sock *inet_csk_reqsk_queue_add(struct sock *sk,
struct request_sock *req,
struct sock *child)
{
struct request_sock_queue *queue = &inet_csk(sk)->icsk_accept_queue;
spin_lock(&queue->rskq_lock);
if (unlikely(sk->sk_state != TCP_LISTEN)) {
inet_child_forget(sk, req, child);
child = NULL;
} else {
req->sk = child;
req->dl_next = NULL;
if (queue->rskq_accept_head == NULL)
WRITE_ONCE(queue->rskq_accept_head, req);
else
queue->rskq_accept_tail->dl_next = req;
queue->rskq_accept_tail = req;
sk_acceptq_added(sk);
}
spin_unlock(&queue->rskq_lock);
return child;
}
EXPORT_SYMBOL(inet_csk_reqsk_queue_add);
struct sock *inet_csk_complete_hashdance(struct sock *sk, struct sock *child,
struct request_sock *req, bool own_req)
{
if (own_req) {
inet_csk_reqsk_queue_drop(req->rsk_listener, req);
reqsk_queue_removed(&inet_csk(req->rsk_listener)->icsk_accept_queue, req);
if (sk != req->rsk_listener) {
/* another listening sk has been selected,
* migrate the req to it.
*/
struct request_sock *nreq;
/* hold a refcnt for the nreq->rsk_listener
* which is assigned in inet_reqsk_clone()
*/
sock_hold(sk);
nreq = inet_reqsk_clone(req, sk);
if (!nreq) {
inet_child_forget(sk, req, child);
goto child_put;
}
refcount_set(&nreq->rsk_refcnt, 1);
if (inet_csk_reqsk_queue_add(sk, nreq, child)) {
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMIGRATEREQSUCCESS);
reqsk_migrate_reset(req);
reqsk_put(req);
return child;
}
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMIGRATEREQFAILURE);
reqsk_migrate_reset(nreq);
__reqsk_free(nreq);
} else if (inet_csk_reqsk_queue_add(sk, req, child)) {
return child;
}
}
/* Too bad, another child took ownership of the request, undo. */
child_put:
bh_unlock_sock(child);
sock_put(child);
return NULL;
}
/*
* This routine closes sockets which have been at least partially
* opened, but not yet accepted.
*/
void inet_csk_listen_stop(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct request_sock_queue *queue = &icsk->icsk_accept_queue;
struct request_sock *next, *req;
/* Following specs, it would be better either to send FIN
* (and enter FIN-WAIT-1, it is normal close)
* or to send active reset (abort).
* Certainly, it is pretty dangerous while synflood, but it is
* bad justification for our negligence 8)
* To be honest, we are not able to make either
* of the variants now. --ANK
*/
while ((req = reqsk_queue_remove(queue, sk)) != NULL) {
struct sock *child = req->sk, *nsk;
struct request_sock *nreq;
local_bh_disable();
bh_lock_sock(child);
WARN_ON(sock_owned_by_user(child));
sock_hold(child);
nsk = reuseport_migrate_sock(sk, child, NULL);
if (nsk) {
nreq = inet_reqsk_clone(req, nsk);
if (nreq) {
refcount_set(&nreq->rsk_refcnt, 1);
rcu_read_lock();
if (inet_csk_reqsk_queue_add(nsk, nreq, child)) {
__NET_INC_STATS(sock_net(nsk),
LINUX_MIB_TCPMIGRATEREQSUCCESS);
reqsk_migrate_reset(req);
READ_ONCE(nsk->sk_data_ready)(nsk);
} else {
__NET_INC_STATS(sock_net(nsk),
LINUX_MIB_TCPMIGRATEREQFAILURE);
reqsk_migrate_reset(nreq);
__reqsk_free(nreq);
}
rcu_read_unlock();
/* inet_csk_reqsk_queue_add() has already
* called inet_child_forget() on failure case.
*/
goto skip_child_forget;
}
}
inet_child_forget(sk, req, child);
skip_child_forget:
reqsk_put(req);
bh_unlock_sock(child);
local_bh_enable();
sock_put(child);
cond_resched();
}
if (queue->fastopenq.rskq_rst_head) {
/* Free all the reqs queued in rskq_rst_head. */
spin_lock_bh(&queue->fastopenq.lock);
req = queue->fastopenq.rskq_rst_head;
queue->fastopenq.rskq_rst_head = NULL;
spin_unlock_bh(&queue->fastopenq.lock);
while (req != NULL) {
next = req->dl_next;
reqsk_put(req);
req = next;
}
}
WARN_ON_ONCE(sk->sk_ack_backlog);
}
static struct dst_entry *inet_csk_rebuild_route(struct sock *sk, struct flowi *fl)
{
const struct inet_sock *inet = inet_sk(sk);
struct flowi4 *fl4;
struct rtable *rt;
rcu_read_lock();
fl4 = &fl->u.ip4;
inet_sk_init_flowi4(inet, fl4);
rt = ip_route_output_flow(sock_net(sk), fl4, sk);
if (IS_ERR(rt))
rt = NULL;
if (rt)
sk_setup_caps(sk, &rt->dst);
rcu_read_unlock();
return &rt->dst;
}
struct dst_entry *inet_csk_update_pmtu(struct sock *sk, u32 mtu)
{
struct dst_entry *dst = __sk_dst_check(sk, 0);
struct inet_sock *inet = inet_sk(sk);
if (!dst) {
dst = inet_csk_rebuild_route(sk, &inet->cork.fl);
if (!dst)
goto out;
}
dst->ops->update_pmtu(dst, sk, NULL, mtu, true);
dst = __sk_dst_check(sk, 0);
if (!dst)
dst = inet_csk_rebuild_route(sk, &inet->cork.fl);
out:
return dst;
}
]
|
| 22/2 |
2026/05/17 01:37 |
llm |
expert |
0m
Model:
Tokens:
input: 51250
output: 21
thoughts: 896
Thoughts: **Analyzing a Potential Kernel Bug**
Okay, let's break down this `reqsk_timer_handler` function. My initial thought is to understand its flow and, specifically, how it interacts with the `reqsk_put` function. It looks like the core logic revolves around handling retransmissions of SYN+ACKs and potentially migrating request sockets. I see that the timer handler gets a `request_sock` structure via `timer_container_of`. The key seems to be in the conditional block where retransmissions are decided. If a SYN+ACK resend isn't necessary or has been acknowledged, then there is an increment and a re-arm of the timer, or migration logic is triggered.
At the very end of this handler, regardless of the path, `reqsk_put(oreq)` is called. And, I know that `reqsk_put` itself calls `refcount_dec_and_test`. My concern immediately sharpens: if the reference count somehow underflows, the code ends up calling `reqsk_free`, which could lead to a use-after-free scenario, or worse. The compiler, in this case, might optimize that final `reqsk_put` call into a tail call, meaning it would be effectively replacing the current stack frame with a frame that calls `refcount_warn_saturate`.
If an underflow does occur, the resulting stack trace might show `call_timer_fn` calling `refcount_warn_saturate` instead of the more expected path. This sounds like it could be a significant bug. My gut tells me this could be a race condition.
I need to confirm if other timer handlers have the same pattern of calling `reqsk_put` or `sock_put` at the very end. `reqsk_timer_handler` looks like a prime candidate for this type of issue, so I'm going to scan the code for more such cases. I'll need to check the entire kernel to see if this pattern is repeated. What about something like `tcp_write_timer`? I'll check in `net/ipv4/tcp_timer.c` to see if there are any other potential issues.
|
| 23/2 |
2026/05/17 01:37 |
tool |
grepper |
0m
Args: map[Expression:void tcp_write_timer\(]
Results: map[Output:net/ipv4/tcp_timer.c=694=void tcp_write_timer_handler(struct sock *sk)
--
net/ipv4/tcp_timer.c-728-
net/ipv4/tcp_timer.c:729:static void tcp_write_timer(struct timer_list *t)
net/ipv4/tcp_timer.c-730-{
]
|
| 24/2 |
2026/05/17 01:37 |
llm |
expert |
0m
Model:
Tokens:
input: 52741
output: 25
thoughts: 12
Thoughts: Okay, let's dive into this. Here's my thought process summarized, assuming I'm already pretty familiar with network internals:
**Checking `tcp_write_timer`**
Alright, I need to investigate the `tcp_write_timer`. This suggests we're dealing with issues related to TCP write operations, potentially involving retransmissions, keep-alives, or connection timeouts. My first instinct is to understand its role within the larger TCP stack context. I need to know: what triggers this timer? What actions does it perform when it fires? Where does this timer fit in the state machine of a TCP connection? Is it directly associated with a specific socket or flow control window? Are we seeing unusual behavior in the connections? I would check the code where it is defined, and how it is connected to the other components. Are there any known issues reported related to this timer? What are the potential causes for errors detected by the associated function? Depending on the specifics, I'd want to check associated data structures, and the current state of any TCP connection experiencing problems. Let's see... let's see what we can find.
|
| 25/2 |
2026/05/17 01:37 |
tool |
read-file |
0m
Args: map[File:net/ipv4/tcp_timer.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
* INET An implementation of the TCP/IP protocol suite for the LINUX
* operating system. INET is implemented using the BSD Socket
* interface as the means of communication with the user level.
*
* Implementation of the Transmission Control Protocol(TCP).
*
* Authors: Ross Biro
* Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG>
* Mark Evans, <evansmp@uhura.aston.ac.uk>
* Corey Minyard <wf-rch!minyard@relay.EU.net>
* Florian La Roche, <flla@stud.uni-sb.de>
* Charles Hedrick, <hedrick@klinzhai.rutgers.edu>
* Linus Torvalds, <torvalds@cs.helsinki.fi>
* Alan Cox, <gw4pts@gw4pts.ampr.org>
* Matthew Dillon, <dillon@apollo.west.oic.com>
* Arnt Gulbrandsen, <agulbra@nvg.unit.no>
* Jorge Cwik, <jorge@laser.satlink.net>
*/
#include <linux/module.h>
#include <linux/gfp.h>
#include <net/tcp.h>
#include <net/tcp_ecn.h>
#include <net/rstreason.h>
static u32 tcp_clamp_rto_to_user_timeout(const struct sock *sk)
{
const struct inet_connection_sock *icsk = inet_csk(sk);
const struct tcp_sock *tp = tcp_sk(sk);
u32 elapsed, user_timeout;
s32 remaining;
user_timeout = READ_ONCE(icsk->icsk_user_timeout);
if (!user_timeout)
return icsk->icsk_rto;
elapsed = tcp_time_stamp_ts(tp) - tp->retrans_stamp;
if (tp->tcp_usec_ts)
elapsed /= USEC_PER_MSEC;
remaining = user_timeout - elapsed;
if (remaining <= 0)
return 1; /* user timeout has passed; fire ASAP */
return min_t(u32, icsk->icsk_rto, msecs_to_jiffies(remaining));
}
u32 tcp_clamp_probe0_to_user_timeout(const struct sock *sk, u32 when)
{
const struct inet_connection_sock *icsk = inet_csk(sk);
u32 remaining, user_timeout;
s32 elapsed;
user_timeout = READ_ONCE(icsk->icsk_user_timeout);
if (!user_timeout || !icsk->icsk_probes_tstamp)
return when;
elapsed = tcp_jiffies32 - icsk->icsk_probes_tstamp;
if (unlikely(elapsed < 0))
elapsed = 0;
remaining = msecs_to_jiffies(user_timeout) - elapsed;
remaining = max_t(u32, remaining, TCP_TIMEOUT_MIN);
return min_t(u32, remaining, when);
}
/**
* tcp_write_err() - close socket and save error info
* @sk: The socket the error has appeared on.
*
* Returns: Nothing (void)
*/
static void tcp_write_err(struct sock *sk)
{
tcp_done_with_error(sk, READ_ONCE(sk->sk_err_soft) ? : ETIMEDOUT);
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONTIMEOUT);
}
/**
* tcp_out_of_resources() - Close socket if out of resources
* @sk: pointer to current socket
* @do_reset: send a last packet with reset flag
*
* Do not allow orphaned sockets to eat all our resources.
* This is direct violation of TCP specs, but it is required
* to prevent DoS attacks. It is called when a retransmission timeout
* or zero probe timeout occurs on orphaned socket.
*
* Also close if our net namespace is exiting; in that case there is no
* hope of ever communicating again since all netns interfaces are already
* down (or about to be down), and we need to release our dst references,
* which have been moved to the netns loopback interface, so the namespace
* can finish exiting. This condition is only possible if we are a kernel
* socket, as those do not hold references to the namespace.
*
* Criteria is still not confirmed experimentally and may change.
* We kill the socket, if:
* 1. If number of orphaned sockets exceeds an administratively configured
* limit.
* 2. If we have strong memory pressure.
* 3. If our net namespace is exiting.
*/
static int tcp_out_of_resources(struct sock *sk, bool do_reset)
{
struct tcp_sock *tp = tcp_sk(sk);
int shift = 0;
/* If peer does not open window for long time, or did not transmit
* anything for long time, penalize it. */
if ((s32)(tcp_jiffies32 - tp->lsndtime) > 2*tcp_rto_max(sk) || !do_reset)
shift++;
/* If some dubious ICMP arrived, penalize even more. */
if (READ_ONCE(sk->sk_err_soft))
shift++;
if (tcp_check_oom(sk, shift)) {
/* Catch exceptional cases, when connection requires reset.
* 1. Last segment was sent recently. */
if ((s32)(tcp_jiffies32 - tp->lsndtime) <= TCP_TIMEWAIT_LEN ||
/* 2. Window is closed. */
(!tp->snd_wnd && !tp->packets_out))
do_reset = true;
if (do_reset)
tcp_send_active_reset(sk, GFP_ATOMIC,
SK_RST_REASON_TCP_ABORT_ON_MEMORY);
tcp_done(sk);
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONMEMORY);
return 1;
}
if (!check_net(sock_net(sk))) {
/* Not possible to send reset; just close */
tcp_done(sk);
return 1;
}
return 0;
}
/**
* tcp_orphan_retries() - Returns maximal number of retries on an orphaned socket
* @sk: Pointer to the current socket.
* @alive: bool, socket alive state
*/
static int tcp_orphan_retries(struct sock *sk, bool alive)
{
int retries = READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_orphan_retries); /* May be zero. */
/* We know from an ICMP that something is wrong. */
if (READ_ONCE(sk->sk_err_soft) && !alive)
retries = 0;
/* However, if socket sent something recently, select some safe
* number of retries. 8 corresponds to >100 seconds with minimal
* RTO of 200msec. */
if (retries == 0 && alive)
retries = 8;
return retries;
}
static void tcp_mtu_probing(struct inet_connection_sock *icsk, struct sock *sk)
{
const struct net *net = sock_net(sk);
int mss;
/* Black hole detection */
if (!READ_ONCE(net->ipv4.sysctl_tcp_mtu_probing))
return;
if (!icsk->icsk_mtup.enabled) {
icsk->icsk_mtup.enabled = 1;
icsk->icsk_mtup.probe_timestamp = tcp_jiffies32;
} else {
mss = tcp_mtu_to_mss(sk, icsk->icsk_mtup.search_low) >> 1;
mss = min(READ_ONCE(net->ipv4.sysctl_tcp_base_mss), mss);
mss = max(mss, READ_ONCE(net->ipv4.sysctl_tcp_mtu_probe_floor));
mss = max(mss, READ_ONCE(net->ipv4.sysctl_tcp_min_snd_mss));
icsk->icsk_mtup.search_low = tcp_mss_to_mtu(sk, mss);
}
tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
}
static unsigned int tcp_model_timeout(struct sock *sk,
unsigned int boundary,
unsigned int rto_base)
{
unsigned int linear_backoff_thresh, timeout;
linear_backoff_thresh = ilog2(tcp_rto_max(sk) / rto_base);
if (boundary <= linear_backoff_thresh)
timeout = ((2 << boundary) - 1) * rto_base;
else
timeout = ((2 << linear_backoff_thresh) - 1) * rto_base +
(boundary - linear_backoff_thresh) * tcp_rto_max(sk);
return jiffies_to_msecs(timeout);
}
/**
* retransmits_timed_out() - returns true if this connection has timed out
* @sk: The current socket
* @boundary: max number of retransmissions
* @timeout: A custom timeout value.
* If set to 0 the default timeout is calculated and used.
* Using TCP_RTO_MIN and the number of unsuccessful retransmits.
*
* The default "timeout" value this function can calculate and use
* is equivalent to the timeout of a TCP Connection
* after "boundary" unsuccessful, exponentially backed-off
* retransmissions with an initial RTO of TCP_RTO_MIN.
*/
static bool retransmits_timed_out(struct sock *sk,
unsigned int boundary,
unsigned int timeout)
{
struct tcp_sock *tp = tcp_sk(sk);
unsigned int start_ts, delta;
if (!inet_csk(sk)->icsk_retransmits)
return false;
start_ts = tp->retrans_stamp;
if (likely(timeout == 0)) {
unsigned int rto_base = TCP_RTO_MIN;
if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV))
rto_base = tcp_timeout_init(sk);
timeout = tcp_model_timeout(sk, boundary, rto_base);
}
if (tp->tcp_usec_ts) {
/* delta maybe off up to a jiffy due to timer granularity. */
delta = tp->tcp_mstamp - start_ts + jiffies_to_usecs(1);
return (s32)(delta - timeout * USEC_PER_MSEC) >= 0;
}
return (s32)(tcp_time_stamp_ts(tp) - start_ts - timeout) >= 0;
}
/* A write timeout has occurred. Process the after effects. */
static int tcp_write_timeout(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct tcp_sock *tp = tcp_sk(sk);
struct net *net = sock_net(sk);
bool expired = false, do_reset;
int retry_until, max_retransmits;
if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) {
if (icsk->icsk_retransmits)
__dst_negative_advice(sk);
/* Paired with WRITE_ONCE() in tcp_sock_set_syncnt() */
retry_until = READ_ONCE(icsk->icsk_syn_retries) ? :
READ_ONCE(net->ipv4.sysctl_tcp_syn_retries);
max_retransmits = retry_until;
if (sk->sk_state == TCP_SYN_SENT)
max_retransmits += READ_ONCE(net->ipv4.sysctl_tcp_syn_linear_timeouts);
expired = icsk->icsk_retransmits >= max_retransmits;
} else {
if (retransmits_timed_out(sk, READ_ONCE(net->ipv4.sysctl_tcp_retries1), 0)) {
/* Black hole detection */
tcp_mtu_probing(icsk, sk);
__dst_negative_advice(sk);
}
retry_until = READ_ONCE(net->ipv4.sysctl_tcp_retries2);
if (sock_flag(sk, SOCK_DEAD)) {
const bool alive = icsk->icsk_rto < tcp_rto_max(sk);
retry_until = tcp_orphan_retries(sk, alive);
do_reset = alive ||
!retransmits_timed_out(sk, retry_until, 0);
if (tcp_out_of_resources(sk, do_reset))
return 1;
}
}
if (!expired)
expired = retransmits_timed_out(sk, retry_until,
READ_ONCE(icsk->icsk_user_timeout));
tcp_fastopen_active_detect_blackhole(sk, expired);
mptcp_active_detect_blackhole(sk, expired);
if (BPF_SOCK_OPS_TEST_FLAG(tp, BPF_SOCK_OPS_RTO_CB_FLAG))
tcp_call_bpf_3arg(sk, BPF_SOCK_OPS_RTO_CB,
icsk->icsk_retransmits,
icsk->icsk_rto, (int)expired);
if (expired) {
/* Has it gone just too far? */
tcp_write_err(sk);
return 1;
}
if (sk_rethink_txhash(sk)) {
WRITE_ONCE(tp->timeout_rehash, tp->timeout_rehash + 1);
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPTIMEOUTREHASH);
}
return 0;
}
/* Called with BH disabled */
void tcp_delack_timer_handler(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct tcp_sock *tp = tcp_sk(sk);
if ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN))
return;
/* Handling the sack compression case */
if (tp->compressed_ack) {
tcp_mstamp_refresh(tp);
tcp_sack_compress_send_ack(sk);
return;
}
if (!(icsk->icsk_ack.pending & ICSK_ACK_TIMER))
return;
if (time_after(icsk_delack_timeout(icsk), jiffies)) {
sk_reset_timer(sk, &icsk->icsk_delack_timer,
icsk_delack_timeout(icsk));
return;
}
icsk->icsk_ack.pending &= ~ICSK_ACK_TIMER;
if (inet_csk_ack_scheduled(sk)) {
if (!inet_csk_in_pingpong_mode(sk)) {
/* Delayed ACK missed: inflate ATO. */
icsk->icsk_ack.ato = min_t(u32, icsk->icsk_ack.ato << 1, icsk->icsk_rto);
} else {
/* Delayed ACK missed: leave pingpong mode and
* deflate ATO.
*/
inet_csk_exit_pingpong_mode(sk);
icsk->icsk_ack.ato = TCP_ATO_MIN;
}
tcp_mstamp_refresh(tp);
tcp_send_ack(sk);
__NET_INC_STATS(sock_net(sk), LINUX_MIB_DELAYEDACKS);
}
}
/**
* tcp_delack_timer() - The TCP delayed ACK timeout handler
* @t: Pointer to the timer. (gets casted to struct sock *)
*
* This function gets (indirectly) called when the kernel timer for a TCP packet
* of this socket expires. Calls tcp_delack_timer_handler() to do the actual work.
*
* Returns: Nothing (void)
*/
static void tcp_delack_timer(struct timer_list *t)
{
struct inet_connection_sock *icsk =
timer_container_of(icsk, t, icsk_delack_timer);
struct sock *sk = &icsk->icsk_inet.sk;
/* Avoid taking socket spinlock if there is no ACK to send.
* The compressed_ack check is racy, but a separate hrtimer
* will take care of it eventually.
*/
if (!(smp_load_acquire(&icsk->icsk_ack.pending) & ICSK_ACK_TIMER) &&
!READ_ONCE(tcp_sk(sk)->compressed_ack))
goto out;
bh_lock_sock(sk);
if (!sock_owned_by_user(sk)) {
tcp_delack_timer_handler(sk);
} else {
__NET_INC_STATS(sock_net(sk), LINUX_MIB_DELAYEDACKLOCKED);
/* deleguate our work to tcp_release_cb() */
if (!test_and_set_bit(TCP_DELACK_TIMER_DEFERRED, &sk->sk_tsq_flags))
sock_hold(sk);
}
bh_unlock_sock(sk);
out:
sock_put(sk);
}
static void tcp_probe_timer(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct sk_buff *skb = tcp_send_head(sk);
struct tcp_sock *tp = tcp_sk(sk);
int max_probes;
if (tp->packets_out || !skb) {
WRITE_ONCE(icsk->icsk_probes_out, 0);
icsk->icsk_probes_tstamp = 0;
return;
}
/* RFC 1122 4.2.2.17 requires the sender to stay open indefinitely as
* long as the receiver continues to respond probes. We support this by
* default and reset icsk_probes_out with incoming ACKs. But if the
* socket is orphaned or the user specifies TCP_USER_TIMEOUT, we
* kill the socket when the retry count and the time exceeds the
* corresponding system limit. We also implement similar policy when
* we use RTO to probe window in tcp_retransmit_timer().
*/
if (!icsk->icsk_probes_tstamp) {
icsk->icsk_probes_tstamp = tcp_jiffies32;
} else {
u32 user_timeout = READ_ONCE(icsk->icsk_user_timeout);
if (user_timeout &&
(s32)(tcp_jiffies32 - icsk->icsk_probes_tstamp) >=
msecs_to_jiffies(user_timeout))
goto abort;
}
max_probes = READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_retries2);
if (sock_flag(sk, SOCK_DEAD)) {
unsigned int rto_max = tcp_rto_max(sk);
const bool alive = inet_csk_rto_backoff(icsk, rto_max) < rto_max;
max_probes = tcp_orphan_retries(sk, alive);
if (!alive && icsk->icsk_backoff >= max_probes)
goto abort;
if (tcp_out_of_resources(sk, true))
return;
}
if (icsk->icsk_probes_out >= max_probes) {
abort: tcp_write_err(sk);
} else {
/* Only send another probe if we didn't close things up. */
tcp_send_probe0(sk);
}
}
static void tcp_update_rto_stats(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct tcp_sock *tp = tcp_sk(sk);
if (!icsk->icsk_retransmits) {
tp->total_rto_recoveries++;
tp->rto_stamp = tcp_time_stamp_ms(tp);
}
WRITE_ONCE(icsk->icsk_retransmits, icsk->icsk_retransmits + 1);
tp->total_rto++;
}
/*
* Timer for Fast Open socket to retransmit SYNACK. Note that the
* sk here is the child socket, not the parent (listener) socket.
*/
static void tcp_fastopen_synack_timer(struct sock *sk, struct request_sock *req)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct tcp_sock *tp = tcp_sk(sk);
int max_retries;
tcp_syn_ack_timeout(req);
/* Add one more retry for fastopen.
* Paired with WRITE_ONCE() in tcp_sock_set_syncnt()
*/
max_retries = READ_ONCE(icsk->icsk_syn_retries) ? :
READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_synack_retries) + 1;
if (req->num_timeout >= max_retries) {
tcp_write_err(sk);
return;
}
/* Lower cwnd after certain SYNACK timeout like tcp_init_transfer() */
if (icsk->icsk_retransmits == 1)
tcp_enter_loss(sk);
/* XXX (TFO) - Unlike regular SYN-ACK retransmit, we ignore error
* returned from rtx_syn_ack() to make it more persistent like
* regular retransmit because if the child socket has been accepted
* it's not good to give up too easily.
*/
tcp_rtx_synack(sk, req);
if (req->num_retrans > 1 && tcp_rsk(req)->accecn_ok)
tcp_rsk(req)->accecn_fail_mode |= TCP_ACCECN_ACE_FAIL_SEND;
req->num_timeout++;
tcp_update_rto_stats(sk);
if (!tp->retrans_stamp)
tp->retrans_stamp = tcp_time_stamp_ts(tp);
tcp_reset_xmit_timer(sk, ICSK_TIME_RETRANS,
req->timeout << req->num_timeout, false);
}
static bool tcp_rtx_probe0_timed_out(const struct sock *sk,
const struct sk_buff *skb,
u32 rtx_delta)
{
const struct inet_connection_sock *icsk = inet_csk(sk);
u32 user_timeout = READ_ONCE(icsk->icsk_user_timeout);
const struct tcp_sock *tp = tcp_sk(sk);
int timeout = tcp_rto_max(sk) * 2;
s32 rcv_delta;
if (user_timeout) {
/* If user application specified a TCP_USER_TIMEOUT,
* it does not want win 0 packets to 'reset the timer'
* while retransmits are not making progress.
*/
if (rtx_delta > user_timeout)
return true;
timeout = min_t(u32, timeout, msecs_to_jiffies(user_timeout));
}
/* Note: timer interrupt might have been delayed by at least one jiffy,
* and tp->rcv_tstamp might very well have been written recently.
* rcv_delta can thus be negative.
*/
rcv_delta = tcp_timeout_expires(sk) - tp->rcv_tstamp;
if (rcv_delta <= timeout)
return false;
return msecs_to_jiffies(rtx_delta) > timeout;
}
/**
* tcp_retransmit_timer() - The TCP retransmit timeout handler
* @sk: Pointer to the current socket.
*
* This function gets called when the kernel timer for a TCP packet
* of this socket expires.
*
* It handles retransmission, timer adjustment and other necessary measures.
*
* Returns: Nothing (void)
*/
void tcp_retransmit_timer(struct sock *sk)
{
struct tcp_sock *tp = tcp_sk(sk);
struct net *net = sock_net(sk);
struct inet_connection_sock *icsk = inet_csk(sk);
struct request_sock *req;
struct sk_buff *skb;
req = rcu_dereference_protected(tp->fastopen_rsk,
lockdep_sock_is_held(sk));
if (req) {
WARN_ON_ONCE(sk->sk_state != TCP_SYN_RECV &&
sk->sk_state != TCP_FIN_WAIT1);
tcp_fastopen_synack_timer(sk, req);
/* Before we receive ACK to our SYN-ACK don't retransmit
* anything else (e.g., data or FIN segments).
*/
return;
}
if (!tp->packets_out)
return;
skb = tcp_rtx_queue_head(sk);
if (WARN_ON_ONCE(!skb))
return;
if (!tp->snd_wnd && !sock_flag(sk, SOCK_DEAD) &&
!((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV))) {
/* Receiver dastardly shrinks window. Our retransmits
* become zero probes, but we should not timeout this
* connection. If the socket is an orphan, time it out,
* we cannot allow such beasts to hang infinitely.
*/
struct inet_sock *inet = inet_sk(sk);
u32 rtx_delta;
rtx_delta = tcp_time_stamp_ts(tp) - (tp->retrans_stamp ?:
tcp_skb_timestamp_ts(tp->tcp_usec_ts, skb));
if (tp->tcp_usec_ts)
rtx_delta /= USEC_PER_MSEC;
if (sk->sk_family == AF_INET) {
net_dbg_ratelimited("Probing zero-window on %pI4:%u/%u, seq=%u:%u, recv %ums ago, lasting %ums\n",
&inet->inet_daddr, ntohs(inet->inet_dport),
inet->inet_num, tp->snd_una, tp->snd_nxt,
jiffies_to_msecs(jiffies - tp->rcv_tstamp),
rtx_delta);
}
#if IS_ENABLED(CONFIG_IPV6)
else if (sk->sk_family == AF_INET6) {
net_dbg_ratelimited("Probing zero-window on %pI6:%u/%u, seq=%u:%u, recv %ums ago, lasting %ums\n",
&sk->sk_v6_daddr, ntohs(inet->inet_dport),
inet->inet_num, tp->snd_una, tp->snd_nxt,
jiffies_to_msecs(jiffies - tp->rcv_tstamp),
rtx_delta);
}
#endif
if (tcp_rtx_probe0_timed_out(sk, skb, rtx_delta)) {
tcp_write_err(sk);
goto out;
}
tcp_enter_loss(sk);
tcp_retransmit_skb(sk, skb, 1);
__sk_dst_reset(sk);
goto out_reset_timer;
}
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPTIMEOUTS);
if (tcp_write_timeout(sk))
goto out;
if (icsk->icsk_retransmits == 0) {
int mib_idx = 0;
if (icsk->icsk_ca_state == TCP_CA_Recovery) {
if (tcp_is_sack(tp))
mib_idx = LINUX_MIB_TCPSACKRECOVERYFAIL;
else
mib_idx = LINUX_MIB_TCPRENORECOVERYFAIL;
} else if (icsk->icsk_ca_state == TCP_CA_Loss) {
mib_idx = LINUX_MIB_TCPLOSSFAILURES;
} else if ((icsk->icsk_ca_state == TCP_CA_Disorder) ||
tp->sacked_out) {
if (tcp_is_sack(tp))
mib_idx = LINUX_MIB_TCPSACKFAILURES;
else
mib_idx = LINUX_MIB_TCPRENOFAILURES;
}
if (mib_idx)
__NET_INC_STATS(sock_net(sk), mib_idx);
}
tcp_enter_loss(sk);
tcp_update_rto_stats(sk);
if (tcp_retransmit_skb(sk, tcp_rtx_queue_head(sk), 1) > 0) {
/* Retransmission failed because of local congestion,
* Let senders fight for local resources conservatively.
*/
tcp_reset_xmit_timer(sk, ICSK_TIME_RETRANS,
TCP_RESOURCE_PROBE_INTERVAL,
false);
goto out;
}
/* Increase the timeout each time we retransmit. Note that
* we do not increase the rtt estimate. rto is initialized
* from rtt, but increases here. Jacobson (SIGCOMM 88) suggests
* that doubling rto each time is the least we can get away with.
* In KA9Q, Karn uses this for the first few times, and then
* goes to quadratic. netBSD doubles, but only goes up to *64,
* and clamps at 1 to 64 sec afterwards. Note that 120 sec is
* defined in the protocol as the maximum possible RTT. I guess
* we'll have to use something other than TCP to talk to the
* University of Mars.
*
* PAWS allows us longer timeouts and large windows, so once
* implemented ftp to mars will work nicely. We will have to fix
* the 120 second clamps though!
*/
out_reset_timer:
/* If stream is thin, use linear timeouts. Since 'icsk_backoff' is
* used to reset timer, set to 0. Recalculate 'icsk_rto' as this
* might be increased if the stream oscillates between thin and thick,
* thus the old value might already be too high compared to the value
* set by 'tcp_set_rto' in tcp_input.c which resets the rto without
* backoff. Limit to TCP_THIN_LINEAR_RETRIES before initiating
* exponential backoff behaviour to avoid continue hammering
* linear-timeout retransmissions into a black hole
*/
if (sk->sk_state == TCP_ESTABLISHED &&
(tp->thin_lto || READ_ONCE(net->ipv4.sysctl_tcp_thin_linear_timeouts)) &&
tcp_stream_is_thin(tp) &&
icsk->icsk_retransmits <= TCP_THIN_LINEAR_RETRIES) {
icsk->icsk_backoff = 0;
icsk->icsk_rto = clamp(__tcp_set_rto(tp),
tcp_rto_min(sk),
tcp_rto_max(sk));
} else if (sk->sk_state != TCP_SYN_SENT ||
tp->total_rto >
READ_ONCE(net->ipv4.sysctl_tcp_syn_linear_timeouts)) {
/* Use normal (exponential) backoff unless linear timeouts are
* activated.
*/
icsk->icsk_backoff++;
icsk->icsk_rto = min(icsk->icsk_rto << 1, tcp_rto_max(sk));
}
tcp_reset_xmit_timer(sk, ICSK_TIME_RETRANS,
tcp_clamp_rto_to_user_timeout(sk), false);
if (retransmits_timed_out(sk, READ_ONCE(net->ipv4.sysctl_tcp_retries1) + 1, 0))
__sk_dst_reset(sk);
out:;
}
/* Called with bottom-half processing disabled.
* Called by tcp_write_timer() and tcp_release_cb().
*/
void tcp_write_timer_handler(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
int event;
if (((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN)) ||
!icsk->icsk_pending)
return;
if (time_after(tcp_timeout_expires(sk), jiffies)) {
sk_reset_timer(sk, &sk->tcp_retransmit_timer,
tcp_timeout_expires(sk));
return;
}
tcp_mstamp_refresh(tcp_sk(sk));
event = icsk->icsk_pending;
switch (event) {
case ICSK_TIME_REO_TIMEOUT:
tcp_rack_reo_timeout(sk);
break;
case ICSK_TIME_LOSS_PROBE:
tcp_send_loss_probe(sk);
break;
case ICSK_TIME_RETRANS:
smp_store_release(&icsk->icsk_pending, 0);
tcp_retransmit_timer(sk);
break;
case ICSK_TIME_PROBE0:
smp_store_release(&icsk->icsk_pending, 0);
tcp_probe_timer(sk);
break;
}
}
static void tcp_write_timer(struct timer_list *t)
{
struct sock *sk = timer_container_of(sk, t, tcp_retransmit_timer);
/* Avoid locking the socket when there is no pending event. */
if (!smp_load_acquire(&inet_csk(sk)->icsk_pending))
goto out;
bh_lock_sock(sk);
if (!sock_owned_by_user(sk)) {
tcp_write_timer_handler(sk);
} else {
/* delegate our work to tcp_release_cb() */
if (!test_and_set_bit(TCP_WRITE_TIMER_DEFERRED, &sk->sk_tsq_flags))
sock_hold(sk);
}
bh_unlock_sock(sk);
out:
sock_put(sk);
}
void tcp_syn_ack_timeout(const struct request_sock *req)
{
struct net *net = read_pnet(&inet_rsk(req)->ireq_net);
__NET_INC_STATS(net, LINUX_MIB_TCPTIMEOUTS);
}
void tcp_reset_keepalive_timer(struct sock *sk, unsigned long len)
{
sk_reset_timer(sk, &inet_csk(sk)->icsk_keepalive_timer, jiffies + len);
}
static void tcp_delete_keepalive_timer(struct sock *sk)
{
sk_stop_timer(sk, &inet_csk(sk)->icsk_keepalive_timer);
}
void tcp_set_keepalive(struct sock *sk, int val)
{
if ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN))
return;
if (val && !sock_flag(sk, SOCK_KEEPOPEN))
tcp_reset_keepalive_timer(sk, keepalive_time_when(tcp_sk(sk)));
else if (!val)
tcp_delete_keepalive_timer(sk);
}
static void tcp_keepalive_timer(struct timer_list *t)
{
struct inet_connection_sock *icsk =
timer_container_of(icsk, t, icsk_keepalive_timer);
struct sock *sk = &icsk->icsk_inet.sk;
struct tcp_sock *tp = tcp_sk(sk);
u32 elapsed;
/* Only process if socket is not in use. */
bh_lock_sock(sk);
if (sock_owned_by_user(sk)) {
/* Try again later. */
tcp_reset_keepalive_timer(sk, HZ/20);
goto out;
}
if (sk->sk_state == TCP_LISTEN) {
pr_err("Hmm... keepalive on a LISTEN ???\n");
goto out;
}
tcp_mstamp_refresh(tp);
if (sk->sk_state == TCP_FIN_WAIT2 && sock_flag(sk, SOCK_DEAD)) {
if (READ_ONCE(tp->linger2) >= 0) {
const int tmo = tcp_fin_time(sk) - TCP_TIMEWAIT_LEN;
if (tmo > 0) {
tcp_time_wait(sk, TCP_FIN_WAIT2, tmo);
goto out;
}
}
tcp_send_active_reset(sk, GFP_ATOMIC, SK_RST_REASON_TCP_STATE);
goto death;
}
if (!sock_flag(sk, SOCK_KEEPOPEN) ||
((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_SYN_SENT)))
goto out;
elapsed = keepalive_time_when(tp);
/* It is alive without keepalive 8) */
if (tp->packets_out || !tcp_write_queue_empty(sk))
goto resched;
elapsed = keepalive_time_elapsed(tp);
if (elapsed >= keepalive_time_when(tp)) {
u32 user_timeout = READ_ONCE(icsk->icsk_user_timeout);
/* If the TCP_USER_TIMEOUT option is enabled, use that
* to determine when to timeout instead.
*/
if ((user_timeout != 0 &&
elapsed >= msecs_to_jiffies(user_timeout) &&
icsk->icsk_probes_out > 0) ||
(user_timeout == 0 &&
icsk->icsk_probes_out >= keepalive_probes(tp))) {
tcp_send_active_reset(sk, GFP_ATOMIC,
SK_RST_REASON_TCP_KEEPALIVE_TIMEOUT);
tcp_write_err(sk);
goto out;
}
if (tcp_write_wakeup(sk, LINUX_MIB_TCPKEEPALIVE) <= 0) {
WRITE_ONCE(icsk->icsk_probes_out, icsk->icsk_probes_out + 1);
elapsed = keepalive_intvl_when(tp);
} else {
/* If keepalive was lost due to local congestion,
* try harder.
*/
elapsed = TCP_RESOURCE_PROBE_INTERVAL;
}
} else {
/* It is tp->rcv_tstamp + keepalive_time_when(tp) */
elapsed = keepalive_time_when(tp) - elapsed;
}
resched:
tcp_reset_keepalive_timer(sk, elapsed);
goto out;
death:
tcp_done(sk);
out:
bh_unlock_sock(sk);
sock_put(sk);
}
static enum hrtimer_restart tcp_compressed_ack_kick(struct hrtimer *timer)
{
struct tcp_sock *tp = container_of(timer, struct tcp_sock, compressed_ack_timer);
struct sock *sk = (struct sock *)tp;
bh_lock_sock(sk);
if (!sock_owned_by_user(sk)) {
if (tp->compressed_ack) {
/* Since we have to send one ack finally,
* subtract one from tp->compressed_ack to keep
* LINUX_MIB_TCPACKCOMPRESSED accurate.
*/
tp->compressed_ack--;
tcp_mstamp_refresh(tp);
tcp_send_ack(sk);
}
} else {
if (!test_and_set_bit(TCP_DELACK_TIMER_DEFERRED,
&sk->sk_tsq_flags))
sock_hold(sk);
}
bh_unlock_sock(sk);
sock_put(sk);
return HRTIMER_NORESTART;
}
void tcp_init_xmit_timers(struct sock *sk)
{
inet_csk_init_xmit_timers(sk, &tcp_write_timer, &tcp_delack_timer,
&tcp_keepalive_timer);
hrtimer_setup(&tcp_sk(sk)->pacing_timer, tcp_pace_kick, CLOCK_MONOTONIC,
HRTIMER_MODE_ABS_PINNED_SOFT);
hrtimer_setup(&tcp_sk(sk)->compressed_ack_timer, tcp_compressed_ack_kick, CLOCK_MONOTONIC,
HRTIMER_MODE_REL_PINNED_SOFT);
}
]
|
| 26/2 |
2026/05/17 01:37 |
llm |
expert |
0m
Model:
Tokens:
input: 63414
output: 27
thoughts: 3787
Thoughts: **Deep Dive into Socket Destruction and Timer Interactions**
Okay, I've got it. I'm focusing on why a socket's reference count might underflow, leading to a "use-after-free" situation, specifically tied to timers. The core issue seems to revolve around the interaction between socket destruction and concurrent timer execution.
First, I looked at how timers are handled, specifically the common pattern of `sock_put(sk)` at the end of timer callbacks (`tcp_write_timer`, `tcp_delack_timer`, `tcp_keepalive_timer`). `sock_put` decrements `sk->sk_refcnt` and frees the socket if the count reaches zero. Crucially, if the reference count *underflows* – meaning the socket was already freed – a warning is triggered. This direct call to `refcount_warn_saturate` in the stack trace points directly to these timer callbacks, which I identified earlier as candidates.
A critical point: the `refcount_t: underflow` error means the socket's refcount was *already* zero when the timer tried to release its reference. This is what caused the UAF and the warning. So the timer is not getting a reference to the socket properly. How?
When I looked into the logic of how timers work, it seems that if a timer is *scheduled*, it holds a reference to the socket using `sock_hold`. Conversely, `sk_stop_timer` uses `timer_delete`, which deletes a timer. If `timer_delete` *succeeds* in deleting a pending timer, it does call `sock_put`. If the timer is currently *running*, `timer_delete` returns false, and then `sock_put` is NOT called, because the timer *itself* must call `sock_put` when it finishes. This difference is important. The socket *won't* be freed until the running timer finishes and calls `sock_put`. But if someone else drops the socket, this is the root cause. This is a potential source of the bug.
I then considered what happens during socket teardown. The key function is `inet_csk_clear_xmit_timers`, which calls `sk_stop_timer` on various timers. The timers include the same ones in the UAF. But here's the kicker: `inet_csk_clear_xmit_timers` uses `sk_stop_timer`, which uses `timer_delete`. If a timer is running concurrently, `timer_delete` returns false and `sock_put` is *not* called there. The timer is supposed to call `sock_put` at the end. I then traced `inet_csk_clear_xmit_timers` to the function `tcp_v4_destroy_sock`, which gets called by `sk->sk_prot->destroy(sk)`. This latter function is called by `__sk_destruct`, which is called when `sk->sk_refcnt` is 0, which means the socket is being freed.
But here's a crucial point: `__sk_destruct` doesn't *directly* call the `destroy` function! It's `sk_common_release` that calls it. This can happen from a number of states.
The main thread of execution when a socket is destroyed is:
1. Socket calls `tcp_close` or `tcp_done`.
2. `tcp_close` / `tcp_done` calls `inet_csk_destroy_sock`.
3. `inet_csk_destroy_sock` calls `sk->sk_prot->destroy(sk)` (which is `tcp_v4_destroy_sock`) *before* the final `sock_put`.
4. `tcp_v4_destroy_sock` calls `inet_csk_clear_xmit_timers`, which uses `sk_stop_timer` which uses `timer_delete` which does *not* wait for the timer to finish! If the timer is still running, then we can have this race.
Here's the problem: if `tcp_v4_destroy_sock` is called and *returns*, and the timer is *still* running, and if the final reference is dropped, the socket is freed by `inet_csk_destroy_sock`’s `sock_put` *before* the timer finishes.
So, the issue is a race condition: `tcp_v4_destroy_sock` (and the `inet_csk_clear_xmit_timers` call within it) can run concurrently with the timers, leading to a premature `sock_put` and potential UAF. The key to fixing this lies in coordinating the destruction of the
|
| 27/2 |
2026/05/17 01:38 |
tool |
grepper |
0m
Args: map[Expression:\.destroy.*=.*tcp_v4_destroy_sock]
Results: map[Output:net/ipv4/tcp_ipv4.c=3337=struct proto tcp_prot = {
--
net/ipv4/tcp_ipv4.c-3346- .init = tcp_v4_init_sock,
net/ipv4/tcp_ipv4.c:3347: .destroy = tcp_v4_destroy_sock,
net/ipv4/tcp_ipv4.c-3348- .shutdown = tcp_shutdown,
--
net/ipv6/tcp_ipv6.c=2251=struct proto tcpv6_prot = {
--
net/ipv6/tcp_ipv6.c-2260- .init = tcp_v6_init_sock,
net/ipv6/tcp_ipv6.c:2261: .destroy = tcp_v4_destroy_sock,
net/ipv6/tcp_ipv6.c-2262- .shutdown = tcp_shutdown,
]
|
| 28/2 |
2026/05/17 01:38 |
llm |
expert |
0m
Model:
Tokens:
input: 68505
output: 26
thoughts: 157
Thoughts: **Dissecting the TCP Socket Destruction Process**
Okay, so I'm tracing the socket destruction flow, and it looks like `tcp_v4_destroy_sock` is indeed the destination function called via `sk->sk_prot->destroy`, as I expected. Now, I see `inet_csk_destroy_sock` is called from `tcp_done` or `tcp_close`. Wait, am I understanding this correctly? `tcp_done` then calls `inet_csk_destroy_sock`, which, in turn, calls the destroy function for the specific protocol, so `sk->sk_prot->destroy(sk)`. But... I'm seeing `tcp_v4_destroy_sock` *also* being called from `sk_common_release`. I need to clarify this. Hmm... `tcp_close` is the socket's close function (`sk->sk_prot->close`). Hang on, `tcp_close` doesn't seem to be calling `sk_common_release`. Ok, I need to verify how `tcp_v4_destroy_sock` is being called to confirm that I understand the call stacks. Let's delve into that function.
|
| 29/2 |
2026/05/17 01:38 |
tool |
read-file |
0m
Args: map[File:net/ipv4/tcp_ipv4.c]
Results: map[Contents:// SPDX-License-Identifier: GPL-2.0-or-later
/*
* INET An implementation of the TCP/IP protocol suite for the LINUX
* operating system. INET is implemented using the BSD Socket
* interface as the means of communication with the user level.
*
* Implementation of the Transmission Control Protocol(TCP).
*
* IPv4 specific functions
*
* code split from:
* linux/ipv4/tcp.c
* linux/ipv4/tcp_input.c
* linux/ipv4/tcp_output.c
*
* See tcp.c for author information
*/
/*
* Changes:
* David S. Miller : New socket lookup architecture.
* This code is dedicated to John Dyson.
* David S. Miller : Change semantics of established hash,
* half is devoted to TIME_WAIT sockets
* and the rest go in the other half.
* Andi Kleen : Add support for syncookies and fixed
* some bugs: ip options weren't passed to
* the TCP layer, missed a check for an
* ACK bit.
* Andi Kleen : Implemented fast path mtu discovery.
* Fixed many serious bugs in the
* request_sock handling and moved
* most of it into the af independent code.
* Added tail drop and some other bugfixes.
* Added new listen semantics.
* Mike McLagan : Routing by source
* Juan Jose Ciarlante: ip_dynaddr bits
* Andi Kleen: various fixes.
* Vitaly E. Lavrov : Transparent proxy revived after year
* coma.
* Andi Kleen : Fix new listen.
* Andi Kleen : Fix accept error reporting.
* YOSHIFUJI Hideaki @USAGI and: Support IPV6_V6ONLY socket option, which
* Alexey Kuznetsov allow both IPv4 and IPv6 sockets to bind
* a single port at the same time.
*/
#define pr_fmt(fmt) "TCP: " fmt
#include <linux/bottom_half.h>
#include <linux/types.h>
#include <linux/fcntl.h>
#include <linux/module.h>
#include <linux/random.h>
#include <linux/cache.h>
#include <linux/fips.h>
#include <linux/jhash.h>
#include <linux/init.h>
#include <linux/times.h>
#include <linux/slab.h>
#include <linux/sched.h>
#include <linux/sock_diag.h>
#include <net/aligned_data.h>
#include <net/net_namespace.h>
#include <net/icmp.h>
#include <net/inet_hashtables.h>
#include <net/tcp.h>
#include <net/tcp_ecn.h>
#include <net/transp_v6.h>
#include <net/ipv6.h>
#include <net/inet_common.h>
#include <net/inet_ecn.h>
#include <net/timewait_sock.h>
#include <net/xfrm.h>
#include <net/secure_seq.h>
#include <net/busy_poll.h>
#include <net/rstreason.h>
#include <net/psp.h>
#include <linux/inet.h>
#include <linux/ipv6.h>
#include <linux/stddef.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
#include <linux/inetdevice.h>
#include <linux/btf_ids.h>
#include <linux/skbuff_ref.h>
#include <crypto/md5.h>
#include <crypto/utils.h>
#include <trace/events/tcp.h>
#ifdef CONFIG_TCP_MD5SIG
static void tcp_v4_md5_hash_hdr(char *md5_hash, const struct tcp_md5sig_key *key,
__be32 daddr, __be32 saddr, const struct tcphdr *th);
#endif
struct inet_hashinfo tcp_hashinfo;
static DEFINE_PER_CPU(struct sock_bh_locked, ipv4_tcp_sk) = {
.bh_lock = INIT_LOCAL_LOCK(bh_lock),
};
static DEFINE_MUTEX(tcp_exit_batch_mutex);
INDIRECT_CALLABLE_SCOPE union tcp_seq_and_ts_off
tcp_v4_init_seq_and_ts_off(const struct net *net, const struct sk_buff *skb)
{
return secure_tcp_seq_and_ts_off(net,
ip_hdr(skb)->daddr,
ip_hdr(skb)->saddr,
tcp_hdr(skb)->dest,
tcp_hdr(skb)->source);
}
int tcp_twsk_unique(struct sock *sk, struct sock *sktw, void *twp)
{
int reuse = READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_tw_reuse);
const struct inet_timewait_sock *tw = inet_twsk(sktw);
const struct tcp_timewait_sock *tcptw = tcp_twsk(sktw);
struct tcp_sock *tp = tcp_sk(sk);
int ts_recent_stamp;
u32 reuse_thresh;
if (READ_ONCE(tw->tw_substate) == TCP_FIN_WAIT2)
reuse = 0;
if (reuse == 2) {
/* Still does not detect *everything* that goes through
* lo, since we require a loopback src or dst address
* or direct binding to 'lo' interface.
*/
bool loopback = false;
if (tw->tw_bound_dev_if == LOOPBACK_IFINDEX)
loopback = true;
#if IS_ENABLED(CONFIG_IPV6)
if (tw->tw_family == AF_INET6) {
if (ipv6_addr_loopback(&tw->tw_v6_daddr) ||
ipv6_addr_v4mapped_loopback(&tw->tw_v6_daddr) ||
ipv6_addr_loopback(&tw->tw_v6_rcv_saddr) ||
ipv6_addr_v4mapped_loopback(&tw->tw_v6_rcv_saddr))
loopback = true;
} else
#endif
{
if (ipv4_is_loopback(tw->tw_daddr) ||
ipv4_is_loopback(tw->tw_rcv_saddr))
loopback = true;
}
if (!loopback)
reuse = 0;
}
/* With PAWS, it is safe from the viewpoint
of data integrity. Even without PAWS it is safe provided sequence
spaces do not overlap i.e. at data rates <= 80Mbit/sec.
Actually, the idea is close to VJ's one, only timestamp cache is
held not per host, but per port pair and TW bucket is used as state
holder.
If TW bucket has been already destroyed we fall back to VJ's scheme
and use initial timestamp retrieved from peer table.
*/
ts_recent_stamp = READ_ONCE(tcptw->tw_ts_recent_stamp);
reuse_thresh = READ_ONCE(tw->tw_entry_stamp) +
READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_tw_reuse_delay);
if (ts_recent_stamp &&
(!twp || (reuse && time_after32(tcp_clock_ms(), reuse_thresh)))) {
/* inet_twsk_hashdance_schedule() sets sk_refcnt after putting twsk
* and releasing the bucket lock.
*/
if (unlikely(!refcount_inc_not_zero(&sktw->sk_refcnt)))
return 0;
/* In case of repair and re-using TIME-WAIT sockets we still
* want to be sure that it is safe as above but honor the
* sequence numbers and time stamps set as part of the repair
* process.
*
* Without this check re-using a TIME-WAIT socket with TCP
* repair would accumulate a -1 on the repair assigned
* sequence number. The first time it is reused the sequence
* is -1, the second time -2, etc. This fixes that issue
* without appearing to create any others.
*/
if (likely(!tp->repair)) {
u32 seq = tcptw->tw_snd_nxt + 65535 + 2;
if (!seq)
seq = 1;
WRITE_ONCE(tp->write_seq, seq);
tp->rx_opt.ts_recent = READ_ONCE(tcptw->tw_ts_recent);
tp->rx_opt.ts_recent_stamp = ts_recent_stamp;
}
return 1;
}
return 0;
}
static int tcp_v4_pre_connect(struct sock *sk, struct sockaddr_unsized *uaddr,
int addr_len)
{
/* This check is replicated from tcp_v4_connect() and intended to
* prevent BPF program called below from accessing bytes that are out
* of the bound specified by user in addr_len.
*/
if (addr_len < sizeof(struct sockaddr_in))
return -EINVAL;
sock_owned_by_me(sk);
return BPF_CGROUP_RUN_PROG_INET4_CONNECT(sk, uaddr, &addr_len);
}
/* This will initiate an outgoing connection. */
int tcp_v4_connect(struct sock *sk, struct sockaddr_unsized *uaddr, int addr_len)
{
struct sockaddr_in *usin = (struct sockaddr_in *)uaddr;
struct inet_timewait_death_row *tcp_death_row;
struct inet_sock *inet = inet_sk(sk);
struct tcp_sock *tp = tcp_sk(sk);
struct ip_options_rcu *inet_opt;
struct net *net = sock_net(sk);
__be16 orig_sport, orig_dport;
__be32 daddr, nexthop;
struct flowi4 *fl4;
struct rtable *rt;
int err;
if (addr_len < sizeof(struct sockaddr_in))
return -EINVAL;
if (usin->sin_family != AF_INET)
return -EAFNOSUPPORT;
nexthop = daddr = usin->sin_addr.s_addr;
inet_opt = rcu_dereference_protected(inet->inet_opt,
lockdep_sock_is_held(sk));
if (inet_opt && inet_opt->opt.srr) {
if (!daddr)
return -EINVAL;
nexthop = inet_opt->opt.faddr;
}
orig_sport = inet->inet_sport;
orig_dport = usin->sin_port;
fl4 = &inet->cork.fl.u.ip4;
rt = ip_route_connect(fl4, nexthop, inet->inet_saddr,
sk->sk_bound_dev_if, IPPROTO_TCP, orig_sport,
orig_dport, sk);
if (IS_ERR(rt)) {
err = PTR_ERR(rt);
if (err == -ENETUNREACH)
IP_INC_STATS(net, IPSTATS_MIB_OUTNOROUTES);
return err;
}
if (rt->rt_flags & (RTCF_MULTICAST | RTCF_BROADCAST)) {
ip_rt_put(rt);
return -ENETUNREACH;
}
if (!inet_opt || !inet_opt->opt.srr)
daddr = fl4->daddr;
tcp_death_row = &sock_net(sk)->ipv4.tcp_death_row;
if (!inet->inet_saddr) {
err = inet_bhash2_update_saddr(sk, &fl4->saddr, AF_INET);
if (err) {
ip_rt_put(rt);
return err;
}
} else {
sk_rcv_saddr_set(sk, inet->inet_saddr);
}
if (tp->rx_opt.ts_recent_stamp && inet->inet_daddr != daddr) {
/* Reset inherited state */
tp->rx_opt.ts_recent = 0;
tp->rx_opt.ts_recent_stamp = 0;
if (likely(!tp->repair))
WRITE_ONCE(tp->write_seq, 0);
}
inet->inet_dport = usin->sin_port;
sk_daddr_set(sk, daddr);
inet_csk(sk)->icsk_ext_hdr_len = psp_sk_overhead(sk);
if (inet_opt)
inet_csk(sk)->icsk_ext_hdr_len += inet_opt->opt.optlen;
tp->rx_opt.mss_clamp = TCP_MSS_DEFAULT;
/* Socket identity is still unknown (sport may be zero).
* However we set state to SYN-SENT and not releasing socket
* lock select source port, enter ourselves into the hash tables and
* complete initialization after this.
*/
tcp_set_state(sk, TCP_SYN_SENT);
err = inet_hash_connect(tcp_death_row, sk);
if (err)
goto failure;
sk_set_txhash(sk);
rt = ip_route_newports(fl4, rt, orig_sport, orig_dport,
inet->inet_sport, inet->inet_dport, sk);
if (IS_ERR(rt)) {
err = PTR_ERR(rt);
rt = NULL;
goto failure;
}
tp->tcp_usec_ts = dst_tcp_usec_ts(&rt->dst);
/* OK, now commit destination to socket. */
sk->sk_gso_type = SKB_GSO_TCPV4;
sk_setup_caps(sk, &rt->dst);
rt = NULL;
if (likely(!tp->repair)) {
union tcp_seq_and_ts_off st;
st = secure_tcp_seq_and_ts_off(net,
inet->inet_saddr,
inet->inet_daddr,
inet->inet_sport,
usin->sin_port);
if (!tp->write_seq)
WRITE_ONCE(tp->write_seq, st.seq);
WRITE_ONCE(tp->tsoffset, st.ts_off);
}
atomic_set(&inet->inet_id, get_random_u16());
if (tcp_fastopen_defer_connect(sk, &err))
return err;
if (err)
goto failure;
err = tcp_connect(sk);
if (err)
goto failure;
return 0;
failure:
/*
* This unhashes the socket and releases the local port,
* if necessary.
*/
tcp_set_state(sk, TCP_CLOSE);
inet_bhash2_reset_saddr(sk);
ip_rt_put(rt);
sk->sk_route_caps = 0;
inet->inet_dport = 0;
return err;
}
/*
* This routine reacts to ICMP_FRAG_NEEDED mtu indications as defined in RFC1191.
* It can be called through tcp_release_cb() if socket was owned by user
* at the time tcp_v4_err() was called to handle ICMP message.
*/
void tcp_v4_mtu_reduced(struct sock *sk)
{
struct inet_sock *inet = inet_sk(sk);
struct dst_entry *dst;
u32 mtu, dmtu;
if ((1 << sk->sk_state) & (TCPF_LISTEN | TCPF_CLOSE))
return;
mtu = READ_ONCE(tcp_sk(sk)->mtu_info);
dst = inet_csk_update_pmtu(sk, mtu);
if (!dst)
return;
/* Something is about to be wrong... Remember soft error
* for the case, if this connection will not able to recover.
*/
dmtu = dst4_mtu(dst);
if (mtu < dmtu && ip_dont_fragment(sk, dst))
WRITE_ONCE(sk->sk_err_soft, EMSGSIZE);
if (inet->pmtudisc != IP_PMTUDISC_DONT &&
ip_sk_accept_pmtu(sk) &&
inet_csk(sk)->icsk_pmtu_cookie > dmtu) {
tcp_sync_mss(sk, dmtu);
/* Resend the TCP packet because it's
* clear that the old packet has been
* dropped. This is the new "fast" path mtu
* discovery.
*/
tcp_simple_retransmit(sk);
} /* else let the usual retransmit timer handle it */
}
static void do_redirect(struct sk_buff *skb, struct sock *sk)
{
struct dst_entry *dst = __sk_dst_check(sk, 0);
if (dst)
dst->ops->redirect(dst, sk, skb);
}
/* handle ICMP messages on TCP_NEW_SYN_RECV request sockets */
void tcp_req_err(struct sock *sk, u32 seq, bool abort)
{
struct request_sock *req = inet_reqsk(sk);
struct net *net = sock_net(sk);
/* ICMPs are not backlogged, hence we cannot get
* an established socket here.
*/
if (seq != tcp_rsk(req)->snt_isn) {
__NET_INC_STATS(net, LINUX_MIB_OUTOFWINDOWICMPS);
} else if (abort) {
/*
* Still in SYN_RECV, just remove it silently.
* There is no good way to pass the error to the newly
* created socket, and POSIX does not want network
* errors returned from accept().
*/
inet_csk_reqsk_queue_drop(req->rsk_listener, req);
tcp_listendrop(req->rsk_listener);
}
reqsk_put(req);
}
/* TCP-LD (RFC 6069) logic */
void tcp_ld_RTO_revert(struct sock *sk, u32 seq)
{
struct inet_connection_sock *icsk = inet_csk(sk);
struct tcp_sock *tp = tcp_sk(sk);
struct sk_buff *skb;
s32 remaining;
u32 delta_us;
if (sock_owned_by_user(sk))
return;
if (seq != tp->snd_una || !icsk->icsk_retransmits ||
!icsk->icsk_backoff)
return;
skb = tcp_rtx_queue_head(sk);
if (WARN_ON_ONCE(!skb))
return;
icsk->icsk_backoff--;
icsk->icsk_rto = tp->srtt_us ? __tcp_set_rto(tp) : TCP_TIMEOUT_INIT;
icsk->icsk_rto = inet_csk_rto_backoff(icsk, tcp_rto_max(sk));
tcp_mstamp_refresh(tp);
delta_us = (u32)(tp->tcp_mstamp - tcp_skb_timestamp_us(skb));
remaining = icsk->icsk_rto - usecs_to_jiffies(delta_us);
if (remaining > 0) {
tcp_reset_xmit_timer(sk, ICSK_TIME_RETRANS, remaining, false);
} else {
/* RTO revert clocked out retransmission.
* Will retransmit now.
*/
tcp_retransmit_timer(sk);
}
}
/*
* This routine is called by the ICMP module when it gets some
* sort of error condition. If err < 0 then the socket should
* be closed and the error returned to the user. If err > 0
* it's just the icmp type << 8 | icmp code. After adjustment
* header points to the first 8 bytes of the tcp header. We need
* to find the appropriate port.
*
* The locking strategy used here is very "optimistic". When
* someone else accesses the socket the ICMP is just dropped
* and for some paths there is no check at all.
* A more general error queue to queue errors for later handling
* is probably better.
*
*/
int tcp_v4_err(struct sk_buff *skb, u32 info)
{
const struct iphdr *iph = (const struct iphdr *)skb->data;
struct tcphdr *th = (struct tcphdr *)(skb->data + (iph->ihl << 2));
struct net *net = dev_net_rcu(skb->dev);
const int type = icmp_hdr(skb)->type;
const int code = icmp_hdr(skb)->code;
struct request_sock *fastopen;
struct tcp_sock *tp;
u32 seq, snd_una;
struct sock *sk;
int err;
sk = __inet_lookup_established(net, iph->daddr, th->dest, iph->saddr,
ntohs(th->source), inet_iif(skb), 0);
if (!sk) {
__ICMP_INC_STATS(net, ICMP_MIB_INERRORS);
return -ENOENT;
}
if (sk->sk_state == TCP_TIME_WAIT) {
/* To increase the counter of ignored icmps for TCP-AO */
tcp_ao_ignore_icmp(sk, AF_INET, type, code);
inet_twsk_put(inet_twsk(sk));
return 0;
}
seq = ntohl(th->seq);
if (sk->sk_state == TCP_NEW_SYN_RECV) {
tcp_req_err(sk, seq, type == ICMP_PARAMETERPROB ||
type == ICMP_TIME_EXCEEDED ||
(type == ICMP_DEST_UNREACH &&
(code == ICMP_NET_UNREACH ||
code == ICMP_HOST_UNREACH)));
return 0;
}
if (tcp_ao_ignore_icmp(sk, AF_INET, type, code)) {
sock_put(sk);
return 0;
}
bh_lock_sock(sk);
/* If too many ICMPs get dropped on busy
* servers this needs to be solved differently.
* We do take care of PMTU discovery (RFC1191) special case :
* we can receive locally generated ICMP messages while socket is held.
*/
if (sock_owned_by_user(sk)) {
if (!(type == ICMP_DEST_UNREACH && code == ICMP_FRAG_NEEDED))
__NET_INC_STATS(net, LINUX_MIB_LOCKDROPPEDICMPS);
}
if (sk->sk_state == TCP_CLOSE)
goto out;
if (static_branch_unlikely(&ip4_min_ttl)) {
/* min_ttl can be changed concurrently from do_ip_setsockopt() */
if (unlikely(iph->ttl < READ_ONCE(inet_sk(sk)->min_ttl))) {
__NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP);
goto out;
}
}
tp = tcp_sk(sk);
/* XXX (TFO) - tp->snd_una should be ISN (tcp_create_openreq_child() */
fastopen = rcu_dereference(tp->fastopen_rsk);
snd_una = fastopen ? tcp_rsk(fastopen)->snt_isn : tp->snd_una;
if (sk->sk_state != TCP_LISTEN &&
!between(seq, snd_una, tp->snd_nxt)) {
__NET_INC_STATS(net, LINUX_MIB_OUTOFWINDOWICMPS);
goto out;
}
switch (type) {
case ICMP_REDIRECT:
if (!sock_owned_by_user(sk))
do_redirect(skb, sk);
goto out;
case ICMP_SOURCE_QUENCH:
/* Just silently ignore these. */
goto out;
case ICMP_PARAMETERPROB:
err = EPROTO;
break;
case ICMP_DEST_UNREACH:
if (code > NR_ICMP_UNREACH)
goto out;
if (code == ICMP_FRAG_NEEDED) { /* PMTU discovery (RFC1191) */
/* We are not interested in TCP_LISTEN and open_requests
* (SYN-ACKs send out by Linux are always <576bytes so
* they should go through unfragmented).
*/
if (sk->sk_state == TCP_LISTEN)
goto out;
WRITE_ONCE(tp->mtu_info, info);
if (!sock_owned_by_user(sk)) {
tcp_v4_mtu_reduced(sk);
} else {
if (!test_and_set_bit(TCP_MTU_REDUCED_DEFERRED, &sk->sk_tsq_flags))
sock_hold(sk);
}
goto out;
}
err = icmp_err_convert[code].errno;
/* check if this ICMP message allows revert of backoff.
* (see RFC 6069)
*/
if (!fastopen &&
(code == ICMP_NET_UNREACH || code == ICMP_HOST_UNREACH))
tcp_ld_RTO_revert(sk, seq);
break;
case ICMP_TIME_EXCEEDED:
err = EHOSTUNREACH;
break;
default:
goto out;
}
switch (sk->sk_state) {
case TCP_SYN_SENT:
case TCP_SYN_RECV:
/* Only in fast or simultaneous open. If a fast open socket is
* already accepted it is treated as a connected one below.
*/
if (fastopen && !fastopen->sk)
break;
ip_icmp_error(sk, skb, err, th->dest, info, (u8 *)th);
if (!sock_owned_by_user(sk))
tcp_done_with_error(sk, err);
else
WRITE_ONCE(sk->sk_err_soft, err);
goto out;
}
/* If we've already connected we will keep trying
* until we time out, or the user gives up.
*
* rfc1122 4.2.3.9 allows to consider as hard errors
* only PROTO_UNREACH and PORT_UNREACH (well, FRAG_FAILED too,
* but it is obsoleted by pmtu discovery).
*
* Note, that in modern internet, where routing is unreliable
* and in each dark corner broken firewalls sit, sending random
* errors ordered by their masters even this two messages finally lose
* their original sense (even Linux sends invalid PORT_UNREACHs)
*
* Now we are in compliance with RFCs.
* --ANK (980905)
*/
if (!sock_owned_by_user(sk) &&
inet_test_bit(RECVERR, sk)) {
WRITE_ONCE(sk->sk_err, err);
sk_error_report(sk);
} else { /* Only an error on timeout */
WRITE_ONCE(sk->sk_err_soft, err);
}
out:
bh_unlock_sock(sk);
sock_put(sk);
return 0;
}
#define REPLY_OPTIONS_LEN (MAX_TCP_OPTION_SPACE / sizeof(__be32))
static bool tcp_v4_ao_sign_reset(const struct sock *sk, struct sk_buff *skb,
const struct tcp_ao_hdr *aoh,
struct ip_reply_arg *arg, struct tcphdr *reply,
__be32 reply_options[REPLY_OPTIONS_LEN])
{
#ifdef CONFIG_TCP_AO
int sdif = tcp_v4_sdif(skb);
int dif = inet_iif(skb);
int l3index = sdif ? dif : 0;
bool allocated_traffic_key;
struct tcp_ao_key *key;
char *traffic_key;
bool drop = true;
u32 ao_sne = 0;
u8 keyid;
rcu_read_lock();
if (tcp_ao_prepare_reset(sk, skb, aoh, l3index, ntohl(reply->seq),
&key, &traffic_key, &allocated_traffic_key,
&keyid, &ao_sne))
goto out;
reply_options[0] = htonl((TCPOPT_AO << 24) | (tcp_ao_len(key) << 16) |
(aoh->rnext_keyid << 8) | keyid);
arg->iov[0].iov_len += tcp_ao_len_aligned(key);
reply->doff = arg->iov[0].iov_len / 4;
if (tcp_ao_hash_hdr(AF_INET, (char *)&reply_options[1],
key, traffic_key,
(union tcp_ao_addr *)&ip_hdr(skb)->saddr,
(union tcp_ao_addr *)&ip_hdr(skb)->daddr,
reply, ao_sne))
goto out;
drop = false;
out:
rcu_read_unlock();
if (allocated_traffic_key)
kfree(traffic_key);
return drop;
#else
return true;
#endif
}
/*
* This routine will send an RST to the other tcp.
*
* Someone asks: why I NEVER use socket parameters (TOS, TTL etc.)
* for reset.
* Answer: if a packet caused RST, it is not for a socket
* existing in our system, if it is matched to a socket,
* it is just duplicate segment or bug in other side's TCP.
* So that we build reply only basing on parameters
* arrived with segment.
* Exception: precedence violation. We do not implement it in any case.
*/
static void tcp_v4_send_reset(const struct sock *sk, struct sk_buff *skb,
enum sk_rst_reason reason)
{
const struct tcphdr *th = tcp_hdr(skb);
struct {
struct tcphdr th;
__be32 opt[REPLY_OPTIONS_LEN];
} rep;
const __u8 *md5_hash_location = NULL;
const struct tcp_ao_hdr *aoh;
struct ip_reply_arg arg;
#ifdef CONFIG_TCP_MD5SIG
struct tcp_md5sig_key *key = NULL;
unsigned char newhash[16];
struct sock *sk1 = NULL;
#endif
u64 transmit_time = 0;
struct sock *ctl_sk;
struct net *net;
u32 txhash = 0;
/* Never send a reset in response to a reset. */
if (th->rst)
return;
/* If sk not NULL, it means we did a successful lookup and incoming
* route had to be correct. prequeue might have dropped our dst.
*/
if (!sk && skb_rtable(skb)->rt_type != RTN_LOCAL)
return;
/* Swap the send and the receive. */
memset(&rep, 0, sizeof(rep));
rep.th.dest = th->source;
rep.th.source = th->dest;
rep.th.doff = sizeof(struct tcphdr) / 4;
rep.th.rst = 1;
if (th->ack) {
rep.th.seq = th->ack_seq;
} else {
rep.th.ack = 1;
rep.th.ack_seq = htonl(ntohl(th->seq) + th->syn + th->fin +
skb->len - (th->doff << 2));
}
memset(&arg, 0, sizeof(arg));
arg.iov[0].iov_base = (unsigned char *)&rep;
arg.iov[0].iov_len = sizeof(rep.th);
net = sk ? sock_net(sk) : skb_dst_dev_net_rcu(skb);
/* Invalid TCP option size or twice included auth */
if (tcp_parse_auth_options(tcp_hdr(skb), &md5_hash_location, &aoh))
return;
if (aoh && tcp_v4_ao_sign_reset(sk, skb, aoh, &arg, &rep.th, rep.opt))
return;
#ifdef CONFIG_TCP_MD5SIG
rcu_read_lock();
if (sk && sk_fullsock(sk)) {
const union tcp_md5_addr *addr;
int l3index;
/* sdif set, means packet ingressed via a device
* in an L3 domain and inet_iif is set to it.
*/
l3index = tcp_v4_sdif(skb) ? inet_iif(skb) : 0;
addr = (union tcp_md5_addr *)&ip_hdr(skb)->saddr;
key = tcp_md5_do_lookup(sk, l3index, addr, AF_INET);
} else if (md5_hash_location) {
const union tcp_md5_addr *addr;
int sdif = tcp_v4_sdif(skb);
int dif = inet_iif(skb);
int l3index;
/*
* active side is lost. Try to find listening socket through
* source port, and then find md5 key through listening socket.
* we are not loose security here:
* Incoming packet is checked with md5 hash with finding key,
* no RST generated if md5 hash doesn't match.
*/
sk1 = __inet_lookup_listener(net, NULL, 0, ip_hdr(skb)->saddr,
th->source, ip_hdr(skb)->daddr,
ntohs(th->source), dif, sdif);
/* don't send rst if it can't find key */
if (!sk1)
goto out;
/* sdif set, means packet ingressed via a device
* in an L3 domain and dif is set to it.
*/
l3index = sdif ? dif : 0;
addr = (union tcp_md5_addr *)&ip_hdr(skb)->saddr;
key = tcp_md5_do_lookup(sk1, l3index, addr, AF_INET);
if (!key)
goto out;
tcp_v4_md5_hash_skb(newhash, key, NULL, skb);
if (crypto_memneq(md5_hash_location, newhash, 16))
goto out;
}
if (key) {
rep.opt[0] = htonl((TCPOPT_NOP << 24) |
(TCPOPT_NOP << 16) |
(TCPOPT_MD5SIG << 8) |
TCPOLEN_MD5SIG);
/* Update length and the length the header thinks exists */
arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
rep.th.doff = arg.iov[0].iov_len / 4;
tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[1],
key, ip_hdr(skb)->saddr,
ip_hdr(skb)->daddr, &rep.th);
}
#endif
/* Can't co-exist with TCPMD5, hence check rep.opt[0] */
if (rep.opt[0] == 0) {
__be32 mrst = mptcp_reset_option(skb);
if (mrst) {
rep.opt[0] = mrst;
arg.iov[0].iov_len += sizeof(mrst);
rep.th.doff = arg.iov[0].iov_len / 4;
}
}
arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
ip_hdr(skb)->saddr, /* XXX */
arg.iov[0].iov_len, IPPROTO_TCP, 0);
arg.csumoffset = offsetof(struct tcphdr, check) / 2;
arg.flags = (sk && inet_sk_transparent(sk)) ? IP_REPLY_ARG_NOSRCCHECK : 0;
/* When socket is gone, all binding information is lost.
* routing might fail in this case. No choice here, if we choose to force
* input interface, we will misroute in case of asymmetric route.
*/
if (sk)
arg.bound_dev_if = sk->sk_bound_dev_if;
trace_tcp_send_reset(sk, skb, reason);
BUILD_BUG_ON(offsetof(struct sock, sk_bound_dev_if) !=
offsetof(struct inet_timewait_sock, tw_bound_dev_if));
/* ECN bits of TW reset are cleared */
arg.tos = ip_hdr(skb)->tos & ~INET_ECN_MASK;
arg.uid = sock_net_uid(net, sk && sk_fullsock(sk) ? sk : NULL);
local_bh_disable();
local_lock_nested_bh(&ipv4_tcp_sk.bh_lock);
ctl_sk = this_cpu_read(ipv4_tcp_sk.sock);
sock_net_set(ctl_sk, net);
if (sk) {
ctl_sk->sk_mark = (sk->sk_state == TCP_TIME_WAIT) ?
inet_twsk(sk)->tw_mark : READ_ONCE(sk->sk_mark);
ctl_sk->sk_priority = (sk->sk_state == TCP_TIME_WAIT) ?
inet_twsk(sk)->tw_priority : READ_ONCE(sk->sk_priority);
transmit_time = tcp_transmit_time(sk);
xfrm_sk_clone_policy(ctl_sk, sk);
txhash = (sk->sk_state == TCP_TIME_WAIT) ?
inet_twsk(sk)->tw_txhash : sk->sk_txhash;
} else {
ctl_sk->sk_mark = 0;
ctl_sk->sk_priority = 0;
}
ip_send_unicast_reply(ctl_sk, sk,
skb, &TCP_SKB_CB(skb)->header.h4.opt,
ip_hdr(skb)->saddr, ip_hdr(skb)->daddr,
&arg, arg.iov[0].iov_len,
transmit_time, txhash);
xfrm_sk_free_policy(ctl_sk);
sock_net_set(ctl_sk, &init_net);
__TCP_INC_STATS(net, TCP_MIB_OUTSEGS);
__TCP_INC_STATS(net, TCP_MIB_OUTRSTS);
local_unlock_nested_bh(&ipv4_tcp_sk.bh_lock);
local_bh_enable();
#ifdef CONFIG_TCP_MD5SIG
out:
rcu_read_unlock();
#endif
}
/* The code following below sending ACKs in SYN-RECV and TIME-WAIT states
outside socket context is ugly, certainly. What can I do?
*/
static void tcp_v4_send_ack(const struct sock *sk,
struct sk_buff *skb, u32 seq, u32 ack,
u32 win, u32 tsval, u32 tsecr, int oif,
struct tcp_key *key,
int reply_flags, u8 tos, u32 txhash)
{
const struct tcphdr *th = tcp_hdr(skb);
struct {
struct tcphdr th;
__be32 opt[(MAX_TCP_OPTION_SPACE >> 2)];
} rep;
struct net *net = sock_net(sk);
struct ip_reply_arg arg;
struct sock *ctl_sk;
u64 transmit_time;
memset(&rep.th, 0, sizeof(struct tcphdr));
memset(&arg, 0, sizeof(arg));
arg.iov[0].iov_base = (unsigned char *)&rep;
arg.iov[0].iov_len = sizeof(rep.th);
if (tsecr) {
rep.opt[0] = htonl((TCPOPT_NOP << 24) | (TCPOPT_NOP << 16) |
(TCPOPT_TIMESTAMP << 8) |
TCPOLEN_TIMESTAMP);
rep.opt[1] = htonl(tsval);
rep.opt[2] = htonl(tsecr);
arg.iov[0].iov_len += TCPOLEN_TSTAMP_ALIGNED;
}
/* Swap the send and the receive. */
rep.th.dest = th->source;
rep.th.source = th->dest;
rep.th.doff = arg.iov[0].iov_len / 4;
rep.th.seq = htonl(seq);
rep.th.ack_seq = htonl(ack);
rep.th.ack = 1;
rep.th.window = htons(win);
#ifdef CONFIG_TCP_MD5SIG
if (tcp_key_is_md5(key)) {
int offset = (tsecr) ? 3 : 0;
rep.opt[offset++] = htonl((TCPOPT_NOP << 24) |
(TCPOPT_NOP << 16) |
(TCPOPT_MD5SIG << 8) |
TCPOLEN_MD5SIG);
arg.iov[0].iov_len += TCPOLEN_MD5SIG_ALIGNED;
rep.th.doff = arg.iov[0].iov_len/4;
tcp_v4_md5_hash_hdr((__u8 *) &rep.opt[offset],
key->md5_key, ip_hdr(skb)->saddr,
ip_hdr(skb)->daddr, &rep.th);
}
#endif
#ifdef CONFIG_TCP_AO
if (tcp_key_is_ao(key)) {
int offset = (tsecr) ? 3 : 0;
rep.opt[offset++] = htonl((TCPOPT_AO << 24) |
(tcp_ao_len(key->ao_key) << 16) |
(key->ao_key->sndid << 8) |
key->rcv_next);
arg.iov[0].iov_len += tcp_ao_len_aligned(key->ao_key);
rep.th.doff = arg.iov[0].iov_len / 4;
tcp_ao_hash_hdr(AF_INET, (char *)&rep.opt[offset],
key->ao_key, key->traffic_key,
(union tcp_ao_addr *)&ip_hdr(skb)->saddr,
(union tcp_ao_addr *)&ip_hdr(skb)->daddr,
&rep.th, key->sne);
}
#endif
arg.flags = reply_flags;
arg.csum = csum_tcpudp_nofold(ip_hdr(skb)->daddr,
ip_hdr(skb)->saddr, /* XXX */
arg.iov[0].iov_len, IPPROTO_TCP, 0);
arg.csumoffset = offsetof(struct tcphdr, check) / 2;
if (oif)
arg.bound_dev_if = oif;
arg.tos = tos;
arg.uid = sock_net_uid(net, sk_fullsock(sk) ? sk : NULL);
local_bh_disable();
local_lock_nested_bh(&ipv4_tcp_sk.bh_lock);
ctl_sk = this_cpu_read(ipv4_tcp_sk.sock);
sock_net_set(ctl_sk, net);
ctl_sk->sk_mark = (sk->sk_state == TCP_TIME_WAIT) ?
inet_twsk(sk)->tw_mark : READ_ONCE(sk->sk_mark);
ctl_sk->sk_priority = (sk->sk_state == TCP_TIME_WAIT) ?
inet_twsk(sk)->tw_priority : READ_ONCE(sk->sk_priority);
transmit_time = tcp_transmit_time(sk);
ip_send_unicast_reply(ctl_sk, sk,
skb, &TCP_SKB_CB(skb)->header.h4.opt,
ip_hdr(skb)->saddr, ip_hdr(skb)->daddr,
&arg, arg.iov[0].iov_len,
transmit_time, txhash);
sock_net_set(ctl_sk, &init_net);
__TCP_INC_STATS(net, TCP_MIB_OUTSEGS);
local_unlock_nested_bh(&ipv4_tcp_sk.bh_lock);
local_bh_enable();
}
static void tcp_v4_timewait_ack(struct sock *sk, struct sk_buff *skb,
enum tcp_tw_status tw_status)
{
struct inet_timewait_sock *tw = inet_twsk(sk);
struct tcp_timewait_sock *tcptw = tcp_twsk(sk);
struct tcp_key key = {};
u8 tos = tw->tw_tos;
/* Cleaning only ECN bits of TW ACKs of oow data or is paws_reject,
* while not cleaning ECN bits of other TW ACKs to avoid these ACKs
* being placed in a different service queues (Classic rather than L4S)
*/
if (tw_status == TCP_TW_ACK_OOW)
tos &= ~INET_ECN_MASK;
#ifdef CONFIG_TCP_AO
struct tcp_ao_info *ao_info;
if (static_branch_unlikely(&tcp_ao_needed.key)) {
/* FIXME: the segment to-be-acked is not verified yet */
ao_info = rcu_dereference(tcptw->ao_info);
if (ao_info) {
const struct tcp_ao_hdr *aoh;
if (tcp_parse_auth_options(tcp_hdr(skb), NULL, &aoh)) {
inet_twsk_put(tw);
return;
}
if (aoh)
key.ao_key = tcp_ao_established_key(sk, ao_info,
aoh->rnext_keyid, -1);
}
}
if (key.ao_key) {
struct tcp_ao_key *rnext_key;
key.traffic_key = snd_other_key(key.ao_key);
key.sne = READ_ONCE(ao_info->snd_sne);
rnext_key = READ_ONCE(ao_info->rnext_key);
key.rcv_next = rnext_key->rcvid;
key.type = TCP_KEY_AO;
#else
if (0) {
#endif
} else if (static_branch_tcp_md5()) {
key.md5_key = tcp_twsk_md5_key(tcptw);
if (key.md5_key)
key.type = TCP_KEY_MD5;
}
tcp_v4_send_ack(sk, skb,
tcptw->tw_snd_nxt, READ_ONCE(tcptw->tw_rcv_nxt),
tcptw->tw_rcv_wnd >> tw->tw_rcv_wscale,
tcp_tw_tsval(tcptw),
READ_ONCE(tcptw->tw_ts_recent),
tw->tw_bound_dev_if, &key,
tw->tw_transparent ? IP_REPLY_ARG_NOSRCCHECK : 0,
tos,
tw->tw_txhash);
inet_twsk_put(tw);
}
static void tcp_v4_reqsk_send_ack(const struct sock *sk, struct sk_buff *skb,
struct request_sock *req)
{
struct tcp_key key = {};
/* sk->sk_state == TCP_LISTEN -> for regular TCP_SYN_RECV
* sk->sk_state == TCP_SYN_RECV -> for Fast Open.
*/
u32 seq = (sk->sk_state == TCP_LISTEN) ? tcp_rsk(req)->snt_isn + 1 :
tcp_sk(sk)->snd_nxt;
#ifdef CONFIG_TCP_AO
if (static_branch_unlikely(&tcp_ao_needed.key) &&
tcp_rsk_used_ao(req)) {
const union tcp_md5_addr *addr;
const struct tcp_ao_hdr *aoh;
int l3index;
/* Invalid TCP option size or twice included auth */
if (tcp_parse_auth_options(tcp_hdr(skb), NULL, &aoh))
return;
if (!aoh)
return;
addr = (union tcp_md5_addr *)&ip_hdr(skb)->saddr;
l3index = tcp_v4_sdif(skb) ? inet_iif(skb) : 0;
key.ao_key = tcp_ao_do_lookup(sk, l3index, addr, AF_INET,
aoh->rnext_keyid, -1);
if (unlikely(!key.ao_key)) {
/* Send ACK with any matching MKT for the peer */
key.ao_key = tcp_ao_do_lookup(sk, l3index, addr, AF_INET, -1, -1);
/* Matching key disappeared (user removed the key?)
* let the handshake timeout.
*/
if (!key.ao_key) {
net_info_ratelimited("TCP-AO key for (%pI4, %d)->(%pI4, %d) suddenly disappeared, won't ACK new connection\n",
addr,
ntohs(tcp_hdr(skb)->source),
&ip_hdr(skb)->daddr,
ntohs(tcp_hdr(skb)->dest));
return;
}
}
key.traffic_key = kmalloc(tcp_ao_digest_size(key.ao_key), GFP_ATOMIC);
if (!key.traffic_key)
return;
key.type = TCP_KEY_AO;
key.rcv_next = aoh->keyid;
tcp_v4_ao_calc_key_rsk(key.ao_key, key.traffic_key, req);
#else
if (0) {
#endif
} else if (static_branch_tcp_md5()) {
const union tcp_md5_addr *addr;
int l3index;
addr = (union tcp_md5_addr *)&ip_hdr(skb)->saddr;
l3index = tcp_v4_sdif(skb) ? inet_iif(skb) : 0;
key.md5_key = tcp_md5_do_lookup(sk, l3index, addr, AF_INET);
if (key.md5_key)
key.type = TCP_KEY_MD5;
}
/* Cleaning ECN bits of TW ACKs of oow data or is paws_reject */
tcp_v4_send_ack(sk, skb, seq,
tcp_rsk(req)->rcv_nxt,
tcp_synack_window(req) >> inet_rsk(req)->rcv_wscale,
tcp_rsk_tsval(tcp_rsk(req)),
req->ts_recent,
0, &key,
inet_rsk(req)->no_srccheck ? IP_REPLY_ARG_NOSRCCHECK : 0,
ip_hdr(skb)->tos & ~INET_ECN_MASK,
READ_ONCE(tcp_rsk(req)->txhash));
if (tcp_key_is_ao(&key))
kfree(key.traffic_key);
}
/*
* Send a SYN-ACK after having received a SYN.
* This still operates on a request_sock only, not on a big
* socket.
*/
static int tcp_v4_send_synack(const struct sock *sk, struct dst_entry *dst,
struct flowi *fl,
struct request_sock *req,
struct tcp_fastopen_cookie *foc,
enum tcp_synack_type synack_type,
struct sk_buff *syn_skb)
{
struct inet_request_sock *ireq = inet_rsk(req);
struct flowi4 fl4;
int err = -1;
struct sk_buff *skb;
u8 tos;
/* First, grab a route. */
if (!dst && (dst = inet_csk_route_req(sk, &fl4, req)) == NULL)
return -1;
skb = tcp_make_synack(sk, dst, req, foc, synack_type, syn_skb);
if (skb) {
tcp_rsk(req)->syn_ect_snt = inet_sk(sk)->tos & INET_ECN_MASK;
__tcp_v4_send_check(skb, ireq->ir_loc_addr, ireq->ir_rmt_addr);
tos = READ_ONCE(inet_sk(sk)->tos);
if (READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_reflect_tos))
tos = (tcp_rsk(req)->syn_tos & ~INET_ECN_MASK) |
(tos & INET_ECN_MASK);
if (!INET_ECN_is_capable(tos) &&
tcp_bpf_ca_needs_ecn((struct sock *)req))
tos |= INET_ECN_ECT_0;
rcu_read_lock();
err = ip_build_and_send_pkt(skb, sk, ireq->ir_loc_addr,
ireq->ir_rmt_addr,
rcu_dereference(ireq->ireq_opt),
tos);
rcu_read_unlock();
err = net_xmit_eval(err);
}
return err;
}
/*
* IPv4 request_sock destructor.
*/
static void tcp_v4_reqsk_destructor(struct request_sock *req)
{
kfree(rcu_dereference_protected(inet_rsk(req)->ireq_opt, 1));
}
#ifdef CONFIG_TCP_MD5SIG
/*
* RFC2385 MD5 checksumming requires a mapping of
* IP address->MD5 Key.
* We need to maintain these in the sk structure.
*/
DEFINE_STATIC_KEY_DEFERRED_FALSE(tcp_md5_needed, HZ);
static bool better_md5_match(struct tcp_md5sig_key *old, struct tcp_md5sig_key *new)
{
if (!old)
return true;
/* l3index always overrides non-l3index */
if (old->l3index && new->l3index == 0)
return false;
if (old->l3index == 0 && new->l3index)
return true;
return old->prefixlen < new->prefixlen;
}
/* Find the Key structure for an address. */
struct tcp_md5sig_key *__tcp_md5_do_lookup(const struct sock *sk, int l3index,
const union tcp_md5_addr *addr,
int family, bool any_l3index)
{
const struct tcp_sock *tp = tcp_sk(sk);
struct tcp_md5sig_key *key;
const struct tcp_md5sig_info *md5sig;
__be32 mask;
struct tcp_md5sig_key *best_match = NULL;
bool match;
/* caller either holds rcu_read_lock() or socket lock */
md5sig = rcu_dereference_check(tp->md5sig_info,
lockdep_sock_is_held(sk));
if (!md5sig)
return NULL;
hlist_for_each_entry_rcu(key, &md5sig->head, node,
lockdep_sock_is_held(sk)) {
if (key->family != family)
continue;
if (!any_l3index && key->flags & TCP_MD5SIG_FLAG_IFINDEX &&
key->l3index != l3index)
continue;
if (family == AF_INET) {
mask = inet_make_mask(key->prefixlen);
match = (key->addr.a4.s_addr & mask) ==
(addr->a4.s_addr & mask);
#if IS_ENABLED(CONFIG_IPV6)
} else if (family == AF_INET6) {
match = ipv6_prefix_equal(&key->addr.a6, &addr->a6,
key->prefixlen);
#endif
} else {
match = false;
}
if (match && better_md5_match(best_match, key))
best_match = key;
}
return best_match;
}
static struct tcp_md5sig_key *tcp_md5_do_lookup_exact(const struct sock *sk,
const union tcp_md5_addr *addr,
int family, u8 prefixlen,
int l3index, u8 flags)
{
const struct tcp_sock *tp = tcp_sk(sk);
struct tcp_md5sig_key *key;
unsigned int size = sizeof(struct in_addr);
const struct tcp_md5sig_info *md5sig;
/* caller either holds rcu_read_lock() or socket lock */
md5sig = rcu_dereference_check(tp->md5sig_info,
lockdep_sock_is_held(sk));
if (!md5sig)
return NULL;
#if IS_ENABLED(CONFIG_IPV6)
if (family == AF_INET6)
size = sizeof(struct in6_addr);
#endif
hlist_for_each_entry_rcu(key, &md5sig->head, node,
lockdep_sock_is_held(sk)) {
if (key->family != family)
continue;
if ((key->flags & TCP_MD5SIG_FLAG_IFINDEX) != (flags & TCP_MD5SIG_FLAG_IFINDEX))
continue;
if (key->l3index != l3index)
continue;
if (!memcmp(&key->addr, addr, size) &&
key->prefixlen == prefixlen)
return key;
}
return NULL;
}
struct tcp_md5sig_key *tcp_v4_md5_lookup(const struct sock *sk,
const struct sock *addr_sk)
{
const union tcp_md5_addr *addr;
int l3index;
l3index = l3mdev_master_ifindex_by_index(sock_net(sk),
addr_sk->sk_bound_dev_if);
addr = (const union tcp_md5_addr *)&addr_sk->sk_daddr;
return tcp_md5_do_lookup(sk, l3index, addr, AF_INET);
}
static int tcp_md5sig_info_add(struct sock *sk, gfp_t gfp)
{
struct tcp_sock *tp = tcp_sk(sk);
struct tcp_md5sig_info *md5sig;
md5sig = kmalloc_obj(*md5sig, gfp);
if (!md5sig)
return -ENOMEM;
sk_gso_disable(sk);
INIT_HLIST_HEAD(&md5sig->head);
rcu_assign_pointer(tp->md5sig_info, md5sig);
return 0;
}
/* This can be called on a newly created socket, from other files */
static int __tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,
int family, u8 prefixlen, int l3index, u8 flags,
const u8 *newkey, u8 newkeylen, gfp_t gfp)
{
/* Add Key to the list */
struct tcp_md5sig_key *key;
struct tcp_sock *tp = tcp_sk(sk);
struct tcp_md5sig_info *md5sig;
key = tcp_md5_do_lookup_exact(sk, addr, family, prefixlen, l3index, flags);
if (key) {
/* Pre-existing entry - just update that one.
* Note that the key might be used concurrently.
* data_race() is telling kcsan that we do not care of
* key mismatches, since changing MD5 key on live flows
* can lead to packet drops.
*/
data_race(memcpy(key->key, newkey, newkeylen));
/* Pairs with READ_ONCE() in tcp_md5_hash_key().
* Also note that a reader could catch new key->keylen value
* but old key->key[], this is the reason we use __GFP_ZERO
* at sock_kmalloc() time below these lines.
*/
WRITE_ONCE(key->keylen, newkeylen);
return 0;
}
md5sig = rcu_dereference_protected(tp->md5sig_info,
lockdep_sock_is_held(sk));
key = sock_kmalloc(sk, sizeof(*key), gfp | __GFP_ZERO);
if (!key)
return -ENOMEM;
memcpy(key->key, newkey, newkeylen);
key->keylen = newkeylen;
key->family = family;
key->prefixlen = prefixlen;
key->l3index = l3index;
key->flags = flags;
memcpy(&key->addr, addr,
(IS_ENABLED(CONFIG_IPV6) && family == AF_INET6) ? sizeof(struct in6_addr) :
sizeof(struct in_addr));
hlist_add_head_rcu(&key->node, &md5sig->head);
return 0;
}
int tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr,
int family, u8 prefixlen, int l3index, u8 flags,
const u8 *newkey, u8 newkeylen)
{
struct tcp_sock *tp = tcp_sk(sk);
if (!rcu_dereference_protected(tp->md5sig_info, lockdep_sock_is_held(sk))) {
if (fips_enabled) {
pr_warn_once("TCP-MD5 support is disabled due to FIPS\n");
return -EOPNOTSUPP;
}
if (tcp_md5sig_info_add(sk, GFP_KERNEL))
return -ENOMEM;
if (!static_branch_inc(&tcp_md5_needed.key)) {
struct tcp_md5sig_info *md5sig;
md5sig = rcu_dereference_protected(tp->md5sig_info, lockdep_sock_is_held(sk));
rcu_assign_pointer(tp->md5sig_info, NULL);
kfree_rcu(md5sig, rcu);
return -EUSERS;
}
}
return __tcp_md5_do_add(sk, addr, family, prefixlen, l3index, flags,
newkey, newkeylen, GFP_KERNEL);
}
int tcp_md5_key_copy(struct sock *sk, const union tcp_md5_addr *addr,
int family, u8 prefixlen, int l3index,
struct tcp_md5sig_key *key)
{
struct tcp_sock *tp = tcp_sk(sk);
if (!rcu_dereference_protected(tp->md5sig_info, lockdep_sock_is_held(sk))) {
if (tcp_md5sig_info_add(sk, sk_gfp_mask(sk, GFP_ATOMIC)))
return -ENOMEM;
if (!static_key_fast_inc_not_disabled(&tcp_md5_needed.key.key)) {
struct tcp_md5sig_info *md5sig;
md5sig = rcu_dereference_protected(tp->md5sig_info, lockdep_sock_is_held(sk));
net_warn_ratelimited("Too many TCP-MD5 keys in the system\n");
rcu_assign_pointer(tp->md5sig_info, NULL);
kfree_rcu(md5sig, rcu);
return -EUSERS;
}
}
return __tcp_md5_do_add(sk, addr, family, prefixlen, l3index,
key->flags, key->key, key->keylen,
sk_gfp_mask(sk, GFP_ATOMIC));
}
int tcp_md5_do_del(struct sock *sk, const union tcp_md5_addr *addr, int family,
u8 prefixlen, int l3index, u8 flags)
{
struct tcp_md5sig_key *key;
key = tcp_md5_do_lookup_exact(sk, addr, family, prefixlen, l3index, flags);
if (!key)
return -ENOENT;
hlist_del_rcu(&key->node);
atomic_sub(sizeof(*key), &sk->sk_omem_alloc);
kfree_rcu(key, rcu);
return 0;
}
void tcp_clear_md5_list(struct sock *sk)
{
struct tcp_sock *tp = tcp_sk(sk);
struct tcp_md5sig_key *key;
struct hlist_node *n;
struct tcp_md5sig_info *md5sig;
md5sig = rcu_dereference_protected(tp->md5sig_info, 1);
hlist_for_each_entry_safe(key, n, &md5sig->head, node) {
hlist_del(&key->node);
atomic_sub(sizeof(*key), &sk->sk_omem_alloc);
kfree(key);
}
}
static int tcp_v4_parse_md5_keys(struct sock *sk, int optname,
sockptr_t optval, int optlen)
{
struct tcp_md5sig cmd;
struct sockaddr_in *sin = (struct sockaddr_in *)&cmd.tcpm_addr;
const union tcp_md5_addr *addr;
u8 prefixlen = 32;
int l3index = 0;
bool l3flag;
u8 flags;
if (optlen < sizeof(cmd))
return -EINVAL;
if (copy_from_sockptr(&cmd, optval, sizeof(cmd)))
return -EFAULT;
if (sin->sin_family != AF_INET)
return -EINVAL;
flags = cmd.tcpm_flags & TCP_MD5SIG_FLAG_IFINDEX;
l3flag = cmd.tcpm_flags & TCP_MD5SIG_FLAG_IFINDEX;
if (optname == TCP_MD5SIG_EXT &&
cmd.tcpm_flags & TCP_MD5SIG_FLAG_PREFIX) {
prefixlen = cmd.tcpm_prefixlen;
if (prefixlen > 32)
return -EINVAL;
}
if (optname == TCP_MD5SIG_EXT && cmd.tcpm_ifindex &&
cmd.tcpm_flags & TCP_MD5SIG_FLAG_IFINDEX) {
struct net_device *dev;
rcu_read_lock();
dev = dev_get_by_index_rcu(sock_net(sk), cmd.tcpm_ifindex);
if (dev && netif_is_l3_master(dev))
l3index = dev->ifindex;
rcu_read_unlock();
/* ok to reference set/not set outside of rcu;
* right now device MUST be an L3 master
*/
if (!dev || !l3index)
return -EINVAL;
}
addr = (union tcp_md5_addr *)&sin->sin_addr.s_addr;
if (!cmd.tcpm_keylen)
return tcp_md5_do_del(sk, addr, AF_INET, prefixlen, l3index, flags);
if (cmd.tcpm_keylen > TCP_MD5SIG_MAXKEYLEN)
return -EINVAL;
/* Don't allow keys for peers that have a matching TCP-AO key.
* See the comment in tcp_ao_add_cmd()
*/
if (tcp_ao_required(sk, addr, AF_INET, l3flag ? l3index : -1, false))
return -EKEYREJECTED;
return tcp_md5_do_add(sk, addr, AF_INET, prefixlen, l3index, flags,
cmd.tcpm_key, cmd.tcpm_keylen);
}
static void tcp_v4_md5_hash_headers(struct md5_ctx *ctx,
__be32 daddr, __be32 saddr,
const struct tcphdr *th, int nbytes)
{
struct {
struct tcp4_pseudohdr ip;
struct tcphdr tcp;
} h;
h.ip.saddr = saddr;
h.ip.daddr = daddr;
h.ip.pad = 0;
h.ip.protocol = IPPROTO_TCP;
h.ip.len = cpu_to_be16(nbytes);
h.tcp = *th;
h.tcp.check = 0;
md5_update(ctx, (const u8 *)&h, sizeof(h.ip) + sizeof(h.tcp));
}
static noinline_for_stack void
tcp_v4_md5_hash_hdr(char *md5_hash, const struct tcp_md5sig_key *key,
__be32 daddr, __be32 saddr, const struct tcphdr *th)
{
struct md5_ctx ctx;
md5_init(&ctx);
tcp_v4_md5_hash_headers(&ctx, daddr, saddr, th, th->doff << 2);
tcp_md5_hash_key(&ctx, key);
md5_final(&ctx, md5_hash);
}
noinline_for_stack void
tcp_v4_md5_hash_skb(char *md5_hash, const struct tcp_md5sig_key *key,
const struct sock *sk, const struct sk_buff *skb)
{
const struct tcphdr *th = tcp_hdr(skb);
__be32 saddr, daddr;
struct md5_ctx ctx;
if (sk) { /* valid for establish/request sockets */
saddr = sk->sk_rcv_saddr;
daddr = sk->sk_daddr;
} else {
const struct iphdr *iph = ip_hdr(skb);
saddr = iph->saddr;
daddr = iph->daddr;
}
md5_init(&ctx);
tcp_v4_md5_hash_headers(&ctx, daddr, saddr, th, skb->len);
tcp_md5_hash_skb_data(&ctx, skb, th->doff << 2);
tcp_md5_hash_key(&ctx, key);
md5_final(&ctx, md5_hash);
}
#endif
static void tcp_v4_init_req(struct request_sock *req,
const struct sock *sk_listener,
struct sk_buff *skb)
{
struct inet_request_sock *ireq = inet_rsk(req);
struct net *net = sock_net(sk_listener);
sk_rcv_saddr_set(req_to_sk(req), ip_hdr(skb)->daddr);
sk_daddr_set(req_to_sk(req), ip_hdr(skb)->saddr);
RCU_INIT_POINTER(ireq->ireq_opt, tcp_v4_save_options(net, skb));
}
static struct dst_entry *tcp_v4_route_req(const struct sock *sk,
struct sk_buff *skb,
struct flowi *fl,
struct request_sock *req,
u32 tw_isn)
{
tcp_v4_init_req(req, sk, skb);
if (security_inet_conn_request(sk, skb, req))
return NULL;
return inet_csk_route_req(sk, &fl->u.ip4, req);
}
struct request_sock_ops tcp_request_sock_ops __read_mostly = {
.family = PF_INET,
.obj_size = sizeof(struct tcp_request_sock),
.send_ack = tcp_v4_reqsk_send_ack,
.destructor = tcp_v4_reqsk_destructor,
.send_reset = tcp_v4_send_reset,
};
const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops = {
.mss_clamp = TCP_MSS_DEFAULT,
#ifdef CONFIG_TCP_MD5SIG
.req_md5_lookup = tcp_v4_md5_lookup,
.calc_md5_hash = tcp_v4_md5_hash_skb,
#endif
#ifdef CONFIG_TCP_AO
.ao_lookup = tcp_v4_ao_lookup_rsk,
.ao_calc_key = tcp_v4_ao_calc_key_rsk,
.ao_synack_hash = tcp_v4_ao_synack_hash,
#endif
#ifdef CONFIG_SYN_COOKIES
.cookie_init_seq = cookie_v4_init_sequence,
#endif
.route_req = tcp_v4_route_req,
.init_seq_and_ts_off = tcp_v4_init_seq_and_ts_off,
.send_synack = tcp_v4_send_synack,
};
int tcp_v4_conn_request(struct sock *sk, struct sk_buff *skb)
{
/* Never answer to SYNs send to broadcast or multicast */
if (skb_rtable(skb)->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST))
goto drop;
return tcp_conn_request(&tcp_request_sock_ops,
&tcp_request_sock_ipv4_ops, sk, skb);
drop:
tcp_listendrop(sk);
return 0;
}
/*
* The three way handshake has completed - we got a valid synack -
* now create the new socket.
*/
struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
struct request_sock *req,
struct dst_entry *dst,
struct request_sock *req_unhash,
bool *own_req,
void (*opt_child_init)(struct sock *newsk,
const struct sock *sk))
{
struct inet_request_sock *ireq;
bool found_dup_sk = false;
struct inet_sock *newinet;
struct tcp_sock *newtp;
struct sock *newsk;
#ifdef CONFIG_TCP_MD5SIG
const union tcp_md5_addr *addr;
struct tcp_md5sig_key *key;
int l3index;
#endif
struct ip_options_rcu *inet_opt;
if (sk_acceptq_is_full(sk))
goto exit_overflow;
newsk = tcp_create_openreq_child(sk, req, skb);
if (!newsk)
goto exit_nonewsk;
newsk->sk_gso_type = SKB_GSO_TCPV4;
inet_sk_rx_dst_set(newsk, skb);
newtp = tcp_sk(newsk);
newinet = inet_sk(newsk);
ireq = inet_rsk(req);
inet_opt = rcu_dereference(ireq->ireq_opt);
RCU_INIT_POINTER(newinet->inet_opt, inet_opt);
newinet->mc_index = inet_iif(skb);
newinet->mc_ttl = ip_hdr(skb)->ttl;
newinet->rcv_tos = ip_hdr(skb)->tos;
inet_csk(newsk)->icsk_ext_hdr_len = 0;
if (inet_opt)
inet_csk(newsk)->icsk_ext_hdr_len = inet_opt->opt.optlen;
atomic_set(&newinet->inet_id, get_random_u16());
/* Set ToS of the new socket based upon the value of incoming SYN.
* ECT bits are set later in tcp_init_transfer().
*/
if (READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_reflect_tos))
newinet->tos = tcp_rsk(req)->syn_tos & ~INET_ECN_MASK;
if (!dst) {
dst = inet_csk_route_child_sock(sk, newsk, req);
if (!dst)
goto put_and_exit;
} else {
/* syncookie case : see end of cookie_v4_check() */
}
sk_setup_caps(newsk, dst);
#if IS_ENABLED(CONFIG_IPV6)
if (opt_child_init)
opt_child_init(newsk, sk);
#endif
tcp_ca_openreq_child(newsk, dst);
tcp_sync_mss(newsk, dst4_mtu(dst));
newtp->advmss = tcp_mss_clamp(tcp_sk(sk), dst_metric_advmss(dst));
tcp_initialize_rcv_mss(newsk);
#ifdef CONFIG_TCP_MD5SIG
l3index = l3mdev_master_ifindex_by_index(sock_net(sk), ireq->ir_iif);
/* Copy over the MD5 key from the original socket */
addr = (union tcp_md5_addr *)&newinet->inet_daddr;
key = tcp_md5_do_lookup(sk, l3index, addr, AF_INET);
if (key && !tcp_rsk_used_ao(req)) {
if (tcp_md5_key_copy(newsk, addr, AF_INET, 32, l3index, key))
goto put_and_exit;
sk_gso_disable(newsk);
}
#endif
#ifdef CONFIG_TCP_AO
if (tcp_ao_copy_all_matching(sk, newsk, req, skb, AF_INET))
goto put_and_exit; /* OOM, release back memory */
#endif
if (__inet_inherit_port(sk, newsk) < 0)
goto put_and_exit;
*own_req = inet_ehash_nolisten(newsk, req_to_sk(req_unhash),
&found_dup_sk);
if (likely(*own_req)) {
tcp_move_syn(newtp, req);
ireq->ireq_opt = NULL;
} else {
newinet->inet_opt = NULL;
if (!req_unhash && found_dup_sk) {
/* This code path should only be executed in the
* syncookie case only
*/
bh_unlock_sock(newsk);
sock_put(newsk);
newsk = NULL;
}
}
return newsk;
exit_overflow:
NET_INC_STATS(sock_net(sk), LINUX_MIB_LISTENOVERFLOWS);
exit_nonewsk:
dst_release(dst);
exit:
tcp_listendrop(sk);
return NULL;
put_and_exit:
newinet->inet_opt = NULL;
inet_csk_prepare_forced_close(newsk);
tcp_done(newsk);
goto exit;
}
static struct sock *tcp_v4_cookie_check(struct sock *sk, struct sk_buff *skb)
{
#ifdef CONFIG_SYN_COOKIES
const struct tcphdr *th = tcp_hdr(skb);
if (!th->syn)
sk = cookie_v4_check(sk, skb);
#endif
return sk;
}
u16 tcp_v4_get_syncookie(struct sock *sk, struct iphdr *iph,
struct tcphdr *th, u32 *cookie)
{
u16 mss = 0;
#ifdef CONFIG_SYN_COOKIES
mss = tcp_get_syncookie_mss(&tcp_request_sock_ops,
&tcp_request_sock_ipv4_ops, sk, th);
if (mss) {
*cookie = __cookie_v4_init_sequence(iph, th, &mss);
tcp_synq_overflow(sk);
}
#endif
return mss;
}
INDIRECT_CALLABLE_DECLARE(struct dst_entry *ipv4_dst_check(struct dst_entry *,
u32));
/* The socket must have it's spinlock held when we get
* here, unless it is a TCP_LISTEN socket.
*
* We have a potential double-lock case here, so even when
* doing backlog processing we use the BH locking scheme.
* This is because we cannot sleep with the original spinlock
* held.
*/
int tcp_v4_do_rcv(struct sock *sk, struct sk_buff *skb)
{
enum skb_drop_reason reason;
struct sock *rsk;
reason = psp_sk_rx_policy_check(sk, skb);
if (reason)
goto err_discard;
if (sk->sk_state == TCP_ESTABLISHED) { /* Fast path */
struct dst_entry *dst;
dst = rcu_dereference_protected(sk->sk_rx_dst,
lockdep_sock_is_held(sk));
sock_rps_save_rxhash(sk, skb);
sk_mark_napi_id(sk, skb);
if (dst && unlikely(dst != skb_dst(skb))) {
if (sk->sk_rx_dst_ifindex != skb->skb_iif ||
!INDIRECT_CALL_1(dst->ops->check, ipv4_dst_check,
dst, 0)) {
RCU_INIT_POINTER(sk->sk_rx_dst, NULL);
dst_release(dst);
}
}
tcp_rcv_established(sk, skb);
return 0;
}
if (tcp_checksum_complete(skb))
goto csum_err;
if (sk->sk_state == TCP_LISTEN) {
struct sock *nsk = tcp_v4_cookie_check(sk, skb);
if (!nsk)
return 0;
if (nsk != sk) {
reason = tcp_child_process(sk, nsk, skb);
if (reason) {
rsk = nsk;
goto reset;
}
return 0;
}
} else
sock_rps_save_rxhash(sk, skb);
reason = tcp_rcv_state_process(sk, skb);
if (reason) {
rsk = sk;
goto reset;
}
return 0;
reset:
tcp_v4_send_reset(rsk, skb, sk_rst_convert_drop_reason(reason));
discard:
sk_skb_reason_drop(sk, skb, reason);
/* Be careful here. If this function gets more complicated and
* gcc suffers from register pressure on the x86, sk (in %ebx)
* might be destroyed here. This current version compiles correctly,
* but you have been warned.
*/
return 0;
csum_err:
reason = SKB_DROP_REASON_TCP_CSUM;
trace_tcp_bad_csum(skb);
TCP_INC_STATS(sock_net(sk), TCP_MIB_CSUMERRORS);
err_discard:
TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
goto discard;
}
EXPORT_SYMBOL(tcp_v4_do_rcv);
enum skb_drop_reason tcp_add_backlog(struct sock *sk, struct sk_buff *skb)
{
u32 tail_gso_size, tail_gso_segs;
struct skb_shared_info *shinfo;
const struct tcphdr *th;
struct tcphdr *thtail;
struct sk_buff *tail;
unsigned int hdrlen;
bool fragstolen;
u32 gso_segs;
u32 gso_size;
u64 limit;
int delta;
int err;
/* In case all data was pulled from skb frags (in __pskb_pull_tail()),
* we can fix skb->truesize to its real value to avoid future drops.
* This is valid because skb is not yet charged to the socket.
* It has been noticed pure SACK packets were sometimes dropped
* (if cooked by drivers without copybreak feature).
*/
skb_condense(skb);
tcp_cleanup_skb(skb);
if (unlikely(tcp_checksum_complete(skb))) {
bh_unlock_sock(sk);
trace_tcp_bad_csum(skb);
__TCP_INC_STATS(sock_net(sk), TCP_MIB_CSUMERRORS);
__TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
return SKB_DROP_REASON_TCP_CSUM;
}
/* Attempt coalescing to last skb in backlog, even if we are
* above the limits.
* This is okay because skb capacity is limited to MAX_SKB_FRAGS.
*/
th = (const struct tcphdr *)skb->data;
hdrlen = th->doff * 4;
tail = sk->sk_backlog.tail;
if (!tail)
goto no_coalesce;
thtail = (struct tcphdr *)tail->data;
if (TCP_SKB_CB(tail)->end_seq != TCP_SKB_CB(skb)->seq ||
TCP_SKB_CB(tail)->ip_dsfield != TCP_SKB_CB(skb)->ip_dsfield ||
((TCP_SKB_CB(tail)->tcp_flags |
TCP_SKB_CB(skb)->tcp_flags) & (TCPHDR_SYN | TCPHDR_RST | TCPHDR_URG)) ||
!((TCP_SKB_CB(tail)->tcp_flags &
TCP_SKB_CB(skb)->tcp_flags) & TCPHDR_ACK) ||
((TCP_SKB_CB(tail)->tcp_flags ^
TCP_SKB_CB(skb)->tcp_flags) &
(TCPHDR_ECE | TCPHDR_CWR | TCPHDR_AE)) ||
!tcp_skb_can_collapse_rx(tail, skb) ||
thtail->doff != th->doff ||
memcmp(thtail + 1, th + 1, hdrlen - sizeof(*th)) ||
/* prior to PSP Rx policy check, retain exact PSP metadata */
psp_skb_coalesce_diff(tail, skb))
goto no_coalesce;
__skb_pull(skb, hdrlen);
shinfo = skb_shinfo(skb);
gso_size = shinfo->gso_size ?: skb->len;
gso_segs = shinfo->gso_segs ?: 1;
shinfo = skb_shinfo(tail);
tail_gso_size = shinfo->gso_size ?: (tail->len - hdrlen);
tail_gso_segs = shinfo->gso_segs ?: 1;
if (skb_try_coalesce(tail, skb, &fragstolen, &delta)) {
TCP_SKB_CB(tail)->end_seq = TCP_SKB_CB(skb)->end_seq;
if (likely(!before(TCP_SKB_CB(skb)->ack_seq, TCP_SKB_CB(tail)->ack_seq))) {
TCP_SKB_CB(tail)->ack_seq = TCP_SKB_CB(skb)->ack_seq;
thtail->window = th->window;
}
/* We have to update both TCP_SKB_CB(tail)->tcp_flags and
* thtail->fin, so that the fast path in tcp_rcv_established()
* is not entered if we append a packet with a FIN.
* SYN, RST, URG are not present.
* ACK is set on both packets.
* PSH : we do not really care in TCP stack,
* at least for 'GRO' packets.
*/
thtail->fin |= th->fin;
TCP_SKB_CB(tail)->tcp_flags |= TCP_SKB_CB(skb)->tcp_flags;
if (TCP_SKB_CB(skb)->has_rxtstamp) {
TCP_SKB_CB(tail)->has_rxtstamp = true;
tail->tstamp = skb->tstamp;
skb_hwtstamps(tail)->hwtstamp = skb_hwtstamps(skb)->hwtstamp;
}
/* Not as strict as GRO. We only need to carry mss max value */
shinfo->gso_size = max(gso_size, tail_gso_size);
shinfo->gso_segs = min_t(u32, gso_segs + tail_gso_segs, 0xFFFF);
sk->sk_backlog.len += delta;
__NET_INC_STATS(sock_net(sk),
LINUX_MIB_TCPBACKLOGCOALESCE);
kfree_skb_partial(skb, fragstolen);
return SKB_NOT_DROPPED_YET;
}
__skb_push(skb, hdrlen);
no_coalesce:
/* sk->sk_backlog.len is reset only at the end of __release_sock().
* Both sk->sk_backlog.len and sk->sk_rmem_alloc could reach
* sk_rcvbuf in normal conditions.
*/
limit = ((u64)READ_ONCE(sk->sk_rcvbuf)) << 1;
limit += ((u32)READ_ONCE(sk->sk_sndbuf)) >> 1;
/* Only socket owner can try to collapse/prune rx queues
* to reduce memory overhead, so add a little headroom here.
* Few sockets backlog are possibly concurrently non empty.
*/
limit += 64 * 1024;
limit = min_t(u64, limit, UINT_MAX);
err = sk_add_backlog(sk, skb, limit);
if (unlikely(err)) {
bh_unlock_sock(sk);
if (err == -ENOMEM) {
__NET_INC_STATS(sock_net(sk), LINUX_MIB_PFMEMALLOCDROP);
return SKB_DROP_REASON_PFMEMALLOC;
}
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPBACKLOGDROP);
return SKB_DROP_REASON_SOCKET_BACKLOG;
}
return SKB_NOT_DROPPED_YET;
}
static void tcp_v4_restore_cb(struct sk_buff *skb)
{
memmove(IPCB(skb), &TCP_SKB_CB(skb)->header.h4,
sizeof(struct inet_skb_parm));
}
static void tcp_v4_fill_cb(struct sk_buff *skb, const struct iphdr *iph,
const struct tcphdr *th)
{
/* This is tricky : We move IPCB at its correct location into TCP_SKB_CB()
* barrier() makes sure compiler wont play fool^Waliasing games.
*/
memmove(&TCP_SKB_CB(skb)->header.h4, IPCB(skb),
sizeof(struct inet_skb_parm));
barrier();
TCP_SKB_CB(skb)->seq = ntohl(th->seq);
TCP_SKB_CB(skb)->end_seq = (TCP_SKB_CB(skb)->seq + th->syn + th->fin +
skb->len - th->doff * 4);
TCP_SKB_CB(skb)->ack_seq = ntohl(th->ack_seq);
TCP_SKB_CB(skb)->tcp_flags = tcp_flags_ntohs(th);
TCP_SKB_CB(skb)->ip_dsfield = ipv4_get_dsfield(iph);
TCP_SKB_CB(skb)->sacked = 0;
TCP_SKB_CB(skb)->has_rxtstamp =
skb->tstamp || skb_hwtstamps(skb)->hwtstamp;
}
/*
* From tcp_input.c
*/
int tcp_v4_rcv(struct sk_buff *skb)
{
struct net *net = dev_net_rcu(skb->dev);
enum skb_drop_reason drop_reason;
enum tcp_tw_status tw_status;
int sdif = inet_sdif(skb);
int dif = inet_iif(skb);
const struct iphdr *iph;
const struct tcphdr *th;
struct sock *sk = NULL;
bool refcounted;
int ret;
u32 isn;
drop_reason = SKB_DROP_REASON_NOT_SPECIFIED;
if (skb->pkt_type != PACKET_HOST)
goto discard_it;
/* Count it even if it's bad */
__TCP_INC_STATS(net, TCP_MIB_INSEGS);
if (!pskb_may_pull(skb, sizeof(struct tcphdr)))
goto discard_it;
th = (const struct tcphdr *)skb->data;
if (unlikely(th->doff < sizeof(struct tcphdr) / 4)) {
drop_reason = SKB_DROP_REASON_PKT_TOO_SMALL;
goto bad_packet;
}
if (!pskb_may_pull(skb, th->doff * 4))
goto discard_it;
/* An explanation is required here, I think.
* Packet length and doff are validated by header prediction,
* provided case of th->doff==0 is eliminated.
* So, we defer the checks. */
if (skb_checksum_init(skb, IPPROTO_TCP, inet_compute_pseudo))
goto csum_error;
th = (const struct tcphdr *)skb->data;
iph = ip_hdr(skb);
lookup:
sk = __inet_lookup_skb(skb, __tcp_hdrlen(th), th->source,
th->dest, sdif, &refcounted);
if (!sk)
goto no_tcp_socket;
if (sk->sk_state == TCP_TIME_WAIT)
goto do_time_wait;
if (sk->sk_state == TCP_NEW_SYN_RECV) {
struct request_sock *req = inet_reqsk(sk);
bool req_stolen = false;
struct sock *nsk;
sk = req->rsk_listener;
if (!xfrm4_policy_check(sk, XFRM_POLICY_IN, skb))
drop_reason = SKB_DROP_REASON_XFRM_POLICY;
else
drop_reason = tcp_inbound_hash(sk, req, skb,
&iph->saddr, &iph->daddr,
AF_INET, dif, sdif);
if (unlikely(drop_reason)) {
sk_drops_skbadd(sk, skb);
reqsk_put(req);
goto discard_it;
}
if (tcp_checksum_complete(skb)) {
reqsk_put(req);
goto csum_error;
}
if (unlikely(sk->sk_state != TCP_LISTEN)) {
nsk = reuseport_migrate_sock(sk, req_to_sk(req), skb);
if (!nsk) {
inet_csk_reqsk_queue_drop_and_put(sk, req);
goto lookup;
}
sk = nsk;
/* reuseport_migrate_sock() has already held one sk_refcnt
* before returning.
*/
} else {
/* We own a reference on the listener, increase it again
* as we might lose it too soon.
*/
sock_hold(sk);
}
refcounted = true;
nsk = NULL;
drop_reason = tcp_filter(sk, skb);
if (!drop_reason) {
th = (const struct tcphdr *)skb->data;
iph = ip_hdr(skb);
tcp_v4_fill_cb(skb, iph, th);
nsk = tcp_check_req(sk, skb, req, false, &req_stolen,
&drop_reason);
}
if (!nsk) {
reqsk_put(req);
if (req_stolen) {
/* Another cpu got exclusive access to req
* and created a full blown socket.
* Try to feed this packet to this socket
* instead of discarding it.
*/
tcp_v4_restore_cb(skb);
sock_put(sk);
goto lookup;
}
goto discard_and_relse;
}
nf_reset_ct(skb);
if (nsk == sk) {
reqsk_put(req);
tcp_v4_restore_cb(skb);
} else {
drop_reason = tcp_child_process(sk, nsk, skb);
if (drop_reason) {
enum sk_rst_reason rst_reason;
rst_reason = sk_rst_convert_drop_reason(drop_reason);
tcp_v4_send_reset(nsk, skb, rst_reason);
goto discard_and_relse;
}
sock_put(sk);
return 0;
}
}
process:
if (static_branch_unlikely(&ip4_min_ttl)) {
/* min_ttl can be changed concurrently from do_ip_setsockopt() */
if (unlikely(iph->ttl < READ_ONCE(inet_sk(sk)->min_ttl))) {
__NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP);
drop_reason = SKB_DROP_REASON_TCP_MINTTL;
goto discard_and_relse;
}
}
if (!xfrm4_policy_check(sk, XFRM_POLICY_IN, skb)) {
drop_reason = SKB_DROP_REASON_XFRM_POLICY;
goto discard_and_relse;
}
drop_reason = tcp_inbound_hash(sk, NULL, skb, &iph->saddr, &iph->daddr,
AF_INET, dif, sdif);
if (drop_reason)
goto discard_and_relse;
nf_reset_ct(skb);
drop_reason = tcp_filter(sk, skb);
if (drop_reason)
goto discard_and_relse;
th = (const struct tcphdr *)skb->data;
iph = ip_hdr(skb);
tcp_v4_fill_cb(skb, iph, th);
skb->dev = NULL;
if (sk->sk_state == TCP_LISTEN) {
ret = tcp_v4_do_rcv(sk, skb);
goto put_and_return;
}
sk_incoming_cpu_update(sk);
bh_lock_sock_nested(sk);
tcp_segs_in(tcp_sk(sk), skb);
ret = 0;
if (!sock_owned_by_user(sk)) {
ret = tcp_v4_do_rcv(sk, skb);
} else {
drop_reason = tcp_add_backlog(sk, skb);
if (drop_reason)
goto discard_and_relse;
}
bh_unlock_sock(sk);
put_and_return:
if (refcounted)
sock_put(sk);
return ret;
no_tcp_socket:
drop_reason = SKB_DROP_REASON_NO_SOCKET;
if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb))
goto discard_it;
tcp_v4_fill_cb(skb, iph, th);
if (tcp_checksum_complete(skb)) {
csum_error:
drop_reason = SKB_DROP_REASON_TCP_CSUM;
trace_tcp_bad_csum(skb);
__TCP_INC_STATS(net, TCP_MIB_CSUMERRORS);
bad_packet:
__TCP_INC_STATS(net, TCP_MIB_INERRS);
} else {
tcp_v4_send_reset(NULL, skb, sk_rst_convert_drop_reason(drop_reason));
}
discard_it:
SKB_DR_OR(drop_reason, NOT_SPECIFIED);
/* Discard frame. */
sk_skb_reason_drop(sk, skb, drop_reason);
return 0;
discard_and_relse:
sk_drops_skbadd(sk, skb);
if (refcounted)
sock_put(sk);
goto discard_it;
do_time_wait:
if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb)) {
drop_reason = SKB_DROP_REASON_XFRM_POLICY;
inet_twsk_put(inet_twsk(sk));
goto discard_it;
}
tcp_v4_fill_cb(skb, iph, th);
if (tcp_checksum_complete(skb)) {
inet_twsk_put(inet_twsk(sk));
goto csum_error;
}
tw_status = tcp_timewait_state_process(inet_twsk(sk), skb, th, &isn,
&drop_reason);
switch (tw_status) {
case TCP_TW_SYN: {
struct sock *sk2 = inet_lookup_listener(net, skb, __tcp_hdrlen(th),
iph->saddr, th->source,
iph->daddr, th->dest,
inet_iif(skb),
sdif);
if (sk2) {
inet_twsk_deschedule_put(inet_twsk(sk));
sk = sk2;
tcp_v4_restore_cb(skb);
refcounted = false;
__this_cpu_write(tcp_tw_isn, isn);
goto process;
}
drop_reason = psp_twsk_rx_policy_check(inet_twsk(sk), skb);
if (drop_reason)
break;
}
/* to ACK */
fallthrough;
case TCP_TW_ACK:
case TCP_TW_ACK_OOW:
tcp_v4_timewait_ack(sk, skb, tw_status);
break;
case TCP_TW_RST:
tcp_v4_send_reset(sk, skb, SK_RST_REASON_TCP_TIMEWAIT_SOCKET);
inet_twsk_deschedule_put(inet_twsk(sk));
goto discard_it;
case TCP_TW_SUCCESS:;
}
goto discard_it;
}
static struct timewait_sock_ops tcp_timewait_sock_ops = {
.twsk_obj_size = sizeof(struct tcp_timewait_sock),
};
void inet_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb)
{
struct dst_entry *dst = skb_dst(skb);
if (dst && dst_hold_safe(dst)) {
rcu_assign_pointer(sk->sk_rx_dst, dst);
sk->sk_rx_dst_ifindex = skb->skb_iif;
}
}
const struct inet_connection_sock_af_ops ipv4_specific = {
.queue_xmit = ip_queue_xmit,
.rebuild_header = inet_sk_rebuild_header,
.sk_rx_dst_set = inet_sk_rx_dst_set,
.conn_request = tcp_v4_conn_request,
.syn_recv_sock = tcp_v4_syn_recv_sock,
.net_header_len = sizeof(struct iphdr),
.setsockopt = ip_setsockopt,
.getsockopt = ip_getsockopt,
.mtu_reduced = tcp_v4_mtu_reduced,
};
#if defined(CONFIG_TCP_MD5SIG) || defined(CONFIG_TCP_AO)
static const struct tcp_sock_af_ops tcp_sock_ipv4_specific = {
#ifdef CONFIG_TCP_MD5SIG
.md5_lookup = tcp_v4_md5_lookup,
.calc_md5_hash = tcp_v4_md5_hash_skb,
.md5_parse = tcp_v4_parse_md5_keys,
#endif
#ifdef CONFIG_TCP_AO
.ao_lookup = tcp_v4_ao_lookup,
.calc_ao_hash = tcp_v4_ao_hash_skb,
.ao_parse = tcp_v4_parse_ao,
.ao_calc_key_sk = tcp_v4_ao_calc_key_sk,
#endif
};
static void tcp4_destruct_sock(struct sock *sk)
{
tcp_md5_destruct_sock(sk);
tcp_ao_destroy_sock(sk, false);
inet_sock_destruct(sk);
}
#endif
/* NOTE: A lot of things set to zero explicitly by call to
* sk_alloc() so need not be done here.
*/
static int tcp_v4_init_sock(struct sock *sk)
{
struct inet_connection_sock *icsk = inet_csk(sk);
tcp_init_sock(sk);
icsk->icsk_af_ops = &ipv4_specific;
#if defined(CONFIG_TCP_MD5SIG) || defined(CONFIG_TCP_AO)
tcp_sk(sk)->af_specific = &tcp_sock_ipv4_specific;
sk->sk_destruct = tcp4_destruct_sock;
#endif
return 0;
}
static void tcp_release_user_frags(struct sock *sk)
{
#ifdef CONFIG_PAGE_POOL
unsigned long index;
void *netmem;
xa_for_each(&sk->sk_user_frags, index, netmem)
WARN_ON_ONCE(!napi_pp_put_page((__force netmem_ref)netmem));
#endif
}
void tcp_v4_destroy_sock(struct sock *sk)
{
struct tcp_sock *tp = tcp_sk(sk);
tcp_release_user_frags(sk);
xa_destroy(&sk->sk_user_frags);
trace_tcp_destroy_sock(sk);
tcp_clear_xmit_timers(sk);
tcp_cleanup_congestion_control(sk);
tcp_cleanup_ulp(sk);
/* Cleanup up the write buffer. */
tcp_write_queue_purge(sk);
/* Check if we want to disable active TFO */
tcp_fastopen_active_disable_ofo_check(sk);
/* Cleans up our, hopefully empty, out_of_order_queue. */
skb_rbtree_purge(&tp->out_of_order_queue);
/* Clean up a referenced TCP bind bucket. */
if (inet_csk(sk)->icsk_bind_hash)
inet_put_port(sk);
BUG_ON(rcu_access_pointer(tp->fastopen_rsk));
/* If socket is aborted during connect operation */
tcp_free_fastopen_req(tp);
tcp_fastopen_destroy_cipher(sk);
tcp_saved_syn_free(tp);
sk_sockets_allocated_dec(sk);
}
#ifdef CONFIG_PROC_FS
/* Proc filesystem TCP sock list dumping. */
static unsigned short seq_file_family(const struct seq_file *seq);
static bool seq_sk_match(struct seq_file *seq, const struct sock *sk)
{
unsigned short family = seq_file_family(seq);
/* AF_UNSPEC is used as a match all */
return ((family == AF_UNSPEC || family == sk->sk_family) &&
net_eq(sock_net(sk), seq_file_net(seq)));
}
/* Find a non empty bucket (starting from st->bucket)
* and return the first sk from it.
*/
static void *listening_get_first(struct seq_file *seq)
{
struct inet_hashinfo *hinfo = seq_file_net(seq)->ipv4.tcp_death_row.hashinfo;
struct tcp_iter_state *st = seq->private;
st->offset = 0;
for (; st->bucket <= hinfo->lhash2_mask; st->bucket++) {
struct inet_listen_hashbucket *ilb2;
struct hlist_nulls_node *node;
struct sock *sk;
ilb2 = &hinfo->lhash2[st->bucket];
if (hlist_nulls_empty(&ilb2->nulls_head))
continue;
spin_lock(&ilb2->lock);
sk_nulls_for_each(sk, node, &ilb2->nulls_head) {
if (seq_sk_match(seq, sk))
return sk;
}
spin_unlock(&ilb2->lock);
}
return NULL;
}
/* Find the next sk of "cur" within the same bucket (i.e. st->bucket).
* If "cur" is the last one in the st->bucket,
* call listening_get_first() to return the first sk of the next
* non empty bucket.
*/
static void *listening_get_next(struct seq_file *seq, void *cur)
{
struct tcp_iter_state *st = seq->private;
struct inet_listen_hashbucket *ilb2;
struct hlist_nulls_node *node;
struct inet_hashinfo *hinfo;
struct sock *sk = cur;
++st->num;
++st->offset;
sk = sk_nulls_next(sk);
sk_nulls_for_each_from(sk, node) {
if (seq_sk_match(seq, sk))
return sk;
}
hinfo = seq_file_net(seq)->ipv4.tcp_death_row.hashinfo;
ilb2 = &hinfo->lhash2[st->bucket];
spin_unlock(&ilb2->lock);
++st->bucket;
return listening_get_first(seq);
}
static void *listening_get_idx(struct seq_file *seq, loff_t *pos)
{
struct tcp_iter_state *st = seq->private;
void *rc;
st->bucket = 0;
st->offset = 0;
rc = listening_get_first(seq);
while (rc && *pos) {
rc = listening_get_next(seq, rc);
--*pos;
}
return rc;
}
static inline bool empty_bucket(struct inet_hashinfo *hinfo,
const struct tcp_iter_state *st)
{
return hlist_nulls_empty(&hinfo->ehash[st->bucket].chain);
}
/*
* Get first established socket starting from bucket given in st->bucket.
* If st->bucket is zero, the very first socket in the hash is returned.
*/
static void *established_get_first(struct seq_file *seq)
{
struct inet_hashinfo *hinfo = seq_file_net(seq)->ipv4.tcp_death_row.hashinfo;
struct tcp_iter_state *st = seq->private;
st->offset = 0;
for (; st->bucket <= hinfo->ehash_mask; ++st->bucket) {
struct sock *sk;
struct hlist_nulls_node *node;
spinlock_t *lock = inet_ehash_lockp(hinfo, st->bucket);
cond_resched();
/* Lockless fast path for the common case of empty buckets */
if (empty_bucket(hinfo, st))
continue;
spin_lock_bh(lock);
sk_nulls_for_each(sk, node, &hinfo->ehash[st->bucket].chain) {
if (seq_sk_match(seq, sk))
return sk;
}
spin_unlock_bh(lock);
}
return NULL;
}
static void *established_get_next(struct seq_file *seq, void *cur)
{
struct inet_hashinfo *hinfo = seq_file_net(seq)->ipv4.tcp_death_row.hashinfo;
struct tcp_iter_state *st = seq->private;
struct hlist_nulls_node *node;
struct sock *sk = cur;
++st->num;
++st->offset;
sk = sk_nulls_next(sk);
sk_nulls_for_each_from(sk, node) {
if (seq_sk_match(seq, sk))
return sk;
}
spin_unlock_bh(inet_ehash_lockp(hinfo, st->bucket));
++st->bucket;
return established_get_first(seq);
}
static void *established_get_idx(struct seq_file *seq, loff_t pos)
{
struct tcp_iter_state *st = seq->private;
void *rc;
st->bucket = 0;
rc = established_get_first(seq);
while (rc && pos) {
rc = established_get_next(seq, rc);
--pos;
}
return rc;
}
static void *tcp_get_idx(struct seq_file *seq, loff_t pos)
{
void *rc;
struct tcp_iter_state *st = seq->private;
st->state = TCP_SEQ_STATE_LISTENING;
rc = listening_get_idx(seq, &pos);
if (!rc) {
st->state = TCP_SEQ_STATE_ESTABLISHED;
rc = established_get_idx(seq, pos);
}
return rc;
}
static void *tcp_seek_last_pos(struct seq_file *seq)
{
struct inet_hashinfo *hinfo = seq_file_net(seq)->ipv4.tcp_death_row.hashinfo;
struct tcp_iter_state *st = seq->private;
int bucket = st->bucket;
int offset = st->offset;
int orig_num = st->num;
void *rc = NULL;
switch (st->state) {
case TCP_SEQ_STATE_LISTENING:
if (st->bucket > hinfo->lhash2_mask)
break;
rc = listening_get_first(seq);
while (offset-- && rc && bucket == st->bucket)
rc = listening_get_next(seq, rc);
if (rc)
break;
st->bucket = 0;
st->state = TCP_SEQ_STATE_ESTABLISHED;
fallthrough;
case TCP_SEQ_STATE_ESTABLISHED:
if (st->bucket > hinfo->ehash_mask)
break;
rc = established_get_first(seq);
while (offset-- && rc && bucket == st->bucket)
rc = established_get_next(seq, rc);
}
st->num = orig_num;
return rc;
}
void *tcp_seq_start(struct seq_file *seq, loff_t *pos)
{
struct tcp_iter_state *st = seq->private;
void *rc;
if (*pos && *pos == st->last_pos) {
rc = tcp_seek_last_pos(seq);
if (rc)
goto out;
}
st->state = TCP_SEQ_STATE_LISTENING;
st->num = 0;
st->bucket = 0;
st->offset = 0;
rc = *pos ? tcp_get_idx(seq, *pos - 1) : SEQ_START_TOKEN;
out:
st->last_pos = *pos;
return rc;
}
void *tcp_seq_next(struct seq_file *seq, void *v, loff_t *pos)
{
struct tcp_iter_state *st = seq->private;
void *rc = NULL;
if (v == SEQ_START_TOKEN) {
rc = tcp_get_idx(seq, 0);
goto out;
}
switch (st->state) {
case TCP_SEQ_STATE_LISTENING:
rc = listening_get_next(seq, v);
if (!rc) {
st->state = TCP_SEQ_STATE_ESTABLISHED;
st->bucket = 0;
st->offset = 0;
rc = established_get_first(seq);
}
break;
case TCP_SEQ_STATE_ESTABLISHED:
rc = established_get_next(seq, v);
break;
}
out:
++*pos;
st->last_pos = *pos;
return rc;
}
void tcp_seq_stop(struct seq_file *seq, void *v)
{
struct inet_hashinfo *hinfo = seq_file_net(seq)->ipv4.tcp_death_row.hashinfo;
struct tcp_iter_state *st = seq->private;
switch (st->state) {
case TCP_SEQ_STATE_LISTENING:
if (v != SEQ_START_TOKEN)
spin_unlock(&hinfo->lhash2[st->bucket].lock);
break;
case TCP_SEQ_STATE_ESTABLISHED:
if (v)
spin_unlock_bh(inet_ehash_lockp(hinfo, st->bucket));
break;
}
}
static void get_openreq4(const struct request_sock *req,
struct seq_file *f, int i)
{
const struct inet_request_sock *ireq = inet_rsk(req);
long delta = req->rsk_timer.expires - jiffies;
seq_printf(f, "%4d: %08X:%04X %08X:%04X"
" %02X %08X:%08X %02X:%08lX %08X %5u %8d %u %d %pK",
i,
ireq->ir_loc_addr,
ireq->ir_num,
ireq->ir_rmt_addr,
ntohs(ireq->ir_rmt_port),
TCP_SYN_RECV,
0, 0, /* could print option size, but that is af dependent. */
1, /* timers active (only the expire timer) */
jiffies_delta_to_clock_t(delta),
req->num_timeout,
from_kuid_munged(seq_user_ns(f),
sk_uid(req->rsk_listener)),
0, /* non standard timer */
0, /* open_requests have no inode */
0,
req);
}
static void get_tcp4_sock(struct sock *sk, struct seq_file *f, int i)
{
int timer_active;
unsigned long timer_expires;
const struct tcp_sock *tp = tcp_sk(sk);
const struct inet_connection_sock *icsk = inet_csk(sk);
const struct inet_sock *inet = inet_sk(sk);
const struct fastopen_queue *fastopenq = &icsk->icsk_accept_queue.fastopenq;
__be32 dest = inet->inet_daddr;
__be32 src = inet->inet_rcv_saddr;
__u16 destp = ntohs(inet->inet_dport);
__u16 srcp = ntohs(inet->inet_sport);
u8 icsk_pending;
int rx_queue;
int state;
icsk_pending = smp_load_acquire(&icsk->icsk_pending);
if (icsk_pending == ICSK_TIME_RETRANS ||
icsk_pending == ICSK_TIME_REO_TIMEOUT ||
icsk_pending == ICSK_TIME_LOSS_PROBE) {
timer_active = 1;
timer_expires = tcp_timeout_expires(sk);
} else if (icsk_pending == ICSK_TIME_PROBE0) {
timer_active = 4;
timer_expires = tcp_timeout_expires(sk);
} else if (timer_pending(&icsk->icsk_keepalive_timer)) {
timer_active = 2;
timer_expires = icsk->icsk_keepalive_timer.expires;
} else {
timer_active = 0;
timer_expires = jiffies;
}
state = inet_sk_state_load(sk);
if (state == TCP_LISTEN)
rx_queue = READ_ONCE(sk->sk_ack_backlog);
else
/* Because we don't lock the socket,
* we might find a transient negative value.
*/
rx_queue = max_t(int, READ_ONCE(tp->rcv_nxt) -
READ_ONCE(tp->copied_seq), 0);
seq_printf(f, "%4d: %08X:%04X %08X:%04X %02X %08X:%08X %02X:%08lX "
"%08X %5u %8d %llu %d %pK %lu %lu %u %u %d",
i, src, srcp, dest, destp, state,
READ_ONCE(tp->write_seq) - tp->snd_una,
rx_queue,
timer_active,
jiffies_delta_to_clock_t(timer_expires - jiffies),
READ_ONCE(icsk->icsk_retransmits),
from_kuid_munged(seq_user_ns(f), sk_uid(sk)),
READ_ONCE(icsk->icsk_probes_out),
sock_i_ino(sk),
refcount_read(&sk->sk_refcnt), sk,
jiffies_to_clock_t(icsk->icsk_rto),
jiffies_to_clock_t(icsk->icsk_ack.ato),
(icsk->icsk_ack.quick << 1) | inet_csk_in_pingpong_mode(sk),
tcp_snd_cwnd(tp),
state == TCP_LISTEN ?
fastopenq->max_qlen :
(tcp_in_initial_slowstart(tp) ? -1 : tp->snd_ssthresh));
}
static void get_timewait4_sock(const struct inet_timewait_sock *tw,
struct seq_file *f, int i)
{
long delta = tw->tw_timer.expires - jiffies;
__be32 dest, src;
__u16 destp, srcp;
dest = tw->tw_daddr;
src = tw->tw_rcv_saddr;
destp = ntohs(tw->tw_dport);
srcp = ntohs(tw->tw_sport);
seq_printf(f, "%4d: %08X:%04X %08X:%04X"
" %02X %08X:%08X %02X:%08lX %08X %5d %8d %d %d %pK",
i, src, srcp, dest, destp, READ_ONCE(tw->tw_substate), 0, 0,
3, jiffies_delta_to_clock_t(delta), 0, 0, 0, 0,
refcount_read(&tw->tw_refcnt), tw);
}
#define TMPSZ 150
static int tcp4_seq_show(struct seq_file *seq, void *v)
{
struct tcp_iter_state *st;
struct sock *sk = v;
seq_setwidth(seq, TMPSZ - 1);
if (v == SEQ_START_TOKEN) {
seq_puts(seq, " sl local_address rem_address st tx_queue "
"rx_queue tr tm->when retrnsmt uid timeout "
"inode");
goto out;
}
st = seq->private;
if (sk->sk_state == TCP_TIME_WAIT)
get_timewait4_sock(v, seq, st->num);
else if (sk->sk_state == TCP_NEW_SYN_RECV)
get_openreq4(v, seq, st->num);
else
get_tcp4_sock(v, seq, st->num);
out:
seq_pad(seq, '\n');
return 0;
}
#ifdef CONFIG_BPF_SYSCALL
union bpf_tcp_iter_batch_item {
struct sock *sk;
__u64 cookie;
};
struct bpf_tcp_iter_state {
struct tcp_iter_state state;
unsigned int cur_sk;
unsigned int end_sk;
unsigned int max_sk;
union bpf_tcp_iter_batch_item *batch;
};
struct bpf_iter__tcp {
__bpf_md_ptr(struct bpf_iter_meta *, meta);
__bpf_md_ptr(struct sock_common *, sk_common);
uid_t uid __aligned(8);
};
static int tcp_prog_seq_show(struct bpf_prog *prog, struct bpf_iter_meta *meta,
struct sock_common *sk_common, uid_t uid)
{
struct bpf_iter__tcp ctx;
meta->seq_num--; /* skip SEQ_START_TOKEN */
ctx.meta = meta;
ctx.sk_common = sk_common;
ctx.uid = uid;
return bpf_iter_run_prog(prog, &ctx);
}
static void bpf_iter_tcp_put_batch(struct bpf_tcp_iter_state *iter)
{
union bpf_tcp_iter_batch_item *item;
unsigned int cur_sk = iter->cur_sk;
__u64 cookie;
/* Remember the cookies of the sockets we haven't seen yet, so we can
* pick up where we left off next time around.
*/
while (cur_sk < iter->end_sk) {
item = &iter->batch[cur_sk++];
cookie = sock_gen_cookie(item->sk);
sock_gen_put(item->sk);
item->cookie = cookie;
}
}
static int bpf_iter_tcp_realloc_batch(struct bpf_tcp_iter_state *iter,
unsigned int new_batch_sz, gfp_t flags)
{
union bpf_tcp_iter_batch_item *new_batch;
new_batch = kvmalloc(sizeof(*new_batch) * new_batch_sz,
flags | __GFP_NOWARN);
if (!new_batch)
return -ENOMEM;
memcpy(new_batch, iter->batch, sizeof(*iter->batch) * iter->end_sk);
kvfree(iter->batch);
iter->batch = new_batch;
iter->max_sk = new_batch_sz;
return 0;
}
static struct sock *bpf_iter_tcp_resume_bucket(struct sock *first_sk,
union bpf_tcp_iter_batch_item *cookies,
int n_cookies)
{
struct hlist_nulls_node *node;
struct sock *sk;
int i;
for (i = 0; i < n_cookies; i++) {
sk = first_sk;
sk_nulls_for_each_from(sk, node)
if (cookies[i].cookie == atomic64_read(&sk->sk_cookie))
return sk;
}
return NULL;
}
static struct sock *bpf_iter_tcp_resume_listening(struct seq_file *seq)
{
struct inet_hashinfo *hinfo = seq_file_net(seq)->ipv4.tcp_death_row.hashinfo;
struct bpf_tcp_iter_state *iter = seq->private;
struct tcp_iter_state *st = &iter->state;
unsigned int find_cookie = iter->cur_sk;
unsigned int end_cookie = iter->end_sk;
int resume_bucket = st->bucket;
struct sock *sk;
if (end_cookie && find_cookie == end_cookie)
++st->bucket;
sk = listening_get_first(seq);
iter->cur_sk = 0;
iter->end_sk = 0;
if (sk && st->bucket == resume_bucket && end_cookie) {
sk = bpf_iter_tcp_resume_bucket(sk, &iter->batch[find_cookie],
end_cookie - find_cookie);
if (!sk) {
spin_unlock(&hinfo->lhash2[st->bucket].lock);
++st->bucket;
sk = listening_get_first(seq);
}
}
return sk;
}
static struct sock *bpf_iter_tcp_resume_established(struct seq_file *seq)
{
struct inet_hashinfo *hinfo = seq_file_net(seq)->ipv4.tcp_death_row.hashinfo;
struct bpf_tcp_iter_state *iter = seq->private;
struct tcp_iter_state *st = &iter->state;
unsigned int find_cookie = iter->cur_sk;
unsigned int end_cookie = iter->end_sk;
int resume_bucket = st->bucket;
struct sock *sk;
if (end_cookie && find_cookie == end_cookie)
++st->bucket;
sk = established_get_first(seq);
iter->cur_sk = 0;
iter->end_sk = 0;
if (sk && st->bucket == resume_bucket && end_cookie) {
sk = bpf_iter_tcp_resume_bucket(sk, &iter->batch[find_cookie],
end_cookie - find_cookie);
if (!sk) {
spin_unlock_bh(inet_ehash_lockp(hinfo, st->bucket));
++st->bucket;
sk = established_get_first(seq);
}
}
return sk;
}
static struct sock *bpf_iter_tcp_resume(struct seq_file *seq)
{
struct bpf_tcp_iter_state *iter = seq->private;
struct tcp_iter_state *st = &iter->state;
struct sock *sk = NULL;
switch (st->state) {
case TCP_SEQ_STATE_LISTENING:
sk = bpf_iter_tcp_resume_listening(seq);
if (sk)
break;
st->bucket = 0;
st->state = TCP_SEQ_STATE_ESTABLISHED;
fallthrough;
case TCP_SEQ_STATE_ESTABLISHED:
sk = bpf_iter_tcp_resume_established(seq);
break;
}
return sk;
}
static unsigned int bpf_iter_tcp_listening_batch(struct seq_file *seq,
struct sock **start_sk)
{
struct bpf_tcp_iter_state *iter = seq->private;
struct hlist_nulls_node *node;
unsigned int expected = 1;
struct sock *sk;
sock_hold(*start_sk);
iter->batch[iter->end_sk++].sk = *start_sk;
sk = sk_nulls_next(*start_sk);
*start_sk = NULL;
sk_nulls_for_each_from(sk, node) {
if (seq_sk_match(seq, sk)) {
if (iter->end_sk < iter->max_sk) {
sock_hold(sk);
iter->batch[iter->end_sk++].sk = sk;
} else if (!*start_sk) {
/* Remember where we left off. */
*start_sk = sk;
}
expected++;
}
}
return expected;
}
static unsigned int bpf_iter_tcp_established_batch(struct seq_file *seq,
struct sock **start_sk)
{
struct bpf_tcp_iter_state *iter = seq->private;
struct hlist_nulls_node *node;
unsigned int expected = 1;
struct sock *sk;
sock_hold(*start_sk);
iter->batch[iter->end_sk++].sk = *start_sk;
sk = sk_nulls_next(*start_sk);
*start_sk = NULL;
sk_nulls_for_each_from(sk, node) {
if (seq_sk_match(seq, sk)) {
if (iter->end_sk < iter->max_sk) {
sock_hold(sk);
iter->batch[iter->end_sk++].sk = sk;
} else if (!*start_sk) {
/* Remember where we left off. */
*start_sk = sk;
}
expected++;
}
}
return expected;
}
static unsigned int bpf_iter_fill_batch(struct seq_file *seq,
struct sock **start_sk)
{
struct bpf_tcp_iter_state *iter = seq->private;
struct tcp_iter_state *st = &iter->state;
if (st->state == TCP_SEQ_STATE_LISTENING)
return bpf_iter_tcp_listening_batch(seq, start_sk);
else
return bpf_iter_tcp_established_batch(seq, start_sk);
}
static void bpf_iter_tcp_unlock_bucket(struct seq_file *seq)
{
struct inet_hashinfo *hinfo = seq_file_net(seq)->ipv4.tcp_death_row.hashinfo;
struct bpf_tcp_iter_state *iter = seq->private;
struct tcp_iter_state *st = &iter->state;
if (st->state == TCP_SEQ_STATE_LISTENING)
spin_unlock(&hinfo->lhash2[st->bucket].lock);
else
spin_unlock_bh(inet_ehash_lockp(hinfo, st->bucket));
}
static struct sock *bpf_iter_tcp_batch(struct seq_file *seq)
{
struct bpf_tcp_iter_state *iter = seq->private;
unsigned int expected;
struct sock *sk;
int err;
sk = bpf_iter_tcp_resume(seq);
if (!sk)
return NULL; /* Done */
expected = bpf_iter_fill_batch(seq, &sk);
if (likely(iter->end_sk == expected))
goto done;
/* Batch size was too small. */
bpf_iter_tcp_unlock_bucket(seq);
bpf_iter_tcp_put_batch(iter);
err = bpf_iter_tcp_realloc_batch(iter, expected * 3 / 2,
GFP_USER);
if (err)
return ERR_PTR(err);
sk = bpf_iter_tcp_resume(seq);
if (!sk)
return NULL; /* Done */
expected = bpf_iter_fill_batch(seq, &sk);
if (likely(iter->end_sk == expected))
goto done;
/* Batch size was still too small. Hold onto the lock while we try
* again with a larger batch to make sure the current bucket's size
* does not change in the meantime.
*/
err = bpf_iter_tcp_realloc_batch(iter, expected, GFP_NOWAIT);
if (err) {
bpf_iter_tcp_unlock_bucket(seq);
return ERR_PTR(err);
}
expected = bpf_iter_fill_batch(seq, &sk);
WARN_ON_ONCE(iter->end_sk != expected);
done:
bpf_iter_tcp_unlock_bucket(seq);
return iter->batch[0].sk;
}
static void *bpf_iter_tcp_seq_start(struct seq_file *seq, loff_t *pos)
{
/* bpf iter does not support lseek, so it always
* continue from where it was stop()-ped.
*/
if (*pos)
return bpf_iter_tcp_batch(seq);
return SEQ_START_TOKEN;
}
static void *bpf_iter_tcp_seq_next(struct seq_file *seq, void *v, loff_t *pos)
{
struct bpf_tcp_iter_state *iter = seq->private;
struct tcp_iter_state *st = &iter->state;
struct sock *sk;
/* Whenever seq_next() is called, the iter->cur_sk is
* done with seq_show(), so advance to the next sk in
* the batch.
*/
if (iter->cur_sk < iter->end_sk) {
/* Keeping st->num consistent in tcp_iter_state.
* bpf_iter_tcp does not use st->num.
* meta.seq_num is used instead.
*/
st->num++;
sock_gen_put(iter->batch[iter->cur_sk++].sk);
}
if (iter->cur_sk < iter->end_sk)
sk = iter->batch[iter->cur_sk].sk;
else
sk = bpf_iter_tcp_batch(seq);
++*pos;
/* Keeping st->last_pos consistent in tcp_iter_state.
* bpf iter does not do lseek, so st->last_pos always equals to *pos.
*/
st->last_pos = *pos;
return sk;
}
static int bpf_iter_tcp_seq_show(struct seq_file *seq, void *v)
{
struct bpf_iter_meta meta;
struct bpf_prog *prog;
struct sock *sk = v;
uid_t uid;
int ret;
if (v == SEQ_START_TOKEN)
return 0;
if (sk_fullsock(sk))
lock_sock(sk);
if (unlikely(sk_unhashed(sk))) {
ret = SEQ_SKIP;
goto unlock;
}
if (sk->sk_state == TCP_TIME_WAIT) {
uid = 0;
} else if (sk->sk_state == TCP_NEW_SYN_RECV) {
const struct request_sock *req = v;
uid = from_kuid_munged(seq_user_ns(seq),
sk_uid(req->rsk_listener));
} else {
uid = from_kuid_munged(seq_user_ns(seq), sk_uid(sk));
}
meta.seq = seq;
prog = bpf_iter_get_info(&meta, false);
ret = tcp_prog_seq_show(prog, &meta, v, uid);
unlock:
if (sk_fullsock(sk))
release_sock(sk);
return ret;
}
static void bpf_iter_tcp_seq_stop(struct seq_file *seq, void *v)
{
struct bpf_tcp_iter_state *iter = seq->private;
struct bpf_iter_meta meta;
struct bpf_prog *prog;
if (!v) {
meta.seq = seq;
prog = bpf_iter_get_info(&meta, true);
if (prog)
(void)tcp_prog_seq_show(prog, &meta, v, 0);
}
if (iter->cur_sk < iter->end_sk)
bpf_iter_tcp_put_batch(iter);
}
static const struct seq_operations bpf_iter_tcp_seq_ops = {
.show = bpf_iter_tcp_seq_show,
.start = bpf_iter_tcp_seq_start,
.next = bpf_iter_tcp_seq_next,
.stop = bpf_iter_tcp_seq_stop,
};
#endif
static unsigned short seq_file_family(const struct seq_file *seq)
{
const struct tcp_seq_afinfo *afinfo;
#ifdef CONFIG_BPF_SYSCALL
/* Iterated from bpf_iter. Let the bpf prog to filter instead. */
if (seq->op == &bpf_iter_tcp_seq_ops)
return AF_UNSPEC;
#endif
/* Iterated from proc fs */
afinfo = pde_data(file_inode(seq->file));
return afinfo->family;
}
static const struct seq_operations tcp4_seq_ops = {
.show = tcp4_seq_show,
.start = tcp_seq_start,
.next = tcp_seq_next,
.stop = tcp_seq_stop,
};
static struct tcp_seq_afinfo tcp4_seq_afinfo = {
.family = AF_INET,
};
static int __net_init tcp4_proc_init_net(struct net *net)
{
if (!proc_create_net_data("tcp", 0444, net->proc_net, &tcp4_seq_ops,
sizeof(struct tcp_iter_state), &tcp4_seq_afinfo))
return -ENOMEM;
return 0;
}
static void __net_exit tcp4_proc_exit_net(struct net *net)
{
remove_proc_entry("tcp", net->proc_net);
}
static struct pernet_operations tcp4_net_ops = {
.init = tcp4_proc_init_net,
.exit = tcp4_proc_exit_net,
};
int __init tcp4_proc_init(void)
{
return register_pernet_subsys(&tcp4_net_ops);
}
void tcp4_proc_exit(void)
{
unregister_pernet_subsys(&tcp4_net_ops);
}
#endif /* CONFIG_PROC_FS */
struct proto tcp_prot = {
.name = "TCP",
.owner = THIS_MODULE,
.close = tcp_close,
.pre_connect = tcp_v4_pre_connect,
.connect = tcp_v4_connect,
.disconnect = tcp_disconnect,
.accept = inet_csk_accept,
.ioctl = tcp_ioctl,
.init = tcp_v4_init_sock,
.destroy = tcp_v4_destroy_sock,
.shutdown = tcp_shutdown,
.setsockopt = tcp_setsockopt,
.getsockopt = tcp_getsockopt,
.bpf_bypass_getsockopt = tcp_bpf_bypass_getsockopt,
.keepalive = tcp_set_keepalive,
.recvmsg = tcp_recvmsg,
.sendmsg = tcp_sendmsg,
.splice_eof = tcp_splice_eof,
.backlog_rcv = tcp_v4_do_rcv,
.release_cb = tcp_release_cb,
.hash = inet_hash,
.unhash = inet_unhash,
.get_port = inet_csk_get_port,
.put_port = inet_put_port,
#ifdef CONFIG_BPF_SYSCALL
.psock_update_sk_prot = tcp_bpf_update_proto,
#endif
.enter_memory_pressure = tcp_enter_memory_pressure,
.leave_memory_pressure = tcp_leave_memory_pressure,
.stream_memory_free = tcp_stream_memory_free,
.sockets_allocated = &tcp_sockets_allocated,
.memory_allocated = &net_aligned_data.tcp_memory_allocated,
.per_cpu_fw_alloc = &tcp_memory_per_cpu_fw_alloc,
.memory_pressure = &tcp_memory_pressure,
.sysctl_mem = sysctl_tcp_mem,
.sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_tcp_wmem),
.sysctl_rmem_offset = offsetof(struct net, ipv4.sysctl_tcp_rmem),
.max_header = MAX_TCP_HEADER,
.obj_size = sizeof(struct tcp_sock),
.freeptr_offset = offsetof(struct tcp_sock,
inet_conn.icsk_inet.sk.sk_freeptr),
.slab_flags = SLAB_TYPESAFE_BY_RCU,
.twsk_prot = &tcp_timewait_sock_ops,
.rsk_prot = &tcp_request_sock_ops,
.h.hashinfo = NULL,
.no_autobind = true,
.diag_destroy = tcp_abort,
};
EXPORT_SYMBOL(tcp_prot);
static void __net_exit tcp_sk_exit(struct net *net)
{
if (net->ipv4.tcp_congestion_control)
bpf_module_put(net->ipv4.tcp_congestion_control,
net->ipv4.tcp_congestion_control->owner);
}
static void __net_init tcp_set_hashinfo(struct net *net)
{
struct inet_hashinfo *hinfo;
unsigned int ehash_entries;
struct net *old_net;
if (net_eq(net, &init_net))
goto fallback;
old_net = current->nsproxy->net_ns;
ehash_entries = READ_ONCE(old_net->ipv4.sysctl_tcp_child_ehash_entries);
if (!ehash_entries)
goto fallback;
ehash_entries = roundup_pow_of_two(ehash_entries);
hinfo = inet_pernet_hashinfo_alloc(&tcp_hashinfo, ehash_entries);
if (!hinfo) {
pr_warn("Failed to allocate TCP ehash (entries: %u) "
"for a netns, fallback to the global one\n",
ehash_entries);
fallback:
hinfo = &tcp_hashinfo;
ehash_entries = tcp_hashinfo.ehash_mask + 1;
}
net->ipv4.tcp_death_row.hashinfo = hinfo;
net->ipv4.tcp_death_row.sysctl_max_tw_buckets = ehash_entries / 2;
net->ipv4.sysctl_max_syn_backlog = max(128U, ehash_entries / 128);
}
static int __net_init tcp_sk_init(struct net *net)
{
net->ipv4.sysctl_tcp_ecn = TCP_ECN_IN_ECN_OUT_NOECN;
net->ipv4.sysctl_tcp_ecn_option = TCP_ACCECN_OPTION_FULL;
net->ipv4.sysctl_tcp_ecn_option_beacon = TCP_ACCECN_OPTION_BEACON;
net->ipv4.sysctl_tcp_ecn_fallback = 1;
net->ipv4.sysctl_tcp_base_mss = TCP_BASE_MSS;
net->ipv4.sysctl_tcp_min_snd_mss = TCP_MIN_SND_MSS;
net->ipv4.sysctl_tcp_probe_threshold = TCP_PROBE_THRESHOLD;
net->ipv4.sysctl_tcp_probe_interval = TCP_PROBE_INTERVAL;
net->ipv4.sysctl_tcp_mtu_probe_floor = TCP_MIN_SND_MSS;
net->ipv4.sysctl_tcp_keepalive_time = TCP_KEEPALIVE_TIME;
net->ipv4.sysctl_tcp_keepalive_probes = TCP_KEEPALIVE_PROBES;
net->ipv4.sysctl_tcp_keepalive_intvl = TCP_KEEPALIVE_INTVL;
net->ipv4.sysctl_tcp_syn_retries = TCP_SYN_RETRIES;
net->ipv4.sysctl_tcp_synack_retries = TCP_SYNACK_RETRIES;
net->ipv4.sysctl_tcp_syncookies = 1;
net->ipv4.sysctl_tcp_reordering = TCP_FASTRETRANS_THRESH;
net->ipv4.sysctl_tcp_retries1 = TCP_RETR1;
net->ipv4.sysctl_tcp_retries2 = TCP_RETR2;
net->ipv4.sysctl_tcp_orphan_retries = 0;
net->ipv4.sysctl_tcp_fin_timeout = TCP_FIN_TIMEOUT;
net->ipv4.sysctl_tcp_notsent_lowat = UINT_MAX;
net->ipv4.sysctl_tcp_tw_reuse = 2;
net->ipv4.sysctl_tcp_tw_reuse_delay = 1 * MSEC_PER_SEC;
net->ipv4.sysctl_tcp_no_ssthresh_metrics_save = 1;
refcount_set(&net->ipv4.tcp_death_row.tw_refcount, 1);
tcp_set_hashinfo(net);
net->ipv4.sysctl_tcp_sack = 1;
net->ipv4.sysctl_tcp_window_scaling = 1;
net->ipv4.sysctl_tcp_timestamps = 1;
net->ipv4.sysctl_tcp_early_retrans = 3;
net->ipv4.sysctl_tcp_recovery = TCP_RACK_LOSS_DETECTION;
net->ipv4.sysctl_tcp_slow_start_after_idle = 1; /* By default, RFC2861 behavior. */
net->ipv4.sysctl_tcp_retrans_collapse = 1;
net->ipv4.sysctl_tcp_max_reordering = 300;
net->ipv4.sysctl_tcp_dsack = 1;
net->ipv4.sysctl_tcp_app_win = 31;
net->ipv4.sysctl_tcp_adv_win_scale = 1;
net->ipv4.sysctl_tcp_frto = 2;
net->ipv4.sysctl_tcp_moderate_rcvbuf = 1;
net->ipv4.sysctl_tcp_rcvbuf_low_rtt = USEC_PER_MSEC;
/* This limits the percentage of the congestion window which we
* will allow a single TSO frame to consume. Building TSO frames
* which are too large can cause TCP streams to be bursty.
*/
net->ipv4.sysctl_tcp_tso_win_divisor = 3;
/* Default TSQ limit of 4 MB */
net->ipv4.sysctl_tcp_limit_output_bytes = 4 << 20;
/* rfc5961 challenge ack rate limiting, per net-ns, disabled by default. */
net->ipv4.sysctl_tcp_challenge_ack_limit = INT_MAX;
net->ipv4.sysctl_tcp_min_tso_segs = 2;
net->ipv4.sysctl_tcp_tso_rtt_log = 9; /* 2^9 = 512 usec */
net->ipv4.sysctl_tcp_min_rtt_wlen = 300;
net->ipv4.sysctl_tcp_autocorking = 1;
net->ipv4.sysctl_tcp_invalid_ratelimit = HZ/2;
net->ipv4.sysctl_tcp_pacing_ss_ratio = 200;
net->ipv4.sysctl_tcp_pacing_ca_ratio = 120;
if (net != &init_net) {
memcpy(net->ipv4.sysctl_tcp_rmem,
init_net.ipv4.sysctl_tcp_rmem,
sizeof(init_net.ipv4.sysctl_tcp_rmem));
memcpy(net->ipv4.sysctl_tcp_wmem,
init_net.ipv4.sysctl_tcp_wmem,
sizeof(init_net.ipv4.sysctl_tcp_wmem));
}
net->ipv4.sysctl_tcp_comp_sack_delay_ns = NSEC_PER_MSEC;
net->ipv4.sysctl_tcp_comp_sack_slack_ns = 10 * NSEC_PER_USEC;
net->ipv4.sysctl_tcp_comp_sack_nr = 44;
net->ipv4.sysctl_tcp_comp_sack_rtt_percent = 33;
net->ipv4.sysctl_tcp_backlog_ack_defer = 1;
net->ipv4.sysctl_tcp_fastopen = TFO_CLIENT_ENABLE;
net->ipv4.sysctl_tcp_fastopen_blackhole_timeout = 0;
atomic_set(&net->ipv4.tfo_active_disable_times, 0);
/* Set default values for PLB */
net->ipv4.sysctl_tcp_plb_enabled = 0; /* Disabled by default */
net->ipv4.sysctl_tcp_plb_idle_rehash_rounds = 3;
net->ipv4.sysctl_tcp_plb_rehash_rounds = 12;
net->ipv4.sysctl_tcp_plb_suspend_rto_sec = 60;
/* Default congestion threshold for PLB to mark a round is 50% */
net->ipv4.sysctl_tcp_plb_cong_thresh = (1 << TCP_PLB_SCALE) / 2;
/* Reno is always built in */
if (!net_eq(net, &init_net) &&
bpf_try_module_get(init_net.ipv4.tcp_congestion_control,
init_net.ipv4.tcp_congestion_control->owner))
net->ipv4.tcp_congestion_control = init_net.ipv4.tcp_congestion_control;
else
net->ipv4.tcp_congestion_control = &tcp_reno;
net->ipv4.sysctl_tcp_syn_linear_timeouts = 4;
net->ipv4.sysctl_tcp_shrink_window = 0;
net->ipv4.sysctl_tcp_pingpong_thresh = 1;
net->ipv4.sysctl_tcp_rto_min_us = jiffies_to_usecs(TCP_RTO_MIN);
net->ipv4.sysctl_tcp_rto_max_ms = TCP_RTO_MAX_SEC * MSEC_PER_SEC;
return 0;
}
static void __net_exit tcp_sk_exit_batch(struct list_head *net_exit_list)
{
struct net *net;
/* make sure concurrent calls to tcp_sk_exit_batch from net_cleanup_work
* and failed setup_net error unwinding path are serialized.
*
* tcp_twsk_purge() handles twsk in any dead netns, not just those in
* net_exit_list, the thread that dismantles a particular twsk must
* do so without other thread progressing to refcount_dec_and_test() of
* tcp_death_row.tw_refcount.
*/
mutex_lock(&tcp_exit_batch_mutex);
tcp_twsk_purge(net_exit_list);
list_for_each_entry(net, net_exit_list, exit_list) {
inet_pernet_hashinfo_free(net->ipv4.tcp_death_row.hashinfo);
WARN_ON_ONCE(!refcount_dec_and_test(&net->ipv4.tcp_death_row.tw_refcount));
tcp_fastopen_ctx_destroy(net);
}
mutex_unlock(&tcp_exit_batch_mutex);
}
static struct pernet_operations __net_initdata tcp_sk_ops = {
.init = tcp_sk_init,
.exit = tcp_sk_exit,
.exit_batch = tcp_sk_exit_batch,
};
#if defined(CONFIG_BPF_SYSCALL) && defined(CONFIG_PROC_FS)
DEFINE_BPF_ITER_FUNC(tcp, struct bpf_iter_meta *meta,
struct sock_common *sk_common, uid_t uid)
#define INIT_BATCH_SZ 16
static int bpf_iter_init_tcp(void *priv_data, struct bpf_iter_aux_info *aux)
{
struct bpf_tcp_iter_state *iter = priv_data;
int err;
err = bpf_iter_init_seq_net(priv_data, aux);
if (err)
return err;
err = bpf_iter_tcp_realloc_batch(iter, INIT_BATCH_SZ, GFP_USER);
if (err) {
bpf_iter_fini_seq_net(priv_data);
return err;
}
return 0;
}
static void bpf_iter_fini_tcp(void *priv_data)
{
struct bpf_tcp_iter_state *iter = priv_data;
bpf_iter_fini_seq_net(priv_data);
kvfree(iter->batch);
}
static const struct bpf_iter_seq_info tcp_seq_info = {
.seq_ops = &bpf_iter_tcp_seq_ops,
.init_seq_private = bpf_iter_init_tcp,
.fini_seq_private = bpf_iter_fini_tcp,
.seq_priv_size = sizeof(struct bpf_tcp_iter_state),
};
static const struct bpf_func_proto *
bpf_iter_tcp_get_func_proto(enum bpf_func_id func_id,
const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_setsockopt:
return &bpf_sk_setsockopt_proto;
case BPF_FUNC_getsockopt:
return &bpf_sk_getsockopt_proto;
default:
return NULL;
}
}
static struct bpf_iter_reg tcp_reg_info = {
.target = "tcp",
.ctx_arg_info_size = 1,
.ctx_arg_info = {
{ offsetof(struct bpf_iter__tcp, sk_common),
PTR_TO_BTF_ID_OR_NULL | PTR_TRUSTED },
},
.get_func_proto = bpf_iter_tcp_get_func_proto,
.seq_info = &tcp_seq_info,
};
static void __init bpf_iter_register(void)
{
tcp_reg_info.ctx_arg_info[0].btf_id = btf_sock_ids[BTF_SOCK_TYPE_SOCK_COMMON];
if (bpf_iter_reg_target(&tcp_reg_info))
pr_warn("Warning: could not register bpf iterator tcp\n");
}
#endif
void __init tcp_v4_init(void)
{
int cpu, res;
for_each_possible_cpu(cpu) {
struct sock *sk;
res = inet_ctl_sock_create(&sk, PF_INET, SOCK_RAW,
IPPROTO_TCP, &init_net);
if (res)
panic("Failed to create the TCP control socket.\n");
sock_set_flag(sk, SOCK_USE_WRITE_QUEUE);
/* Please enforce IP_DF and IPID==0 for RST and
* ACK sent in SYN-RECV and TIME-WAIT state.
*/
inet_sk(sk)->pmtudisc = IP_PMTUDISC_DO;
sk->sk_clockid = CLOCK_MONOTONIC;
per_cpu(ipv4_tcp_sk.sock, cpu) = sk;
}
if (register_pernet_subsys(&tcp_sk_ops))
panic("Failed to create the TCP control socket.\n");
#if defined(CONFIG_BPF_SYSCALL) && defined(CONFIG_PROC_FS)
bpf_iter_register();
#endif
}
]
|
| 30/2 |
2026/05/17 01:38 |
llm |
expert |
2m
Model:
Tokens:
input: 109783
output: 66
thoughts: 8407
Thoughts: **Analysis of a TCP Request Socket Migration Vulnerability**
Okay, let's break this down. We have a potentially serious problem in the kernel's TCP implementation, specifically within the request socket migration logic associated with `SO_REUSEPORT`. Here's what's happening, or rather, *could* be happening, in a situation that is concerning. We're dealing with the interaction between timers, reference counting, and the migration of TCP request sockets (`request_sock`).
Essentially, if a TCP connection is established, and then the listener socket is closed, this initiates a migration of the "request sockets" to the remaining listening sockets in the reuseport group. The function `reqsk_timer_handler` handles the request socket timeouts.
Now, we're particularly focused on the code path where a "request socket" (`nreq`) is cloned from an existing socket (`oreq`) during a `SO_REUSEPORT` migration, and subsequently, `inet_ehash_insert` fails (because of a race or existing state). If this insertion into the hash table fails, we see `__inet_csk_reqsk_queue_drop` called, which is intended to clean up the request socket, but the cleanup process goes wrong. Critically, and this is where the vulnerability lies: if `__inet_csk_reqsk_queue_drop` is called due to a failure in `inet_ehash_insert` with `from_timer=true`, the code *doesn't* delete the timer associated with `nreq`! The logic assumes that it's running from a timer, and so avoids a double-free on timer delete. But the *nreq* timer has *not* fired yet. It was just set, not running! So, if the *nreq* timer is not deleted, and `nreq` is freed by `__reqsk_free`, we've got a classic UAF waiting to happen.
The problem hinges on the `from_timer` flag. It's passed as true, incorrectly assuming the current timer handler is `nreq` when it's `oreq`. Because of this, the `timer_delete_sync` is bypassed and the timer for `nreq` is left pending. The code then proceeds to free the memory associated with `nreq` using `__reqsk_free`, which causes the request socket to be freed while the timer remains pending.
Later, that same timer fires. At that point, the timer handler, `reqsk_timer_handler`, tries to access the memory of the freed `nreq`. This is a textbook use-after-free, and the stack trace matches perfectly the bug report! The attacker potentially controls what's in that memory, leading to RCE.
This vulnerability is easily exploitable, and it can be triggered by unprivileged users. It also requires `SO_REUSEPORT`, and can happen when the server process is closed or reloaded (which is the case if you close the listener socket after a SYN-ACK, creating `oreq`, then fire another SYN-ACK with the same tuple, and then an ACK which closes the migration, and inserts the full socket, removing `oreq` from the table before the UAF can trigger). Given `SO_REUSEPORT` and a remote attacker, this creates a Remote Code Execution condition. It's a network-based vulnerability, easily remotely triggered, and with a high potential impact. The exploit setup does require a race condition, which makes it less reliable but still a considerable risk.
|
| 31/2 |
2026/05/17 01:40 |
tool |
set-results |
0m
Args: map[DenialOfService:true Exploitable:true FilesystemTrigger:false NetworkTrigger:true PeripheralTrigger:false RemoteTrigger:true Unprivileged:true UserNamespace:true VMGuestTrigger:false VMHostTrigger:false]
Results: map[DenialOfService:true Exploitable:true FilesystemTrigger:false NetworkTrigger:true PeripheralTrigger:false RemoteTrigger:true Unprivileged:true UserNamespace:true VMGuestTrigger:false VMHostTrigger:false]
|
| 32/2 |
2026/05/17 01:40 |
llm |
expert |
0m
Model:
Tokens:
input: 119080
output: 756
thoughts: 652
Thoughts: **My Analysis of the TCP Request Socket Use-After-Free Vulnerability**
Okay, here's my breakdown of this bug, as I see it. The core issue lies within the `net/ipv4/inet_connection_sock.c` file, specifically the `reqsk_timer_handler` function. This handler deals with retransmissions of SYN-ACKs related to TCP request sockets, a process I'm very familiar with. The scenario is this: a listener socket is closed, triggering a migration of pending requests to other sockets in the `SO_REUSEPORT` group.
The problem arises during this migration. They clone the original request socket (`oreq`) into a new one (`nreq`). A timer is scheduled for `nreq`, which is expected. Then, they attempt to insert `nreq` into the ehash table using `inet_ehash_insert()`. Now, here's the tricky part. If that insertion fails—say, another CPU gets there first and removes `oreq` due to receiving the final ACK— the code correctly handles the failure, by dropping `nreq`.
However, the problem is in the way they do the "drop." They call `__inet_csk_reqsk_queue_drop(sk_listener, nreq, true)`. The `true` parameter is crucial, it indicates that they *don't* need to delete the timer synchronously because they believe the running timer is for this dropped socket. The critical flaw is that the timer *still* running belongs to the *original* socket `oreq`, not `nreq`! That is the first timer, which the function ignores and does not cancel.
So, the code then *immediately* frees the memory allocated to `nreq` using `__reqsk_free(nreq)`. When this old timer eventually fires, it calls `reqsk_timer_handler` on the *freed* `request_sock` object, which is now corrupted, leading to a classic Use-After-Free (UAF) situation. The warning "refcount_t: underflow; use-after-free" confirms my initial suspicion. `reqsk_put()` is called on an already freed object.
Now, from a security perspective, this is serious. The `request_sock` structure contains function pointers, like `rsk_ops`, and other memory pointers. An attacker could potentially reallocate that freed memory and inject controlled data, opening the door to control flow hijacking or arbitrary memory read/write operations. In short, it's exploitable.
Moreover, it's not just a security risk, it's a denial-of-service vulnerability. The memory corruption caused by the UAF will trigger kernel warnings or, worse, cause the system to panic. The best part is this is unprivileged and can be triggered in a user namespace. The attack vector is a remote network attack! Any user can create TCP sockets, use `SO_REUSEPORT`, and by closing a socket can trigger the migration. An attacker can remotely trigger this against servers that use `SO_REUSEPORT`, like Nginx or HAProxy, by precisely timing the TCP handshake and ACK packets to create the race condition.
|
| 33/1 |
2026/05/17 01:41 |
action |
format-explanation |
0m
Results: map[Explanation:Based on the analysis of the bug report and the kernel source code, here is the
assessment of the vulnerability:
### Bug Explanation
The bug is a **Use-After-Free (UAF)** in the TCP request socket timer handling,
specifically within `reqsk_timer_handler()` in
`net/ipv4/inet_connection_sock.c`.
When a TCP listener socket is closed but has pending incomplete connections
(request sockets), the kernel attempts to migrate these request sockets to
another listener socket in the same `SO_REUSEPORT` group.
During this migration:
1. The original request socket (`oreq`) is cloned into a new request socket
(`nreq`).
2. A timer is scheduled for the new request socket via
`mod_timer(&req->rsk_timer, ...)` (where `req` points to `nreq`).
3. The code attempts to replace `oreq` with `nreq` in the established hash table
by calling `inet_ehash_insert()`.
If `inet_ehash_insert()` fails—which can happen if another CPU concurrently
receives the final ACK from the client and removes `oreq` from the hash
table—the code enters an error path to clean up `nreq`.
It calls `__inet_csk_reqsk_queue_drop(sk_listener, nreq, true)`. The `true`
argument is passed for the `from_timer` parameter, which instructs the function
**not** to call `timer_delete_sync()`. The developer mistakenly assumed that the
timer being dropped was the currently running timer (which is
`oreq->rsk_timer`), but the timer that needs to be deleted is the newly
scheduled `nreq->rsk_timer`.
Because `timer_delete_sync()` is skipped, `nreq->rsk_timer` is left pending. The
error path then immediately frees `nreq` by calling `__reqsk_free(nreq)`. When
the pending timer eventually fires, it executes `reqsk_timer_handler()` on the
freed `nreq` object. The warning in the bug report (`refcount_t: underflow;
use-after-free.`) is triggered when the timer handler finishes and calls
`reqsk_put()` on the already freed memory.
### Security Impact
* **Exploitable:** **Yes.** The UAF involves `struct request_sock`, which
contains function pointers (e.g., `rsk_ops`) and pointers to other kernel
structures. An attacker who successfully reallocates the freed memory with
controlled data can hijack the kernel control flow to achieve privilege
escalation or arbitrary code execution.
* **Denial of Service:** **Yes.** The Use-After-Free leads to memory corruption,
kernel warnings, and ultimately kernel panics.
* **Accessible From Unprivileged Processes / User Namespaces:** **Yes.** An
unprivileged local user can easily trigger this by creating TCP sockets with
`SO_REUSEPORT`, initiating connections to them, and closing the listeners to
force the migration race condition.
* **Ethernet Network / Remote Trigger:** **Yes.** This vulnerability can be
triggered **remotely** against a server running a service that utilizes
`SO_REUSEPORT` (such as Nginx or HAProxy). If the service reloads its
configuration (closing old sockets and opening new ones), a remote attacker can
send a SYN packet to create a request socket, and then precisely time the final
ACK packet to win the race against the timer handler's `inet_ehash_insert()`,
triggering the UAF.]
|