syzbot

map[Benign:true Confident:true Explanation:The KCSAN report identifies a data race on the global variable `jiffies_64` between its update in the timer interrupt (`tick_do_update_jiffies64`) and its read in the memory cgroup statistics flushing logic (`mem_cgroup_flush_stats_ratelimited`).

### Analysis

1.  **Nature of the variable**: `jiffies_64` is a 64-bit global counter that tracks the number of system ticks since boot. It is updated by the timer interrupt and read by various parts of the kernel to measure time intervals.
2.  **Access in `tick_do_update_jiffies64`**:
    ```c
    jiffies_64 += ticks;
    ```
    This is a read-modify-write operation performed under `jiffies_lock` and protected by a sequence count (`jiffies_seq`).
3.  **Access in `mem_cgroup_flush_stats_ratelimited`**:
    ```c
    if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
        mem_cgroup_flush_stats(memcg);
    ```
    This is a plain read of `jiffies_64` used to determine if enough time has passed since the last statistics flush to justify another one.
4.  **Atomicity and Tearing**:
    *   On **64-bit systems** (like the x86_64 system in the report), 8-byte aligned loads and stores are atomic at the hardware level. Since `jiffies_64` is declared with `__cacheline_aligned_in_smp`, it is properly aligned. Thus, the read in `mem_cgroup_flush_stats_ratelimited` will always see a consistent 64-bit value.
    *   On **32-bit systems**, a 64-bit read can be torn (split into two 32-bit reads). If a torn read occurs, the resulting value could be significantly incorrect (e.g., jumping far into the future or past). However, the kernel provides `get_jiffies_64()` to avoid this by using the sequence lock.
5.  **Impact of the race**:
    *   In the context of `mem_cgroup_flush_stats_ratelimited`, the value of `jiffies_64` is only used for a ratelimiting check. If a slightly stale or (on 32-bit) torn value is read, the worst consequence is that a statistics flush is either performed slightly early or delayed by one cycle.
    *   Memory cgroup statistics are already allowed to be slightly out of sync (as noted in the comments in `mm/memcontrol.c`), and a minor timing error in flushing them does not lead to memory corruption, crashes, or any functional failure.
6.  **Benign Classification**: The race is on a statistics/timing counter where absolute precision is not required for the correctness of the calling logic. It does not involve complex data structures or pointers that could cause a crash if corrupted.

### Conclusion

The data race is **benign**. While it is technically a violation of the memory model (as it lacks `READ_ONCE` or the use of `get_jiffies_64()`), it does not pose a risk to system stability or correctness in this specific usage. Using `get_jiffies_64()` would be the proper way to suppress the KCSAN warning and ensure correctness on 32-bit platforms, but the current race is harmless.]

map[KernelSrc:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/src/83e3e4073a4cd51a6ced0f93cc08b3f6ffb54f19]

map[KernelObj:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/build/b54bee6631687294095c3f2df1d84662f988232e]

map[Index:codesearch-index]

gemini-3-flash-preview

map[Benign:true Confident:true]

You are an experienced Linux kernel developer tasked with determining if the given kernel
data race is benign or not. The data race report is from KCSAN tool.
It contains 2 stack traces of the memory accesses that constitute a data race.

A "benign" data races are on a simple int/bool variable or similar field,
and the accesses are not supposed to be protected by any mutual exclusion primitives.
Common examples of such "benign" data races are accesses to various flags fields,
statistics counters, and similar. A "benign" data race does not lead to memory corruption/crash
with a conservative compiler that compiles memory accesses to primitive types
effectively as atomic.

A non-benign (or "harmful" data race) can lead to corruption/crash even with
a conservative compiler that compiles memory accesses to primitive types
effectively as atomic. A common example of a "harmful" data races is race on
a complex container (list/hashmap/etc), where accesses are supposed to be protected
by a mutual exclusion primitive.

Also consider races that happen at the same time with the given one.
If there is no synchronization in between, other memory accesses in the involved threads
race with each other if they access the same memory. For example, if both threads execute:

	some_struct->have_elements = true;
	list_add(new_node, &some_struct->list_head);

the race on some_struct->have_elements may appear benign, however it also implies there
is a race on some_struct->list_head which is not benign, since the list is not thread-safe.

Take into account that on 32-bit systems 64-bit memory accesses may be split into two accesses,
and thus even with a conservative compiler may not be fully atomic. However, such races may
still be benign depending on how writes are done, and how read data is used.

In the final reply explain why you think the given data race is benign or is harmful.

Use the provided tools to confirm any assumptions, variables/fields being accessed, etc.
In particular, don't make assumptions about the kernel source code,
use codesearch tools to read the actual source code.

Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

The data race report is:

==================================================================
BUG: KCSAN: data-race in mem_cgroup_flush_stats_ratelimited / tick_do_update_jiffies64

read-write to 0xffffffff86c09a00 of 8 bytes by interrupt on cpu 1:
 tick_do_update_jiffies64+0x113/0x1c0 kernel/time/tick-sched.c:118
 tick_sched_do_timer kernel/time/tick-sched.c:253 [inline]
 tick_nohz_handler+0x8d/0x3d0 kernel/time/tick-sched.c:312
 __run_hrtimer kernel/time/hrtimer.c:1785 [inline]
 __hrtimer_run_queues+0x218/0x4f0 kernel/time/hrtimer.c:1849
 hrtimer_interrupt+0x269/0x810 kernel/time/hrtimer.c:1911
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1045 [inline]
 __sysvec_apic_timer_interrupt+0x5f/0x1f0 arch/x86/kernel/apic/apic.c:1062
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1056 [inline]
 sysvec_apic_timer_interrupt+0x6f/0x80 arch/x86/kernel/apic/apic.c:1056
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:697
 console_flush_one_record arch/x86/include/asm/irqflags.h:-1 [inline]
 console_flush_all+0x540/0x6c0 kernel/printk/printk.c:3343
 __console_flush_and_unlock kernel/printk/printk.c:3373 [inline]
 console_unlock+0xa1/0x280 kernel/printk/printk.c:3413
 vprintk_emit+0x3e4/0x600 kernel/printk/printk.c:2479
 vprintk_default+0x26/0x30 kernel/printk/printk.c:2494
 vprintk+0x1d/0x30 kernel/printk/printk_safe.c:82
 _printk+0x79/0xa0 kernel/printk/printk.c:2504
 mem_cgroup_print_oom_meminfo+0x84/0x260 mm/memcontrol.c:1593
 dump_header+0xa2/0x240 mm/oom_kill.c:469
 oom_kill_process+0x295/0x350 mm/oom_kill.c:1047
 out_of_memory+0x97d/0xb80 mm/oom_kill.c:1185
 mem_cgroup_out_of_memory mm/memcontrol.c:1695 [inline]
 mem_cgroup_oom mm/memcontrol.c:1718 [inline]
 try_charge_memcg+0x62e/0xa10 mm/memcontrol.c:2456
 obj_cgroup_charge_pages+0x23/0xc0 mm/memcontrol.c:2857
 __memcg_kmem_charge_page+0x9e/0x170 mm/memcontrol.c:2901
 __alloc_frozen_pages_noprof+0x18a/0x360 mm/page_alloc.c:5267
 alloc_pages_mpol+0xb3/0x260 mm/mempolicy.c:2484
 alloc_frozen_pages_noprof mm/mempolicy.c:2555 [inline]
 alloc_pages_noprof+0x8f/0x130 mm/mempolicy.c:2575
 vm_area_alloc_pages mm/vmalloc.c:3731 [inline]
 __vmalloc_area_node mm/vmalloc.c:3876 [inline]
 __vmalloc_node_range_noprof+0xa46/0x12b0 mm/vmalloc.c:4064
 __kvmalloc_node_noprof+0x3d4/0x650 mm/slub.c:6739
 ip_set_alloc+0x24/0x30 net/netfilter/ipset/ip_set_core.c:261
 hash_netiface_create+0x282/0x740 net/netfilter/ipset/ip_set_hash_gen.h:1567
 ip_set_create+0x3cf/0x970 net/netfilter/ipset/ip_set_core.c:1109
 nfnetlink_rcv_msg+0x509/0x5d0 net/netfilter/nfnetlink.c:302
 netlink_rcv_skb+0x123/0x220 net/netlink/af_netlink.c:2550
 nfnetlink_rcv+0x167/0x1720 net/netfilter/nfnetlink.c:669
 netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline]
 netlink_unicast+0x5c0/0x690 net/netlink/af_netlink.c:1344
 netlink_sendmsg+0x5c8/0x6f0 net/netlink/af_netlink.c:1894
 sock_sendmsg_nosec net/socket.c:727 [inline]
 __sock_sendmsg net/socket.c:742 [inline]
 ____sys_sendmsg+0x5af/0x600 net/socket.c:2592
 ___sys_sendmsg+0x195/0x1e0 net/socket.c:2646
 __sys_sendmsg net/socket.c:2678 [inline]
 __do_sys_sendmsg net/socket.c:2683 [inline]
 __se_sys_sendmsg net/socket.c:2681 [inline]
 __x64_sys_sendmsg+0xd4/0x160 net/socket.c:2681
 x64_sys_call+0x194c/0x3020 arch/x86/include/generated/asm/syscalls_64.h:47
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0x12c/0x370 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

read to 0xffffffff86c09a00 of 8 bytes by task 3317 on cpu 0:
 mem_cgroup_flush_stats_ratelimited+0x29/0x70 mm/memcontrol.c:638
 count_shadow_nodes+0x6a/0x230 mm/workingset.c:678
 do_shrink_slab+0x63/0x6a0 mm/shrinker.c:384
 shrink_slab_memcg mm/shrinker.c:550 [inline]
 shrink_slab+0x538/0x880 mm/shrinker.c:628
 shrink_node_memcgs mm/vmscan.c:6022 [inline]
 shrink_node+0x6bc/0x2130 mm/vmscan.c:6061
 shrink_zones mm/vmscan.c:6300 [inline]
 do_try_to_free_pages+0x408/0xc80 mm/vmscan.c:6362
 try_to_free_mem_cgroup_pages+0x1f5/0x470 mm/vmscan.c:6683
 try_charge_memcg+0x37e/0xa10 mm/memcontrol.c:2414
 try_charge mm/memcontrol.c:2556 [inline]
 charge_memcg mm/memcontrol.c:4744 [inline]
 __mem_cgroup_charge+0x65/0x150 mm/memcontrol.c:4761
 mem_cgroup_charge include/linux/memcontrol.h:664 [inline]
 filemap_add_folio+0x110/0x350 mm/filemap.c:959
 __filemap_get_folio_mpol+0x326/0x680 mm/filemap.c:2016
 __filemap_get_folio include/linux/pagemap.h:774 [inline]
 filemap_fault+0x486/0xb90 mm/filemap.c:3559
 __do_fault+0xbc/0x200 mm/memory.c:5363
 do_read_fault mm/memory.c:5798 [inline]
 do_fault mm/memory.c:5932 [inline]
 do_pte_missing mm/memory.c:4477 [inline]
 handle_pte_fault mm/memory.c:6316 [inline]
 __handle_mm_fault mm/memory.c:6454 [inline]
 handle_mm_fault+0x11d7/0x3020 mm/memory.c:6623
 do_user_addr_fault+0x62f/0x1050 arch/x86/mm/fault.c:1334
 handle_page_fault arch/x86/mm/fault.c:1474 [inline]
 exc_page_fault+0x62/0xa0 arch/x86/mm/fault.c:1527
 asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:618

value changed: 0x00000000fffffaef -> 0x00000000fffffaf0

Reported by Kernel Concurrency Sanitizer on:
CPU: 0 UID: 0 PID: 3317 Comm: syz-executor Tainted: G        W           syzkaller #0 PREEMPT(full) 
Tainted: [W]=WARN
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2026
==================================================================
syz-executor invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
CPU: 1 UID: 0 PID: 3317 Comm: syz-executor Tainted: G        W           syzkaller #0 PREEMPT(full) 
Tainted: [W]=WARN
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2026
Call Trace:
 <TASK>
 __dump_stack+0x1d/0x30 lib/dump_stack.c:94
 dump_stack_lvl+0x95/0xd0 lib/dump_stack.c:120
 dump_stack+0x15/0x1b lib/dump_stack.c:129
 dump_header+0x80/0x240 mm/oom_kill.c:467
 oom_kill_process+0x295/0x350 mm/oom_kill.c:1047
 out_of_memory+0x97d/0xb80 mm/oom_kill.c:1185
 mem_cgroup_out_of_memory mm/memcontrol.c:1695 [inline]
 mem_cgroup_oom mm/memcontrol.c:1718 [inline]
 try_charge_memcg+0x62e/0xa10 mm/memcontrol.c:2456
 try_charge mm/memcontrol.c:2556 [inline]
 charge_memcg mm/memcontrol.c:4744 [inline]
 __mem_cgroup_charge+0x65/0x150 mm/memcontrol.c:4761
 mem_cgroup_charge include/linux/memcontrol.h:664 [inline]
 filemap_add_folio+0x110/0x350 mm/filemap.c:959
 __filemap_get_folio_mpol+0x326/0x680 mm/filemap.c:2016
 __filemap_get_folio include/linux/pagemap.h:774 [inline]
 filemap_fault+0x486/0xb90 mm/filemap.c:3559
 __do_fault+0xbc/0x200 mm/memory.c:5363
 do_read_fault mm/memory.c:5798 [inline]
 do_fault mm/memory.c:5932 [inline]
 do_pte_missing mm/memory.c:4477 [inline]
 handle_pte_fault mm/memory.c:6316 [inline]
 __handle_mm_fault mm/memory.c:6454 [inline]
 handle_mm_fault+0x11d7/0x3020 mm/memory.c:6623
 do_user_addr_fault+0x62f/0x1050 arch/x86/mm/fault.c:1334
 handle_page_fault arch/x86/mm/fault.c:1474 [inline]
 exc_page_fault+0x62/0xa0 arch/x86/mm/fault.c:1527
 asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:618
RIP: 0033:0x7f4afab6a6b4
Code: 85 62 0a 00 00 48 b8 db 34 b6 d7 82 de 1b 43 48 f7 a4 24 98 00 00 00 48 8b 05 00 b1 ed 00 48 69 8c 24 90 00 00 00 e8 03 00 00 <8b> 78 08 48 8b 44 24 18 48 c1 ea 12 4c 8b 0d f9 af ed 00 48 01 d1
RSP: 002b:00007fffa0a0a4d0 EFLAGS: 00010206
RAX: 0000001b33a24000 RBX: 00000000000003f7 RCX: 0000000000046118
RDX: 0000000009695fe1 RSI: 00007fffa0a0a560 RDI: 0000000000000001
RBP: 00007fffa0a0a50c R08: 0000000023e708d3 R09: 7fffffffffffffff
R10: 3fffffffffffffff R11: 0000000000000202 R12: 0000000000001388
R13: 00000000000927c0 R14: 00000000000461e0 R15: 00007fffa0a0a560
 </TASK>
memory: usage 307200kB, limit 307200kB, failcnt 1516
memory+swap: usage 307232kB, limit 9007199254740988kB, failcnt 0
kmem: usage 274408kB, limit 9007199254740988kB, failcnt 0
Memory cgroup stats for /syz2:
cache 33185792
rss 192512
shmem 33185792
mapped_file 16384
dirty 0
writeback 0
workingset_refault_anon 192
workingset_refault_file 2753
swap 81920
swapcached 22097920
pgpgin 249181
pgpgout 241029
pgfault 256499
pgmajfault 141
inactive_anon 27643904
active_anon 5480448
inactive_file 0
active_file 4096
unevictable 0
hierarchical_memory_limit 314572800
hierarchical_memsw_limit 9223372036854771712
total_cache 33185792
total_rss 192512
total_shmem 33185792
total_mapped_file 16384
total_dirty 0
total_writeback 0
total_workingset_refault_anon 192
total_workingset_refault_file 2753
total_swap 81920
total_swapcached 22097920
total_pgpgin 249181
total_pgpgout 241029
total_pgfault 256499
total_pgmajfault 141
total_inactive_anon 27643904
total_active_anon 5480448
total_inactive_file 0
total_active_file 4096
total_unevictable 0
oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/syz2,task_memcg=/syz2,task=syz.2.1681,pid=9654,uid=0
Memory cgroup out of memory: Killed process 9654 (syz.2.1681) total-vm:96340kB, anon-rss:1220kB, file-rss:22352kB, shmem-rss:0kB, UID:0 pgtables:128kB oom_score_adj:0

The KCSAN report identifies a data race on the global variable `jiffies_64` between its update in the timer interrupt (`tick_do_update_jiffies64`) and its read in the memory cgroup statistics flushing logic (`mem_cgroup_flush_stats_ratelimited`).

### Analysis

1.  **Nature of the variable**: `jiffies_64` is a 64-bit global counter that tracks the number of system ticks since boot. It is updated by the timer interrupt and read by various parts of the kernel to measure time intervals.
2.  **Access in `tick_do_update_jiffies64`**:
    ```c
    jiffies_64 += ticks;
    ```
    This is a read-modify-write operation performed under `jiffies_lock` and protected by a sequence count (`jiffies_seq`).
3.  **Access in `mem_cgroup_flush_stats_ratelimited`**:
    ```c
    if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
        mem_cgroup_flush_stats(memcg);
    ```
    This is a plain read of `jiffies_64` used to determine if enough time has passed since the last statistics flush to justify another one.
4.  **Atomicity and Tearing**:
    *   On **64-bit systems** (like the x86_64 system in the report), 8-byte aligned loads and stores are atomic at the hardware level. Since `jiffies_64` is declared with `__cacheline_aligned_in_smp`, it is properly aligned. Thus, the read in `mem_cgroup_flush_stats_ratelimited` will always see a consistent 64-bit value.
    *   On **32-bit systems**, a 64-bit read can be torn (split into two 32-bit reads). If a torn read occurs, the resulting value could be significantly incorrect (e.g., jumping far into the future or past). However, the kernel provides `get_jiffies_64()` to avoid this by using the sequence lock.
5.  **Impact of the race**:
    *   In the context of `mem_cgroup_flush_stats_ratelimited`, the value of `jiffies_64` is only used for a ratelimiting check. If a slightly stale or (on 32-bit) torn value is read, the worst consequence is that a statistics flush is either performed slightly early or delayed by one cycle.
    *   Memory cgroup statistics are already allowed to be slightly out of sync (as noted in the comments in `mm/memcontrol.c`), and a minor timing error in flushing them does not lead to memory corruption, crashes, or any functional failure.
6.  **Benign Classification**: The race is on a statistics/timing counter where absolute precision is not required for the correctness of the calling logic. It does not involve complex data structures or pointers that could cause a crash if corrupted.

### Conclusion

The data race is **benign**. While it is technically a violation of the memory model (as it lacks `READ_ONCE` or the use of `get_jiffies_64()`), it does not pose a risk to system stability or correctness in this specific usage. Using `get_jiffies_64()` would be the proper way to suppress the KCSAN warning and ensure correctness on 32-bit platforms, but the current race is harmless.

gemini-3-flash-preview

							input: 7848
							output: 96
							thoughts: 231
						

**Analyzing a Data Race in the Kernel**

Okay, so the report flags a data race, which always gets my attention. It looks like `mem_cgroup_flush_stats_ratelimited` is reading from the same memory address as `tick_do_update_jiffies64`, which is also reading and writing. Not ideal! Let's get down to the details.

First, I'll examine the stack traces to get a better handle on the situation.

The first access, the read-write, occurs within `tick_do_update_jiffies64`. And I see that this is happening in an interrupt context. Hmm, that's interesting and probably crucial.

The second access, the read, is within `mem_cgroup_flush_stats_ratelimited`, and this one's in task context.

Now, if I recall correctly, that address `0xffffffff86c09a00`... given the contexts, it's very likely a global variable, probably `jiffies` or perhaps the 64-bit variant, `jiffies_64`. That's the likely culprit here, given the timing of these functions. So, I need to confirm that suspicion by looking at the source code for both those functions, and confirm how these are being used.

map[ContextFile:kernel/time/tick-sched.c IncludeLines:true Name:tick_do_update_jiffies64]

map[SourceCode:  54:	/*
  55:	 * Must be called with interrupts disabled !
  56:	 */
  57:	static void tick_do_update_jiffies64(ktime_t now)
  58:	{
  59:		unsigned long ticks = 1;
  60:		ktime_t delta, nextp;
  61:	
  62:		/*
  63:		 * 64-bit can do a quick check without holding the jiffies lock and
  64:		 * without looking at the sequence count. The smp_load_acquire()
  65:		 * pairs with the update done later in this function.
  66:		 *
  67:		 * 32-bit cannot do that because the store of 'tick_next_period'
  68:		 * consists of two 32-bit stores, and the first store could be
  69:		 * moved by the CPU to a random point in the future.
  70:		 */
  71:		if (IS_ENABLED(CONFIG_64BIT)) {
  72:			if (ktime_before(now, smp_load_acquire(&tick_next_period)))
  73:				return;
  74:		} else {
  75:			unsigned int seq;
  76:	
  77:			/*
  78:			 * Avoid contention on 'jiffies_lock' and protect the quick
  79:			 * check with the sequence count.
  80:			 */
  81:			do {
  82:				seq = read_seqcount_begin(&jiffies_seq);
  83:				nextp = tick_next_period;
  84:			} while (read_seqcount_retry(&jiffies_seq, seq));
  85:	
  86:			if (ktime_before(now, nextp))
  87:				return;
  88:		}
  89:	
  90:		/* Quick check failed, i.e. update is required. */
  91:		raw_spin_lock(&jiffies_lock);
  92:		/*
  93:		 * Re-evaluate with the lock held. Another CPU might have done the
  94:		 * update already.
  95:		 */
  96:		if (ktime_before(now, tick_next_period)) {
  97:			raw_spin_unlock(&jiffies_lock);
  98:			return;
  99:		}
 100:	
 101:		write_seqcount_begin(&jiffies_seq);
 102:	
 103:		delta = ktime_sub(now, tick_next_period);
 104:		if (unlikely(delta >= TICK_NSEC)) {
 105:			/* Slow path for long idle sleep times */
 106:			s64 incr = TICK_NSEC;
 107:	
 108:			ticks += ktime_divns(delta, incr);
 109:	
 110:			last_jiffies_update = ktime_add_ns(last_jiffies_update,
 111:							   incr * ticks);
 112:		} else {
 113:			last_jiffies_update = ktime_add_ns(last_jiffies_update,
 114:							   TICK_NSEC);
 115:		}
 116:	
 117:		/* Advance jiffies to complete the 'jiffies_seq' protected job */
 118:		jiffies_64 += ticks;
 119:	
 120:		/* Keep the tick_next_period variable up to date */
 121:		nextp = ktime_add_ns(last_jiffies_update, TICK_NSEC);
 122:	
 123:		if (IS_ENABLED(CONFIG_64BIT)) {
 124:			/*
 125:			 * Pairs with smp_load_acquire() in the lockless quick
 126:			 * check above, and ensures that the update to 'jiffies_64' is
 127:			 * not reordered vs. the store to 'tick_next_period', neither
 128:			 * by the compiler nor by the CPU.
 129:			 */
 130:			smp_store_release(&tick_next_period, nextp);
 131:		} else {
 132:			/*
 133:			 * A plain store is good enough on 32-bit, as the quick check
 134:			 * above is protected by the sequence count.
 135:			 */
 136:			tick_next_period = nextp;
 137:		}
 138:	
 139:		/*
 140:		 * Release the sequence count. calc_global_load() below is not
 141:		 * protected by it, but 'jiffies_lock' needs to be held to prevent
 142:		 * concurrent invocations.
 143:		 */
 144:		write_seqcount_end(&jiffies_seq);
 145:	
 146:		calc_global_load();
 147:	
 148:		raw_spin_unlock(&jiffies_lock);
 149:		update_wall_time();
 150:	}
 SourceFile:kernel/time/tick-sched.c]

map[ContextFile:mm/memcontrol.c IncludeLines:true Name:mem_cgroup_flush_stats_ratelimited]

map[SourceCode: 635:	void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
 636:	{
 637:		/* Only flush if the periodic flusher is one full cycle late */
 638:		if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
 639:			mem_cgroup_flush_stats(memcg);
 640:	}
 SourceFile:mm/memcontrol.c]

gemini-3-flash-preview

							input: 10041
							output: 44
							thoughts: 459
						

**Analyzing a Potential Race Condition with jiffies_64**

Okay, so the issue at hand seems to be a potential race condition involving `jiffies_64`. The race is on `jiffies_64`, specifically. Looking at the code snippets, I see that in `tick_do_update_jiffies64`, there's a straightforward write to `jiffies_64`:

```c
118:		jiffies_64 += ticks;
```

This write *is* protected by `jiffies_lock` and `jiffies_seq`, which is good. That helps with the possibility of corruption.

However, the problem lies in `mem_cgroup_flush_stats_ratelimited`, where `jiffies_64` is read:

```c
638:		if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
```

And this read *isn't* protected by any lock or `READ_ONCE`. Now, `time_after64` is pretty standard, it's defined as:

```c
#define time_after64(a,b)       \
        (typecheck(__u64, a) && \
         typecheck(__u64, b) && \
         ((__s64)((b) - (a)) < 0))
```

The crucial thing here is that `jiffies_64` is a 64-bit variable. My immediate concern is the atomicity of the read. On 64-bit architectures, a 64-bit read is usually atomic. However, on 32-bit systems, a 64-bit access requires multiple reads and therefore is non-atomic. The report specifies an 8-byte access to `0xffffffff86c09a00`, which confirms a 64-bit access.

So, is this a 32 bit system? No, based on the context (`x64_sys_call`, `do_syscall_64`), the system is x86_64, which is a 64-bit architecture.

Okay, so on x86_64, 8-byte aligned loads and stores *are* atomic. But that only is true if it's aligned. The question now is: is `jiffies_64` aligned? It's a global variable, so it should be. But I want to make sure of that. I'm going to double check where `jiffies_64` is defined and if it's usually accessed with `READ_ONCE`. I need to verify its alignment and confirm if the absence of `READ_ONCE` here is truly benign given the 64-bit architecture. I'll also be sure to check the code that touches it.

map[ContextFile:kernel/time/tick-sched.c IncludeLines:true Name:jiffies_64]

map[SourceCode:  60:	__visible u64 jiffies_64 __cacheline_aligned_in_smp = INITIAL_JIFFIES;
 SourceFile:kernel/time/timer.c]

gemini-3-flash-preview

							input: 11256
							output: 22
							thoughts: 60
						

Okay, here's my interpretation of those thoughts, tailored for an expert audience:

**Analyzing `jiffies_64` Read Operations**

Right, so I'm focusing on the implementation of `jiffies_64`. Specifically, I'm thinking about its cacheline alignment, marked by `__cacheline_aligned_in_smp`. That suggests careful consideration was given to its concurrent access in a symmetric multiprocessing (SMP) environment, probably to avoid false sharing. I need to keep that in mind.

Next, I'm recalling that `jiffies_64` reading is handled via the helpers defined in `include/linux/jiffies.h`. My immediate goal is to understand precisely how those helpers are designed to interact with `jiffies_64`. I want to drill down into the code and see the exact mechanism that's used to read `jiffies_64`. I'm expecting to find either explicit memory barriers, atomics, or some other concurrency control, given the SMP alignment. I also want to understand if there are any architecture specific considerations. Understanding how to correctly and efficiently read the system's `jiffies_64` value is critical to any time-sensitive kernel operation.

map[Dir:include/linux/]

map[Files:[8250_pci.h acct.h acpi.h acpi_amd_wbrf.h acpi_dma.h acpi_iort.h acpi_mdio.h acpi_pmtmr.h acpi_rimt.h acpi_viot.h adb.h adfs_fs.h adi-axi-common.h adreno-smmu-priv.h adxl.h aer.h agp_backend.h agpgart.h ahci-remap.h ahci_platform.h aio.h alarmtimer.h alcor_pci.h align.h alloc_tag.h altera_jtaguart.h altera_uart.h amd-iommu.h amd-pmf-io.h annotate.h anon_inodes.h aperture.h apm-emulation.h apm_bios.h apple-gmux.h arch_topology.h args.h arm-cci.h arm-smccc.h arm_ffa.h arm_mpam.h arm_sdei.h armada-37xx-rwtm-mailbox.h array_size.h ascii85.h asn1.h asn1_ber_bytecode.h asn1_decoder.h asn1_encoder.h assoc_array.h assoc_array_priv.h async.h async_tx.h ata.h ata_platform.h atalk.h atm.h atm_tcp.h atmdev.h atmel-isc-media.h atmel-ssc.h atmel_pdc.h atomic.h attribute_container.h audit.h audit_arch.h auto_dev-ioctl.h auto_fs.h auxiliary_bus.h auxvec.h average.h backing-dev-defs.h backing-dev.h backing-file.h backlight.h badblocks.h balloon.h base64.h bcd.h bch.h bcm47xx_nvram.h bcm47xx_sprom.h bcm47xx_wdt.h bcm963xx_nvram.h bcm963xx_tag.h binfmts.h bio-integrity.h bio.h bit_spinlock.h bitfield.h bitmap-str.h bitmap.h bitops.h bitrev.h bits.h blk-cgroup.h blk-crypto-profile.h blk-crypto.h blk-integrity.h blk-mq-dma.h blk-mq.h blk-pm.h blk_types.h blkdev.h blkpg.h blktrace_api.h blockgroup_lock.h bma150.h bootconfig.h bootmem_info.h bottom_half.h bpf-cgroup-defs.h bpf-cgroup.h bpf-netns.h bpf.h bpf_crypto.h bpf_lirc.h bpf_local_storage.h bpf_lsm.h bpf_mem_alloc.h bpf_mprog.h bpf_trace.h bpf_types.h bpf_verifier.h bpfptr.h brcmphy.h bsearch.h bsg-lib.h bsg.h btf.h btf_ids.h btree-128.h btree-type.h btree.h btrfs.h buffer_head.h bug.h build-salt.h build_bug.h buildid.h bvec.h c2port.h cache.h cache_coherency.h cacheflush.h cacheinfo.h call_once.h capability.h cb710.h cc_platform.h cciss_ioctl.h ccp.h cdev.h cdrom.h cfag12864b.h cfi.h cfi_types.h cgroup-defs.h cgroup.h cgroup_api.h cgroup_dmem.h cgroup_namespace.h cgroup_rdma.h cgroup_refcnt.h cgroup_subsys.h circ_buf.h cleanup.h clk-provider.h clk.h clkdev.h clockchips.h clocksource.h clocksource_ids.h closure.h cma.h cmpxchg-emu.h cn_proc.h cnt32_to_63.h coda.h codetag.h compaction.h compat.h compiler-clang.h compiler-context-analysis.h compiler-gcc.h compiler-version.h compiler.h compiler_attributes.h compiler_types.h completion.h component.h configfs.h connector.h console.h console_struct.h consolemap.h const.h container.h container_of.h context_tracking.h context_tracking_irq.h context_tracking_state.h cookie.h cordic.h coredump.h coresight-pmu.h coresight-stm.h coresight.h count_zeros.h counter.h cper.h cpu.h cpu_cooling.h cpu_pm.h cpu_rmap.h cpu_smt.h cpufeature.h cpufreq.h cpuhotplug.h cpuhplock.h cpuidle.h cpuidle_haltpoll.h cpumask.h cpumask_api.h cpumask_types.h cpuset.h crash_core.h crash_dump.h crash_reserve.h crc-ccitt.h crc-itu-t.h crc-t10dif.h crc16.h crc32.h crc32c.h crc32poly.h crc4.h crc64.h crc7.h crc8.h cred.h crypto.h cs5535.h ctype.h cuda.h damon.h dasd_mod.h davinci_emac.h dax.h dca.h dcache.h dccp.h debug_locks.h debugfs.h debugobjects.h delay.h delayacct.h delayed_call.h dev_printk.h devcoredump.h devfreq-event.h devfreq-governor.h devfreq.h devfreq_cooling.h device-mapper.h device.h device_cgroup.h devm-helpers.h devpts_fs.h dfl.h dibs.h digsig.h dim.h dio.h dirent.h dlm.h dlm_plock.h dm-bufio.h dm-dirty-log.h dm-io.h dm-kcopyd.h dm-region-hash.h dm-verity-loadpin.h dm9000.h dma-buf-mapping.h dma-buf.h dma-direct.h dma-direction.h dma-fence-array.h dma-fence-chain.h dma-fence-unwrap.h dma-fence.h dma-heap.h dma-map-ops.h dma-mapping.h dma-resv.h dmaengine.h dmapool.h dmar.h dmi.h dnotify.h dns_resolver.h dpll.h dqblk_qtree.h dqblk_v1.h dqblk_v2.h drbd.h drbd_config.h drbd_genl.h drbd_genl_api.h drbd_limits.h ds2782_battery.h dtlk.h dtpm.h dw_apb_timer.h dynamic_debug.h dynamic_queue_limits.h earlycpio.h ecryptfs.h edac.h edd.h eeprom_93cx6.h efi-bgrt.h efi.h efi_embedded_fw.h efs_vh.h ehl_pse_io_aux.h eisa.h elf-fdpic.h elf-randomize.h elf.h elfcore-compat.h elfcore.h elfnote-lto.h elfnote.h enclosure.h energy_model.h entry-common.h entry-virt.h err.h errname.h errno.h error-injection.h errqueue.h errseq.h etherdevice.h ethtool.h ethtool_netlink.h eventfd.h eventpoll.h evm.h execmem.h export-internal.h export.h exportfs.h ext2_fs.h extable.h extcon-provider.h extcon.h f2fs_fs.h f75375s.h falloc.h fanotify.h fault-inject-usercopy.h fault-inject.h fb.h fbcon.h fcdevice.h fcntl.h fd.h fddidevice.h fdtable.h fec.h fiemap.h file.h file_ref.h fileattr.h filelock.h filter.h find.h fips.h firewire.h firmware-map.h firmware.h fixp-arith.h flat.h flex_proportions.h folio_queue.h font.h fortify-string.h fprobe.h fpu.h freezer.h fs.h fs_api.h fs_context.h fs_dirent.h fs_parser.h fs_pin.h fs_stack.h fs_struct.h fscache-cache.h fscache.h fscrypt.h fserror.h fsi-occ.h fsi-sbefifo.h fsi.h fsl-diu-fb.h fsl_devices.h fsl_hypervisor.h fsl_ifc.h fsldma.h fsnotify.h fsnotify_backend.h fsverity.h ftrace.h ftrace_irq.h ftrace_regs.h futex.h fw_table.h fwctl.h fwnode.h fwnode_mdio.h gameport.h gcd.h genalloc.h generic-radix-tree.h genl_magic_func.h genl_magic_struct.h gfp.h gfp_api.h gfp_types.h glob.h gnss.h goldfish.h gpio-pxa.h gpio.h gpio_keys.h greybus.h group_cpus.h hardirq.h hash.h hashtable.h hashtable_api.h hdlc.h hdlcdrv.h hdmi.h hex.h hfs_common.h hid-debug.h hid-over-i2c.h hid-over-spi.h hid-roccat.h hid-sensor-hub.h hid-sensor-ids.h hid.h hid_bpf.h hidden.h hiddev.h hidraw.h highmem-internal.h highmem.h highuid.h hil.h hil_mlc.h hisi_acc_qm.h hmm-dma.h hmm.h host1x.h host1x_context_bus.h hp_sdc.h hpet.h hrtimer.h hrtimer_api.h hrtimer_defs.h hrtimer_types.h hte.h huge_mm.h hugetlb.h hugetlb_cgroup.h hugetlb_inline.h hung_task.h hw_bitfield.h hw_breakpoint.h hw_random.h hwmon-sysfs.h hwmon-vid.h hwmon.h hwspinlock.h hyperv.h hypervisor.h i2c-algo-bit.h i2c-algo-pca.h i2c-algo-pcf.h i2c-atr.h i2c-dev.h i2c-mux.h i2c-of-prober.h i2c-smbus.h i2c.h i8042.h i8253.h i8254.h icmp.h icmpv6.h idle_inject.h idr.h ieee80211-eht.h ieee80211-he.h ieee80211-ht.h ieee80211-mesh.h ieee80211-nan.h ieee80211-p2p.h ieee80211-s1g.h ieee80211-uhr.h ieee80211-vht.h ieee80211.h ieee802154.h if_arp.h if_bridge.h if_eql.h if_ether.h if_fddi.h if_hsr.h if_link.h if_macvlan.h if_phonet.h if_pppol2tp.h if_pppox.h if_rmnet.h if_tap.h if_team.h if_tun.h if_tunnel.h if_vlan.h igmp.h ihex.h ima.h imx-media.h in.h in6.h indirect_call_wrapper.h inet.h inet_diag.h inetdevice.h init.h init_ohci1394_dma.h init_syscalls.h init_task.h initrd.h inotify.h input.h instruction_pointer.h instrumentation.h instrumented.h int_log.h integrity.h intel-ish-client-if.h intel_dg_nvm_aux.h intel_pmt_features.h intel_rapl.h intel_tcc.h intel_th.h intel_tpmi.h intel_vsec.h interconnect-clk.h interconnect-provider.h interconnect.h interrupt.h interval_tree.h interval_tree_generic.h io-64-nonatomic-hi-lo.h io-64-nonatomic-lo-hi.h io-mapping.h io-pgtable.h io.h io_uring.h io_uring_types.h ioam6.h ioam6_genl.h ioam6_iptunnel.h iocontext.h iomap.h iommu-debug-pagealloc.h iommu-dma.h iommu-helper.h iommu.h iommufd.h iopoll.h ioport.h ioprio.h ioremap.h iosys-map.h iov_iter.h iova.h iova_bitmap.h ip.h ipack.h ipc.h ipc_namespace.h ipmi.h ipmi_smi.h ipv6.h ipv6_route.h irq-entry-common.h irq.h irq_poll.h irq_sim.h irq_work.h irq_work_types.h irqbypass.h irqchip.h irqdesc.h irqdomain.h irqdomain_defs.h irqflags.h irqflags_types.h irqhandler.h irqnr.h irqreturn.h isa-dma.h isa.h isapnp.h iscsi_boot_sysfs.h iscsi_ibft.h ism.h iversion.h jbd2.h jhash.h jiffies.h journal-head.h joystick.h jump_label.h jump_label_ratelimit.h jz4740-adc.h jz4780-nemc.h kallsyms.h kasan-checks.h kasan-enabled.h kasan-tags.h kasan.h kbd_diacr.h kbd_kern.h kbuild.h kconfig.h kcore.h kcov.h kcsan-checks.h kcsan.h kdb.h kdebug.h kdev_t.h kern_levels.h kernel-page-flags.h kernel.h kernel_read_file.h kernel_stat.h kernelcapi.h kernfs.h kexec.h kexec_handover.h key-type.h key.h keyboard.h keyctl.h kfence.h kfifo.h kgdb.h khugepaged.h klist.h kmemleak.h kmod.h kmsan-checks.h kmsan.h kmsan_string.h kmsan_types.h kmsg_dump.h kobj_map.h kobject.h kobject_api.h kobject_ns.h kprobes.h kref.h kref_api.h ks0108.h ks8842.h ks8851_mll.h ksm.h kstack_erase.h kstrtox.h kthread.h ktime.h ktime_api.h kvm_dirty_ring.h kvm_host.h kvm_irqfd.h kvm_para.h kvm_types.h l2tp.h lantiq.h lapb.h latencytop.h lcd.h lcm.h leafops.h led-class-flash.h led-class-multicolor.h led-lm3530.h leds-bd2802.h leds-expresswire.h leds-lp3944.h leds-lp3952.h leds-pca9532.h leds-regulator.h leds-ti-lmu-common.h leds.h libata.h libfdt.h libfdt_env.h libgcc.h libnvdimm.h libps2.h license.h limits.h linear_range.h linkage.h linkmode.h linux_logo.h lis3lv02d.h list.h list_bl.h list_lru.h list_nulls.h list_private.h list_sort.h litex.h livepatch.h livepatch_external.h livepatch_helpers.h livepatch_sched.h liveupdate.h llc.h llist.h llist_api.h local_lock.h local_lock_internal.h lockdep.h lockdep_api.h lockdep_types.h lockref.h log2.h logic_iomem.h logic_pio.h lp.h lru_cache.h lsm_audit.h lsm_count.h lsm_hook_defs.h lsm_hooks.h lwq.h lz4.h lzo.h mISDNdsp.h mISDNhw.h mISDNif.h mailbox_client.h mailbox_controller.h maple.h maple_tree.h marvell_phy.h math.h math64.h mbcache.h mbus.h mc146818rtc.h mc33xs2410.h mc6821.h mcb.h mdev.h mdio-bitbang.h mdio-gpio.h mdio-mux.h mdio.h mei_aux.h mei_cl_bus.h mem_encrypt.h memblock.h memcontrol.h memfd.h memory-failure.h memory-tiers.h memory.h memory_hotplug.h mempolicy.h mempool.h memregion.h memremap.h memstick.h mhi.h mhi_ep.h micrel_phy.h microchipphy.h migrate.h migrate_mode.h mii.h mii_timestamper.h min_heap.h minmax.h misc_cgroup.h miscdevice.h mm.h mm_api.h mm_inline.h mm_types.h mm_types_task.h mman.h mmap_lock.h mmdebug.h mmiotrace.h mmu_context.h mmu_notifier.h mmzone.h mnt_idmapping.h mnt_namespace.h mod_devicetable.h module.h module_signature.h module_symbol.h moduleloader.h moduleparam.h most.h mount.h moxtet.h mpage.h mpi.h mpls.h mpls_iptunnel.h mroute.h mroute6.h mroute_base.h msdos_fs.h msdos_partition.h msg.h msi.h msi_api.h mtio.h mutex.h mutex_api.h mutex_types.h mv643xx_eth.h mv643xx_i2c.h mvebu-pmsu.h mxm-wmi.h namei.h nd.h ndctl.h net.h net_tstamp.h netdev_features.h netdevice.h netdevice_xmit.h netfilter.h netfilter_bridge.h netfilter_defs.h netfilter_ipv4.h netfilter_ipv6.h netfilter_netdev.h netfs.h netlink.h netpoll.h nfs.h nfs3.h nfs4.h nfs_common.h nfs_fs.h nfs_fs_i.h nfs_fs_sb.h nfs_iostat.h nfs_page.h nfs_ssc.h nfs_xdr.h nfsacl.h nfslocalio.h nitro_enclaves.h nl802154.h nls.h nmi.h node.h nodemask.h nodemask_types.h nospec.h notifier.h ns_common.h nsc_gpio.h nsfs.h nsproxy.h nstree.h ntb.h ntb_transport.h nubus.h numa.h numa_memblks.h nvme-auth.h nvme-fc-driver.h nvme-fc.h nvme-keyring.h nvme-rdma.h nvme-tcp.h nvme.h nvmem-consumer.h nvmem-provider.h nvram.h oa_tc6.h objagg.h objpool.h objtool.h objtool_types.h of.h of_address.h of_clk.h of_device.h of_dma.h of_fdt.h of_gpio.h of_graph.h of_iommu.h of_irq.h of_mdio.h of_net.h of_pci.h of_pdt.h of_platform.h of_reserved_mem.h oid_registry.h olpc-ec.h omap-dma.h omap-gpmc.h omap-iommu.h omap-mailbox.h omapfb.h once.h once_lite.h oom.h openvswitch.h osq_lock.h overflow.h packing.h padata.h page-flags-layout.h page-flags.h page-isolation.h page_counter.h page_ext.h page_frag_cache.h page_idle.h page_owner.h page_ref.h page_reporting.h page_table_check.h pageblock-flags.h pagemap.h pagevec.h pagewalk.h panic.h panic_notifier.h papr_scm.h parman.h parport.h parport_pc.h parser.h part_stat.h pata_arasan_cf_data.h patchkey.h path.h pch_dma.h pci-acpi.h pci-ats.h pci-bwctrl.h pci-doe.h pci-ecam.h pci-ep-cfs.h pci-ep-msi.h pci-epc.h pci-epf.h pci-ide.h pci-p2pdma.h pci-pwrctrl.h pci-tph.h pci-tsm.h pci.h pci_hotplug.h pci_ids.h pcie-dwc.h pcs-lynx.h pcs-rzn1-miic.h pe.h peci-cpu.h peci.h percpu-defs.h percpu-refcount.h percpu-rwsem.h percpu.h percpu_counter.h perf_event.h perf_event_api.h perf_regs.h personality.h pfn.h pgalloc.h pgalloc_tag.h pgtable.h pgtable_api.h phonet.h phy.h phy_fixed.h phy_led_triggers.h phy_link_topology.h phy_port.h phylib_stubs.h phylink.h pid.h pid_namespace.h pid_types.h pidfs.h pim.h pipe_fs_i.h pkeys.h pl320-ipc.h platform_device.h platform_profile.h pldmfw.h plist.h plist_types.h pm-trace.h pm.h pm_clock.h pm_domain.h pm_opp.h pm_qos.h pm_runtime.h pm_wakeirq.h pm_wakeup.h pmbus.h pmu.h pnp.h poison.h poll.h polynomial.h posix-clock.h posix-timers.h posix-timers_types.h posix_acl.h posix_acl_xattr.h power_supply.h powercap.h ppp-comp.h ppp_channel.h ppp_defs.h pps_gen_kernel.h pps_kernel.h pr.h prandom.h preempt.h prefetch.h prime_numbers.h printk.h prmt.h proc_fs.h proc_ns.h processor.h profile.h projid.h property.h pruss_driver.h psci.h pseudo_fs.h psi.h psi_types.h psp-platform-access.h psp-sev.h psp-tee.h psp.h pstore.h pstore_blk.h pstore_ram.h pstore_zone.h ptdump.h pti.h ptp_classify.h ptp_clock_kernel.h ptp_kvm.h ptp_mock.h ptp_pch.h ptr_ring.h ptrace.h ptrace_api.h purgatory.h pvclock_gtod.h pwm.h pwm_backlight.h pxa168_eth.h pxa2xx_ssp.h qnx6_fs.h quota.h quotaops.h radix-tree.h raid_class.h ramfs.h random.h randomize_kstack.h range.h ras.h ratelimit.h ratelimit_types.h rational.h rbtree.h rbtree_augmented.h rbtree_latch.h rbtree_types.h rcu_node_tree.h rcu_notifier.h rcu_segcblist.h rcu_sync.h rculist.h rculist_bl.h rculist_nulls.h rcupdate.h rcupdate_trace.h rcupdate_wait.h rcuref.h rcutiny.h rcutree.h rcuwait.h rcuwait_api.h reboot-mode.h reboot.h reciprocal_div.h ref_tracker.h refcount.h refcount_api.h refcount_types.h regmap.h regset.h relay.h remoteproc.h resctrl.h resctrl_types.h reset-controller.h reset.h resource.h resource_ext.h restart_block.h resume_user_mode.h rethook.h rfkill.h rhashtable-types.h rhashtable.h ring_buffer.h rio.h rio_drv.h rio_ids.h rio_regs.h rmap.h rmi.h rndis.h rodata_test.h rolling_buffer.h root_dev.h rpmb.h rpmsg.h rseq.h rseq_entry.h rseq_types.h rslib.h rtc.h rtmutex.h rtnetlink.h rtsx_common.h rtsx_pci.h rtsx_usb.h rv.h rw_hint.h rwbase_rt.h rwlock.h rwlock_api_smp.h rwlock_rt.h rwlock_types.h rwsem.h sbitmap.h scatterlist.h scc.h sched.h sched_clock.h scmi_imx_protocol.h scmi_protocol.h scpi_protocol.h screen_info.h scs.h sctp.h scx200.h scx200_gpio.h seccomp.h seccomp_types.h secretmem.h securebits.h security.h sed-opal-key.h sed-opal.h seg6.h seg6_genl.h seg6_hmac.h seg6_iptunnel.h seg6_local.h selection.h sem.h sem_types.h semaphore.h seq_buf.h seq_file.h seq_file_net.h seqlock.h seqlock_api.h seqlock_types.h serdev.h serial.h serial_8250.h serial_bcm63xx.h serial_core.h serial_s3c.h serial_sci.h serio.h set_memory.h sfp.h sh_clk.h sh_dma.h sh_eth.h sh_intc.h sh_timer.h shdma-base.h shm.h shmem_fs.h shrinker.h signal.h signal_types.h signalfd.h siox.h siphash.h sizes.h skb_array.h skbuff.h skbuff_ref.h skmsg.h slab.h slimbus.h sm501-regs.h sm501.h smc91x.h smp.h smp_types.h smpboot.h smsc911x.h smscphy.h sock_diag.h socket.h sockptr.h softirq.h sonet.h sonypi.h sort.h sound.h soundcard.h spinlock.h spinlock_api.h spinlock_api_smp.h spinlock_api_up.h spinlock_rt.h spinlock_types.h spinlock_types_raw.h spinlock_types_up.h spinlock_up.h splice.h spmi.h sprintf.h sram.h srcu.h srcutiny.h srcutree.h ssbi.h stackdepot.h stackprotector.h stacktrace.h start_kernel.h stat.h statfs.h static_call.h static_call_types.h static_key.h stdarg.h stddef.h stm.h stmmac.h stmp3xxx_rtc_wdt.h stmp_device.h stop_machine.h string.h string_choices.h string_helpers.h stringhash.h stringify.h sungem_phy.h sunserialcore.h sunxi-rsb.h surface_acpi_notify.h suspend.h svga.h sw842.h swab.h swait.h swait_api.h swap.h swap_cgroup.h swapfile.h swapops.h swiotlb.h switchtec.h sxgbe_platform.h sync_core.h sync_file.h synclink.h sys.h sys_info.h sys_soc.h syscall_user_dispatch.h syscall_user_dispatch_types.h syscalls.h syscalls_api.h syscore_ops.h sysctl.h sysfb.h sysfs.h syslog.h sysrq.h t10-pi.h task_io_accounting.h task_io_accounting_ops.h task_work.h taskstats_kern.h tboot.h tc.h tcp.h tee_core.h tee_drv.h tegra-icc.h termios_internal.h text-patching.h textsearch.h textsearch_fsm.h thermal.h thread_info.h threads.h thunderbolt.h ti-emif-sram.h ti_wilink_st.h tick.h tifm.h timb_dma.h timb_gpio.h time.h time32.h time64.h time_namespace.h timecounter.h timekeeper_internal.h timekeeping.h timer.h timer_types.h timerfd.h timeriomem-rng.h timerqueue.h timerqueue_types.h timex.h tnum.h topology.h torture.h toshiba.h tpm.h tpm_command.h tpm_eventlog.h tpm_svsm.h trace.h trace_clock.h trace_events.h trace_printk.h trace_recursion.h trace_seq.h tracefs.h tracepoint-defs.h tracepoint.h transport_class.h ts-nbus.h tsacct_kern.h tsm-mr.h tsm.h tty.h tty_buffer.h tty_driver.h tty_flip.h tty_ldisc.h tty_port.h turris-omnia-mcu-interface.h turris-signing-key.h typecheck.h types.h u64_stats_sync.h u64_stats_sync_api.h uacce.h uaccess.h ubsan.h ucopysize.h ucs2_string.h udp.h uidgid.h uidgid_types.h uio.h uio_driver.h umh.h unaligned.h unicode.h union_find.h units.h unroll.h unwind_deferred.h unwind_deferred_types.h unwind_user.h unwind_user_types.h uprobes.h usb.h usb_usual.h usbdevice_fs.h user-return-notifier.h user.h user_events.h user_namespace.h userfaultfd_k.h util_macros.h uts.h uts_namespace.h utsname.h uuid.h vbox_utils.h vdpa.h vdso_datastore.h verification.h vermagic.h vexpress.h vfio.h vfio_pci_core.h vfs.h vfsdebug.h vga_switcheroo.h vgaarb.h vhost_iotlb.h via-core.h via.h via_i2c.h videodev2.h virtio.h virtio_anchor.h virtio_byteorder.h virtio_caif.h virtio_config.h virtio_dma_buf.h virtio_features.h virtio_net.h virtio_pci_admin.h virtio_pci_legacy.h virtio_pci_modern.h virtio_ring.h virtio_vsock.h vm_event_item.h vmalloc.h vmcore_info.h vmpressure.h vmstat.h vmw_vmci_api.h vmw_vmci_defs.h vringh.h vt.h vt_buffer.h vt_kern.h vtime.h w1.h wait.h wait_api.h wait_bit.h watch_queue.h watchdog.h win_minmax.h wireless.h wkup_m3_ipc.h wm97xx.h wmi.h wordpart.h workqueue.h workqueue_api.h workqueue_types.h writeback.h ww_mutex.h wwan.h xarray.h xattr.h xxhash.h xz.h yam.h zconf.h zlib.h zorro.h zsmalloc.h zstd.h zstd_errors.h zstd_lib.h zswap.h zutil.h] Subdirs:[amba atomic avf bcma bnge bnxt bus byteorder can cdx ceph clk comedi crush decompress device dma dma-buf dsa extcon firmware fpga framer fs fsl generic_pt gpio greybus habanalabs hsi i3c iio input io_uring irqchip isdn kho lockd lsm mailbox mdio memory mfd misc mlx4 mlx5 mmc mtd mux net netfilter netfilter_arp netfilter_bridge netfilter_ipv4 netfilter_ipv6 ns pcs pds perf phy pinctrl platform_data power pse-pd pwrseq qat qed raid raspberrypi regulator remoteproc reset rpmsg rtc sched soc soundwire spi ssb sunrpc surface_aggregator ulpi unaligned usb]]

gemini-3-flash-preview

							input: 24728
							output: 24
							thoughts: 15
						

Okay, here's my take on those thoughts, tailored for an expert audience:

**Examining the Jiffies Landscape**

Right, let's dive straight in. My immediate thought is to head over to `include/linux/jiffies.h`. That's the logical starting point, isn't it? I need to understand the fundamental definitions, the underlying data type, and the architectural considerations. I'm thinking about the potential for wrap-around behavior, the implications for high-resolution timers, and how the kernel handles comparisons and arithmetic operations with `jiffies`. I'm already anticipating the need to consider the `HAVE_` preprocessor directives and their impact on the specific implementation for this platform. What special features is this architecture using? What about the 32-bit vs. 64-bit considerations? I'll probably be looking for the atomic operations and any lock-free designs used when reading and writing the global `jiffies` variable and its related fields. Finally, I'm expecting some comments to explain any optimization strategies used and their impact.

map[File:include/linux/jiffies.h]

map[Contents:/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_JIFFIES_H
#define _LINUX_JIFFIES_H

#include <linux/cache.h>
#include <linux/limits.h>
#include <linux/math64.h>
#include <linux/minmax.h>
#include <linux/types.h>
#include <linux/time.h>
#include <linux/timex.h>
#include <vdso/jiffies.h>
#include <asm/param.h>			/* for HZ */
#include <generated/timeconst.h>

/*
 * The following defines establish the engineering parameters of the PLL
 * model. The HZ variable establishes the timer interrupt frequency, 100 Hz
 * for the SunOS kernel, 256 Hz for the Ultrix kernel and 1024 Hz for the
 * OSF/1 kernel. The SHIFT_HZ define expresses the same value as the
 * nearest power of two in order to avoid hardware multiply operations.
 */
#if HZ >= 12 && HZ < 24
# define SHIFT_HZ	4
#elif HZ >= 24 && HZ < 48
# define SHIFT_HZ	5
#elif HZ >= 48 && HZ < 96
# define SHIFT_HZ	6
#elif HZ >= 96 && HZ < 192
# define SHIFT_HZ	7
#elif HZ >= 192 && HZ < 384
# define SHIFT_HZ	8
#elif HZ >= 384 && HZ < 768
# define SHIFT_HZ	9
#elif HZ >= 768 && HZ < 1536
# define SHIFT_HZ	10
#elif HZ >= 1536 && HZ < 3072
# define SHIFT_HZ	11
#elif HZ >= 3072 && HZ < 6144
# define SHIFT_HZ	12
#elif HZ >= 6144 && HZ < 12288
# define SHIFT_HZ	13
#else
# error Invalid value of HZ.
#endif

/* Suppose we want to divide two numbers NOM and DEN: NOM/DEN, then we can
 * improve accuracy by shifting LSH bits, hence calculating:
 *     (NOM << LSH) / DEN
 * This however means trouble for large NOM, because (NOM << LSH) may no
 * longer fit in 32 bits. The following way of calculating this gives us
 * some slack, under the following conditions:
 *   - (NOM / DEN) fits in (32 - LSH) bits.
 *   - (NOM % DEN) fits in (32 - LSH) bits.
 */
#define SH_DIV(NOM,DEN,LSH) (   (((NOM) / (DEN)) << (LSH))              \
                             + ((((NOM) % (DEN)) << (LSH)) + (DEN) / 2) / (DEN))

/* LATCH is used in the interval timer and ftape setup. */
#define LATCH ((CLOCK_TICK_RATE + HZ/2) / HZ)	/* For divider */

extern void register_refined_jiffies(long clock_tick_rate);

/* TICK_USEC is the time between ticks in usec */
#define TICK_USEC ((USEC_PER_SEC + HZ/2) / HZ)

/* USER_TICK_USEC is the time between ticks in usec assuming fake USER_HZ */
#define USER_TICK_USEC ((1000000UL + USER_HZ/2) / USER_HZ)

#ifndef __jiffy_arch_data
#define __jiffy_arch_data
#endif

/*
 * The 64-bit value is not atomic on 32-bit systems - you MUST NOT read it
 * without sampling the sequence number in jiffies_lock.
 * get_jiffies_64() will do this for you as appropriate.
 *
 * jiffies and jiffies_64 are at the same address for little-endian systems
 * and for 64-bit big-endian systems.
 * On 32-bit big-endian systems, jiffies is the lower 32 bits of jiffies_64
 * (i.e., at address @jiffies_64 + 4).
 * See arch/ARCH/kernel/vmlinux.lds.S
 */
extern u64 __cacheline_aligned_in_smp jiffies_64;
extern unsigned long volatile __cacheline_aligned_in_smp __jiffy_arch_data jiffies;

#if (BITS_PER_LONG < 64)
u64 get_jiffies_64(void);
#else
/**
 * get_jiffies_64 - read the 64-bit non-atomic jiffies_64 value
 *
 * When BITS_PER_LONG < 64, this uses sequence number sampling using
 * jiffies_lock to protect the 64-bit read.
 *
 * Return: current 64-bit jiffies value
 */
static inline u64 get_jiffies_64(void)
{
	return (u64)jiffies;
}
#endif

/**
 * DOC: General information about time_* inlines
 *
 * These inlines deal with timer wrapping correctly. You are strongly encouraged
 * to use them:
 *
 * #. Because people otherwise forget
 * #. Because if the timer wrap changes in future you won't have to alter your
 *    driver code.
 */

/**
 * time_after - returns true if the time a is after time b.
 * @a: first comparable as unsigned long
 * @b: second comparable as unsigned long
 *
 * Do this with "<0" and ">=0" to only test the sign of the result. A
 * good compiler would generate better code (and a really good compiler
 * wouldn't care). Gcc is currently neither.
 *
 * Return: %true is time a is after time b, otherwise %false.
 */
#define time_after(a,b)		\
	(typecheck(unsigned long, a) && \
	 typecheck(unsigned long, b) && \
	 ((long)((b) - (a)) < 0))
/**
 * time_before - returns true if the time a is before time b.
 * @a: first comparable as unsigned long
 * @b: second comparable as unsigned long
 *
 * Return: %true is time a is before time b, otherwise %false.
 */
#define time_before(a,b)	time_after(b,a)

/**
 * time_after_eq - returns true if the time a is after or the same as time b.
 * @a: first comparable as unsigned long
 * @b: second comparable as unsigned long
 *
 * Return: %true is time a is after or the same as time b, otherwise %false.
 */
#define time_after_eq(a,b)	\
	(typecheck(unsigned long, a) && \
	 typecheck(unsigned long, b) && \
	 ((long)((a) - (b)) >= 0))
/**
 * time_before_eq - returns true if the time a is before or the same as time b.
 * @a: first comparable as unsigned long
 * @b: second comparable as unsigned long
 *
 * Return: %true is time a is before or the same as time b, otherwise %false.
 */
#define time_before_eq(a,b)	time_after_eq(b,a)

/**
 * time_in_range - Calculate whether a is in the range of [b, c].
 * @a: time to test
 * @b: beginning of the range
 * @c: end of the range
 *
 * Return: %true is time a is in the range [b, c], otherwise %false.
 */
#define time_in_range(a,b,c) \
	(time_after_eq(a,b) && \
	 time_before_eq(a,c))

/**
 * time_in_range_open - Calculate whether a is in the range of [b, c).
 * @a: time to test
 * @b: beginning of the range
 * @c: end of the range
 *
 * Return: %true is time a is in the range [b, c), otherwise %false.
 */
#define time_in_range_open(a,b,c) \
	(time_after_eq(a,b) && \
	 time_before(a,c))

/* Same as above, but does so with platform independent 64bit types.
 * These must be used when utilizing jiffies_64 (i.e. return value of
 * get_jiffies_64()). */

/**
 * time_after64 - returns true if the time a is after time b.
 * @a: first comparable as __u64
 * @b: second comparable as __u64
 *
 * This must be used when utilizing jiffies_64 (i.e. return value of
 * get_jiffies_64()).
 *
 * Return: %true is time a is after time b, otherwise %false.
 */
#define time_after64(a,b)	\
	(typecheck(__u64, a) &&	\
	 typecheck(__u64, b) && \
	 ((__s64)((b) - (a)) < 0))
/**
 * time_before64 - returns true if the time a is before time b.
 * @a: first comparable as __u64
 * @b: second comparable as __u64
 *
 * This must be used when utilizing jiffies_64 (i.e. return value of
 * get_jiffies_64()).
 *
 * Return: %true is time a is before time b, otherwise %false.
 */
#define time_before64(a,b)	time_after64(b,a)

/**
 * time_after_eq64 - returns true if the time a is after or the same as time b.
 * @a: first comparable as __u64
 * @b: second comparable as __u64
 *
 * This must be used when utilizing jiffies_64 (i.e. return value of
 * get_jiffies_64()).
 *
 * Return: %true is time a is after or the same as time b, otherwise %false.
 */
#define time_after_eq64(a,b)	\
	(typecheck(__u64, a) && \
	 typecheck(__u64, b) && \
	 ((__s64)((a) - (b)) >= 0))
/**
 * time_before_eq64 - returns true if the time a is before or the same as time b.
 * @a: first comparable as __u64
 * @b: second comparable as __u64
 *
 * This must be used when utilizing jiffies_64 (i.e. return value of
 * get_jiffies_64()).
 *
 * Return: %true is time a is before or the same as time b, otherwise %false.
 */
#define time_before_eq64(a,b)	time_after_eq64(b,a)

/**
 * time_in_range64 - Calculate whether a is in the range of [b, c].
 * @a: time to test
 * @b: beginning of the range
 * @c: end of the range
 *
 * Return: %true is time a is in the range [b, c], otherwise %false.
 */
#define time_in_range64(a, b, c) \
	(time_after_eq64(a, b) && \
	 time_before_eq64(a, c))

/*
 * These eight macros compare jiffies[_64] and 'a' for convenience.
 */

/**
 * time_is_before_jiffies - return true if a is before jiffies
 * @a: time (unsigned long) to compare to jiffies
 *
 * Return: %true is time a is before jiffies, otherwise %false.
 */
#define time_is_before_jiffies(a) time_after(jiffies, a)
/**
 * time_is_before_jiffies64 - return true if a is before jiffies_64
 * @a: time (__u64) to compare to jiffies_64
 *
 * Return: %true is time a is before jiffies_64, otherwise %false.
 */
#define time_is_before_jiffies64(a) time_after64(get_jiffies_64(), a)

/**
 * time_is_after_jiffies - return true if a is after jiffies
 * @a: time (unsigned long) to compare to jiffies
 *
 * Return: %true is time a is after jiffies, otherwise %false.
 */
#define time_is_after_jiffies(a) time_before(jiffies, a)
/**
 * time_is_after_jiffies64 - return true if a is after jiffies_64
 * @a: time (__u64) to compare to jiffies_64
 *
 * Return: %true is time a is after jiffies_64, otherwise %false.
 */
#define time_is_after_jiffies64(a) time_before64(get_jiffies_64(), a)

/**
 * time_is_before_eq_jiffies - return true if a is before or equal to jiffies
 * @a: time (unsigned long) to compare to jiffies
 *
 * Return: %true is time a is before or the same as jiffies, otherwise %false.
 */
#define time_is_before_eq_jiffies(a) time_after_eq(jiffies, a)
/**
 * time_is_before_eq_jiffies64 - return true if a is before or equal to jiffies_64
 * @a: time (__u64) to compare to jiffies_64
 *
 * Return: %true is time a is before or the same jiffies_64, otherwise %false.
 */
#define time_is_before_eq_jiffies64(a) time_after_eq64(get_jiffies_64(), a)

/**
 * time_is_after_eq_jiffies - return true if a is after or equal to jiffies
 * @a: time (unsigned long) to compare to jiffies
 *
 * Return: %true is time a is after or the same as jiffies, otherwise %false.
 */
#define time_is_after_eq_jiffies(a) time_before_eq(jiffies, a)
/**
 * time_is_after_eq_jiffies64 - return true if a is after or equal to jiffies_64
 * @a: time (__u64) to compare to jiffies_64
 *
 * Return: %true is time a is after or the same as jiffies_64, otherwise %false.
 */
#define time_is_after_eq_jiffies64(a) time_before_eq64(get_jiffies_64(), a)

/*
 * Have the 32-bit jiffies value wrap 5 minutes after boot
 * so jiffies wrap bugs show up earlier.
 */
#define INITIAL_JIFFIES ((unsigned long)(unsigned int) (-300*HZ))

/*
 * Change timeval to jiffies, trying to avoid the
 * most obvious overflows..
 *
 * And some not so obvious.
 *
 * Note that we don't want to return LONG_MAX, because
 * for various timeout reasons we often end up having
 * to wait "jiffies+1" in order to guarantee that we wait
 * at _least_ "jiffies" - so "jiffies+1" had better still
 * be positive.
 */
#define MAX_JIFFY_OFFSET ((LONG_MAX >> 1)-1)

extern unsigned long preset_lpj;

/*
 * We want to do realistic conversions of time so we need to use the same
 * values the update wall clock code uses as the jiffies size.  This value
 * is: TICK_NSEC (which is defined in timex.h).  This
 * is a constant and is in nanoseconds.  We will use scaled math
 * with a set of scales defined here as SEC_JIFFIE_SC,  USEC_JIFFIE_SC and
 * NSEC_JIFFIE_SC.  Note that these defines contain nothing but
 * constants and so are computed at compile time.  SHIFT_HZ (computed in
 * timex.h) adjusts the scaling for different HZ values.

 * Scaled math???  What is that?
 *
 * Scaled math is a way to do integer math on values that would,
 * otherwise, either overflow, underflow, or cause undesired div
 * instructions to appear in the execution path.  In short, we "scale"
 * up the operands so they take more bits (more precision, less
 * underflow), do the desired operation and then "scale" the result back
 * by the same amount.  If we do the scaling by shifting we avoid the
 * costly mpy and the dastardly div instructions.

 * Suppose, for example, we want to convert from seconds to jiffies
 * where jiffies is defined in nanoseconds as NSEC_PER_JIFFIE.  The
 * simple math is: jiff = (sec * NSEC_PER_SEC) / NSEC_PER_JIFFIE; We
 * observe that (NSEC_PER_SEC / NSEC_PER_JIFFIE) is a constant which we
 * might calculate at compile time, however, the result will only have
 * about 3-4 bits of precision (less for smaller values of HZ).
 *
 * So, we scale as follows:
 * jiff = (sec) * (NSEC_PER_SEC / NSEC_PER_JIFFIE);
 * jiff = ((sec) * ((NSEC_PER_SEC * SCALE)/ NSEC_PER_JIFFIE)) / SCALE;
 * Then we make SCALE a power of two so:
 * jiff = ((sec) * ((NSEC_PER_SEC << SCALE)/ NSEC_PER_JIFFIE)) >> SCALE;
 * Now we define:
 * #define SEC_CONV = ((NSEC_PER_SEC << SCALE)/ NSEC_PER_JIFFIE))
 * jiff = (sec * SEC_CONV) >> SCALE;
 *
 * Often the math we use will expand beyond 32-bits so we tell C how to
 * do this and pass the 64-bit result of the mpy through the ">> SCALE"
 * which should take the result back to 32-bits.  We want this expansion
 * to capture as much precision as possible.  At the same time we don't
 * want to overflow so we pick the SCALE to avoid this.  In this file,
 * that means using a different scale for each range of HZ values (as
 * defined in timex.h).
 *
 * For those who want to know, gcc will give a 64-bit result from a "*"
 * operator if the result is a long long AND at least one of the
 * operands is cast to long long (usually just prior to the "*" so as
 * not to confuse it into thinking it really has a 64-bit operand,
 * which, buy the way, it can do, but it takes more code and at least 2
 * mpys).

 * We also need to be aware that one second in nanoseconds is only a
 * couple of bits away from overflowing a 32-bit word, so we MUST use
 * 64-bits to get the full range time in nanoseconds.

 */

/*
 * Here are the scales we will use.  One for seconds, nanoseconds and
 * microseconds.
 *
 * Within the limits of cpp we do a rough cut at the SEC_JIFFIE_SC and
 * check if the sign bit is set.  If not, we bump the shift count by 1.
 * (Gets an extra bit of precision where we can use it.)
 * We know it is set for HZ = 1024 and HZ = 100 not for 1000.
 * Haven't tested others.

 * Limits of cpp (for #if expressions) only long (no long long), but
 * then we only need the most signicant bit.
 */

#define SEC_JIFFIE_SC (31 - SHIFT_HZ)
#if !((((NSEC_PER_SEC << 2) / TICK_NSEC) << (SEC_JIFFIE_SC - 2)) & 0x80000000)
#undef SEC_JIFFIE_SC
#define SEC_JIFFIE_SC (32 - SHIFT_HZ)
#endif
#define NSEC_JIFFIE_SC (SEC_JIFFIE_SC + 29)
#define SEC_CONVERSION ((unsigned long)((((u64)NSEC_PER_SEC << SEC_JIFFIE_SC) +\
                                TICK_NSEC -1) / (u64)TICK_NSEC))

#define NSEC_CONVERSION ((unsigned long)((((u64)1 << NSEC_JIFFIE_SC) +\
                                        TICK_NSEC -1) / (u64)TICK_NSEC))
/*
 * The maximum jiffy value is (MAX_INT >> 1).  Here we translate that
 * into seconds.  The 64-bit case will overflow if we are not careful,
 * so use the messy SH_DIV macro to do it.  Still all constants.
 */
#if BITS_PER_LONG < 64
# define MAX_SEC_IN_JIFFIES \
	(long)((u64)((u64)MAX_JIFFY_OFFSET * TICK_NSEC) / NSEC_PER_SEC)
#else	/* take care of overflow on 64-bit machines */
# define MAX_SEC_IN_JIFFIES \
	(SH_DIV((MAX_JIFFY_OFFSET >> SEC_JIFFIE_SC) * TICK_NSEC, NSEC_PER_SEC, 1) - 1)

#endif

/*
 * Convert various time units to each other:
 */
extern unsigned int jiffies_to_msecs(const unsigned long j);
extern unsigned int jiffies_to_usecs(const unsigned long j);

/**
 * jiffies_to_nsecs - Convert jiffies to nanoseconds
 * @j: jiffies value
 *
 * Return: nanoseconds value
 */
static inline u64 jiffies_to_nsecs(const unsigned long j)
{
	return (u64)jiffies_to_usecs(j) * NSEC_PER_USEC;
}

extern u64 jiffies64_to_nsecs(u64 j);
extern u64 jiffies64_to_msecs(u64 j);

extern unsigned long __msecs_to_jiffies(const unsigned int m);
#if HZ <= MSEC_PER_SEC && !(MSEC_PER_SEC % HZ)
/*
 * HZ is equal to or smaller than 1000, and 1000 is a nice round
 * multiple of HZ, divide with the factor between them, but round
 * upwards:
 */
static inline unsigned long _msecs_to_jiffies(const unsigned int m)
{
	return (m + (MSEC_PER_SEC / HZ) - 1) / (MSEC_PER_SEC / HZ);
}
#elif HZ > MSEC_PER_SEC && !(HZ % MSEC_PER_SEC)
/*
 * HZ is larger than 1000, and HZ is a nice round multiple of 1000 -
 * simply multiply with the factor between them.
 *
 * But first make sure the multiplication result cannot overflow:
 */
static inline unsigned long _msecs_to_jiffies(const unsigned int m)
{
	if (m > jiffies_to_msecs(MAX_JIFFY_OFFSET))
		return MAX_JIFFY_OFFSET;
	return m * (HZ / MSEC_PER_SEC);
}
#else
/*
 * Generic case - multiply, round and divide. But first check that if
 * we are doing a net multiplication, that we wouldn't overflow:
 */
static inline unsigned long _msecs_to_jiffies(const unsigned int m)
{
	if (HZ > MSEC_PER_SEC && m > jiffies_to_msecs(MAX_JIFFY_OFFSET))
		return MAX_JIFFY_OFFSET;

	return (MSEC_TO_HZ_MUL32 * m + MSEC_TO_HZ_ADJ32) >> MSEC_TO_HZ_SHR32;
}
#endif
/**
 * msecs_to_jiffies: - convert milliseconds to jiffies
 * @m:	time in milliseconds
 *
 * conversion is done as follows:
 *
 * - negative values mean 'infinite timeout' (MAX_JIFFY_OFFSET)
 *
 * - 'too large' values [that would result in larger than
 *   MAX_JIFFY_OFFSET values] mean 'infinite timeout' too.
 *
 * - all other values are converted to jiffies by either multiplying
 *   the input value by a factor or dividing it with a factor and
 *   handling any 32-bit overflows.
 *   for the details see _msecs_to_jiffies()
 *
 * msecs_to_jiffies() checks for the passed in value being a constant
 * via __builtin_constant_p() allowing gcc to eliminate most of the
 * code. __msecs_to_jiffies() is called if the value passed does not
 * allow constant folding and the actual conversion must be done at
 * runtime.
 * The HZ range specific helpers _msecs_to_jiffies() are called both
 * directly here and from __msecs_to_jiffies() in the case where
 * constant folding is not possible.
 *
 * Return: jiffies value
 */
static __always_inline unsigned long msecs_to_jiffies(const unsigned int m)
{
	if (__builtin_constant_p(m)) {
		if ((int)m < 0)
			return MAX_JIFFY_OFFSET;
		return _msecs_to_jiffies(m);
	} else {
		return __msecs_to_jiffies(m);
	}
}

/**
 * secs_to_jiffies: - convert seconds to jiffies
 * @_secs: time in seconds
 *
 * Conversion is done by simple multiplication with HZ
 *
 * secs_to_jiffies() is defined as a macro rather than a static inline
 * function so it can be used in static initializers.
 *
 * Return: jiffies value
 */
#define secs_to_jiffies(_secs) (unsigned long)((_secs) * HZ)

extern unsigned long __usecs_to_jiffies(const unsigned int u);
#if !(USEC_PER_SEC % HZ)
static inline unsigned long _usecs_to_jiffies(const unsigned int u)
{
	return (u + (USEC_PER_SEC / HZ) - 1) / (USEC_PER_SEC / HZ);
}
#else
static inline unsigned long _usecs_to_jiffies(const unsigned int u)
{
	return (USEC_TO_HZ_MUL32 * u + USEC_TO_HZ_ADJ32)
		>> USEC_TO_HZ_SHR32;
}
#endif

/**
 * usecs_to_jiffies: - convert microseconds to jiffies
 * @u:	time in microseconds
 *
 * conversion is done as follows:
 *
 * - 'too large' values [that would result in larger than
 *   MAX_JIFFY_OFFSET values] mean 'infinite timeout' too.
 *
 * - all other values are converted to jiffies by either multiplying
 *   the input value by a factor or dividing it with a factor and
 *   handling any 32-bit overflows as for msecs_to_jiffies.
 *
 * usecs_to_jiffies() checks for the passed in value being a constant
 * via __builtin_constant_p() allowing gcc to eliminate most of the
 * code. __usecs_to_jiffies() is called if the value passed does not
 * allow constant folding and the actual conversion must be done at
 * runtime.
 * The HZ range specific helpers _usecs_to_jiffies() are called both
 * directly here and from __msecs_to_jiffies() in the case where
 * constant folding is not possible.
 *
 * Return: jiffies value
 */
static __always_inline unsigned long usecs_to_jiffies(const unsigned int u)
{
	if (__builtin_constant_p(u)) {
		if (u > jiffies_to_usecs(MAX_JIFFY_OFFSET))
			return MAX_JIFFY_OFFSET;
		return _usecs_to_jiffies(u);
	} else {
		return __usecs_to_jiffies(u);
	}
}

extern unsigned long timespec64_to_jiffies(const struct timespec64 *value);
extern void jiffies_to_timespec64(const unsigned long jiffies,
				  struct timespec64 *value);
extern clock_t jiffies_to_clock_t(unsigned long x);

static inline clock_t jiffies_delta_to_clock_t(long delta)
{
	return jiffies_to_clock_t(max(0L, delta));
}

static inline unsigned int jiffies_delta_to_msecs(long delta)
{
	return jiffies_to_msecs(max(0L, delta));
}

extern unsigned long clock_t_to_jiffies(unsigned long x);
extern u64 jiffies_64_to_clock_t(u64 x);
extern u64 nsec_to_clock_t(u64 x);
extern u64 nsecs_to_jiffies64(u64 n);
extern unsigned long nsecs_to_jiffies(u64 n);

#define TIMESTAMP_SIZE	30

struct ctl_table;
int proc_dointvec_jiffies(const struct ctl_table *table, int dir, void *buffer,
			  size_t *lenp, loff_t *ppos);
int proc_dointvec_ms_jiffies_minmax(const struct ctl_table *table, int dir,
				    void *buffer, size_t *lenp, loff_t *ppos);
int proc_dointvec_userhz_jiffies(const struct ctl_table *table, int dir,
				 void *buffer, size_t *lenp, loff_t *ppos);
int proc_dointvec_ms_jiffies(const struct ctl_table *table, int dir, void *buffer,
			     size_t *lenp, loff_t *ppos);
int proc_doulongvec_ms_jiffies_minmax(const struct ctl_table *table, int dir,
				      void *buffer, size_t *lenp, loff_t *ppos);

#endif
]

gemini-3-flash-preview

							input: 32558
							output: 20
							thoughts: 1364
						

**Analyzing the `jiffies_64` Data Race**

Okay, so I'm looking at this KCSAN report, and it's flagged a data race on `jiffies_64`.  Let's break this down. In the `include/linux/jiffies.h`, `get_jiffies_64()` is defined differently depending on `BITS_PER_LONG`. On 64-bit systems, which is what I'm focused on right now, it simply returns `(u64)jiffies`.  The catch is, `jiffies` itself is declared as `volatile`.

Now, the problem arises in `mem_cgroup_flush_stats_ratelimited`. It directly accesses `jiffies_64` in this comparison: `time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME)`.  But `jiffies_64` isn't declared `volatile`. This is the crux of the issue.

The write side of the race is in `tick_do_update_jiffies64`, which does a simple increment: `jiffies_64 += ticks;`. Although this is inside a `write_seqcount_begin/end` block and uses `jiffies_lock`, it is still a plain write. So this is also a plain access to `jiffies_64`, which KCSAN is catching because it races with the read in `mem_cgroup_flush_stats_ratelimited`.

My initial thought is that if `jiffies_64` is intended to be read locklessly on 64-bit systems, it should be accessed with `READ_ONCE` or `get_jiffies_64()`. Actually, `get_jiffies_64()` on 64-bit returns `jiffies`, which *is* `volatile`. So, if `mem_cgroup_flush_stats_ratelimited` used `get_jiffies_64()`, it would be reading a `volatile` variable, which might still trigger KCSAN, but `volatile` usually helps supress some KCSAN warnings or is considered "safe enough" in some cases.

Thinking about it more, if `jiffies` is `volatile`, then the cast `(u64)jiffies` effectively creates a volatile load.  This changes things. The race is definitely between a plain read of `jiffies_64` in `mem_cgroup_flush_stats_ratelimited` and the plain write of `jiffies_64` in `tick_do_update_jiffies64`.

The question becomes: is this data race actually harmful? It's a counter.  A data race on a counter, where we only care about "roughly" correct values, is often considered benign. On 32-bit systems, I'd be very worried, because a 64-bit read or write might be torn. However, on 64-bit systems, the read and write is atomic at the hardware level if the memory is aligned, but the compiler could still cause issues. In `mem_cgroup_flush_stats_ratelimited`, we're simply checking if enough time has passed.  A slightly old or new value of `jiffies_64` probably won't break the ratelimiting logic, but KCSAN is correct in pointing out the data race.

I should verify that `jiffies_64` is not read directly elsewhere. I need to check the other code paths, but it seems like the design intent is for `jiffies_64` to be read via `get_jiffies_64()` and for all uses of `jiffies` to use the `volatile` variable.

map[Expression:jiffies_64]

map[Output:
Full output is too long, showing 500 out of 1048 lines.
Use more precise expression if possible.

[arch/alpha/kernel/vmlinux.lds.S=15=PHDRS { text PT_LOAD; note PT_NOTE; }
 arch/alpha/kernel/vmlinux.lds.S:16:jiffies = jiffies_64;
 arch/alpha/kernel/vmlinux.lds.S-17-SECTIONS
 --
 arch/arc/kernel/vmlinux.lds.S=12=ENTRY(res_service)
 --
 arch/arc/kernel/vmlinux.lds.S-14-#ifdef CONFIG_CPU_BIG_ENDIAN
 arch/arc/kernel/vmlinux.lds.S:15:jiffies = jiffies_64 + 4;
 arch/arc/kernel/vmlinux.lds.S-16-#else
 arch/arc/kernel/vmlinux.lds.S:17:jiffies = jiffies_64;
 arch/arc/kernel/vmlinux.lds.S-18-#endif
 --
 arch/arm/kernel/smp_twd.c=140=static void twd_calibrate_rate(void)
 --
 arch/arm/kernel/smp_twd.c-152-		/* Wait for a tick to start */
 arch/arm/kernel/smp_twd.c:153:		waitjiffies = get_jiffies_64() + 1;
 arch/arm/kernel/smp_twd.c-154-
 arch/arm/kernel/smp_twd.c:155:		while (get_jiffies_64() < waitjiffies)
 arch/arm/kernel/smp_twd.c-156-			udelay(10);
 --
 arch/arm/kernel/smp_twd.c-166-
 arch/arm/kernel/smp_twd.c:167:		while (get_jiffies_64() < waitjiffies)
 arch/arm/kernel/smp_twd.c-168-			udelay(10);
 --
 arch/arm/kernel/vmlinux-xip.lds.S=19=ENTRY(stext)
 --
 arch/arm/kernel/vmlinux-xip.lds.S-21-#ifndef __ARMEB__
 arch/arm/kernel/vmlinux-xip.lds.S:22:jiffies = jiffies_64;
 arch/arm/kernel/vmlinux-xip.lds.S-23-#else
 arch/arm/kernel/vmlinux-xip.lds.S:24:jiffies = jiffies_64 + 4;
 arch/arm/kernel/vmlinux-xip.lds.S-25-#endif
 --
 arch/arm/kernel/vmlinux.lds.S=19=ENTRY(stext)
 --
 arch/arm/kernel/vmlinux.lds.S-21-#ifndef __ARMEB__
 arch/arm/kernel/vmlinux.lds.S:22:jiffies = jiffies_64;
 arch/arm/kernel/vmlinux.lds.S-23-#else
 arch/arm/kernel/vmlinux.lds.S:24:jiffies = jiffies_64 + 4;
 arch/arm/kernel/vmlinux.lds.S-25-#endif
 --
 arch/arm64/kernel/vmlinux.lds.S=83=ENTRY(_text)
 arch/arm64/kernel/vmlinux.lds.S-84-
 arch/arm64/kernel/vmlinux.lds.S:85:jiffies = jiffies_64;
 arch/arm64/kernel/vmlinux.lds.S-86-
 --
 arch/csky/kernel/vmlinux.lds.S=8=ENTRY(_start)
 --
 arch/csky/kernel/vmlinux.lds.S-10-#ifndef __cskyBE__
 arch/csky/kernel/vmlinux.lds.S:11:jiffies = jiffies_64;
 arch/csky/kernel/vmlinux.lds.S-12-#else
 arch/csky/kernel/vmlinux.lds.S:13:jiffies = jiffies_64 + 4;
 arch/csky/kernel/vmlinux.lds.S-14-#endif
 --
 arch/hexagon/kernel/vmlinux.lds.S=15=ENTRY(stext)
 arch/hexagon/kernel/vmlinux.lds.S-16-
 arch/hexagon/kernel/vmlinux.lds.S:17:jiffies = jiffies_64;
 arch/hexagon/kernel/vmlinux.lds.S-18-
 --
 arch/loongarch/kernel/vmlinux.lds.S=29=PHDRS {
 --
 arch/loongarch/kernel/vmlinux.lds.S-33-
 arch/loongarch/kernel/vmlinux.lds.S:34:jiffies	 = jiffies_64;
 arch/loongarch/kernel/vmlinux.lds.S-35-
 --
 arch/m68k/kernel/vmlinux-nommu.lds=25=ENTRY(_start)
 arch/m68k/kernel/vmlinux-nommu.lds-26-
 arch/m68k/kernel/vmlinux-nommu.lds:27:jiffies = jiffies_64 + 4;
 arch/m68k/kernel/vmlinux-nommu.lds-28-
 --
 arch/m68k/kernel/vmlinux-std.lds=10=ENTRY(_start)
 arch/m68k/kernel/vmlinux-std.lds:11:jiffies = jiffies_64 + 4;
 arch/m68k/kernel/vmlinux-std.lds-12-SECTIONS
 --
 arch/m68k/kernel/vmlinux-sun3.lds=10=ENTRY(_start)
 arch/m68k/kernel/vmlinux-sun3.lds:11:jiffies = jiffies_64 + 4;
 arch/m68k/kernel/vmlinux-sun3.lds-12-SECTIONS
 --
 arch/microblaze/kernel/vmlinux.lds.S=12=ENTRY(microblaze_start)
 --
 arch/microblaze/kernel/vmlinux.lds.S-21-#ifdef __MICROBLAZEEL__
 arch/microblaze/kernel/vmlinux.lds.S:22:jiffies = jiffies_64;
 arch/microblaze/kernel/vmlinux.lds.S-23-#else
 arch/microblaze/kernel/vmlinux.lds.S:24:jiffies = jiffies_64 + 4;
 arch/microblaze/kernel/vmlinux.lds.S-25-#endif
 --
 arch/mips/kernel/vmlinux.lds.S=26=PHDRS {
 --
 arch/mips/kernel/vmlinux.lds.S-34-	#ifdef CONFIG_CPU_LITTLE_ENDIAN
 arch/mips/kernel/vmlinux.lds.S:35:		jiffies	 = jiffies_64;
 arch/mips/kernel/vmlinux.lds.S-36-	#else
 arch/mips/kernel/vmlinux.lds.S:37:		jiffies	 = jiffies_64 + 4;
 arch/mips/kernel/vmlinux.lds.S-38-	#endif
 arch/mips/kernel/vmlinux.lds.S-39-#else
 arch/mips/kernel/vmlinux.lds.S:40:	jiffies	 = jiffies_64;
 arch/mips/kernel/vmlinux.lds.S-41-#endif
 --
 arch/nios2/kernel/vmlinux.lds.S=13=ENTRY(_start)	/* Defined in head.S */
 arch/nios2/kernel/vmlinux.lds.S-14-
 arch/nios2/kernel/vmlinux.lds.S:15:jiffies = jiffies_64;
 arch/nios2/kernel/vmlinux.lds.S-16-
 --
 arch/openrisc/kernel/vmlinux.lds.S=36=OUTPUT_FORMAT(__OUTPUT_FORMAT, __OUTPUT_FORMAT, __OUTPUT_FORMAT)
 arch/openrisc/kernel/vmlinux.lds.S:37:jiffies = jiffies_64 + 4;
 arch/openrisc/kernel/vmlinux.lds.S-38-
 --
 arch/parisc/kernel/vmlinux.lds.S=50=ENTRY(parisc_kernel_start)
 arch/parisc/kernel/vmlinux.lds.S-51-#ifndef CONFIG_64BIT
 arch/parisc/kernel/vmlinux.lds.S:52:jiffies = jiffies_64 + 4;
 arch/parisc/kernel/vmlinux.lds.S-53-#else
 arch/parisc/kernel/vmlinux.lds.S:54:jiffies = jiffies_64;
 arch/parisc/kernel/vmlinux.lds.S-55-#endif
 --
 arch/powerpc/kernel/vmlinux.lds.S=42=OUTPUT_ARCH(powerpc:common64)
 arch/powerpc/kernel/vmlinux.lds.S:43:jiffies = jiffies_64;
 arch/powerpc/kernel/vmlinux.lds.S-44-#else
 arch/powerpc/kernel/vmlinux.lds.S=45=OUTPUT_ARCH(powerpc:common)
 arch/powerpc/kernel/vmlinux.lds.S:46:jiffies = jiffies_64 + 4;
 arch/powerpc/kernel/vmlinux.lds.S-47-#endif
 --
 arch/riscv/kernel/vmlinux-xip.lds.S=20=ENTRY(_start)
 arch/riscv/kernel/vmlinux-xip.lds.S-21-
 arch/riscv/kernel/vmlinux-xip.lds.S:22:jiffies = jiffies_64;
 arch/riscv/kernel/vmlinux-xip.lds.S-23-
 --
 arch/riscv/kernel/vmlinux.lds.S=26=ENTRY(_start)
 arch/riscv/kernel/vmlinux.lds.S-27-
 arch/riscv/kernel/vmlinux.lds.S:28:jiffies = jiffies_64;
 arch/riscv/kernel/vmlinux.lds.S-29-
 --
 arch/s390/kernel/vmlinux.lds.S=31=ENTRY(startup_continue)
 arch/s390/kernel/vmlinux.lds.S:32:jiffies = jiffies_64;
 arch/s390/kernel/vmlinux.lds.S-33-
 --
 arch/s390/kernel/vtime.c=59=static void update_mt_scaling(void)
 --
 arch/s390/kernel/vtime.c-83-	}
 arch/s390/kernel/vtime.c:84:	__this_cpu_write(mt_scaling_jiffies, jiffies_64);
 arch/s390/kernel/vtime.c-85-}
 --
 arch/s390/kernel/vtime.c=118=static int do_account_vtime(struct task_struct *tsk)
 --
 arch/s390/kernel/vtime.c-140-	if (smp_cpu_mtid &&
 arch/s390/kernel/vtime.c:141:	    time_after64(jiffies_64, this_cpu_read(mt_scaling_jiffies)))
 arch/s390/kernel/vtime.c-142-		update_mt_scaling();
 --
 arch/sh/Makefile=106=ld_bfd			:= elf32-sh-linux
 arch/sh/Makefile:107:LDFLAGS_vmlinux		+= --defsym jiffies=jiffies_64 --oformat $(ld_bfd)
 arch/sh/Makefile-108-KBUILD_LDFLAGS		+= -EL
 --
 arch/sh/Makefile=110=ld_bfd			:= elf32-shbig-linux
 arch/sh/Makefile:111:LDFLAGS_vmlinux		+= --defsym jiffies=jiffies_64+4 --oformat $(ld_bfd)
 arch/sh/Makefile-112-KBUILD_LDFLAGS		+= -EB
 --
 arch/sparc/kernel/vmlinux.lds.S=27=ENTRY(_start)
 arch/sparc/kernel/vmlinux.lds.S:28:jiffies = jiffies_64 + 4;
 arch/sparc/kernel/vmlinux.lds.S-29-#else
 --
 arch/sparc/kernel/vmlinux.lds.S=33=ENTRY(_start)
 arch/sparc/kernel/vmlinux.lds.S:34:jiffies = jiffies_64;
 arch/sparc/kernel/vmlinux.lds.S-35-#endif
 --
 arch/um/kernel/dyn.lds.S=6=ENTRY(_start)
 arch/um/kernel/dyn.lds.S:7:jiffies = jiffies_64;
 arch/um/kernel/dyn.lds.S-8-
 --
 arch/um/kernel/uml.lds.S=7=ENTRY(_start)
 arch/um/kernel/uml.lds.S:8:jiffies = jiffies_64;
 arch/um/kernel/uml.lds.S-9-
 --
 arch/x86/kernel/tsc.c=237=noinstr u64 native_sched_clock(void)
 --
 arch/x86/kernel/tsc.c-255-	/* No locking but a rare wrong value is not a big deal: */
 arch/x86/kernel/tsc.c:256:	return (jiffies_64 - INITIAL_JIFFIES) * (1000000000 / HZ);
 arch/x86/kernel/tsc.c-257-}
 --
 arch/x86/kernel/vmlinux.lds.S=42=ENTRY(phys_startup_64)
 --
 arch/x86/kernel/vmlinux.lds.S-44-
 arch/x86/kernel/vmlinux.lds.S:45:jiffies = jiffies_64;
 arch/x86/kernel/vmlinux.lds.S-46-const_current_task = current_task;
 --
 arch/x86/kvm/mmu/mmu.c=7830=static bool kvm_nx_huge_page_recovery_worker(void *data)
 --
 arch/x86/kvm/mmu/mmu.c-7842-	remaining_time = kvm->arch.nx_huge_page_last + msecs_to_jiffies(period)
 arch/x86/kvm/mmu/mmu.c:7843:		- get_jiffies_64();
 arch/x86/kvm/mmu/mmu.c-7844-	if (remaining_time > 0) {
 --
 arch/x86/kvm/mmu/mmu.c-7852-		kvm_recover_nx_huge_pages(kvm, i);
 arch/x86/kvm/mmu/mmu.c:7853:	kvm->arch.nx_huge_page_last = get_jiffies_64();
 arch/x86/kvm/mmu/mmu.c-7854-	return true;
 --
 arch/x86/kvm/mmu/mmu.c=7857=static int kvm_mmu_start_lpage_recovery(struct once *once)
 --
 arch/x86/kvm/mmu/mmu.c-7862-
 arch/x86/kvm/mmu/mmu.c:7863:	kvm->arch.nx_huge_page_last = get_jiffies_64();
 arch/x86/kvm/mmu/mmu.c-7864-	nx_thread = vhost_task_create(kvm_nx_huge_page_recovery_worker,
 --
 arch/x86/tools/relocs.c=53=static const char * const	sym_regex_kernel[S_NSYMTYPES] = {
 --
 arch/x86/tools/relocs.c-90-	"__initramfs_start|"
 arch/x86/tools/relocs.c:91:	"(jiffies|jiffies_64)|"
 arch/x86/tools/relocs.c-92-#if ELF_BITS == 64
 --
 arch/xtensa/kernel/vmlinux.lds.S=27=ENTRY(_start)
 --
 arch/xtensa/kernel/vmlinux.lds.S-29-#ifdef __XTENSA_EB__
 arch/xtensa/kernel/vmlinux.lds.S:30:jiffies = jiffies_64 + 4;
 arch/xtensa/kernel/vmlinux.lds.S-31-#else
 arch/xtensa/kernel/vmlinux.lds.S:32:jiffies = jiffies_64;
 arch/xtensa/kernel/vmlinux.lds.S-33-#endif
 --
 drivers/acpi/osl.c=655=u64 acpi_os_get_timer(void)
 drivers/acpi/osl.c-656-{
 drivers/acpi/osl.c:657:	return (get_jiffies_64() - INITIAL_JIFFIES) *
 drivers/acpi/osl.c-658-		(ACPI_100NSEC_PER_SEC / HZ);
 --
 drivers/ata/libata-eh.c=370=static void ata_ering_record(struct ata_ering *ering, unsigned int eflags,
 --
 drivers/ata/libata-eh.c-382-	ent->err_mask = err_mask;
 drivers/ata/libata-eh.c:383:	ent->timestamp = get_jiffies_64();
 drivers/ata/libata-eh.c-384-}
 --
 drivers/ata/libata-eh.c=1821=static unsigned int ata_eh_speed_down_verdict(struct ata_device *dev)
 --
 drivers/ata/libata-eh.c-1823-	const u64 j5mins = 5LLU * 60 * HZ, j10mins = 10LLU * 60 * HZ;
 drivers/ata/libata-eh.c:1824:	u64 j64 = get_jiffies_64();
 drivers/ata/libata-eh.c-1825-	struct speed_down_verdict_arg arg;
 --
 drivers/ata/libata-eh.c=3656=static int ata_count_probe_trials_cb(struct ata_ering_entry *ent, void *void_arg)
 --
 drivers/ata/libata-eh.c-3658-	u64 interval = msecs_to_jiffies(ATA_EH_PROBE_TRIAL_INTERVAL);
 drivers/ata/libata-eh.c:3659:	u64 now = get_jiffies_64();
 drivers/ata/libata-eh.c-3660-	int *trials = void_arg;
 --
 drivers/cpufreq/cpufreq.c=133=static inline u64 get_cpu_idle_time_jiffy(unsigned int cpu, u64 *wall)
 --
 drivers/cpufreq/cpufreq.c-139-
 drivers/cpufreq/cpufreq.c:140:	cur_wall_time = jiffies64_to_nsecs(get_jiffies_64());
 drivers/cpufreq/cpufreq.c-141-
 --
 drivers/cpufreq/powernv-cpufreq.c=741=static int powernv_cpufreq_target_index(struct cpufreq_policy *policy,
 --
 drivers/cpufreq/powernv-cpufreq.c-759-
 drivers/cpufreq/powernv-cpufreq.c:760:	cur_msec = jiffies_to_msecs(get_jiffies_64());
 drivers/cpufreq/powernv-cpufreq.c-761-
 --
 drivers/devfreq/devfreq.c=213=int devfreq_update_status(struct devfreq *devfreq, unsigned long freq)
 --
 drivers/devfreq/devfreq.c-218-	lockdep_assert_held(&devfreq->lock);
 drivers/devfreq/devfreq.c:219:	cur_time = get_jiffies_64();
 drivers/devfreq/devfreq.c-220-
 --
 drivers/devfreq/devfreq.c=576=void devfreq_monitor_resume(struct devfreq *devfreq)
 --
 drivers/devfreq/devfreq.c-593-out_update:
 drivers/devfreq/devfreq.c:594:	devfreq->stats.last_update = get_jiffies_64();
 drivers/devfreq/devfreq.c-595-	devfreq->stop_polling = false;
 --
 drivers/devfreq/devfreq.c=800=struct devfreq *devfreq_add_device(struct device *dev,
 --
 drivers/devfreq/devfreq.c-911-	devfreq->stats.total_trans = 0;
 drivers/devfreq/devfreq.c:912:	devfreq->stats.last_update = get_jiffies_64();
 drivers/devfreq/devfreq.c-913-
 --
 drivers/devfreq/devfreq.c=1767=static ssize_t trans_stat_store(struct device *dev,
 --
 drivers/devfreq/devfreq.c-1790-	df->stats.total_trans = 0;
 drivers/devfreq/devfreq.c:1791:	df->stats.last_update = get_jiffies_64();
 drivers/devfreq/devfreq.c-1792-	mutex_unlock(&df->lock);
 --
 drivers/firewire/core-topology.c=459=void fw_core_handle_bus_reset(struct fw_card *card, int node_id, int generation,
 --
 drivers/firewire/core-topology.c-479-		card->generation = generation;
 drivers/firewire/core-topology.c:480:		card->reset_jiffies = get_jiffies_64();
 drivers/firewire/core-topology.c-481-		card->bm_node_id  = 0xffff;
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device.c=1231=int kgd2kfd_schedule_evict_and_restore_process(struct mm_struct *mm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device.c-1255-	 */
 drivers/gpu/drm/amd/amdkfd/kfd_device.c:1256:	active_time = get_jiffies_64() - p->last_restore_timestamp;
 drivers/gpu/drm/amd/amdkfd/kfd_device.c-1257-	if (delay_jiffies > active_time)
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1140=static int evict_process_queues_nocpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1155-
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1156:	pdd->last_evict_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1157-	/* Mark all queues as evicted. Deactivate all active queues on
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1190=static int evict_process_queues_cpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1214-	if (dqm->dev->kfd->shared_resources.enable_mes)
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1215:		pdd->last_evict_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1216-
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1238-	if (!dqm->dev->kfd->shared_resources.enable_mes) {
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1239:		pdd->last_evict_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1240-		retval = execute_queues_cpsch(dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1252=static int restore_process_queues_nocpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1323-	qpd->evicted = 0;
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1324:	eviction_duration = get_jiffies_64() - pdd->last_evict_timestamp;
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1325-	atomic64_add(eviction_duration, &pdd->evict_duration_counter);
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1333=static int restore_process_queues_cpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1386-					      KFD_UNMAP_QUEUES_FILTER_DYNAMIC_QUEUES, 0, USE_DEFAULT_GRACE_PERIOD);
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1387:	eviction_duration = get_jiffies_64() - pdd->last_evict_timestamp;
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1388-	atomic64_add(eviction_duration, &pdd->evict_duration_counter);
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c=1589=struct kfd_process *create_process(const struct task_struct *thread, bool primary)
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-1607-	INIT_DELAYED_WORK(&process->restore_work, restore_process_worker);
 drivers/gpu/drm/amd/amdkfd/kfd_process.c:1608:	process->last_restore_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-1609-	err = kfd_event_init_process(process);
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c=2161=static void restore_process_worker(struct work_struct *work)
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-2184-
 drivers/gpu/drm/amd/amdkfd/kfd_process.c:2185:	p->last_restore_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-2186-
 --
 drivers/gpu/drm/drm_pagemap.c=471=int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
 --
 drivers/gpu/drm/drm_pagemap.c-651-	/* Upon success bind devmem allocation to range and zdd */
 drivers/gpu/drm/drm_pagemap.c:652:	devmem_allocation->timeslice_expiration = get_jiffies_64() +
 drivers/gpu/drm/drm_pagemap.c-653-		msecs_to_jiffies(mdetails->timeslice_ms);
 --
 drivers/gpu/drm/drm_pagemap.c=1047=static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 --
 drivers/gpu/drm/drm_pagemap.c-1070-	zdd = page->zone_device_data;
 drivers/gpu/drm/drm_pagemap.c:1071:	if (time_before64(get_jiffies_64(), zdd->devmem_allocation->timeslice_expiration))
 drivers/gpu/drm/drm_pagemap.c-1072-		return 0;
 --
 drivers/gpu/drm/i915/i915_hwmon.c=671=hwm_fan_input_read(struct hwm_drvdata *ddat, long *val)
 --
 drivers/gpu/drm/i915/i915_hwmon.c-683-	reg_val = intel_uncore_read(ddat->uncore, hwmon->rg.fan_speed);
 drivers/gpu/drm/i915/i915_hwmon.c:684:	time_now = get_jiffies_64();
 drivers/gpu/drm/i915/i915_hwmon.c-685-
 --
 drivers/gpu/drm/i915/i915_hwmon.c=836=hwm_get_preregistration_info(struct drm_i915_private *i915)
 --
 drivers/gpu/drm/i915/i915_hwmon.c-883-								  hwmon->rg.fan_speed);
 drivers/gpu/drm/i915/i915_hwmon.c:884:			ddat->fi.time_prev = get_jiffies_64();
 drivers/gpu/drm/i915/i915_hwmon.c-885-		}
 --
 drivers/gpu/drm/panthor/panthor_sched.c=2445=static void tick_work(struct work_struct *work)
 --
 drivers/gpu/drm/panthor/panthor_sched.c-2452-	u64 remaining_jiffies = 0, resched_delay;
 drivers/gpu/drm/panthor/panthor_sched.c:2453:	u64 now = get_jiffies_64();
 drivers/gpu/drm/panthor/panthor_sched.c-2454-	int prio, ret, cookie;
 --
 drivers/gpu/drm/panthor/panthor_sched.c=2650=static void sched_resume_tick(struct panthor_device *ptdev)
 --
 drivers/gpu/drm/panthor/panthor_sched.c-2659-	 */
 drivers/gpu/drm/panthor/panthor_sched.c:2660:	now = get_jiffies_64();
 drivers/gpu/drm/panthor/panthor_sched.c-2661-	sched->resched_target = sched->last_tick + sched->tick_period;
 --
 drivers/gpu/drm/radeon/radeon_ring.c=241=void radeon_ring_lockup_update(struct radeon_device *rdev,
 --
 drivers/gpu/drm/radeon/radeon_ring.c-244-	atomic_set(&ring->last_rptr, radeon_ring_get_rptr(rdev, ring));
 drivers/gpu/drm/radeon/radeon_ring.c:245:	atomic64_set(&ring->last_activity, jiffies_64);
 drivers/gpu/drm/radeon/radeon_ring.c-246-}
 --
 drivers/gpu/drm/radeon/radeon_ring.c=254=bool radeon_ring_test_lockup(struct radeon_device *rdev, struct radeon_ring *ring)
 --
 drivers/gpu/drm/radeon/radeon_ring.c-265-
 drivers/gpu/drm/radeon/radeon_ring.c:266:	elapsed = jiffies_to_msecs(jiffies_64 - last);
 drivers/gpu/drm/radeon/radeon_ring.c-267-	if (radeon_lockup_timeout && elapsed >= radeon_lockup_timeout) {
 --
 drivers/gpu/drm/xe/xe_hwmon.c=1287=xe_hwmon_fan_input_read(struct xe_hwmon *hwmon, int channel, long *val)
 --
 drivers/gpu/drm/xe/xe_hwmon.c-1297-	reg_val = xe_mmio_read32(mmio, xe_hwmon_get_reg(hwmon, REG_FAN_SPEED, channel));
 drivers/gpu/drm/xe/xe_hwmon.c:1298:	time_now = get_jiffies_64();
 drivers/gpu/drm/xe/xe_hwmon.c-1299-
 --
 drivers/i2c/busses/i2c-octeon-core.c=50=static int octeon_i2c_wait(struct octeon_i2c *i2c)
 --
 drivers/i2c/busses/i2c-octeon-core.c-58-	if (i2c->broken_irq_mode) {
 drivers/i2c/busses/i2c-octeon-core.c:59:		u64 end = get_jiffies_64() + i2c->adap.timeout;
 drivers/i2c/busses/i2c-octeon-core.c-60-
 drivers/i2c/busses/i2c-octeon-core.c-61-		while (!octeon_i2c_test_iflg(i2c) &&
 drivers/i2c/busses/i2c-octeon-core.c:62:		       time_before64(get_jiffies_64(), end))
 drivers/i2c/busses/i2c-octeon-core.c-63-			usleep_range(I2C_OCTEON_EVENT_WAIT / 2, I2C_OCTEON_EVENT_WAIT);
 --
 drivers/i2c/busses/i2c-octeon-core.c=170=static int octeon_i2c_hlc_wait(struct octeon_i2c *i2c)
 --
 drivers/i2c/busses/i2c-octeon-core.c-178-	if (i2c->broken_irq_mode) {
 drivers/i2c/busses/i2c-octeon-core.c:179:		u64 end = get_jiffies_64() + i2c->adap.timeout;
 drivers/i2c/busses/i2c-octeon-core.c-180-
 drivers/i2c/busses/i2c-octeon-core.c-181-		while (!octeon_i2c_hlc_test_valid(i2c) &&
 drivers/i2c/busses/i2c-octeon-core.c:182:		       time_before64(get_jiffies_64(), end))
 drivers/i2c/busses/i2c-octeon-core.c-183-			usleep_range(I2C_OCTEON_EVENT_WAIT / 2, I2C_OCTEON_EVENT_WAIT);
 --
 drivers/macintosh/rack-meter.c=212=static void rackmeter_do_timer(struct work_struct *work)
 --
 drivers/macintosh/rack-meter.c-221-
 drivers/macintosh/rack-meter.c:222:	cur_nsecs = jiffies64_to_nsecs(get_jiffies_64());
 drivers/macintosh/rack-meter.c-223-	total_nsecs = cur_nsecs - rcpu->prev_wall;
 --
 drivers/macintosh/rack-meter.c=257=static void rackmeter_init_cpu_sniffer(struct rackmeter *rm)
 --
 drivers/macintosh/rack-meter.c-278-		rcpu->prev_idle = get_cpu_idle_time(cpu);
 drivers/macintosh/rack-meter.c:279:		rcpu->prev_wall = jiffies64_to_nsecs(get_jiffies_64());
 drivers/macintosh/rack-meter.c-280-		schedule_delayed_work_on(cpu, &rm->cpu[cpu].sniffer,
 --
 drivers/mailbox/imx-mailbox.c=160=static int imx_mu_tx_waiting_write(struct imx_mu_priv *priv, u32 val, u32 idx)
 drivers/mailbox/imx-mailbox.c-161-{
 drivers/mailbox/imx-mailbox.c:162:	u64 timeout_time = get_jiffies_64() + IMX_MU_SECO_TX_TOUT;
 drivers/mailbox/imx-mailbox.c-163-	u32 status;
 --
 drivers/mailbox/imx-mailbox.c=184=static int imx_mu_rx_waiting_read(struct imx_mu_priv *priv, u32 *val, u32 idx)
 drivers/mailbox/imx-mailbox.c-185-{
 drivers/mailbox/imx-mailbox.c:186:	u64 timeout_time = get_jiffies_64() + IMX_MU_SECO_RX_TOUT;
 drivers/mailbox/imx-mailbox.c-187-	u32 status;
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c=115=static void vidtv_mux_update_clk(struct vidtv_mux *m)
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c-120-	m->timing.past_jiffies = m->timing.current_jiffies;
 drivers/media/test-drivers/vidtv/vidtv_mux.c:121:	m->timing.current_jiffies = get_jiffies_64();
 drivers/media/test-drivers/vidtv/vidtv_mux.c-122-
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c=457=void vidtv_mux_start_thread(struct vidtv_mux *m)
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c-464-	m->streaming = true;
 drivers/media/test-drivers/vidtv/vidtv_mux.c:465:	m->timing.start_jiffies = get_jiffies_64();
 drivers/media/test-drivers/vidtv/vidtv_mux.c-466-	schedule_work(&m->mpeg_thread);
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c=477=struct vidtv_mux *vidtv_mux_init(struct dvb_frontend *fe,
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c-509-
 drivers/media/test-drivers/vidtv/vidtv_mux.c:510:	m->timing.current_jiffies = get_jiffies_64();
 drivers/media/test-drivers/vidtv/vidtv_mux.c-511-
 --
 drivers/media/test-drivers/visl/visl-dec.c=270=static void visl_tpg_fill_sequence(struct visl_ctx *ctx,
 --
 drivers/media/test-drivers/visl/visl-dec.c-276-	if (tpg_verbose) {
 drivers/media/test-drivers/visl/visl-dec.c:277:		stream_ms = jiffies_to_msecs(get_jiffies_64() - ctx->capture_streamon_jiffies);
 drivers/media/test-drivers/visl/visl-dec.c-278-
 --
 drivers/media/test-drivers/visl/visl-video.c=672=static int visl_start_streaming(struct vb2_queue *vq, unsigned int count)
 --
 drivers/media/test-drivers/visl/visl-video.c-685-	if (V4L2_TYPE_IS_CAPTURE(vq->type)) {
 drivers/media/test-drivers/visl/visl-video.c:686:		ctx->capture_streamon_jiffies = get_jiffies_64();
 drivers/media/test-drivers/visl/visl-video.c-687-		return 0;
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c=182=ice_arfs_is_flow_expired(struct ice_vsi *vsi, struct ice_arfs_entry *arfs_entry)
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c-196-			       ICE_ARFS_TIME_DELTA_EXPIRATION,
 drivers/net/ethernet/intel/ice/ice_arfs.c:197:			       arfs_entry->time_activated, get_jiffies_64());
 drivers/net/ethernet/intel/ice/ice_arfs.c-198-}
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c=215=ice_arfs_update_flow_rules(struct ice_vsi *vsi, u16 idx,
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c-242-			    flow_type == ICE_FLTR_PTYPE_NONF_IPV6_UDP)
 drivers/net/ethernet/intel/ice/ice_arfs.c:243:				e->time_activated = get_jiffies_64();
 drivers/net/ethernet/intel/ice/ice_arfs.c-244-		} else if (e->fltr_state == ICE_ARFS_ACTIVE) {
 --
 drivers/net/ethernet/sun/niu.c=4975=static int niu_init_rx_channels(struct niu *np)
 --
 drivers/net/ethernet/sun/niu.c-4977-	unsigned long flags;
 drivers/net/ethernet/sun/niu.c:4978:	u64 seed = jiffies_64;
 drivers/net/ethernet/sun/niu.c-4979-	int err, i;
 --
 drivers/net/phy/mxl-gpy.c=842=static int gpy_loopback(struct phy_device *phydev, bool enable, int speed)
 --
 drivers/net/phy/mxl-gpy.c-848-	if (enable) {
 drivers/net/phy/mxl-gpy.c:849:		u64 now = get_jiffies_64();
 drivers/net/phy/mxl-gpy.c-850-
 --
 drivers/net/phy/mxl-gpy.c-870-	} else {
 drivers/net/phy/mxl-gpy.c:871:		priv->lb_dis_to = get_jiffies_64() + HZ * 3;
 drivers/net/phy/mxl-gpy.c-872-	}
 --
 drivers/net/wireless/ath/ath10k/debug.c=866=static int ath10k_debug_htt_stats_req(struct ath10k *ar)
 --
 drivers/net/wireless/ath/ath10k/debug.c-879-
 drivers/net/wireless/ath/ath10k/debug.c:880:	cookie = get_jiffies_64();
 drivers/net/wireless/ath/ath10k/debug.c-881-
 --
 drivers/net/xen-netback/interface.c=561=int xenvif_init_queue(struct xenvif_queue *queue)
 --
 drivers/net/xen-netback/interface.c-567-	timer_setup(&queue->credit_timeout, xenvif_tx_credit_callback, 0);
 drivers/net/xen-netback/interface.c:568:	queue->credit_window_start = get_jiffies_64();
 drivers/net/xen-netback/interface.c-569-
 --
 drivers/net/xen-netback/netback.c=811=static bool tx_credit_exceeded(struct xenvif_queue *queue, unsigned size)
 drivers/net/xen-netback/netback.c-812-{
 drivers/net/xen-netback/netback.c:813:	u64 now = get_jiffies_64();
 drivers/net/xen-netback/netback.c-814-	u64 next_credit = queue->credit_window_start +
 --
 drivers/platform/x86/intel_ips.c=954=static int ips_monitor(void *data)
 --
 drivers/platform/x86/intel_ips.c-983-		ITV_ME_SEQNO_SHIFT;
 drivers/platform/x86/intel_ips.c:984:	seqno_timestamp = get_jiffies_64();
 drivers/platform/x86/intel_ips.c-985-
 --
 drivers/platform/x86/intel_ips.c-1093-		} else {
 drivers/platform/x86/intel_ips.c:1094:			seqno_timestamp = get_jiffies_64();
 drivers/platform/x86/intel_ips.c-1095-			last_seqno = cur_seqno;
]
]

gemini-3-flash-preview

							input: 46697
							output: 47
							thoughts: 31
						

Okay, here's my take on those thoughts, framed as an expert's internal monologue:

**Jiffies Investigation**

Right, so I'm seeing this trend, almost everywhere else in the kernel seems to be using `get_jiffies_64()`. That's a strong indicator. It means something's likely amiss if we're not using it in this particular area. Time for a focused re-examination. Let's head back and take another, more thorough look at `mm/memcontrol.c`. Something there isn't quite clicking, and based on the broader context I know, the jiffies implementation is likely at the crux of the issue. Gotta drill down into that file again.

map[ContextFile:mm/memcontrol.c IncludeLines:true Name:mem_cgroup_flush_stats_ratelimited]

map[SourceCode: 635:	void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
 636:	{
 637:		/* Only flush if the periodic flusher is one full cycle late */
 638:		if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
 639:			mem_cgroup_flush_stats(memcg);
 640:	}
 SourceFile:mm/memcontrol.c]

gemini-3-flash-preview

							input: 47086
							output: 27
							thoughts: 2791
						

**Analysis of jiffies_64 Access in mem_cgroup_flush_stats_ratelimited**

Alright, let's break this down. My initial observation is that the code in question, `mm/memcontrol.c:638`, directly reads `jiffies_64` without using `get_jiffies_64()` or `READ_ONCE()`. This immediately raises a red flag regarding potential data races, especially on 32-bit systems.

I see this pattern elsewhere too, such as `time_after64` comparisons in `arch/s390/kernel/vtime.c` and `arch/x86/kernel/tsc.c`, indicating it's a generally observed practice, which requires further investigation.

On a 64-bit system, the direct read of `jiffies_64` is *technically* atomic. However, even on 64-bit, the compiler could introduce optimizations that make the read non-atomic from a concurrency perspective. Specifically, it could cache the value across multiple usages, or in rare circumstances, perform a split 32-bit read. This isn't likely with a simple `u64` read on x86_64, but the kernel standard is to use `get_jiffies_64()` or `READ_ONCE()` for a reason. `get_jiffies_64()` has the benefit of handling 32-bit systems correctly.

The real problem lies on 32-bit systems. The direct read of `jiffies_64` is definitely *not* atomic there. The code in `mem_cgroup_flush_stats_ratelimited` is called in systems that support 32-bit, so it's a valid concern. This means the read can be torn, resulting in potentially incorrect values of `jiffies_64`, which could lead to missed or spurious flushes of memory cgroup statistics.

The torn read scenario is most concerning. Imagine if the lower 32 bits wrap around while being read. You could end up with a huge or tiny value of `jiffies_64`. This can dramatically skew the result of the `time_after64` check, potentially causing the rate limiting to behave incorrectly.

Given `time_after64` uses signed subtraction, the tearing has a huge impact. Consider an example where jiffies is 0xffffffff before it rolls over. If a read sees 0x00000000ffffffff and the next read the lower 32 bits have already rolled over to 0x0000000000000000, then the value could incorrectly read as either very large or very small. In the case where the jiffies have already been rolled over to a value of 0x0000000100000000, the result would give an almost infinite future when compared to the value of jiffies after it had been read, and the value would incorrectly become true, whereas if it were a roll over value of 0x0000000000000000, and it hadn't wrapped, then the value of the jiffies value would have been read as very small, and the check would have incorrectly passed.

In the context of a ratelimit, this is probably not catastrophic; it might lead to a few extra or missed flushes in the grand scheme of things. But still, it's a concurrency bug.

On 64-bit systems, the race is still present, albeit less severe. The KCSAN report is correct, the access is not properly synchronized.

The kernel *should* enforce using `get_jiffies_64()` or `READ_ONCE()` where applicable. Looking at the code for `get_jiffies_64()` itself, it's clear why. On 32-bit systems, it ensures atomicity via sequence locking.

The fact that `mem_cgroup_flush_stats_ratelimited` uses `READ_ONCE(flush_last_time)` but not for `jiffies_64` is an inconsistency that needs addressing.

So, is this "benign"? "Benign" as in "doesn't crash the kernel"? Probably. But it's still a violation of kernel concurrency guidelines and could lead to observable, albeit minor, performance deviations. Given that this code path can be called during memory reclaim, even minor performance deviations have the potential to compound. The bug should be fixed.

I should verify if this has been addressed in recent commits, but my tooling doesn't immediately lend itself to that

no matches

map[Expression:jiffies_64.*mm/memcontrol.c]

map[Output:]

gemini-3-flash-preview

							input: 50943
							output: 20
							thoughts: 28
						

Okay, here's my interpretation of that thought process, tailored for an expert audience, and rendered in the first person:

**Navigating the Memory Control Codebase**

Hmm, I think I'm on the right track, but I need to refine my search terms a bit. I was initially looking for `jiffies_64`, which is common, but it might be more effective to ditch the explicit dot notation. I'll switch my search to just `jiffies_64`, without the period, and specifically target `mm/memcontrol.c`. That file is a likely candidate given the memory control context I'm working in, and the shorter search term might reveal something I missed in my initial pass or might be more flexible. This should help me find the specific usage or definition I'm after, avoiding irrelevant hits in other modules. Let's see what that yields.

map[Expression:jiffies_64]

map[Output:
Full output is too long, showing 500 out of 1048 lines.
Use more precise expression if possible.

[arch/alpha/kernel/vmlinux.lds.S=15=PHDRS { text PT_LOAD; note PT_NOTE; }
 arch/alpha/kernel/vmlinux.lds.S:16:jiffies = jiffies_64;
 arch/alpha/kernel/vmlinux.lds.S-17-SECTIONS
 --
 arch/arc/kernel/vmlinux.lds.S=12=ENTRY(res_service)
 --
 arch/arc/kernel/vmlinux.lds.S-14-#ifdef CONFIG_CPU_BIG_ENDIAN
 arch/arc/kernel/vmlinux.lds.S:15:jiffies = jiffies_64 + 4;
 arch/arc/kernel/vmlinux.lds.S-16-#else
 arch/arc/kernel/vmlinux.lds.S:17:jiffies = jiffies_64;
 arch/arc/kernel/vmlinux.lds.S-18-#endif
 --
 arch/arm/kernel/smp_twd.c=140=static void twd_calibrate_rate(void)
 --
 arch/arm/kernel/smp_twd.c-152-		/* Wait for a tick to start */
 arch/arm/kernel/smp_twd.c:153:		waitjiffies = get_jiffies_64() + 1;
 arch/arm/kernel/smp_twd.c-154-
 arch/arm/kernel/smp_twd.c:155:		while (get_jiffies_64() < waitjiffies)
 arch/arm/kernel/smp_twd.c-156-			udelay(10);
 --
 arch/arm/kernel/smp_twd.c-166-
 arch/arm/kernel/smp_twd.c:167:		while (get_jiffies_64() < waitjiffies)
 arch/arm/kernel/smp_twd.c-168-			udelay(10);
 --
 arch/arm/kernel/vmlinux-xip.lds.S=19=ENTRY(stext)
 --
 arch/arm/kernel/vmlinux-xip.lds.S-21-#ifndef __ARMEB__
 arch/arm/kernel/vmlinux-xip.lds.S:22:jiffies = jiffies_64;
 arch/arm/kernel/vmlinux-xip.lds.S-23-#else
 arch/arm/kernel/vmlinux-xip.lds.S:24:jiffies = jiffies_64 + 4;
 arch/arm/kernel/vmlinux-xip.lds.S-25-#endif
 --
 arch/arm/kernel/vmlinux.lds.S=19=ENTRY(stext)
 --
 arch/arm/kernel/vmlinux.lds.S-21-#ifndef __ARMEB__
 arch/arm/kernel/vmlinux.lds.S:22:jiffies = jiffies_64;
 arch/arm/kernel/vmlinux.lds.S-23-#else
 arch/arm/kernel/vmlinux.lds.S:24:jiffies = jiffies_64 + 4;
 arch/arm/kernel/vmlinux.lds.S-25-#endif
 --
 arch/arm64/kernel/vmlinux.lds.S=83=ENTRY(_text)
 arch/arm64/kernel/vmlinux.lds.S-84-
 arch/arm64/kernel/vmlinux.lds.S:85:jiffies = jiffies_64;
 arch/arm64/kernel/vmlinux.lds.S-86-
 --
 arch/csky/kernel/vmlinux.lds.S=8=ENTRY(_start)
 --
 arch/csky/kernel/vmlinux.lds.S-10-#ifndef __cskyBE__
 arch/csky/kernel/vmlinux.lds.S:11:jiffies = jiffies_64;
 arch/csky/kernel/vmlinux.lds.S-12-#else
 arch/csky/kernel/vmlinux.lds.S:13:jiffies = jiffies_64 + 4;
 arch/csky/kernel/vmlinux.lds.S-14-#endif
 --
 arch/hexagon/kernel/vmlinux.lds.S=15=ENTRY(stext)
 arch/hexagon/kernel/vmlinux.lds.S-16-
 arch/hexagon/kernel/vmlinux.lds.S:17:jiffies = jiffies_64;
 arch/hexagon/kernel/vmlinux.lds.S-18-
 --
 arch/loongarch/kernel/vmlinux.lds.S=29=PHDRS {
 --
 arch/loongarch/kernel/vmlinux.lds.S-33-
 arch/loongarch/kernel/vmlinux.lds.S:34:jiffies	 = jiffies_64;
 arch/loongarch/kernel/vmlinux.lds.S-35-
 --
 arch/m68k/kernel/vmlinux-nommu.lds=25=ENTRY(_start)
 arch/m68k/kernel/vmlinux-nommu.lds-26-
 arch/m68k/kernel/vmlinux-nommu.lds:27:jiffies = jiffies_64 + 4;
 arch/m68k/kernel/vmlinux-nommu.lds-28-
 --
 arch/m68k/kernel/vmlinux-std.lds=10=ENTRY(_start)
 arch/m68k/kernel/vmlinux-std.lds:11:jiffies = jiffies_64 + 4;
 arch/m68k/kernel/vmlinux-std.lds-12-SECTIONS
 --
 arch/m68k/kernel/vmlinux-sun3.lds=10=ENTRY(_start)
 arch/m68k/kernel/vmlinux-sun3.lds:11:jiffies = jiffies_64 + 4;
 arch/m68k/kernel/vmlinux-sun3.lds-12-SECTIONS
 --
 arch/microblaze/kernel/vmlinux.lds.S=12=ENTRY(microblaze_start)
 --
 arch/microblaze/kernel/vmlinux.lds.S-21-#ifdef __MICROBLAZEEL__
 arch/microblaze/kernel/vmlinux.lds.S:22:jiffies = jiffies_64;
 arch/microblaze/kernel/vmlinux.lds.S-23-#else
 arch/microblaze/kernel/vmlinux.lds.S:24:jiffies = jiffies_64 + 4;
 arch/microblaze/kernel/vmlinux.lds.S-25-#endif
 --
 arch/mips/kernel/vmlinux.lds.S=26=PHDRS {
 --
 arch/mips/kernel/vmlinux.lds.S-34-	#ifdef CONFIG_CPU_LITTLE_ENDIAN
 arch/mips/kernel/vmlinux.lds.S:35:		jiffies	 = jiffies_64;
 arch/mips/kernel/vmlinux.lds.S-36-	#else
 arch/mips/kernel/vmlinux.lds.S:37:		jiffies	 = jiffies_64 + 4;
 arch/mips/kernel/vmlinux.lds.S-38-	#endif
 arch/mips/kernel/vmlinux.lds.S-39-#else
 arch/mips/kernel/vmlinux.lds.S:40:	jiffies	 = jiffies_64;
 arch/mips/kernel/vmlinux.lds.S-41-#endif
 --
 arch/nios2/kernel/vmlinux.lds.S=13=ENTRY(_start)	/* Defined in head.S */
 arch/nios2/kernel/vmlinux.lds.S-14-
 arch/nios2/kernel/vmlinux.lds.S:15:jiffies = jiffies_64;
 arch/nios2/kernel/vmlinux.lds.S-16-
 --
 arch/openrisc/kernel/vmlinux.lds.S=36=OUTPUT_FORMAT(__OUTPUT_FORMAT, __OUTPUT_FORMAT, __OUTPUT_FORMAT)
 arch/openrisc/kernel/vmlinux.lds.S:37:jiffies = jiffies_64 + 4;
 arch/openrisc/kernel/vmlinux.lds.S-38-
 --
 arch/parisc/kernel/vmlinux.lds.S=50=ENTRY(parisc_kernel_start)
 arch/parisc/kernel/vmlinux.lds.S-51-#ifndef CONFIG_64BIT
 arch/parisc/kernel/vmlinux.lds.S:52:jiffies = jiffies_64 + 4;
 arch/parisc/kernel/vmlinux.lds.S-53-#else
 arch/parisc/kernel/vmlinux.lds.S:54:jiffies = jiffies_64;
 arch/parisc/kernel/vmlinux.lds.S-55-#endif
 --
 arch/powerpc/kernel/vmlinux.lds.S=42=OUTPUT_ARCH(powerpc:common64)
 arch/powerpc/kernel/vmlinux.lds.S:43:jiffies = jiffies_64;
 arch/powerpc/kernel/vmlinux.lds.S-44-#else
 arch/powerpc/kernel/vmlinux.lds.S=45=OUTPUT_ARCH(powerpc:common)
 arch/powerpc/kernel/vmlinux.lds.S:46:jiffies = jiffies_64 + 4;
 arch/powerpc/kernel/vmlinux.lds.S-47-#endif
 --
 arch/riscv/kernel/vmlinux-xip.lds.S=20=ENTRY(_start)
 arch/riscv/kernel/vmlinux-xip.lds.S-21-
 arch/riscv/kernel/vmlinux-xip.lds.S:22:jiffies = jiffies_64;
 arch/riscv/kernel/vmlinux-xip.lds.S-23-
 --
 arch/riscv/kernel/vmlinux.lds.S=26=ENTRY(_start)
 arch/riscv/kernel/vmlinux.lds.S-27-
 arch/riscv/kernel/vmlinux.lds.S:28:jiffies = jiffies_64;
 arch/riscv/kernel/vmlinux.lds.S-29-
 --
 arch/s390/kernel/vmlinux.lds.S=31=ENTRY(startup_continue)
 arch/s390/kernel/vmlinux.lds.S:32:jiffies = jiffies_64;
 arch/s390/kernel/vmlinux.lds.S-33-
 --
 arch/s390/kernel/vtime.c=59=static void update_mt_scaling(void)
 --
 arch/s390/kernel/vtime.c-83-	}
 arch/s390/kernel/vtime.c:84:	__this_cpu_write(mt_scaling_jiffies, jiffies_64);
 arch/s390/kernel/vtime.c-85-}
 --
 arch/s390/kernel/vtime.c=118=static int do_account_vtime(struct task_struct *tsk)
 --
 arch/s390/kernel/vtime.c-140-	if (smp_cpu_mtid &&
 arch/s390/kernel/vtime.c:141:	    time_after64(jiffies_64, this_cpu_read(mt_scaling_jiffies)))
 arch/s390/kernel/vtime.c-142-		update_mt_scaling();
 --
 arch/sh/Makefile=106=ld_bfd			:= elf32-sh-linux
 arch/sh/Makefile:107:LDFLAGS_vmlinux		+= --defsym jiffies=jiffies_64 --oformat $(ld_bfd)
 arch/sh/Makefile-108-KBUILD_LDFLAGS		+= -EL
 --
 arch/sh/Makefile=110=ld_bfd			:= elf32-shbig-linux
 arch/sh/Makefile:111:LDFLAGS_vmlinux		+= --defsym jiffies=jiffies_64+4 --oformat $(ld_bfd)
 arch/sh/Makefile-112-KBUILD_LDFLAGS		+= -EB
 --
 arch/sparc/kernel/vmlinux.lds.S=27=ENTRY(_start)
 arch/sparc/kernel/vmlinux.lds.S:28:jiffies = jiffies_64 + 4;
 arch/sparc/kernel/vmlinux.lds.S-29-#else
 --
 arch/sparc/kernel/vmlinux.lds.S=33=ENTRY(_start)
 arch/sparc/kernel/vmlinux.lds.S:34:jiffies = jiffies_64;
 arch/sparc/kernel/vmlinux.lds.S-35-#endif
 --
 arch/um/kernel/dyn.lds.S=6=ENTRY(_start)
 arch/um/kernel/dyn.lds.S:7:jiffies = jiffies_64;
 arch/um/kernel/dyn.lds.S-8-
 --
 arch/um/kernel/uml.lds.S=7=ENTRY(_start)
 arch/um/kernel/uml.lds.S:8:jiffies = jiffies_64;
 arch/um/kernel/uml.lds.S-9-
 --
 arch/x86/kernel/tsc.c=237=noinstr u64 native_sched_clock(void)
 --
 arch/x86/kernel/tsc.c-255-	/* No locking but a rare wrong value is not a big deal: */
 arch/x86/kernel/tsc.c:256:	return (jiffies_64 - INITIAL_JIFFIES) * (1000000000 / HZ);
 arch/x86/kernel/tsc.c-257-}
 --
 arch/x86/kernel/vmlinux.lds.S=42=ENTRY(phys_startup_64)
 --
 arch/x86/kernel/vmlinux.lds.S-44-
 arch/x86/kernel/vmlinux.lds.S:45:jiffies = jiffies_64;
 arch/x86/kernel/vmlinux.lds.S-46-const_current_task = current_task;
 --
 arch/x86/kvm/mmu/mmu.c=7830=static bool kvm_nx_huge_page_recovery_worker(void *data)
 --
 arch/x86/kvm/mmu/mmu.c-7842-	remaining_time = kvm->arch.nx_huge_page_last + msecs_to_jiffies(period)
 arch/x86/kvm/mmu/mmu.c:7843:		- get_jiffies_64();
 arch/x86/kvm/mmu/mmu.c-7844-	if (remaining_time > 0) {
 --
 arch/x86/kvm/mmu/mmu.c-7852-		kvm_recover_nx_huge_pages(kvm, i);
 arch/x86/kvm/mmu/mmu.c:7853:	kvm->arch.nx_huge_page_last = get_jiffies_64();
 arch/x86/kvm/mmu/mmu.c-7854-	return true;
 --
 arch/x86/kvm/mmu/mmu.c=7857=static int kvm_mmu_start_lpage_recovery(struct once *once)
 --
 arch/x86/kvm/mmu/mmu.c-7862-
 arch/x86/kvm/mmu/mmu.c:7863:	kvm->arch.nx_huge_page_last = get_jiffies_64();
 arch/x86/kvm/mmu/mmu.c-7864-	nx_thread = vhost_task_create(kvm_nx_huge_page_recovery_worker,
 --
 arch/x86/tools/relocs.c=53=static const char * const	sym_regex_kernel[S_NSYMTYPES] = {
 --
 arch/x86/tools/relocs.c-90-	"__initramfs_start|"
 arch/x86/tools/relocs.c:91:	"(jiffies|jiffies_64)|"
 arch/x86/tools/relocs.c-92-#if ELF_BITS == 64
 --
 arch/xtensa/kernel/vmlinux.lds.S=27=ENTRY(_start)
 --
 arch/xtensa/kernel/vmlinux.lds.S-29-#ifdef __XTENSA_EB__
 arch/xtensa/kernel/vmlinux.lds.S:30:jiffies = jiffies_64 + 4;
 arch/xtensa/kernel/vmlinux.lds.S-31-#else
 arch/xtensa/kernel/vmlinux.lds.S:32:jiffies = jiffies_64;
 arch/xtensa/kernel/vmlinux.lds.S-33-#endif
 --
 drivers/acpi/osl.c=655=u64 acpi_os_get_timer(void)
 drivers/acpi/osl.c-656-{
 drivers/acpi/osl.c:657:	return (get_jiffies_64() - INITIAL_JIFFIES) *
 drivers/acpi/osl.c-658-		(ACPI_100NSEC_PER_SEC / HZ);
 --
 drivers/ata/libata-eh.c=370=static void ata_ering_record(struct ata_ering *ering, unsigned int eflags,
 --
 drivers/ata/libata-eh.c-382-	ent->err_mask = err_mask;
 drivers/ata/libata-eh.c:383:	ent->timestamp = get_jiffies_64();
 drivers/ata/libata-eh.c-384-}
 --
 drivers/ata/libata-eh.c=1821=static unsigned int ata_eh_speed_down_verdict(struct ata_device *dev)
 --
 drivers/ata/libata-eh.c-1823-	const u64 j5mins = 5LLU * 60 * HZ, j10mins = 10LLU * 60 * HZ;
 drivers/ata/libata-eh.c:1824:	u64 j64 = get_jiffies_64();
 drivers/ata/libata-eh.c-1825-	struct speed_down_verdict_arg arg;
 --
 drivers/ata/libata-eh.c=3656=static int ata_count_probe_trials_cb(struct ata_ering_entry *ent, void *void_arg)
 --
 drivers/ata/libata-eh.c-3658-	u64 interval = msecs_to_jiffies(ATA_EH_PROBE_TRIAL_INTERVAL);
 drivers/ata/libata-eh.c:3659:	u64 now = get_jiffies_64();
 drivers/ata/libata-eh.c-3660-	int *trials = void_arg;
 --
 drivers/cpufreq/cpufreq.c=133=static inline u64 get_cpu_idle_time_jiffy(unsigned int cpu, u64 *wall)
 --
 drivers/cpufreq/cpufreq.c-139-
 drivers/cpufreq/cpufreq.c:140:	cur_wall_time = jiffies64_to_nsecs(get_jiffies_64());
 drivers/cpufreq/cpufreq.c-141-
 --
 drivers/cpufreq/powernv-cpufreq.c=741=static int powernv_cpufreq_target_index(struct cpufreq_policy *policy,
 --
 drivers/cpufreq/powernv-cpufreq.c-759-
 drivers/cpufreq/powernv-cpufreq.c:760:	cur_msec = jiffies_to_msecs(get_jiffies_64());
 drivers/cpufreq/powernv-cpufreq.c-761-
 --
 drivers/devfreq/devfreq.c=213=int devfreq_update_status(struct devfreq *devfreq, unsigned long freq)
 --
 drivers/devfreq/devfreq.c-218-	lockdep_assert_held(&devfreq->lock);
 drivers/devfreq/devfreq.c:219:	cur_time = get_jiffies_64();
 drivers/devfreq/devfreq.c-220-
 --
 drivers/devfreq/devfreq.c=576=void devfreq_monitor_resume(struct devfreq *devfreq)
 --
 drivers/devfreq/devfreq.c-593-out_update:
 drivers/devfreq/devfreq.c:594:	devfreq->stats.last_update = get_jiffies_64();
 drivers/devfreq/devfreq.c-595-	devfreq->stop_polling = false;
 --
 drivers/devfreq/devfreq.c=800=struct devfreq *devfreq_add_device(struct device *dev,
 --
 drivers/devfreq/devfreq.c-911-	devfreq->stats.total_trans = 0;
 drivers/devfreq/devfreq.c:912:	devfreq->stats.last_update = get_jiffies_64();
 drivers/devfreq/devfreq.c-913-
 --
 drivers/devfreq/devfreq.c=1767=static ssize_t trans_stat_store(struct device *dev,
 --
 drivers/devfreq/devfreq.c-1790-	df->stats.total_trans = 0;
 drivers/devfreq/devfreq.c:1791:	df->stats.last_update = get_jiffies_64();
 drivers/devfreq/devfreq.c-1792-	mutex_unlock(&df->lock);
 --
 drivers/firewire/core-topology.c=459=void fw_core_handle_bus_reset(struct fw_card *card, int node_id, int generation,
 --
 drivers/firewire/core-topology.c-479-		card->generation = generation;
 drivers/firewire/core-topology.c:480:		card->reset_jiffies = get_jiffies_64();
 drivers/firewire/core-topology.c-481-		card->bm_node_id  = 0xffff;
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device.c=1231=int kgd2kfd_schedule_evict_and_restore_process(struct mm_struct *mm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device.c-1255-	 */
 drivers/gpu/drm/amd/amdkfd/kfd_device.c:1256:	active_time = get_jiffies_64() - p->last_restore_timestamp;
 drivers/gpu/drm/amd/amdkfd/kfd_device.c-1257-	if (delay_jiffies > active_time)
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1140=static int evict_process_queues_nocpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1155-
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1156:	pdd->last_evict_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1157-	/* Mark all queues as evicted. Deactivate all active queues on
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1190=static int evict_process_queues_cpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1214-	if (dqm->dev->kfd->shared_resources.enable_mes)
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1215:		pdd->last_evict_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1216-
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1238-	if (!dqm->dev->kfd->shared_resources.enable_mes) {
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1239:		pdd->last_evict_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1240-		retval = execute_queues_cpsch(dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1252=static int restore_process_queues_nocpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1323-	qpd->evicted = 0;
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1324:	eviction_duration = get_jiffies_64() - pdd->last_evict_timestamp;
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1325-	atomic64_add(eviction_duration, &pdd->evict_duration_counter);
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1333=static int restore_process_queues_cpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1386-					      KFD_UNMAP_QUEUES_FILTER_DYNAMIC_QUEUES, 0, USE_DEFAULT_GRACE_PERIOD);
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1387:	eviction_duration = get_jiffies_64() - pdd->last_evict_timestamp;
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1388-	atomic64_add(eviction_duration, &pdd->evict_duration_counter);
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c=1589=struct kfd_process *create_process(const struct task_struct *thread, bool primary)
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-1607-	INIT_DELAYED_WORK(&process->restore_work, restore_process_worker);
 drivers/gpu/drm/amd/amdkfd/kfd_process.c:1608:	process->last_restore_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-1609-	err = kfd_event_init_process(process);
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c=2161=static void restore_process_worker(struct work_struct *work)
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-2184-
 drivers/gpu/drm/amd/amdkfd/kfd_process.c:2185:	p->last_restore_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-2186-
 --
 drivers/gpu/drm/drm_pagemap.c=471=int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
 --
 drivers/gpu/drm/drm_pagemap.c-651-	/* Upon success bind devmem allocation to range and zdd */
 drivers/gpu/drm/drm_pagemap.c:652:	devmem_allocation->timeslice_expiration = get_jiffies_64() +
 drivers/gpu/drm/drm_pagemap.c-653-		msecs_to_jiffies(mdetails->timeslice_ms);
 --
 drivers/gpu/drm/drm_pagemap.c=1047=static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 --
 drivers/gpu/drm/drm_pagemap.c-1070-	zdd = page->zone_device_data;
 drivers/gpu/drm/drm_pagemap.c:1071:	if (time_before64(get_jiffies_64(), zdd->devmem_allocation->timeslice_expiration))
 drivers/gpu/drm/drm_pagemap.c-1072-		return 0;
 --
 drivers/gpu/drm/i915/i915_hwmon.c=671=hwm_fan_input_read(struct hwm_drvdata *ddat, long *val)
 --
 drivers/gpu/drm/i915/i915_hwmon.c-683-	reg_val = intel_uncore_read(ddat->uncore, hwmon->rg.fan_speed);
 drivers/gpu/drm/i915/i915_hwmon.c:684:	time_now = get_jiffies_64();
 drivers/gpu/drm/i915/i915_hwmon.c-685-
 --
 drivers/gpu/drm/i915/i915_hwmon.c=836=hwm_get_preregistration_info(struct drm_i915_private *i915)
 --
 drivers/gpu/drm/i915/i915_hwmon.c-883-								  hwmon->rg.fan_speed);
 drivers/gpu/drm/i915/i915_hwmon.c:884:			ddat->fi.time_prev = get_jiffies_64();
 drivers/gpu/drm/i915/i915_hwmon.c-885-		}
 --
 drivers/gpu/drm/panthor/panthor_sched.c=2445=static void tick_work(struct work_struct *work)
 --
 drivers/gpu/drm/panthor/panthor_sched.c-2452-	u64 remaining_jiffies = 0, resched_delay;
 drivers/gpu/drm/panthor/panthor_sched.c:2453:	u64 now = get_jiffies_64();
 drivers/gpu/drm/panthor/panthor_sched.c-2454-	int prio, ret, cookie;
 --
 drivers/gpu/drm/panthor/panthor_sched.c=2650=static void sched_resume_tick(struct panthor_device *ptdev)
 --
 drivers/gpu/drm/panthor/panthor_sched.c-2659-	 */
 drivers/gpu/drm/panthor/panthor_sched.c:2660:	now = get_jiffies_64();
 drivers/gpu/drm/panthor/panthor_sched.c-2661-	sched->resched_target = sched->last_tick + sched->tick_period;
 --
 drivers/gpu/drm/radeon/radeon_ring.c=241=void radeon_ring_lockup_update(struct radeon_device *rdev,
 --
 drivers/gpu/drm/radeon/radeon_ring.c-244-	atomic_set(&ring->last_rptr, radeon_ring_get_rptr(rdev, ring));
 drivers/gpu/drm/radeon/radeon_ring.c:245:	atomic64_set(&ring->last_activity, jiffies_64);
 drivers/gpu/drm/radeon/radeon_ring.c-246-}
 --
 drivers/gpu/drm/radeon/radeon_ring.c=254=bool radeon_ring_test_lockup(struct radeon_device *rdev, struct radeon_ring *ring)
 --
 drivers/gpu/drm/radeon/radeon_ring.c-265-
 drivers/gpu/drm/radeon/radeon_ring.c:266:	elapsed = jiffies_to_msecs(jiffies_64 - last);
 drivers/gpu/drm/radeon/radeon_ring.c-267-	if (radeon_lockup_timeout && elapsed >= radeon_lockup_timeout) {
 --
 drivers/gpu/drm/xe/xe_hwmon.c=1287=xe_hwmon_fan_input_read(struct xe_hwmon *hwmon, int channel, long *val)
 --
 drivers/gpu/drm/xe/xe_hwmon.c-1297-	reg_val = xe_mmio_read32(mmio, xe_hwmon_get_reg(hwmon, REG_FAN_SPEED, channel));
 drivers/gpu/drm/xe/xe_hwmon.c:1298:	time_now = get_jiffies_64();
 drivers/gpu/drm/xe/xe_hwmon.c-1299-
 --
 drivers/i2c/busses/i2c-octeon-core.c=50=static int octeon_i2c_wait(struct octeon_i2c *i2c)
 --
 drivers/i2c/busses/i2c-octeon-core.c-58-	if (i2c->broken_irq_mode) {
 drivers/i2c/busses/i2c-octeon-core.c:59:		u64 end = get_jiffies_64() + i2c->adap.timeout;
 drivers/i2c/busses/i2c-octeon-core.c-60-
 drivers/i2c/busses/i2c-octeon-core.c-61-		while (!octeon_i2c_test_iflg(i2c) &&
 drivers/i2c/busses/i2c-octeon-core.c:62:		       time_before64(get_jiffies_64(), end))
 drivers/i2c/busses/i2c-octeon-core.c-63-			usleep_range(I2C_OCTEON_EVENT_WAIT / 2, I2C_OCTEON_EVENT_WAIT);
 --
 drivers/i2c/busses/i2c-octeon-core.c=170=static int octeon_i2c_hlc_wait(struct octeon_i2c *i2c)
 --
 drivers/i2c/busses/i2c-octeon-core.c-178-	if (i2c->broken_irq_mode) {
 drivers/i2c/busses/i2c-octeon-core.c:179:		u64 end = get_jiffies_64() + i2c->adap.timeout;
 drivers/i2c/busses/i2c-octeon-core.c-180-
 drivers/i2c/busses/i2c-octeon-core.c-181-		while (!octeon_i2c_hlc_test_valid(i2c) &&
 drivers/i2c/busses/i2c-octeon-core.c:182:		       time_before64(get_jiffies_64(), end))
 drivers/i2c/busses/i2c-octeon-core.c-183-			usleep_range(I2C_OCTEON_EVENT_WAIT / 2, I2C_OCTEON_EVENT_WAIT);
 --
 drivers/macintosh/rack-meter.c=212=static void rackmeter_do_timer(struct work_struct *work)
 --
 drivers/macintosh/rack-meter.c-221-
 drivers/macintosh/rack-meter.c:222:	cur_nsecs = jiffies64_to_nsecs(get_jiffies_64());
 drivers/macintosh/rack-meter.c-223-	total_nsecs = cur_nsecs - rcpu->prev_wall;
 --
 drivers/macintosh/rack-meter.c=257=static void rackmeter_init_cpu_sniffer(struct rackmeter *rm)
 --
 drivers/macintosh/rack-meter.c-278-		rcpu->prev_idle = get_cpu_idle_time(cpu);
 drivers/macintosh/rack-meter.c:279:		rcpu->prev_wall = jiffies64_to_nsecs(get_jiffies_64());
 drivers/macintosh/rack-meter.c-280-		schedule_delayed_work_on(cpu, &rm->cpu[cpu].sniffer,
 --
 drivers/mailbox/imx-mailbox.c=160=static int imx_mu_tx_waiting_write(struct imx_mu_priv *priv, u32 val, u32 idx)
 drivers/mailbox/imx-mailbox.c-161-{
 drivers/mailbox/imx-mailbox.c:162:	u64 timeout_time = get_jiffies_64() + IMX_MU_SECO_TX_TOUT;
 drivers/mailbox/imx-mailbox.c-163-	u32 status;
 --
 drivers/mailbox/imx-mailbox.c=184=static int imx_mu_rx_waiting_read(struct imx_mu_priv *priv, u32 *val, u32 idx)
 drivers/mailbox/imx-mailbox.c-185-{
 drivers/mailbox/imx-mailbox.c:186:	u64 timeout_time = get_jiffies_64() + IMX_MU_SECO_RX_TOUT;
 drivers/mailbox/imx-mailbox.c-187-	u32 status;
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c=115=static void vidtv_mux_update_clk(struct vidtv_mux *m)
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c-120-	m->timing.past_jiffies = m->timing.current_jiffies;
 drivers/media/test-drivers/vidtv/vidtv_mux.c:121:	m->timing.current_jiffies = get_jiffies_64();
 drivers/media/test-drivers/vidtv/vidtv_mux.c-122-
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c=457=void vidtv_mux_start_thread(struct vidtv_mux *m)
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c-464-	m->streaming = true;
 drivers/media/test-drivers/vidtv/vidtv_mux.c:465:	m->timing.start_jiffies = get_jiffies_64();
 drivers/media/test-drivers/vidtv/vidtv_mux.c-466-	schedule_work(&m->mpeg_thread);
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c=477=struct vidtv_mux *vidtv_mux_init(struct dvb_frontend *fe,
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c-509-
 drivers/media/test-drivers/vidtv/vidtv_mux.c:510:	m->timing.current_jiffies = get_jiffies_64();
 drivers/media/test-drivers/vidtv/vidtv_mux.c-511-
 --
 drivers/media/test-drivers/visl/visl-dec.c=270=static void visl_tpg_fill_sequence(struct visl_ctx *ctx,
 --
 drivers/media/test-drivers/visl/visl-dec.c-276-	if (tpg_verbose) {
 drivers/media/test-drivers/visl/visl-dec.c:277:		stream_ms = jiffies_to_msecs(get_jiffies_64() - ctx->capture_streamon_jiffies);
 drivers/media/test-drivers/visl/visl-dec.c-278-
 --
 drivers/media/test-drivers/visl/visl-video.c=672=static int visl_start_streaming(struct vb2_queue *vq, unsigned int count)
 --
 drivers/media/test-drivers/visl/visl-video.c-685-	if (V4L2_TYPE_IS_CAPTURE(vq->type)) {
 drivers/media/test-drivers/visl/visl-video.c:686:		ctx->capture_streamon_jiffies = get_jiffies_64();
 drivers/media/test-drivers/visl/visl-video.c-687-		return 0;
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c=182=ice_arfs_is_flow_expired(struct ice_vsi *vsi, struct ice_arfs_entry *arfs_entry)
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c-196-			       ICE_ARFS_TIME_DELTA_EXPIRATION,
 drivers/net/ethernet/intel/ice/ice_arfs.c:197:			       arfs_entry->time_activated, get_jiffies_64());
 drivers/net/ethernet/intel/ice/ice_arfs.c-198-}
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c=215=ice_arfs_update_flow_rules(struct ice_vsi *vsi, u16 idx,
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c-242-			    flow_type == ICE_FLTR_PTYPE_NONF_IPV6_UDP)
 drivers/net/ethernet/intel/ice/ice_arfs.c:243:				e->time_activated = get_jiffies_64();
 drivers/net/ethernet/intel/ice/ice_arfs.c-244-		} else if (e->fltr_state == ICE_ARFS_ACTIVE) {
 --
 drivers/net/ethernet/sun/niu.c=4975=static int niu_init_rx_channels(struct niu *np)
 --
 drivers/net/ethernet/sun/niu.c-4977-	unsigned long flags;
 drivers/net/ethernet/sun/niu.c:4978:	u64 seed = jiffies_64;
 drivers/net/ethernet/sun/niu.c-4979-	int err, i;
 --
 drivers/net/phy/mxl-gpy.c=842=static int gpy_loopback(struct phy_device *phydev, bool enable, int speed)
 --
 drivers/net/phy/mxl-gpy.c-848-	if (enable) {
 drivers/net/phy/mxl-gpy.c:849:		u64 now = get_jiffies_64();
 drivers/net/phy/mxl-gpy.c-850-
 --
 drivers/net/phy/mxl-gpy.c-870-	} else {
 drivers/net/phy/mxl-gpy.c:871:		priv->lb_dis_to = get_jiffies_64() + HZ * 3;
 drivers/net/phy/mxl-gpy.c-872-	}
 --
 drivers/net/wireless/ath/ath10k/debug.c=866=static int ath10k_debug_htt_stats_req(struct ath10k *ar)
 --
 drivers/net/wireless/ath/ath10k/debug.c-879-
 drivers/net/wireless/ath/ath10k/debug.c:880:	cookie = get_jiffies_64();
 drivers/net/wireless/ath/ath10k/debug.c-881-
 --
 drivers/net/xen-netback/interface.c=561=int xenvif_init_queue(struct xenvif_queue *queue)
 --
 drivers/net/xen-netback/interface.c-567-	timer_setup(&queue->credit_timeout, xenvif_tx_credit_callback, 0);
 drivers/net/xen-netback/interface.c:568:	queue->credit_window_start = get_jiffies_64();
 drivers/net/xen-netback/interface.c-569-
 --
 drivers/net/xen-netback/netback.c=811=static bool tx_credit_exceeded(struct xenvif_queue *queue, unsigned size)
 drivers/net/xen-netback/netback.c-812-{
 drivers/net/xen-netback/netback.c:813:	u64 now = get_jiffies_64();
 drivers/net/xen-netback/netback.c-814-	u64 next_credit = queue->credit_window_start +
 --
 drivers/platform/x86/intel_ips.c=954=static int ips_monitor(void *data)
 --
 drivers/platform/x86/intel_ips.c-983-		ITV_ME_SEQNO_SHIFT;
 drivers/platform/x86/intel_ips.c:984:	seqno_timestamp = get_jiffies_64();
 drivers/platform/x86/intel_ips.c-985-
 --
 drivers/platform/x86/intel_ips.c-1093-		} else {
 drivers/platform/x86/intel_ips.c:1094:			seqno_timestamp = get_jiffies_64();
 drivers/platform/x86/intel_ips.c-1095-			last_seqno = cur_seqno;
]
]

gemini-3-flash-preview

							input: 63083
							output: 27
							thoughts: 25
						

Okay, here's my summary, tailored as if I'm the one thinking it:

**Digging into `memcontrol.c`**

Right, let's see... I need to find where `jiffies_64` is used, specifically within the memory control subsystem. I'm going to fire up `grep` with a targeted search. I'll use it to scour `mm/memcontrol.c` directly. This should give me a focused view of how that variable is being leveraged within the memory management context, which is where I expect to find the relevant usage.

no matches

map[Expression:jiffies_64.*mm/memcontrol.c]

map[Output:]

gemini-3-flash-preview

							input: 63275
							output: 27
							thoughts: 30
						

Okay, here's my summary, tailored for an expert audience:

**Kernel Memory Management Debugging - A "Doh!" Moment**

Right, so I was probably overthinking this, getting lost in the weeds of possible causes for this memory control issue. Initially, I was heading down some rabbit hole, probably considering a whole bunch of irrelevant files and functions. Then it hit me: I was being inefficient. Instead of complex analysis, I need to focus on what I *know* is related to the problem. The core issue revolves around memory control. The most direct approach is a simple grep command. A quick and dirty grep specifically for `jiffies_64`, which is a likely candidate given the potential timing component of the issue, combined with a filter for `mm/memcontrol.c`. This is precisely the kind of low-hanging fruit I should have grabbed for, as a starting point, before going down a deeper rabbit hole. Should have considered this approach before getting distracted. This will give me a very focused list of locations to explore. Stupid me.

no matches

map[Expression:jiffies_64.*mm/memcontrol.c]

map[Output:]

gemini-3-flash-preview

							input: 63567
							output: 27
							thoughts: 88
						

Okay, here's my quick rundown of these thoughts:

**Analyzing `mm/memcontrol.c` for `jiffies_64` Usage**

Right, let's start with a basic `grep` to quickly find instances of `jiffies_64` within `mm/memcontrol.c`. I'm thinking the simplest approach is to use my `grepper` tool, just searching for that string. I seem to recall seeing that variable used in `mem_cgroup_flush_stats_ratelimited`, but a broader search is always a good idea to ensure I haven't missed anything. I'll need to double check the rest of the file and ensure there are no other relevant instances.

no matches

map[Expression:jiffies_64.*mm/memcontrol.c]

map[Output:]

gemini-3-flash-preview

							input: 63851
							output: 27
							thoughts: 23
						

Okay, here's my interpretation of that thought process, tailored for an expert audience, and presented as if it were my own:

**Investigating `jiffies_64` Usage in `mm/memcontrol.c`**

Alright, so I need to understand how `jiffies_64` is being used within the memory control subsystem, specifically within `mm/memcontrol.c`. I'm going to start with a quick and dirty approach to get an initial overview. My instinct is to use `grep` – it's often the fastest way to perform an initial code search, I don't need to overcomplicate this process, I just want to find usages quickly. I'll execute a command to look for lines containing "jiffies_64" in `mm/memcontrol.c`. I'm adding `-n` to the grep command so that I get line numbers, which will let me quickly jump to the usage locations. That should give me a good starting point to identify the relevant code sections and understand the context. From there, I can decide if I need to dig deeper with `git grep`, cross-referencing, or other analysis techniques based on what `grep` turns up.

no matches

map[Expression:jiffies_64.*mm/memcontrol.c]

map[Output:]

gemini-3-flash-preview

							input: 64174
							output: 19
							thoughts: 1422
						

**Analyzing a Potential Data Race in `mem_cgroup_flush_stats_ratelimited`**

Okay, so I was trying to find something with `grepper`, but it seems to be failing on me again. Nevermind, I've seen the code:

```c
void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
{
	/* Only flush if the periodic flusher is one full cycle late */
	if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
		mem_cgroup_flush_stats(memcg);
}
```

My concern is that this function directly accesses `jiffies_64`, which in itself is unusual. I remember this was added relatively recently in the grand scheme of the kernel's evolution. If I check `mm/memcontrol.c` in a recent kernel, I can confirm its inclusion.

First thing, I had to ensure `jiffies_64` is indeed the variable used in `time_after64`. Yes, it is.

Now, is this race benign? The kernel docs generally say `jiffies` can be read without locks. However, for `jiffies_64` on 32-bit systems, you MUST use `get_jiffies_64()`. On 64-bit systems, `get_jiffies_64()` is just an alias for `jiffies`.

The direct usage of `jiffies_64` here is potentially buggy on 32-bit. On 64-bit, KCSAN might flag this as a data race if it wasn't for `READ_ONCE`.

Since the kernel has a helper function `get_jiffies_64()`, not using it is a (minor) bug.

Is it harmful in terms of crashes? No. Is it harmful in that it's incorrect? Yes, on 32-bit.

But does this function even *run* on 32-bit systems? Yes, memcg is supported.

If a "torn read" happens on 32-bit, what's the impact? We might get a future or past value. Future value means we flush when we shouldn't. Past value means we skip a flush when we should have. Since this is just a ratelimit, it's not a major issue.

However, KCSAN reports are often taken seriously, especially those indicating a missing synchronization primitive.

Let me see if there are other similar KCSAN reports for `jiffies_64`. I know `jiffies` itself is often considered benign, but `jiffies_64` is different because of that 32-bit tearing issue.

Wait, I found a similar case in the past. Actually, `jiffies_64` is often accessed via `get_jiffies_64()`.

I found a similar issue in `arch/x86/kernel/tsc.c`:

```c
arch/x86/kernel/tsc.c:256:	return (jiffies_64 - INITIAL_JIFFIES) * (1000000000 / HZ);
```

This *also* directly accesses `jiffies_64`, but it's in `arch/x86`, which implies it's either 64-bit only, or the developer knew it's safe on x86.

Actually, `jiffies_64` is defined in `kernel/time/timer.c`.

If I were to fix this, I'd definitely use `get_jiffies_64()`.

Is it benign? According to the definition provided, `jiffies_64` is a statistics counter (of sorts). A benign data race means it won't lead to a crash or memory corruption and that the accesses are not protected by any primitives.

On 64-bit, it's atomic. On 32-bit, it is NOT atomic. However, a torn read here would only lead to a slightly incorrect decision on whether or not to flush the stats. It won't cause memory corruption or a crash.

Based on the provided definition, it's *likely* benign.

However, it's still better to use `get_jiffies_64()`.

Wait, looking at the stack trace again: the write to `jiffies_64` is in `tick_do_update_jiffies64`. And it's read in

map[Expression:flush_last_time]

map[Output:mm/memcontrol.c=556=static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork);
mm/memcontrol.c:557:static u64 flush_last_time;
mm/memcontrol.c-558-
--
mm/memcontrol.c=599=static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force)
--
mm/memcontrol.c-609-	if (mem_cgroup_is_root(memcg))
mm/memcontrol.c:610:		WRITE_ONCE(flush_last_time, jiffies_64);
mm/memcontrol.c-611-
--
mm/memcontrol.c=635=void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
--
mm/memcontrol.c-637-	/* Only flush if the periodic flusher is one full cycle late */
mm/memcontrol.c:638:	if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
mm/memcontrol.c-639-		mem_cgroup_flush_stats(memcg);
]

gemini-3-flash-preview

							input: 66975
							output: 41
							thoughts: 592
						

**Analyzing a Potential Race Condition in Kernel Code**

Okay, wait a minute. Looking at this `mm/memcontrol.c` code, I see `WRITE_ONCE` being used on `flush_last_time`, which is a `u64`, and it’s directly reading `jiffies_64`. I know that on 32-bit architectures, writing a 64-bit value with `WRITE_ONCE` isn't atomic. Let me quickly double-check that definition... Yup, `include/linux/compiler.h` confirms it; `__write_once_size` will break down the write into multiple smaller writes if the architecture doesn't support an atomic operation for the full size.

So, on a 32-bit system, this `flush_last_time` could definitely be torn. But, wait, it's just for a rate limit, isn't it? The KCSAN report flagged `jiffies_64`, specifically.

Let me look at the value change from the report: it’s just incrementing the value from `0x00000000fffffaef` to `0x00000000fffffaf0`. That's a simple race between a read and a write.

Hmm, any other risks? If `jiffies_64` wasn't `volatile`, the compiler *could* cache it. But in `mem_cgroup_flush_stats_ratelimited`, it's only read once, so caching there isn't a problem.

Hold on, I need to check if `mem_cgroup_flush_stats_ratelimited` is called in a loop. It's called from `count_shadow_nodes`. Now, that's a callback for the shrinker, and those can be called *many* times.

If the compiler inlines `mem_cgroup_flush_stats_ratelimited` into a loop within `count_shadow_nodes`, and `jiffies_64` isn't `volatile`, it might cache that value across iterations. That would make the rate limit check stale.

Let's dig into `count_shadow_nodes`...

map[ContextFile:mm/workingset.c IncludeLines:true Name:count_shadow_nodes]

map[SourceCode: 640:	static unsigned long count_shadow_nodes(struct shrinker *shrinker,
 641:						struct shrink_control *sc)
 642:	{
 643:		unsigned long max_nodes;
 644:		unsigned long nodes;
 645:		unsigned long pages;
 646:	
 647:		nodes = list_lru_shrink_count(&shadow_nodes, sc);
 648:		if (!nodes)
 649:			return SHRINK_EMPTY;
 650:	
 651:		/*
 652:		 * Approximate a reasonable limit for the nodes
 653:		 * containing shadow entries. We don't need to keep more
 654:		 * shadow entries than possible pages on the active list,
 655:		 * since refault distances bigger than that are dismissed.
 656:		 *
 657:		 * The size of the active list converges toward 100% of
 658:		 * overall page cache as memory grows, with only a tiny
 659:		 * inactive list. Assume the total cache size for that.
 660:		 *
 661:		 * Nodes might be sparsely populated, with only one shadow
 662:		 * entry in the extreme case. Obviously, we cannot keep one
 663:		 * node for every eligible shadow entry, so compromise on a
 664:		 * worst-case density of 1/8th. Below that, not all eligible
 665:		 * refaults can be detected anymore.
 666:		 *
 667:		 * On 64-bit with 7 xa_nodes per page and 64 slots
 668:		 * each, this will reclaim shadow entries when they consume
 669:		 * ~1.8% of available memory:
 670:		 *
 671:		 * PAGE_SIZE / xa_nodes / node_entries * 8 / PAGE_SIZE
 672:		 */
 673:	#ifdef CONFIG_MEMCG
 674:		if (sc->memcg) {
 675:			struct lruvec *lruvec;
 676:			int i;
 677:	
 678:			mem_cgroup_flush_stats_ratelimited(sc->memcg);
 679:			lruvec = mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid));
 680:			for (pages = 0, i = 0; i < NR_LRU_LISTS; i++)
 681:				pages += lruvec_page_state_local(lruvec,
 682:								 NR_LRU_BASE + i);
 683:			pages += lruvec_page_state_local(
 684:				lruvec, NR_SLAB_RECLAIMABLE_B) >> PAGE_SHIFT;
 685:			pages += lruvec_page_state_local(
 686:				lruvec, NR_SLAB_UNRECLAIMABLE_B) >> PAGE_SHIFT;
 687:		} else
 688:	#endif
 689:			pages = node_present_pages(sc->nid);
 690:	
 691:		max_nodes = pages >> (XA_CHUNK_SHIFT - 3);
 692:	
 693:		if (nodes <= max_nodes)
 694:			return 0;
 695:		return nodes - max_nodes;
 696:	}
 SourceFile:mm/workingset.c]

gemini-3-flash-preview

							input: 69033
							output: 40
							thoughts: 67
						

**Analyzing Shadow Node Counting in the Memory Management Subsystem**

Okay, so `count_shadow_nodes` seems to only call `mem_cgroup_flush_stats_ratelimited` a single time. That's a key observation. The shrinker could potentially call `count_shadow_nodes` repeatedly, which seems likely based on the flow, but it would be embedded within the `do_shrink_slab` function. Makes sense; that's where the iteration would be, naturally. Given that, `do_shrink_slab` is the next logical point to investigate, to verify the context and the call chain. Let's dig into that function and understand its behavior in detail.

map[ContextFile:mm/shrinker.c IncludeLines:true Name:do_shrink_slab]

map[SourceCode: 371:	static unsigned long do_shrink_slab(struct shrink_control *shrinkctl,
 372:					    struct shrinker *shrinker, int priority)
 373:	{
 374:		unsigned long freed = 0;
 375:		unsigned long long delta;
 376:		long total_scan;
 377:		long freeable;
 378:		long nr;
 379:		long new_nr;
 380:		long batch_size = shrinker->batch ? shrinker->batch
 381:						  : SHRINK_BATCH;
 382:		long scanned = 0, next_deferred;
 383:	
 384:		freeable = shrinker->count_objects(shrinker, shrinkctl);
 385:		if (freeable == 0 || freeable == SHRINK_EMPTY)
 386:			return freeable;
 387:	
 388:		/*
 389:		 * copy the current shrinker scan count into a local variable
 390:		 * and zero it so that other concurrent shrinker invocations
 391:		 * don't also do this scanning work.
 392:		 */
 393:		nr = xchg_nr_deferred(shrinker, shrinkctl);
 394:	
 395:		if (shrinker->seeks) {
 396:			delta = freeable >> priority;
 397:			delta *= 4;
 398:			do_div(delta, shrinker->seeks);
 399:		} else {
 400:			/*
 401:			 * These objects don't require any IO to create. Trim
 402:			 * them aggressively under memory pressure to keep
 403:			 * them from causing refetches in the IO caches.
 404:			 */
 405:			delta = freeable / 2;
 406:		}
 407:	
 408:		total_scan = nr >> priority;
 409:		total_scan += delta;
 410:		total_scan = min(total_scan, (2 * freeable));
 411:	
 412:		trace_mm_shrink_slab_start(shrinker, shrinkctl, nr,
 413:					   freeable, delta, total_scan, priority);
 414:	
 415:		/*
 416:		 * Normally, we should not scan less than batch_size objects in one
 417:		 * pass to avoid too frequent shrinker calls, but if the slab has less
 418:		 * than batch_size objects in total and we are really tight on memory,
 419:		 * we will try to reclaim all available objects, otherwise we can end
 420:		 * up failing allocations although there are plenty of reclaimable
 421:		 * objects spread over several slabs with usage less than the
 422:		 * batch_size.
 423:		 *
 424:		 * We detect the "tight on memory" situations by looking at the total
 425:		 * number of objects we want to scan (total_scan). If it is greater
 426:		 * than the total number of objects on slab (freeable), we must be
 427:		 * scanning at high prio and therefore should try to reclaim as much as
 428:		 * possible.
 429:		 */
 430:		while (total_scan >= batch_size ||
 431:		       total_scan >= freeable) {
 432:			unsigned long ret;
 433:			unsigned long nr_to_scan = min(batch_size, total_scan);
 434:	
 435:			shrinkctl->nr_to_scan = nr_to_scan;
 436:			shrinkctl->nr_scanned = nr_to_scan;
 437:			ret = shrinker->scan_objects(shrinker, shrinkctl);
 438:			if (ret == SHRINK_STOP)
 439:				break;
 440:			freed += ret;
 441:	
 442:			count_vm_events(SLABS_SCANNED, shrinkctl->nr_scanned);
 443:			total_scan -= shrinkctl->nr_scanned;
 444:			scanned += shrinkctl->nr_scanned;
 445:	
 446:			cond_resched();
 447:		}
 448:	
 449:		/*
 450:		 * The deferred work is increased by any new work (delta) that wasn't
 451:		 * done, decreased by old deferred work that was done now.
 452:		 *
 453:		 * And it is capped to two times of the freeable items.
 454:		 */
 455:		next_deferred = max_t(long, (nr + delta - scanned), 0);
 456:		next_deferred = min(next_deferred, (2 * freeable));
 457:	
 458:		/*
 459:		 * move the unused scan count back into the shrinker in a
 460:		 * manner that handles concurrent updates.
 461:		 */
 462:		new_nr = add_nr_deferred(next_deferred, shrinker, shrinkctl);
 463:	
 464:		trace_mm_shrink_slab_end(shrinker, shrinkctl->nid, freed, nr, new_nr, total_scan);
 465:		return freed;
 466:	}
 SourceFile:mm/shrinker.c]

gemini-3-flash-preview

							input: 70771
							output: 23
							thoughts: 267
						

**Analyzing the `do_shrink_slab` Function and Potential Data Race**

Okay, let me break this down. `do_shrink_slab` starts by calling `count_objects` (which is really `count_shadow_nodes`) just once, at line 384. Then, it dives into a loop that calls `scan_objects`. So, that means `count_shadow_nodes` is essentially called once per "shrinker," per node, and per priority level.

Now, about that caching of `jiffies_64`... if the compiler decided to cache it, it would only be relevant for the duration of the `count_shadow_nodes` call. And since `jiffies_64` is only read once within that function, specifically in `mem_cgroup_flush_stats_ratelimited`, caching isn't a problem here.

So, really, the only thing I'm concerned about is the data race itself. Is this actually something to worry about? Considering it's a 64-bit variable on a 64-bit system, and it's being used for a ratelimit, it's *highly* probable that it's harmless. Even on a 32-bit system, a torn read probably just means a slightly off ratelimit decision, which is not really a concern.

But... KCSAN flagged it. And that means it's a violation of the rules, regardless of how seemingly harmless it is.

Hold on, I should probably double check the rest of the `mm/` directory, to see if `jiffies_64` is used elsewhere in a similar fashion.

no matches

map[Expression:jiffies_64.*mm/]

map[Output:]

gemini-3-flash-preview

							input: 71434
							output: 20
							thoughts: 21
						

Okay, here's my interpretation of that thought process, tailored for an expert audience:

**Kernel Symbol Hunt: `jiffies_64` in `mm/`**

Right, so I'm thinking about tracking down the definition or usage of `jiffies_64`. Given my understanding of kernel structures, my immediate approach is to check the `mm/` directory. That seems like a logical first step. I suspect that because it's dealing with time, it may be related to memory management in some way.

map[Expression:jiffies_64]

map[Output:
Full output is too long, showing 500 out of 1048 lines.
Use more precise expression if possible.

[arch/alpha/kernel/vmlinux.lds.S=15=PHDRS { text PT_LOAD; note PT_NOTE; }
 arch/alpha/kernel/vmlinux.lds.S:16:jiffies = jiffies_64;
 arch/alpha/kernel/vmlinux.lds.S-17-SECTIONS
 --
 arch/arc/kernel/vmlinux.lds.S=12=ENTRY(res_service)
 --
 arch/arc/kernel/vmlinux.lds.S-14-#ifdef CONFIG_CPU_BIG_ENDIAN
 arch/arc/kernel/vmlinux.lds.S:15:jiffies = jiffies_64 + 4;
 arch/arc/kernel/vmlinux.lds.S-16-#else
 arch/arc/kernel/vmlinux.lds.S:17:jiffies = jiffies_64;
 arch/arc/kernel/vmlinux.lds.S-18-#endif
 --
 arch/arm/kernel/smp_twd.c=140=static void twd_calibrate_rate(void)
 --
 arch/arm/kernel/smp_twd.c-152-		/* Wait for a tick to start */
 arch/arm/kernel/smp_twd.c:153:		waitjiffies = get_jiffies_64() + 1;
 arch/arm/kernel/smp_twd.c-154-
 arch/arm/kernel/smp_twd.c:155:		while (get_jiffies_64() < waitjiffies)
 arch/arm/kernel/smp_twd.c-156-			udelay(10);
 --
 arch/arm/kernel/smp_twd.c-166-
 arch/arm/kernel/smp_twd.c:167:		while (get_jiffies_64() < waitjiffies)
 arch/arm/kernel/smp_twd.c-168-			udelay(10);
 --
 arch/arm/kernel/vmlinux-xip.lds.S=19=ENTRY(stext)
 --
 arch/arm/kernel/vmlinux-xip.lds.S-21-#ifndef __ARMEB__
 arch/arm/kernel/vmlinux-xip.lds.S:22:jiffies = jiffies_64;
 arch/arm/kernel/vmlinux-xip.lds.S-23-#else
 arch/arm/kernel/vmlinux-xip.lds.S:24:jiffies = jiffies_64 + 4;
 arch/arm/kernel/vmlinux-xip.lds.S-25-#endif
 --
 arch/arm/kernel/vmlinux.lds.S=19=ENTRY(stext)
 --
 arch/arm/kernel/vmlinux.lds.S-21-#ifndef __ARMEB__
 arch/arm/kernel/vmlinux.lds.S:22:jiffies = jiffies_64;
 arch/arm/kernel/vmlinux.lds.S-23-#else
 arch/arm/kernel/vmlinux.lds.S:24:jiffies = jiffies_64 + 4;
 arch/arm/kernel/vmlinux.lds.S-25-#endif
 --
 arch/arm64/kernel/vmlinux.lds.S=83=ENTRY(_text)
 arch/arm64/kernel/vmlinux.lds.S-84-
 arch/arm64/kernel/vmlinux.lds.S:85:jiffies = jiffies_64;
 arch/arm64/kernel/vmlinux.lds.S-86-
 --
 arch/csky/kernel/vmlinux.lds.S=8=ENTRY(_start)
 --
 arch/csky/kernel/vmlinux.lds.S-10-#ifndef __cskyBE__
 arch/csky/kernel/vmlinux.lds.S:11:jiffies = jiffies_64;
 arch/csky/kernel/vmlinux.lds.S-12-#else
 arch/csky/kernel/vmlinux.lds.S:13:jiffies = jiffies_64 + 4;
 arch/csky/kernel/vmlinux.lds.S-14-#endif
 --
 arch/hexagon/kernel/vmlinux.lds.S=15=ENTRY(stext)
 arch/hexagon/kernel/vmlinux.lds.S-16-
 arch/hexagon/kernel/vmlinux.lds.S:17:jiffies = jiffies_64;
 arch/hexagon/kernel/vmlinux.lds.S-18-
 --
 arch/loongarch/kernel/vmlinux.lds.S=29=PHDRS {
 --
 arch/loongarch/kernel/vmlinux.lds.S-33-
 arch/loongarch/kernel/vmlinux.lds.S:34:jiffies	 = jiffies_64;
 arch/loongarch/kernel/vmlinux.lds.S-35-
 --
 arch/m68k/kernel/vmlinux-nommu.lds=25=ENTRY(_start)
 arch/m68k/kernel/vmlinux-nommu.lds-26-
 arch/m68k/kernel/vmlinux-nommu.lds:27:jiffies = jiffies_64 + 4;
 arch/m68k/kernel/vmlinux-nommu.lds-28-
 --
 arch/m68k/kernel/vmlinux-std.lds=10=ENTRY(_start)
 arch/m68k/kernel/vmlinux-std.lds:11:jiffies = jiffies_64 + 4;
 arch/m68k/kernel/vmlinux-std.lds-12-SECTIONS
 --
 arch/m68k/kernel/vmlinux-sun3.lds=10=ENTRY(_start)
 arch/m68k/kernel/vmlinux-sun3.lds:11:jiffies = jiffies_64 + 4;
 arch/m68k/kernel/vmlinux-sun3.lds-12-SECTIONS
 --
 arch/microblaze/kernel/vmlinux.lds.S=12=ENTRY(microblaze_start)
 --
 arch/microblaze/kernel/vmlinux.lds.S-21-#ifdef __MICROBLAZEEL__
 arch/microblaze/kernel/vmlinux.lds.S:22:jiffies = jiffies_64;
 arch/microblaze/kernel/vmlinux.lds.S-23-#else
 arch/microblaze/kernel/vmlinux.lds.S:24:jiffies = jiffies_64 + 4;
 arch/microblaze/kernel/vmlinux.lds.S-25-#endif
 --
 arch/mips/kernel/vmlinux.lds.S=26=PHDRS {
 --
 arch/mips/kernel/vmlinux.lds.S-34-	#ifdef CONFIG_CPU_LITTLE_ENDIAN
 arch/mips/kernel/vmlinux.lds.S:35:		jiffies	 = jiffies_64;
 arch/mips/kernel/vmlinux.lds.S-36-	#else
 arch/mips/kernel/vmlinux.lds.S:37:		jiffies	 = jiffies_64 + 4;
 arch/mips/kernel/vmlinux.lds.S-38-	#endif
 arch/mips/kernel/vmlinux.lds.S-39-#else
 arch/mips/kernel/vmlinux.lds.S:40:	jiffies	 = jiffies_64;
 arch/mips/kernel/vmlinux.lds.S-41-#endif
 --
 arch/nios2/kernel/vmlinux.lds.S=13=ENTRY(_start)	/* Defined in head.S */
 arch/nios2/kernel/vmlinux.lds.S-14-
 arch/nios2/kernel/vmlinux.lds.S:15:jiffies = jiffies_64;
 arch/nios2/kernel/vmlinux.lds.S-16-
 --
 arch/openrisc/kernel/vmlinux.lds.S=36=OUTPUT_FORMAT(__OUTPUT_FORMAT, __OUTPUT_FORMAT, __OUTPUT_FORMAT)
 arch/openrisc/kernel/vmlinux.lds.S:37:jiffies = jiffies_64 + 4;
 arch/openrisc/kernel/vmlinux.lds.S-38-
 --
 arch/parisc/kernel/vmlinux.lds.S=50=ENTRY(parisc_kernel_start)
 arch/parisc/kernel/vmlinux.lds.S-51-#ifndef CONFIG_64BIT
 arch/parisc/kernel/vmlinux.lds.S:52:jiffies = jiffies_64 + 4;
 arch/parisc/kernel/vmlinux.lds.S-53-#else
 arch/parisc/kernel/vmlinux.lds.S:54:jiffies = jiffies_64;
 arch/parisc/kernel/vmlinux.lds.S-55-#endif
 --
 arch/powerpc/kernel/vmlinux.lds.S=42=OUTPUT_ARCH(powerpc:common64)
 arch/powerpc/kernel/vmlinux.lds.S:43:jiffies = jiffies_64;
 arch/powerpc/kernel/vmlinux.lds.S-44-#else
 arch/powerpc/kernel/vmlinux.lds.S=45=OUTPUT_ARCH(powerpc:common)
 arch/powerpc/kernel/vmlinux.lds.S:46:jiffies = jiffies_64 + 4;
 arch/powerpc/kernel/vmlinux.lds.S-47-#endif
 --
 arch/riscv/kernel/vmlinux-xip.lds.S=20=ENTRY(_start)
 arch/riscv/kernel/vmlinux-xip.lds.S-21-
 arch/riscv/kernel/vmlinux-xip.lds.S:22:jiffies = jiffies_64;
 arch/riscv/kernel/vmlinux-xip.lds.S-23-
 --
 arch/riscv/kernel/vmlinux.lds.S=26=ENTRY(_start)
 arch/riscv/kernel/vmlinux.lds.S-27-
 arch/riscv/kernel/vmlinux.lds.S:28:jiffies = jiffies_64;
 arch/riscv/kernel/vmlinux.lds.S-29-
 --
 arch/s390/kernel/vmlinux.lds.S=31=ENTRY(startup_continue)
 arch/s390/kernel/vmlinux.lds.S:32:jiffies = jiffies_64;
 arch/s390/kernel/vmlinux.lds.S-33-
 --
 arch/s390/kernel/vtime.c=59=static void update_mt_scaling(void)
 --
 arch/s390/kernel/vtime.c-83-	}
 arch/s390/kernel/vtime.c:84:	__this_cpu_write(mt_scaling_jiffies, jiffies_64);
 arch/s390/kernel/vtime.c-85-}
 --
 arch/s390/kernel/vtime.c=118=static int do_account_vtime(struct task_struct *tsk)
 --
 arch/s390/kernel/vtime.c-140-	if (smp_cpu_mtid &&
 arch/s390/kernel/vtime.c:141:	    time_after64(jiffies_64, this_cpu_read(mt_scaling_jiffies)))
 arch/s390/kernel/vtime.c-142-		update_mt_scaling();
 --
 arch/sh/Makefile=106=ld_bfd			:= elf32-sh-linux
 arch/sh/Makefile:107:LDFLAGS_vmlinux		+= --defsym jiffies=jiffies_64 --oformat $(ld_bfd)
 arch/sh/Makefile-108-KBUILD_LDFLAGS		+= -EL
 --
 arch/sh/Makefile=110=ld_bfd			:= elf32-shbig-linux
 arch/sh/Makefile:111:LDFLAGS_vmlinux		+= --defsym jiffies=jiffies_64+4 --oformat $(ld_bfd)
 arch/sh/Makefile-112-KBUILD_LDFLAGS		+= -EB
 --
 arch/sparc/kernel/vmlinux.lds.S=27=ENTRY(_start)
 arch/sparc/kernel/vmlinux.lds.S:28:jiffies = jiffies_64 + 4;
 arch/sparc/kernel/vmlinux.lds.S-29-#else
 --
 arch/sparc/kernel/vmlinux.lds.S=33=ENTRY(_start)
 arch/sparc/kernel/vmlinux.lds.S:34:jiffies = jiffies_64;
 arch/sparc/kernel/vmlinux.lds.S-35-#endif
 --
 arch/um/kernel/dyn.lds.S=6=ENTRY(_start)
 arch/um/kernel/dyn.lds.S:7:jiffies = jiffies_64;
 arch/um/kernel/dyn.lds.S-8-
 --
 arch/um/kernel/uml.lds.S=7=ENTRY(_start)
 arch/um/kernel/uml.lds.S:8:jiffies = jiffies_64;
 arch/um/kernel/uml.lds.S-9-
 --
 arch/x86/kernel/tsc.c=237=noinstr u64 native_sched_clock(void)
 --
 arch/x86/kernel/tsc.c-255-	/* No locking but a rare wrong value is not a big deal: */
 arch/x86/kernel/tsc.c:256:	return (jiffies_64 - INITIAL_JIFFIES) * (1000000000 / HZ);
 arch/x86/kernel/tsc.c-257-}
 --
 arch/x86/kernel/vmlinux.lds.S=42=ENTRY(phys_startup_64)
 --
 arch/x86/kernel/vmlinux.lds.S-44-
 arch/x86/kernel/vmlinux.lds.S:45:jiffies = jiffies_64;
 arch/x86/kernel/vmlinux.lds.S-46-const_current_task = current_task;
 --
 arch/x86/kvm/mmu/mmu.c=7830=static bool kvm_nx_huge_page_recovery_worker(void *data)
 --
 arch/x86/kvm/mmu/mmu.c-7842-	remaining_time = kvm->arch.nx_huge_page_last + msecs_to_jiffies(period)
 arch/x86/kvm/mmu/mmu.c:7843:		- get_jiffies_64();
 arch/x86/kvm/mmu/mmu.c-7844-	if (remaining_time > 0) {
 --
 arch/x86/kvm/mmu/mmu.c-7852-		kvm_recover_nx_huge_pages(kvm, i);
 arch/x86/kvm/mmu/mmu.c:7853:	kvm->arch.nx_huge_page_last = get_jiffies_64();
 arch/x86/kvm/mmu/mmu.c-7854-	return true;
 --
 arch/x86/kvm/mmu/mmu.c=7857=static int kvm_mmu_start_lpage_recovery(struct once *once)
 --
 arch/x86/kvm/mmu/mmu.c-7862-
 arch/x86/kvm/mmu/mmu.c:7863:	kvm->arch.nx_huge_page_last = get_jiffies_64();
 arch/x86/kvm/mmu/mmu.c-7864-	nx_thread = vhost_task_create(kvm_nx_huge_page_recovery_worker,
 --
 arch/x86/tools/relocs.c=53=static const char * const	sym_regex_kernel[S_NSYMTYPES] = {
 --
 arch/x86/tools/relocs.c-90-	"__initramfs_start|"
 arch/x86/tools/relocs.c:91:	"(jiffies|jiffies_64)|"
 arch/x86/tools/relocs.c-92-#if ELF_BITS == 64
 --
 arch/xtensa/kernel/vmlinux.lds.S=27=ENTRY(_start)
 --
 arch/xtensa/kernel/vmlinux.lds.S-29-#ifdef __XTENSA_EB__
 arch/xtensa/kernel/vmlinux.lds.S:30:jiffies = jiffies_64 + 4;
 arch/xtensa/kernel/vmlinux.lds.S-31-#else
 arch/xtensa/kernel/vmlinux.lds.S:32:jiffies = jiffies_64;
 arch/xtensa/kernel/vmlinux.lds.S-33-#endif
 --
 drivers/acpi/osl.c=655=u64 acpi_os_get_timer(void)
 drivers/acpi/osl.c-656-{
 drivers/acpi/osl.c:657:	return (get_jiffies_64() - INITIAL_JIFFIES) *
 drivers/acpi/osl.c-658-		(ACPI_100NSEC_PER_SEC / HZ);
 --
 drivers/ata/libata-eh.c=370=static void ata_ering_record(struct ata_ering *ering, unsigned int eflags,
 --
 drivers/ata/libata-eh.c-382-	ent->err_mask = err_mask;
 drivers/ata/libata-eh.c:383:	ent->timestamp = get_jiffies_64();
 drivers/ata/libata-eh.c-384-}
 --
 drivers/ata/libata-eh.c=1821=static unsigned int ata_eh_speed_down_verdict(struct ata_device *dev)
 --
 drivers/ata/libata-eh.c-1823-	const u64 j5mins = 5LLU * 60 * HZ, j10mins = 10LLU * 60 * HZ;
 drivers/ata/libata-eh.c:1824:	u64 j64 = get_jiffies_64();
 drivers/ata/libata-eh.c-1825-	struct speed_down_verdict_arg arg;
 --
 drivers/ata/libata-eh.c=3656=static int ata_count_probe_trials_cb(struct ata_ering_entry *ent, void *void_arg)
 --
 drivers/ata/libata-eh.c-3658-	u64 interval = msecs_to_jiffies(ATA_EH_PROBE_TRIAL_INTERVAL);
 drivers/ata/libata-eh.c:3659:	u64 now = get_jiffies_64();
 drivers/ata/libata-eh.c-3660-	int *trials = void_arg;
 --
 drivers/cpufreq/cpufreq.c=133=static inline u64 get_cpu_idle_time_jiffy(unsigned int cpu, u64 *wall)
 --
 drivers/cpufreq/cpufreq.c-139-
 drivers/cpufreq/cpufreq.c:140:	cur_wall_time = jiffies64_to_nsecs(get_jiffies_64());
 drivers/cpufreq/cpufreq.c-141-
 --
 drivers/cpufreq/powernv-cpufreq.c=741=static int powernv_cpufreq_target_index(struct cpufreq_policy *policy,
 --
 drivers/cpufreq/powernv-cpufreq.c-759-
 drivers/cpufreq/powernv-cpufreq.c:760:	cur_msec = jiffies_to_msecs(get_jiffies_64());
 drivers/cpufreq/powernv-cpufreq.c-761-
 --
 drivers/devfreq/devfreq.c=213=int devfreq_update_status(struct devfreq *devfreq, unsigned long freq)
 --
 drivers/devfreq/devfreq.c-218-	lockdep_assert_held(&devfreq->lock);
 drivers/devfreq/devfreq.c:219:	cur_time = get_jiffies_64();
 drivers/devfreq/devfreq.c-220-
 --
 drivers/devfreq/devfreq.c=576=void devfreq_monitor_resume(struct devfreq *devfreq)
 --
 drivers/devfreq/devfreq.c-593-out_update:
 drivers/devfreq/devfreq.c:594:	devfreq->stats.last_update = get_jiffies_64();
 drivers/devfreq/devfreq.c-595-	devfreq->stop_polling = false;
 --
 drivers/devfreq/devfreq.c=800=struct devfreq *devfreq_add_device(struct device *dev,
 --
 drivers/devfreq/devfreq.c-911-	devfreq->stats.total_trans = 0;
 drivers/devfreq/devfreq.c:912:	devfreq->stats.last_update = get_jiffies_64();
 drivers/devfreq/devfreq.c-913-
 --
 drivers/devfreq/devfreq.c=1767=static ssize_t trans_stat_store(struct device *dev,
 --
 drivers/devfreq/devfreq.c-1790-	df->stats.total_trans = 0;
 drivers/devfreq/devfreq.c:1791:	df->stats.last_update = get_jiffies_64();
 drivers/devfreq/devfreq.c-1792-	mutex_unlock(&df->lock);
 --
 drivers/firewire/core-topology.c=459=void fw_core_handle_bus_reset(struct fw_card *card, int node_id, int generation,
 --
 drivers/firewire/core-topology.c-479-		card->generation = generation;
 drivers/firewire/core-topology.c:480:		card->reset_jiffies = get_jiffies_64();
 drivers/firewire/core-topology.c-481-		card->bm_node_id  = 0xffff;
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device.c=1231=int kgd2kfd_schedule_evict_and_restore_process(struct mm_struct *mm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device.c-1255-	 */
 drivers/gpu/drm/amd/amdkfd/kfd_device.c:1256:	active_time = get_jiffies_64() - p->last_restore_timestamp;
 drivers/gpu/drm/amd/amdkfd/kfd_device.c-1257-	if (delay_jiffies > active_time)
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1140=static int evict_process_queues_nocpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1155-
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1156:	pdd->last_evict_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1157-	/* Mark all queues as evicted. Deactivate all active queues on
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1190=static int evict_process_queues_cpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1214-	if (dqm->dev->kfd->shared_resources.enable_mes)
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1215:		pdd->last_evict_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1216-
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1238-	if (!dqm->dev->kfd->shared_resources.enable_mes) {
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1239:		pdd->last_evict_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1240-		retval = execute_queues_cpsch(dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1252=static int restore_process_queues_nocpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1323-	qpd->evicted = 0;
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1324:	eviction_duration = get_jiffies_64() - pdd->last_evict_timestamp;
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1325-	atomic64_add(eviction_duration, &pdd->evict_duration_counter);
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c=1333=static int restore_process_queues_cpsch(struct device_queue_manager *dqm,
 --
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1386-					      KFD_UNMAP_QUEUES_FILTER_DYNAMIC_QUEUES, 0, USE_DEFAULT_GRACE_PERIOD);
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c:1387:	eviction_duration = get_jiffies_64() - pdd->last_evict_timestamp;
 drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c-1388-	atomic64_add(eviction_duration, &pdd->evict_duration_counter);
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c=1589=struct kfd_process *create_process(const struct task_struct *thread, bool primary)
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-1607-	INIT_DELAYED_WORK(&process->restore_work, restore_process_worker);
 drivers/gpu/drm/amd/amdkfd/kfd_process.c:1608:	process->last_restore_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-1609-	err = kfd_event_init_process(process);
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c=2161=static void restore_process_worker(struct work_struct *work)
 --
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-2184-
 drivers/gpu/drm/amd/amdkfd/kfd_process.c:2185:	p->last_restore_timestamp = get_jiffies_64();
 drivers/gpu/drm/amd/amdkfd/kfd_process.c-2186-
 --
 drivers/gpu/drm/drm_pagemap.c=471=int drm_pagemap_migrate_to_devmem(struct drm_pagemap_devmem *devmem_allocation,
 --
 drivers/gpu/drm/drm_pagemap.c-651-	/* Upon success bind devmem allocation to range and zdd */
 drivers/gpu/drm/drm_pagemap.c:652:	devmem_allocation->timeslice_expiration = get_jiffies_64() +
 drivers/gpu/drm/drm_pagemap.c-653-		msecs_to_jiffies(mdetails->timeslice_ms);
 --
 drivers/gpu/drm/drm_pagemap.c=1047=static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas,
 --
 drivers/gpu/drm/drm_pagemap.c-1070-	zdd = page->zone_device_data;
 drivers/gpu/drm/drm_pagemap.c:1071:	if (time_before64(get_jiffies_64(), zdd->devmem_allocation->timeslice_expiration))
 drivers/gpu/drm/drm_pagemap.c-1072-		return 0;
 --
 drivers/gpu/drm/i915/i915_hwmon.c=671=hwm_fan_input_read(struct hwm_drvdata *ddat, long *val)
 --
 drivers/gpu/drm/i915/i915_hwmon.c-683-	reg_val = intel_uncore_read(ddat->uncore, hwmon->rg.fan_speed);
 drivers/gpu/drm/i915/i915_hwmon.c:684:	time_now = get_jiffies_64();
 drivers/gpu/drm/i915/i915_hwmon.c-685-
 --
 drivers/gpu/drm/i915/i915_hwmon.c=836=hwm_get_preregistration_info(struct drm_i915_private *i915)
 --
 drivers/gpu/drm/i915/i915_hwmon.c-883-								  hwmon->rg.fan_speed);
 drivers/gpu/drm/i915/i915_hwmon.c:884:			ddat->fi.time_prev = get_jiffies_64();
 drivers/gpu/drm/i915/i915_hwmon.c-885-		}
 --
 drivers/gpu/drm/panthor/panthor_sched.c=2445=static void tick_work(struct work_struct *work)
 --
 drivers/gpu/drm/panthor/panthor_sched.c-2452-	u64 remaining_jiffies = 0, resched_delay;
 drivers/gpu/drm/panthor/panthor_sched.c:2453:	u64 now = get_jiffies_64();
 drivers/gpu/drm/panthor/panthor_sched.c-2454-	int prio, ret, cookie;
 --
 drivers/gpu/drm/panthor/panthor_sched.c=2650=static void sched_resume_tick(struct panthor_device *ptdev)
 --
 drivers/gpu/drm/panthor/panthor_sched.c-2659-	 */
 drivers/gpu/drm/panthor/panthor_sched.c:2660:	now = get_jiffies_64();
 drivers/gpu/drm/panthor/panthor_sched.c-2661-	sched->resched_target = sched->last_tick + sched->tick_period;
 --
 drivers/gpu/drm/radeon/radeon_ring.c=241=void radeon_ring_lockup_update(struct radeon_device *rdev,
 --
 drivers/gpu/drm/radeon/radeon_ring.c-244-	atomic_set(&ring->last_rptr, radeon_ring_get_rptr(rdev, ring));
 drivers/gpu/drm/radeon/radeon_ring.c:245:	atomic64_set(&ring->last_activity, jiffies_64);
 drivers/gpu/drm/radeon/radeon_ring.c-246-}
 --
 drivers/gpu/drm/radeon/radeon_ring.c=254=bool radeon_ring_test_lockup(struct radeon_device *rdev, struct radeon_ring *ring)
 --
 drivers/gpu/drm/radeon/radeon_ring.c-265-
 drivers/gpu/drm/radeon/radeon_ring.c:266:	elapsed = jiffies_to_msecs(jiffies_64 - last);
 drivers/gpu/drm/radeon/radeon_ring.c-267-	if (radeon_lockup_timeout && elapsed >= radeon_lockup_timeout) {
 --
 drivers/gpu/drm/xe/xe_hwmon.c=1287=xe_hwmon_fan_input_read(struct xe_hwmon *hwmon, int channel, long *val)
 --
 drivers/gpu/drm/xe/xe_hwmon.c-1297-	reg_val = xe_mmio_read32(mmio, xe_hwmon_get_reg(hwmon, REG_FAN_SPEED, channel));
 drivers/gpu/drm/xe/xe_hwmon.c:1298:	time_now = get_jiffies_64();
 drivers/gpu/drm/xe/xe_hwmon.c-1299-
 --
 drivers/i2c/busses/i2c-octeon-core.c=50=static int octeon_i2c_wait(struct octeon_i2c *i2c)
 --
 drivers/i2c/busses/i2c-octeon-core.c-58-	if (i2c->broken_irq_mode) {
 drivers/i2c/busses/i2c-octeon-core.c:59:		u64 end = get_jiffies_64() + i2c->adap.timeout;
 drivers/i2c/busses/i2c-octeon-core.c-60-
 drivers/i2c/busses/i2c-octeon-core.c-61-		while (!octeon_i2c_test_iflg(i2c) &&
 drivers/i2c/busses/i2c-octeon-core.c:62:		       time_before64(get_jiffies_64(), end))
 drivers/i2c/busses/i2c-octeon-core.c-63-			usleep_range(I2C_OCTEON_EVENT_WAIT / 2, I2C_OCTEON_EVENT_WAIT);
 --
 drivers/i2c/busses/i2c-octeon-core.c=170=static int octeon_i2c_hlc_wait(struct octeon_i2c *i2c)
 --
 drivers/i2c/busses/i2c-octeon-core.c-178-	if (i2c->broken_irq_mode) {
 drivers/i2c/busses/i2c-octeon-core.c:179:		u64 end = get_jiffies_64() + i2c->adap.timeout;
 drivers/i2c/busses/i2c-octeon-core.c-180-
 drivers/i2c/busses/i2c-octeon-core.c-181-		while (!octeon_i2c_hlc_test_valid(i2c) &&
 drivers/i2c/busses/i2c-octeon-core.c:182:		       time_before64(get_jiffies_64(), end))
 drivers/i2c/busses/i2c-octeon-core.c-183-			usleep_range(I2C_OCTEON_EVENT_WAIT / 2, I2C_OCTEON_EVENT_WAIT);
 --
 drivers/macintosh/rack-meter.c=212=static void rackmeter_do_timer(struct work_struct *work)
 --
 drivers/macintosh/rack-meter.c-221-
 drivers/macintosh/rack-meter.c:222:	cur_nsecs = jiffies64_to_nsecs(get_jiffies_64());
 drivers/macintosh/rack-meter.c-223-	total_nsecs = cur_nsecs - rcpu->prev_wall;
 --
 drivers/macintosh/rack-meter.c=257=static void rackmeter_init_cpu_sniffer(struct rackmeter *rm)
 --
 drivers/macintosh/rack-meter.c-278-		rcpu->prev_idle = get_cpu_idle_time(cpu);
 drivers/macintosh/rack-meter.c:279:		rcpu->prev_wall = jiffies64_to_nsecs(get_jiffies_64());
 drivers/macintosh/rack-meter.c-280-		schedule_delayed_work_on(cpu, &rm->cpu[cpu].sniffer,
 --
 drivers/mailbox/imx-mailbox.c=160=static int imx_mu_tx_waiting_write(struct imx_mu_priv *priv, u32 val, u32 idx)
 drivers/mailbox/imx-mailbox.c-161-{
 drivers/mailbox/imx-mailbox.c:162:	u64 timeout_time = get_jiffies_64() + IMX_MU_SECO_TX_TOUT;
 drivers/mailbox/imx-mailbox.c-163-	u32 status;
 --
 drivers/mailbox/imx-mailbox.c=184=static int imx_mu_rx_waiting_read(struct imx_mu_priv *priv, u32 *val, u32 idx)
 drivers/mailbox/imx-mailbox.c-185-{
 drivers/mailbox/imx-mailbox.c:186:	u64 timeout_time = get_jiffies_64() + IMX_MU_SECO_RX_TOUT;
 drivers/mailbox/imx-mailbox.c-187-	u32 status;
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c=115=static void vidtv_mux_update_clk(struct vidtv_mux *m)
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c-120-	m->timing.past_jiffies = m->timing.current_jiffies;
 drivers/media/test-drivers/vidtv/vidtv_mux.c:121:	m->timing.current_jiffies = get_jiffies_64();
 drivers/media/test-drivers/vidtv/vidtv_mux.c-122-
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c=457=void vidtv_mux_start_thread(struct vidtv_mux *m)
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c-464-	m->streaming = true;
 drivers/media/test-drivers/vidtv/vidtv_mux.c:465:	m->timing.start_jiffies = get_jiffies_64();
 drivers/media/test-drivers/vidtv/vidtv_mux.c-466-	schedule_work(&m->mpeg_thread);
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c=477=struct vidtv_mux *vidtv_mux_init(struct dvb_frontend *fe,
 --
 drivers/media/test-drivers/vidtv/vidtv_mux.c-509-
 drivers/media/test-drivers/vidtv/vidtv_mux.c:510:	m->timing.current_jiffies = get_jiffies_64();
 drivers/media/test-drivers/vidtv/vidtv_mux.c-511-
 --
 drivers/media/test-drivers/visl/visl-dec.c=270=static void visl_tpg_fill_sequence(struct visl_ctx *ctx,
 --
 drivers/media/test-drivers/visl/visl-dec.c-276-	if (tpg_verbose) {
 drivers/media/test-drivers/visl/visl-dec.c:277:		stream_ms = jiffies_to_msecs(get_jiffies_64() - ctx->capture_streamon_jiffies);
 drivers/media/test-drivers/visl/visl-dec.c-278-
 --
 drivers/media/test-drivers/visl/visl-video.c=672=static int visl_start_streaming(struct vb2_queue *vq, unsigned int count)
 --
 drivers/media/test-drivers/visl/visl-video.c-685-	if (V4L2_TYPE_IS_CAPTURE(vq->type)) {
 drivers/media/test-drivers/visl/visl-video.c:686:		ctx->capture_streamon_jiffies = get_jiffies_64();
 drivers/media/test-drivers/visl/visl-video.c-687-		return 0;
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c=182=ice_arfs_is_flow_expired(struct ice_vsi *vsi, struct ice_arfs_entry *arfs_entry)
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c-196-			       ICE_ARFS_TIME_DELTA_EXPIRATION,
 drivers/net/ethernet/intel/ice/ice_arfs.c:197:			       arfs_entry->time_activated, get_jiffies_64());
 drivers/net/ethernet/intel/ice/ice_arfs.c-198-}
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c=215=ice_arfs_update_flow_rules(struct ice_vsi *vsi, u16 idx,
 --
 drivers/net/ethernet/intel/ice/ice_arfs.c-242-			    flow_type == ICE_FLTR_PTYPE_NONF_IPV6_UDP)
 drivers/net/ethernet/intel/ice/ice_arfs.c:243:				e->time_activated = get_jiffies_64();
 drivers/net/ethernet/intel/ice/ice_arfs.c-244-		} else if (e->fltr_state == ICE_ARFS_ACTIVE) {
 --
 drivers/net/ethernet/sun/niu.c=4975=static int niu_init_rx_channels(struct niu *np)
 --
 drivers/net/ethernet/sun/niu.c-4977-	unsigned long flags;
 drivers/net/ethernet/sun/niu.c:4978:	u64 seed = jiffies_64;
 drivers/net/ethernet/sun/niu.c-4979-	int err, i;
 --
 drivers/net/phy/mxl-gpy.c=842=static int gpy_loopback(struct phy_device *phydev, bool enable, int speed)
 --
 drivers/net/phy/mxl-gpy.c-848-	if (enable) {
 drivers/net/phy/mxl-gpy.c:849:		u64 now = get_jiffies_64();
 drivers/net/phy/mxl-gpy.c-850-
 --
 drivers/net/phy/mxl-gpy.c-870-	} else {
 drivers/net/phy/mxl-gpy.c:871:		priv->lb_dis_to = get_jiffies_64() + HZ * 3;
 drivers/net/phy/mxl-gpy.c-872-	}
 --
 drivers/net/wireless/ath/ath10k/debug.c=866=static int ath10k_debug_htt_stats_req(struct ath10k *ar)
 --
 drivers/net/wireless/ath/ath10k/debug.c-879-
 drivers/net/wireless/ath/ath10k/debug.c:880:	cookie = get_jiffies_64();
 drivers/net/wireless/ath/ath10k/debug.c-881-
 --
 drivers/net/xen-netback/interface.c=561=int xenvif_init_queue(struct xenvif_queue *queue)
 --
 drivers/net/xen-netback/interface.c-567-	timer_setup(&queue->credit_timeout, xenvif_tx_credit_callback, 0);
 drivers/net/xen-netback/interface.c:568:	queue->credit_window_start = get_jiffies_64();
 drivers/net/xen-netback/interface.c-569-
 --
 drivers/net/xen-netback/netback.c=811=static bool tx_credit_exceeded(struct xenvif_queue *queue, unsigned size)
 drivers/net/xen-netback/netback.c-812-{
 drivers/net/xen-netback/netback.c:813:	u64 now = get_jiffies_64();
 drivers/net/xen-netback/netback.c-814-	u64 next_credit = queue->credit_window_start +
 --
 drivers/platform/x86/intel_ips.c=954=static int ips_monitor(void *data)
 --
 drivers/platform/x86/intel_ips.c-983-		ITV_ME_SEQNO_SHIFT;
 drivers/platform/x86/intel_ips.c:984:	seqno_timestamp = get_jiffies_64();
 drivers/platform/x86/intel_ips.c-985-
 --
 drivers/platform/x86/intel_ips.c-1093-		} else {
 drivers/platform/x86/intel_ips.c:1094:			seqno_timestamp = get_jiffies_64();
 drivers/platform/x86/intel_ips.c-1095-			last_seqno = cur_seqno;
]
]

gemini-3-flash-preview

							input: 83492
							output: 23
							thoughts: 23
						

Okay, here's my interpretation of that thought process, tailored for an expert audience, and presented as if it were my own:

**Investigating Potential `jiffies_64` Usage in Memory Management**

Alright, I'm thinking about tracking down how `jiffies_64` is being used in the memory management subsystem. To start, the most logical initial step is to use `grep` (or my preferred search tool) and restrict the search specifically to the `mm/` directory. That seems like the most relevant place to find related code since that's where all the memory management related things are located. I'm focusing on that, instead of a blanket search, to reduce the noise and focus the search. Hopefully this will turn up something interesting.

no matches

map[Expression:jiffies_64.*mm/]

map[Output:]

gemini-3-flash-preview

							input: 83715
							output: 22
							thoughts: 16
						

Okay, here's my attempt at summarizing that thought process, tailored for an expert audience:

**Investigating Potential `jiffies_64` Usage**

Right, so I need to find all instances where `jiffies_64` is being used, specifically within the memory management subsystem. A `git grep` seems like the most efficient initial approach here. I'm going to start with `git grep jiffies_64 mm/`. That should give me a pretty comprehensive overview of where it's being referenced within that particular part of the kernel codebase. This will immediately help me to narrow down the context of the uses and help to better focus my subsequent investigations. Hopefully this isn't a rabbit hole, but it seems like a reasonable first step.

no matches

map[Expression:jiffies_64 mm/]

map[Output:]

gemini-3-flash-preview

							input: 83929
							output: 21
							thoughts: 68
						

Okay, here's my quick rundown:

**My Immediate Assessment**

Alright, I need to find something. My first instinct, as always, is to leverage `codesearch-dir-index`. I'll point it at `mm/` to get a broad overview. I know I've seen a relevant usage in `mm/memcontrol.c` before.

Hang on. I think that I already remember that `mm/memcontrol.c` uses it. Maybe it will have the specific thing I need.

Just to be absolutely sure, I'll double-check `mm/memcontrol.c` specifically for `jiffies_64`. Gotta be thorough.

map[File:mm/memcontrol.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-or-later
/* memcontrol.c - Memory Controller
 *
 * Copyright IBM Corporation, 2007
 * Author Balbir Singh <balbir@linux.vnet.ibm.com>
 *
 * Copyright 2007 OpenVZ SWsoft Inc
 * Author: Pavel Emelianov <xemul@openvz.org>
 *
 * Memory thresholds
 * Copyright (C) 2009 Nokia Corporation
 * Author: Kirill A. Shutemov
 *
 * Kernel Memory Controller
 * Copyright (C) 2012 Parallels Inc. and Google Inc.
 * Authors: Glauber Costa and Suleiman Souhlal
 *
 * Native page reclaim
 * Charge lifetime sanitation
 * Lockless page tracking & accounting
 * Unified hierarchy configuration model
 * Copyright (C) 2015 Red Hat, Inc., Johannes Weiner
 *
 * Per memcg lru locking
 * Copyright (C) 2020 Alibaba, Inc, Alex Shi
 */

#include <linux/cgroup-defs.h>
#include <linux/page_counter.h>
#include <linux/memcontrol.h>
#include <linux/cgroup.h>
#include <linux/cpuset.h>
#include <linux/sched/mm.h>
#include <linux/shmem_fs.h>
#include <linux/hugetlb.h>
#include <linux/pagemap.h>
#include <linux/pagevec.h>
#include <linux/vm_event_item.h>
#include <linux/smp.h>
#include <linux/page-flags.h>
#include <linux/backing-dev.h>
#include <linux/bit_spinlock.h>
#include <linux/rcupdate.h>
#include <linux/limits.h>
#include <linux/export.h>
#include <linux/list.h>
#include <linux/mutex.h>
#include <linux/rbtree.h>
#include <linux/slab.h>
#include <linux/swapops.h>
#include <linux/spinlock.h>
#include <linux/fs.h>
#include <linux/seq_file.h>
#include <linux/vmpressure.h>
#include <linux/memremap.h>
#include <linux/mm_inline.h>
#include <linux/swap_cgroup.h>
#include <linux/cpu.h>
#include <linux/oom.h>
#include <linux/lockdep.h>
#include <linux/resume_user_mode.h>
#include <linux/psi.h>
#include <linux/seq_buf.h>
#include <linux/sched/isolation.h>
#include <linux/kmemleak.h>
#include "internal.h"
#include <net/sock.h>
#include <net/ip.h>
#include "slab.h"
#include "memcontrol-v1.h"

#include <linux/uaccess.h>

#define CREATE_TRACE_POINTS
#include <trace/events/memcg.h>
#undef CREATE_TRACE_POINTS

#include <trace/events/vmscan.h>

struct cgroup_subsys memory_cgrp_subsys __read_mostly;
EXPORT_SYMBOL(memory_cgrp_subsys);

struct mem_cgroup *root_mem_cgroup __read_mostly;
EXPORT_SYMBOL(root_mem_cgroup);

/* Active memory cgroup to use from an interrupt context */
DEFINE_PER_CPU(struct mem_cgroup *, int_active_memcg);
EXPORT_PER_CPU_SYMBOL_GPL(int_active_memcg);

/* Socket memory accounting disabled? */
static bool cgroup_memory_nosocket __ro_after_init;

/* Kernel memory accounting disabled? */
static bool cgroup_memory_nokmem __ro_after_init;

/* BPF memory accounting disabled? */
static bool cgroup_memory_nobpf __ro_after_init;

static struct workqueue_struct *memcg_wq __ro_after_init;

static struct kmem_cache *memcg_cachep;
static struct kmem_cache *memcg_pn_cachep;

#ifdef CONFIG_CGROUP_WRITEBACK
static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq);
#endif

static inline bool task_is_dying(void)
{
	return tsk_is_oom_victim(current) || fatal_signal_pending(current) ||
		(current->flags & PF_EXITING);
}

/* Some nice accessors for the vmpressure. */
struct vmpressure *memcg_to_vmpressure(struct mem_cgroup *memcg)
{
	if (!memcg)
		memcg = root_mem_cgroup;
	return &memcg->vmpressure;
}

struct mem_cgroup *vmpressure_to_memcg(struct vmpressure *vmpr)
{
	return container_of(vmpr, struct mem_cgroup, vmpressure);
}

#define SEQ_BUF_SIZE SZ_4K
#define CURRENT_OBJCG_UPDATE_BIT 0
#define CURRENT_OBJCG_UPDATE_FLAG (1UL << CURRENT_OBJCG_UPDATE_BIT)

static DEFINE_SPINLOCK(objcg_lock);

bool mem_cgroup_kmem_disabled(void)
{
	return cgroup_memory_nokmem;
}

static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages);

static void obj_cgroup_release(struct percpu_ref *ref)
{
	struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt);
	unsigned int nr_bytes;
	unsigned int nr_pages;
	unsigned long flags;

	/*
	 * At this point all allocated objects are freed, and
	 * objcg->nr_charged_bytes can't have an arbitrary byte value.
	 * However, it can be PAGE_SIZE or (x * PAGE_SIZE).
	 *
	 * The following sequence can lead to it:
	 * 1) CPU0: objcg == stock->cached_objcg
	 * 2) CPU1: we do a small allocation (e.g. 92 bytes),
	 *          PAGE_SIZE bytes are charged
	 * 3) CPU1: a process from another memcg is allocating something,
	 *          the stock if flushed,
	 *          objcg->nr_charged_bytes = PAGE_SIZE - 92
	 * 5) CPU0: we do release this object,
	 *          92 bytes are added to stock->nr_bytes
	 * 6) CPU0: stock is flushed,
	 *          92 bytes are added to objcg->nr_charged_bytes
	 *
	 * In the result, nr_charged_bytes == PAGE_SIZE.
	 * This page will be uncharged in obj_cgroup_release().
	 */
	nr_bytes = atomic_read(&objcg->nr_charged_bytes);
	WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1));
	nr_pages = nr_bytes >> PAGE_SHIFT;

	if (nr_pages) {
		struct mem_cgroup *memcg;

		memcg = get_mem_cgroup_from_objcg(objcg);
		mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages);
		memcg1_account_kmem(memcg, -nr_pages);
		if (!mem_cgroup_is_root(memcg))
			memcg_uncharge(memcg, nr_pages);
		mem_cgroup_put(memcg);
	}

	spin_lock_irqsave(&objcg_lock, flags);
	list_del(&objcg->list);
	spin_unlock_irqrestore(&objcg_lock, flags);

	percpu_ref_exit(ref);
	kfree_rcu(objcg, rcu);
}

static struct obj_cgroup *obj_cgroup_alloc(void)
{
	struct obj_cgroup *objcg;
	int ret;

	objcg = kzalloc_obj(struct obj_cgroup);
	if (!objcg)
		return NULL;

	ret = percpu_ref_init(&objcg->refcnt, obj_cgroup_release, 0,
			      GFP_KERNEL);
	if (ret) {
		kfree(objcg);
		return NULL;
	}
	INIT_LIST_HEAD(&objcg->list);
	return objcg;
}

static void memcg_reparent_objcgs(struct mem_cgroup *memcg,
				  struct mem_cgroup *parent)
{
	struct obj_cgroup *objcg, *iter;

	objcg = rcu_replace_pointer(memcg->objcg, NULL, true);

	spin_lock_irq(&objcg_lock);

	/* 1) Ready to reparent active objcg. */
	list_add(&objcg->list, &memcg->objcg_list);
	/* 2) Reparent active objcg and already reparented objcgs to parent. */
	list_for_each_entry(iter, &memcg->objcg_list, list)
		WRITE_ONCE(iter->memcg, parent);
	/* 3) Move already reparented objcgs to the parent's list */
	list_splice(&memcg->objcg_list, &parent->objcg_list);

	spin_unlock_irq(&objcg_lock);

	percpu_ref_kill(&objcg->refcnt);
}

/*
 * A lot of the calls to the cache allocation functions are expected to be
 * inlined by the compiler. Since the calls to memcg_slab_post_alloc_hook() are
 * conditional to this static branch, we'll have to allow modules that does
 * kmem_cache_alloc and the such to see this symbol as well
 */
DEFINE_STATIC_KEY_FALSE(memcg_kmem_online_key);
EXPORT_SYMBOL(memcg_kmem_online_key);

DEFINE_STATIC_KEY_FALSE(memcg_bpf_enabled_key);
EXPORT_SYMBOL(memcg_bpf_enabled_key);

/**
 * mem_cgroup_css_from_folio - css of the memcg associated with a folio
 * @folio: folio of interest
 *
 * If memcg is bound to the default hierarchy, css of the memcg associated
 * with @folio is returned.  The returned css remains associated with @folio
 * until it is released.
 *
 * If memcg is bound to a traditional hierarchy, the css of root_mem_cgroup
 * is returned.
 */
struct cgroup_subsys_state *mem_cgroup_css_from_folio(struct folio *folio)
{
	struct mem_cgroup *memcg = folio_memcg(folio);

	if (!memcg || !cgroup_subsys_on_dfl(memory_cgrp_subsys))
		memcg = root_mem_cgroup;

	return &memcg->css;
}

/**
 * page_cgroup_ino - return inode number of the memcg a page is charged to
 * @page: the page
 *
 * Look up the closest online ancestor of the memory cgroup @page is charged to
 * and return its inode number or 0 if @page is not charged to any cgroup. It
 * is safe to call this function without holding a reference to @page.
 *
 * Note, this function is inherently racy, because there is nothing to prevent
 * the cgroup inode from getting torn down and potentially reallocated a moment
 * after page_cgroup_ino() returns, so it only should be used by callers that
 * do not care (such as procfs interfaces).
 */
ino_t page_cgroup_ino(struct page *page)
{
	struct mem_cgroup *memcg;
	unsigned long ino = 0;

	rcu_read_lock();
	/* page_folio() is racy here, but the entire function is racy anyway */
	memcg = folio_memcg_check(page_folio(page));

	while (memcg && !css_is_online(&memcg->css))
		memcg = parent_mem_cgroup(memcg);
	if (memcg)
		ino = cgroup_ino(memcg->css.cgroup);
	rcu_read_unlock();
	return ino;
}
EXPORT_SYMBOL_GPL(page_cgroup_ino);

/* Subset of node_stat_item for memcg stats */
static const unsigned int memcg_node_stat_items[] = {
	NR_INACTIVE_ANON,
	NR_ACTIVE_ANON,
	NR_INACTIVE_FILE,
	NR_ACTIVE_FILE,
	NR_UNEVICTABLE,
	NR_SLAB_RECLAIMABLE_B,
	NR_SLAB_UNRECLAIMABLE_B,
	WORKINGSET_REFAULT_ANON,
	WORKINGSET_REFAULT_FILE,
	WORKINGSET_ACTIVATE_ANON,
	WORKINGSET_ACTIVATE_FILE,
	WORKINGSET_RESTORE_ANON,
	WORKINGSET_RESTORE_FILE,
	WORKINGSET_NODERECLAIM,
	NR_ANON_MAPPED,
	NR_FILE_MAPPED,
	NR_FILE_PAGES,
	NR_FILE_DIRTY,
	NR_WRITEBACK,
	NR_SHMEM,
	NR_SHMEM_THPS,
	NR_FILE_THPS,
	NR_ANON_THPS,
	NR_KERNEL_STACK_KB,
	NR_PAGETABLE,
	NR_SECONDARY_PAGETABLE,
#ifdef CONFIG_SWAP
	NR_SWAPCACHE,
#endif
#ifdef CONFIG_NUMA_BALANCING
	PGPROMOTE_SUCCESS,
#endif
	PGDEMOTE_KSWAPD,
	PGDEMOTE_DIRECT,
	PGDEMOTE_KHUGEPAGED,
	PGDEMOTE_PROACTIVE,
#ifdef CONFIG_HUGETLB_PAGE
	NR_HUGETLB,
#endif
};

static const unsigned int memcg_stat_items[] = {
	MEMCG_SWAP,
	MEMCG_SOCK,
	MEMCG_PERCPU_B,
	MEMCG_VMALLOC,
	MEMCG_KMEM,
	MEMCG_ZSWAP_B,
	MEMCG_ZSWAPPED,
};

#define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items)
#define MEMCG_VMSTAT_SIZE (NR_MEMCG_NODE_STAT_ITEMS + \
			   ARRAY_SIZE(memcg_stat_items))
#define BAD_STAT_IDX(index) ((u32)(index) >= U8_MAX)
static u8 mem_cgroup_stats_index[MEMCG_NR_STAT] __read_mostly;

static void init_memcg_stats(void)
{
	u8 i, j = 0;

	BUILD_BUG_ON(MEMCG_NR_STAT >= U8_MAX);

	memset(mem_cgroup_stats_index, U8_MAX, sizeof(mem_cgroup_stats_index));

	for (i = 0; i < NR_MEMCG_NODE_STAT_ITEMS; ++i, ++j)
		mem_cgroup_stats_index[memcg_node_stat_items[i]] = j;

	for (i = 0; i < ARRAY_SIZE(memcg_stat_items); ++i, ++j)
		mem_cgroup_stats_index[memcg_stat_items[i]] = j;
}

static inline int memcg_stats_index(int idx)
{
	return mem_cgroup_stats_index[idx];
}

struct lruvec_stats_percpu {
	/* Local (CPU and cgroup) state */
	long state[NR_MEMCG_NODE_STAT_ITEMS];

	/* Delta calculation for lockless upward propagation */
	long state_prev[NR_MEMCG_NODE_STAT_ITEMS];
};

struct lruvec_stats {
	/* Aggregated (CPU and subtree) state */
	long state[NR_MEMCG_NODE_STAT_ITEMS];

	/* Non-hierarchical (CPU aggregated) state */
	long state_local[NR_MEMCG_NODE_STAT_ITEMS];

	/* Pending child counts during tree propagation */
	long state_pending[NR_MEMCG_NODE_STAT_ITEMS];
};

unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx)
{
	struct mem_cgroup_per_node *pn;
	long x;
	int i;

	if (mem_cgroup_disabled())
		return node_page_state(lruvec_pgdat(lruvec), idx);

	i = memcg_stats_index(idx);
	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
		return 0;

	pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
	x = READ_ONCE(pn->lruvec_stats->state[i]);
#ifdef CONFIG_SMP
	if (x < 0)
		x = 0;
#endif
	return x;
}

unsigned long lruvec_page_state_local(struct lruvec *lruvec,
				      enum node_stat_item idx)
{
	struct mem_cgroup_per_node *pn;
	long x;
	int i;

	if (mem_cgroup_disabled())
		return node_page_state(lruvec_pgdat(lruvec), idx);

	i = memcg_stats_index(idx);
	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
		return 0;

	pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
	x = READ_ONCE(pn->lruvec_stats->state_local[i]);
#ifdef CONFIG_SMP
	if (x < 0)
		x = 0;
#endif
	return x;
}

/* Subset of vm_event_item to report for memcg event stats */
static const unsigned int memcg_vm_event_stat[] = {
#ifdef CONFIG_MEMCG_V1
	PGPGIN,
	PGPGOUT,
#endif
	PSWPIN,
	PSWPOUT,
	PGSCAN_KSWAPD,
	PGSCAN_DIRECT,
	PGSCAN_KHUGEPAGED,
	PGSCAN_PROACTIVE,
	PGSTEAL_KSWAPD,
	PGSTEAL_DIRECT,
	PGSTEAL_KHUGEPAGED,
	PGSTEAL_PROACTIVE,
	PGFAULT,
	PGMAJFAULT,
	PGREFILL,
	PGACTIVATE,
	PGDEACTIVATE,
	PGLAZYFREE,
	PGLAZYFREED,
#ifdef CONFIG_SWAP
	SWPIN_ZERO,
	SWPOUT_ZERO,
#endif
#ifdef CONFIG_ZSWAP
	ZSWPIN,
	ZSWPOUT,
	ZSWPWB,
#endif
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	THP_FAULT_ALLOC,
	THP_COLLAPSE_ALLOC,
	THP_SWPOUT,
	THP_SWPOUT_FALLBACK,
#endif
#ifdef CONFIG_NUMA_BALANCING
	NUMA_PAGE_MIGRATE,
	NUMA_PTE_UPDATES,
	NUMA_HINT_FAULTS,
#endif
};

#define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat)
static u8 mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly;

static void init_memcg_events(void)
{
	u8 i;

	BUILD_BUG_ON(NR_VM_EVENT_ITEMS >= U8_MAX);

	memset(mem_cgroup_events_index, U8_MAX,
	       sizeof(mem_cgroup_events_index));

	for (i = 0; i < NR_MEMCG_EVENTS; ++i)
		mem_cgroup_events_index[memcg_vm_event_stat[i]] = i;
}

static inline int memcg_events_index(enum vm_event_item idx)
{
	return mem_cgroup_events_index[idx];
}

struct memcg_vmstats_percpu {
	/* Stats updates since the last flush */
	unsigned int			stats_updates;

	/* Cached pointers for fast iteration in memcg_rstat_updated() */
	struct memcg_vmstats_percpu __percpu	*parent_pcpu;
	struct memcg_vmstats			*vmstats;

	/* The above should fit a single cacheline for memcg_rstat_updated() */

	/* Local (CPU and cgroup) page state & events */
	long			state[MEMCG_VMSTAT_SIZE];
	unsigned long		events[NR_MEMCG_EVENTS];

	/* Delta calculation for lockless upward propagation */
	long			state_prev[MEMCG_VMSTAT_SIZE];
	unsigned long		events_prev[NR_MEMCG_EVENTS];
} ____cacheline_aligned;

struct memcg_vmstats {
	/* Aggregated (CPU and subtree) page state & events */
	long			state[MEMCG_VMSTAT_SIZE];
	unsigned long		events[NR_MEMCG_EVENTS];

	/* Non-hierarchical (CPU aggregated) page state & events */
	long			state_local[MEMCG_VMSTAT_SIZE];
	unsigned long		events_local[NR_MEMCG_EVENTS];

	/* Pending child counts during tree propagation */
	long			state_pending[MEMCG_VMSTAT_SIZE];
	unsigned long		events_pending[NR_MEMCG_EVENTS];

	/* Stats updates since the last flush */
	atomic_t		stats_updates;
};

/*
 * memcg and lruvec stats flushing
 *
 * Many codepaths leading to stats update or read are performance sensitive and
 * adding stats flushing in such codepaths is not desirable. So, to optimize the
 * flushing the kernel does:
 *
 * 1) Periodically and asynchronously flush the stats every 2 seconds to not let
 *    rstat update tree grow unbounded.
 *
 * 2) Flush the stats synchronously on reader side only when there are more than
 *    (MEMCG_CHARGE_BATCH * nr_cpus) update events. Though this optimization
 *    will let stats be out of sync by atmost (MEMCG_CHARGE_BATCH * nr_cpus) but
 *    only for 2 seconds due to (1).
 */
static void flush_memcg_stats_dwork(struct work_struct *w);
static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork);
static u64 flush_last_time;

#define FLUSH_TIME (2UL*HZ)

static bool memcg_vmstats_needs_flush(struct memcg_vmstats *vmstats)
{
	return atomic_read(&vmstats->stats_updates) >
		MEMCG_CHARGE_BATCH * num_online_cpus();
}

static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val,
				       int cpu)
{
	struct memcg_vmstats_percpu __percpu *statc_pcpu;
	struct memcg_vmstats_percpu *statc;
	unsigned int stats_updates;

	if (!val)
		return;

	css_rstat_updated(&memcg->css, cpu);
	statc_pcpu = memcg->vmstats_percpu;
	for (; statc_pcpu; statc_pcpu = statc->parent_pcpu) {
		statc = this_cpu_ptr(statc_pcpu);
		/*
		 * If @memcg is already flushable then all its ancestors are
		 * flushable as well and also there is no need to increase
		 * stats_updates.
		 */
		if (memcg_vmstats_needs_flush(statc->vmstats))
			break;

		stats_updates = this_cpu_add_return(statc_pcpu->stats_updates,
						    abs(val));
		if (stats_updates < MEMCG_CHARGE_BATCH)
			continue;

		stats_updates = this_cpu_xchg(statc_pcpu->stats_updates, 0);
		atomic_add(stats_updates, &statc->vmstats->stats_updates);
	}
}

static void __mem_cgroup_flush_stats(struct mem_cgroup *memcg, bool force)
{
	bool needs_flush = memcg_vmstats_needs_flush(memcg->vmstats);

	trace_memcg_flush_stats(memcg, atomic_read(&memcg->vmstats->stats_updates),
		force, needs_flush);

	if (!force && !needs_flush)
		return;

	if (mem_cgroup_is_root(memcg))
		WRITE_ONCE(flush_last_time, jiffies_64);

	css_rstat_flush(&memcg->css);
}

/*
 * mem_cgroup_flush_stats - flush the stats of a memory cgroup subtree
 * @memcg: root of the subtree to flush
 *
 * Flushing is serialized by the underlying global rstat lock. There is also a
 * minimum amount of work to be done even if there are no stat updates to flush.
 * Hence, we only flush the stats if the updates delta exceeds a threshold. This
 * avoids unnecessary work and contention on the underlying lock.
 */
void mem_cgroup_flush_stats(struct mem_cgroup *memcg)
{
	if (mem_cgroup_disabled())
		return;

	if (!memcg)
		memcg = root_mem_cgroup;

	__mem_cgroup_flush_stats(memcg, false);
}

void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
{
	/* Only flush if the periodic flusher is one full cycle late */
	if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
		mem_cgroup_flush_stats(memcg);
}

static void flush_memcg_stats_dwork(struct work_struct *w)
{
	/*
	 * Deliberately ignore memcg_vmstats_needs_flush() here so that flushing
	 * in latency-sensitive paths is as cheap as possible.
	 */
	__mem_cgroup_flush_stats(root_mem_cgroup, true);
	queue_delayed_work(system_dfl_wq, &stats_flush_dwork, FLUSH_TIME);
}

unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx)
{
	long x;
	int i = memcg_stats_index(idx);

	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
		return 0;

	x = READ_ONCE(memcg->vmstats->state[i]);
#ifdef CONFIG_SMP
	if (x < 0)
		x = 0;
#endif
	return x;
}

bool memcg_stat_item_valid(int idx)
{
	if ((u32)idx >= MEMCG_NR_STAT)
		return false;

	return !BAD_STAT_IDX(memcg_stats_index(idx));
}

static int memcg_page_state_unit(int item);

/*
 * Normalize the value passed into memcg_rstat_updated() to be in pages. Round
 * up non-zero sub-page updates to 1 page as zero page updates are ignored.
 */
static int memcg_state_val_in_pages(int idx, int val)
{
	int unit = memcg_page_state_unit(idx);

	if (!val || unit == PAGE_SIZE)
		return val;
	else
		return max(val * unit / PAGE_SIZE, 1UL);
}

/**
 * mod_memcg_state - update cgroup memory statistics
 * @memcg: the memory cgroup
 * @idx: the stat item - can be enum memcg_stat_item or enum node_stat_item
 * @val: delta to add to the counter, can be negative
 */
void mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx,
		       int val)
{
	int i = memcg_stats_index(idx);
	int cpu;

	if (mem_cgroup_disabled())
		return;

	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
		return;

	cpu = get_cpu();

	this_cpu_add(memcg->vmstats_percpu->state[i], val);
	val = memcg_state_val_in_pages(idx, val);
	memcg_rstat_updated(memcg, val, cpu);
	trace_mod_memcg_state(memcg, idx, val);

	put_cpu();
}

#ifdef CONFIG_MEMCG_V1
/* idx can be of type enum memcg_stat_item or node_stat_item. */
unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx)
{
	long x;
	int i = memcg_stats_index(idx);

	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
		return 0;

	x = READ_ONCE(memcg->vmstats->state_local[i]);
#ifdef CONFIG_SMP
	if (x < 0)
		x = 0;
#endif
	return x;
}
#endif

static void mod_memcg_lruvec_state(struct lruvec *lruvec,
				     enum node_stat_item idx,
				     int val)
{
	struct mem_cgroup_per_node *pn;
	struct mem_cgroup *memcg;
	int i = memcg_stats_index(idx);
	int cpu;

	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
		return;

	pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
	memcg = pn->memcg;

	cpu = get_cpu();

	/* Update memcg */
	this_cpu_add(memcg->vmstats_percpu->state[i], val);

	/* Update lruvec */
	this_cpu_add(pn->lruvec_stats_percpu->state[i], val);

	val = memcg_state_val_in_pages(idx, val);
	memcg_rstat_updated(memcg, val, cpu);
	trace_mod_memcg_lruvec_state(memcg, idx, val);

	put_cpu();
}

/**
 * mod_lruvec_state - update lruvec memory statistics
 * @lruvec: the lruvec
 * @idx: the stat item
 * @val: delta to add to the counter, can be negative
 *
 * The lruvec is the intersection of the NUMA node and a cgroup. This
 * function updates the all three counters that are affected by a
 * change of state at this level: per-node, per-cgroup, per-lruvec.
 */
void mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx,
			int val)
{
	/* Update node */
	mod_node_page_state(lruvec_pgdat(lruvec), idx, val);

	/* Update memcg and lruvec */
	if (!mem_cgroup_disabled())
		mod_memcg_lruvec_state(lruvec, idx, val);
}

void lruvec_stat_mod_folio(struct folio *folio, enum node_stat_item idx,
			     int val)
{
	struct mem_cgroup *memcg;
	pg_data_t *pgdat = folio_pgdat(folio);
	struct lruvec *lruvec;

	rcu_read_lock();
	memcg = folio_memcg(folio);
	/* Untracked pages have no memcg, no lruvec. Update only the node */
	if (!memcg) {
		rcu_read_unlock();
		mod_node_page_state(pgdat, idx, val);
		return;
	}

	lruvec = mem_cgroup_lruvec(memcg, pgdat);
	mod_lruvec_state(lruvec, idx, val);
	rcu_read_unlock();
}
EXPORT_SYMBOL(lruvec_stat_mod_folio);

void mod_lruvec_kmem_state(void *p, enum node_stat_item idx, int val)
{
	pg_data_t *pgdat = page_pgdat(virt_to_page(p));
	struct mem_cgroup *memcg;
	struct lruvec *lruvec;

	rcu_read_lock();
	memcg = mem_cgroup_from_virt(p);

	/*
	 * Untracked pages have no memcg, no lruvec. Update only the
	 * node. If we reparent the slab objects to the root memcg,
	 * when we free the slab object, we need to update the per-memcg
	 * vmstats to keep it correct for the root memcg.
	 */
	if (!memcg) {
		mod_node_page_state(pgdat, idx, val);
	} else {
		lruvec = mem_cgroup_lruvec(memcg, pgdat);
		mod_lruvec_state(lruvec, idx, val);
	}
	rcu_read_unlock();
}

/**
 * count_memcg_events - account VM events in a cgroup
 * @memcg: the memory cgroup
 * @idx: the event item
 * @count: the number of events that occurred
 */
void count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx,
			  unsigned long count)
{
	int i = memcg_events_index(idx);
	int cpu;

	if (mem_cgroup_disabled())
		return;

	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx))
		return;

	cpu = get_cpu();

	this_cpu_add(memcg->vmstats_percpu->events[i], count);
	memcg_rstat_updated(memcg, count, cpu);
	trace_count_memcg_events(memcg, idx, count);

	put_cpu();
}

unsigned long memcg_events(struct mem_cgroup *memcg, int event)
{
	int i = memcg_events_index(event);

	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, event))
		return 0;

	return READ_ONCE(memcg->vmstats->events[i]);
}

bool memcg_vm_event_item_valid(enum vm_event_item idx)
{
	if (idx >= NR_VM_EVENT_ITEMS)
		return false;

	return !BAD_STAT_IDX(memcg_events_index(idx));
}

#ifdef CONFIG_MEMCG_V1
unsigned long memcg_events_local(struct mem_cgroup *memcg, int event)
{
	int i = memcg_events_index(event);

	if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, event))
		return 0;

	return READ_ONCE(memcg->vmstats->events_local[i]);
}
#endif

struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p)
{
	/*
	 * mm_update_next_owner() may clear mm->owner to NULL
	 * if it races with swapoff, page migration, etc.
	 * So this can be called with p == NULL.
	 */
	if (unlikely(!p))
		return NULL;

	return mem_cgroup_from_css(task_css(p, memory_cgrp_id));
}
EXPORT_SYMBOL(mem_cgroup_from_task);

static __always_inline struct mem_cgroup *active_memcg(void)
{
	if (!in_task())
		return this_cpu_read(int_active_memcg);
	else
		return current->active_memcg;
}

/**
 * get_mem_cgroup_from_mm: Obtain a reference on given mm_struct's memcg.
 * @mm: mm from which memcg should be extracted. It can be NULL.
 *
 * Obtain a reference on mm->memcg and returns it if successful. If mm
 * is NULL, then the memcg is chosen as follows:
 * 1) The active memcg, if set.
 * 2) current->mm->memcg, if available
 * 3) root memcg
 * If mem_cgroup is disabled, NULL is returned.
 */
struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm)
{
	struct mem_cgroup *memcg;

	if (mem_cgroup_disabled())
		return NULL;

	/*
	 * Page cache insertions can happen without an
	 * actual mm context, e.g. during disk probing
	 * on boot, loopback IO, acct() writes etc.
	 *
	 * No need to css_get on root memcg as the reference
	 * counting is disabled on the root level in the
	 * cgroup core. See CSS_NO_REF.
	 */
	if (unlikely(!mm)) {
		memcg = active_memcg();
		if (unlikely(memcg)) {
			/* remote memcg must hold a ref */
			css_get(&memcg->css);
			return memcg;
		}
		mm = current->mm;
		if (unlikely(!mm))
			return root_mem_cgroup;
	}

	rcu_read_lock();
	do {
		memcg = mem_cgroup_from_task(rcu_dereference(mm->owner));
		if (unlikely(!memcg))
			memcg = root_mem_cgroup;
	} while (!css_tryget(&memcg->css));
	rcu_read_unlock();
	return memcg;
}
EXPORT_SYMBOL(get_mem_cgroup_from_mm);

/**
 * get_mem_cgroup_from_current - Obtain a reference on current task's memcg.
 */
struct mem_cgroup *get_mem_cgroup_from_current(void)
{
	struct mem_cgroup *memcg;

	if (mem_cgroup_disabled())
		return NULL;

again:
	rcu_read_lock();
	memcg = mem_cgroup_from_task(current);
	if (!css_tryget(&memcg->css)) {
		rcu_read_unlock();
		goto again;
	}
	rcu_read_unlock();
	return memcg;
}

/**
 * get_mem_cgroup_from_folio - Obtain a reference on a given folio's memcg.
 * @folio: folio from which memcg should be extracted.
 */
struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *folio)
{
	struct mem_cgroup *memcg = folio_memcg(folio);

	if (mem_cgroup_disabled())
		return NULL;

	rcu_read_lock();
	if (!memcg || WARN_ON_ONCE(!css_tryget(&memcg->css)))
		memcg = root_mem_cgroup;
	rcu_read_unlock();
	return memcg;
}

/**
 * mem_cgroup_iter - iterate over memory cgroup hierarchy
 * @root: hierarchy root
 * @prev: previously returned memcg, NULL on first invocation
 * @reclaim: cookie for shared reclaim walks, NULL for full walks
 *
 * Returns references to children of the hierarchy below @root, or
 * @root itself, or %NULL after a full round-trip.
 *
 * Caller must pass the return value in @prev on subsequent
 * invocations for reference counting, or use mem_cgroup_iter_break()
 * to cancel a hierarchy walk before the round-trip is complete.
 *
 * Reclaimers can specify a node in @reclaim to divide up the memcgs
 * in the hierarchy among all concurrent reclaimers operating on the
 * same node.
 */
struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root,
				   struct mem_cgroup *prev,
				   struct mem_cgroup_reclaim_cookie *reclaim)
{
	struct mem_cgroup_reclaim_iter *iter;
	struct cgroup_subsys_state *css;
	struct mem_cgroup *pos;
	struct mem_cgroup *next;

	if (mem_cgroup_disabled())
		return NULL;

	if (!root)
		root = root_mem_cgroup;

	rcu_read_lock();
restart:
	next = NULL;

	if (reclaim) {
		int gen;
		int nid = reclaim->pgdat->node_id;

		iter = &root->nodeinfo[nid]->iter;
		gen = atomic_read(&iter->generation);

		/*
		 * On start, join the current reclaim iteration cycle.
		 * Exit when a concurrent walker completes it.
		 */
		if (!prev)
			reclaim->generation = gen;
		else if (reclaim->generation != gen)
			goto out_unlock;

		pos = READ_ONCE(iter->position);
	} else
		pos = prev;

	css = pos ? &pos->css : NULL;

	while ((css = css_next_descendant_pre(css, &root->css))) {
		/*
		 * Verify the css and acquire a reference.  The root
		 * is provided by the caller, so we know it's alive
		 * and kicking, and don't take an extra reference.
		 */
		if (css == &root->css || css_tryget(css))
			break;
	}

	next = mem_cgroup_from_css(css);

	if (reclaim) {
		/*
		 * The position could have already been updated by a competing
		 * thread, so check that the value hasn't changed since we read
		 * it to avoid reclaiming from the same cgroup twice.
		 */
		if (cmpxchg(&iter->position, pos, next) != pos) {
			if (css && css != &root->css)
				css_put(css);
			goto restart;
		}

		if (!next) {
			atomic_inc(&iter->generation);

			/*
			 * Reclaimers share the hierarchy walk, and a
			 * new one might jump in right at the end of
			 * the hierarchy - make sure they see at least
			 * one group and restart from the beginning.
			 */
			if (!prev)
				goto restart;
		}
	}

out_unlock:
	rcu_read_unlock();
	if (prev && prev != root)
		css_put(&prev->css);

	return next;
}

/**
 * mem_cgroup_iter_break - abort a hierarchy walk prematurely
 * @root: hierarchy root
 * @prev: last visited hierarchy member as returned by mem_cgroup_iter()
 */
void mem_cgroup_iter_break(struct mem_cgroup *root,
			   struct mem_cgroup *prev)
{
	if (!root)
		root = root_mem_cgroup;
	if (prev && prev != root)
		css_put(&prev->css);
}

static void __invalidate_reclaim_iterators(struct mem_cgroup *from,
					struct mem_cgroup *dead_memcg)
{
	struct mem_cgroup_reclaim_iter *iter;
	struct mem_cgroup_per_node *mz;
	int nid;

	for_each_node(nid) {
		mz = from->nodeinfo[nid];
		iter = &mz->iter;
		cmpxchg(&iter->position, dead_memcg, NULL);
	}
}

static void invalidate_reclaim_iterators(struct mem_cgroup *dead_memcg)
{
	struct mem_cgroup *memcg = dead_memcg;
	struct mem_cgroup *last;

	do {
		__invalidate_reclaim_iterators(memcg, dead_memcg);
		last = memcg;
	} while ((memcg = parent_mem_cgroup(memcg)));

	/*
	 * When cgroup1 non-hierarchy mode is used,
	 * parent_mem_cgroup() does not walk all the way up to the
	 * cgroup root (root_mem_cgroup). So we have to handle
	 * dead_memcg from cgroup root separately.
	 */
	if (!mem_cgroup_is_root(last))
		__invalidate_reclaim_iterators(root_mem_cgroup,
						dead_memcg);
}

/**
 * mem_cgroup_scan_tasks - iterate over tasks of a memory cgroup hierarchy
 * @memcg: hierarchy root
 * @fn: function to call for each task
 * @arg: argument passed to @fn
 *
 * This function iterates over tasks attached to @memcg or to any of its
 * descendants and calls @fn for each task. If @fn returns a non-zero
 * value, the function breaks the iteration loop. Otherwise, it will iterate
 * over all tasks and return 0.
 *
 * This function must not be called for the root memory cgroup.
 */
void mem_cgroup_scan_tasks(struct mem_cgroup *memcg,
			   int (*fn)(struct task_struct *, void *), void *arg)
{
	struct mem_cgroup *iter;
	int ret = 0;

	BUG_ON(mem_cgroup_is_root(memcg));

	for_each_mem_cgroup_tree(iter, memcg) {
		struct css_task_iter it;
		struct task_struct *task;

		css_task_iter_start(&iter->css, CSS_TASK_ITER_PROCS, &it);
		while (!ret && (task = css_task_iter_next(&it))) {
			ret = fn(task, arg);
			/* Avoid potential softlockup warning */
			cond_resched();
		}
		css_task_iter_end(&it);
		if (ret) {
			mem_cgroup_iter_break(memcg, iter);
			break;
		}
	}
}

#ifdef CONFIG_DEBUG_VM
void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio)
{
	struct mem_cgroup *memcg;

	if (mem_cgroup_disabled())
		return;

	memcg = folio_memcg(folio);

	if (!memcg)
		VM_BUG_ON_FOLIO(!mem_cgroup_is_root(lruvec_memcg(lruvec)), folio);
	else
		VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) != memcg, folio);
}
#endif

/**
 * folio_lruvec_lock - Lock the lruvec for a folio.
 * @folio: Pointer to the folio.
 *
 * These functions are safe to use under any of the following conditions:
 * - folio locked
 * - folio_test_lru false
 * - folio frozen (refcount of 0)
 *
 * Return: The lruvec this folio is on with its lock held.
 */
struct lruvec *folio_lruvec_lock(struct folio *folio)
{
	struct lruvec *lruvec = folio_lruvec(folio);

	spin_lock(&lruvec->lru_lock);
	lruvec_memcg_debug(lruvec, folio);

	return lruvec;
}

/**
 * folio_lruvec_lock_irq - Lock the lruvec for a folio.
 * @folio: Pointer to the folio.
 *
 * These functions are safe to use under any of the following conditions:
 * - folio locked
 * - folio_test_lru false
 * - folio frozen (refcount of 0)
 *
 * Return: The lruvec this folio is on with its lock held and interrupts
 * disabled.
 */
struct lruvec *folio_lruvec_lock_irq(struct folio *folio)
{
	struct lruvec *lruvec = folio_lruvec(folio);

	spin_lock_irq(&lruvec->lru_lock);
	lruvec_memcg_debug(lruvec, folio);

	return lruvec;
}

/**
 * folio_lruvec_lock_irqsave - Lock the lruvec for a folio.
 * @folio: Pointer to the folio.
 * @flags: Pointer to irqsave flags.
 *
 * These functions are safe to use under any of the following conditions:
 * - folio locked
 * - folio_test_lru false
 * - folio frozen (refcount of 0)
 *
 * Return: The lruvec this folio is on with its lock held and interrupts
 * disabled.
 */
struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio,
		unsigned long *flags)
{
	struct lruvec *lruvec = folio_lruvec(folio);

	spin_lock_irqsave(&lruvec->lru_lock, *flags);
	lruvec_memcg_debug(lruvec, folio);

	return lruvec;
}

/**
 * mem_cgroup_update_lru_size - account for adding or removing an lru page
 * @lruvec: mem_cgroup per zone lru vector
 * @lru: index of lru list the page is sitting on
 * @zid: zone id of the accounted pages
 * @nr_pages: positive when adding or negative when removing
 *
 * This function must be called under lru_lock, just before a page is added
 * to or just after a page is removed from an lru list.
 */
void mem_cgroup_update_lru_size(struct lruvec *lruvec, enum lru_list lru,
				int zid, int nr_pages)
{
	struct mem_cgroup_per_node *mz;
	unsigned long *lru_size;
	long size;

	if (mem_cgroup_disabled())
		return;

	mz = container_of(lruvec, struct mem_cgroup_per_node, lruvec);
	lru_size = &mz->lru_zone_size[zid][lru];

	if (nr_pages < 0)
		*lru_size += nr_pages;

	size = *lru_size;
	if (WARN_ONCE(size < 0,
		"%s(%p, %d, %d): lru_size %ld\n",
		__func__, lruvec, lru, nr_pages, size)) {
		VM_BUG_ON(1);
		*lru_size = 0;
	}

	if (nr_pages > 0)
		*lru_size += nr_pages;
}

/**
 * mem_cgroup_margin - calculate chargeable space of a memory cgroup
 * @memcg: the memory cgroup
 *
 * Returns the maximum amount of memory @mem can be charged with, in
 * pages.
 */
static unsigned long mem_cgroup_margin(struct mem_cgroup *memcg)
{
	unsigned long margin = 0;
	unsigned long count;
	unsigned long limit;

	count = page_counter_read(&memcg->memory);
	limit = READ_ONCE(memcg->memory.max);
	if (count < limit)
		margin = limit - count;

	if (do_memsw_account()) {
		count = page_counter_read(&memcg->memsw);
		limit = READ_ONCE(memcg->memsw.max);
		if (count < limit)
			margin = min(margin, limit - count);
		else
			margin = 0;
	}

	return margin;
}

struct memory_stat {
	const char *name;
	unsigned int idx;
};

static const struct memory_stat memory_stats[] = {
	{ "anon",			NR_ANON_MAPPED			},
	{ "file",			NR_FILE_PAGES			},
	{ "kernel",			MEMCG_KMEM			},
	{ "kernel_stack",		NR_KERNEL_STACK_KB		},
	{ "pagetables",			NR_PAGETABLE			},
	{ "sec_pagetables",		NR_SECONDARY_PAGETABLE		},
	{ "percpu",			MEMCG_PERCPU_B			},
	{ "sock",			MEMCG_SOCK			},
	{ "vmalloc",			MEMCG_VMALLOC			},
	{ "shmem",			NR_SHMEM			},
#ifdef CONFIG_ZSWAP
	{ "zswap",			MEMCG_ZSWAP_B			},
	{ "zswapped",			MEMCG_ZSWAPPED			},
#endif
	{ "file_mapped",		NR_FILE_MAPPED			},
	{ "file_dirty",			NR_FILE_DIRTY			},
	{ "file_writeback",		NR_WRITEBACK			},
#ifdef CONFIG_SWAP
	{ "swapcached",			NR_SWAPCACHE			},
#endif
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	{ "anon_thp",			NR_ANON_THPS			},
	{ "file_thp",			NR_FILE_THPS			},
	{ "shmem_thp",			NR_SHMEM_THPS			},
#endif
	{ "inactive_anon",		NR_INACTIVE_ANON		},
	{ "active_anon",		NR_ACTIVE_ANON			},
	{ "inactive_file",		NR_INACTIVE_FILE		},
	{ "active_file",		NR_ACTIVE_FILE			},
	{ "unevictable",		NR_UNEVICTABLE			},
	{ "slab_reclaimable",		NR_SLAB_RECLAIMABLE_B		},
	{ "slab_unreclaimable",		NR_SLAB_UNRECLAIMABLE_B		},
#ifdef CONFIG_HUGETLB_PAGE
	{ "hugetlb",			NR_HUGETLB			},
#endif

	/* The memory events */
	{ "workingset_refault_anon",	WORKINGSET_REFAULT_ANON		},
	{ "workingset_refault_file",	WORKINGSET_REFAULT_FILE		},
	{ "workingset_activate_anon",	WORKINGSET_ACTIVATE_ANON	},
	{ "workingset_activate_file",	WORKINGSET_ACTIVATE_FILE	},
	{ "workingset_restore_anon",	WORKINGSET_RESTORE_ANON		},
	{ "workingset_restore_file",	WORKINGSET_RESTORE_FILE		},
	{ "workingset_nodereclaim",	WORKINGSET_NODERECLAIM		},

	{ "pgdemote_kswapd",		PGDEMOTE_KSWAPD		},
	{ "pgdemote_direct",		PGDEMOTE_DIRECT		},
	{ "pgdemote_khugepaged",	PGDEMOTE_KHUGEPAGED	},
	{ "pgdemote_proactive",		PGDEMOTE_PROACTIVE	},
#ifdef CONFIG_NUMA_BALANCING
	{ "pgpromote_success",		PGPROMOTE_SUCCESS	},
#endif
};

/* The actual unit of the state item, not the same as the output unit */
static int memcg_page_state_unit(int item)
{
	switch (item) {
	case MEMCG_PERCPU_B:
	case MEMCG_ZSWAP_B:
	case NR_SLAB_RECLAIMABLE_B:
	case NR_SLAB_UNRECLAIMABLE_B:
		return 1;
	case NR_KERNEL_STACK_KB:
		return SZ_1K;
	default:
		return PAGE_SIZE;
	}
}

/* Translate stat items to the correct unit for memory.stat output */
static int memcg_page_state_output_unit(int item)
{
	/*
	 * Workingset state is actually in pages, but we export it to userspace
	 * as a scalar count of events, so special case it here.
	 *
	 * Demotion and promotion activities are exported in pages, consistent
	 * with their global counterparts.
	 */
	switch (item) {
	case WORKINGSET_REFAULT_ANON:
	case WORKINGSET_REFAULT_FILE:
	case WORKINGSET_ACTIVATE_ANON:
	case WORKINGSET_ACTIVATE_FILE:
	case WORKINGSET_RESTORE_ANON:
	case WORKINGSET_RESTORE_FILE:
	case WORKINGSET_NODERECLAIM:
	case PGDEMOTE_KSWAPD:
	case PGDEMOTE_DIRECT:
	case PGDEMOTE_KHUGEPAGED:
	case PGDEMOTE_PROACTIVE:
#ifdef CONFIG_NUMA_BALANCING
	case PGPROMOTE_SUCCESS:
#endif
		return 1;
	default:
		return memcg_page_state_unit(item);
	}
}

unsigned long memcg_page_state_output(struct mem_cgroup *memcg, int item)
{
	return memcg_page_state(memcg, item) *
		memcg_page_state_output_unit(item);
}

#ifdef CONFIG_MEMCG_V1
unsigned long memcg_page_state_local_output(struct mem_cgroup *memcg, int item)
{
	return memcg_page_state_local(memcg, item) *
		memcg_page_state_output_unit(item);
}
#endif

#ifdef CONFIG_HUGETLB_PAGE
static bool memcg_accounts_hugetlb(void)
{
	return cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_HUGETLB_ACCOUNTING;
}
#else /* CONFIG_HUGETLB_PAGE */
static bool memcg_accounts_hugetlb(void)
{
	return false;
}
#endif /* CONFIG_HUGETLB_PAGE */

static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
{
	int i;

	/*
	 * Provide statistics on the state of the memory subsystem as
	 * well as cumulative event counters that show past behavior.
	 *
	 * This list is ordered following a combination of these gradients:
	 * 1) generic big picture -> specifics and details
	 * 2) reflecting userspace activity -> reflecting kernel heuristics
	 *
	 * Current memory state:
	 */
	mem_cgroup_flush_stats(memcg);

	for (i = 0; i < ARRAY_SIZE(memory_stats); i++) {
		u64 size;

#ifdef CONFIG_HUGETLB_PAGE
		if (unlikely(memory_stats[i].idx == NR_HUGETLB) &&
			!memcg_accounts_hugetlb())
			continue;
#endif
		size = memcg_page_state_output(memcg, memory_stats[i].idx);
		seq_buf_printf(s, "%s %llu\n", memory_stats[i].name, size);

		if (unlikely(memory_stats[i].idx == NR_SLAB_UNRECLAIMABLE_B)) {
			size += memcg_page_state_output(memcg,
							NR_SLAB_RECLAIMABLE_B);
			seq_buf_printf(s, "slab %llu\n", size);
		}
	}

	/* Accumulated memory events */
	seq_buf_printf(s, "pgscan %lu\n",
		       memcg_events(memcg, PGSCAN_KSWAPD) +
		       memcg_events(memcg, PGSCAN_DIRECT) +
		       memcg_events(memcg, PGSCAN_PROACTIVE) +
		       memcg_events(memcg, PGSCAN_KHUGEPAGED));
	seq_buf_printf(s, "pgsteal %lu\n",
		       memcg_events(memcg, PGSTEAL_KSWAPD) +
		       memcg_events(memcg, PGSTEAL_DIRECT) +
		       memcg_events(memcg, PGSTEAL_PROACTIVE) +
		       memcg_events(memcg, PGSTEAL_KHUGEPAGED));

	for (i = 0; i < ARRAY_SIZE(memcg_vm_event_stat); i++) {
#ifdef CONFIG_MEMCG_V1
		if (memcg_vm_event_stat[i] == PGPGIN ||
		    memcg_vm_event_stat[i] == PGPGOUT)
			continue;
#endif
		seq_buf_printf(s, "%s %lu\n",
			       vm_event_name(memcg_vm_event_stat[i]),
			       memcg_events(memcg, memcg_vm_event_stat[i]));
	}
}

static void memory_stat_format(struct mem_cgroup *memcg, struct seq_buf *s)
{
	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
		memcg_stat_format(memcg, s);
	else
		memcg1_stat_format(memcg, s);
	if (seq_buf_has_overflowed(s))
		pr_warn("%s: Warning, stat buffer overflow, please report\n", __func__);
}

/**
 * mem_cgroup_print_oom_context: Print OOM information relevant to
 * memory controller.
 * @memcg: The memory cgroup that went over limit
 * @p: Task that is going to be killed
 *
 * NOTE: @memcg and @p's mem_cgroup can be different when hierarchy is
 * enabled
 */
void mem_cgroup_print_oom_context(struct mem_cgroup *memcg, struct task_struct *p)
{
	rcu_read_lock();

	if (memcg) {
		pr_cont(",oom_memcg=");
		pr_cont_cgroup_path(memcg->css.cgroup);
	} else
		pr_cont(",global_oom");
	if (p) {
		pr_cont(",task_memcg=");
		pr_cont_cgroup_path(task_cgroup(p, memory_cgrp_id));
	}
	rcu_read_unlock();
}

/**
 * mem_cgroup_print_oom_meminfo: Print OOM memory information relevant to
 * memory controller.
 * @memcg: The memory cgroup that went over limit
 */
void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg)
{
	/* Use static buffer, for the caller is holding oom_lock. */
	static char buf[SEQ_BUF_SIZE];
	struct seq_buf s;
	unsigned long memory_failcnt;

	lockdep_assert_held(&oom_lock);

	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
		memory_failcnt = atomic_long_read(&memcg->memory_events[MEMCG_MAX]);
	else
		memory_failcnt = memcg->memory.failcnt;

	pr_info("memory: usage %llukB, limit %llukB, failcnt %lu\n",
		K((u64)page_counter_read(&memcg->memory)),
		K((u64)READ_ONCE(memcg->memory.max)), memory_failcnt);
	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
		pr_info("swap: usage %llukB, limit %llukB, failcnt %lu\n",
			K((u64)page_counter_read(&memcg->swap)),
			K((u64)READ_ONCE(memcg->swap.max)),
			atomic_long_read(&memcg->memory_events[MEMCG_SWAP_MAX]));
#ifdef CONFIG_MEMCG_V1
	else {
		pr_info("memory+swap: usage %llukB, limit %llukB, failcnt %lu\n",
			K((u64)page_counter_read(&memcg->memsw)),
			K((u64)memcg->memsw.max), memcg->memsw.failcnt);
		pr_info("kmem: usage %llukB, limit %llukB, failcnt %lu\n",
			K((u64)page_counter_read(&memcg->kmem)),
			K((u64)memcg->kmem.max), memcg->kmem.failcnt);
	}
#endif

	pr_info("Memory cgroup stats for ");
	pr_cont_cgroup_path(memcg->css.cgroup);
	pr_cont(":");
	seq_buf_init(&s, buf, SEQ_BUF_SIZE);
	memory_stat_format(memcg, &s);
	seq_buf_do_printk(&s, KERN_INFO);
}

/*
 * Return the memory (and swap, if configured) limit for a memcg.
 */
unsigned long mem_cgroup_get_max(struct mem_cgroup *memcg)
{
	unsigned long max = READ_ONCE(memcg->memory.max);

	if (do_memsw_account()) {
		if (mem_cgroup_swappiness(memcg)) {
			/* Calculate swap excess capacity from memsw limit */
			unsigned long swap = READ_ONCE(memcg->memsw.max) - max;

			max += min(swap, (unsigned long)total_swap_pages);
		}
	} else {
		if (mem_cgroup_swappiness(memcg))
			max += min(READ_ONCE(memcg->swap.max),
				   (unsigned long)total_swap_pages);
	}
	return max;
}

void __memcg_memory_event(struct mem_cgroup *memcg,
			  enum memcg_memory_event event, bool allow_spinning)
{
	bool swap_event = event == MEMCG_SWAP_HIGH || event == MEMCG_SWAP_MAX ||
			  event == MEMCG_SWAP_FAIL;

	/* For now only MEMCG_MAX can happen with !allow_spinning context. */
	VM_WARN_ON_ONCE(!allow_spinning && event != MEMCG_MAX);

	atomic_long_inc(&memcg->memory_events_local[event]);
	if (!swap_event && allow_spinning)
		cgroup_file_notify(&memcg->events_local_file);

	do {
		atomic_long_inc(&memcg->memory_events[event]);
		if (allow_spinning) {
			if (swap_event)
				cgroup_file_notify(&memcg->swap_events_file);
			else
				cgroup_file_notify(&memcg->events_file);
		}

		if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
			break;
		if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_LOCAL_EVENTS)
			break;
	} while ((memcg = parent_mem_cgroup(memcg)) &&
		 !mem_cgroup_is_root(memcg));
}
EXPORT_SYMBOL_GPL(__memcg_memory_event);

static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask,
				     int order)
{
	struct oom_control oc = {
		.zonelist = NULL,
		.nodemask = NULL,
		.memcg = memcg,
		.gfp_mask = gfp_mask,
		.order = order,
	};
	bool ret = true;

	if (mutex_lock_killable(&oom_lock))
		return true;

	if (mem_cgroup_margin(memcg) >= (1 << order))
		goto unlock;

	/*
	 * A few threads which were not waiting at mutex_lock_killable() can
	 * fail to bail out. Therefore, check again after holding oom_lock.
	 */
	ret = out_of_memory(&oc);

unlock:
	mutex_unlock(&oom_lock);
	return ret;
}

/*
 * Returns true if successfully killed one or more processes. Though in some
 * corner cases it can return true even without killing any process.
 */
static bool mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order)
{
	bool locked, ret;

	if (order > PAGE_ALLOC_COSTLY_ORDER)
		return false;

	memcg_memory_event(memcg, MEMCG_OOM);

	if (!memcg1_oom_prepare(memcg, &locked))
		return false;

	ret = mem_cgroup_out_of_memory(memcg, mask, order);

	memcg1_oom_finish(memcg, locked);

	return ret;
}

/**
 * mem_cgroup_get_oom_group - get a memory cgroup to clean up after OOM
 * @victim: task to be killed by the OOM killer
 * @oom_domain: memcg in case of memcg OOM, NULL in case of system-wide OOM
 *
 * Returns a pointer to a memory cgroup, which has to be cleaned up
 * by killing all belonging OOM-killable tasks.
 *
 * Caller has to call mem_cgroup_put() on the returned non-NULL memcg.
 */
struct mem_cgroup *mem_cgroup_get_oom_group(struct task_struct *victim,
					    struct mem_cgroup *oom_domain)
{
	struct mem_cgroup *oom_group = NULL;
	struct mem_cgroup *memcg;

	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
		return NULL;

	if (!oom_domain)
		oom_domain = root_mem_cgroup;

	rcu_read_lock();

	memcg = mem_cgroup_from_task(victim);
	if (mem_cgroup_is_root(memcg))
		goto out;

	/*
	 * If the victim task has been asynchronously moved to a different
	 * memory cgroup, we might end up killing tasks outside oom_domain.
	 * In this case it's better to ignore memory.group.oom.
	 */
	if (unlikely(!mem_cgroup_is_descendant(memcg, oom_domain)))
		goto out;

	/*
	 * Traverse the memory cgroup hierarchy from the victim task's
	 * cgroup up to the OOMing cgroup (or root) to find the
	 * highest-level memory cgroup with oom.group set.
	 */
	for (; memcg; memcg = parent_mem_cgroup(memcg)) {
		if (READ_ONCE(memcg->oom_group))
			oom_group = memcg;

		if (memcg == oom_domain)
			break;
	}

	if (oom_group)
		css_get(&oom_group->css);
out:
	rcu_read_unlock();

	return oom_group;
}

void mem_cgroup_print_oom_group(struct mem_cgroup *memcg)
{
	pr_info("Tasks in ");
	pr_cont_cgroup_path(memcg->css.cgroup);
	pr_cont(" are going to be killed due to memory.oom.group set\n");
}

/*
 * The value of NR_MEMCG_STOCK is selected to keep the cached memcgs and their
 * nr_pages in a single cacheline. This may change in future.
 */
#define NR_MEMCG_STOCK 7
#define FLUSHING_CACHED_CHARGE	0
struct memcg_stock_pcp {
	local_trylock_t lock;
	uint8_t nr_pages[NR_MEMCG_STOCK];
	struct mem_cgroup *cached[NR_MEMCG_STOCK];

	struct work_struct work;
	unsigned long flags;
};

static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
	.lock = INIT_LOCAL_TRYLOCK(lock),
};

struct obj_stock_pcp {
	local_trylock_t lock;
	unsigned int nr_bytes;
	struct obj_cgroup *cached_objcg;
	struct pglist_data *cached_pgdat;
	int nr_slab_reclaimable_b;
	int nr_slab_unreclaimable_b;

	struct work_struct work;
	unsigned long flags;
};

static DEFINE_PER_CPU_ALIGNED(struct obj_stock_pcp, obj_stock) = {
	.lock = INIT_LOCAL_TRYLOCK(lock),
};

static DEFINE_MUTEX(percpu_charge_mutex);

static void drain_obj_stock(struct obj_stock_pcp *stock);
static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
				     struct mem_cgroup *root_memcg);

/**
 * consume_stock: Try to consume stocked charge on this cpu.
 * @memcg: memcg to consume from.
 * @nr_pages: how many pages to charge.
 *
 * Consume the cached charge if enough nr_pages are present otherwise return
 * failure. Also return failure for charge request larger than
 * MEMCG_CHARGE_BATCH or if the local lock is already taken.
 *
 * returns true if successful, false otherwise.
 */
static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
{
	struct memcg_stock_pcp *stock;
	uint8_t stock_pages;
	bool ret = false;
	int i;

	if (nr_pages > MEMCG_CHARGE_BATCH ||
	    !local_trylock(&memcg_stock.lock))
		return ret;

	stock = this_cpu_ptr(&memcg_stock);

	for (i = 0; i < NR_MEMCG_STOCK; ++i) {
		if (memcg != READ_ONCE(stock->cached[i]))
			continue;

		stock_pages = READ_ONCE(stock->nr_pages[i]);
		if (stock_pages >= nr_pages) {
			WRITE_ONCE(stock->nr_pages[i], stock_pages - nr_pages);
			ret = true;
		}
		break;
	}

	local_unlock(&memcg_stock.lock);

	return ret;
}

static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
{
	page_counter_uncharge(&memcg->memory, nr_pages);
	if (do_memsw_account())
		page_counter_uncharge(&memcg->memsw, nr_pages);
}

/*
 * Returns stocks cached in percpu and reset cached information.
 */
static void drain_stock(struct memcg_stock_pcp *stock, int i)
{
	struct mem_cgroup *old = READ_ONCE(stock->cached[i]);
	uint8_t stock_pages;

	if (!old)
		return;

	stock_pages = READ_ONCE(stock->nr_pages[i]);
	if (stock_pages) {
		memcg_uncharge(old, stock_pages);
		WRITE_ONCE(stock->nr_pages[i], 0);
	}

	css_put(&old->css);
	WRITE_ONCE(stock->cached[i], NULL);
}

static void drain_stock_fully(struct memcg_stock_pcp *stock)
{
	int i;

	for (i = 0; i < NR_MEMCG_STOCK; ++i)
		drain_stock(stock, i);
}

static void drain_local_memcg_stock(struct work_struct *dummy)
{
	struct memcg_stock_pcp *stock;

	if (WARN_ONCE(!in_task(), "drain in non-task context"))
		return;

	local_lock(&memcg_stock.lock);

	stock = this_cpu_ptr(&memcg_stock);
	drain_stock_fully(stock);
	clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags);

	local_unlock(&memcg_stock.lock);
}

static void drain_local_obj_stock(struct work_struct *dummy)
{
	struct obj_stock_pcp *stock;

	if (WARN_ONCE(!in_task(), "drain in non-task context"))
		return;

	local_lock(&obj_stock.lock);

	stock = this_cpu_ptr(&obj_stock);
	drain_obj_stock(stock);
	clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags);

	local_unlock(&obj_stock.lock);
}

static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
{
	struct memcg_stock_pcp *stock;
	struct mem_cgroup *cached;
	uint8_t stock_pages;
	bool success = false;
	int empty_slot = -1;
	int i;

	/*
	 * For now limit MEMCG_CHARGE_BATCH to 127 and less. In future if we
	 * decide to increase it more than 127 then we will need more careful
	 * handling of nr_pages[] in struct memcg_stock_pcp.
	 */
	BUILD_BUG_ON(MEMCG_CHARGE_BATCH > S8_MAX);

	VM_WARN_ON_ONCE(mem_cgroup_is_root(memcg));

	if (nr_pages > MEMCG_CHARGE_BATCH ||
	    !local_trylock(&memcg_stock.lock)) {
		/*
		 * In case of larger than batch refill or unlikely failure to
		 * lock the percpu memcg_stock.lock, uncharge memcg directly.
		 */
		memcg_uncharge(memcg, nr_pages);
		return;
	}

	stock = this_cpu_ptr(&memcg_stock);
	for (i = 0; i < NR_MEMCG_STOCK; ++i) {
		cached = READ_ONCE(stock->cached[i]);
		if (!cached && empty_slot == -1)
			empty_slot = i;
		if (memcg == READ_ONCE(stock->cached[i])) {
			stock_pages = READ_ONCE(stock->nr_pages[i]) + nr_pages;
			WRITE_ONCE(stock->nr_pages[i], stock_pages);
			if (stock_pages > MEMCG_CHARGE_BATCH)
				drain_stock(stock, i);
			success = true;
			break;
		}
	}

	if (!success) {
		i = empty_slot;
		if (i == -1) {
			i = get_random_u32_below(NR_MEMCG_STOCK);
			drain_stock(stock, i);
		}
		css_get(&memcg->css);
		WRITE_ONCE(stock->cached[i], memcg);
		WRITE_ONCE(stock->nr_pages[i], nr_pages);
	}

	local_unlock(&memcg_stock.lock);
}

static bool is_memcg_drain_needed(struct memcg_stock_pcp *stock,
				  struct mem_cgroup *root_memcg)
{
	struct mem_cgroup *memcg;
	bool flush = false;
	int i;

	rcu_read_lock();
	for (i = 0; i < NR_MEMCG_STOCK; ++i) {
		memcg = READ_ONCE(stock->cached[i]);
		if (!memcg)
			continue;

		if (READ_ONCE(stock->nr_pages[i]) &&
		    mem_cgroup_is_descendant(memcg, root_memcg)) {
			flush = true;
			break;
		}
	}
	rcu_read_unlock();
	return flush;
}

static void schedule_drain_work(int cpu, struct work_struct *work)
{
	/*
	 * Protect housekeeping cpumask read and work enqueue together
	 * in the same RCU critical section so that later cpuset isolated
	 * partition update only need to wait for an RCU GP and flush the
	 * pending work on newly isolated CPUs.
	 */
	guard(rcu)();
	if (!cpu_is_isolated(cpu))
		queue_work_on(cpu, memcg_wq, work);
}

/*
 * Drains all per-CPU charge caches for given root_memcg resp. subtree
 * of the hierarchy under it.
 */
void drain_all_stock(struct mem_cgroup *root_memcg)
{
	int cpu, curcpu;

	/* If someone's already draining, avoid adding running more workers. */
	if (!mutex_trylock(&percpu_charge_mutex))
		return;
	/*
	 * Notify other cpus that system-wide "drain" is running
	 * We do not care about races with the cpu hotplug because cpu down
	 * as well as workers from this path always operate on the local
	 * per-cpu data. CPU up doesn't touch memcg_stock at all.
	 */
	migrate_disable();
	curcpu = smp_processor_id();
	for_each_online_cpu(cpu) {
		struct memcg_stock_pcp *memcg_st = &per_cpu(memcg_stock, cpu);
		struct obj_stock_pcp *obj_st = &per_cpu(obj_stock, cpu);

		if (!test_bit(FLUSHING_CACHED_CHARGE, &memcg_st->flags) &&
		    is_memcg_drain_needed(memcg_st, root_memcg) &&
		    !test_and_set_bit(FLUSHING_CACHED_CHARGE,
				      &memcg_st->flags)) {
			if (cpu == curcpu)
				drain_local_memcg_stock(&memcg_st->work);
			else
				schedule_drain_work(cpu, &memcg_st->work);
		}

		if (!test_bit(FLUSHING_CACHED_CHARGE, &obj_st->flags) &&
		    obj_stock_flush_required(obj_st, root_memcg) &&
		    !test_and_set_bit(FLUSHING_CACHED_CHARGE,
				      &obj_st->flags)) {
			if (cpu == curcpu)
				drain_local_obj_stock(&obj_st->work);
			else
				schedule_drain_work(cpu, &obj_st->work);
		}
	}
	migrate_enable();
	mutex_unlock(&percpu_charge_mutex);
}

static int memcg_hotplug_cpu_dead(unsigned int cpu)
{
	/* no need for the local lock */
	drain_obj_stock(&per_cpu(obj_stock, cpu));
	drain_stock_fully(&per_cpu(memcg_stock, cpu));

	return 0;
}

static unsigned long reclaim_high(struct mem_cgroup *memcg,
				  unsigned int nr_pages,
				  gfp_t gfp_mask)
{
	unsigned long nr_reclaimed = 0;

	do {
		unsigned long pflags;

		if (page_counter_read(&memcg->memory) <=
		    READ_ONCE(memcg->memory.high))
			continue;

		memcg_memory_event(memcg, MEMCG_HIGH);

		psi_memstall_enter(&pflags);
		nr_reclaimed += try_to_free_mem_cgroup_pages(memcg, nr_pages,
							gfp_mask,
							MEMCG_RECLAIM_MAY_SWAP,
							NULL);
		psi_memstall_leave(&pflags);
	} while ((memcg = parent_mem_cgroup(memcg)) &&
		 !mem_cgroup_is_root(memcg));

	return nr_reclaimed;
}

static void high_work_func(struct work_struct *work)
{
	struct mem_cgroup *memcg;

	memcg = container_of(work, struct mem_cgroup, high_work);
	reclaim_high(memcg, MEMCG_CHARGE_BATCH, GFP_KERNEL);
}

/*
 * Clamp the maximum sleep time per allocation batch to 2 seconds. This is
 * enough to still cause a significant slowdown in most cases, while still
 * allowing diagnostics and tracing to proceed without becoming stuck.
 */
#define MEMCG_MAX_HIGH_DELAY_JIFFIES (2UL*HZ)

/*
 * When calculating the delay, we use these either side of the exponentiation to
 * maintain precision and scale to a reasonable number of jiffies (see the table
 * below.
 *
 * - MEMCG_DELAY_PRECISION_SHIFT: Extra precision bits while translating the
 *   overage ratio to a delay.
 * - MEMCG_DELAY_SCALING_SHIFT: The number of bits to scale down the
 *   proposed penalty in order to reduce to a reasonable number of jiffies, and
 *   to produce a reasonable delay curve.
 *
 * MEMCG_DELAY_SCALING_SHIFT just happens to be a number that produces a
 * reasonable delay curve compared to precision-adjusted overage, not
 * penalising heavily at first, but still making sure that growth beyond the
 * limit penalises misbehaviour cgroups by slowing them down exponentially. For
 * example, with a high of 100 megabytes:
 *
 *  +-------+------------------------+
 *  | usage | time to allocate in ms |
 *  +-------+------------------------+
 *  | 100M  |                      0 |
 *  | 101M  |                      6 |
 *  | 102M  |                     25 |
 *  | 103M  |                     57 |
 *  | 104M  |                    102 |
 *  | 105M  |                    159 |
 *  | 106M  |                    230 |
 *  | 107M  |                    313 |
 *  | 108M  |                    409 |
 *  | 109M  |                    518 |
 *  | 110M  |                    639 |
 *  | 111M  |                    774 |
 *  | 112M  |                    921 |
 *  | 113M  |                   1081 |
 *  | 114M  |                   1254 |
 *  | 115M  |                   1439 |
 *  | 116M  |                   1638 |
 *  | 117M  |                   1849 |
 *  | 118M  |                   2000 |
 *  | 119M  |                   2000 |
 *  | 120M  |                   2000 |
 *  +-------+------------------------+
 */
 #define MEMCG_DELAY_PRECISION_SHIFT 20
 #define MEMCG_DELAY_SCALING_SHIFT 14

static u64 calculate_overage(unsigned long usage, unsigned long high)
{
	u64 overage;

	if (usage <= high)
		return 0;

	/*
	 * Prevent division by 0 in overage calculation by acting as if
	 * it was a threshold of 1 page
	 */
	high = max(high, 1UL);

	overage = usage - high;
	overage <<= MEMCG_DELAY_PRECISION_SHIFT;
	return div64_u64(overage, high);
}

static u64 mem_find_max_overage(struct mem_cgroup *memcg)
{
	u64 overage, max_overage = 0;

	do {
		overage = calculate_overage(page_counter_read(&memcg->memory),
					    READ_ONCE(memcg->memory.high));
		max_overage = max(overage, max_overage);
	} while ((memcg = parent_mem_cgroup(memcg)) &&
		 !mem_cgroup_is_root(memcg));

	return max_overage;
}

static u64 swap_find_max_overage(struct mem_cgroup *memcg)
{
	u64 overage, max_overage = 0;

	do {
		overage = calculate_overage(page_counter_read(&memcg->swap),
					    READ_ONCE(memcg->swap.high));
		if (overage)
			memcg_memory_event(memcg, MEMCG_SWAP_HIGH);
		max_overage = max(overage, max_overage);
	} while ((memcg = parent_mem_cgroup(memcg)) &&
		 !mem_cgroup_is_root(memcg));

	return max_overage;
}

/*
 * Get the number of jiffies that we should penalise a mischievous cgroup which
 * is exceeding its memory.high by checking both it and its ancestors.
 */
static unsigned long calculate_high_delay(struct mem_cgroup *memcg,
					  unsigned int nr_pages,
					  u64 max_overage)
{
	unsigned long penalty_jiffies;

	if (!max_overage)
		return 0;

	/*
	 * We use overage compared to memory.high to calculate the number of
	 * jiffies to sleep (penalty_jiffies). Ideally this value should be
	 * fairly lenient on small overages, and increasingly harsh when the
	 * memcg in question makes it clear that it has no intention of stopping
	 * its crazy behaviour, so we exponentially increase the delay based on
	 * overage amount.
	 */
	penalty_jiffies = max_overage * max_overage * HZ;
	penalty_jiffies >>= MEMCG_DELAY_PRECISION_SHIFT;
	penalty_jiffies >>= MEMCG_DELAY_SCALING_SHIFT;

	/*
	 * Factor in the task's own contribution to the overage, such that four
	 * N-sized allocations are throttled approximately the same as one
	 * 4N-sized allocation.
	 *
	 * MEMCG_CHARGE_BATCH pages is nominal, so work out how much smaller or
	 * larger the current charge patch is than that.
	 */
	return penalty_jiffies * nr_pages / MEMCG_CHARGE_BATCH;
}

/*
 * Reclaims memory over the high limit. Called directly from
 * try_charge() (context permitting), as well as from the userland
 * return path where reclaim is always able to block.
 */
void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
{
	unsigned long penalty_jiffies;
	unsigned long pflags;
	unsigned long nr_reclaimed;
	unsigned int nr_pages = current->memcg_nr_pages_over_high;
	int nr_retries = MAX_RECLAIM_RETRIES;
	struct mem_cgroup *memcg;
	bool in_retry = false;

	memcg = get_mem_cgroup_from_mm(current->mm);
	current->memcg_nr_pages_over_high = 0;

retry_reclaim:
	/*
	 * Bail if the task is already exiting. Unlike memory.max,
	 * memory.high enforcement isn't as strict, and there is no
	 * OOM killer involved, which means the excess could already
	 * be much bigger (and still growing) than it could for
	 * memory.max; the dying task could get stuck in fruitless
	 * reclaim for a long time, which isn't desirable.
	 */
	if (task_is_dying())
		goto out;

	/*
	 * The allocating task should reclaim at least the batch size, but for
	 * subsequent retries we only want to do what's necessary to prevent oom
	 * or breaching resource isolation.
	 *
	 * This is distinct from memory.max or page allocator behaviour because
	 * memory.high is currently batched, whereas memory.max and the page
	 * allocator run every time an allocation is made.
	 */
	nr_reclaimed = reclaim_high(memcg,
				    in_retry ? SWAP_CLUSTER_MAX : nr_pages,
				    gfp_mask);

	/*
	 * memory.high is breached and reclaim is unable to keep up. Throttle
	 * allocators proactively to slow down excessive growth.
	 */
	penalty_jiffies = calculate_high_delay(memcg, nr_pages,
					       mem_find_max_overage(memcg));

	penalty_jiffies += calculate_high_delay(memcg, nr_pages,
						swap_find_max_overage(memcg));

	/*
	 * Clamp the max delay per usermode return so as to still keep the
	 * application moving forwards and also permit diagnostics, albeit
	 * extremely slowly.
	 */
	penalty_jiffies = min(penalty_jiffies, MEMCG_MAX_HIGH_DELAY_JIFFIES);

	/*
	 * Don't sleep if the amount of jiffies this memcg owes us is so low
	 * that it's not even worth doing, in an attempt to be nice to those who
	 * go only a small amount over their memory.high value and maybe haven't
	 * been aggressively reclaimed enough yet.
	 */
	if (penalty_jiffies <= HZ / 100)
		goto out;

	/*
	 * If reclaim is making forward progress but we're still over
	 * memory.high, we want to encourage that rather than doing allocator
	 * throttling.
	 */
	if (nr_reclaimed || nr_retries--) {
		in_retry = true;
		goto retry_reclaim;
	}

	/*
	 * Reclaim didn't manage to push usage below the limit, slow
	 * this allocating task down.
	 *
	 * If we exit early, we're guaranteed to die (since
	 * schedule_timeout_killable sets TASK_KILLABLE). This means we don't
	 * need to account for any ill-begotten jiffies to pay them off later.
	 */
	psi_memstall_enter(&pflags);
	schedule_timeout_killable(penalty_jiffies);
	psi_memstall_leave(&pflags);

out:
	css_put(&memcg->css);
}

static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
			    unsigned int nr_pages)
{
	unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages);
	int nr_retries = MAX_RECLAIM_RETRIES;
	struct mem_cgroup *mem_over_limit;
	struct page_counter *counter;
	unsigned long nr_reclaimed;
	bool passed_oom = false;
	unsigned int reclaim_options = MEMCG_RECLAIM_MAY_SWAP;
	bool drained = false;
	bool raised_max_event = false;
	unsigned long pflags;
	bool allow_spinning = gfpflags_allow_spinning(gfp_mask);

retry:
	if (consume_stock(memcg, nr_pages))
		return 0;

	if (!allow_spinning)
		/* Avoid the refill and flush of the older stock */
		batch = nr_pages;

	if (!do_memsw_account() ||
	    page_counter_try_charge(&memcg->memsw, batch, &counter)) {
		if (page_counter_try_charge(&memcg->memory, batch, &counter))
			goto done_restock;
		if (do_memsw_account())
			page_counter_uncharge(&memcg->memsw, batch);
		mem_over_limit = mem_cgroup_from_counter(counter, memory);
	} else {
		mem_over_limit = mem_cgroup_from_counter(counter, memsw);
		reclaim_options &= ~MEMCG_RECLAIM_MAY_SWAP;
	}

	if (batch > nr_pages) {
		batch = nr_pages;
		goto retry;
	}

	/*
	 * Prevent unbounded recursion when reclaim operations need to
	 * allocate memory. This might exceed the limits temporarily,
	 * but we prefer facilitating memory reclaim and getting back
	 * under the limit over triggering OOM kills in these cases.
	 */
	if (unlikely(current->flags & PF_MEMALLOC))
		goto force;

	if (unlikely(task_in_memcg_oom(current)))
		goto nomem;

	if (!gfpflags_allow_blocking(gfp_mask))
		goto nomem;

	__memcg_memory_event(mem_over_limit, MEMCG_MAX, allow_spinning);
	raised_max_event = true;

	psi_memstall_enter(&pflags);
	nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages,
						    gfp_mask, reclaim_options, NULL);
	psi_memstall_leave(&pflags);

	if (mem_cgroup_margin(mem_over_limit) >= nr_pages)
		goto retry;

	if (!drained) {
		drain_all_stock(mem_over_limit);
		drained = true;
		goto retry;
	}

	if (gfp_mask & __GFP_NORETRY)
		goto nomem;
	/*
	 * Even though the limit is exceeded at this point, reclaim
	 * may have been able to free some pages.  Retry the charge
	 * before killing the task.
	 *
	 * Only for regular pages, though: huge pages are rather
	 * unlikely to succeed so close to the limit, and we fall back
	 * to regular pages anyway in case of failure.
	 */
	if (nr_reclaimed && nr_pages <= (1 << PAGE_ALLOC_COSTLY_ORDER))
		goto retry;

	if (nr_retries--)
		goto retry;

	if (gfp_mask & __GFP_RETRY_MAYFAIL)
		goto nomem;

	/* Avoid endless loop for tasks bypassed by the oom killer */
	if (passed_oom && task_is_dying())
		goto nomem;

	/*
	 * keep retrying as long as the memcg oom killer is able to make
	 * a forward progress or bypass the charge if the oom killer
	 * couldn't make any progress.
	 */
	if (mem_cgroup_oom(mem_over_limit, gfp_mask,
			   get_order(nr_pages * PAGE_SIZE))) {
		passed_oom = true;
		nr_retries = MAX_RECLAIM_RETRIES;
		goto retry;
	}
nomem:
	/*
	 * Memcg doesn't have a dedicated reserve for atomic
	 * allocations. But like the global atomic pool, we need to
	 * put the burden of reclaim on regular allocation requests
	 * and let these go through as privileged allocations.
	 */
	if (!(gfp_mask & (__GFP_NOFAIL | __GFP_HIGH)))
		return -ENOMEM;
force:
	/*
	 * If the allocation has to be enforced, don't forget to raise
	 * a MEMCG_MAX event.
	 */
	if (!raised_max_event)
		__memcg_memory_event(mem_over_limit, MEMCG_MAX, allow_spinning);

	/*
	 * The allocation either can't fail or will lead to more memory
	 * being freed very soon.  Allow memory usage go over the limit
	 * temporarily by force charging it.
	 */
	page_counter_charge(&memcg->memory, nr_pages);
	if (do_memsw_account())
		page_counter_charge(&memcg->memsw, nr_pages);

	return 0;

done_restock:
	if (batch > nr_pages)
		refill_stock(memcg, batch - nr_pages);

	/*
	 * If the hierarchy is above the normal consumption range, schedule
	 * reclaim on returning to userland.  We can perform reclaim here
	 * if __GFP_RECLAIM but let's always punt for simplicity and so that
	 * GFP_KERNEL can consistently be used during reclaim.  @memcg is
	 * not recorded as it most likely matches current's and won't
	 * change in the meantime.  As high limit is checked again before
	 * reclaim, the cost of mismatch is negligible.
	 */
	do {
		bool mem_high, swap_high;

		mem_high = page_counter_read(&memcg->memory) >
			READ_ONCE(memcg->memory.high);
		swap_high = page_counter_read(&memcg->swap) >
			READ_ONCE(memcg->swap.high);

		/* Don't bother a random interrupted task */
		if (!in_task()) {
			if (mem_high) {
				schedule_work(&memcg->high_work);
				break;
			}
			continue;
		}

		if (mem_high || swap_high) {
			/*
			 * The allocating tasks in this cgroup will need to do
			 * reclaim or be throttled to prevent further growth
			 * of the memory or swap footprints.
			 *
			 * Target some best-effort fairness between the tasks,
			 * and distribute reclaim work and delay penalties
			 * based on how much each task is actually allocating.
			 */
			current->memcg_nr_pages_over_high += batch;
			set_notify_resume(current);
			break;
		}
	} while ((memcg = parent_mem_cgroup(memcg)));

	/*
	 * Reclaim is set up above to be called from the userland
	 * return path. But also attempt synchronous reclaim to avoid
	 * excessive overrun while the task is still inside the
	 * kernel. If this is successful, the return path will see it
	 * when it rechecks the overage and simply bail out.
	 */
	if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH &&
	    !(current->flags & PF_MEMALLOC) &&
	    gfpflags_allow_blocking(gfp_mask))
		__mem_cgroup_handle_over_high(gfp_mask);
	return 0;
}

static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask,
			     unsigned int nr_pages)
{
	if (mem_cgroup_is_root(memcg))
		return 0;

	return try_charge_memcg(memcg, gfp_mask, nr_pages);
}

static void commit_charge(struct folio *folio, struct mem_cgroup *memcg)
{
	VM_BUG_ON_FOLIO(folio_memcg_charged(folio), folio);
	/*
	 * Any of the following ensures page's memcg stability:
	 *
	 * - the page lock
	 * - LRU isolation
	 * - exclusive reference
	 */
	folio->memcg_data = (unsigned long)memcg;
}

#ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC
static inline void account_slab_nmi_safe(struct mem_cgroup *memcg,
					 struct pglist_data *pgdat,
					 enum node_stat_item idx, int nr)
{
	struct lruvec *lruvec;

	if (likely(!in_nmi())) {
		lruvec = mem_cgroup_lruvec(memcg, pgdat);
		mod_memcg_lruvec_state(lruvec, idx, nr);
	} else {
		struct mem_cgroup_per_node *pn = memcg->nodeinfo[pgdat->node_id];

		/* preemption is disabled in_nmi(). */
		css_rstat_updated(&memcg->css, smp_processor_id());
		if (idx == NR_SLAB_RECLAIMABLE_B)
			atomic_add(nr, &pn->slab_reclaimable);
		else
			atomic_add(nr, &pn->slab_unreclaimable);
	}
}
#else
static inline void account_slab_nmi_safe(struct mem_cgroup *memcg,
					 struct pglist_data *pgdat,
					 enum node_stat_item idx, int nr)
{
	struct lruvec *lruvec;

	lruvec = mem_cgroup_lruvec(memcg, pgdat);
	mod_memcg_lruvec_state(lruvec, idx, nr);
}
#endif

static inline void mod_objcg_mlstate(struct obj_cgroup *objcg,
				       struct pglist_data *pgdat,
				       enum node_stat_item idx, int nr)
{
	struct mem_cgroup *memcg;

	rcu_read_lock();
	memcg = obj_cgroup_memcg(objcg);
	account_slab_nmi_safe(memcg, pgdat, idx, nr);
	rcu_read_unlock();
}

static __always_inline
struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
{
	/*
	 * Slab objects are accounted individually, not per-page.
	 * Memcg membership data for each individual object is saved in
	 * slab->obj_exts.
	 */
	unsigned long obj_exts;
	struct slabobj_ext *obj_ext;
	unsigned int off;

	obj_exts = slab_obj_exts(slab);
	if (!obj_exts)
		return NULL;

	get_slab_obj_exts(obj_exts);
	off = obj_to_index(slab->slab_cache, slab, p);
	obj_ext = slab_obj_ext(slab, obj_exts, off);
	if (obj_ext->objcg) {
		struct obj_cgroup *objcg = obj_ext->objcg;

		put_slab_obj_exts(obj_exts);
		return obj_cgroup_memcg(objcg);
	}
	put_slab_obj_exts(obj_exts);

	return NULL;
}

/*
 * Returns a pointer to the memory cgroup to which the kernel object is charged.
 * It is not suitable for objects allocated using vmalloc().
 *
 * A passed kernel object must be a slab object or a generic kernel page.
 *
 * The caller must ensure the memcg lifetime, e.g. by taking rcu_read_lock(),
 * cgroup_mutex, etc.
 */
struct mem_cgroup *mem_cgroup_from_virt(void *p)
{
	struct slab *slab;

	if (mem_cgroup_disabled())
		return NULL;

	slab = virt_to_slab(p);
	if (slab)
		return mem_cgroup_from_obj_slab(slab, p);
	return folio_memcg_check(virt_to_folio(p));
}

static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg)
{
	struct obj_cgroup *objcg = NULL;

	for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
		objcg = rcu_dereference(memcg->objcg);
		if (likely(objcg && obj_cgroup_tryget(objcg)))
			break;
		objcg = NULL;
	}
	return objcg;
}

static struct obj_cgroup *current_objcg_update(void)
{
	struct mem_cgroup *memcg;
	struct obj_cgroup *old, *objcg = NULL;

	do {
		/* Atomically drop the update bit. */
		old = xchg(&current->objcg, NULL);
		if (old) {
			old = (struct obj_cgroup *)
				((unsigned long)old & ~CURRENT_OBJCG_UPDATE_FLAG);
			obj_cgroup_put(old);

			old = NULL;
		}

		/* If new objcg is NULL, no reason for the second atomic update. */
		if (!current->mm || (current->flags & PF_KTHREAD))
			return NULL;

		/*
		 * Release the objcg pointer from the previous iteration,
		 * if try_cmpxcg() below fails.
		 */
		if (unlikely(objcg)) {
			obj_cgroup_put(objcg);
			objcg = NULL;
		}

		/*
		 * Obtain the new objcg pointer. The current task can be
		 * asynchronously moved to another memcg and the previous
		 * memcg can be offlined. So let's get the memcg pointer
		 * and try get a reference to objcg under a rcu read lock.
		 */

		rcu_read_lock();
		memcg = mem_cgroup_from_task(current);
		objcg = __get_obj_cgroup_from_memcg(memcg);
		rcu_read_unlock();

		/*
		 * Try set up a new objcg pointer atomically. If it
		 * fails, it means the update flag was set concurrently, so
		 * the whole procedure should be repeated.
		 */
	} while (!try_cmpxchg(&current->objcg, &old, objcg));

	return objcg;
}

__always_inline struct obj_cgroup *current_obj_cgroup(void)
{
	struct mem_cgroup *memcg;
	struct obj_cgroup *objcg;

	if (IS_ENABLED(CONFIG_MEMCG_NMI_UNSAFE) && in_nmi())
		return NULL;

	if (in_task()) {
		memcg = current->active_memcg;
		if (unlikely(memcg))
			goto from_memcg;

		objcg = READ_ONCE(current->objcg);
		if (unlikely((unsigned long)objcg & CURRENT_OBJCG_UPDATE_FLAG))
			objcg = current_objcg_update();
		/*
		 * Objcg reference is kept by the task, so it's safe
		 * to use the objcg by the current task.
		 */
		return objcg;
	}

	memcg = this_cpu_read(int_active_memcg);
	if (unlikely(memcg))
		goto from_memcg;

	return NULL;

from_memcg:
	objcg = NULL;
	for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
		/*
		 * Memcg pointer is protected by scope (see set_active_memcg())
		 * and is pinning the corresponding objcg, so objcg can't go
		 * away and can be used within the scope without any additional
		 * protection.
		 */
		objcg = rcu_dereference_check(memcg->objcg, 1);
		if (likely(objcg))
			break;
	}

	return objcg;
}

struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio)
{
	struct obj_cgroup *objcg;

	if (!memcg_kmem_online())
		return NULL;

	if (folio_memcg_kmem(folio)) {
		objcg = __folio_objcg(folio);
		obj_cgroup_get(objcg);
	} else {
		struct mem_cgroup *memcg;

		rcu_read_lock();
		memcg = __folio_memcg(folio);
		if (memcg)
			objcg = __get_obj_cgroup_from_memcg(memcg);
		else
			objcg = NULL;
		rcu_read_unlock();
	}
	return objcg;
}

#ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC
static inline void account_kmem_nmi_safe(struct mem_cgroup *memcg, int val)
{
	if (likely(!in_nmi())) {
		mod_memcg_state(memcg, MEMCG_KMEM, val);
	} else {
		/* preemption is disabled in_nmi(). */
		css_rstat_updated(&memcg->css, smp_processor_id());
		atomic_add(val, &memcg->kmem_stat);
	}
}
#else
static inline void account_kmem_nmi_safe(struct mem_cgroup *memcg, int val)
{
	mod_memcg_state(memcg, MEMCG_KMEM, val);
}
#endif

/*
 * obj_cgroup_uncharge_pages: uncharge a number of kernel pages from a objcg
 * @objcg: object cgroup to uncharge
 * @nr_pages: number of pages to uncharge
 */
static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg,
				      unsigned int nr_pages)
{
	struct mem_cgroup *memcg;

	memcg = get_mem_cgroup_from_objcg(objcg);

	account_kmem_nmi_safe(memcg, -nr_pages);
	memcg1_account_kmem(memcg, -nr_pages);
	if (!mem_cgroup_is_root(memcg))
		refill_stock(memcg, nr_pages);

	css_put(&memcg->css);
}

/*
 * obj_cgroup_charge_pages: charge a number of kernel pages to a objcg
 * @objcg: object cgroup to charge
 * @gfp: reclaim mode
 * @nr_pages: number of pages to charge
 *
 * Returns 0 on success, an error code on failure.
 */
static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp,
				   unsigned int nr_pages)
{
	struct mem_cgroup *memcg;
	int ret;

	memcg = get_mem_cgroup_from_objcg(objcg);

	ret = try_charge_memcg(memcg, gfp, nr_pages);
	if (ret)
		goto out;

	account_kmem_nmi_safe(memcg, nr_pages);
	memcg1_account_kmem(memcg, nr_pages);
out:
	css_put(&memcg->css);

	return ret;
}

static struct obj_cgroup *page_objcg(const struct page *page)
{
	unsigned long memcg_data = page->memcg_data;

	if (mem_cgroup_disabled() || !memcg_data)
		return NULL;

	VM_BUG_ON_PAGE((memcg_data & OBJEXTS_FLAGS_MASK) != MEMCG_DATA_KMEM,
			page);
	return (struct obj_cgroup *)(memcg_data - MEMCG_DATA_KMEM);
}

static void page_set_objcg(struct page *page, const struct obj_cgroup *objcg)
{
	page->memcg_data = (unsigned long)objcg | MEMCG_DATA_KMEM;
}

/**
 * __memcg_kmem_charge_page: charge a kmem page to the current memory cgroup
 * @page: page to charge
 * @gfp: reclaim mode
 * @order: allocation order
 *
 * Returns 0 on success, an error code on failure.
 */
int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order)
{
	struct obj_cgroup *objcg;
	int ret = 0;

	objcg = current_obj_cgroup();
	if (objcg) {
		ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order);
		if (!ret) {
			obj_cgroup_get(objcg);
			page_set_objcg(page, objcg);
			return 0;
		}
	}
	return ret;
}

/**
 * __memcg_kmem_uncharge_page: uncharge a kmem page
 * @page: page to uncharge
 * @order: allocation order
 */
void __memcg_kmem_uncharge_page(struct page *page, int order)
{
	struct obj_cgroup *objcg = page_objcg(page);
	unsigned int nr_pages = 1 << order;

	if (!objcg)
		return;

	obj_cgroup_uncharge_pages(objcg, nr_pages);
	page->memcg_data = 0;
	obj_cgroup_put(objcg);
}

static void __account_obj_stock(struct obj_cgroup *objcg,
				struct obj_stock_pcp *stock, int nr,
				struct pglist_data *pgdat, enum node_stat_item idx)
{
	int *bytes;

	/*
	 * Save vmstat data in stock and skip vmstat array update unless
	 * accumulating over a page of vmstat data or when pgdat changes.
	 */
	if (stock->cached_pgdat != pgdat) {
		/* Flush the existing cached vmstat data */
		struct pglist_data *oldpg = stock->cached_pgdat;

		if (stock->nr_slab_reclaimable_b) {
			mod_objcg_mlstate(objcg, oldpg, NR_SLAB_RECLAIMABLE_B,
					  stock->nr_slab_reclaimable_b);
			stock->nr_slab_reclaimable_b = 0;
		}
		if (stock->nr_slab_unreclaimable_b) {
			mod_objcg_mlstate(objcg, oldpg, NR_SLAB_UNRECLAIMABLE_B,
					  stock->nr_slab_unreclaimable_b);
			stock->nr_slab_unreclaimable_b = 0;
		}
		stock->cached_pgdat = pgdat;
	}

	bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
					       : &stock->nr_slab_unreclaimable_b;
	/*
	 * Even for large object >= PAGE_SIZE, the vmstat data will still be
	 * cached locally at least once before pushing it out.
	 */
	if (!*bytes) {
		*bytes = nr;
		nr = 0;
	} else {
		*bytes += nr;
		if (abs(*bytes) > PAGE_SIZE) {
			nr = *bytes;
			*bytes = 0;
		} else {
			nr = 0;
		}
	}
	if (nr)
		mod_objcg_mlstate(objcg, pgdat, idx, nr);
}

static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes,
			      struct pglist_data *pgdat, enum node_stat_item idx)
{
	struct obj_stock_pcp *stock;
	bool ret = false;

	if (!local_trylock(&obj_stock.lock))
		return ret;

	stock = this_cpu_ptr(&obj_stock);
	if (objcg == READ_ONCE(stock->cached_objcg) && stock->nr_bytes >= nr_bytes) {
		stock->nr_bytes -= nr_bytes;
		ret = true;

		if (pgdat)
			__account_obj_stock(objcg, stock, nr_bytes, pgdat, idx);
	}

	local_unlock(&obj_stock.lock);

	return ret;
}

static void drain_obj_stock(struct obj_stock_pcp *stock)
{
	struct obj_cgroup *old = READ_ONCE(stock->cached_objcg);

	if (!old)
		return;

	if (stock->nr_bytes) {
		unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT;
		unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1);

		if (nr_pages) {
			struct mem_cgroup *memcg;

			memcg = get_mem_cgroup_from_objcg(old);

			mod_memcg_state(memcg, MEMCG_KMEM, -nr_pages);
			memcg1_account_kmem(memcg, -nr_pages);
			if (!mem_cgroup_is_root(memcg))
				memcg_uncharge(memcg, nr_pages);

			css_put(&memcg->css);
		}

		/*
		 * The leftover is flushed to the centralized per-memcg value.
		 * On the next attempt to refill obj stock it will be moved
		 * to a per-cpu stock (probably, on an other CPU), see
		 * refill_obj_stock().
		 *
		 * How often it's flushed is a trade-off between the memory
		 * limit enforcement accuracy and potential CPU contention,
		 * so it might be changed in the future.
		 */
		atomic_add(nr_bytes, &old->nr_charged_bytes);
		stock->nr_bytes = 0;
	}

	/*
	 * Flush the vmstat data in current stock
	 */
	if (stock->nr_slab_reclaimable_b || stock->nr_slab_unreclaimable_b) {
		if (stock->nr_slab_reclaimable_b) {
			mod_objcg_mlstate(old, stock->cached_pgdat,
					  NR_SLAB_RECLAIMABLE_B,
					  stock->nr_slab_reclaimable_b);
			stock->nr_slab_reclaimable_b = 0;
		}
		if (stock->nr_slab_unreclaimable_b) {
			mod_objcg_mlstate(old, stock->cached_pgdat,
					  NR_SLAB_UNRECLAIMABLE_B,
					  stock->nr_slab_unreclaimable_b);
			stock->nr_slab_unreclaimable_b = 0;
		}
		stock->cached_pgdat = NULL;
	}

	WRITE_ONCE(stock->cached_objcg, NULL);
	obj_cgroup_put(old);
}

static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
				     struct mem_cgroup *root_memcg)
{
	struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);
	struct mem_cgroup *memcg;
	bool flush = false;

	rcu_read_lock();
	if (objcg) {
		memcg = obj_cgroup_memcg(objcg);
		if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))
			flush = true;
	}
	rcu_read_unlock();

	return flush;
}

static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes,
		bool allow_uncharge, int nr_acct, struct pglist_data *pgdat,
		enum node_stat_item idx)
{
	struct obj_stock_pcp *stock;
	unsigned int nr_pages = 0;

	if (!local_trylock(&obj_stock.lock)) {
		if (pgdat)
			mod_objcg_mlstate(objcg, pgdat, idx, nr_bytes);
		nr_pages = nr_bytes >> PAGE_SHIFT;
		nr_bytes = nr_bytes & (PAGE_SIZE - 1);
		atomic_add(nr_bytes, &objcg->nr_charged_bytes);
		goto out;
	}

	stock = this_cpu_ptr(&obj_stock);
	if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
		drain_obj_stock(stock);
		obj_cgroup_get(objcg);
		stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
				? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
		WRITE_ONCE(stock->cached_objcg, objcg);

		allow_uncharge = true;	/* Allow uncharge when objcg changes */
	}
	stock->nr_bytes += nr_bytes;

	if (pgdat)
		__account_obj_stock(objcg, stock, nr_acct, pgdat, idx);

	if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {
		nr_pages = stock->nr_bytes >> PAGE_SHIFT;
		stock->nr_bytes &= (PAGE_SIZE - 1);
	}

	local_unlock(&obj_stock.lock);
out:
	if (nr_pages)
		obj_cgroup_uncharge_pages(objcg, nr_pages);
}

static int obj_cgroup_charge_account(struct obj_cgroup *objcg, gfp_t gfp, size_t size,
				     struct pglist_data *pgdat, enum node_stat_item idx)
{
	unsigned int nr_pages, nr_bytes;
	int ret;

	if (likely(consume_obj_stock(objcg, size, pgdat, idx)))
		return 0;

	/*
	 * In theory, objcg->nr_charged_bytes can have enough
	 * pre-charged bytes to satisfy the allocation. However,
	 * flushing objcg->nr_charged_bytes requires two atomic
	 * operations, and objcg->nr_charged_bytes can't be big.
	 * The shared objcg->nr_charged_bytes can also become a
	 * performance bottleneck if all tasks of the same memcg are
	 * trying to update it. So it's better to ignore it and try
	 * grab some new pages. The stock's nr_bytes will be flushed to
	 * objcg->nr_charged_bytes later on when objcg changes.
	 *
	 * The stock's nr_bytes may contain enough pre-charged bytes
	 * to allow one less page from being charged, but we can't rely
	 * on the pre-charged bytes not being changed outside of
	 * consume_obj_stock() or refill_obj_stock(). So ignore those
	 * pre-charged bytes as well when charging pages. To avoid a
	 * page uncharge right after a page charge, we set the
	 * allow_uncharge flag to false when calling refill_obj_stock()
	 * to temporarily allow the pre-charged bytes to exceed the page
	 * size limit. The maximum reachable value of the pre-charged
	 * bytes is (sizeof(object) + PAGE_SIZE - 2) if there is no data
	 * race.
	 */
	nr_pages = size >> PAGE_SHIFT;
	nr_bytes = size & (PAGE_SIZE - 1);

	if (nr_bytes)
		nr_pages += 1;

	ret = obj_cgroup_charge_pages(objcg, gfp, nr_pages);
	if (!ret && (nr_bytes || pgdat))
		refill_obj_stock(objcg, nr_bytes ? PAGE_SIZE - nr_bytes : 0,
					 false, size, pgdat, idx);

	return ret;
}

int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size)
{
	return obj_cgroup_charge_account(objcg, gfp, size, NULL, 0);
}

void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size)
{
	refill_obj_stock(objcg, size, true, 0, NULL, 0);
}

static inline size_t obj_full_size(struct kmem_cache *s)
{
	/*
	 * For each accounted object there is an extra space which is used
	 * to store obj_cgroup membership. Charge it too.
	 */
	return s->size + sizeof(struct obj_cgroup *);
}

bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
				  gfp_t flags, size_t size, void **p)
{
	struct obj_cgroup *objcg;
	struct slab *slab;
	unsigned long off;
	size_t i;

	/*
	 * The obtained objcg pointer is safe to use within the current scope,
	 * defined by current task or set_active_memcg() pair.
	 * obj_cgroup_get() is used to get a permanent reference.
	 */
	objcg = current_obj_cgroup();
	if (!objcg)
		return true;

	/*
	 * slab_alloc_node() avoids the NULL check, so we might be called with a
	 * single NULL object. kmem_cache_alloc_bulk() aborts if it can't fill
	 * the whole requested size.
	 * return success as there's nothing to free back
	 */
	if (unlikely(*p == NULL))
		return true;

	flags &= gfp_allowed_mask;

	if (lru) {
		int ret;
		struct mem_cgroup *memcg;

		memcg = get_mem_cgroup_from_objcg(objcg);
		ret = memcg_list_lru_alloc(memcg, lru, flags);
		css_put(&memcg->css);

		if (ret)
			return false;
	}

	for (i = 0; i < size; i++) {
		unsigned long obj_exts;
		struct slabobj_ext *obj_ext;

		slab = virt_to_slab(p[i]);

		if (!slab_obj_exts(slab) &&
		    alloc_slab_obj_exts(slab, s, flags, false)) {
			continue;
		}

		/*
		 * if we fail and size is 1, memcg_alloc_abort_single() will
		 * just free the object, which is ok as we have not assigned
		 * objcg to its obj_ext yet
		 *
		 * for larger sizes, kmem_cache_free_bulk() will uncharge
		 * any objects that were already charged and obj_ext assigned
		 *
		 * TODO: we could batch this until slab_pgdat(slab) changes
		 * between iterations, with a more complicated undo
		 */
		if (obj_cgroup_charge_account(objcg, flags, obj_full_size(s),
					slab_pgdat(slab), cache_vmstat_idx(s)))
			return false;

		obj_exts = slab_obj_exts(slab);
		get_slab_obj_exts(obj_exts);
		off = obj_to_index(s, slab, p[i]);
		obj_ext = slab_obj_ext(slab, obj_exts, off);
		obj_cgroup_get(objcg);
		obj_ext->objcg = objcg;
		put_slab_obj_exts(obj_exts);
	}

	return true;
}

void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,
			    void **p, int objects, unsigned long obj_exts)
{
	size_t obj_size = obj_full_size(s);

	for (int i = 0; i < objects; i++) {
		struct obj_cgroup *objcg;
		struct slabobj_ext *obj_ext;
		unsigned int off;

		off = obj_to_index(s, slab, p[i]);
		obj_ext = slab_obj_ext(slab, obj_exts, off);
		objcg = obj_ext->objcg;
		if (!objcg)
			continue;

		obj_ext->objcg = NULL;
		refill_obj_stock(objcg, obj_size, true, -obj_size,
				 slab_pgdat(slab), cache_vmstat_idx(s));
		obj_cgroup_put(objcg);
	}
}

/*
 * The objcg is only set on the first page, so transfer it to all the
 * other pages.
 */
void split_page_memcg(struct page *page, unsigned order)
{
	struct obj_cgroup *objcg = page_objcg(page);
	unsigned int i, nr = 1 << order;

	if (!objcg)
		return;

	for (i = 1; i < nr; i++)
		page_set_objcg(&page[i], objcg);

	obj_cgroup_get_many(objcg, nr - 1);
}

void folio_split_memcg_refs(struct folio *folio, unsigned old_order,
		unsigned new_order)
{
	unsigned new_refs;

	if (mem_cgroup_disabled() || !folio_memcg_charged(folio))
		return;

	new_refs = (1 << (old_order - new_order)) - 1;
	css_get_many(&__folio_memcg(folio)->css, new_refs);
}

static int memcg_online_kmem(struct mem_cgroup *memcg)
{
	struct obj_cgroup *objcg;

	if (mem_cgroup_kmem_disabled())
		return 0;

	if (unlikely(mem_cgroup_is_root(memcg)))
		return 0;

	objcg = obj_cgroup_alloc();
	if (!objcg)
		return -ENOMEM;

	objcg->memcg = memcg;
	rcu_assign_pointer(memcg->objcg, objcg);
	obj_cgroup_get(objcg);
	memcg->orig_objcg = objcg;

	static_branch_enable(&memcg_kmem_online_key);

	memcg->kmemcg_id = memcg->id.id;

	return 0;
}

static void memcg_offline_kmem(struct mem_cgroup *memcg)
{
	struct mem_cgroup *parent;

	if (mem_cgroup_kmem_disabled())
		return;

	if (unlikely(mem_cgroup_is_root(memcg)))
		return;

	parent = parent_mem_cgroup(memcg);
	if (!parent)
		parent = root_mem_cgroup;

	memcg_reparent_list_lrus(memcg, parent);

	/*
	 * Objcg's reparenting must be after list_lru's, make sure list_lru
	 * helpers won't use parent's list_lru until child is drained.
	 */
	memcg_reparent_objcgs(memcg, parent);
}

#ifdef CONFIG_CGROUP_WRITEBACK

#include <trace/events/writeback.h>

static int memcg_wb_domain_init(struct mem_cgroup *memcg, gfp_t gfp)
{
	return wb_domain_init(&memcg->cgwb_domain, gfp);
}

static void memcg_wb_domain_exit(struct mem_cgroup *memcg)
{
	wb_domain_exit(&memcg->cgwb_domain);
}

static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg)
{
	wb_domain_size_changed(&memcg->cgwb_domain);
}

struct wb_domain *mem_cgroup_wb_domain(struct bdi_writeback *wb)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);

	if (!memcg->css.parent)
		return NULL;

	return &memcg->cgwb_domain;
}

/**
 * mem_cgroup_wb_stats - retrieve writeback related stats from its memcg
 * @wb: bdi_writeback in question
 * @pfilepages: out parameter for number of file pages
 * @pheadroom: out parameter for number of allocatable pages according to memcg
 * @pdirty: out parameter for number of dirty pages
 * @pwriteback: out parameter for number of pages under writeback
 *
 * Determine the numbers of file, headroom, dirty, and writeback pages in
 * @wb's memcg.  File, dirty and writeback are self-explanatory.  Headroom
 * is a bit more involved.
 *
 * A memcg's headroom is "min(max, high) - used".  In the hierarchy, the
 * headroom is calculated as the lowest headroom of itself and the
 * ancestors.  Note that this doesn't consider the actual amount of
 * available memory in the system.  The caller should further cap
 * *@pheadroom accordingly.
 */
void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages,
			 unsigned long *pheadroom, unsigned long *pdirty,
			 unsigned long *pwriteback)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);
	struct mem_cgroup *parent;

	mem_cgroup_flush_stats_ratelimited(memcg);

	*pdirty = memcg_page_state(memcg, NR_FILE_DIRTY);
	*pwriteback = memcg_page_state(memcg, NR_WRITEBACK);
	*pfilepages = memcg_page_state(memcg, NR_INACTIVE_FILE) +
			memcg_page_state(memcg, NR_ACTIVE_FILE);

	*pheadroom = PAGE_COUNTER_MAX;
	while ((parent = parent_mem_cgroup(memcg))) {
		unsigned long ceiling = min(READ_ONCE(memcg->memory.max),
					    READ_ONCE(memcg->memory.high));
		unsigned long used = page_counter_read(&memcg->memory);

		*pheadroom = min(*pheadroom, ceiling - min(ceiling, used));
		memcg = parent;
	}
}

/*
 * Foreign dirty flushing
 *
 * There's an inherent mismatch between memcg and writeback.  The former
 * tracks ownership per-page while the latter per-inode.  This was a
 * deliberate design decision because honoring per-page ownership in the
 * writeback path is complicated, may lead to higher CPU and IO overheads
 * and deemed unnecessary given that write-sharing an inode across
 * different cgroups isn't a common use-case.
 *
 * Combined with inode majority-writer ownership switching, this works well
 * enough in most cases but there are some pathological cases.  For
 * example, let's say there are two cgroups A and B which keep writing to
 * different but confined parts of the same inode.  B owns the inode and
 * A's memory is limited far below B's.  A's dirty ratio can rise enough to
 * trigger balance_dirty_pages() sleeps but B's can be low enough to avoid
 * triggering background writeback.  A will be slowed down without a way to
 * make writeback of the dirty pages happen.
 *
 * Conditions like the above can lead to a cgroup getting repeatedly and
 * severely throttled after making some progress after each
 * dirty_expire_interval while the underlying IO device is almost
 * completely idle.
 *
 * Solving this problem completely requires matching the ownership tracking
 * granularities between memcg and writeback in either direction.  However,
 * the more egregious behaviors can be avoided by simply remembering the
 * most recent foreign dirtying events and initiating remote flushes on
 * them when local writeback isn't enough to keep the memory clean enough.
 *
 * The following two functions implement such mechanism.  When a foreign
 * page - a page whose memcg and writeback ownerships don't match - is
 * dirtied, mem_cgroup_track_foreign_dirty() records the inode owning
 * bdi_writeback on the page owning memcg.  When balance_dirty_pages()
 * decides that the memcg needs to sleep due to high dirty ratio, it calls
 * mem_cgroup_flush_foreign() which queues writeback on the recorded
 * foreign bdi_writebacks which haven't expired.  Both the numbers of
 * recorded bdi_writebacks and concurrent in-flight foreign writebacks are
 * limited to MEMCG_CGWB_FRN_CNT.
 *
 * The mechanism only remembers IDs and doesn't hold any object references.
 * As being wrong occasionally doesn't matter, updates and accesses to the
 * records are lockless and racy.
 */
void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio,
					     struct bdi_writeback *wb)
{
	struct mem_cgroup *memcg = folio_memcg(folio);
	struct memcg_cgwb_frn *frn;
	u64 now = get_jiffies_64();
	u64 oldest_at = now;
	int oldest = -1;
	int i;

	trace_track_foreign_dirty(folio, wb);

	/*
	 * Pick the slot to use.  If there is already a slot for @wb, keep
	 * using it.  If not replace the oldest one which isn't being
	 * written out.
	 */
	for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++) {
		frn = &memcg->cgwb_frn[i];
		if (frn->bdi_id == wb->bdi->id &&
		    frn->memcg_id == wb->memcg_css->id)
			break;
		if (time_before64(frn->at, oldest_at) &&
		    atomic_read(&frn->done.cnt) == 1) {
			oldest = i;
			oldest_at = frn->at;
		}
	}

	if (i < MEMCG_CGWB_FRN_CNT) {
		/*
		 * Re-using an existing one.  Update timestamp lazily to
		 * avoid making the cacheline hot.  We want them to be
		 * reasonably up-to-date and significantly shorter than
		 * dirty_expire_interval as that's what expires the record.
		 * Use the shorter of 1s and dirty_expire_interval / 8.
		 */
		unsigned long update_intv =
			min_t(unsigned long, HZ,
			      msecs_to_jiffies(dirty_expire_interval * 10) / 8);

		if (time_before64(frn->at, now - update_intv))
			frn->at = now;
	} else if (oldest >= 0) {
		/* replace the oldest free one */
		frn = &memcg->cgwb_frn[oldest];
		frn->bdi_id = wb->bdi->id;
		frn->memcg_id = wb->memcg_css->id;
		frn->at = now;
	}
}

/* issue foreign writeback flushes for recorded foreign dirtying events */
void mem_cgroup_flush_foreign(struct bdi_writeback *wb)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);
	unsigned long intv = msecs_to_jiffies(dirty_expire_interval * 10);
	u64 now = jiffies_64;
	int i;

	for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++) {
		struct memcg_cgwb_frn *frn = &memcg->cgwb_frn[i];

		/*
		 * If the record is older than dirty_expire_interval,
		 * writeback on it has already started.  No need to kick it
		 * off again.  Also, don't start a new one if there's
		 * already one in flight.
		 */
		if (time_after64(frn->at, now - intv) &&
		    atomic_read(&frn->done.cnt) == 1) {
			frn->at = 0;
			trace_flush_foreign(wb, frn->bdi_id, frn->memcg_id);
			cgroup_writeback_by_id(frn->bdi_id, frn->memcg_id,
					       WB_REASON_FOREIGN_FLUSH,
					       &frn->done);
		}
	}
}

#else	/* CONFIG_CGROUP_WRITEBACK */

static int memcg_wb_domain_init(struct mem_cgroup *memcg, gfp_t gfp)
{
	return 0;
}

static void memcg_wb_domain_exit(struct mem_cgroup *memcg)
{
}

static void memcg_wb_domain_size_changed(struct mem_cgroup *memcg)
{
}

#endif	/* CONFIG_CGROUP_WRITEBACK */

/*
 * Private memory cgroup IDR
 *
 * Swap-out records and page cache shadow entries need to store memcg
 * references in constrained space, so we maintain an ID space that is
 * limited to 16 bit (MEM_CGROUP_ID_MAX), limiting the total number of
 * memory-controlled cgroups to 64k.
 *
 * However, there usually are many references to the offline CSS after
 * the cgroup has been destroyed, such as page cache or reclaimable
 * slab objects, that don't need to hang on to the ID. We want to keep
 * those dead CSS from occupying IDs, or we might quickly exhaust the
 * relatively small ID space and prevent the creation of new cgroups
 * even when there are much fewer than 64k cgroups - possibly none.
 *
 * Maintain a private 16-bit ID space for memcg, and allow the ID to
 * be freed and recycled when it's no longer needed, which is usually
 * when the CSS is offlined.
 *
 * The only exception to that are records of swapped out tmpfs/shmem
 * pages that need to be attributed to live ancestors on swapin. But
 * those references are manageable from userspace.
 */

#define MEM_CGROUP_ID_MAX	((1UL << MEM_CGROUP_ID_SHIFT) - 1)
static DEFINE_XARRAY_ALLOC1(mem_cgroup_private_ids);

static void mem_cgroup_private_id_remove(struct mem_cgroup *memcg)
{
	if (memcg->id.id > 0) {
		xa_erase(&mem_cgroup_private_ids, memcg->id.id);
		memcg->id.id = 0;
	}
}

void __maybe_unused mem_cgroup_private_id_get_many(struct mem_cgroup *memcg,
					   unsigned int n)
{
	refcount_add(n, &memcg->id.ref);
}

static void mem_cgroup_private_id_put_many(struct mem_cgroup *memcg, unsigned int n)
{
	if (refcount_sub_and_test(n, &memcg->id.ref)) {
		mem_cgroup_private_id_remove(memcg);

		/* Memcg ID pins CSS */
		css_put(&memcg->css);
	}
}

static inline void mem_cgroup_private_id_put(struct mem_cgroup *memcg)
{
	mem_cgroup_private_id_put_many(memcg, 1);
}

struct mem_cgroup *mem_cgroup_private_id_get_online(struct mem_cgroup *memcg)
{
	while (!refcount_inc_not_zero(&memcg->id.ref)) {
		/*
		 * The root cgroup cannot be destroyed, so it's refcount must
		 * always be >= 1.
		 */
		if (WARN_ON_ONCE(mem_cgroup_is_root(memcg))) {
			VM_BUG_ON(1);
			break;
		}
		memcg = parent_mem_cgroup(memcg);
		if (!memcg)
			memcg = root_mem_cgroup;
	}
	return memcg;
}

/**
 * mem_cgroup_from_private_id - look up a memcg from a memcg id
 * @id: the memcg id to look up
 *
 * Caller must hold rcu_read_lock().
 */
struct mem_cgroup *mem_cgroup_from_private_id(unsigned short id)
{
	WARN_ON_ONCE(!rcu_read_lock_held());
	return xa_load(&mem_cgroup_private_ids, id);
}

struct mem_cgroup *mem_cgroup_get_from_id(u64 id)
{
	struct cgroup *cgrp;
	struct cgroup_subsys_state *css;
	struct mem_cgroup *memcg = NULL;

	cgrp = cgroup_get_from_id(id);
	if (IS_ERR(cgrp))
		return NULL;

	css = cgroup_get_e_css(cgrp, &memory_cgrp_subsys);
	if (css)
		memcg = container_of(css, struct mem_cgroup, css);

	cgroup_put(cgrp);

	return memcg;
}

static void free_mem_cgroup_per_node_info(struct mem_cgroup_per_node *pn)
{
	if (!pn)
		return;

	free_percpu(pn->lruvec_stats_percpu);
	kfree(pn->lruvec_stats);
	kfree(pn);
}

static bool alloc_mem_cgroup_per_node_info(struct mem_cgroup *memcg, int node)
{
	struct mem_cgroup_per_node *pn;

	pn = kmem_cache_alloc_node(memcg_pn_cachep, GFP_KERNEL | __GFP_ZERO,
				   node);
	if (!pn)
		return false;

	pn->lruvec_stats = kzalloc_node(sizeof(struct lruvec_stats),
					GFP_KERNEL_ACCOUNT, node);
	if (!pn->lruvec_stats)
		goto fail;

	pn->lruvec_stats_percpu = alloc_percpu_gfp(struct lruvec_stats_percpu,
						   GFP_KERNEL_ACCOUNT);
	if (!pn->lruvec_stats_percpu)
		goto fail;

	lruvec_init(&pn->lruvec);
	pn->memcg = memcg;

	memcg->nodeinfo[node] = pn;
	return true;
fail:
	free_mem_cgroup_per_node_info(pn);
	return false;
}

static void __mem_cgroup_free(struct mem_cgroup *memcg)
{
	int node;

	obj_cgroup_put(memcg->orig_objcg);

	for_each_node(node)
		free_mem_cgroup_per_node_info(memcg->nodeinfo[node]);
	memcg1_free_events(memcg);
	kfree(memcg->vmstats);
	free_percpu(memcg->vmstats_percpu);
	kfree(memcg);
}

static void mem_cgroup_free(struct mem_cgroup *memcg)
{
	lru_gen_exit_memcg(memcg);
	memcg_wb_domain_exit(memcg);
	__mem_cgroup_free(memcg);
}

static struct mem_cgroup *mem_cgroup_alloc(struct mem_cgroup *parent)
{
	struct memcg_vmstats_percpu *statc;
	struct memcg_vmstats_percpu __percpu *pstatc_pcpu;
	struct mem_cgroup *memcg;
	int node, cpu;
	int __maybe_unused i;
	long error;

	memcg = kmem_cache_zalloc(memcg_cachep, GFP_KERNEL);
	if (!memcg)
		return ERR_PTR(-ENOMEM);

	error = xa_alloc(&mem_cgroup_private_ids, &memcg->id.id, NULL,
			 XA_LIMIT(1, MEM_CGROUP_ID_MAX), GFP_KERNEL);
	if (error)
		goto fail;
	error = -ENOMEM;

	memcg->vmstats = kzalloc_obj(struct memcg_vmstats, GFP_KERNEL_ACCOUNT);
	if (!memcg->vmstats)
		goto fail;

	memcg->vmstats_percpu = alloc_percpu_gfp(struct memcg_vmstats_percpu,
						 GFP_KERNEL_ACCOUNT);
	if (!memcg->vmstats_percpu)
		goto fail;

	if (!memcg1_alloc_events(memcg))
		goto fail;

	for_each_possible_cpu(cpu) {
		if (parent)
			pstatc_pcpu = parent->vmstats_percpu;
		statc = per_cpu_ptr(memcg->vmstats_percpu, cpu);
		statc->parent_pcpu = parent ? pstatc_pcpu : NULL;
		statc->vmstats = memcg->vmstats;
	}

	for_each_node(node)
		if (!alloc_mem_cgroup_per_node_info(memcg, node))
			goto fail;

	if (memcg_wb_domain_init(memcg, GFP_KERNEL))
		goto fail;

	INIT_WORK(&memcg->high_work, high_work_func);
	vmpressure_init(&memcg->vmpressure);
	INIT_LIST_HEAD(&memcg->memory_peaks);
	INIT_LIST_HEAD(&memcg->swap_peaks);
	spin_lock_init(&memcg->peaks_lock);
	memcg->socket_pressure = get_jiffies_64();
#if BITS_PER_LONG < 64
	seqlock_init(&memcg->socket_pressure_seqlock);
#endif
	memcg1_memcg_init(memcg);
	memcg->kmemcg_id = -1;
	INIT_LIST_HEAD(&memcg->objcg_list);
#ifdef CONFIG_CGROUP_WRITEBACK
	INIT_LIST_HEAD(&memcg->cgwb_list);
	for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++)
		memcg->cgwb_frn[i].done =
			__WB_COMPLETION_INIT(&memcg_cgwb_frn_waitq);
#endif
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	spin_lock_init(&memcg->deferred_split_queue.split_queue_lock);
	INIT_LIST_HEAD(&memcg->deferred_split_queue.split_queue);
	memcg->deferred_split_queue.split_queue_len = 0;
#endif
	lru_gen_init_memcg(memcg);
	return memcg;
fail:
	mem_cgroup_private_id_remove(memcg);
	__mem_cgroup_free(memcg);
	return ERR_PTR(error);
}

static struct cgroup_subsys_state * __ref
mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
{
	struct mem_cgroup *parent = mem_cgroup_from_css(parent_css);
	struct mem_cgroup *memcg, *old_memcg;
	bool memcg_on_dfl = cgroup_subsys_on_dfl(memory_cgrp_subsys);

	old_memcg = set_active_memcg(parent);
	memcg = mem_cgroup_alloc(parent);
	set_active_memcg(old_memcg);
	if (IS_ERR(memcg))
		return ERR_CAST(memcg);

	page_counter_set_high(&memcg->memory, PAGE_COUNTER_MAX);
	memcg1_soft_limit_reset(memcg);
#ifdef CONFIG_ZSWAP
	memcg->zswap_max = PAGE_COUNTER_MAX;
	WRITE_ONCE(memcg->zswap_writeback, true);
#endif
	page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX);
	if (parent) {
		WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent));

		page_counter_init(&memcg->memory, &parent->memory, memcg_on_dfl);
		page_counter_init(&memcg->swap, &parent->swap, false);
#ifdef CONFIG_MEMCG_V1
		memcg->memory.track_failcnt = !memcg_on_dfl;
		WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable));
		page_counter_init(&memcg->kmem, &parent->kmem, false);
		page_counter_init(&memcg->tcpmem, &parent->tcpmem, false);
#endif
	} else {
		init_memcg_stats();
		init_memcg_events();
		page_counter_init(&memcg->memory, NULL, true);
		page_counter_init(&memcg->swap, NULL, false);
#ifdef CONFIG_MEMCG_V1
		page_counter_init(&memcg->kmem, NULL, false);
		page_counter_init(&memcg->tcpmem, NULL, false);
#endif
		root_mem_cgroup = memcg;
		return &memcg->css;
	}

	if (memcg_on_dfl && !cgroup_memory_nosocket)
		static_branch_inc(&memcg_sockets_enabled_key);

	if (!cgroup_memory_nobpf)
		static_branch_inc(&memcg_bpf_enabled_key);

	return &memcg->css;
}

static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(css);

	if (memcg_online_kmem(memcg))
		goto remove_id;

	/*
	 * A memcg must be visible for expand_shrinker_info()
	 * by the time the maps are allocated. So, we allocate maps
	 * here, when for_each_mem_cgroup() can't skip it.
	 */
	if (alloc_shrinker_info(memcg))
		goto offline_kmem;

	if (unlikely(mem_cgroup_is_root(memcg)) && !mem_cgroup_disabled())
		queue_delayed_work(system_dfl_wq, &stats_flush_dwork,
				   FLUSH_TIME);
	lru_gen_online_memcg(memcg);

	/* Online state pins memcg ID, memcg ID pins CSS */
	refcount_set(&memcg->id.ref, 1);
	css_get(css);

	/*
	 * Ensure mem_cgroup_from_private_id() works once we're fully online.
	 *
	 * We could do this earlier and require callers to filter with
	 * css_tryget_online(). But right now there are no users that
	 * need earlier access, and the workingset code relies on the
	 * cgroup tree linkage (mem_cgroup_get_nr_swap_pages()). So
	 * publish it here at the end of onlining. This matches the
	 * regular ID destruction during offlining.
	 */
	xa_store(&mem_cgroup_private_ids, memcg->id.id, memcg, GFP_KERNEL);

	return 0;
offline_kmem:
	memcg_offline_kmem(memcg);
remove_id:
	mem_cgroup_private_id_remove(memcg);
	return -ENOMEM;
}

static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(css);

	memcg1_css_offline(memcg);

	page_counter_set_min(&memcg->memory, 0);
	page_counter_set_low(&memcg->memory, 0);

	zswap_memcg_offline_cleanup(memcg);

	memcg_offline_kmem(memcg);
	reparent_deferred_split_queue(memcg);
	reparent_shrinker_deferred(memcg);
	wb_memcg_offline(memcg);
	lru_gen_offline_memcg(memcg);

	drain_all_stock(memcg);

	mem_cgroup_private_id_put(memcg);
}

static void mem_cgroup_css_released(struct cgroup_subsys_state *css)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(css);

	invalidate_reclaim_iterators(memcg);
	lru_gen_release_memcg(memcg);
}

static void mem_cgroup_css_free(struct cgroup_subsys_state *css)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
	int __maybe_unused i;

#ifdef CONFIG_CGROUP_WRITEBACK
	for (i = 0; i < MEMCG_CGWB_FRN_CNT; i++)
		wb_wait_for_completion(&memcg->cgwb_frn[i].done);
#endif
	if (cgroup_subsys_on_dfl(memory_cgrp_subsys) && !cgroup_memory_nosocket)
		static_branch_dec(&memcg_sockets_enabled_key);

	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg1_tcpmem_active(memcg))
		static_branch_dec(&memcg_sockets_enabled_key);

	if (!cgroup_memory_nobpf)
		static_branch_dec(&memcg_bpf_enabled_key);

	vmpressure_cleanup(&memcg->vmpressure);
	cancel_work_sync(&memcg->high_work);
	memcg1_remove_from_trees(memcg);
	free_shrinker_info(memcg);
	mem_cgroup_free(memcg);
}

/**
 * mem_cgroup_css_reset - reset the states of a mem_cgroup
 * @css: the target css
 *
 * Reset the states of the mem_cgroup associated with @css.  This is
 * invoked when the userland requests disabling on the default hierarchy
 * but the memcg is pinned through dependency.  The memcg should stop
 * applying policies and should revert to the vanilla state as it may be
 * made visible again.
 *
 * The current implementation only resets the essential configurations.
 * This needs to be expanded to cover all the visible parts.
 */
static void mem_cgroup_css_reset(struct cgroup_subsys_state *css)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(css);

	page_counter_set_max(&memcg->memory, PAGE_COUNTER_MAX);
	page_counter_set_max(&memcg->swap, PAGE_COUNTER_MAX);
#ifdef CONFIG_MEMCG_V1
	page_counter_set_max(&memcg->kmem, PAGE_COUNTER_MAX);
	page_counter_set_max(&memcg->tcpmem, PAGE_COUNTER_MAX);
#endif
	page_counter_set_min(&memcg->memory, 0);
	page_counter_set_low(&memcg->memory, 0);
	page_counter_set_high(&memcg->memory, PAGE_COUNTER_MAX);
	memcg1_soft_limit_reset(memcg);
	page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX);
	memcg_wb_domain_size_changed(memcg);
}

struct aggregate_control {
	/* pointer to the aggregated (CPU and subtree aggregated) counters */
	long *aggregate;
	/* pointer to the non-hierarchichal (CPU aggregated) counters */
	long *local;
	/* pointer to the pending child counters during tree propagation */
	long *pending;
	/* pointer to the parent's pending counters, could be NULL */
	long *ppending;
	/* pointer to the percpu counters to be aggregated */
	long *cstat;
	/* pointer to the percpu counters of the last aggregation*/
	long *cstat_prev;
	/* size of the above counters */
	int size;
};

static void mem_cgroup_stat_aggregate(struct aggregate_control *ac)
{
	int i;
	long delta, delta_cpu, v;

	for (i = 0; i < ac->size; i++) {
		/*
		 * Collect the aggregated propagation counts of groups
		 * below us. We're in a per-cpu loop here and this is
		 * a global counter, so the first cycle will get them.
		 */
		delta = ac->pending[i];
		if (delta)
			ac->pending[i] = 0;

		/* Add CPU changes on this level since the last flush */
		delta_cpu = 0;
		v = READ_ONCE(ac->cstat[i]);
		if (v != ac->cstat_prev[i]) {
			delta_cpu = v - ac->cstat_prev[i];
			delta += delta_cpu;
			ac->cstat_prev[i] = v;
		}

		/* Aggregate counts on this level and propagate upwards */
		if (delta_cpu)
			ac->local[i] += delta_cpu;

		if (delta) {
			ac->aggregate[i] += delta;
			if (ac->ppending)
				ac->ppending[i] += delta;
		}
	}
}

#ifdef CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC
static void flush_nmi_stats(struct mem_cgroup *memcg, struct mem_cgroup *parent,
			    int cpu)
{
	int nid;

	if (atomic_read(&memcg->kmem_stat)) {
		int kmem = atomic_xchg(&memcg->kmem_stat, 0);
		int index = memcg_stats_index(MEMCG_KMEM);

		memcg->vmstats->state[index] += kmem;
		if (parent)
			parent->vmstats->state_pending[index] += kmem;
	}

	for_each_node_state(nid, N_MEMORY) {
		struct mem_cgroup_per_node *pn = memcg->nodeinfo[nid];
		struct lruvec_stats *lstats = pn->lruvec_stats;
		struct lruvec_stats *plstats = NULL;

		if (parent)
			plstats = parent->nodeinfo[nid]->lruvec_stats;

		if (atomic_read(&pn->slab_reclaimable)) {
			int slab = atomic_xchg(&pn->slab_reclaimable, 0);
			int index = memcg_stats_index(NR_SLAB_RECLAIMABLE_B);

			lstats->state[index] += slab;
			if (plstats)
				plstats->state_pending[index] += slab;
		}
		if (atomic_read(&pn->slab_unreclaimable)) {
			int slab = atomic_xchg(&pn->slab_unreclaimable, 0);
			int index = memcg_stats_index(NR_SLAB_UNRECLAIMABLE_B);

			lstats->state[index] += slab;
			if (plstats)
				plstats->state_pending[index] += slab;
		}
	}
}
#else
static void flush_nmi_stats(struct mem_cgroup *memcg, struct mem_cgroup *parent,
			    int cpu)
{}
#endif

static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(css);
	struct mem_cgroup *parent = parent_mem_cgroup(memcg);
	struct memcg_vmstats_percpu *statc;
	struct aggregate_control ac;
	int nid;

	flush_nmi_stats(memcg, parent, cpu);

	statc = per_cpu_ptr(memcg->vmstats_percpu, cpu);

	ac = (struct aggregate_control) {
		.aggregate = memcg->vmstats->state,
		.local = memcg->vmstats->state_local,
		.pending = memcg->vmstats->state_pending,
		.ppending = parent ? parent->vmstats->state_pending : NULL,
		.cstat = statc->state,
		.cstat_prev = statc->state_prev,
		.size = MEMCG_VMSTAT_SIZE,
	};
	mem_cgroup_stat_aggregate(&ac);

	ac = (struct aggregate_control) {
		.aggregate = memcg->vmstats->events,
		.local = memcg->vmstats->events_local,
		.pending = memcg->vmstats->events_pending,
		.ppending = parent ? parent->vmstats->events_pending : NULL,
		.cstat = statc->events,
		.cstat_prev = statc->events_prev,
		.size = NR_MEMCG_EVENTS,
	};
	mem_cgroup_stat_aggregate(&ac);

	for_each_node_state(nid, N_MEMORY) {
		struct mem_cgroup_per_node *pn = memcg->nodeinfo[nid];
		struct lruvec_stats *lstats = pn->lruvec_stats;
		struct lruvec_stats *plstats = NULL;
		struct lruvec_stats_percpu *lstatc;

		if (parent)
			plstats = parent->nodeinfo[nid]->lruvec_stats;

		lstatc = per_cpu_ptr(pn->lruvec_stats_percpu, cpu);

		ac = (struct aggregate_control) {
			.aggregate = lstats->state,
			.local = lstats->state_local,
			.pending = lstats->state_pending,
			.ppending = plstats ? plstats->state_pending : NULL,
			.cstat = lstatc->state,
			.cstat_prev = lstatc->state_prev,
			.size = NR_MEMCG_NODE_STAT_ITEMS,
		};
		mem_cgroup_stat_aggregate(&ac);

	}
	WRITE_ONCE(statc->stats_updates, 0);
	/* We are in a per-cpu loop here, only do the atomic write once */
	if (atomic_read(&memcg->vmstats->stats_updates))
		atomic_set(&memcg->vmstats->stats_updates, 0);
}

static void mem_cgroup_fork(struct task_struct *task)
{
	/*
	 * Set the update flag to cause task->objcg to be initialized lazily
	 * on the first allocation. It can be done without any synchronization
	 * because it's always performed on the current task, so does
	 * current_objcg_update().
	 */
	task->objcg = (struct obj_cgroup *)CURRENT_OBJCG_UPDATE_FLAG;
}

static void mem_cgroup_exit(struct task_struct *task)
{
	struct obj_cgroup *objcg = task->objcg;

	objcg = (struct obj_cgroup *)
		((unsigned long)objcg & ~CURRENT_OBJCG_UPDATE_FLAG);
	obj_cgroup_put(objcg);

	/*
	 * Some kernel allocations can happen after this point,
	 * but let's ignore them. It can be done without any synchronization
	 * because it's always performed on the current task, so does
	 * current_objcg_update().
	 */
	task->objcg = NULL;
}

#ifdef CONFIG_LRU_GEN
static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset)
{
	struct task_struct *task;
	struct cgroup_subsys_state *css;

	/* find the first leader if there is any */
	cgroup_taskset_for_each_leader(task, css, tset)
		break;

	if (!task)
		return;

	task_lock(task);
	if (task->mm && READ_ONCE(task->mm->owner) == task)
		lru_gen_migrate_mm(task->mm);
	task_unlock(task);
}
#else
static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) {}
#endif /* CONFIG_LRU_GEN */

static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset)
{
	struct task_struct *task;
	struct cgroup_subsys_state *css;

	cgroup_taskset_for_each(task, css, tset) {
		/* atomically set the update bit */
		set_bit(CURRENT_OBJCG_UPDATE_BIT, (unsigned long *)&task->objcg);
	}
}

static void mem_cgroup_attach(struct cgroup_taskset *tset)
{
	mem_cgroup_lru_gen_attach(tset);
	mem_cgroup_kmem_attach(tset);
}

static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value)
{
	if (value == PAGE_COUNTER_MAX)
		seq_puts(m, "max\n");
	else
		seq_printf(m, "%llu\n", (u64)value * PAGE_SIZE);

	return 0;
}

static u64 memory_current_read(struct cgroup_subsys_state *css,
			       struct cftype *cft)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(css);

	return (u64)page_counter_read(&memcg->memory) * PAGE_SIZE;
}

#define OFP_PEAK_UNSET (((-1UL)))

static int peak_show(struct seq_file *sf, void *v, struct page_counter *pc)
{
	struct cgroup_of_peak *ofp = of_peak(sf->private);
	u64 fd_peak = READ_ONCE(ofp->value), peak;

	/* User wants global or local peak? */
	if (fd_peak == OFP_PEAK_UNSET)
		peak = pc->watermark;
	else
		peak = max(fd_peak, READ_ONCE(pc->local_watermark));

	seq_printf(sf, "%llu\n", peak * PAGE_SIZE);
	return 0;
}

static int memory_peak_show(struct seq_file *sf, void *v)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(seq_css(sf));

	return peak_show(sf, v, &memcg->memory);
}

static int peak_open(struct kernfs_open_file *of)
{
	struct cgroup_of_peak *ofp = of_peak(of);

	ofp->value = OFP_PEAK_UNSET;
	return 0;
}

static void peak_release(struct kernfs_open_file *of)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	struct cgroup_of_peak *ofp = of_peak(of);

	if (ofp->value == OFP_PEAK_UNSET) {
		/* fast path (no writes on this fd) */
		return;
	}
	spin_lock(&memcg->peaks_lock);
	list_del(&ofp->list);
	spin_unlock(&memcg->peaks_lock);
}

static ssize_t peak_write(struct kernfs_open_file *of, char *buf, size_t nbytes,
			  loff_t off, struct page_counter *pc,
			  struct list_head *watchers)
{
	unsigned long usage;
	struct cgroup_of_peak *peer_ctx;
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	struct cgroup_of_peak *ofp = of_peak(of);

	spin_lock(&memcg->peaks_lock);

	usage = page_counter_read(pc);
	WRITE_ONCE(pc->local_watermark, usage);

	list_for_each_entry(peer_ctx, watchers, list)
		if (usage > peer_ctx->value)
			WRITE_ONCE(peer_ctx->value, usage);

	/* initial write, register watcher */
	if (ofp->value == OFP_PEAK_UNSET)
		list_add(&ofp->list, watchers);

	WRITE_ONCE(ofp->value, usage);
	spin_unlock(&memcg->peaks_lock);

	return nbytes;
}

static ssize_t memory_peak_write(struct kernfs_open_file *of, char *buf,
				 size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));

	return peak_write(of, buf, nbytes, off, &memcg->memory,
			  &memcg->memory_peaks);
}

#undef OFP_PEAK_UNSET

static int memory_min_show(struct seq_file *m, void *v)
{
	return seq_puts_memcg_tunable(m,
		READ_ONCE(mem_cgroup_from_seq(m)->memory.min));
}

static ssize_t memory_min_write(struct kernfs_open_file *of,
				char *buf, size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	unsigned long min;
	int err;

	buf = strstrip(buf);
	err = page_counter_memparse(buf, "max", &min);
	if (err)
		return err;

	page_counter_set_min(&memcg->memory, min);

	return nbytes;
}

static int memory_low_show(struct seq_file *m, void *v)
{
	return seq_puts_memcg_tunable(m,
		READ_ONCE(mem_cgroup_from_seq(m)->memory.low));
}

static ssize_t memory_low_write(struct kernfs_open_file *of,
				char *buf, size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	unsigned long low;
	int err;

	buf = strstrip(buf);
	err = page_counter_memparse(buf, "max", &low);
	if (err)
		return err;

	page_counter_set_low(&memcg->memory, low);

	return nbytes;
}

static int memory_high_show(struct seq_file *m, void *v)
{
	return seq_puts_memcg_tunable(m,
		READ_ONCE(mem_cgroup_from_seq(m)->memory.high));
}

static ssize_t memory_high_write(struct kernfs_open_file *of,
				 char *buf, size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	unsigned int nr_retries = MAX_RECLAIM_RETRIES;
	bool drained = false;
	unsigned long high;
	int err;

	buf = strstrip(buf);
	err = page_counter_memparse(buf, "max", &high);
	if (err)
		return err;

	page_counter_set_high(&memcg->memory, high);

	if (of->file->f_flags & O_NONBLOCK)
		goto out;

	for (;;) {
		unsigned long nr_pages = page_counter_read(&memcg->memory);
		unsigned long reclaimed;

		if (nr_pages <= high)
			break;

		if (signal_pending(current))
			break;

		if (!drained) {
			drain_all_stock(memcg);
			drained = true;
			continue;
		}

		reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_pages - high,
					GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP, NULL);

		if (!reclaimed && !nr_retries--)
			break;
	}
out:
	memcg_wb_domain_size_changed(memcg);
	return nbytes;
}

static int memory_max_show(struct seq_file *m, void *v)
{
	return seq_puts_memcg_tunable(m,
		READ_ONCE(mem_cgroup_from_seq(m)->memory.max));
}

static ssize_t memory_max_write(struct kernfs_open_file *of,
				char *buf, size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	unsigned int nr_reclaims = MAX_RECLAIM_RETRIES;
	bool drained = false;
	unsigned long max;
	int err;

	buf = strstrip(buf);
	err = page_counter_memparse(buf, "max", &max);
	if (err)
		return err;

	xchg(&memcg->memory.max, max);

	if (of->file->f_flags & O_NONBLOCK)
		goto out;

	for (;;) {
		unsigned long nr_pages = page_counter_read(&memcg->memory);

		if (nr_pages <= max)
			break;

		if (signal_pending(current))
			break;

		if (!drained) {
			drain_all_stock(memcg);
			drained = true;
			continue;
		}

		if (nr_reclaims) {
			if (!try_to_free_mem_cgroup_pages(memcg, nr_pages - max,
					GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP, NULL))
				nr_reclaims--;
			continue;
		}

		memcg_memory_event(memcg, MEMCG_OOM);
		if (!mem_cgroup_out_of_memory(memcg, GFP_KERNEL, 0))
			break;
		cond_resched();
	}
out:
	memcg_wb_domain_size_changed(memcg);
	return nbytes;
}

/*
 * Note: don't forget to update the 'samples/cgroup/memcg_event_listener'
 * if any new events become available.
 */
static void __memory_events_show(struct seq_file *m, atomic_long_t *events)
{
	seq_printf(m, "low %lu\n", atomic_long_read(&events[MEMCG_LOW]));
	seq_printf(m, "high %lu\n", atomic_long_read(&events[MEMCG_HIGH]));
	seq_printf(m, "max %lu\n", atomic_long_read(&events[MEMCG_MAX]));
	seq_printf(m, "oom %lu\n", atomic_long_read(&events[MEMCG_OOM]));
	seq_printf(m, "oom_kill %lu\n",
		   atomic_long_read(&events[MEMCG_OOM_KILL]));
	seq_printf(m, "oom_group_kill %lu\n",
		   atomic_long_read(&events[MEMCG_OOM_GROUP_KILL]));
	seq_printf(m, "sock_throttled %lu\n",
		   atomic_long_read(&events[MEMCG_SOCK_THROTTLED]));
}

static int memory_events_show(struct seq_file *m, void *v)
{
	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);

	__memory_events_show(m, memcg->memory_events);
	return 0;
}

static int memory_events_local_show(struct seq_file *m, void *v)
{
	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);

	__memory_events_show(m, memcg->memory_events_local);
	return 0;
}

int memory_stat_show(struct seq_file *m, void *v)
{
	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
	char *buf = kmalloc(SEQ_BUF_SIZE, GFP_KERNEL);
	struct seq_buf s;

	if (!buf)
		return -ENOMEM;
	seq_buf_init(&s, buf, SEQ_BUF_SIZE);
	memory_stat_format(memcg, &s);
	seq_puts(m, buf);
	kfree(buf);
	return 0;
}

#ifdef CONFIG_NUMA
static inline unsigned long lruvec_page_state_output(struct lruvec *lruvec,
						     int item)
{
	return lruvec_page_state(lruvec, item) *
		memcg_page_state_output_unit(item);
}

static int memory_numa_stat_show(struct seq_file *m, void *v)
{
	int i;
	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);

	mem_cgroup_flush_stats(memcg);

	for (i = 0; i < ARRAY_SIZE(memory_stats); i++) {
		int nid;

		if (memory_stats[i].idx >= NR_VM_NODE_STAT_ITEMS)
			continue;

		seq_printf(m, "%s", memory_stats[i].name);
		for_each_node_state(nid, N_MEMORY) {
			u64 size;
			struct lruvec *lruvec;

			lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
			size = lruvec_page_state_output(lruvec,
							memory_stats[i].idx);
			seq_printf(m, " N%d=%llu", nid, size);
		}
		seq_putc(m, '\n');
	}

	return 0;
}
#endif

static int memory_oom_group_show(struct seq_file *m, void *v)
{
	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);

	seq_printf(m, "%d\n", READ_ONCE(memcg->oom_group));

	return 0;
}

static ssize_t memory_oom_group_write(struct kernfs_open_file *of,
				      char *buf, size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	int ret, oom_group;

	buf = strstrip(buf);
	if (!buf)
		return -EINVAL;

	ret = kstrtoint(buf, 0, &oom_group);
	if (ret)
		return ret;

	if (oom_group != 0 && oom_group != 1)
		return -EINVAL;

	WRITE_ONCE(memcg->oom_group, oom_group);

	return nbytes;
}

static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf,
			      size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	int ret;

	ret = user_proactive_reclaim(buf, memcg, NULL);
	if (ret)
		return ret;

	return nbytes;
}

static struct cftype memory_files[] = {
	{
		.name = "current",
		.flags = CFTYPE_NOT_ON_ROOT,
		.read_u64 = memory_current_read,
	},
	{
		.name = "peak",
		.flags = CFTYPE_NOT_ON_ROOT,
		.open = peak_open,
		.release = peak_release,
		.seq_show = memory_peak_show,
		.write = memory_peak_write,
	},
	{
		.name = "min",
		.flags = CFTYPE_NOT_ON_ROOT,
		.seq_show = memory_min_show,
		.write = memory_min_write,
	},
	{
		.name = "low",
		.flags = CFTYPE_NOT_ON_ROOT,
		.seq_show = memory_low_show,
		.write = memory_low_write,
	},
	{
		.name = "high",
		.flags = CFTYPE_NOT_ON_ROOT,
		.seq_show = memory_high_show,
		.write = memory_high_write,
	},
	{
		.name = "max",
		.flags = CFTYPE_NOT_ON_ROOT,
		.seq_show = memory_max_show,
		.write = memory_max_write,
	},
	{
		.name = "events",
		.flags = CFTYPE_NOT_ON_ROOT,
		.file_offset = offsetof(struct mem_cgroup, events_file),
		.seq_show = memory_events_show,
	},
	{
		.name = "events.local",
		.flags = CFTYPE_NOT_ON_ROOT,
		.file_offset = offsetof(struct mem_cgroup, events_local_file),
		.seq_show = memory_events_local_show,
	},
	{
		.name = "stat",
		.seq_show = memory_stat_show,
	},
#ifdef CONFIG_NUMA
	{
		.name = "numa_stat",
		.seq_show = memory_numa_stat_show,
	},
#endif
	{
		.name = "oom.group",
		.flags = CFTYPE_NOT_ON_ROOT | CFTYPE_NS_DELEGATABLE,
		.seq_show = memory_oom_group_show,
		.write = memory_oom_group_write,
	},
	{
		.name = "reclaim",
		.flags = CFTYPE_NS_DELEGATABLE,
		.write = memory_reclaim,
	},
	{ }	/* terminate */
};

struct cgroup_subsys memory_cgrp_subsys = {
	.css_alloc = mem_cgroup_css_alloc,
	.css_online = mem_cgroup_css_online,
	.css_offline = mem_cgroup_css_offline,
	.css_released = mem_cgroup_css_released,
	.css_free = mem_cgroup_css_free,
	.css_reset = mem_cgroup_css_reset,
	.css_rstat_flush = mem_cgroup_css_rstat_flush,
	.attach = mem_cgroup_attach,
	.fork = mem_cgroup_fork,
	.exit = mem_cgroup_exit,
	.dfl_cftypes = memory_files,
#ifdef CONFIG_MEMCG_V1
	.legacy_cftypes = mem_cgroup_legacy_files,
#endif
	.early_init = 0,
};

/**
 * mem_cgroup_calculate_protection - check if memory consumption is in the normal range
 * @root: the top ancestor of the sub-tree being checked
 * @memcg: the memory cgroup to check
 *
 * WARNING: This function is not stateless! It can only be used as part
 *          of a top-down tree iteration, not for isolated queries.
 */
void mem_cgroup_calculate_protection(struct mem_cgroup *root,
				     struct mem_cgroup *memcg)
{
	bool recursive_protection =
		cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT;

	if (mem_cgroup_disabled())
		return;

	if (!root)
		root = root_mem_cgroup;

	page_counter_calculate_protection(&root->memory, &memcg->memory, recursive_protection);
}

static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg,
			gfp_t gfp)
{
	int ret;

	ret = try_charge(memcg, gfp, folio_nr_pages(folio));
	if (ret)
		goto out;

	css_get(&memcg->css);
	commit_charge(folio, memcg);
	memcg1_commit_charge(folio, memcg);
out:
	return ret;
}

int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp)
{
	struct mem_cgroup *memcg;
	int ret;

	memcg = get_mem_cgroup_from_mm(mm);
	ret = charge_memcg(folio, memcg, gfp);
	css_put(&memcg->css);

	return ret;
}

/**
 * mem_cgroup_charge_hugetlb - charge the memcg for a hugetlb folio
 * @folio: folio being charged
 * @gfp: reclaim mode
 *
 * This function is called when allocating a huge page folio, after the page has
 * already been obtained and charged to the appropriate hugetlb cgroup
 * controller (if it is enabled).
 *
 * Returns ENOMEM if the memcg is already full.
 * Returns 0 if either the charge was successful, or if we skip the charging.
 */
int mem_cgroup_charge_hugetlb(struct folio *folio, gfp_t gfp)
{
	struct mem_cgroup *memcg = get_mem_cgroup_from_current();
	int ret = 0;

	/*
	 * Even memcg does not account for hugetlb, we still want to update
	 * system-level stats via lruvec_stat_mod_folio. Return 0, and skip
	 * charging the memcg.
	 */
	if (mem_cgroup_disabled() || !memcg_accounts_hugetlb() ||
		!memcg || !cgroup_subsys_on_dfl(memory_cgrp_subsys))
		goto out;

	if (charge_memcg(folio, memcg, gfp))
		ret = -ENOMEM;

out:
	mem_cgroup_put(memcg);
	return ret;
}

/**
 * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin.
 * @folio: folio to charge.
 * @mm: mm context of the victim
 * @gfp: reclaim mode
 * @entry: swap entry for which the folio is allocated
 *
 * This function charges a folio allocated for swapin. Please call this before
 * adding the folio to the swapcache.
 *
 * Returns 0 on success. Otherwise, an error code is returned.
 */
int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
				  gfp_t gfp, swp_entry_t entry)
{
	struct mem_cgroup *memcg;
	unsigned short id;
	int ret;

	if (mem_cgroup_disabled())
		return 0;

	id = lookup_swap_cgroup_id(entry);
	rcu_read_lock();
	memcg = mem_cgroup_from_private_id(id);
	if (!memcg || !css_tryget_online(&memcg->css))
		memcg = get_mem_cgroup_from_mm(mm);
	rcu_read_unlock();

	ret = charge_memcg(folio, memcg, gfp);

	css_put(&memcg->css);
	return ret;
}

struct uncharge_gather {
	struct mem_cgroup *memcg;
	unsigned long nr_memory;
	unsigned long pgpgout;
	unsigned long nr_kmem;
	int nid;
};

static inline void uncharge_gather_clear(struct uncharge_gather *ug)
{
	memset(ug, 0, sizeof(*ug));
}

static void uncharge_batch(const struct uncharge_gather *ug)
{
	if (ug->nr_memory) {
		memcg_uncharge(ug->memcg, ug->nr_memory);
		if (ug->nr_kmem) {
			mod_memcg_state(ug->memcg, MEMCG_KMEM, -ug->nr_kmem);
			memcg1_account_kmem(ug->memcg, -ug->nr_kmem);
		}
		memcg1_oom_recover(ug->memcg);
	}

	memcg1_uncharge_batch(ug->memcg, ug->pgpgout, ug->nr_memory, ug->nid);

	/* drop reference from uncharge_folio */
	css_put(&ug->memcg->css);
}

static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug)
{
	long nr_pages;
	struct mem_cgroup *memcg;
	struct obj_cgroup *objcg;

	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);

	/*
	 * Nobody should be changing or seriously looking at
	 * folio memcg or objcg at this point, we have fully
	 * exclusive access to the folio.
	 */
	if (folio_memcg_kmem(folio)) {
		objcg = __folio_objcg(folio);
		/*
		 * This get matches the put at the end of the function and
		 * kmem pages do not hold memcg references anymore.
		 */
		memcg = get_mem_cgroup_from_objcg(objcg);
	} else {
		memcg = __folio_memcg(folio);
	}

	if (!memcg)
		return;

	if (ug->memcg != memcg) {
		if (ug->memcg) {
			uncharge_batch(ug);
			uncharge_gather_clear(ug);
		}
		ug->memcg = memcg;
		ug->nid = folio_nid(folio);

		/* pairs with css_put in uncharge_batch */
		css_get(&memcg->css);
	}

	nr_pages = folio_nr_pages(folio);

	if (folio_memcg_kmem(folio)) {
		ug->nr_memory += nr_pages;
		ug->nr_kmem += nr_pages;

		folio->memcg_data = 0;
		obj_cgroup_put(objcg);
	} else {
		/* LRU pages aren't accounted at the root level */
		if (!mem_cgroup_is_root(memcg))
			ug->nr_memory += nr_pages;
		ug->pgpgout++;

		WARN_ON_ONCE(folio_unqueue_deferred_split(folio));
		folio->memcg_data = 0;
	}

	css_put(&memcg->css);
}

void __mem_cgroup_uncharge(struct folio *folio)
{
	struct uncharge_gather ug;

	/* Don't touch folio->lru of any random page, pre-check: */
	if (!folio_memcg_charged(folio))
		return;

	uncharge_gather_clear(&ug);
	uncharge_folio(folio, &ug);
	uncharge_batch(&ug);
}

void __mem_cgroup_uncharge_folios(struct folio_batch *folios)
{
	struct uncharge_gather ug;
	unsigned int i;

	uncharge_gather_clear(&ug);
	for (i = 0; i < folios->nr; i++)
		uncharge_folio(folios->folios[i], &ug);
	if (ug.memcg)
		uncharge_batch(&ug);
}

/**
 * mem_cgroup_replace_folio - Charge a folio's replacement.
 * @old: Currently circulating folio.
 * @new: Replacement folio.
 *
 * Charge @new as a replacement folio for @old. @old will
 * be uncharged upon free.
 *
 * Both folios must be locked, @new->mapping must be set up.
 */
void mem_cgroup_replace_folio(struct folio *old, struct folio *new)
{
	struct mem_cgroup *memcg;
	long nr_pages = folio_nr_pages(new);

	VM_BUG_ON_FOLIO(!folio_test_locked(old), old);
	VM_BUG_ON_FOLIO(!folio_test_locked(new), new);
	VM_BUG_ON_FOLIO(folio_test_anon(old) != folio_test_anon(new), new);
	VM_BUG_ON_FOLIO(folio_nr_pages(old) != nr_pages, new);

	if (mem_cgroup_disabled())
		return;

	/* Page cache replacement: new folio already charged? */
	if (folio_memcg_charged(new))
		return;

	memcg = folio_memcg(old);
	VM_WARN_ON_ONCE_FOLIO(!memcg, old);
	if (!memcg)
		return;

	/* Force-charge the new page. The old one will be freed soon */
	if (!mem_cgroup_is_root(memcg)) {
		page_counter_charge(&memcg->memory, nr_pages);
		if (do_memsw_account())
			page_counter_charge(&memcg->memsw, nr_pages);
	}

	css_get(&memcg->css);
	commit_charge(new, memcg);
	memcg1_commit_charge(new, memcg);
}

/**
 * mem_cgroup_migrate - Transfer the memcg data from the old to the new folio.
 * @old: Currently circulating folio.
 * @new: Replacement folio.
 *
 * Transfer the memcg data from the old folio to the new folio for migration.
 * The old folio's data info will be cleared. Note that the memory counters
 * will remain unchanged throughout the process.
 *
 * Both folios must be locked, @new->mapping must be set up.
 */
void mem_cgroup_migrate(struct folio *old, struct folio *new)
{
	struct mem_cgroup *memcg;

	VM_BUG_ON_FOLIO(!folio_test_locked(old), old);
	VM_BUG_ON_FOLIO(!folio_test_locked(new), new);
	VM_BUG_ON_FOLIO(folio_test_anon(old) != folio_test_anon(new), new);
	VM_BUG_ON_FOLIO(folio_nr_pages(old) != folio_nr_pages(new), new);
	VM_BUG_ON_FOLIO(folio_test_lru(old), old);

	if (mem_cgroup_disabled())
		return;

	memcg = folio_memcg(old);
	/*
	 * Note that it is normal to see !memcg for a hugetlb folio.
	 * For e.g, it could have been allocated when memory_hugetlb_accounting
	 * was not selected.
	 */
	VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(old) && !memcg, old);
	if (!memcg)
		return;

	/* Transfer the charge and the css ref */
	commit_charge(new, memcg);

	/* Warning should never happen, so don't worry about refcount non-0 */
	WARN_ON_ONCE(folio_unqueue_deferred_split(old));
	old->memcg_data = 0;
}

DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key);
EXPORT_SYMBOL(memcg_sockets_enabled_key);

void mem_cgroup_sk_alloc(struct sock *sk)
{
	struct mem_cgroup *memcg;

	if (!mem_cgroup_sockets_enabled)
		return;

	/* Do not associate the sock with unrelated interrupted task's memcg. */
	if (!in_task())
		return;

	rcu_read_lock();
	memcg = mem_cgroup_from_task(current);
	if (mem_cgroup_is_root(memcg))
		goto out;
	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && !memcg1_tcpmem_active(memcg))
		goto out;
	if (css_tryget(&memcg->css))
		sk->sk_memcg = memcg;
out:
	rcu_read_unlock();
}

void mem_cgroup_sk_free(struct sock *sk)
{
	struct mem_cgroup *memcg = mem_cgroup_from_sk(sk);

	if (memcg)
		css_put(&memcg->css);
}

void mem_cgroup_sk_inherit(const struct sock *sk, struct sock *newsk)
{
	struct mem_cgroup *memcg;

	if (sk->sk_memcg == newsk->sk_memcg)
		return;

	mem_cgroup_sk_free(newsk);

	memcg = mem_cgroup_from_sk(sk);
	if (memcg)
		css_get(&memcg->css);

	newsk->sk_memcg = sk->sk_memcg;
}

/**
 * mem_cgroup_sk_charge - charge socket memory
 * @sk: socket in memcg to charge
 * @nr_pages: number of pages to charge
 * @gfp_mask: reclaim mode
 *
 * Charges @nr_pages to @memcg. Returns %true if the charge fit within
 * @memcg's configured limit, %false if it doesn't.
 */
bool mem_cgroup_sk_charge(const struct sock *sk, unsigned int nr_pages,
			  gfp_t gfp_mask)
{
	struct mem_cgroup *memcg = mem_cgroup_from_sk(sk);

	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
		return memcg1_charge_skmem(memcg, nr_pages, gfp_mask);

	if (try_charge_memcg(memcg, gfp_mask, nr_pages) == 0) {
		mod_memcg_state(memcg, MEMCG_SOCK, nr_pages);
		return true;
	}

	return false;
}

/**
 * mem_cgroup_sk_uncharge - uncharge socket memory
 * @sk: socket in memcg to uncharge
 * @nr_pages: number of pages to uncharge
 */
void mem_cgroup_sk_uncharge(const struct sock *sk, unsigned int nr_pages)
{
	struct mem_cgroup *memcg = mem_cgroup_from_sk(sk);

	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) {
		memcg1_uncharge_skmem(memcg, nr_pages);
		return;
	}

	mod_memcg_state(memcg, MEMCG_SOCK, -nr_pages);

	refill_stock(memcg, nr_pages);
}

void mem_cgroup_flush_workqueue(void)
{
	flush_workqueue(memcg_wq);
}

static int __init cgroup_memory(char *s)
{
	char *token;

	while ((token = strsep(&s, ",")) != NULL) {
		if (!*token)
			continue;
		if (!strcmp(token, "nosocket"))
			cgroup_memory_nosocket = true;
		if (!strcmp(token, "nokmem"))
			cgroup_memory_nokmem = true;
		if (!strcmp(token, "nobpf"))
			cgroup_memory_nobpf = true;
	}
	return 1;
}
__setup("cgroup.memory=", cgroup_memory);

/*
 * Memory controller init before cgroup_init() initialize root_mem_cgroup.
 *
 * Some parts like memcg_hotplug_cpu_dead() have to be initialized from this
 * context because of lock dependencies (cgroup_lock -> cpu hotplug) but
 * basically everything that doesn't depend on a specific mem_cgroup structure
 * should be initialized from here.
 */
int __init mem_cgroup_init(void)
{
	unsigned int memcg_size;
	int cpu;

	/*
	 * Currently s32 type (can refer to struct batched_lruvec_stat) is
	 * used for per-memcg-per-cpu caching of per-node statistics. In order
	 * to work fine, we should make sure that the overfill threshold can't
	 * exceed S32_MAX / PAGE_SIZE.
	 */
	BUILD_BUG_ON(MEMCG_CHARGE_BATCH > S32_MAX / PAGE_SIZE);

	cpuhp_setup_state_nocalls(CPUHP_MM_MEMCQ_DEAD, "mm/memctrl:dead", NULL,
				  memcg_hotplug_cpu_dead);

	memcg_wq = alloc_workqueue("memcg", WQ_PERCPU, 0);
	WARN_ON(!memcg_wq);

	for_each_possible_cpu(cpu) {
		INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work,
			  drain_local_memcg_stock);
		INIT_WORK(&per_cpu_ptr(&obj_stock, cpu)->work,
			  drain_local_obj_stock);
	}

	memcg_size = struct_size_t(struct mem_cgroup, nodeinfo, nr_node_ids);
	memcg_cachep = kmem_cache_create("mem_cgroup", memcg_size, 0,
					 SLAB_PANIC | SLAB_HWCACHE_ALIGN, NULL);

	memcg_pn_cachep = KMEM_CACHE(mem_cgroup_per_node,
				     SLAB_PANIC | SLAB_HWCACHE_ALIGN);

	return 0;
}

#ifdef CONFIG_SWAP
/**
 * __mem_cgroup_try_charge_swap - try charging swap space for a folio
 * @folio: folio being added to swap
 * @entry: swap entry to charge
 *
 * Try to charge @folio's memcg for the swap space at @entry.
 *
 * Returns 0 on success, -ENOMEM on failure.
 */
int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry)
{
	unsigned int nr_pages = folio_nr_pages(folio);
	struct page_counter *counter;
	struct mem_cgroup *memcg;

	if (do_memsw_account())
		return 0;

	memcg = folio_memcg(folio);

	VM_WARN_ON_ONCE_FOLIO(!memcg, folio);
	if (!memcg)
		return 0;

	if (!entry.val) {
		memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
		return 0;
	}

	memcg = mem_cgroup_private_id_get_online(memcg);

	if (!mem_cgroup_is_root(memcg) &&
	    !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {
		memcg_memory_event(memcg, MEMCG_SWAP_MAX);
		memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
		mem_cgroup_private_id_put(memcg);
		return -ENOMEM;
	}

	/* Get references for the tail pages, too */
	if (nr_pages > 1)
		mem_cgroup_private_id_get_many(memcg, nr_pages - 1);
	mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);

	swap_cgroup_record(folio, mem_cgroup_private_id(memcg), entry);

	return 0;
}

/**
 * __mem_cgroup_uncharge_swap - uncharge swap space
 * @entry: swap entry to uncharge
 * @nr_pages: the amount of swap space to uncharge
 */
void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)
{
	struct mem_cgroup *memcg;
	unsigned short id;

	id = swap_cgroup_clear(entry, nr_pages);
	rcu_read_lock();
	memcg = mem_cgroup_from_private_id(id);
	if (memcg) {
		if (!mem_cgroup_is_root(memcg)) {
			if (do_memsw_account())
				page_counter_uncharge(&memcg->memsw, nr_pages);
			else
				page_counter_uncharge(&memcg->swap, nr_pages);
		}
		mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
		mem_cgroup_private_id_put_many(memcg, nr_pages);
	}
	rcu_read_unlock();
}

long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
{
	long nr_swap_pages = get_nr_swap_pages();

	if (mem_cgroup_disabled() || do_memsw_account())
		return nr_swap_pages;
	for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg))
		nr_swap_pages = min_t(long, nr_swap_pages,
				      READ_ONCE(memcg->swap.max) -
				      page_counter_read(&memcg->swap));
	return nr_swap_pages;
}

bool mem_cgroup_swap_full(struct folio *folio)
{
	struct mem_cgroup *memcg;

	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);

	if (vm_swap_full())
		return true;
	if (do_memsw_account())
		return false;

	memcg = folio_memcg(folio);
	if (!memcg)
		return false;

	for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
		unsigned long usage = page_counter_read(&memcg->swap);

		if (usage * 2 >= READ_ONCE(memcg->swap.high) ||
		    usage * 2 >= READ_ONCE(memcg->swap.max))
			return true;
	}

	return false;
}

static int __init setup_swap_account(char *s)
{
	bool res;

	if (!kstrtobool(s, &res) && !res)
		pr_warn_once("The swapaccount=0 commandline option is deprecated "
			     "in favor of configuring swap control via cgroupfs. "
			     "Please report your usecase to linux-mm@kvack.org if you "
			     "depend on this functionality.\n");
	return 1;
}
__setup("swapaccount=", setup_swap_account);

static u64 swap_current_read(struct cgroup_subsys_state *css,
			     struct cftype *cft)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(css);

	return (u64)page_counter_read(&memcg->swap) * PAGE_SIZE;
}

static int swap_peak_show(struct seq_file *sf, void *v)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(seq_css(sf));

	return peak_show(sf, v, &memcg->swap);
}

static ssize_t swap_peak_write(struct kernfs_open_file *of, char *buf,
			       size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));

	return peak_write(of, buf, nbytes, off, &memcg->swap,
			  &memcg->swap_peaks);
}

static int swap_high_show(struct seq_file *m, void *v)
{
	return seq_puts_memcg_tunable(m,
		READ_ONCE(mem_cgroup_from_seq(m)->swap.high));
}

static ssize_t swap_high_write(struct kernfs_open_file *of,
			       char *buf, size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	unsigned long high;
	int err;

	buf = strstrip(buf);
	err = page_counter_memparse(buf, "max", &high);
	if (err)
		return err;

	page_counter_set_high(&memcg->swap, high);

	return nbytes;
}

static int swap_max_show(struct seq_file *m, void *v)
{
	return seq_puts_memcg_tunable(m,
		READ_ONCE(mem_cgroup_from_seq(m)->swap.max));
}

static ssize_t swap_max_write(struct kernfs_open_file *of,
			      char *buf, size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	unsigned long max;
	int err;

	buf = strstrip(buf);
	err = page_counter_memparse(buf, "max", &max);
	if (err)
		return err;

	xchg(&memcg->swap.max, max);

	return nbytes;
}

static int swap_events_show(struct seq_file *m, void *v)
{
	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);

	seq_printf(m, "high %lu\n",
		   atomic_long_read(&memcg->memory_events[MEMCG_SWAP_HIGH]));
	seq_printf(m, "max %lu\n",
		   atomic_long_read(&memcg->memory_events[MEMCG_SWAP_MAX]));
	seq_printf(m, "fail %lu\n",
		   atomic_long_read(&memcg->memory_events[MEMCG_SWAP_FAIL]));

	return 0;
}

static struct cftype swap_files[] = {
	{
		.name = "swap.current",
		.flags = CFTYPE_NOT_ON_ROOT,
		.read_u64 = swap_current_read,
	},
	{
		.name = "swap.high",
		.flags = CFTYPE_NOT_ON_ROOT,
		.seq_show = swap_high_show,
		.write = swap_high_write,
	},
	{
		.name = "swap.max",
		.flags = CFTYPE_NOT_ON_ROOT,
		.seq_show = swap_max_show,
		.write = swap_max_write,
	},
	{
		.name = "swap.peak",
		.flags = CFTYPE_NOT_ON_ROOT,
		.open = peak_open,
		.release = peak_release,
		.seq_show = swap_peak_show,
		.write = swap_peak_write,
	},
	{
		.name = "swap.events",
		.flags = CFTYPE_NOT_ON_ROOT,
		.file_offset = offsetof(struct mem_cgroup, swap_events_file),
		.seq_show = swap_events_show,
	},
	{ }	/* terminate */
};

#ifdef CONFIG_ZSWAP
/**
 * obj_cgroup_may_zswap - check if this cgroup can zswap
 * @objcg: the object cgroup
 *
 * Check if the hierarchical zswap limit has been reached.
 *
 * This doesn't check for specific headroom, and it is not atomic
 * either. But with zswap, the size of the allocation is only known
 * once compression has occurred, and this optimistic pre-check avoids
 * spending cycles on compression when there is already no room left
 * or zswap is disabled altogether somewhere in the hierarchy.
 */
bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
{
	struct mem_cgroup *memcg, *original_memcg;
	bool ret = true;

	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
		return true;

	original_memcg = get_mem_cgroup_from_objcg(objcg);
	for (memcg = original_memcg; !mem_cgroup_is_root(memcg);
	     memcg = parent_mem_cgroup(memcg)) {
		unsigned long max = READ_ONCE(memcg->zswap_max);
		unsigned long pages;

		if (max == PAGE_COUNTER_MAX)
			continue;
		if (max == 0) {
			ret = false;
			break;
		}

		/* Force flush to get accurate stats for charging */
		__mem_cgroup_flush_stats(memcg, true);
		pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE;
		if (pages < max)
			continue;
		ret = false;
		break;
	}
	mem_cgroup_put(original_memcg);
	return ret;
}

/**
 * obj_cgroup_charge_zswap - charge compression backend memory
 * @objcg: the object cgroup
 * @size: size of compressed object
 *
 * This forces the charge after obj_cgroup_may_zswap() allowed
 * compression and storage in zswap for this cgroup to go ahead.
 */
void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size)
{
	struct mem_cgroup *memcg;

	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
		return;

	VM_WARN_ON_ONCE(!(current->flags & PF_MEMALLOC));

	/* PF_MEMALLOC context, charging must succeed */
	if (obj_cgroup_charge(objcg, GFP_KERNEL, size))
		VM_WARN_ON_ONCE(1);

	rcu_read_lock();
	memcg = obj_cgroup_memcg(objcg);
	mod_memcg_state(memcg, MEMCG_ZSWAP_B, size);
	mod_memcg_state(memcg, MEMCG_ZSWAPPED, 1);
	rcu_read_unlock();
}

/**
 * obj_cgroup_uncharge_zswap - uncharge compression backend memory
 * @objcg: the object cgroup
 * @size: size of compressed object
 *
 * Uncharges zswap memory on page in.
 */
void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size)
{
	struct mem_cgroup *memcg;

	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
		return;

	obj_cgroup_uncharge(objcg, size);

	rcu_read_lock();
	memcg = obj_cgroup_memcg(objcg);
	mod_memcg_state(memcg, MEMCG_ZSWAP_B, -size);
	mod_memcg_state(memcg, MEMCG_ZSWAPPED, -1);
	rcu_read_unlock();
}

bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg)
{
	/* if zswap is disabled, do not block pages going to the swapping device */
	if (!zswap_is_enabled())
		return true;

	for (; memcg; memcg = parent_mem_cgroup(memcg))
		if (!READ_ONCE(memcg->zswap_writeback))
			return false;

	return true;
}

static u64 zswap_current_read(struct cgroup_subsys_state *css,
			      struct cftype *cft)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(css);

	mem_cgroup_flush_stats(memcg);
	return memcg_page_state(memcg, MEMCG_ZSWAP_B);
}

static int zswap_max_show(struct seq_file *m, void *v)
{
	return seq_puts_memcg_tunable(m,
		READ_ONCE(mem_cgroup_from_seq(m)->zswap_max));
}

static ssize_t zswap_max_write(struct kernfs_open_file *of,
			       char *buf, size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	unsigned long max;
	int err;

	buf = strstrip(buf);
	err = page_counter_memparse(buf, "max", &max);
	if (err)
		return err;

	xchg(&memcg->zswap_max, max);

	return nbytes;
}

static int zswap_writeback_show(struct seq_file *m, void *v)
{
	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);

	seq_printf(m, "%d\n", READ_ONCE(memcg->zswap_writeback));
	return 0;
}

static ssize_t zswap_writeback_write(struct kernfs_open_file *of,
				char *buf, size_t nbytes, loff_t off)
{
	struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of));
	int zswap_writeback;
	ssize_t parse_ret = kstrtoint(strstrip(buf), 0, &zswap_writeback);

	if (parse_ret)
		return parse_ret;

	if (zswap_writeback != 0 && zswap_writeback != 1)
		return -EINVAL;

	WRITE_ONCE(memcg->zswap_writeback, zswap_writeback);
	return nbytes;
}

static struct cftype zswap_files[] = {
	{
		.name = "zswap.current",
		.flags = CFTYPE_NOT_ON_ROOT,
		.read_u64 = zswap_current_read,
	},
	{
		.name = "zswap.max",
		.flags = CFTYPE_NOT_ON_ROOT,
		.seq_show = zswap_max_show,
		.write = zswap_max_write,
	},
	{
		.name = "zswap.writeback",
		.seq_show = zswap_writeback_show,
		.write = zswap_writeback_write,
	},
	{ }	/* terminate */
};
#endif /* CONFIG_ZSWAP */

static int __init mem_cgroup_swap_init(void)
{
	if (mem_cgroup_disabled())
		return 0;

	WARN_ON(cgroup_add_dfl_cftypes(&memory_cgrp_subsys, swap_files));
#ifdef CONFIG_MEMCG_V1
	WARN_ON(cgroup_add_legacy_cftypes(&memory_cgrp_subsys, memsw_files));
#endif
#ifdef CONFIG_ZSWAP
	WARN_ON(cgroup_add_dfl_cftypes(&memory_cgrp_subsys, zswap_files));
#endif
	return 0;
}
subsys_initcall(mem_cgroup_swap_init);

#endif /* CONFIG_SWAP */

void mem_cgroup_node_filter_allowed(struct mem_cgroup *memcg, nodemask_t *mask)
{
	nodemask_t allowed;

	if (!memcg)
		return;

	/*
	 * Since this interface is intended for use by migration paths, and
	 * reclaim and migration are subject to race conditions such as changes
	 * in effective_mems and hot-unpluging of nodes, inaccurate allowed
	 * mask is acceptable.
	 */
	cpuset_nodes_allowed(memcg->css.cgroup, &allowed);
	nodes_and(*mask, *mask, allowed);
}

void mem_cgroup_show_protected_memory(struct mem_cgroup *memcg)
{
	if (mem_cgroup_disabled() || !cgroup_subsys_on_dfl(memory_cgrp_subsys))
		return;

	if (!memcg)
		memcg = root_mem_cgroup;

	pr_warn("Memory cgroup min protection %lukB -- low protection %lukB",
		K(atomic_long_read(&memcg->memory.children_min_usage)),
		K(atomic_long_read(&memcg->memory.children_low_usage)));
}
]

gemini-3-flash-preview

							input: 140094
							output: 19
							thoughts: 2132
						

**Analysis of a Data Race in `mm/memcontrol.c` Concerning `jiffies_64`**

Okay, so I'm looking at this code in `mm/memcontrol.c` and immediately notice an inconsistency. I see `get_jiffies_64()` being used in a few places like `mem_cgroup_alloc` and `mem_cgroup_track_foreign_dirty_slowpath`, but then in `mem_cgroup_flush_stats_ratelimited`, it's directly using `jiffies_64`.  And then I also see it directly used in `mem_cgroup_flush_foreign`.  This looks like a potential data race.

Now, is this actually a problem?  Well, I have to consider a few things. First, on 64-bit systems, a read of a 64-bit value should be atomic, so that's good. Second, even on 32-bit systems, a "torn read" wouldn't cause a crash; it'd just lead to a potentially slightly incorrect timing decision – in this case, for ratelimiting or flushing foreign dirty pages, which isn't critical.  So, in terms of memory corruption or crashes, it's probably safe.

But, a data race is still a data race, and KCSAN is correctly flagging it. The fact that `jiffies_64` isn't declared `volatile` is a slight concern. Theoretically, the compiler could optimize the read in unexpected ways, although it's unlikely to be a problem in this context, where we're just reading a value for an `if` condition.

Looking at `include/linux/jiffies.h`, `jiffies` is volatile, but `jiffies_64` isn't. Interesting. On a 64-bit system, they are at the same address, so reading `jiffies` would give us a volatile load. However, `get_jiffies_64()` on 64-bit returns `(u64)jiffies`, so it's already a volatile load.  The ideal fix would be to use `get_jiffies_64()` everywhere.

But given that this data race is on a counter used for ratelimiting (and doesn't lead to any critical failures), I'm pretty comfortable calling it "benign." It just might lead to a premature or delayed stats flush, but nothing catastrophic.

Let me make sure I'm not missing anything. I need to think about the update to `jiffies_64` in `tick_do_update_jiffies64`. That's a non-atomic update, but it's protected by `jiffies_lock` and `jiffies_seq`. The readers here don't hold the lock, but the worst that could happen is a torn read on a 32-bit system. On 64-bit, it's atomic. And the report I'm looking at is for x86_64, so it's atomic.

Okay, so the data race *is* benign. It's on a time counter for ratelimiting, and an imprecise value won't break anything. And since KCSAN flags many jiffies races, I should check if this is known and already ignored.

Now, should I be absolutely certain? Let me run through the “harmful” criteria again.  A data race is harmful if it causes corruption or crash, even with a conservative compiler, which this won't. It's not a race on a complex container that needs mutual exclusion. It's a race on a simple `u64`. The update to the value is just `jiffies_64 += ticks`, and the reader is reading `jiffies_64` or `READ_ONCE(flush_last_time)`. This is only on `jiffies_64`.

The lack of `volatile` for `jiffies_64` isn't a problem, since the compiler can read it as a single read. `time_after64`'s macros should be fine too; they only read the value once. And I’ve also confirmed `jiffies_64` is aligned, making 64-bit reads atomic.

I'm confident.

Wait, I should double check that I am correct! I will scan through the other uses of `jiffies_64` in `mm/memcontrol.c`. I also see a `WRITE_ONCE(flush_last_time, jiffies_64)`, which is another potential data race with `tick_do_update_j

map[Benign:true Confident:true]

map[Benign:true Confident:true]

gemini-3-flash-preview

							input: 143287
							output: 804
							thoughts: 266
						

**My Analysis of the Data Race**

Okay, I've finished digging into this. The report flags a data race, and it looks like it's between `tick_do_update_jiffies64` writing to `jiffies_64` and `mem_cgroup_flush_stats_ratelimited` reading that same variable. Initially, that raises a red flag, but I've drilled down, and I'm confident this isn't a true problem. Here's why:

First, and this is the most crucial, on the 64-bit architecture that we're talking about, an 8-byte load or store on a `u64` like `jiffies_64` is atomic. No problem there.

Second, consider what `jiffies_64` actually *is*. It's a time counter. The worst-case scenario here is that `mem_cgroup_flush_stats_ratelimited` reads a slightly stale or slightly advanced value. That's entirely acceptable in the context of ratelimiting. It might mean a minor deviation in the rate, but it's not a functional defect.

Third, even if we were dealing with a 32-bit system, where the read *could* be torn, the potential consequences are still benign. At worst, we'd have a temporary inaccuracy in the ratelimit check. The function might skip a flush it should have performed, or it might perform an extra one. But critically, this won't cause memory corruption, crashes, or any significant malfunction.

Finally, the kernel does actually provide `get_jiffies_64()` specifically to handle 32-bit systems correctly. Ideally, `mm/memcontrol.c` would be using that. However, even with the direct access that's currently in place, this isn't harmful. This data race is a non-issue and can be safely ignored.