syzbot

map[Benign:true Confident:true Explanation:The data race reported is between `has_bh_in_lru()` (reading a pointer) and `__invalidate_bh_lrus()` (writing `NULL` to the same pointer).

### Analysis

1.  **Functionality**:
    *   `has_bh_in_lru()` is a predicate function used to check if a specific CPU's buffer-head LRU cache contains any entries. It is used as an optimization in `invalidate_bh_lrus()` and `lru_add_drain_all()` to decide whether an IPI (Inter-Processor Interrupt) needs to be sent to that CPU to flush its cache.
    *   `__invalidate_bh_lrus()` is the function that actually clears the LRU cache by iterating through the entries, calling `brelse()` on them, and setting the pointers to `NULL`.

2.  **The Race**:
    *   The race occurs when one CPU (CPU 1 in the report) is executing `invalidate_bh_lrus()` (e.g., during a filesystem unmount) and checks `has_bh_in_lru()` for another CPU (CPU 0), while CPU 0 is simultaneously clearing its own LRU cache via `invalidate_bh_lrus_cpu()` (triggered by a workqueue).

3.  **Why it is benign**:
    *   **Atomic Access**: The variable being accessed is a pointer (`struct buffer_head *`). On 64-bit systems like the one in the report, 8-byte aligned pointer writes and reads are atomic. CPU 1 will either see a valid pointer or `NULL`.
    *   **No Dereference**: `has_bh_in_lru()` only checks if the pointer is non-NULL (`if (b->bhs[i])`). It does not dereference the pointer, so there is no risk of a use-after-free or NULL pointer dereference even if the buffer head was just released.
    *   **Safe Outcomes**:
        *   If `has_bh_in_lru()` reads the pointer before it is cleared, it returns `true`, and an IPI is sent to CPU 0. CPU 0 will then run the invalidation again, see that the LRU is already empty (or becoming empty), and do nothing. This is a harmless extra IPI.
        *   If `has_bh_in_lru()` reads `NULL` because the pointer was just cleared, it returns `false`, and no IPI is sent. This is correct because the LRU is already being cleared.
    *   **Optimization Pattern**: This is a common pattern in the kernel where a lockless "quick check" is performed to avoid expensive operations (like IPIs). Such races are considered benign as they do not affect the eventual consistency or stability of the system.

### Conclusion
The data race is **benign**. It is a racy optimization check on a pointer value that does not lead to memory corruption, crashes, or incorrect logic, as both sides of the race are working towards the same goal of emptying the LRU cache.]

map[KernelSrc:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/src/22b53f5d3c32fa7512dcfc9c7f3ae37c008c3955]

map[KernelObj:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/build/f088e7c04026f6805eaa66f1d1ae7669f331d79f]

map[Index:codesearch-index]

gemini-3-flash-preview

map[Benign:true Confident:true]

You are an experienced Linux kernel developer tasked with determining if the given kernel
data race is benign or not. The data race report is from KCSAN tool.
It contains 2 stack traces of the memory accesses that constitute a data race.

A "benign" data races are on a simple int/bool variable or similar field,
and the accesses are not supposed to be protected by any mutual exclusion primitives.
Common examples of such "benign" data races are accesses to various flags fields,
statistics counters, and similar. A "benign" data race does not lead to memory corruption/crash
with a conservative compiler that compiles memory accesses to primitive types
effectively as atomic.

A non-benign (or "harmful" data race) can lead to corruption/crash even with
a conservative compiler that compiles memory accesses to primitive types
effectively as atomic. A common example of a "harmful" data races is race on
a complex container (list/hashmap/etc), where accesses are supposed to be protected
by a mutual exclusion primitive.

Also consider races that happen at the same time with the given one.
If there is no synchronization in between, other memory accesses in the involved threads
race with each other if they access the same memory. For example, if both threads execute:

	some_struct->have_elements = true;
	list_add(new_node, &some_struct->list_head);

the race on some_struct->have_elements may appear benign, however it also implies there
is a race on some_struct->list_head which is not benign, since the list is not thread-safe.

Take into account that on 32-bit systems 64-bit memory accesses may be split into two accesses,
and thus even with a conservative compiler may not be fully atomic. However, such races may
still be benign depending on how writes are done, and how read data is used.

In the final reply explain why you think the given data race is benign or is harmful.

Use the provided tools to confirm any assumptions, variables/fields being accessed, etc.
In particular, don't make assumptions about the kernel source code,
use codesearch tools to read the actual source code.

Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

The data race report is:

EXT4-fs (loop4): unmounting filesystem 00000000-0000-0006-0000-000000000000.
EXT4-fs (loop4): unmounting filesystem 00000000-0000-0000-0000-000000000000.
==================================================================
BUG: KCSAN: data-race in has_bh_in_lru / invalidate_bh_lrus_cpu

write to 0xffff888237c27760 of 8 bytes by task 3409 on cpu 0:
 __invalidate_bh_lrus fs/buffer.c:1526 [inline]
 invalidate_bh_lrus_cpu+0x83/0xc0 fs/buffer.c:1571
 lru_add_and_bh_lrus_drain mm/swap.c:753 [inline]
 lru_add_drain_per_cpu+0x55/0x70 mm/swap.c:772
 process_one_work kernel/workqueue.c:3275 [inline]
 process_scheduled_works+0x4de/0x9e0 kernel/workqueue.c:3358
 worker_thread+0x581/0x770 kernel/workqueue.c:3439
 kthread+0x22a/0x280 kernel/kthread.c:467
 ret_from_fork+0x150/0x360 arch/x86/kernel/process.c:158
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245

read to 0xffff888237c27760 of 8 bytes by task 3315 on cpu 1:
 has_bh_in_lru+0x35/0x1f0 fs/buffer.c:1548
 smp_call_function_many_cond+0x2c1/0xc30 kernel/smp.c:821
 on_each_cpu_cond_mask+0x3c/0x80 kernel/smp.c:1043
 on_each_cpu_cond include/linux/smp.h:106 [inline]
 invalidate_bh_lrus+0x2a/0x30 fs/buffer.c:1557
 invalidate_bdev+0x42/0x70 block/bdev.c:100
 ext4_put_super+0x624/0x7d0 fs/ext4/super.c:1353
 generic_shutdown_super+0xee/0x220 fs/super.c:646
 kill_block_super+0x2a/0x70 fs/super.c:1725
 ext4_kill_sb+0x42/0x80 fs/ext4/super.c:7459
 deactivate_locked_super+0x75/0x1c0 fs/super.c:476
 deactivate_super+0x97/0xa0 fs/super.c:509
 cleanup_mnt+0x2bb/0x330 fs/namespace.c:1312
 __cleanup_mnt+0x19/0x20 fs/namespace.c:1319
 task_work_run+0x130/0x1a0 kernel/task_work.c:233
 resume_user_mode_work include/linux/resume_user_mode.h:50 [inline]
 __exit_to_user_mode_loop kernel/entry/common.c:67 [inline]
 exit_to_user_mode_loop+0x1f4/0x6f0 kernel/entry/common.c:98
 __exit_to_user_mode_prepare include/linux/irq-entry-common.h:226 [inline]
 syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:256 [inline]
 syscall_exit_to_user_mode include/linux/entry-common.h:325 [inline]
 do_syscall_64+0x249/0x370 arch/x86/entry/syscall_64.c:100
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

value changed: 0xffff88810645fc30 -> 0x0000000000000000

Reported by Kernel Concurrency Sanitizer on:
CPU: 1 UID: 0 PID: 3315 Comm: syz-executor Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2026
==================================================================

The data race reported is between `has_bh_in_lru()` (reading a pointer) and `__invalidate_bh_lrus()` (writing `NULL` to the same pointer).

### Analysis

1.  **Functionality**:
    *   `has_bh_in_lru()` is a predicate function used to check if a specific CPU's buffer-head LRU cache contains any entries. It is used as an optimization in `invalidate_bh_lrus()` and `lru_add_drain_all()` to decide whether an IPI (Inter-Processor Interrupt) needs to be sent to that CPU to flush its cache.
    *   `__invalidate_bh_lrus()` is the function that actually clears the LRU cache by iterating through the entries, calling `brelse()` on them, and setting the pointers to `NULL`.

2.  **The Race**:
    *   The race occurs when one CPU (CPU 1 in the report) is executing `invalidate_bh_lrus()` (e.g., during a filesystem unmount) and checks `has_bh_in_lru()` for another CPU (CPU 0), while CPU 0 is simultaneously clearing its own LRU cache via `invalidate_bh_lrus_cpu()` (triggered by a workqueue).

3.  **Why it is benign**:
    *   **Atomic Access**: The variable being accessed is a pointer (`struct buffer_head *`). On 64-bit systems like the one in the report, 8-byte aligned pointer writes and reads are atomic. CPU 1 will either see a valid pointer or `NULL`.
    *   **No Dereference**: `has_bh_in_lru()` only checks if the pointer is non-NULL (`if (b->bhs[i])`). It does not dereference the pointer, so there is no risk of a use-after-free or NULL pointer dereference even if the buffer head was just released.
    *   **Safe Outcomes**:
        *   If `has_bh_in_lru()` reads the pointer before it is cleared, it returns `true`, and an IPI is sent to CPU 0. CPU 0 will then run the invalidation again, see that the LRU is already empty (or becoming empty), and do nothing. This is a harmless extra IPI.
        *   If `has_bh_in_lru()` reads `NULL` because the pointer was just cleared, it returns `false`, and no IPI is sent. This is correct because the LRU is already being cleared.
    *   **Optimization Pattern**: This is a common pattern in the kernel where a lockless "quick check" is performed to avoid expensive operations (like IPIs). Such races are considered benign as they do not affect the eventual consistency or stability of the system.

### Conclusion
The data race is **benign**. It is a racy optimization check on a pointer value that does not lead to memory corruption, crashes, or incorrect logic, as both sides of the race are working towards the same goal of emptying the LRU cache.

gemini-3-flash-preview

							input: 4293
							output: 40
							thoughts: 134
						

**Analyzing a Data Race in the Buffer Cache**

Okay, so the report flagged a pretty nasty data race, specifically between `__invalidate_bh_lrus` (triggered through `invalidate_bh_lrus_cpu`) and `has_bh_in_lru`. The race is happening on that memory address: `0xffff888237c27760`. Interesting, and it's an 8-byte access, so we're talking about a pointer, most likely. The value shifted from `0xffff88810645fc30` to `0x0000000000000000`, a clear indication a pointer is being nulled out.

Time to dive into `fs/buffer.c` and see what the heck is going on with that buffer cache implementation. I need to understand the logic flow, what data structures are involved, and where these two functions are being called. My immediate concern is that a buffer head is being invalidated while another thread is still attempting to use it. Looks like this could result in serious data corruption or even a kernel panic, so I need to track down the root cause of this race as quickly as possible.

map[Dir:fs/]

map[Files:[aio.c anon_inodes.c attr.c backing-file.c bad_inode.c binfmt_elf.c binfmt_elf_fdpic.c binfmt_flat.c binfmt_misc.c binfmt_script.c bpf_fs_kfuncs.c buffer.c char_dev.c compat_binfmt_elf.c coredump.c d_path.c dax.c dcache.c direct-io.c drop_caches.c eventfd.c eventpoll.c exec.c fcntl.c fhandle.c file.c file_attr.c file_table.c filesystems.c fs-writeback.c fs_context.c fs_dirent.c fs_parser.c fs_pin.c fs_struct.c fserror.c fsopen.c init.c inode.c internal.h ioctl.c kernel_read_file.c libfs.c locks.c mbcache.c mnt_idmapping.c mount.h mpage.c namei.c namespace.c nsfs.c nullfs.c open.c pidfs.c pipe.c pnode.c pnode.h posix_acl.c proc_namespace.c read_write.c readdir.c remap_range.c select.c seq_file.c signalfd.c splice.c stack.c stat.c statfs.c super.c sync.c sysctls.c timerfd.c userfaultfd.c utimes.c xattr.c] Subdirs:[9p adfs affs afs autofs befs bfs btrfs cachefiles ceph coda configfs cramfs crypto debugfs devpts dlm ecryptfs efivarfs efs erofs exfat exportfs ext2 ext4 f2fs fat freevxfs fuse gfs2 hfs hfsplus hostfs hpfs hugetlbfs iomap isofs jbd2 jffs2 jfs kernfs lockd minix netfs nfs nfs_common nfsd nilfs2 nls notify ntfs3 ocfs2 omfs openpromfs orangefs overlayfs proc pstore qnx4 qnx6 quota ramfs resctrl romfs smb squashfs sysfs tests tracefs ubifs udf ufs unicode vboxsf verity xfs zonefs]]

map[File:fs/buffer.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
 *  linux/fs/buffer.c
 *
 *  Copyright (C) 1991, 1992, 2002  Linus Torvalds
 */

/*
 * Start bdflush() with kernel_thread not syscall - Paul Gortmaker, 12/95
 *
 * Removed a lot of unnecessary code and simplified things now that
 * the buffer cache isn't our primary cache - Andrew Tridgell 12/96
 *
 * Speed up hash, lru, and free list operations.  Use gfp() for allocating
 * hash table, use SLAB cache for buffer heads. SMP threading.  -DaveM
 *
 * Added 32k buffer block sizes - these are required older ARM systems. - RMK
 *
 * async buffer flushing, 1999 Andrea Arcangeli <andrea@suse.de>
 */

#include <linux/kernel.h>
#include <linux/sched/signal.h>
#include <linux/syscalls.h>
#include <linux/fs.h>
#include <linux/iomap.h>
#include <linux/mm.h>
#include <linux/percpu.h>
#include <linux/slab.h>
#include <linux/capability.h>
#include <linux/blkdev.h>
#include <linux/blk-crypto.h>
#include <linux/file.h>
#include <linux/quotaops.h>
#include <linux/highmem.h>
#include <linux/export.h>
#include <linux/backing-dev.h>
#include <linux/writeback.h>
#include <linux/hash.h>
#include <linux/suspend.h>
#include <linux/buffer_head.h>
#include <linux/task_io_accounting_ops.h>
#include <linux/bio.h>
#include <linux/cpu.h>
#include <linux/bitops.h>
#include <linux/mpage.h>
#include <linux/bit_spinlock.h>
#include <linux/pagevec.h>
#include <linux/sched/mm.h>
#include <trace/events/block.h>
#include <linux/fscrypt.h>
#include <linux/fsverity.h>
#include <linux/sched/isolation.h>

#include "internal.h"

static int fsync_buffers_list(spinlock_t *lock, struct list_head *list);
static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh,
			  enum rw_hint hint, struct writeback_control *wbc);

#define BH_ENTRY(list) list_entry((list), struct buffer_head, b_assoc_buffers)

inline void touch_buffer(struct buffer_head *bh)
{
	trace_block_touch_buffer(bh);
	folio_mark_accessed(bh->b_folio);
}
EXPORT_SYMBOL(touch_buffer);

void __lock_buffer(struct buffer_head *bh)
{
	wait_on_bit_lock_io(&bh->b_state, BH_Lock, TASK_UNINTERRUPTIBLE);
}
EXPORT_SYMBOL(__lock_buffer);

void unlock_buffer(struct buffer_head *bh)
{
	clear_bit_unlock(BH_Lock, &bh->b_state);
	smp_mb__after_atomic();
	wake_up_bit(&bh->b_state, BH_Lock);
}
EXPORT_SYMBOL(unlock_buffer);

/*
 * Returns if the folio has dirty or writeback buffers. If all the buffers
 * are unlocked and clean then the folio_test_dirty information is stale. If
 * any of the buffers are locked, it is assumed they are locked for IO.
 */
void buffer_check_dirty_writeback(struct folio *folio,
				     bool *dirty, bool *writeback)
{
	struct buffer_head *head, *bh;
	*dirty = false;
	*writeback = false;

	BUG_ON(!folio_test_locked(folio));

	head = folio_buffers(folio);
	if (!head)
		return;

	if (folio_test_writeback(folio))
		*writeback = true;

	bh = head;
	do {
		if (buffer_locked(bh))
			*writeback = true;

		if (buffer_dirty(bh))
			*dirty = true;

		bh = bh->b_this_page;
	} while (bh != head);
}

/*
 * Block until a buffer comes unlocked.  This doesn't stop it
 * from becoming locked again - you have to lock it yourself
 * if you want to preserve its state.
 */
void __wait_on_buffer(struct buffer_head * bh)
{
	wait_on_bit_io(&bh->b_state, BH_Lock, TASK_UNINTERRUPTIBLE);
}
EXPORT_SYMBOL(__wait_on_buffer);

static void buffer_io_error(struct buffer_head *bh, char *msg)
{
	if (!test_bit(BH_Quiet, &bh->b_state))
		printk_ratelimited(KERN_ERR
			"Buffer I/O error on dev %pg, logical block %llu%s\n",
			bh->b_bdev, (unsigned long long)bh->b_blocknr, msg);
}

/*
 * End-of-IO handler helper function which does not touch the bh after
 * unlocking it.
 * Note: unlock_buffer() sort-of does touch the bh after unlocking it, but
 * a race there is benign: unlock_buffer() only use the bh's address for
 * hashing after unlocking the buffer, so it doesn't actually touch the bh
 * itself.
 */
static void __end_buffer_read_notouch(struct buffer_head *bh, int uptodate)
{
	if (uptodate) {
		set_buffer_uptodate(bh);
	} else {
		/* This happens, due to failed read-ahead attempts. */
		clear_buffer_uptodate(bh);
	}
	unlock_buffer(bh);
}

/*
 * Default synchronous end-of-IO handler..  Just mark it up-to-date and
 * unlock the buffer.
 */
void end_buffer_read_sync(struct buffer_head *bh, int uptodate)
{
	put_bh(bh);
	__end_buffer_read_notouch(bh, uptodate);
}
EXPORT_SYMBOL(end_buffer_read_sync);

void end_buffer_write_sync(struct buffer_head *bh, int uptodate)
{
	if (uptodate) {
		set_buffer_uptodate(bh);
	} else {
		buffer_io_error(bh, ", lost sync page write");
		mark_buffer_write_io_error(bh);
		clear_buffer_uptodate(bh);
	}
	unlock_buffer(bh);
	put_bh(bh);
}
EXPORT_SYMBOL(end_buffer_write_sync);

static struct buffer_head *
__find_get_block_slow(struct block_device *bdev, sector_t block, bool atomic)
{
	struct address_space *bd_mapping = bdev->bd_mapping;
	const int blkbits = bd_mapping->host->i_blkbits;
	struct buffer_head *ret = NULL;
	pgoff_t index;
	struct buffer_head *bh;
	struct buffer_head *head;
	struct folio *folio;
	int all_mapped = 1;
	static DEFINE_RATELIMIT_STATE(last_warned, HZ, 1);

	index = ((loff_t)block << blkbits) / PAGE_SIZE;
	folio = __filemap_get_folio(bd_mapping, index, FGP_ACCESSED, 0);
	if (IS_ERR(folio))
		goto out;

	/*
	 * Folio lock protects the buffers. Callers that cannot block
	 * will fallback to serializing vs try_to_free_buffers() via
	 * the i_private_lock.
	 */
	if (atomic)
		spin_lock(&bd_mapping->i_private_lock);
	else
		folio_lock(folio);

	head = folio_buffers(folio);
	if (!head)
		goto out_unlock;
	/*
	 * Upon a noref migration, the folio lock serializes here;
	 * otherwise bail.
	 */
	if (test_bit_acquire(BH_Migrate, &head->b_state)) {
		WARN_ON(!atomic);
		goto out_unlock;
	}

	bh = head;
	do {
		if (!buffer_mapped(bh))
			all_mapped = 0;
		else if (bh->b_blocknr == block) {
			ret = bh;
			get_bh(bh);
			goto out_unlock;
		}
		bh = bh->b_this_page;
	} while (bh != head);

	/* we might be here because some of the buffers on this page are
	 * not mapped.  This is due to various races between
	 * file io on the block device and getblk.  It gets dealt with
	 * elsewhere, don't buffer_error if we had some unmapped buffers
	 */
	ratelimit_set_flags(&last_warned, RATELIMIT_MSG_ON_RELEASE);
	if (all_mapped && __ratelimit(&last_warned)) {
		printk("__find_get_block_slow() failed. block=%llu, "
		       "b_blocknr=%llu, b_state=0x%08lx, b_size=%zu, "
		       "device %pg blocksize: %d\n",
		       (unsigned long long)block,
		       (unsigned long long)bh->b_blocknr,
		       bh->b_state, bh->b_size, bdev,
		       1 << blkbits);
	}
out_unlock:
	if (atomic)
		spin_unlock(&bd_mapping->i_private_lock);
	else
		folio_unlock(folio);
	folio_put(folio);
out:
	return ret;
}

static void end_buffer_async_read(struct buffer_head *bh, int uptodate)
{
	unsigned long flags;
	struct buffer_head *first;
	struct buffer_head *tmp;
	struct folio *folio;
	int folio_uptodate = 1;

	BUG_ON(!buffer_async_read(bh));

	folio = bh->b_folio;
	if (uptodate) {
		set_buffer_uptodate(bh);
	} else {
		clear_buffer_uptodate(bh);
		buffer_io_error(bh, ", async page read");
	}

	/*
	 * Be _very_ careful from here on. Bad things can happen if
	 * two buffer heads end IO at almost the same time and both
	 * decide that the page is now completely done.
	 */
	first = folio_buffers(folio);
	spin_lock_irqsave(&first->b_uptodate_lock, flags);
	clear_buffer_async_read(bh);
	unlock_buffer(bh);
	tmp = bh;
	do {
		if (!buffer_uptodate(tmp))
			folio_uptodate = 0;
		if (buffer_async_read(tmp)) {
			BUG_ON(!buffer_locked(tmp));
			goto still_busy;
		}
		tmp = tmp->b_this_page;
	} while (tmp != bh);
	spin_unlock_irqrestore(&first->b_uptodate_lock, flags);

	folio_end_read(folio, folio_uptodate);
	return;

still_busy:
	spin_unlock_irqrestore(&first->b_uptodate_lock, flags);
}

struct postprocess_bh_ctx {
	struct work_struct work;
	struct buffer_head *bh;
	struct fsverity_info *vi;
};

static void verify_bh(struct work_struct *work)
{
	struct postprocess_bh_ctx *ctx =
		container_of(work, struct postprocess_bh_ctx, work);
	struct buffer_head *bh = ctx->bh;
	bool valid;

	valid = fsverity_verify_blocks(ctx->vi, bh->b_folio, bh->b_size,
				       bh_offset(bh));
	end_buffer_async_read(bh, valid);
	kfree(ctx);
}

static void decrypt_bh(struct work_struct *work)
{
	struct postprocess_bh_ctx *ctx =
		container_of(work, struct postprocess_bh_ctx, work);
	struct buffer_head *bh = ctx->bh;
	int err;

	err = fscrypt_decrypt_pagecache_blocks(bh->b_folio, bh->b_size,
					       bh_offset(bh));
	if (err == 0 && ctx->vi) {
		/*
		 * We use different work queues for decryption and for verity
		 * because verity may require reading metadata pages that need
		 * decryption, and we shouldn't recurse to the same workqueue.
		 */
		INIT_WORK(&ctx->work, verify_bh);
		fsverity_enqueue_verify_work(&ctx->work);
		return;
	}
	end_buffer_async_read(bh, err == 0);
	kfree(ctx);
}

/*
 * I/O completion handler for block_read_full_folio() - pages
 * which come unlocked at the end of I/O.
 */
static void end_buffer_async_read_io(struct buffer_head *bh, int uptodate)
{
	struct inode *inode = bh->b_folio->mapping->host;
	bool decrypt = fscrypt_inode_uses_fs_layer_crypto(inode);
	struct fsverity_info *vi = NULL;

	/* needed by ext4 */
	if (bh->b_folio->index < DIV_ROUND_UP(inode->i_size, PAGE_SIZE))
		vi = fsverity_get_info(inode);

	/* Decrypt (with fscrypt) and/or verify (with fsverity) if needed. */
	if (uptodate && (decrypt || vi)) {
		struct postprocess_bh_ctx *ctx = kmalloc_obj(*ctx, GFP_ATOMIC);

		if (ctx) {
			ctx->bh = bh;
			ctx->vi = vi;
			if (decrypt) {
				INIT_WORK(&ctx->work, decrypt_bh);
				fscrypt_enqueue_decrypt_work(&ctx->work);
			} else {
				INIT_WORK(&ctx->work, verify_bh);
				fsverity_enqueue_verify_work(&ctx->work);
			}
			return;
		}
		uptodate = 0;
	}
	end_buffer_async_read(bh, uptodate);
}

/*
 * Completion handler for block_write_full_folio() - folios which are unlocked
 * during I/O, and which have the writeback flag cleared upon I/O completion.
 */
static void end_buffer_async_write(struct buffer_head *bh, int uptodate)
{
	unsigned long flags;
	struct buffer_head *first;
	struct buffer_head *tmp;
	struct folio *folio;

	BUG_ON(!buffer_async_write(bh));

	folio = bh->b_folio;
	if (uptodate) {
		set_buffer_uptodate(bh);
	} else {
		buffer_io_error(bh, ", lost async page write");
		mark_buffer_write_io_error(bh);
		clear_buffer_uptodate(bh);
	}

	first = folio_buffers(folio);
	spin_lock_irqsave(&first->b_uptodate_lock, flags);

	clear_buffer_async_write(bh);
	unlock_buffer(bh);
	tmp = bh->b_this_page;
	while (tmp != bh) {
		if (buffer_async_write(tmp)) {
			BUG_ON(!buffer_locked(tmp));
			goto still_busy;
		}
		tmp = tmp->b_this_page;
	}
	spin_unlock_irqrestore(&first->b_uptodate_lock, flags);
	folio_end_writeback(folio);
	return;

still_busy:
	spin_unlock_irqrestore(&first->b_uptodate_lock, flags);
}

/*
 * If a page's buffers are under async readin (end_buffer_async_read
 * completion) then there is a possibility that another thread of
 * control could lock one of the buffers after it has completed
 * but while some of the other buffers have not completed.  This
 * locked buffer would confuse end_buffer_async_read() into not unlocking
 * the page.  So the absence of BH_Async_Read tells end_buffer_async_read()
 * that this buffer is not under async I/O.
 *
 * The page comes unlocked when it has no locked buffer_async buffers
 * left.
 *
 * PageLocked prevents anyone starting new async I/O reads any of
 * the buffers.
 *
 * PageWriteback is used to prevent simultaneous writeout of the same
 * page.
 *
 * PageLocked prevents anyone from starting writeback of a page which is
 * under read I/O (PageWriteback is only ever set against a locked page).
 */
static void mark_buffer_async_read(struct buffer_head *bh)
{
	bh->b_end_io = end_buffer_async_read_io;
	set_buffer_async_read(bh);
}

static void mark_buffer_async_write_endio(struct buffer_head *bh,
					  bh_end_io_t *handler)
{
	bh->b_end_io = handler;
	set_buffer_async_write(bh);
}

void mark_buffer_async_write(struct buffer_head *bh)
{
	mark_buffer_async_write_endio(bh, end_buffer_async_write);
}
EXPORT_SYMBOL(mark_buffer_async_write);


/*
 * fs/buffer.c contains helper functions for buffer-backed address space's
 * fsync functions.  A common requirement for buffer-based filesystems is
 * that certain data from the backing blockdev needs to be written out for
 * a successful fsync().  For example, ext2 indirect blocks need to be
 * written back and waited upon before fsync() returns.
 *
 * The functions mark_buffer_dirty_inode(), fsync_inode_buffers(),
 * inode_has_buffers() and invalidate_inode_buffers() are provided for the
 * management of a list of dependent buffers at ->i_mapping->i_private_list.
 *
 * Locking is a little subtle: try_to_free_buffers() will remove buffers
 * from their controlling inode's queue when they are being freed.  But
 * try_to_free_buffers() will be operating against the *blockdev* mapping
 * at the time, not against the S_ISREG file which depends on those buffers.
 * So the locking for i_private_list is via the i_private_lock in the address_space
 * which backs the buffers.  Which is different from the address_space 
 * against which the buffers are listed.  So for a particular address_space,
 * mapping->i_private_lock does *not* protect mapping->i_private_list!  In fact,
 * mapping->i_private_list will always be protected by the backing blockdev's
 * ->i_private_lock.
 *
 * Which introduces a requirement: all buffers on an address_space's
 * ->i_private_list must be from the same address_space: the blockdev's.
 *
 * address_spaces which do not place buffers at ->i_private_list via these
 * utility functions are free to use i_private_lock and i_private_list for
 * whatever they want.  The only requirement is that list_empty(i_private_list)
 * be true at clear_inode() time.
 *
 * FIXME: clear_inode should not call invalidate_inode_buffers().  The
 * filesystems should do that.  invalidate_inode_buffers() should just go
 * BUG_ON(!list_empty).
 *
 * FIXME: mark_buffer_dirty_inode() is a data-plane operation.  It should
 * take an address_space, not an inode.  And it should be called
 * mark_buffer_dirty_fsync() to clearly define why those buffers are being
 * queued up.
 *
 * FIXME: mark_buffer_dirty_inode() doesn't need to add the buffer to the
 * list if it is already on a list.  Because if the buffer is on a list,
 * it *must* already be on the right one.  If not, the filesystem is being
 * silly.  This will save a ton of locking.  But first we have to ensure
 * that buffers are taken *off* the old inode's list when they are freed
 * (presumably in truncate).  That requires careful auditing of all
 * filesystems (do it inside bforget()).  It could also be done by bringing
 * b_inode back.
 */

/*
 * The buffer's backing address_space's i_private_lock must be held
 */
static void __remove_assoc_queue(struct buffer_head *bh)
{
	list_del_init(&bh->b_assoc_buffers);
	WARN_ON(!bh->b_assoc_map);
	bh->b_assoc_map = NULL;
}

int inode_has_buffers(struct inode *inode)
{
	return !list_empty(&inode->i_data.i_private_list);
}

/*
 * osync is designed to support O_SYNC io.  It waits synchronously for
 * all already-submitted IO to complete, but does not queue any new
 * writes to the disk.
 *
 * To do O_SYNC writes, just queue the buffer writes with write_dirty_buffer
 * as you dirty the buffers, and then use osync_inode_buffers to wait for
 * completion.  Any other dirty buffers which are not yet queued for
 * write will not be flushed to disk by the osync.
 */
static int osync_buffers_list(spinlock_t *lock, struct list_head *list)
{
	struct buffer_head *bh;
	struct list_head *p;
	int err = 0;

	spin_lock(lock);
repeat:
	list_for_each_prev(p, list) {
		bh = BH_ENTRY(p);
		if (buffer_locked(bh)) {
			get_bh(bh);
			spin_unlock(lock);
			wait_on_buffer(bh);
			if (!buffer_uptodate(bh))
				err = -EIO;
			brelse(bh);
			spin_lock(lock);
			goto repeat;
		}
	}
	spin_unlock(lock);
	return err;
}

/**
 * sync_mapping_buffers - write out & wait upon a mapping's "associated" buffers
 * @mapping: the mapping which wants those buffers written
 *
 * Starts I/O against the buffers at mapping->i_private_list, and waits upon
 * that I/O.
 *
 * Basically, this is a convenience function for fsync().
 * @mapping is a file or directory which needs those buffers to be written for
 * a successful fsync().
 */
int sync_mapping_buffers(struct address_space *mapping)
{
	struct address_space *buffer_mapping = mapping->i_private_data;

	if (buffer_mapping == NULL || list_empty(&mapping->i_private_list))
		return 0;

	return fsync_buffers_list(&buffer_mapping->i_private_lock,
					&mapping->i_private_list);
}
EXPORT_SYMBOL(sync_mapping_buffers);

/**
 * generic_buffers_fsync_noflush - generic buffer fsync implementation
 * for simple filesystems with no inode lock
 *
 * @file:	file to synchronize
 * @start:	start offset in bytes
 * @end:	end offset in bytes (inclusive)
 * @datasync:	only synchronize essential metadata if true
 *
 * This is a generic implementation of the fsync method for simple
 * filesystems which track all non-inode metadata in the buffers list
 * hanging off the address_space structure.
 */
int generic_buffers_fsync_noflush(struct file *file, loff_t start, loff_t end,
				  bool datasync)
{
	struct inode *inode = file->f_mapping->host;
	int err;
	int ret;

	err = file_write_and_wait_range(file, start, end);
	if (err)
		return err;

	ret = sync_mapping_buffers(inode->i_mapping);
	if (!(inode_state_read_once(inode) & I_DIRTY_ALL))
		goto out;
	if (datasync && !(inode_state_read_once(inode) & I_DIRTY_DATASYNC))
		goto out;

	err = sync_inode_metadata(inode, 1);
	if (ret == 0)
		ret = err;

out:
	/* check and advance again to catch errors after syncing out buffers */
	err = file_check_and_advance_wb_err(file);
	if (ret == 0)
		ret = err;
	return ret;
}
EXPORT_SYMBOL(generic_buffers_fsync_noflush);

/**
 * generic_buffers_fsync - generic buffer fsync implementation
 * for simple filesystems with no inode lock
 *
 * @file:	file to synchronize
 * @start:	start offset in bytes
 * @end:	end offset in bytes (inclusive)
 * @datasync:	only synchronize essential metadata if true
 *
 * This is a generic implementation of the fsync method for simple
 * filesystems which track all non-inode metadata in the buffers list
 * hanging off the address_space structure. This also makes sure that
 * a device cache flush operation is called at the end.
 */
int generic_buffers_fsync(struct file *file, loff_t start, loff_t end,
			  bool datasync)
{
	struct inode *inode = file->f_mapping->host;
	int ret;

	ret = generic_buffers_fsync_noflush(file, start, end, datasync);
	if (!ret)
		ret = blkdev_issue_flush(inode->i_sb->s_bdev);
	return ret;
}
EXPORT_SYMBOL(generic_buffers_fsync);

/*
 * Called when we've recently written block `bblock', and it is known that
 * `bblock' was for a buffer_boundary() buffer.  This means that the block at
 * `bblock + 1' is probably a dirty indirect block.  Hunt it down and, if it's
 * dirty, schedule it for IO.  So that indirects merge nicely with their data.
 */
void write_boundary_block(struct block_device *bdev,
			sector_t bblock, unsigned blocksize)
{
	struct buffer_head *bh;

	bh = __find_get_block_nonatomic(bdev, bblock + 1, blocksize);
	if (bh) {
		if (buffer_dirty(bh))
			write_dirty_buffer(bh, 0);
		put_bh(bh);
	}
}

void mark_buffer_dirty_inode(struct buffer_head *bh, struct inode *inode)
{
	struct address_space *mapping = inode->i_mapping;
	struct address_space *buffer_mapping = bh->b_folio->mapping;

	mark_buffer_dirty(bh);
	if (!mapping->i_private_data) {
		mapping->i_private_data = buffer_mapping;
	} else {
		BUG_ON(mapping->i_private_data != buffer_mapping);
	}
	if (!bh->b_assoc_map) {
		spin_lock(&buffer_mapping->i_private_lock);
		list_move_tail(&bh->b_assoc_buffers,
				&mapping->i_private_list);
		bh->b_assoc_map = mapping;
		spin_unlock(&buffer_mapping->i_private_lock);
	}
}
EXPORT_SYMBOL(mark_buffer_dirty_inode);

/**
 * block_dirty_folio - Mark a folio as dirty.
 * @mapping: The address space containing this folio.
 * @folio: The folio to mark dirty.
 *
 * Filesystems which use buffer_heads can use this function as their
 * ->dirty_folio implementation.  Some filesystems need to do a little
 * work before calling this function.  Filesystems which do not use
 * buffer_heads should call filemap_dirty_folio() instead.
 *
 * If the folio has buffers, the uptodate buffers are set dirty, to
 * preserve dirty-state coherency between the folio and the buffers.
 * Buffers added to a dirty folio are created dirty.
 *
 * The buffers are dirtied before the folio is dirtied.  There's a small
 * race window in which writeback may see the folio cleanness but not the
 * buffer dirtiness.  That's fine.  If this code were to set the folio
 * dirty before the buffers, writeback could clear the folio dirty flag,
 * see a bunch of clean buffers and we'd end up with dirty buffers/clean
 * folio on the dirty folio list.
 *
 * We use i_private_lock to lock against try_to_free_buffers() while
 * using the folio's buffer list.  This also prevents clean buffers
 * being added to the folio after it was set dirty.
 *
 * Context: May only be called from process context.  Does not sleep.
 * Caller must ensure that @folio cannot be truncated during this call,
 * typically by holding the folio lock or having a page in the folio
 * mapped and holding the page table lock.
 *
 * Return: True if the folio was dirtied; false if it was already dirtied.
 */
bool block_dirty_folio(struct address_space *mapping, struct folio *folio)
{
	struct buffer_head *head;
	bool newly_dirty;

	spin_lock(&mapping->i_private_lock);
	head = folio_buffers(folio);
	if (head) {
		struct buffer_head *bh = head;

		do {
			set_buffer_dirty(bh);
			bh = bh->b_this_page;
		} while (bh != head);
	}
	/*
	 * Lock out page's memcg migration to keep PageDirty
	 * synchronized with per-memcg dirty page counters.
	 */
	newly_dirty = !folio_test_set_dirty(folio);
	spin_unlock(&mapping->i_private_lock);

	if (newly_dirty)
		__folio_mark_dirty(folio, mapping, 1);

	if (newly_dirty)
		__mark_inode_dirty(mapping->host, I_DIRTY_PAGES);

	return newly_dirty;
}
EXPORT_SYMBOL(block_dirty_folio);

/*
 * Write out and wait upon a list of buffers.
 *
 * We have conflicting pressures: we want to make sure that all
 * initially dirty buffers get waited on, but that any subsequently
 * dirtied buffers don't.  After all, we don't want fsync to last
 * forever if somebody is actively writing to the file.
 *
 * Do this in two main stages: first we copy dirty buffers to a
 * temporary inode list, queueing the writes as we go.  Then we clean
 * up, waiting for those writes to complete.
 * 
 * During this second stage, any subsequent updates to the file may end
 * up refiling the buffer on the original inode's dirty list again, so
 * there is a chance we will end up with a buffer queued for write but
 * not yet completed on that list.  So, as a final cleanup we go through
 * the osync code to catch these locked, dirty buffers without requeuing
 * any newly dirty buffers for write.
 */
static int fsync_buffers_list(spinlock_t *lock, struct list_head *list)
{
	struct buffer_head *bh;
	struct address_space *mapping;
	int err = 0, err2;
	struct blk_plug plug;
	LIST_HEAD(tmp);

	blk_start_plug(&plug);

	spin_lock(lock);
	while (!list_empty(list)) {
		bh = BH_ENTRY(list->next);
		mapping = bh->b_assoc_map;
		__remove_assoc_queue(bh);
		/* Avoid race with mark_buffer_dirty_inode() which does
		 * a lockless check and we rely on seeing the dirty bit */
		smp_mb();
		if (buffer_dirty(bh) || buffer_locked(bh)) {
			list_add(&bh->b_assoc_buffers, &tmp);
			bh->b_assoc_map = mapping;
			if (buffer_dirty(bh)) {
				get_bh(bh);
				spin_unlock(lock);
				/*
				 * Ensure any pending I/O completes so that
				 * write_dirty_buffer() actually writes the
				 * current contents - it is a noop if I/O is
				 * still in flight on potentially older
				 * contents.
				 */
				write_dirty_buffer(bh, REQ_SYNC);

				/*
				 * Kick off IO for the previous mapping. Note
				 * that we will not run the very last mapping,
				 * wait_on_buffer() will do that for us
				 * through sync_buffer().
				 */
				brelse(bh);
				spin_lock(lock);
			}
		}
	}

	spin_unlock(lock);
	blk_finish_plug(&plug);
	spin_lock(lock);

	while (!list_empty(&tmp)) {
		bh = BH_ENTRY(tmp.prev);
		get_bh(bh);
		mapping = bh->b_assoc_map;
		__remove_assoc_queue(bh);
		/* Avoid race with mark_buffer_dirty_inode() which does
		 * a lockless check and we rely on seeing the dirty bit */
		smp_mb();
		if (buffer_dirty(bh)) {
			list_add(&bh->b_assoc_buffers,
				 &mapping->i_private_list);
			bh->b_assoc_map = mapping;
		}
		spin_unlock(lock);
		wait_on_buffer(bh);
		if (!buffer_uptodate(bh))
			err = -EIO;
		brelse(bh);
		spin_lock(lock);
	}
	
	spin_unlock(lock);
	err2 = osync_buffers_list(lock, list);
	if (err)
		return err;
	else
		return err2;
}

/*
 * Invalidate any and all dirty buffers on a given inode.  We are
 * probably unmounting the fs, but that doesn't mean we have already
 * done a sync().  Just drop the buffers from the inode list.
 *
 * NOTE: we take the inode's blockdev's mapping's i_private_lock.  Which
 * assumes that all the buffers are against the blockdev.
 */
void invalidate_inode_buffers(struct inode *inode)
{
	if (inode_has_buffers(inode)) {
		struct address_space *mapping = &inode->i_data;
		struct list_head *list = &mapping->i_private_list;
		struct address_space *buffer_mapping = mapping->i_private_data;

		spin_lock(&buffer_mapping->i_private_lock);
		while (!list_empty(list))
			__remove_assoc_queue(BH_ENTRY(list->next));
		spin_unlock(&buffer_mapping->i_private_lock);
	}
}
EXPORT_SYMBOL(invalidate_inode_buffers);

/*
 * Remove any clean buffers from the inode's buffer list.  This is called
 * when we're trying to free the inode itself.  Those buffers can pin it.
 *
 * Returns true if all buffers were removed.
 */
int remove_inode_buffers(struct inode *inode)
{
	int ret = 1;

	if (inode_has_buffers(inode)) {
		struct address_space *mapping = &inode->i_data;
		struct list_head *list = &mapping->i_private_list;
		struct address_space *buffer_mapping = mapping->i_private_data;

		spin_lock(&buffer_mapping->i_private_lock);
		while (!list_empty(list)) {
			struct buffer_head *bh = BH_ENTRY(list->next);
			if (buffer_dirty(bh)) {
				ret = 0;
				break;
			}
			__remove_assoc_queue(bh);
		}
		spin_unlock(&buffer_mapping->i_private_lock);
	}
	return ret;
}

/*
 * Create the appropriate buffers when given a folio for data area and
 * the size of each buffer.. Use the bh->b_this_page linked list to
 * follow the buffers created.  Return NULL if unable to create more
 * buffers.
 *
 * The retry flag is used to differentiate async IO (paging, swapping)
 * which may not fail from ordinary buffer allocations.
 */
struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,
					gfp_t gfp)
{
	struct buffer_head *bh, *head;
	long offset;
	struct mem_cgroup *memcg, *old_memcg;

	/* The folio lock pins the memcg */
	memcg = folio_memcg(folio);
	old_memcg = set_active_memcg(memcg);

	head = NULL;
	offset = folio_size(folio);
	while ((offset -= size) >= 0) {
		bh = alloc_buffer_head(gfp);
		if (!bh)
			goto no_grow;

		bh->b_this_page = head;
		bh->b_blocknr = -1;
		head = bh;

		bh->b_size = size;

		/* Link the buffer to its folio */
		folio_set_bh(bh, folio, offset);
	}
out:
	set_active_memcg(old_memcg);
	return head;
/*
 * In case anything failed, we just free everything we got.
 */
no_grow:
	if (head) {
		do {
			bh = head;
			head = head->b_this_page;
			free_buffer_head(bh);
		} while (head);
	}

	goto out;
}
EXPORT_SYMBOL_GPL(folio_alloc_buffers);

struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size)
{
	gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT;

	return folio_alloc_buffers(page_folio(page), size, gfp);
}
EXPORT_SYMBOL_GPL(alloc_page_buffers);

static inline void link_dev_buffers(struct folio *folio,
		struct buffer_head *head)
{
	struct buffer_head *bh, *tail;

	bh = head;
	do {
		tail = bh;
		bh = bh->b_this_page;
	} while (bh);
	tail->b_this_page = head;
	folio_attach_private(folio, head);
}

static sector_t blkdev_max_block(struct block_device *bdev, unsigned int size)
{
	sector_t retval = ~((sector_t)0);
	loff_t sz = bdev_nr_bytes(bdev);

	if (sz) {
		unsigned int sizebits = blksize_bits(size);
		retval = (sz >> sizebits);
	}
	return retval;
}

/*
 * Initialise the state of a blockdev folio's buffers.
 */ 
static sector_t folio_init_buffers(struct folio *folio,
		struct block_device *bdev, unsigned size)
{
	struct buffer_head *head = folio_buffers(folio);
	struct buffer_head *bh = head;
	bool uptodate = folio_test_uptodate(folio);
	sector_t block = div_u64(folio_pos(folio), size);
	sector_t end_block = blkdev_max_block(bdev, size);

	do {
		if (!buffer_mapped(bh)) {
			bh->b_end_io = NULL;
			bh->b_private = NULL;
			bh->b_bdev = bdev;
			bh->b_blocknr = block;
			if (uptodate)
				set_buffer_uptodate(bh);
			if (block < end_block)
				set_buffer_mapped(bh);
		}
		block++;
		bh = bh->b_this_page;
	} while (bh != head);

	/*
	 * Caller needs to validate requested block against end of device.
	 */
	return end_block;
}

/*
 * Create the page-cache folio that contains the requested block.
 *
 * This is used purely for blockdev mappings.
 *
 * Returns false if we have a failure which cannot be cured by retrying
 * without sleeping.  Returns true if we succeeded, or the caller should retry.
 */
static bool grow_dev_folio(struct block_device *bdev, sector_t block,
		pgoff_t index, unsigned size, gfp_t gfp)
{
	struct address_space *mapping = bdev->bd_mapping;
	struct folio *folio;
	struct buffer_head *bh;
	sector_t end_block = 0;

	folio = __filemap_get_folio(mapping, index,
			FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp);
	if (IS_ERR(folio))
		return false;

	bh = folio_buffers(folio);
	if (bh) {
		if (bh->b_size == size) {
			end_block = folio_init_buffers(folio, bdev, size);
			goto unlock;
		}

		/*
		 * Retrying may succeed; for example the folio may finish
		 * writeback, or buffers may be cleaned.  This should not
		 * happen very often; maybe we have old buffers attached to
		 * this blockdev's page cache and we're trying to change
		 * the block size?
		 */
		if (!try_to_free_buffers(folio)) {
			end_block = ~0ULL;
			goto unlock;
		}
	}

	bh = folio_alloc_buffers(folio, size, gfp | __GFP_ACCOUNT);
	if (!bh)
		goto unlock;

	/*
	 * Link the folio to the buffers and initialise them.  Take the
	 * lock to be atomic wrt __find_get_block(), which does not
	 * run under the folio lock.
	 */
	spin_lock(&mapping->i_private_lock);
	link_dev_buffers(folio, bh);
	end_block = folio_init_buffers(folio, bdev, size);
	spin_unlock(&mapping->i_private_lock);
unlock:
	folio_unlock(folio);
	folio_put(folio);
	return block < end_block;
}

/*
 * Create buffers for the specified block device block's folio.  If
 * that folio was dirty, the buffers are set dirty also.  Returns false
 * if we've hit a permanent error.
 */
static bool grow_buffers(struct block_device *bdev, sector_t block,
		unsigned size, gfp_t gfp)
{
	loff_t pos;

	/*
	 * Check for a block which lies outside our maximum possible
	 * pagecache index.
	 */
	if (check_mul_overflow(block, (sector_t)size, &pos) || pos > MAX_LFS_FILESIZE) {
		printk(KERN_ERR "%s: requested out-of-range block %llu for device %pg\n",
			__func__, (unsigned long long)block,
			bdev);
		return false;
	}

	/* Create a folio with the proper size buffers */
	return grow_dev_folio(bdev, block, pos / PAGE_SIZE, size, gfp);
}

static struct buffer_head *
__getblk_slow(struct block_device *bdev, sector_t block,
	     unsigned size, gfp_t gfp)
{
	bool blocking = gfpflags_allow_blocking(gfp);

	if (WARN_ON_ONCE(!IS_ALIGNED(size, bdev_logical_block_size(bdev)))) {
		printk(KERN_ERR "getblk(): block size %d not aligned to logical block size %d\n",
		       size, bdev_logical_block_size(bdev));
		return NULL;
	}

	for (;;) {
		struct buffer_head *bh;

		if (!grow_buffers(bdev, block, size, gfp))
			return NULL;

		if (blocking)
			bh = __find_get_block_nonatomic(bdev, block, size);
		else
			bh = __find_get_block(bdev, block, size);
		if (bh)
			return bh;
	}
}

/*
 * The relationship between dirty buffers and dirty pages:
 *
 * Whenever a page has any dirty buffers, the page's dirty bit is set, and
 * the page is tagged dirty in the page cache.
 *
 * At all times, the dirtiness of the buffers represents the dirtiness of
 * subsections of the page.  If the page has buffers, the page dirty bit is
 * merely a hint about the true dirty state.
 *
 * When a page is set dirty in its entirety, all its buffers are marked dirty
 * (if the page has buffers).
 *
 * When a buffer is marked dirty, its page is dirtied, but the page's other
 * buffers are not.
 *
 * Also.  When blockdev buffers are explicitly read with bread(), they
 * individually become uptodate.  But their backing page remains not
 * uptodate - even if all of its buffers are uptodate.  A subsequent
 * block_read_full_folio() against that folio will discover all the uptodate
 * buffers, will set the folio uptodate and will perform no I/O.
 */

/**
 * mark_buffer_dirty - mark a buffer_head as needing writeout
 * @bh: the buffer_head to mark dirty
 *
 * mark_buffer_dirty() will set the dirty bit against the buffer, then set
 * its backing page dirty, then tag the page as dirty in the page cache
 * and then attach the address_space's inode to its superblock's dirty
 * inode list.
 *
 * mark_buffer_dirty() is atomic.  It takes bh->b_folio->mapping->i_private_lock,
 * i_pages lock and mapping->host->i_lock.
 */
void mark_buffer_dirty(struct buffer_head *bh)
{
	WARN_ON_ONCE(!buffer_uptodate(bh));

	trace_block_dirty_buffer(bh);

	/*
	 * Very *carefully* optimize the it-is-already-dirty case.
	 *
	 * Don't let the final "is it dirty" escape to before we
	 * perhaps modified the buffer.
	 */
	if (buffer_dirty(bh)) {
		smp_mb();
		if (buffer_dirty(bh))
			return;
	}

	if (!test_set_buffer_dirty(bh)) {
		struct folio *folio = bh->b_folio;
		struct address_space *mapping = NULL;

		if (!folio_test_set_dirty(folio)) {
			mapping = folio->mapping;
			if (mapping)
				__folio_mark_dirty(folio, mapping, 0);
		}
		if (mapping)
			__mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
	}
}
EXPORT_SYMBOL(mark_buffer_dirty);

void mark_buffer_write_io_error(struct buffer_head *bh)
{
	set_buffer_write_io_error(bh);
	/* FIXME: do we need to set this in both places? */
	if (bh->b_folio && bh->b_folio->mapping)
		mapping_set_error(bh->b_folio->mapping, -EIO);
	if (bh->b_assoc_map)
		mapping_set_error(bh->b_assoc_map, -EIO);
}
EXPORT_SYMBOL(mark_buffer_write_io_error);

/**
 * __brelse - Release a buffer.
 * @bh: The buffer to release.
 *
 * This variant of brelse() can be called if @bh is guaranteed to not be NULL.
 */
void __brelse(struct buffer_head *bh)
{
	if (atomic_read(&bh->b_count)) {
		put_bh(bh);
		return;
	}
	WARN(1, KERN_ERR "VFS: brelse: Trying to free free buffer\n");
}
EXPORT_SYMBOL(__brelse);

/**
 * __bforget - Discard any dirty data in a buffer.
 * @bh: The buffer to forget.
 *
 * This variant of bforget() can be called if @bh is guaranteed to not
 * be NULL.
 */
void __bforget(struct buffer_head *bh)
{
	clear_buffer_dirty(bh);
	if (bh->b_assoc_map) {
		struct address_space *buffer_mapping = bh->b_folio->mapping;

		spin_lock(&buffer_mapping->i_private_lock);
		list_del_init(&bh->b_assoc_buffers);
		bh->b_assoc_map = NULL;
		spin_unlock(&buffer_mapping->i_private_lock);
	}
	__brelse(bh);
}
EXPORT_SYMBOL(__bforget);

static struct buffer_head *__bread_slow(struct buffer_head *bh)
{
	lock_buffer(bh);
	if (buffer_uptodate(bh)) {
		unlock_buffer(bh);
		return bh;
	} else {
		get_bh(bh);
		bh->b_end_io = end_buffer_read_sync;
		submit_bh(REQ_OP_READ, bh);
		wait_on_buffer(bh);
		if (buffer_uptodate(bh))
			return bh;
	}
	brelse(bh);
	return NULL;
}

/*
 * Per-cpu buffer LRU implementation.  To reduce the cost of __find_get_block().
 * The bhs[] array is sorted - newest buffer is at bhs[0].  Buffers have their
 * refcount elevated by one when they're in an LRU.  A buffer can only appear
 * once in a particular CPU's LRU.  A single buffer can be present in multiple
 * CPU's LRUs at the same time.
 *
 * This is a transparent caching front-end to sb_bread(), sb_getblk() and
 * sb_find_get_block().
 *
 * The LRUs themselves only need locking against invalidate_bh_lrus.  We use
 * a local interrupt disable for that.
 */

#define BH_LRU_SIZE	16

struct bh_lru {
	struct buffer_head *bhs[BH_LRU_SIZE];
};

static DEFINE_PER_CPU(struct bh_lru, bh_lrus) = {{ NULL }};

#ifdef CONFIG_SMP
#define bh_lru_lock()	local_irq_disable()
#define bh_lru_unlock()	local_irq_enable()
#else
#define bh_lru_lock()	preempt_disable()
#define bh_lru_unlock()	preempt_enable()
#endif

static inline void check_irqs_on(void)
{
#ifdef irqs_disabled
	BUG_ON(irqs_disabled());
#endif
}

/*
 * Install a buffer_head into this cpu's LRU.  If not already in the LRU, it is
 * inserted at the front, and the buffer_head at the back if any is evicted.
 * Or, if already in the LRU it is moved to the front.
 */
static void bh_lru_install(struct buffer_head *bh)
{
	struct buffer_head *evictee = bh;
	struct bh_lru *b;
	int i;

	check_irqs_on();
	bh_lru_lock();

	/*
	 * the refcount of buffer_head in bh_lru prevents dropping the
	 * attached page(i.e., try_to_free_buffers) so it could cause
	 * failing page migration.
	 * Skip putting upcoming bh into bh_lru until migration is done.
	 */
	if (lru_cache_disabled() || cpu_is_isolated(smp_processor_id())) {
		bh_lru_unlock();
		return;
	}

	b = this_cpu_ptr(&bh_lrus);
	for (i = 0; i < BH_LRU_SIZE; i++) {
		swap(evictee, b->bhs[i]);
		if (evictee == bh) {
			bh_lru_unlock();
			return;
		}
	}

	get_bh(bh);
	bh_lru_unlock();
	brelse(evictee);
}

/*
 * Look up the bh in this cpu's LRU.  If it's there, move it to the head.
 */
static struct buffer_head *
lookup_bh_lru(struct block_device *bdev, sector_t block, unsigned size)
{
	struct buffer_head *ret = NULL;
	unsigned int i;

	check_irqs_on();
	bh_lru_lock();
	if (cpu_is_isolated(smp_processor_id())) {
		bh_lru_unlock();
		return NULL;
	}
	for (i = 0; i < BH_LRU_SIZE; i++) {
		struct buffer_head *bh = __this_cpu_read(bh_lrus.bhs[i]);

		if (bh && bh->b_blocknr == block && bh->b_bdev == bdev &&
		    bh->b_size == size) {
			if (i) {
				while (i) {
					__this_cpu_write(bh_lrus.bhs[i],
						__this_cpu_read(bh_lrus.bhs[i - 1]));
					i--;
				}
				__this_cpu_write(bh_lrus.bhs[0], bh);
			}
			get_bh(bh);
			ret = bh;
			break;
		}
	}
	bh_lru_unlock();
	return ret;
}

/*
 * Perform a pagecache lookup for the matching buffer.  If it's there, refresh
 * it in the LRU and mark it as accessed.  If it is not present then return
 * NULL. Atomic context callers may also return NULL if the buffer is being
 * migrated; similarly the page is not marked accessed either.
 */
static struct buffer_head *
find_get_block_common(struct block_device *bdev, sector_t block,
			unsigned size, bool atomic)
{
	struct buffer_head *bh = lookup_bh_lru(bdev, block, size);

	if (bh == NULL) {
		/* __find_get_block_slow will mark the page accessed */
		bh = __find_get_block_slow(bdev, block, atomic);
		if (bh)
			bh_lru_install(bh);
	} else
		touch_buffer(bh);

	return bh;
}

struct buffer_head *
__find_get_block(struct block_device *bdev, sector_t block, unsigned size)
{
	return find_get_block_common(bdev, block, size, true);
}
EXPORT_SYMBOL(__find_get_block);

/* same as __find_get_block() but allows sleeping contexts */
struct buffer_head *
__find_get_block_nonatomic(struct block_device *bdev, sector_t block,
			   unsigned size)
{
	return find_get_block_common(bdev, block, size, false);
}
EXPORT_SYMBOL(__find_get_block_nonatomic);

/**
 * bdev_getblk - Get a buffer_head in a block device's buffer cache.
 * @bdev: The block device.
 * @block: The block number.
 * @size: The size of buffer_heads for this @bdev.
 * @gfp: The memory allocation flags to use.
 *
 * The returned buffer head has its reference count incremented, but is
 * not locked.  The caller should call brelse() when it has finished
 * with the buffer.  The buffer may not be uptodate.  If needed, the
 * caller can bring it uptodate either by reading it or overwriting it.
 *
 * Return: The buffer head, or NULL if memory could not be allocated.
 */
struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block,
		unsigned size, gfp_t gfp)
{
	struct buffer_head *bh;

	if (gfpflags_allow_blocking(gfp))
		bh = __find_get_block_nonatomic(bdev, block, size);
	else
		bh = __find_get_block(bdev, block, size);

	might_alloc(gfp);
	if (bh)
		return bh;

	return __getblk_slow(bdev, block, size, gfp);
}
EXPORT_SYMBOL(bdev_getblk);

/*
 * Do async read-ahead on a buffer..
 */
void __breadahead(struct block_device *bdev, sector_t block, unsigned size)
{
	struct buffer_head *bh = bdev_getblk(bdev, block, size,
			GFP_NOWAIT | __GFP_MOVABLE);

	if (likely(bh)) {
		bh_readahead(bh, REQ_RAHEAD);
		brelse(bh);
	}
}
EXPORT_SYMBOL(__breadahead);

/**
 * __bread_gfp() - Read a block.
 * @bdev: The block device to read from.
 * @block: Block number in units of block size.
 * @size: The block size of this device in bytes.
 * @gfp: Not page allocation flags; see below.
 *
 * You are not expected to call this function.  You should use one of
 * sb_bread(), sb_bread_unmovable() or __bread().
 *
 * Read a specified block, and return the buffer head that refers to it.
 * If @gfp is 0, the memory will be allocated using the block device's
 * default GFP flags.  If @gfp is __GFP_MOVABLE, the memory may be
 * allocated from a movable area.  Do not pass in a complete set of
 * GFP flags.
 *
 * The returned buffer head has its refcount increased.  The caller should
 * call brelse() when it has finished with the buffer.
 *
 * Context: May sleep waiting for I/O.
 * Return: NULL if the block was unreadable.
 */
struct buffer_head *__bread_gfp(struct block_device *bdev, sector_t block,
		unsigned size, gfp_t gfp)
{
	struct buffer_head *bh;

	gfp |= mapping_gfp_constraint(bdev->bd_mapping, ~__GFP_FS);

	/*
	 * Prefer looping in the allocator rather than here, at least that
	 * code knows what it's doing.
	 */
	gfp |= __GFP_NOFAIL;

	bh = bdev_getblk(bdev, block, size, gfp);

	if (likely(bh) && !buffer_uptodate(bh))
		bh = __bread_slow(bh);
	return bh;
}
EXPORT_SYMBOL(__bread_gfp);

static void __invalidate_bh_lrus(struct bh_lru *b)
{
	int i;

	for (i = 0; i < BH_LRU_SIZE; i++) {
		brelse(b->bhs[i]);
		b->bhs[i] = NULL;
	}
}
/*
 * invalidate_bh_lrus() is called rarely - but not only at unmount.
 * This doesn't race because it runs in each cpu either in irq
 * or with preempt disabled.
 */
static void invalidate_bh_lru(void *arg)
{
	struct bh_lru *b = &get_cpu_var(bh_lrus);

	__invalidate_bh_lrus(b);
	put_cpu_var(bh_lrus);
}

bool has_bh_in_lru(int cpu, void *dummy)
{
	struct bh_lru *b = per_cpu_ptr(&bh_lrus, cpu);
	int i;
	
	for (i = 0; i < BH_LRU_SIZE; i++) {
		if (b->bhs[i])
			return true;
	}

	return false;
}

void invalidate_bh_lrus(void)
{
	on_each_cpu_cond(has_bh_in_lru, invalidate_bh_lru, NULL, 1);
}
EXPORT_SYMBOL_GPL(invalidate_bh_lrus);

/*
 * It's called from workqueue context so we need a bh_lru_lock to close
 * the race with preemption/irq.
 */
void invalidate_bh_lrus_cpu(void)
{
	struct bh_lru *b;

	bh_lru_lock();
	b = this_cpu_ptr(&bh_lrus);
	__invalidate_bh_lrus(b);
	bh_lru_unlock();
}

void folio_set_bh(struct buffer_head *bh, struct folio *folio,
		  unsigned long offset)
{
	bh->b_folio = folio;
	BUG_ON(offset >= folio_size(folio));
	if (folio_test_highmem(folio))
		/*
		 * This catches illegal uses and preserves the offset:
		 */
		bh->b_data = (char *)(0 + offset);
	else
		bh->b_data = folio_address(folio) + offset;
}
EXPORT_SYMBOL(folio_set_bh);

/*
 * Called when truncating a buffer on a page completely.
 */

/* Bits that are cleared during an invalidate */
#define BUFFER_FLAGS_DISCARD \
	(1 << BH_Mapped | 1 << BH_New | 1 << BH_Req | \
	 1 << BH_Delay | 1 << BH_Unwritten)

static void discard_buffer(struct buffer_head * bh)
{
	unsigned long b_state;

	lock_buffer(bh);
	clear_buffer_dirty(bh);
	bh->b_bdev = NULL;
	b_state = READ_ONCE(bh->b_state);
	do {
	} while (!try_cmpxchg_relaxed(&bh->b_state, &b_state,
				      b_state & ~BUFFER_FLAGS_DISCARD));
	unlock_buffer(bh);
}

/**
 * block_invalidate_folio - Invalidate part or all of a buffer-backed folio.
 * @folio: The folio which is affected.
 * @offset: start of the range to invalidate
 * @length: length of the range to invalidate
 *
 * block_invalidate_folio() is called when all or part of the folio has been
 * invalidated by a truncate operation.
 *
 * block_invalidate_folio() does not have to release all buffers, but it must
 * ensure that no dirty buffer is left outside @offset and that no I/O
 * is underway against any of the blocks which are outside the truncation
 * point.  Because the caller is about to free (and possibly reuse) those
 * blocks on-disk.
 */
void block_invalidate_folio(struct folio *folio, size_t offset, size_t length)
{
	struct buffer_head *head, *bh, *next;
	size_t curr_off = 0;
	size_t stop = length + offset;

	BUG_ON(!folio_test_locked(folio));

	/*
	 * Check for overflow
	 */
	BUG_ON(stop > folio_size(folio) || stop < length);

	head = folio_buffers(folio);
	if (!head)
		return;

	bh = head;
	do {
		size_t next_off = curr_off + bh->b_size;
		next = bh->b_this_page;

		/*
		 * Are we still fully in range ?
		 */
		if (next_off > stop)
			goto out;

		/*
		 * is this block fully invalidated?
		 */
		if (offset <= curr_off)
			discard_buffer(bh);
		curr_off = next_off;
		bh = next;
	} while (bh != head);

	/*
	 * We release buffers only if the entire folio is being invalidated.
	 * The get_block cached value has been unconditionally invalidated,
	 * so real IO is not possible anymore.
	 */
	if (length == folio_size(folio))
		filemap_release_folio(folio, 0);
out:
	folio_clear_mappedtodisk(folio);
}
EXPORT_SYMBOL(block_invalidate_folio);

/*
 * We attach and possibly dirty the buffers atomically wrt
 * block_dirty_folio() via i_private_lock.  try_to_free_buffers
 * is already excluded via the folio lock.
 */
struct buffer_head *create_empty_buffers(struct folio *folio,
		unsigned long blocksize, unsigned long b_state)
{
	struct buffer_head *bh, *head, *tail;
	gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT | __GFP_NOFAIL;

	head = folio_alloc_buffers(folio, blocksize, gfp);
	bh = head;
	do {
		bh->b_state |= b_state;
		tail = bh;
		bh = bh->b_this_page;
	} while (bh);
	tail->b_this_page = head;

	spin_lock(&folio->mapping->i_private_lock);
	if (folio_test_uptodate(folio) || folio_test_dirty(folio)) {
		bh = head;
		do {
			if (folio_test_dirty(folio))
				set_buffer_dirty(bh);
			if (folio_test_uptodate(folio))
				set_buffer_uptodate(bh);
			bh = bh->b_this_page;
		} while (bh != head);
	}
	folio_attach_private(folio, head);
	spin_unlock(&folio->mapping->i_private_lock);

	return head;
}
EXPORT_SYMBOL(create_empty_buffers);

/**
 * clean_bdev_aliases: clean a range of buffers in block device
 * @bdev: Block device to clean buffers in
 * @block: Start of a range of blocks to clean
 * @len: Number of blocks to clean
 *
 * We are taking a range of blocks for data and we don't want writeback of any
 * buffer-cache aliases starting from return from this function and until the
 * moment when something will explicitly mark the buffer dirty (hopefully that
 * will not happen until we will free that block ;-) We don't even need to mark
 * it not-uptodate - nobody can expect anything from a newly allocated buffer
 * anyway. We used to use unmap_buffer() for such invalidation, but that was
 * wrong. We definitely don't want to mark the alias unmapped, for example - it
 * would confuse anyone who might pick it with bread() afterwards...
 *
 * Also..  Note that bforget() doesn't lock the buffer.  So there can be
 * writeout I/O going on against recently-freed buffers.  We don't wait on that
 * I/O in bforget() - it's more efficient to wait on the I/O only if we really
 * need to.  That happens here.
 */
void clean_bdev_aliases(struct block_device *bdev, sector_t block, sector_t len)
{
	struct address_space *bd_mapping = bdev->bd_mapping;
	const int blkbits = bd_mapping->host->i_blkbits;
	struct folio_batch fbatch;
	pgoff_t index = ((loff_t)block << blkbits) / PAGE_SIZE;
	pgoff_t end;
	int i, count;
	struct buffer_head *bh;
	struct buffer_head *head;

	end = ((loff_t)(block + len - 1) << blkbits) / PAGE_SIZE;
	folio_batch_init(&fbatch);
	while (filemap_get_folios(bd_mapping, &index, end, &fbatch)) {
		count = folio_batch_count(&fbatch);
		for (i = 0; i < count; i++) {
			struct folio *folio = fbatch.folios[i];

			if (!folio_buffers(folio))
				continue;
			/*
			 * We use folio lock instead of bd_mapping->i_private_lock
			 * to pin buffers here since we can afford to sleep and
			 * it scales better than a global spinlock lock.
			 */
			folio_lock(folio);
			/* Recheck when the folio is locked which pins bhs */
			head = folio_buffers(folio);
			if (!head)
				goto unlock_page;
			bh = head;
			do {
				if (!buffer_mapped(bh) || (bh->b_blocknr < block))
					goto next;
				if (bh->b_blocknr >= block + len)
					break;
				clear_buffer_dirty(bh);
				wait_on_buffer(bh);
				clear_buffer_req(bh);
next:
				bh = bh->b_this_page;
			} while (bh != head);
unlock_page:
			folio_unlock(folio);
		}
		folio_batch_release(&fbatch);
		cond_resched();
		/* End of range already reached? */
		if (index > end || !index)
			break;
	}
}
EXPORT_SYMBOL(clean_bdev_aliases);

static struct buffer_head *folio_create_buffers(struct folio *folio,
						struct inode *inode,
						unsigned int b_state)
{
	struct buffer_head *bh;

	BUG_ON(!folio_test_locked(folio));

	bh = folio_buffers(folio);
	if (!bh)
		bh = create_empty_buffers(folio,
				1 << READ_ONCE(inode->i_blkbits), b_state);
	return bh;
}

/*
 * NOTE! All mapped/uptodate combinations are valid:
 *
 *	Mapped	Uptodate	Meaning
 *
 *	No	No		"unknown" - must do get_block()
 *	No	Yes		"hole" - zero-filled
 *	Yes	No		"allocated" - allocated on disk, not read in
 *	Yes	Yes		"valid" - allocated and up-to-date in memory.
 *
 * "Dirty" is valid only with the last case (mapped+uptodate).
 */

/*
 * While block_write_full_folio is writing back the dirty buffers under
 * the page lock, whoever dirtied the buffers may decide to clean them
 * again at any time.  We handle that by only looking at the buffer
 * state inside lock_buffer().
 *
 * If block_write_full_folio() is called for regular writeback
 * (wbc->sync_mode == WB_SYNC_NONE) then it will redirty a page which has a
 * locked buffer.   This only can happen if someone has written the buffer
 * directly, with submit_bh().  At the address_space level PageWriteback
 * prevents this contention from occurring.
 *
 * If block_write_full_folio() is called with wbc->sync_mode ==
 * WB_SYNC_ALL, the writes are posted using REQ_SYNC; this
 * causes the writes to be flagged as synchronous writes.
 */
int __block_write_full_folio(struct inode *inode, struct folio *folio,
			get_block_t *get_block, struct writeback_control *wbc)
{
	int err;
	sector_t block;
	sector_t last_block;
	struct buffer_head *bh, *head;
	size_t blocksize;
	int nr_underway = 0;
	blk_opf_t write_flags = wbc_to_write_flags(wbc);

	head = folio_create_buffers(folio, inode,
				    (1 << BH_Dirty) | (1 << BH_Uptodate));

	/*
	 * Be very careful.  We have no exclusion from block_dirty_folio
	 * here, and the (potentially unmapped) buffers may become dirty at
	 * any time.  If a buffer becomes dirty here after we've inspected it
	 * then we just miss that fact, and the folio stays dirty.
	 *
	 * Buffers outside i_size may be dirtied by block_dirty_folio;
	 * handle that here by just cleaning them.
	 */

	bh = head;
	blocksize = bh->b_size;

	block = div_u64(folio_pos(folio), blocksize);
	last_block = div_u64(i_size_read(inode) - 1, blocksize);

	/*
	 * Get all the dirty buffers mapped to disk addresses and
	 * handle any aliases from the underlying blockdev's mapping.
	 */
	do {
		if (block > last_block) {
			/*
			 * mapped buffers outside i_size will occur, because
			 * this folio can be outside i_size when there is a
			 * truncate in progress.
			 */
			/*
			 * The buffer was zeroed by block_write_full_folio()
			 */
			clear_buffer_dirty(bh);
			set_buffer_uptodate(bh);
		} else if ((!buffer_mapped(bh) || buffer_delay(bh)) &&
			   buffer_dirty(bh)) {
			WARN_ON(bh->b_size != blocksize);
			err = get_block(inode, block, bh, 1);
			if (err)
				goto recover;
			clear_buffer_delay(bh);
			if (buffer_new(bh)) {
				/* blockdev mappings never come here */
				clear_buffer_new(bh);
				clean_bdev_bh_alias(bh);
			}
		}
		bh = bh->b_this_page;
		block++;
	} while (bh != head);

	do {
		if (!buffer_mapped(bh))
			continue;
		/*
		 * If it's a fully non-blocking write attempt and we cannot
		 * lock the buffer then redirty the folio.  Note that this can
		 * potentially cause a busy-wait loop from writeback threads
		 * and kswapd activity, but those code paths have their own
		 * higher-level throttling.
		 */
		if (wbc->sync_mode != WB_SYNC_NONE) {
			lock_buffer(bh);
		} else if (!trylock_buffer(bh)) {
			folio_redirty_for_writepage(wbc, folio);
			continue;
		}
		if (test_clear_buffer_dirty(bh)) {
			mark_buffer_async_write_endio(bh,
				end_buffer_async_write);
		} else {
			unlock_buffer(bh);
		}
	} while ((bh = bh->b_this_page) != head);

	/*
	 * The folio and its buffers are protected by the writeback flag,
	 * so we can drop the bh refcounts early.
	 */
	BUG_ON(folio_test_writeback(folio));
	folio_start_writeback(folio);

	do {
		struct buffer_head *next = bh->b_this_page;
		if (buffer_async_write(bh)) {
			submit_bh_wbc(REQ_OP_WRITE | write_flags, bh,
				      inode->i_write_hint, wbc);
			nr_underway++;
		}
		bh = next;
	} while (bh != head);
	folio_unlock(folio);

	err = 0;
done:
	if (nr_underway == 0) {
		/*
		 * The folio was marked dirty, but the buffers were
		 * clean.  Someone wrote them back by hand with
		 * write_dirty_buffer/submit_bh.  A rare case.
		 */
		folio_end_writeback(folio);

		/*
		 * The folio and buffer_heads can be released at any time from
		 * here on.
		 */
	}
	return err;

recover:
	/*
	 * ENOSPC, or some other error.  We may already have added some
	 * blocks to the file, so we need to write these out to avoid
	 * exposing stale data.
	 * The folio is currently locked and not marked for writeback
	 */
	bh = head;
	/* Recovery: lock and submit the mapped buffers */
	do {
		if (buffer_mapped(bh) && buffer_dirty(bh) &&
		    !buffer_delay(bh)) {
			lock_buffer(bh);
			mark_buffer_async_write_endio(bh,
				end_buffer_async_write);
		} else {
			/*
			 * The buffer may have been set dirty during
			 * attachment to a dirty folio.
			 */
			clear_buffer_dirty(bh);
		}
	} while ((bh = bh->b_this_page) != head);
	BUG_ON(folio_test_writeback(folio));
	mapping_set_error(folio->mapping, err);
	folio_start_writeback(folio);
	do {
		struct buffer_head *next = bh->b_this_page;
		if (buffer_async_write(bh)) {
			clear_buffer_dirty(bh);
			submit_bh_wbc(REQ_OP_WRITE | write_flags, bh,
				      inode->i_write_hint, wbc);
			nr_underway++;
		}
		bh = next;
	} while (bh != head);
	folio_unlock(folio);
	goto done;
}
EXPORT_SYMBOL(__block_write_full_folio);

/*
 * If a folio has any new buffers, zero them out here, and mark them uptodate
 * and dirty so they'll be written out (in order to prevent uninitialised
 * block data from leaking). And clear the new bit.
 */
void folio_zero_new_buffers(struct folio *folio, size_t from, size_t to)
{
	size_t block_start, block_end;
	struct buffer_head *head, *bh;

	BUG_ON(!folio_test_locked(folio));
	head = folio_buffers(folio);
	if (!head)
		return;

	bh = head;
	block_start = 0;
	do {
		block_end = block_start + bh->b_size;

		if (buffer_new(bh)) {
			if (block_end > from && block_start < to) {
				if (!folio_test_uptodate(folio)) {
					size_t start, xend;

					start = max(from, block_start);
					xend = min(to, block_end);

					folio_zero_segment(folio, start, xend);
					set_buffer_uptodate(bh);
				}

				clear_buffer_new(bh);
				mark_buffer_dirty(bh);
			}
		}

		block_start = block_end;
		bh = bh->b_this_page;
	} while (bh != head);
}
EXPORT_SYMBOL(folio_zero_new_buffers);

static int
iomap_to_bh(struct inode *inode, sector_t block, struct buffer_head *bh,
		const struct iomap *iomap)
{
	loff_t offset = (loff_t)block << inode->i_blkbits;

	bh->b_bdev = iomap->bdev;

	/*
	 * Block points to offset in file we need to map, iomap contains
	 * the offset at which the map starts. If the map ends before the
	 * current block, then do not map the buffer and let the caller
	 * handle it.
	 */
	if (offset >= iomap->offset + iomap->length)
		return -EIO;

	switch (iomap->type) {
	case IOMAP_HOLE:
		/*
		 * If the buffer is not up to date or beyond the current EOF,
		 * we need to mark it as new to ensure sub-block zeroing is
		 * executed if necessary.
		 */
		if (!buffer_uptodate(bh) ||
		    (offset >= i_size_read(inode)))
			set_buffer_new(bh);
		return 0;
	case IOMAP_DELALLOC:
		if (!buffer_uptodate(bh) ||
		    (offset >= i_size_read(inode)))
			set_buffer_new(bh);
		set_buffer_uptodate(bh);
		set_buffer_mapped(bh);
		set_buffer_delay(bh);
		return 0;
	case IOMAP_UNWRITTEN:
		/*
		 * For unwritten regions, we always need to ensure that regions
		 * in the block we are not writing to are zeroed. Mark the
		 * buffer as new to ensure this.
		 */
		set_buffer_new(bh);
		set_buffer_unwritten(bh);
		fallthrough;
	case IOMAP_MAPPED:
		if ((iomap->flags & IOMAP_F_NEW) ||
		    offset >= i_size_read(inode)) {
			/*
			 * This can happen if truncating the block device races
			 * with the check in the caller as i_size updates on
			 * block devices aren't synchronized by i_rwsem for
			 * block devices.
			 */
			if (S_ISBLK(inode->i_mode))
				return -EIO;
			set_buffer_new(bh);
		}
		bh->b_blocknr = (iomap->addr + offset - iomap->offset) >>
				inode->i_blkbits;
		set_buffer_mapped(bh);
		return 0;
	default:
		WARN_ON_ONCE(1);
		return -EIO;
	}
}

int __block_write_begin_int(struct folio *folio, loff_t pos, unsigned len,
		get_block_t *get_block, const struct iomap *iomap)
{
	size_t from = offset_in_folio(folio, pos);
	size_t to = from + len;
	struct inode *inode = folio->mapping->host;
	size_t block_start, block_end;
	sector_t block;
	int err = 0;
	size_t blocksize;
	struct buffer_head *bh, *head, *wait[2], **wait_bh=wait;

	BUG_ON(!folio_test_locked(folio));
	BUG_ON(to > folio_size(folio));
	BUG_ON(from > to);

	head = folio_create_buffers(folio, inode, 0);
	blocksize = head->b_size;
	block = div_u64(folio_pos(folio), blocksize);

	for (bh = head, block_start = 0; bh != head || !block_start;
	    block++, block_start=block_end, bh = bh->b_this_page) {
		block_end = block_start + blocksize;
		if (block_end <= from || block_start >= to) {
			if (folio_test_uptodate(folio)) {
				if (!buffer_uptodate(bh))
					set_buffer_uptodate(bh);
			}
			continue;
		}
		if (buffer_new(bh))
			clear_buffer_new(bh);
		if (!buffer_mapped(bh)) {
			WARN_ON(bh->b_size != blocksize);
			if (get_block)
				err = get_block(inode, block, bh, 1);
			else
				err = iomap_to_bh(inode, block, bh, iomap);
			if (err)
				break;

			if (buffer_new(bh)) {
				clean_bdev_bh_alias(bh);
				if (folio_test_uptodate(folio)) {
					clear_buffer_new(bh);
					set_buffer_uptodate(bh);
					mark_buffer_dirty(bh);
					continue;
				}
				if (block_end > to || block_start < from)
					folio_zero_segments(folio,
						to, block_end,
						block_start, from);
				continue;
			}
		}
		if (folio_test_uptodate(folio)) {
			if (!buffer_uptodate(bh))
				set_buffer_uptodate(bh);
			continue; 
		}
		if (!buffer_uptodate(bh) && !buffer_delay(bh) &&
		    !buffer_unwritten(bh) &&
		     (block_start < from || block_end > to)) {
			bh_read_nowait(bh, 0);
			*wait_bh++=bh;
		}
	}
	/*
	 * If we issued read requests - let them complete.
	 */
	while(wait_bh > wait) {
		wait_on_buffer(*--wait_bh);
		if (!buffer_uptodate(*wait_bh))
			err = -EIO;
	}
	if (unlikely(err))
		folio_zero_new_buffers(folio, from, to);
	return err;
}

int __block_write_begin(struct folio *folio, loff_t pos, unsigned len,
		get_block_t *get_block)
{
	return __block_write_begin_int(folio, pos, len, get_block, NULL);
}
EXPORT_SYMBOL(__block_write_begin);

void block_commit_write(struct folio *folio, size_t from, size_t to)
{
	size_t block_start, block_end;
	bool partial = false;
	unsigned blocksize;
	struct buffer_head *bh, *head;

	bh = head = folio_buffers(folio);
	if (!bh)
		return;
	blocksize = bh->b_size;

	block_start = 0;
	do {
		block_end = block_start + blocksize;
		if (block_end <= from || block_start >= to) {
			if (!buffer_uptodate(bh))
				partial = true;
		} else {
			set_buffer_uptodate(bh);
			mark_buffer_dirty(bh);
		}
		if (buffer_new(bh))
			clear_buffer_new(bh);

		block_start = block_end;
		bh = bh->b_this_page;
	} while (bh != head);

	/*
	 * If this is a partial write which happened to make all buffers
	 * uptodate then we can optimize away a bogus read_folio() for
	 * the next read(). Here we 'discover' whether the folio went
	 * uptodate as a result of this (potentially partial) write.
	 */
	if (!partial)
		folio_mark_uptodate(folio);
}
EXPORT_SYMBOL(block_commit_write);

/*
 * block_write_begin takes care of the basic task of block allocation and
 * bringing partial write blocks uptodate first.
 *
 * The filesystem needs to handle block truncation upon failure.
 */
int block_write_begin(struct address_space *mapping, loff_t pos, unsigned len,
		struct folio **foliop, get_block_t *get_block)
{
	pgoff_t index = pos >> PAGE_SHIFT;
	struct folio *folio;
	int status;

	folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN,
			mapping_gfp_mask(mapping));
	if (IS_ERR(folio))
		return PTR_ERR(folio);

	status = __block_write_begin_int(folio, pos, len, get_block, NULL);
	if (unlikely(status)) {
		folio_unlock(folio);
		folio_put(folio);
		folio = NULL;
	}

	*foliop = folio;
	return status;
}
EXPORT_SYMBOL(block_write_begin);

int block_write_end(loff_t pos, unsigned len, unsigned copied,
		struct folio *folio)
{
	size_t start = pos - folio_pos(folio);

	if (unlikely(copied < len)) {
		/*
		 * The buffers that were written will now be uptodate, so
		 * we don't have to worry about a read_folio reading them
		 * and overwriting a partial write. However if we have
		 * encountered a short write and only partially written
		 * into a buffer, it will not be marked uptodate, so a
		 * read_folio might come in and destroy our partial write.
		 *
		 * Do the simplest thing, and just treat any short write to a
		 * non uptodate folio as a zero-length write, and force the
		 * caller to redo the whole thing.
		 */
		if (!folio_test_uptodate(folio))
			copied = 0;

		folio_zero_new_buffers(folio, start+copied, start+len);
	}
	flush_dcache_folio(folio);

	/* This could be a short (even 0-length) commit */
	block_commit_write(folio, start, start + copied);

	return copied;
}
EXPORT_SYMBOL(block_write_end);

int generic_write_end(const struct kiocb *iocb, struct address_space *mapping,
		      loff_t pos, unsigned len, unsigned copied,
		      struct folio *folio, void *fsdata)
{
	struct inode *inode = mapping->host;
	loff_t old_size = inode->i_size;
	bool i_size_changed = false;

	copied = block_write_end(pos, len, copied, folio);

	/*
	 * No need to use i_size_read() here, the i_size cannot change under us
	 * because we hold i_rwsem.
	 *
	 * But it's important to update i_size while still holding folio lock:
	 * page writeout could otherwise come in and zero beyond i_size.
	 */
	if (pos + copied > inode->i_size) {
		i_size_write(inode, pos + copied);
		i_size_changed = true;
	}

	folio_unlock(folio);
	folio_put(folio);

	if (old_size < pos)
		pagecache_isize_extended(inode, old_size, pos);
	/*
	 * Don't mark the inode dirty under page lock. First, it unnecessarily
	 * makes the holding time of page lock longer. Second, it forces lock
	 * ordering of page lock and transaction start for journaling
	 * filesystems.
	 */
	if (i_size_changed)
		mark_inode_dirty(inode);
	return copied;
}
EXPORT_SYMBOL(generic_write_end);

/*
 * block_is_partially_uptodate checks whether buffers within a folio are
 * uptodate or not.
 *
 * Returns true if all buffers which correspond to the specified part
 * of the folio are uptodate.
 */
bool block_is_partially_uptodate(struct folio *folio, size_t from, size_t count)
{
	unsigned block_start, block_end, blocksize;
	unsigned to;
	struct buffer_head *bh, *head;
	bool ret = true;

	head = folio_buffers(folio);
	if (!head)
		return false;
	blocksize = head->b_size;
	to = min(folio_size(folio) - from, count);
	to = from + to;
	if (from < blocksize && to > folio_size(folio) - blocksize)
		return false;

	bh = head;
	block_start = 0;
	do {
		block_end = block_start + blocksize;
		if (block_end > from && block_start < to) {
			if (!buffer_uptodate(bh)) {
				ret = false;
				break;
			}
			if (block_end >= to)
				break;
		}
		block_start = block_end;
		bh = bh->b_this_page;
	} while (bh != head);

	return ret;
}
EXPORT_SYMBOL(block_is_partially_uptodate);

/*
 * Generic "read_folio" function for block devices that have the normal
 * get_block functionality. This is most of the block device filesystems.
 * Reads the folio asynchronously --- the unlock_buffer() and
 * set/clear_buffer_uptodate() functions propagate buffer state into the
 * folio once IO has completed.
 */
int block_read_full_folio(struct folio *folio, get_block_t *get_block)
{
	struct inode *inode = folio->mapping->host;
	sector_t iblock, lblock;
	struct buffer_head *bh, *head, *prev = NULL;
	size_t blocksize;
	int fully_mapped = 1;
	bool page_error = false;
	loff_t limit = i_size_read(inode);

	/* This is needed for ext4. */
	if (IS_ENABLED(CONFIG_FS_VERITY) && IS_VERITY(inode))
		limit = inode->i_sb->s_maxbytes;

	head = folio_create_buffers(folio, inode, 0);
	blocksize = head->b_size;

	iblock = div_u64(folio_pos(folio), blocksize);
	lblock = div_u64(limit + blocksize - 1, blocksize);
	bh = head;

	do {
		if (buffer_uptodate(bh))
			continue;

		if (!buffer_mapped(bh)) {
			int err = 0;

			fully_mapped = 0;
			if (iblock < lblock) {
				WARN_ON(bh->b_size != blocksize);
				err = get_block(inode, iblock, bh, 0);
				if (err)
					page_error = true;
			}
			if (!buffer_mapped(bh)) {
				folio_zero_range(folio, bh_offset(bh),
						blocksize);
				if (!err)
					set_buffer_uptodate(bh);
				continue;
			}
			/*
			 * get_block() might have updated the buffer
			 * synchronously
			 */
			if (buffer_uptodate(bh))
				continue;
		}

		lock_buffer(bh);
		if (buffer_uptodate(bh)) {
			unlock_buffer(bh);
			continue;
		}

		mark_buffer_async_read(bh);
		if (prev)
			submit_bh(REQ_OP_READ, prev);
		prev = bh;
	} while (iblock++, (bh = bh->b_this_page) != head);

	if (fully_mapped)
		folio_set_mappedtodisk(folio);

	/*
	 * All buffers are uptodate or get_block() returned an error
	 * when trying to map them - we must finish the read because
	 * end_buffer_async_read() will never be called on any buffer
	 * in this folio.
	 */
	if (prev)
		submit_bh(REQ_OP_READ, prev);
	else
		folio_end_read(folio, !page_error);

	return 0;
}
EXPORT_SYMBOL(block_read_full_folio);

/* utility function for filesystems that need to do work on expanding
 * truncates.  Uses filesystem pagecache writes to allow the filesystem to
 * deal with the hole.  
 */
int generic_cont_expand_simple(struct inode *inode, loff_t size)
{
	struct address_space *mapping = inode->i_mapping;
	const struct address_space_operations *aops = mapping->a_ops;
	struct folio *folio;
	void *fsdata = NULL;
	int err;

	err = inode_newsize_ok(inode, size);
	if (err)
		goto out;

	err = aops->write_begin(NULL, mapping, size, 0, &folio, &fsdata);
	if (err)
		goto out;

	err = aops->write_end(NULL, mapping, size, 0, 0, folio, fsdata);
	BUG_ON(err > 0);

out:
	return err;
}
EXPORT_SYMBOL(generic_cont_expand_simple);

static int cont_expand_zero(const struct kiocb *iocb,
			    struct address_space *mapping,
			    loff_t pos, loff_t *bytes)
{
	struct inode *inode = mapping->host;
	const struct address_space_operations *aops = mapping->a_ops;
	unsigned int blocksize = i_blocksize(inode);
	struct folio *folio;
	void *fsdata = NULL;
	pgoff_t index, curidx;
	loff_t curpos;
	unsigned zerofrom, offset, len;
	int err = 0;

	index = pos >> PAGE_SHIFT;
	offset = pos & ~PAGE_MASK;

	while (index > (curidx = (curpos = *bytes)>>PAGE_SHIFT)) {
		zerofrom = curpos & ~PAGE_MASK;
		if (zerofrom & (blocksize-1)) {
			*bytes |= (blocksize-1);
			(*bytes)++;
		}
		len = PAGE_SIZE - zerofrom;

		err = aops->write_begin(iocb, mapping, curpos, len,
					    &folio, &fsdata);
		if (err)
			goto out;
		folio_zero_range(folio, offset_in_folio(folio, curpos), len);
		err = aops->write_end(iocb, mapping, curpos, len, len,
						folio, fsdata);
		if (err < 0)
			goto out;
		BUG_ON(err != len);
		err = 0;

		balance_dirty_pages_ratelimited(mapping);

		if (fatal_signal_pending(current)) {
			err = -EINTR;
			goto out;
		}
	}

	/* page covers the boundary, find the boundary offset */
	if (index == curidx) {
		zerofrom = curpos & ~PAGE_MASK;
		/* if we will expand the thing last block will be filled */
		if (offset <= zerofrom) {
			goto out;
		}
		if (zerofrom & (blocksize-1)) {
			*bytes |= (blocksize-1);
			(*bytes)++;
		}
		len = offset - zerofrom;

		err = aops->write_begin(iocb, mapping, curpos, len,
					    &folio, &fsdata);
		if (err)
			goto out;
		folio_zero_range(folio, offset_in_folio(folio, curpos), len);
		err = aops->write_end(iocb, mapping, curpos, len, len,
						folio, fsdata);
		if (err < 0)
			goto out;
		BUG_ON(err != len);
		err = 0;
	}
out:
	return err;
}

/*
 * For moronic filesystems that do not allow holes in file.
 * We may have to extend the file.
 */
int cont_write_begin(const struct kiocb *iocb, struct address_space *mapping,
		     loff_t pos, unsigned len, struct folio **foliop,
		     void **fsdata, get_block_t *get_block, loff_t *bytes)
{
	struct inode *inode = mapping->host;
	unsigned int blocksize = i_blocksize(inode);
	unsigned int zerofrom;
	int err;

	err = cont_expand_zero(iocb, mapping, pos, bytes);
	if (err)
		return err;

	zerofrom = *bytes & ~PAGE_MASK;
	if (pos+len > *bytes && zerofrom & (blocksize-1)) {
		*bytes |= (blocksize-1);
		(*bytes)++;
	}

	return block_write_begin(mapping, pos, len, foliop, get_block);
}
EXPORT_SYMBOL(cont_write_begin);

/*
 * block_page_mkwrite() is not allowed to change the file size as it gets
 * called from a page fault handler when a page is first dirtied. Hence we must
 * be careful to check for EOF conditions here. We set the page up correctly
 * for a written page which means we get ENOSPC checking when writing into
 * holes and correct delalloc and unwritten extent mapping on filesystems that
 * support these features.
 *
 * We are not allowed to take the i_rwsem here so we have to play games to
 * protect against truncate races as the page could now be beyond EOF.  Because
 * truncate writes the inode size before removing pages, once we have the
 * page lock we can determine safely if the page is beyond EOF. If it is not
 * beyond EOF, then the page is guaranteed safe against truncation until we
 * unlock the page.
 *
 * Direct callers of this function should protect against filesystem freezing
 * using sb_start_pagefault() - sb_end_pagefault() functions.
 */
int block_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf,
			 get_block_t get_block)
{
	struct folio *folio = page_folio(vmf->page);
	struct inode *inode = file_inode(vma->vm_file);
	unsigned long end;
	loff_t size;
	int ret;

	folio_lock(folio);
	size = i_size_read(inode);
	if ((folio->mapping != inode->i_mapping) ||
	    (folio_pos(folio) >= size)) {
		/* We overload EFAULT to mean page got truncated */
		ret = -EFAULT;
		goto out_unlock;
	}

	end = folio_size(folio);
	/* folio is wholly or partially inside EOF */
	if (folio_pos(folio) + end > size)
		end = size - folio_pos(folio);

	ret = __block_write_begin_int(folio, 0, end, get_block, NULL);
	if (unlikely(ret))
		goto out_unlock;

	block_commit_write(folio, 0, end);

	folio_mark_dirty(folio);
	folio_wait_stable(folio);
	return 0;
out_unlock:
	folio_unlock(folio);
	return ret;
}
EXPORT_SYMBOL(block_page_mkwrite);

int block_truncate_page(struct address_space *mapping,
			loff_t from, get_block_t *get_block)
{
	pgoff_t index = from >> PAGE_SHIFT;
	unsigned blocksize;
	sector_t iblock;
	size_t offset, length, pos;
	struct inode *inode = mapping->host;
	struct folio *folio;
	struct buffer_head *bh;
	int err = 0;

	blocksize = i_blocksize(inode);
	length = from & (blocksize - 1);

	/* Block boundary? Nothing to do */
	if (!length)
		return 0;

	length = blocksize - length;
	iblock = ((loff_t)index * PAGE_SIZE) >> inode->i_blkbits;

	folio = filemap_grab_folio(mapping, index);
	if (IS_ERR(folio))
		return PTR_ERR(folio);

	bh = folio_buffers(folio);
	if (!bh)
		bh = create_empty_buffers(folio, blocksize, 0);

	/* Find the buffer that contains "offset" */
	offset = offset_in_folio(folio, from);
	pos = blocksize;
	while (offset >= pos) {
		bh = bh->b_this_page;
		iblock++;
		pos += blocksize;
	}

	if (!buffer_mapped(bh)) {
		WARN_ON(bh->b_size != blocksize);
		err = get_block(inode, iblock, bh, 0);
		if (err)
			goto unlock;
		/* unmapped? It's a hole - nothing to do */
		if (!buffer_mapped(bh))
			goto unlock;
	}

	/* Ok, it's mapped. Make sure it's up-to-date */
	if (folio_test_uptodate(folio))
		set_buffer_uptodate(bh);

	if (!buffer_uptodate(bh) && !buffer_delay(bh) && !buffer_unwritten(bh)) {
		err = bh_read(bh, 0);
		/* Uhhuh. Read error. Complain and punt. */
		if (err < 0)
			goto unlock;
	}

	folio_zero_range(folio, offset, length);
	mark_buffer_dirty(bh);

unlock:
	folio_unlock(folio);
	folio_put(folio);

	return err;
}
EXPORT_SYMBOL(block_truncate_page);

/*
 * The generic write folio function for buffer-backed address_spaces
 */
int block_write_full_folio(struct folio *folio, struct writeback_control *wbc,
		void *get_block)
{
	struct inode * const inode = folio->mapping->host;
	loff_t i_size = i_size_read(inode);

	/* Is the folio fully inside i_size? */
	if (folio_next_pos(folio) <= i_size)
		return __block_write_full_folio(inode, folio, get_block, wbc);

	/* Is the folio fully outside i_size? (truncate in progress) */
	if (folio_pos(folio) >= i_size) {
		folio_unlock(folio);
		return 0; /* don't care */
	}

	/*
	 * The folio straddles i_size.  It must be zeroed out on each and every
	 * writeback invocation because it may be mmapped.  "A file is mapped
	 * in multiples of the page size.  For a file that is not a multiple of
	 * the page size, the remaining memory is zeroed when mapped, and
	 * writes to that region are not written out to the file."
	 */
	folio_zero_segment(folio, offset_in_folio(folio, i_size),
			folio_size(folio));
	return __block_write_full_folio(inode, folio, get_block, wbc);
}

sector_t generic_block_bmap(struct address_space *mapping, sector_t block,
			    get_block_t *get_block)
{
	struct inode *inode = mapping->host;
	struct buffer_head tmp = {
		.b_size = i_blocksize(inode),
	};

	get_block(inode, block, &tmp, 0);
	return tmp.b_blocknr;
}
EXPORT_SYMBOL(generic_block_bmap);

static void end_bio_bh_io_sync(struct bio *bio)
{
	struct buffer_head *bh = bio->bi_private;

	if (unlikely(bio_flagged(bio, BIO_QUIET)))
		set_bit(BH_Quiet, &bh->b_state);

	bh->b_end_io(bh, !bio->bi_status);
	bio_put(bio);
}

static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh,
			  enum rw_hint write_hint,
			  struct writeback_control *wbc)
{
	const enum req_op op = opf & REQ_OP_MASK;
	struct bio *bio;

	BUG_ON(!buffer_locked(bh));
	BUG_ON(!buffer_mapped(bh));
	BUG_ON(!bh->b_end_io);
	BUG_ON(buffer_delay(bh));
	BUG_ON(buffer_unwritten(bh));

	/*
	 * Only clear out a write error when rewriting
	 */
	if (test_set_buffer_req(bh) && (op == REQ_OP_WRITE))
		clear_buffer_write_io_error(bh);

	if (buffer_meta(bh))
		opf |= REQ_META;
	if (buffer_prio(bh))
		opf |= REQ_PRIO;

	bio = bio_alloc(bh->b_bdev, 1, opf, GFP_NOIO);

	fscrypt_set_bio_crypt_ctx_bh(bio, bh, GFP_NOIO);

	bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9);
	bio->bi_write_hint = write_hint;

	bio_add_folio_nofail(bio, bh->b_folio, bh->b_size, bh_offset(bh));

	bio->bi_end_io = end_bio_bh_io_sync;
	bio->bi_private = bh;

	/* Take care of bh's that straddle the end of the device */
	guard_bio_eod(bio);

	if (wbc) {
		wbc_init_bio(wbc, bio);
		wbc_account_cgroup_owner(wbc, bh->b_folio, bh->b_size);
	}

	blk_crypto_submit_bio(bio);
}

void submit_bh(blk_opf_t opf, struct buffer_head *bh)
{
	submit_bh_wbc(opf, bh, WRITE_LIFE_NOT_SET, NULL);
}
EXPORT_SYMBOL(submit_bh);

void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags)
{
	lock_buffer(bh);
	if (!test_clear_buffer_dirty(bh)) {
		unlock_buffer(bh);
		return;
	}
	bh->b_end_io = end_buffer_write_sync;
	get_bh(bh);
	submit_bh(REQ_OP_WRITE | op_flags, bh);
}
EXPORT_SYMBOL(write_dirty_buffer);

/*
 * For a data-integrity writeout, we need to wait upon any in-progress I/O
 * and then start new I/O and then wait upon it.  The caller must have a ref on
 * the buffer_head.
 */
int __sync_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags)
{
	WARN_ON(atomic_read(&bh->b_count) < 1);
	lock_buffer(bh);
	if (test_clear_buffer_dirty(bh)) {
		/*
		 * The bh should be mapped, but it might not be if the
		 * device was hot-removed. Not much we can do but fail the I/O.
		 */
		if (!buffer_mapped(bh)) {
			unlock_buffer(bh);
			return -EIO;
		}

		get_bh(bh);
		bh->b_end_io = end_buffer_write_sync;
		submit_bh(REQ_OP_WRITE | op_flags, bh);
		wait_on_buffer(bh);
		if (!buffer_uptodate(bh))
			return -EIO;
	} else {
		unlock_buffer(bh);
	}
	return 0;
}
EXPORT_SYMBOL(__sync_dirty_buffer);

int sync_dirty_buffer(struct buffer_head *bh)
{
	return __sync_dirty_buffer(bh, REQ_SYNC);
}
EXPORT_SYMBOL(sync_dirty_buffer);

static inline int buffer_busy(struct buffer_head *bh)
{
	return atomic_read(&bh->b_count) |
		(bh->b_state & ((1 << BH_Dirty) | (1 << BH_Lock)));
}

static bool
drop_buffers(struct folio *folio, struct buffer_head **buffers_to_free)
{
	struct buffer_head *head = folio_buffers(folio);
	struct buffer_head *bh;

	bh = head;
	do {
		if (buffer_busy(bh))
			goto failed;
		bh = bh->b_this_page;
	} while (bh != head);

	do {
		struct buffer_head *next = bh->b_this_page;

		if (bh->b_assoc_map)
			__remove_assoc_queue(bh);
		bh = next;
	} while (bh != head);
	*buffers_to_free = head;
	folio_detach_private(folio);
	return true;
failed:
	return false;
}

/**
 * try_to_free_buffers - Release buffers attached to this folio.
 * @folio: The folio.
 *
 * If any buffers are in use (dirty, under writeback, elevated refcount),
 * no buffers will be freed.
 *
 * If the folio is dirty but all the buffers are clean then we need to
 * be sure to mark the folio clean as well.  This is because the folio
 * may be against a block device, and a later reattachment of buffers
 * to a dirty folio will set *all* buffers dirty.  Which would corrupt
 * filesystem data on the same device.
 *
 * The same applies to regular filesystem folios: if all the buffers are
 * clean then we set the folio clean and proceed.  To do that, we require
 * total exclusion from block_dirty_folio().  That is obtained with
 * i_private_lock.
 *
 * Exclusion against try_to_free_buffers may be obtained by either
 * locking the folio or by holding its mapping's i_private_lock.
 *
 * Context: Process context.  @folio must be locked.  Will not sleep.
 * Return: true if all buffers attached to this folio were freed.
 */
bool try_to_free_buffers(struct folio *folio)
{
	struct address_space * const mapping = folio->mapping;
	struct buffer_head *buffers_to_free = NULL;
	bool ret = 0;

	BUG_ON(!folio_test_locked(folio));
	if (folio_test_writeback(folio))
		return false;

	/* Misconfigured folio check */
	if (WARN_ON_ONCE(!folio_buffers(folio)))
		return true;

	if (mapping == NULL) {		/* can this still happen? */
		ret = drop_buffers(folio, &buffers_to_free);
		goto out;
	}

	spin_lock(&mapping->i_private_lock);
	ret = drop_buffers(folio, &buffers_to_free);

	/*
	 * If the filesystem writes its buffers by hand (eg ext3)
	 * then we can have clean buffers against a dirty folio.  We
	 * clean the folio here; otherwise the VM will never notice
	 * that the filesystem did any IO at all.
	 *
	 * Also, during truncate, discard_buffer will have marked all
	 * the folio's buffers clean.  We discover that here and clean
	 * the folio also.
	 *
	 * i_private_lock must be held over this entire operation in order
	 * to synchronise against block_dirty_folio and prevent the
	 * dirty bit from being lost.
	 */
	if (ret)
		folio_cancel_dirty(folio);
	spin_unlock(&mapping->i_private_lock);
out:
	if (buffers_to_free) {
		struct buffer_head *bh = buffers_to_free;

		do {
			struct buffer_head *next = bh->b_this_page;
			free_buffer_head(bh);
			bh = next;
		} while (bh != buffers_to_free);
	}
	return ret;
}
EXPORT_SYMBOL(try_to_free_buffers);

/*
 * Buffer-head allocation
 */
static struct kmem_cache *bh_cachep __ro_after_init;

/*
 * Once the number of bh's in the machine exceeds this level, we start
 * stripping them in writeback.
 */
static unsigned long max_buffer_heads __ro_after_init;

int buffer_heads_over_limit;

struct bh_accounting {
	int nr;			/* Number of live bh's */
	int ratelimit;		/* Limit cacheline bouncing */
};

static DEFINE_PER_CPU(struct bh_accounting, bh_accounting) = {0, 0};

static void recalc_bh_state(void)
{
	int i;
	int tot = 0;

	if (__this_cpu_inc_return(bh_accounting.ratelimit) - 1 < 4096)
		return;
	__this_cpu_write(bh_accounting.ratelimit, 0);
	for_each_online_cpu(i)
		tot += per_cpu(bh_accounting, i).nr;
	buffer_heads_over_limit = (tot > max_buffer_heads);
}

struct buffer_head *alloc_buffer_head(gfp_t gfp_flags)
{
	struct buffer_head *ret = kmem_cache_zalloc(bh_cachep, gfp_flags);
	if (ret) {
		INIT_LIST_HEAD(&ret->b_assoc_buffers);
		spin_lock_init(&ret->b_uptodate_lock);
		preempt_disable();
		__this_cpu_inc(bh_accounting.nr);
		recalc_bh_state();
		preempt_enable();
	}
	return ret;
}
EXPORT_SYMBOL(alloc_buffer_head);

void free_buffer_head(struct buffer_head *bh)
{
	BUG_ON(!list_empty(&bh->b_assoc_buffers));
	kmem_cache_free(bh_cachep, bh);
	preempt_disable();
	__this_cpu_dec(bh_accounting.nr);
	recalc_bh_state();
	preempt_enable();
}
EXPORT_SYMBOL(free_buffer_head);

static int buffer_exit_cpu_dead(unsigned int cpu)
{
	int i;
	struct bh_lru *b = &per_cpu(bh_lrus, cpu);

	for (i = 0; i < BH_LRU_SIZE; i++) {
		brelse(b->bhs[i]);
		b->bhs[i] = NULL;
	}
	this_cpu_add(bh_accounting.nr, per_cpu(bh_accounting, cpu).nr);
	per_cpu(bh_accounting, cpu).nr = 0;
	return 0;
}

/**
 * bh_uptodate_or_lock - Test whether the buffer is uptodate
 * @bh: struct buffer_head
 *
 * Return true if the buffer is up-to-date and false,
 * with the buffer locked, if not.
 */
int bh_uptodate_or_lock(struct buffer_head *bh)
{
	if (!buffer_uptodate(bh)) {
		lock_buffer(bh);
		if (!buffer_uptodate(bh))
			return 0;
		unlock_buffer(bh);
	}
	return 1;
}
EXPORT_SYMBOL(bh_uptodate_or_lock);

/**
 * __bh_read - Submit read for a locked buffer
 * @bh: struct buffer_head
 * @op_flags: appending REQ_OP_* flags besides REQ_OP_READ
 * @wait: wait until reading finish
 *
 * Returns zero on success or don't wait, and -EIO on error.
 */
int __bh_read(struct buffer_head *bh, blk_opf_t op_flags, bool wait)
{
	int ret = 0;

	BUG_ON(!buffer_locked(bh));

	get_bh(bh);
	bh->b_end_io = end_buffer_read_sync;
	submit_bh(REQ_OP_READ | op_flags, bh);
	if (wait) {
		wait_on_buffer(bh);
		if (!buffer_uptodate(bh))
			ret = -EIO;
	}
	return ret;
}
EXPORT_SYMBOL(__bh_read);

/**
 * __bh_read_batch - Submit read for a batch of unlocked buffers
 * @nr: entry number of the buffer batch
 * @bhs: a batch of struct buffer_head
 * @op_flags: appending REQ_OP_* flags besides REQ_OP_READ
 * @force_lock: force to get a lock on the buffer if set, otherwise drops any
 *              buffer that cannot lock.
 *
 * Returns zero on success or don't wait, and -EIO on error.
 */
void __bh_read_batch(int nr, struct buffer_head *bhs[],
		     blk_opf_t op_flags, bool force_lock)
{
	int i;

	for (i = 0; i < nr; i++) {
		struct buffer_head *bh = bhs[i];

		if (buffer_uptodate(bh))
			continue;

		if (force_lock)
			lock_buffer(bh);
		else
			if (!trylock_buffer(bh))
				continue;

		if (buffer_uptodate(bh)) {
			unlock_buffer(bh);
			continue;
		}

		bh->b_end_io = end_buffer_read_sync;
		get_bh(bh);
		submit_bh(REQ_OP_READ | op_flags, bh);
	}
}
EXPORT_SYMBOL(__bh_read_batch);

void __init buffer_init(void)
{
	unsigned long nrpages;
	int ret;

	bh_cachep = KMEM_CACHE(buffer_head,
				SLAB_RECLAIM_ACCOUNT|SLAB_PANIC);
	/*
	 * Limit the bh occupancy to 10% of ZONE_NORMAL
	 */
	nrpages = (nr_free_buffer_pages() * 10) / 100;
	max_buffer_heads = nrpages * (PAGE_SIZE / sizeof(struct buffer_head));
	ret = cpuhp_setup_state_nocalls(CPUHP_FS_BUFF_DEAD, "fs/buffer:dead",
					NULL, buffer_exit_cpu_dead);
	WARN_ON(ret < 0);
}
]

gemini-3-flash-preview

							input: 35407
							output: 21
							thoughts: 1016
						

**Data Race Detected in Buffer Head LRU Invalidation**

Okay, so I'm looking at a data race situation here. It seems we have a classic scenario where one CPU (CPU 0) is clearing out entries in a buffer head LRU (Least Recently Used) list while another CPU (CPU 1) is checking if that same LRU list contains any valid entries. This is bad news, as we're potentially reading and writing the same memory location concurrently without proper synchronization.

Specifically, CPU 0 is executing `__invalidate_bh_lrus(b)` which iterates through the `bhs` array of the `bh_lru` struct and sets each entry to `NULL` (Line 1526, `b->bhs[i] = NULL;`). Meanwhile, CPU 1 is running `has_bh_in_lru(cpu, dummy)` which is checking if *any* entry in the `bh_lru` struct's `bhs` array is non-NULL (Line 1548, `if (b->bhs[i])`).

The report confirms this with a write to `0xffff888237c27760` (8 bytes) by CPU 0 and a read from the same address by CPU 1. The key point is that the value *changed* from something non-zero (`0xffff88810645fc30`) to zero (`0x0000000000000000`). This perfectly illustrates the race: CPU 0 is clearing the memory, while CPU 1 is checking.

Now, this is happening within the context of `invalidate_bh_lrus()`, which calls `on_each_cpu_cond`. I know `on_each_cpu_cond` calls `has_bh_in_lru` on each CPU and *only* schedules `invalidate_bh_lru` on a CPU if `has_bh_in_lru` returns true for that CPU. 

Digging deeper, the stack trace for the *write* shows that `invalidate_bh_lrus_cpu` locks the `bh_lru` with `bh_lru_lock()`, which is actually `local_irq_disable()` on SMP systems. The problem isn't inside `__invalidate_bh_lrus`, but instead it lies in the fact that *another* CPU may have a stale copy of the `bh_lru` struct.

The read stack trace shows the `on_each_cpu_cond` call, which I know ends up calling `on_each_cpu_cond_mask`, which calls `has_bh_in_lru` on each CPU. It seems to be checking the LRU for buffer heads *before* invalidating them, which is the intention. However, the data race would appear when `has_bh_in_lru` checks the buffer head, which is then immediately invalidated by another CPU.

The likely issue is that we need a *stronger* synchronization mechanism than just disabling interrupts. There needs to be a critical section that prevents the read and write from happening concurrently. A mutex or a spinlock, for example, would be ideal. I have to look into how the LRU is accessed by other paths, to ensure that this isn't a problem, when the LRU is modified.

map[File:kernel/smp.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
 * Generic helpers for smp ipi calls
 *
 * (C) Jens Axboe <jens.axboe@oracle.com> 2008
 */

#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

#include <linux/irq_work.h>
#include <linux/rcupdate.h>
#include <linux/rculist.h>
#include <linux/kernel.h>
#include <linux/export.h>
#include <linux/percpu.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/gfp.h>
#include <linux/smp.h>
#include <linux/cpu.h>
#include <linux/sched.h>
#include <linux/sched/idle.h>
#include <linux/hypervisor.h>
#include <linux/sched/clock.h>
#include <linux/nmi.h>
#include <linux/sched/debug.h>
#include <linux/jump_label.h>
#include <linux/string_choices.h>

#include <trace/events/ipi.h>
#define CREATE_TRACE_POINTS
#include <trace/events/csd.h>
#undef CREATE_TRACE_POINTS

#include "smpboot.h"
#include "sched/smp.h"

#define CSD_TYPE(_csd)	((_csd)->node.u_flags & CSD_FLAG_TYPE_MASK)

struct call_function_data {
	call_single_data_t	__percpu *csd;
	cpumask_var_t		cpumask;
	cpumask_var_t		cpumask_ipi;
};

static DEFINE_PER_CPU_ALIGNED(struct call_function_data, cfd_data);

static DEFINE_PER_CPU_SHARED_ALIGNED(struct llist_head, call_single_queue);

static DEFINE_PER_CPU(atomic_t, trigger_backtrace) = ATOMIC_INIT(1);

static void __flush_smp_call_function_queue(bool warn_cpu_offline);

int smpcfd_prepare_cpu(unsigned int cpu)
{
	struct call_function_data *cfd = &per_cpu(cfd_data, cpu);

	if (!zalloc_cpumask_var_node(&cfd->cpumask, GFP_KERNEL,
				     cpu_to_node(cpu)))
		return -ENOMEM;
	if (!zalloc_cpumask_var_node(&cfd->cpumask_ipi, GFP_KERNEL,
				     cpu_to_node(cpu))) {
		free_cpumask_var(cfd->cpumask);
		return -ENOMEM;
	}
	cfd->csd = alloc_percpu(call_single_data_t);
	if (!cfd->csd) {
		free_cpumask_var(cfd->cpumask);
		free_cpumask_var(cfd->cpumask_ipi);
		return -ENOMEM;
	}

	return 0;
}

int smpcfd_dead_cpu(unsigned int cpu)
{
	struct call_function_data *cfd = &per_cpu(cfd_data, cpu);

	free_cpumask_var(cfd->cpumask);
	free_cpumask_var(cfd->cpumask_ipi);
	free_percpu(cfd->csd);
	return 0;
}

int smpcfd_dying_cpu(unsigned int cpu)
{
	/*
	 * The IPIs for the smp-call-function callbacks queued by other CPUs
	 * might arrive late, either due to hardware latencies or because this
	 * CPU disabled interrupts (inside stop-machine) before the IPIs were
	 * sent. So flush out any pending callbacks explicitly (without waiting
	 * for the IPIs to arrive), to ensure that the outgoing CPU doesn't go
	 * offline with work still pending.
	 *
	 * This runs with interrupts disabled inside the stopper task invoked by
	 * stop_machine(), ensuring mutually exclusive CPU offlining and IPI flush.
	 */
	__flush_smp_call_function_queue(false);
	irq_work_run();
	return 0;
}

void __init call_function_init(void)
{
	int i;

	for_each_possible_cpu(i)
		init_llist_head(&per_cpu(call_single_queue, i));

	smpcfd_prepare_cpu(smp_processor_id());
}

static __always_inline void
send_call_function_single_ipi(int cpu)
{
	if (call_function_single_prep_ipi(cpu)) {
		trace_ipi_send_cpu(cpu, _RET_IP_,
				   generic_smp_call_function_single_interrupt);
		arch_send_call_function_single_ipi(cpu);
	}
}

static __always_inline void
send_call_function_ipi_mask(struct cpumask *mask)
{
	trace_ipi_send_cpumask(mask, _RET_IP_,
			       generic_smp_call_function_single_interrupt);
	arch_send_call_function_ipi_mask(mask);
}

static __always_inline void
csd_do_func(smp_call_func_t func, void *info, call_single_data_t *csd)
{
	trace_csd_function_entry(func, csd);
	func(info);
	trace_csd_function_exit(func, csd);
}

#ifdef CONFIG_CSD_LOCK_WAIT_DEBUG

static DEFINE_STATIC_KEY_MAYBE(CONFIG_CSD_LOCK_WAIT_DEBUG_DEFAULT, csdlock_debug_enabled);

/*
 * Parse the csdlock_debug= kernel boot parameter.
 *
 * If you need to restore the old "ext" value that once provided
 * additional debugging information, reapply the following commits:
 *
 * de7b09ef658d ("locking/csd_lock: Prepare more CSD lock debugging")
 * a5aabace5fb8 ("locking/csd_lock: Add more data to CSD lock debugging")
 */
static int __init csdlock_debug(char *str)
{
	int ret;
	unsigned int val = 0;

	ret = get_option(&str, &val);
	if (ret) {
		if (val)
			static_branch_enable(&csdlock_debug_enabled);
		else
			static_branch_disable(&csdlock_debug_enabled);
	}

	return 1;
}
__setup("csdlock_debug=", csdlock_debug);

static DEFINE_PER_CPU(call_single_data_t *, cur_csd);
static DEFINE_PER_CPU(smp_call_func_t, cur_csd_func);
static DEFINE_PER_CPU(void *, cur_csd_info);

static ulong csd_lock_timeout = 5000;  /* CSD lock timeout in milliseconds. */
module_param(csd_lock_timeout, ulong, 0644);
static int panic_on_ipistall;  /* CSD panic timeout in milliseconds, 300000 for five minutes. */
module_param(panic_on_ipistall, int, 0644);

static atomic_t csd_bug_count = ATOMIC_INIT(0);

/* Record current CSD work for current CPU, NULL to erase. */
static void __csd_lock_record(call_single_data_t *csd)
{
	if (!csd) {
		smp_mb(); /* NULL cur_csd after unlock. */
		__this_cpu_write(cur_csd, NULL);
		return;
	}
	__this_cpu_write(cur_csd_func, csd->func);
	__this_cpu_write(cur_csd_info, csd->info);
	smp_wmb(); /* func and info before csd. */
	__this_cpu_write(cur_csd, csd);
	smp_mb(); /* Update cur_csd before function call. */
		  /* Or before unlock, as the case may be. */
}

static __always_inline void csd_lock_record(call_single_data_t *csd)
{
	if (static_branch_unlikely(&csdlock_debug_enabled))
		__csd_lock_record(csd);
}

static int csd_lock_wait_getcpu(call_single_data_t *csd)
{
	unsigned int csd_type;

	csd_type = CSD_TYPE(csd);
	if (csd_type == CSD_TYPE_ASYNC || csd_type == CSD_TYPE_SYNC)
		return csd->node.dst; /* Other CSD_TYPE_ values might not have ->dst. */
	return -1;
}

static atomic_t n_csd_lock_stuck;

/**
 * csd_lock_is_stuck - Has a CSD-lock acquisition been stuck too long?
 *
 * Returns @true if a CSD-lock acquisition is stuck and has been stuck
 * long enough for a "non-responsive CSD lock" message to be printed.
 */
bool csd_lock_is_stuck(void)
{
	return !!atomic_read(&n_csd_lock_stuck);
}

/*
 * Complain if too much time spent waiting.  Note that only
 * the CSD_TYPE_SYNC/ASYNC types provide the destination CPU,
 * so waiting on other types gets much less information.
 */
static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, int *bug_id, unsigned long *nmessages)
{
	int cpu = -1;
	int cpux;
	bool firsttime;
	u64 ts2, ts_delta;
	call_single_data_t *cpu_cur_csd;
	unsigned int flags = READ_ONCE(csd->node.u_flags);
	unsigned long long csd_lock_timeout_ns = csd_lock_timeout * NSEC_PER_MSEC;

	if (!(flags & CSD_FLAG_LOCK)) {
		if (!unlikely(*bug_id))
			return true;
		cpu = csd_lock_wait_getcpu(csd);
		pr_alert("csd: CSD lock (#%d) got unstuck on CPU#%02d, CPU#%02d released the lock.\n",
			 *bug_id, raw_smp_processor_id(), cpu);
		atomic_dec(&n_csd_lock_stuck);
		return true;
	}

	ts2 = ktime_get_mono_fast_ns();
	/* How long since we last checked for a stuck CSD lock.*/
	ts_delta = ts2 - *ts1;
	if (likely(ts_delta <= csd_lock_timeout_ns * (*nmessages + 1) *
			       (!*nmessages ? 1 : (ilog2(num_online_cpus()) / 2 + 1)) ||
		   csd_lock_timeout_ns == 0))
		return false;

	if (ts0 > ts2) {
		/* Our own sched_clock went backward; don't blame another CPU. */
		ts_delta = ts0 - ts2;
		pr_alert("sched_clock on CPU %d went backward by %llu ns\n", raw_smp_processor_id(), ts_delta);
		*ts1 = ts2;
		return false;
	}

	firsttime = !*bug_id;
	if (firsttime)
		*bug_id = atomic_inc_return(&csd_bug_count);
	cpu = csd_lock_wait_getcpu(csd);
	if (WARN_ONCE(cpu < 0 || cpu >= nr_cpu_ids, "%s: cpu = %d\n", __func__, cpu))
		cpux = 0;
	else
		cpux = cpu;
	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux)); /* Before func and info. */
	/* How long since this CSD lock was stuck. */
	ts_delta = ts2 - ts0;
	pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",
		 firsttime ? "Detected" : "Continued", *bug_id, raw_smp_processor_id(), (s64)ts_delta,
		 cpu, csd->func, csd->info);
	(*nmessages)++;
	if (firsttime)
		atomic_inc(&n_csd_lock_stuck);
	/*
	 * If the CSD lock is still stuck after 5 minutes, it is unlikely
	 * to become unstuck. Use a signed comparison to avoid triggering
	 * on underflows when the TSC is out of sync between sockets.
	 */
	BUG_ON(panic_on_ipistall > 0 && (s64)ts_delta > ((s64)panic_on_ipistall * NSEC_PER_MSEC));
	if (cpu_cur_csd && csd != cpu_cur_csd) {
		pr_alert("\tcsd: CSD lock (#%d) handling prior %pS(%ps) request.\n",
			 *bug_id, READ_ONCE(per_cpu(cur_csd_func, cpux)),
			 READ_ONCE(per_cpu(cur_csd_info, cpux)));
	} else {
		pr_alert("\tcsd: CSD lock (#%d) %s.\n",
			 *bug_id, !cpu_cur_csd ? "unresponsive" : "handling this request");
	}
	if (cpu >= 0) {
		if (atomic_cmpxchg_acquire(&per_cpu(trigger_backtrace, cpu), 1, 0))
			dump_cpu_task(cpu);
		if (!cpu_cur_csd) {
			pr_alert("csd: Re-sending CSD lock (#%d) IPI from CPU#%02d to CPU#%02d\n", *bug_id, raw_smp_processor_id(), cpu);
			arch_send_call_function_single_ipi(cpu);
		}
	}
	if (firsttime)
		dump_stack();
	*ts1 = ts2;

	return false;
}

/*
 * csd_lock/csd_unlock used to serialize access to per-cpu csd resources
 *
 * For non-synchronous ipi calls the csd can still be in use by the
 * previous function call. For multi-cpu calls its even more interesting
 * as we'll have to ensure no other cpu is observing our csd.
 */
static void __csd_lock_wait(call_single_data_t *csd)
{
	unsigned long nmessages = 0;
	int bug_id = 0;
	u64 ts0, ts1;

	ts1 = ts0 = ktime_get_mono_fast_ns();
	for (;;) {
		if (csd_lock_wait_toolong(csd, ts0, &ts1, &bug_id, &nmessages))
			break;
		cpu_relax();
	}
	smp_acquire__after_ctrl_dep();
}

static __always_inline void csd_lock_wait(call_single_data_t *csd)
{
	if (static_branch_unlikely(&csdlock_debug_enabled)) {
		__csd_lock_wait(csd);
		return;
	}

	smp_cond_load_acquire(&csd->node.u_flags, !(VAL & CSD_FLAG_LOCK));
}
#else
static void csd_lock_record(call_single_data_t *csd)
{
}

static __always_inline void csd_lock_wait(call_single_data_t *csd)
{
	smp_cond_load_acquire(&csd->node.u_flags, !(VAL & CSD_FLAG_LOCK));
}
#endif

static __always_inline void csd_lock(call_single_data_t *csd)
{
	csd_lock_wait(csd);
	csd->node.u_flags |= CSD_FLAG_LOCK;

	/*
	 * prevent CPU from reordering the above assignment
	 * to ->flags with any subsequent assignments to other
	 * fields of the specified call_single_data_t structure:
	 */
	smp_wmb();
}

static __always_inline void csd_unlock(call_single_data_t *csd)
{
	WARN_ON(!(csd->node.u_flags & CSD_FLAG_LOCK));

	/*
	 * ensure we're all done before releasing data:
	 */
	smp_store_release(&csd->node.u_flags, 0);
}

static DEFINE_PER_CPU_SHARED_ALIGNED(call_single_data_t, csd_data);

void __smp_call_single_queue(int cpu, struct llist_node *node)
{
	/*
	 * We have to check the type of the CSD before queueing it, because
	 * once queued it can have its flags cleared by
	 *   flush_smp_call_function_queue()
	 * even if we haven't sent the smp_call IPI yet (e.g. the stopper
	 * executes migration_cpu_stop() on the remote CPU).
	 */
	if (trace_csd_queue_cpu_enabled()) {
		call_single_data_t *csd;
		smp_call_func_t func;

		csd = container_of(node, call_single_data_t, node.llist);
		func = CSD_TYPE(csd) == CSD_TYPE_TTWU ?
			sched_ttwu_pending : csd->func;

		trace_csd_queue_cpu(cpu, _RET_IP_, func, csd);
	}

	/*
	 * The list addition should be visible to the target CPU when it pops
	 * the head of the list to pull the entry off it in the IPI handler
	 * because of normal cache coherency rules implied by the underlying
	 * llist ops.
	 *
	 * If IPIs can go out of order to the cache coherency protocol
	 * in an architecture, sufficient synchronisation should be added
	 * to arch code to make it appear to obey cache coherency WRT
	 * locking and barrier primitives. Generic code isn't really
	 * equipped to do the right thing...
	 */
	if (llist_add(node, &per_cpu(call_single_queue, cpu)))
		send_call_function_single_ipi(cpu);
}

/*
 * Insert a previously allocated call_single_data_t element
 * for execution on the given CPU. data must already have
 * ->func, ->info, and ->flags set.
 */
static int generic_exec_single(int cpu, call_single_data_t *csd)
{
	/*
	 * Preemption already disabled here so stopper cannot run on this CPU,
	 * ensuring mutually exclusive CPU offlining and last IPI flush.
	 */
	if (cpu == smp_processor_id()) {
		smp_call_func_t func = csd->func;
		void *info = csd->info;
		unsigned long flags;

		/*
		 * We can unlock early even for the synchronous on-stack case,
		 * since we're doing this from the same CPU..
		 */
		csd_lock_record(csd);
		csd_unlock(csd);
		local_irq_save(flags);
		csd_do_func(func, info, NULL);
		csd_lock_record(NULL);
		local_irq_restore(flags);
		return 0;
	}

	if ((unsigned)cpu >= nr_cpu_ids || !cpu_online(cpu)) {
		csd_unlock(csd);
		return -ENXIO;
	}

	__smp_call_single_queue(cpu, &csd->node.llist);

	return 0;
}

/**
 * generic_smp_call_function_single_interrupt - Execute SMP IPI callbacks
 *
 * Invoked by arch to handle an IPI for call function single.
 * Must be called with interrupts disabled.
 */
void generic_smp_call_function_single_interrupt(void)
{
	__flush_smp_call_function_queue(true);
}

/**
 * __flush_smp_call_function_queue - Flush pending smp-call-function callbacks
 *
 * @warn_cpu_offline: If set to 'true', warn if callbacks were queued on an
 *		      offline CPU. Skip this check if set to 'false'.
 *
 * Flush any pending smp-call-function callbacks queued on this CPU. This is
 * invoked by the generic IPI handler, as well as by a CPU about to go offline,
 * to ensure that all pending IPI callbacks are run before it goes completely
 * offline.
 *
 * Loop through the call_single_queue and run all the queued callbacks.
 * Must be called with interrupts disabled.
 */
static void __flush_smp_call_function_queue(bool warn_cpu_offline)
{
	call_single_data_t *csd, *csd_next;
	struct llist_node *entry, *prev;
	struct llist_head *head;
	static bool warned;
	atomic_t *tbt;

	lockdep_assert_irqs_disabled();

	/* Allow waiters to send backtrace NMI from here onwards */
	tbt = this_cpu_ptr(&trigger_backtrace);
	atomic_set_release(tbt, 1);

	head = this_cpu_ptr(&call_single_queue);
	entry = llist_del_all(head);
	entry = llist_reverse_order(entry);

	/* There shouldn't be any pending callbacks on an offline CPU. */
	if (unlikely(warn_cpu_offline && !cpu_online(smp_processor_id()) &&
		     !warned && entry != NULL)) {
		warned = true;
		WARN(1, "IPI on offline CPU %d\n", smp_processor_id());

		/*
		 * We don't have to use the _safe() variant here
		 * because we are not invoking the IPI handlers yet.
		 */
		llist_for_each_entry(csd, entry, node.llist) {
			switch (CSD_TYPE(csd)) {
			case CSD_TYPE_ASYNC:
			case CSD_TYPE_SYNC:
			case CSD_TYPE_IRQ_WORK:
				pr_warn("IPI callback %pS sent to offline CPU\n",
					csd->func);
				break;

			case CSD_TYPE_TTWU:
				pr_warn("IPI task-wakeup sent to offline CPU\n");
				break;

			default:
				pr_warn("IPI callback, unknown type %d, sent to offline CPU\n",
					CSD_TYPE(csd));
				break;
			}
		}
	}

	/*
	 * First; run all SYNC callbacks, people are waiting for us.
	 */
	prev = NULL;
	llist_for_each_entry_safe(csd, csd_next, entry, node.llist) {
		/* Do we wait until *after* callback? */
		if (CSD_TYPE(csd) == CSD_TYPE_SYNC) {
			smp_call_func_t func = csd->func;
			void *info = csd->info;

			if (prev) {
				prev->next = &csd_next->node.llist;
			} else {
				entry = &csd_next->node.llist;
			}

			csd_lock_record(csd);
			csd_do_func(func, info, csd);
			csd_unlock(csd);
			csd_lock_record(NULL);
		} else {
			prev = &csd->node.llist;
		}
	}

	if (!entry)
		return;

	/*
	 * Second; run all !SYNC callbacks.
	 */
	prev = NULL;
	llist_for_each_entry_safe(csd, csd_next, entry, node.llist) {
		int type = CSD_TYPE(csd);

		if (type != CSD_TYPE_TTWU) {
			if (prev) {
				prev->next = &csd_next->node.llist;
			} else {
				entry = &csd_next->node.llist;
			}

			if (type == CSD_TYPE_ASYNC) {
				smp_call_func_t func = csd->func;
				void *info = csd->info;

				csd_lock_record(csd);
				csd_unlock(csd);
				csd_do_func(func, info, csd);
				csd_lock_record(NULL);
			} else if (type == CSD_TYPE_IRQ_WORK) {
				irq_work_single(csd);
			}

		} else {
			prev = &csd->node.llist;
		}
	}

	/*
	 * Third; only CSD_TYPE_TTWU is left, issue those.
	 */
	if (entry) {
		csd = llist_entry(entry, typeof(*csd), node.llist);
		csd_do_func(sched_ttwu_pending, entry, csd);
	}
}


/**
 * flush_smp_call_function_queue - Flush pending smp-call-function callbacks
 *				   from task context (idle, migration thread)
 *
 * When TIF_POLLING_NRFLAG is supported and a CPU is in idle and has it
 * set, then remote CPUs can avoid sending IPIs and wake the idle CPU by
 * setting TIF_NEED_RESCHED. The idle task on the woken up CPU has to
 * handle queued SMP function calls before scheduling.
 *
 * The migration thread has to ensure that an eventually pending wakeup has
 * been handled before it migrates a task.
 */
void flush_smp_call_function_queue(void)
{
	unsigned int was_pending;
	unsigned long flags;

	if (llist_empty(this_cpu_ptr(&call_single_queue)))
		return;

	local_irq_save(flags);
	/* Get the already pending soft interrupts for RT enabled kernels */
	was_pending = local_softirq_pending();
	__flush_smp_call_function_queue(true);
	if (local_softirq_pending())
		do_softirq_post_smp_call_flush(was_pending);

	local_irq_restore(flags);
}

/*
 * smp_call_function_single - Run a function on a specific CPU
 * @func: The function to run. This must be fast and non-blocking.
 * @info: An arbitrary pointer to pass to the function.
 * @wait: If true, wait until function has completed on other CPUs.
 *
 * Returns 0 on success, else a negative status code.
 */
int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
			     int wait)
{
	call_single_data_t *csd;
	call_single_data_t csd_stack = {
		.node = { .u_flags = CSD_FLAG_LOCK | CSD_TYPE_SYNC, },
	};
	int this_cpu;
	int err;

	/*
	 * Prevent preemption and reschedule on another CPU, as well as CPU
	 * removal. This prevents stopper from running on this CPU, thus
	 * providing mutual exclusion of the below cpu_online() check and
	 * IPI sending ensuring IPI are not missed by CPU going offline.
	 */
	this_cpu = get_cpu();

	/*
	 * Can deadlock when called with interrupts disabled.
	 * We allow cpu's that are not yet online though, as no one else can
	 * send smp call function interrupt to this cpu and as such deadlocks
	 * can't happen.
	 */
	WARN_ON_ONCE(cpu_online(this_cpu) && irqs_disabled()
		     && !oops_in_progress);

	/*
	 * When @wait we can deadlock when we interrupt between llist_add() and
	 * arch_send_call_function_ipi*(); when !@wait we can deadlock due to
	 * csd_lock() on because the interrupt context uses the same csd
	 * storage.
	 */
	WARN_ON_ONCE(!in_task());

	csd = &csd_stack;
	if (!wait) {
		csd = this_cpu_ptr(&csd_data);
		csd_lock(csd);
	}

	csd->func = func;
	csd->info = info;
#ifdef CONFIG_CSD_LOCK_WAIT_DEBUG
	csd->node.src = smp_processor_id();
	csd->node.dst = cpu;
#endif

	err = generic_exec_single(cpu, csd);

	if (wait)
		csd_lock_wait(csd);

	put_cpu();

	return err;
}
EXPORT_SYMBOL(smp_call_function_single);

/**
 * smp_call_function_single_async() - Run an asynchronous function on a
 * 			         specific CPU.
 * @cpu: The CPU to run on.
 * @csd: Pre-allocated and setup data structure
 *
 * Like smp_call_function_single(), but the call is asynchonous and
 * can thus be done from contexts with disabled interrupts.
 *
 * The caller passes his own pre-allocated data structure
 * (ie: embedded in an object) and is responsible for synchronizing it
 * such that the IPIs performed on the @csd are strictly serialized.
 *
 * If the function is called with one csd which has not yet been
 * processed by previous call to smp_call_function_single_async(), the
 * function will return immediately with -EBUSY showing that the csd
 * object is still in progress.
 *
 * NOTE: Be careful, there is unfortunately no current debugging facility to
 * validate the correctness of this serialization.
 *
 * Return: %0 on success or negative errno value on error
 */
int smp_call_function_single_async(int cpu, call_single_data_t *csd)
{
	int err = 0;

	preempt_disable();

	if (csd->node.u_flags & CSD_FLAG_LOCK) {
		err = -EBUSY;
		goto out;
	}

	csd->node.u_flags = CSD_FLAG_LOCK;
	smp_wmb();

	err = generic_exec_single(cpu, csd);

out:
	preempt_enable();

	return err;
}
EXPORT_SYMBOL_GPL(smp_call_function_single_async);

/*
 * smp_call_function_any - Run a function on any of the given cpus
 * @mask: The mask of cpus it can run on.
 * @func: The function to run. This must be fast and non-blocking.
 * @info: An arbitrary pointer to pass to the function.
 * @wait: If true, wait until function has completed.
 *
 * Returns 0 on success, else a negative status code (if no cpus were online).
 *
 * Selection preference:
 *	1) current cpu if in @mask
 *	2) nearest cpu in @mask, based on NUMA topology
 */
int smp_call_function_any(const struct cpumask *mask,
			  smp_call_func_t func, void *info, int wait)
{
	unsigned int cpu;
	int ret;

	/* Try for same CPU (cheapest) */
	cpu = get_cpu();
	if (!cpumask_test_cpu(cpu, mask))
		cpu = sched_numa_find_nth_cpu(mask, 0, cpu_to_node(cpu));

	ret = smp_call_function_single(cpu, func, info, wait);
	put_cpu();
	return ret;
}
EXPORT_SYMBOL_GPL(smp_call_function_any);

/*
 * Flags to be used as scf_flags argument of smp_call_function_many_cond().
 *
 * %SCF_WAIT:		Wait until function execution is completed
 * %SCF_RUN_LOCAL:	Run also locally if local cpu is set in cpumask
 */
#define SCF_WAIT	(1U << 0)
#define SCF_RUN_LOCAL	(1U << 1)

static void smp_call_function_many_cond(const struct cpumask *mask,
					smp_call_func_t func, void *info,
					unsigned int scf_flags,
					smp_cond_func_t cond_func)
{
	int cpu, last_cpu, this_cpu = smp_processor_id();
	struct call_function_data *cfd;
	bool wait = scf_flags & SCF_WAIT;
	int nr_cpus = 0;
	bool run_remote = false;

	lockdep_assert_preemption_disabled();

	/*
	 * Can deadlock when called with interrupts disabled.
	 * We allow cpu's that are not yet online though, as no one else can
	 * send smp call function interrupt to this cpu and as such deadlocks
	 * can't happen.
	 */
	if (cpu_online(this_cpu) && !oops_in_progress &&
	    !early_boot_irqs_disabled)
		lockdep_assert_irqs_enabled();

	/*
	 * When @wait we can deadlock when we interrupt between llist_add() and
	 * arch_send_call_function_ipi*(); when !@wait we can deadlock due to
	 * csd_lock() on because the interrupt context uses the same csd
	 * storage.
	 */
	WARN_ON_ONCE(!in_task());

	/* Check if we need remote execution, i.e., any CPU excluding this one. */
	if (cpumask_any_and_but(mask, cpu_online_mask, this_cpu) < nr_cpu_ids) {
		cfd = this_cpu_ptr(&cfd_data);
		cpumask_and(cfd->cpumask, mask, cpu_online_mask);
		__cpumask_clear_cpu(this_cpu, cfd->cpumask);

		cpumask_clear(cfd->cpumask_ipi);
		for_each_cpu(cpu, cfd->cpumask) {
			call_single_data_t *csd = per_cpu_ptr(cfd->csd, cpu);

			if (cond_func && !cond_func(cpu, info)) {
				__cpumask_clear_cpu(cpu, cfd->cpumask);
				continue;
			}

			/* Work is enqueued on a remote CPU. */
			run_remote = true;

			csd_lock(csd);
			if (wait)
				csd->node.u_flags |= CSD_TYPE_SYNC;
			csd->func = func;
			csd->info = info;
#ifdef CONFIG_CSD_LOCK_WAIT_DEBUG
			csd->node.src = smp_processor_id();
			csd->node.dst = cpu;
#endif
			trace_csd_queue_cpu(cpu, _RET_IP_, func, csd);

			/*
			 * Kick the remote CPU if this is the first work
			 * item enqueued.
			 */
			if (llist_add(&csd->node.llist, &per_cpu(call_single_queue, cpu))) {
				__cpumask_set_cpu(cpu, cfd->cpumask_ipi);
				nr_cpus++;
				last_cpu = cpu;
			}
		}

		/*
		 * Choose the most efficient way to send an IPI. Note that the
		 * number of CPUs might be zero due to concurrent changes to the
		 * provided mask.
		 */
		if (nr_cpus == 1)
			send_call_function_single_ipi(last_cpu);
		else if (likely(nr_cpus > 1))
			send_call_function_ipi_mask(cfd->cpumask_ipi);
	}

	/* Check if we need local execution. */
	if ((scf_flags & SCF_RUN_LOCAL) && cpumask_test_cpu(this_cpu, mask) &&
	    (!cond_func || cond_func(this_cpu, info))) {
		unsigned long flags;

		local_irq_save(flags);
		csd_do_func(func, info, NULL);
		local_irq_restore(flags);
	}

	if (run_remote && wait) {
		for_each_cpu(cpu, cfd->cpumask) {
			call_single_data_t *csd;

			csd = per_cpu_ptr(cfd->csd, cpu);
			csd_lock_wait(csd);
		}
	}
}

/**
 * smp_call_function_many(): Run a function on a set of CPUs.
 * @mask: The set of cpus to run on (only runs on online subset).
 * @func: The function to run. This must be fast and non-blocking.
 * @info: An arbitrary pointer to pass to the function.
 * @wait: If true, wait (atomically) until function has completed
 *        on other CPUs.
 *
 * You must not call this function with disabled interrupts or from a
 * hardware interrupt handler or from a bottom half handler. Preemption
 * must be disabled when calling this function.
 *
 * @func is not called on the local CPU even if @mask contains it.  Consider
 * using on_each_cpu_cond_mask() instead if this is not desirable.
 */
void smp_call_function_many(const struct cpumask *mask,
			    smp_call_func_t func, void *info, bool wait)
{
	smp_call_function_many_cond(mask, func, info, wait * SCF_WAIT, NULL);
}
EXPORT_SYMBOL(smp_call_function_many);

/**
 * smp_call_function(): Run a function on all other CPUs.
 * @func: The function to run. This must be fast and non-blocking.
 * @info: An arbitrary pointer to pass to the function.
 * @wait: If true, wait (atomically) until function has completed
 *        on other CPUs.
 *
 * Returns 0.
 *
 * If @wait is true, then returns once @func has returned; otherwise
 * it returns just before the target cpu calls @func.
 *
 * You must not call this function with disabled interrupts or from a
 * hardware interrupt handler or from a bottom half handler.
 */
void smp_call_function(smp_call_func_t func, void *info, int wait)
{
	preempt_disable();
	smp_call_function_many(cpu_online_mask, func, info, wait);
	preempt_enable();
}
EXPORT_SYMBOL(smp_call_function);

/* Setup configured maximum number of CPUs to activate */
unsigned int setup_max_cpus = NR_CPUS;
EXPORT_SYMBOL(setup_max_cpus);


/*
 * Setup routine for controlling SMP activation
 *
 * Command-line option of "nosmp" or "maxcpus=0" will disable SMP
 * activation entirely (the MPS table probe still happens, though).
 *
 * Command-line option of "maxcpus=<NUM>", where <NUM> is an integer
 * greater than 0, limits the maximum number of CPUs activated in
 * SMP mode to <NUM>.
 */

void __weak __init arch_disable_smp_support(void) { }

static int __init nosmp(char *str)
{
	setup_max_cpus = 0;
	arch_disable_smp_support();

	return 0;
}

early_param("nosmp", nosmp);

/* this is hard limit */
static int __init nrcpus(char *str)
{
	int nr_cpus;

	if (get_option(&str, &nr_cpus) && nr_cpus > 0 && nr_cpus < nr_cpu_ids)
		set_nr_cpu_ids(nr_cpus);

	return 0;
}

early_param("nr_cpus", nrcpus);

static int __init maxcpus(char *str)
{
	get_option(&str, &setup_max_cpus);
	if (setup_max_cpus == 0)
		arch_disable_smp_support();

	return 0;
}

early_param("maxcpus", maxcpus);

#if (NR_CPUS > 1) && !defined(CONFIG_FORCE_NR_CPUS)
/* Setup number of possible processor ids */
unsigned int nr_cpu_ids __read_mostly = NR_CPUS;
EXPORT_SYMBOL(nr_cpu_ids);
#endif

/* An arch may set nr_cpu_ids earlier if needed, so this would be redundant */
void __init setup_nr_cpu_ids(void)
{
	set_nr_cpu_ids(find_last_bit(cpumask_bits(cpu_possible_mask), NR_CPUS) + 1);
}

/* Called by boot processor to activate the rest. */
void __init smp_init(void)
{
	int num_nodes, num_cpus;

	idle_threads_init();
	cpuhp_threads_init();

	pr_info("Bringing up secondary CPUs ...\n");

	bringup_nonboot_cpus(setup_max_cpus);

	num_nodes = num_online_nodes();
	num_cpus  = num_online_cpus();
	pr_info("Brought up %d node%s, %d CPU%s\n",
		num_nodes, str_plural(num_nodes), num_cpus, str_plural(num_cpus));

	/* Any cleanup work */
	smp_cpus_done(setup_max_cpus);
}

/*
 * on_each_cpu_cond(): Call a function on each processor for which
 * the supplied function cond_func returns true, optionally waiting
 * for all the required CPUs to finish. This may include the local
 * processor.
 * @cond_func:	A callback function that is passed a cpu id and
 *		the info parameter. The function is called
 *		with preemption disabled. The function should
 *		return a boolean value indicating whether to IPI
 *		the specified CPU.
 * @func:	The function to run on all applicable CPUs.
 *		This must be fast and non-blocking.
 * @info:	An arbitrary pointer to pass to both functions.
 * @wait:	If true, wait (atomically) until function has
 *		completed on other CPUs.
 *
 * Preemption is disabled to protect against CPUs going offline but not online.
 * CPUs going online during the call will not be seen or sent an IPI.
 *
 * You must not call this function with disabled interrupts or
 * from a hardware interrupt handler or from a bottom half handler.
 */
void on_each_cpu_cond_mask(smp_cond_func_t cond_func, smp_call_func_t func,
			   void *info, bool wait, const struct cpumask *mask)
{
	unsigned int scf_flags = SCF_RUN_LOCAL;

	if (wait)
		scf_flags |= SCF_WAIT;

	preempt_disable();
	smp_call_function_many_cond(mask, func, info, scf_flags, cond_func);
	preempt_enable();
}
EXPORT_SYMBOL(on_each_cpu_cond_mask);

static void do_nothing(void *unused)
{
}

/**
 * kick_all_cpus_sync - Force all cpus out of idle
 *
 * Used to synchronize the update of pm_idle function pointer. It's
 * called after the pointer is updated and returns after the dummy
 * callback function has been executed on all cpus. The execution of
 * the function can only happen on the remote cpus after they have
 * left the idle function which had been called via pm_idle function
 * pointer. So it's guaranteed that nothing uses the previous pointer
 * anymore.
 */
void kick_all_cpus_sync(void)
{
	/* Make sure the change is visible before we kick the cpus */
	smp_mb();
	smp_call_function(do_nothing, NULL, 1);
}
EXPORT_SYMBOL_GPL(kick_all_cpus_sync);

/**
 * wake_up_all_idle_cpus - break all cpus out of idle
 * wake_up_all_idle_cpus try to break all cpus which is in idle state even
 * including idle polling cpus, for non-idle cpus, we will do nothing
 * for them.
 */
void wake_up_all_idle_cpus(void)
{
	int cpu;

	for_each_possible_cpu(cpu) {
		preempt_disable();
		if (cpu != smp_processor_id() && cpu_online(cpu))
			wake_up_if_idle(cpu);
		preempt_enable();
	}
}
EXPORT_SYMBOL_GPL(wake_up_all_idle_cpus);

/**
 * cpus_peek_for_pending_ipi - Check for pending IPI for CPUs
 * @mask: The CPU mask for the CPUs to check.
 *
 * This function walks through the @mask to check if there are any pending IPIs
 * scheduled, for any of the CPUs in the @mask. It does not guarantee
 * correctness as it only provides a racy snapshot.
 *
 * Returns true if there is a pending IPI scheduled and false otherwise.
 */
bool cpus_peek_for_pending_ipi(const struct cpumask *mask)
{
	unsigned int cpu;

	for_each_cpu(cpu, mask) {
		if (!llist_empty(per_cpu_ptr(&call_single_queue, cpu)))
			return true;
	}

	return false;
}

/**
 * struct smp_call_on_cpu_struct - Call a function on a specific CPU
 * @work: &work_struct
 * @done: &completion to signal
 * @func: function to call
 * @data: function's data argument
 * @ret: return value from @func
 * @cpu: target CPU (%-1 for any CPU)
 *
 * Used to call a function on a specific cpu and wait for it to return.
 * Optionally make sure the call is done on a specified physical cpu via vcpu
 * pinning in order to support virtualized environments.
 */
struct smp_call_on_cpu_struct {
	struct work_struct	work;
	struct completion	done;
	int			(*func)(void *);
	void			*data;
	int			ret;
	int			cpu;
};

static void smp_call_on_cpu_callback(struct work_struct *work)
{
	struct smp_call_on_cpu_struct *sscs;

	sscs = container_of(work, struct smp_call_on_cpu_struct, work);
	if (sscs->cpu >= 0)
		hypervisor_pin_vcpu(sscs->cpu);
	sscs->ret = sscs->func(sscs->data);
	if (sscs->cpu >= 0)
		hypervisor_pin_vcpu(-1);

	complete(&sscs->done);
}

int smp_call_on_cpu(unsigned int cpu, int (*func)(void *), void *par, bool phys)
{
	struct smp_call_on_cpu_struct sscs = {
		.done = COMPLETION_INITIALIZER_ONSTACK(sscs.done),
		.func = func,
		.data = par,
		.cpu  = phys ? cpu : -1,
	};

	INIT_WORK_ONSTACK(&sscs.work, smp_call_on_cpu_callback);

	if (cpu >= nr_cpu_ids || !cpu_online(cpu))
		return -ENXIO;

	queue_work_on(cpu, system_wq, &sscs.work);
	wait_for_completion(&sscs.done);
	destroy_work_on_stack(&sscs.work);

	return sscs.ret;
}
EXPORT_SYMBOL_GPL(smp_call_on_cpu);
]

gemini-3-flash-preview

							input: 49076
							output: 21
							thoughts: 4236
						

**Analyzing the Race Condition in Buffer Head LRU Invalidation**

Okay, so I've been diving into this `smp_call_function_many_cond` function and how it interacts with the buffer head LRU invalidation process. It's calling `has_bh_in_lru` on multiple CPUs, which is a key part of `invalidate_bh_lrus`. Essentially, we're trying to figure out if there's a real data race here, and if it's harmful.

`has_bh_in_lru` is checking if a given CPU's LRU contains any buffer heads, and the potential conflict arises because `__invalidate_bh_lrus` is writing to these LRUs simultaneously. The lock used in `invalidate_bh_lrus_cpu` only provides local protection, which is the source of the race.

The concern is that `has_bh_in_lru` might read a partially-written buffer head pointer in `b->bhs[i]`, or, more subtly, return an incorrect value indicating the LRU is empty when it's not, or vice versa. The writes are all about zeroing out buffer heads.

Initially, I considered the possibility of a "torn" pointer read, but since we're dealing with pointer-sized reads and writes on a 64-bit system, these operations should be atomic. I verified with the KCSAN report that the write size is indeed 8 bytes, meaning atomicity is likely.

The primary race seems to be between `has_bh_in_lru` reading the LRU and `__invalidate_bh_lrus` clearing it. This could lead to a missed invalidation if `has_bh_in_lru` returns `false` (LRU is empty) when there are actually valid entries.

I'm focusing now on the scenarios where `has_bh_in_lru` might incorrectly return `false`. This would only happen if, during its scan, it encounters only `NULL` values. This is where I started to check the other places where `bhs[]` is written to (like `bh_lru_install` and `lookup_bh_lru`), which use locks, but the `has_bh_in_lru` function itself doesn't.

`bh_lru_install` uses `swap` which is *not* atomic! It has to copy the new value into place and copy the old value out. My understanding of the race condition is that this could lead to the following scenario: `has_bh_in_lru` has scanned up to `bhs[j]`, which is empty, while `bh_lru_install` has inserted something at index `0` and shifted some entries, so now the buffer head is actually in `bhs[0]`, while other entries are `NULL`.

If `has_bh_in_lru` returns `false` even though entries are in the LRU, we might miss the invalidation. Now, I have to consider the call sites of this function and if missing an invalidation has serious consequences.

`invalidate_bh_lrus` is called by `invalidate_bdev` which calls `invalidate_mapping_pages`. So, it's called during unmount, or when freeing buffers.

The worst-case scenario is if `has_bh_in_lru` returns `false` when there's an actual entry, and that's the thing that's holding a reference to a buffer head that we *need* to invalidate.

Now, my line of thought is as follows: If `has_bh_in_lru` incorrectly returns false, that means it must have encountered all `NULL` entries. Even if the content of the buffer head is updated later, that entry should be gone since we are invalidating.

My conclusion is that race is there but the current analysis indicates it's likely benign due to the way `invalidate_bh_lrus` is used. However, I still need to verify the code path to ensure this assumption is always valid. Also I will search the codebase to identify more scenarios where `has_bh_in_lru` is used to make sure the conclusions are always accurate.

map[File:block/bdev.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
 *  Copyright (C) 1991, 1992  Linus Torvalds
 *  Copyright (C) 2001  Andrea Arcangeli <andrea@suse.de> SuSE
 *  Copyright (C) 2016 - 2020 Christoph Hellwig
 */

#include <linux/init.h>
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/kmod.h>
#include <linux/major.h>
#include <linux/device_cgroup.h>
#include <linux/blkdev.h>
#include <linux/blk-integrity.h>
#include <linux/backing-dev.h>
#include <linux/module.h>
#include <linux/blkpg.h>
#include <linux/magic.h>
#include <linux/buffer_head.h>
#include <linux/swap.h>
#include <linux/writeback.h>
#include <linux/mount.h>
#include <linux/pseudo_fs.h>
#include <linux/uio.h>
#include <linux/namei.h>
#include <linux/security.h>
#include <linux/part_stat.h>
#include <linux/uaccess.h>
#include <linux/stat.h>
#include "../fs/internal.h"
#include "blk.h"

/* Should we allow writing to mounted block devices? */
static bool bdev_allow_write_mounted = IS_ENABLED(CONFIG_BLK_DEV_WRITE_MOUNTED);

struct bdev_inode {
	struct block_device bdev;
	struct inode vfs_inode;
};

static inline struct bdev_inode *BDEV_I(struct inode *inode)
{
	return container_of(inode, struct bdev_inode, vfs_inode);
}

static inline struct inode *BD_INODE(struct block_device *bdev)
{
	return &container_of(bdev, struct bdev_inode, bdev)->vfs_inode;
}

struct block_device *I_BDEV(struct inode *inode)
{
	return &BDEV_I(inode)->bdev;
}
EXPORT_SYMBOL(I_BDEV);

struct block_device *file_bdev(struct file *bdev_file)
{
	return I_BDEV(bdev_file->f_mapping->host);
}
EXPORT_SYMBOL(file_bdev);

static void bdev_write_inode(struct block_device *bdev)
{
	struct inode *inode = BD_INODE(bdev);
	int ret;

	spin_lock(&inode->i_lock);
	while (inode_state_read(inode) & I_DIRTY) {
		spin_unlock(&inode->i_lock);
		ret = write_inode_now(inode, true);
		if (ret)
			pr_warn_ratelimited(
	"VFS: Dirty inode writeback failed for block device %pg (err=%d).\n",
				bdev, ret);
		spin_lock(&inode->i_lock);
	}
	spin_unlock(&inode->i_lock);
}

/* Kill _all_ buffers and pagecache , dirty or not.. */
static void kill_bdev(struct block_device *bdev)
{
	struct address_space *mapping = bdev->bd_mapping;

	if (mapping_empty(mapping))
		return;

	invalidate_bh_lrus();
	truncate_inode_pages(mapping, 0);
}

/* Invalidate clean unused buffers and pagecache. */
void invalidate_bdev(struct block_device *bdev)
{
	struct address_space *mapping = bdev->bd_mapping;

	if (mapping->nrpages) {
		invalidate_bh_lrus();
		lru_add_drain_all();	/* make sure all lru add caches are flushed */
		invalidate_mapping_pages(mapping, 0, -1);
	}
}
EXPORT_SYMBOL(invalidate_bdev);

/*
 * Drop all buffers & page cache for given bdev range. This function bails
 * with error if bdev has other exclusive owner (such as filesystem).
 */
int truncate_bdev_range(struct block_device *bdev, blk_mode_t mode,
			loff_t lstart, loff_t lend)
{
	/*
	 * If we don't hold exclusive handle for the device, upgrade to it
	 * while we discard the buffer cache to avoid discarding buffers
	 * under live filesystem.
	 */
	if (!(mode & BLK_OPEN_EXCL)) {
		int err = bd_prepare_to_claim(bdev, truncate_bdev_range, NULL);
		if (err)
			goto invalidate;
	}

	truncate_inode_pages_range(bdev->bd_mapping, lstart, lend);
	if (!(mode & BLK_OPEN_EXCL))
		bd_abort_claiming(bdev, truncate_bdev_range);
	return 0;

invalidate:
	/*
	 * Someone else has handle exclusively open. Try invalidating instead.
	 * The 'end' argument is inclusive so the rounding is safe.
	 */
	return invalidate_inode_pages2_range(bdev->bd_mapping,
					     lstart >> PAGE_SHIFT,
					     lend >> PAGE_SHIFT);
}

static void set_init_blocksize(struct block_device *bdev)
{
	unsigned int bsize = bdev_logical_block_size(bdev);
	loff_t size = i_size_read(BD_INODE(bdev));

	while (bsize < PAGE_SIZE) {
		if (size & bsize)
			break;
		bsize <<= 1;
	}
	BD_INODE(bdev)->i_blkbits = blksize_bits(bsize);
	mapping_set_folio_min_order(BD_INODE(bdev)->i_mapping,
				    get_order(bsize));
}

/**
 * bdev_validate_blocksize - check that this block size is acceptable
 * @bdev:	blockdevice to check
 * @block_size:	block size to check
 *
 * For block device users that do not use buffer heads or the block device
 * page cache, make sure that this block size can be used with the device.
 *
 * Return: On success zero is returned, negative error code on failure.
 */
int bdev_validate_blocksize(struct block_device *bdev, int block_size)
{
	if (blk_validate_block_size(block_size))
		return -EINVAL;

	/* Size cannot be smaller than the size supported by the device */
	if (block_size < bdev_logical_block_size(bdev))
		return -EINVAL;

	return 0;
}
EXPORT_SYMBOL_GPL(bdev_validate_blocksize);

int set_blocksize(struct file *file, int size)
{
	struct inode *inode = file->f_mapping->host;
	struct block_device *bdev = I_BDEV(inode);
	int ret;

	ret = bdev_validate_blocksize(bdev, size);
	if (ret)
		return ret;

	if (!file->private_data)
		return -EINVAL;

	/* Don't change the size if it is same as current */
	if (inode->i_blkbits != blksize_bits(size)) {
		/*
		 * Flush and truncate the pagecache before we reconfigure the
		 * mapping geometry because folio sizes are variable now.  If a
		 * reader has already allocated a folio whose size is smaller
		 * than the new min_order but invokes readahead after the new
		 * min_order becomes visible, readahead will think there are
		 * "zero" blocks per folio and crash.  Take the inode and
		 * invalidation locks to avoid racing with
		 * read/write/fallocate.
		 */
		inode_lock(inode);
		filemap_invalidate_lock(inode->i_mapping);

		sync_blockdev(bdev);
		kill_bdev(bdev);

		inode->i_blkbits = blksize_bits(size);
		mapping_set_folio_min_order(inode->i_mapping, get_order(size));
		filemap_invalidate_unlock(inode->i_mapping);
		inode_unlock(inode);
	}
	return 0;
}

EXPORT_SYMBOL(set_blocksize);

static int sb_validate_large_blocksize(struct super_block *sb, int size)
{
	const char *err_str = NULL;

	if (!(sb->s_type->fs_flags & FS_LBS))
		err_str = "not supported by filesystem";
	else if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
		err_str = "is only supported with CONFIG_TRANSPARENT_HUGEPAGE";

	if (!err_str)
		return 0;

	pr_warn_ratelimited("%s: block size(%d) > page size(%lu) %s\n",
				sb->s_type->name, size, PAGE_SIZE, err_str);
	return -EINVAL;
}

int sb_set_blocksize(struct super_block *sb, int size)
{
	if (size > PAGE_SIZE && sb_validate_large_blocksize(sb, size))
		return 0;
	if (set_blocksize(sb->s_bdev_file, size))
		return 0;
	/* If we get here, we know size is validated */
	sb->s_blocksize = size;
	sb->s_blocksize_bits = blksize_bits(size);
	return sb->s_blocksize;
}

EXPORT_SYMBOL(sb_set_blocksize);

int __must_check sb_min_blocksize(struct super_block *sb, int size)
{
	int minsize = bdev_logical_block_size(sb->s_bdev);
	if (size < minsize)
		size = minsize;
	return sb_set_blocksize(sb, size);
}

EXPORT_SYMBOL(sb_min_blocksize);

int sync_blockdev_nowait(struct block_device *bdev)
{
	if (!bdev)
		return 0;
	return filemap_flush(bdev->bd_mapping);
}
EXPORT_SYMBOL_GPL(sync_blockdev_nowait);

/*
 * Write out and wait upon all the dirty data associated with a block
 * device via its mapping.  Does not take the superblock lock.
 */
int sync_blockdev(struct block_device *bdev)
{
	if (!bdev)
		return 0;
	return filemap_write_and_wait(bdev->bd_mapping);
}
EXPORT_SYMBOL(sync_blockdev);

int sync_blockdev_range(struct block_device *bdev, loff_t lstart, loff_t lend)
{
	return filemap_write_and_wait_range(bdev->bd_mapping,
			lstart, lend);
}
EXPORT_SYMBOL(sync_blockdev_range);

/**
 * bdev_freeze - lock a filesystem and force it into a consistent state
 * @bdev:	blockdevice to lock
 *
 * If a superblock is found on this device, we take the s_umount semaphore
 * on it to make sure nobody unmounts until the snapshot creation is done.
 * The reference counter (bd_fsfreeze_count) guarantees that only the last
 * unfreeze process can unfreeze the frozen filesystem actually when multiple
 * freeze requests arrive simultaneously. It counts up in bdev_freeze() and
 * count down in bdev_thaw(). When it becomes 0, thaw_bdev() will unfreeze
 * actually.
 *
 * Return: On success zero is returned, negative error code on failure.
 */
int bdev_freeze(struct block_device *bdev)
{
	int error = 0;

	mutex_lock(&bdev->bd_fsfreeze_mutex);

	if (atomic_inc_return(&bdev->bd_fsfreeze_count) > 1) {
		mutex_unlock(&bdev->bd_fsfreeze_mutex);
		return 0;
	}

	mutex_lock(&bdev->bd_holder_lock);
	if (bdev->bd_holder_ops && bdev->bd_holder_ops->freeze) {
		error = bdev->bd_holder_ops->freeze(bdev);
		lockdep_assert_not_held(&bdev->bd_holder_lock);
	} else {
		mutex_unlock(&bdev->bd_holder_lock);
		error = sync_blockdev(bdev);
	}

	if (error)
		atomic_dec(&bdev->bd_fsfreeze_count);

	mutex_unlock(&bdev->bd_fsfreeze_mutex);
	return error;
}
EXPORT_SYMBOL(bdev_freeze);

/**
 * bdev_thaw - unlock filesystem
 * @bdev:	blockdevice to unlock
 *
 * Unlocks the filesystem and marks it writeable again after bdev_freeze().
 *
 * Return: On success zero is returned, negative error code on failure.
 */
int bdev_thaw(struct block_device *bdev)
{
	int error = -EINVAL, nr_freeze;

	mutex_lock(&bdev->bd_fsfreeze_mutex);

	/*
	 * If this returns < 0 it means that @bd_fsfreeze_count was
	 * already 0 and no decrement was performed.
	 */
	nr_freeze = atomic_dec_if_positive(&bdev->bd_fsfreeze_count);
	if (nr_freeze < 0)
		goto out;

	error = 0;
	if (nr_freeze > 0)
		goto out;

	mutex_lock(&bdev->bd_holder_lock);
	if (bdev->bd_holder_ops && bdev->bd_holder_ops->thaw) {
		error = bdev->bd_holder_ops->thaw(bdev);
		lockdep_assert_not_held(&bdev->bd_holder_lock);
	} else {
		mutex_unlock(&bdev->bd_holder_lock);
	}

	if (error)
		atomic_inc(&bdev->bd_fsfreeze_count);
out:
	mutex_unlock(&bdev->bd_fsfreeze_mutex);
	return error;
}
EXPORT_SYMBOL(bdev_thaw);

/*
 * pseudo-fs
 */

static  __cacheline_aligned_in_smp DEFINE_MUTEX(bdev_lock);
static struct kmem_cache *bdev_cachep __ro_after_init;

static struct inode *bdev_alloc_inode(struct super_block *sb)
{
	struct bdev_inode *ei = alloc_inode_sb(sb, bdev_cachep, GFP_KERNEL);

	if (!ei)
		return NULL;
	memset(&ei->bdev, 0, sizeof(ei->bdev));

	if (security_bdev_alloc(&ei->bdev)) {
		kmem_cache_free(bdev_cachep, ei);
		return NULL;
	}
	return &ei->vfs_inode;
}

static void bdev_free_inode(struct inode *inode)
{
	struct block_device *bdev = I_BDEV(inode);

	free_percpu(bdev->bd_stats);
	kfree(bdev->bd_meta_info);
	security_bdev_free(bdev);

	if (!bdev_is_partition(bdev)) {
		if (bdev->bd_disk && bdev->bd_disk->bdi)
			bdi_put(bdev->bd_disk->bdi);
		kfree(bdev->bd_disk);
	}

	if (MAJOR(bdev->bd_dev) == BLOCK_EXT_MAJOR)
		blk_free_ext_minor(MINOR(bdev->bd_dev));

	kmem_cache_free(bdev_cachep, BDEV_I(inode));
}

static void init_once(void *data)
{
	struct bdev_inode *ei = data;

	inode_init_once(&ei->vfs_inode);
}

static void bdev_evict_inode(struct inode *inode)
{
	truncate_inode_pages_final(&inode->i_data);
	invalidate_inode_buffers(inode); /* is it needed here? */
	clear_inode(inode);
}

static const struct super_operations bdev_sops = {
	.statfs = simple_statfs,
	.alloc_inode = bdev_alloc_inode,
	.free_inode = bdev_free_inode,
	.drop_inode = inode_just_drop,
	.evict_inode = bdev_evict_inode,
};

static int bd_init_fs_context(struct fs_context *fc)
{
	struct pseudo_fs_context *ctx = init_pseudo(fc, BDEVFS_MAGIC);
	if (!ctx)
		return -ENOMEM;
	fc->s_iflags |= SB_I_CGROUPWB;
	ctx->ops = &bdev_sops;
	return 0;
}

static struct file_system_type bd_type = {
	.name		= "bdev",
	.init_fs_context = bd_init_fs_context,
	.kill_sb	= kill_anon_super,
};

struct super_block *blockdev_superblock __ro_after_init;
static struct vfsmount *blockdev_mnt __ro_after_init;
EXPORT_SYMBOL_GPL(blockdev_superblock);

void __init bdev_cache_init(void)
{
	int err;

	bdev_cachep = kmem_cache_create("bdev_cache", sizeof(struct bdev_inode),
			0, (SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|
				SLAB_ACCOUNT|SLAB_PANIC),
			init_once);
	err = register_filesystem(&bd_type);
	if (err)
		panic("Cannot register bdev pseudo-fs");
	blockdev_mnt = kern_mount(&bd_type);
	if (IS_ERR(blockdev_mnt))
		panic("Cannot create bdev pseudo-fs");
	blockdev_superblock = blockdev_mnt->mnt_sb;   /* For writeback */
}

struct block_device *bdev_alloc(struct gendisk *disk, u8 partno)
{
	struct block_device *bdev;
	struct inode *inode;

	inode = new_inode(blockdev_superblock);
	if (!inode)
		return NULL;
	inode->i_mode = S_IFBLK;
	inode->i_rdev = 0;
	inode->i_data.a_ops = &def_blk_aops;
	mapping_set_gfp_mask(&inode->i_data, GFP_USER);

	bdev = I_BDEV(inode);
	mutex_init(&bdev->bd_fsfreeze_mutex);
	spin_lock_init(&bdev->bd_size_lock);
	mutex_init(&bdev->bd_holder_lock);
	atomic_set(&bdev->__bd_flags, partno);
	bdev->bd_mapping = &inode->i_data;
	bdev->bd_queue = disk->queue;
	if (partno && bdev_test_flag(disk->part0, BD_HAS_SUBMIT_BIO))
		bdev_set_flag(bdev, BD_HAS_SUBMIT_BIO);
	bdev->bd_stats = alloc_percpu(struct disk_stats);
	if (!bdev->bd_stats) {
		iput(inode);
		return NULL;
	}
	bdev->bd_disk = disk;
	return bdev;
}

void bdev_set_nr_sectors(struct block_device *bdev, sector_t sectors)
{
	spin_lock(&bdev->bd_size_lock);
	i_size_write(BD_INODE(bdev), (loff_t)sectors << SECTOR_SHIFT);
	bdev->bd_nr_sectors = sectors;
	spin_unlock(&bdev->bd_size_lock);
}

void bdev_add(struct block_device *bdev, dev_t dev)
{
	struct inode *inode = BD_INODE(bdev);
	if (bdev_stable_writes(bdev))
		mapping_set_stable_writes(bdev->bd_mapping);
	bdev->bd_dev = dev;
	inode->i_rdev = dev;
	inode->i_ino = dev;
	insert_inode_hash(inode);
}

void bdev_unhash(struct block_device *bdev)
{
	remove_inode_hash(BD_INODE(bdev));
}

void bdev_drop(struct block_device *bdev)
{
	iput(BD_INODE(bdev));
}

long nr_blockdev_pages(void)
{
	struct inode *inode;
	long ret = 0;

	spin_lock(&blockdev_superblock->s_inode_list_lock);
	list_for_each_entry(inode, &blockdev_superblock->s_inodes, i_sb_list)
		ret += inode->i_mapping->nrpages;
	spin_unlock(&blockdev_superblock->s_inode_list_lock);

	return ret;
}

/**
 * bd_may_claim - test whether a block device can be claimed
 * @bdev: block device of interest
 * @holder: holder trying to claim @bdev
 * @hops: holder ops
 *
 * Test whether @bdev can be claimed by @holder.
 *
 * RETURNS:
 * %true if @bdev can be claimed, %false otherwise.
 */
static bool bd_may_claim(struct block_device *bdev, void *holder,
		const struct blk_holder_ops *hops)
{
	struct block_device *whole = bdev_whole(bdev);

	lockdep_assert_held(&bdev_lock);

	if (bdev->bd_holder) {
		/*
		 * The same holder can always re-claim.
		 */
		if (bdev->bd_holder == holder) {
			if (WARN_ON_ONCE(bdev->bd_holder_ops != hops))
				return false;
			return true;
		}
		return false;
	}

	/*
	 * If the whole devices holder is set to bd_may_claim, a partition on
	 * the device is claimed, but not the whole device.
	 */
	if (whole != bdev &&
	    whole->bd_holder && whole->bd_holder != bd_may_claim)
		return false;
	return true;
}

/**
 * bd_prepare_to_claim - claim a block device
 * @bdev: block device of interest
 * @holder: holder trying to claim @bdev
 * @hops: holder ops.
 *
 * Claim @bdev.  This function fails if @bdev is already claimed by another
 * holder and waits if another claiming is in progress. return, the caller
 * has ownership of bd_claiming and bd_holder[s].
 *
 * RETURNS:
 * 0 if @bdev can be claimed, -EBUSY otherwise.
 */
int bd_prepare_to_claim(struct block_device *bdev, void *holder,
		const struct blk_holder_ops *hops)
{
	struct block_device *whole = bdev_whole(bdev);

	if (WARN_ON_ONCE(!holder))
		return -EINVAL;
retry:
	mutex_lock(&bdev_lock);
	/* if someone else claimed, fail */
	if (!bd_may_claim(bdev, holder, hops)) {
		mutex_unlock(&bdev_lock);
		return -EBUSY;
	}

	/* if claiming is already in progress, wait for it to finish */
	if (whole->bd_claiming) {
		wait_queue_head_t *wq = __var_waitqueue(&whole->bd_claiming);
		DEFINE_WAIT(wait);

		prepare_to_wait(wq, &wait, TASK_UNINTERRUPTIBLE);
		mutex_unlock(&bdev_lock);
		schedule();
		finish_wait(wq, &wait);
		goto retry;
	}

	/* yay, all mine */
	whole->bd_claiming = holder;
	mutex_unlock(&bdev_lock);
	return 0;
}
EXPORT_SYMBOL_GPL(bd_prepare_to_claim); /* only for the loop driver */

static void bd_clear_claiming(struct block_device *whole, void *holder)
{
	lockdep_assert_held(&bdev_lock);
	/* tell others that we're done */
	BUG_ON(whole->bd_claiming != holder);
	whole->bd_claiming = NULL;
	wake_up_var(&whole->bd_claiming);
}

/**
 * bd_finish_claiming - finish claiming of a block device
 * @bdev: block device of interest
 * @holder: holder that has claimed @bdev
 * @hops: block device holder operations
 *
 * Finish exclusive open of a block device. Mark the device as exlusively
 * open by the holder and wake up all waiters for exclusive open to finish.
 */
static void bd_finish_claiming(struct block_device *bdev, void *holder,
		const struct blk_holder_ops *hops)
{
	struct block_device *whole = bdev_whole(bdev);

	mutex_lock(&bdev_lock);
	BUG_ON(!bd_may_claim(bdev, holder, hops));
	/*
	 * Note that for a whole device bd_holders will be incremented twice,
	 * and bd_holder will be set to bd_may_claim before being set to holder
	 */
	whole->bd_holders++;
	whole->bd_holder = bd_may_claim;
	bdev->bd_holders++;
	mutex_lock(&bdev->bd_holder_lock);
	bdev->bd_holder = holder;
	bdev->bd_holder_ops = hops;
	mutex_unlock(&bdev->bd_holder_lock);
	bd_clear_claiming(whole, holder);
	mutex_unlock(&bdev_lock);
}

/**
 * bd_abort_claiming - abort claiming of a block device
 * @bdev: block device of interest
 * @holder: holder that has claimed @bdev
 *
 * Abort claiming of a block device when the exclusive open failed. This can be
 * also used when exclusive open is not actually desired and we just needed
 * to block other exclusive openers for a while.
 */
void bd_abort_claiming(struct block_device *bdev, void *holder)
{
	mutex_lock(&bdev_lock);
	bd_clear_claiming(bdev_whole(bdev), holder);
	mutex_unlock(&bdev_lock);
}
EXPORT_SYMBOL(bd_abort_claiming);

static void bd_end_claim(struct block_device *bdev, void *holder)
{
	struct block_device *whole = bdev_whole(bdev);
	bool unblock = false;

	/*
	 * Release a claim on the device.  The holder fields are protected with
	 * bdev_lock.  open_mutex is used to synchronize disk_holder unlinking.
	 */
	mutex_lock(&bdev_lock);
	WARN_ON_ONCE(bdev->bd_holder != holder);
	WARN_ON_ONCE(--bdev->bd_holders < 0);
	WARN_ON_ONCE(--whole->bd_holders < 0);
	if (!bdev->bd_holders) {
		mutex_lock(&bdev->bd_holder_lock);
		bdev->bd_holder = NULL;
		bdev->bd_holder_ops = NULL;
		mutex_unlock(&bdev->bd_holder_lock);
		if (bdev_test_flag(bdev, BD_WRITE_HOLDER))
			unblock = true;
	}
	if (!whole->bd_holders)
		whole->bd_holder = NULL;
	mutex_unlock(&bdev_lock);

	/*
	 * If this was the last claim, remove holder link and unblock evpoll if
	 * it was a write holder.
	 */
	if (unblock) {
		disk_unblock_events(bdev->bd_disk);
		bdev_clear_flag(bdev, BD_WRITE_HOLDER);
	}
}

static void blkdev_flush_mapping(struct block_device *bdev)
{
	WARN_ON_ONCE(bdev->bd_holders);
	sync_blockdev(bdev);
	kill_bdev(bdev);
	bdev_write_inode(bdev);
}

static void blkdev_put_whole(struct block_device *bdev)
{
	if (atomic_dec_and_test(&bdev->bd_openers))
		blkdev_flush_mapping(bdev);
	if (bdev->bd_disk->fops->release)
		bdev->bd_disk->fops->release(bdev->bd_disk);
}

static int blkdev_get_whole(struct block_device *bdev, blk_mode_t mode)
{
	struct gendisk *disk = bdev->bd_disk;
	int ret;

	if (disk->fops->open) {
		ret = disk->fops->open(disk, mode);
		if (ret) {
			/* avoid ghost partitions on a removed medium */
			if (ret == -ENOMEDIUM &&
			     test_bit(GD_NEED_PART_SCAN, &disk->state))
				bdev_disk_changed(disk, true);
			return ret;
		}
	}

	if (!atomic_read(&bdev->bd_openers))
		set_init_blocksize(bdev);
	atomic_inc(&bdev->bd_openers);
	if (test_bit(GD_NEED_PART_SCAN, &disk->state)) {
		/*
		 * Only return scanning errors if we are called from contexts
		 * that explicitly want them, e.g. the BLKRRPART ioctl.
		 */
		ret = bdev_disk_changed(disk, false);
		if (ret && (mode & BLK_OPEN_STRICT_SCAN)) {
			blkdev_put_whole(bdev);
			return ret;
		}
	}
	return 0;
}

static int blkdev_get_part(struct block_device *part, blk_mode_t mode)
{
	struct gendisk *disk = part->bd_disk;
	int ret;

	ret = blkdev_get_whole(bdev_whole(part), mode);
	if (ret)
		return ret;

	ret = -ENXIO;
	if (!bdev_nr_sectors(part))
		goto out_blkdev_put;

	if (!atomic_read(&part->bd_openers)) {
		disk->open_partitions++;
		set_init_blocksize(part);
	}
	atomic_inc(&part->bd_openers);
	return 0;

out_blkdev_put:
	blkdev_put_whole(bdev_whole(part));
	return ret;
}

int bdev_permission(dev_t dev, blk_mode_t mode, void *holder)
{
	int ret;

	ret = devcgroup_check_permission(DEVCG_DEV_BLOCK,
			MAJOR(dev), MINOR(dev),
			((mode & BLK_OPEN_READ) ? DEVCG_ACC_READ : 0) |
			((mode & BLK_OPEN_WRITE) ? DEVCG_ACC_WRITE : 0));
	if (ret)
		return ret;

	/* Blocking writes requires exclusive opener */
	if (mode & BLK_OPEN_RESTRICT_WRITES && !holder)
		return -EINVAL;

	/*
	 * We're using error pointers to indicate to ->release() when we
	 * failed to open that block device. Also this doesn't make sense.
	 */
	if (WARN_ON_ONCE(IS_ERR(holder)))
		return -EINVAL;

	return 0;
}

static void blkdev_put_part(struct block_device *part)
{
	struct block_device *whole = bdev_whole(part);

	if (atomic_dec_and_test(&part->bd_openers)) {
		blkdev_flush_mapping(part);
		whole->bd_disk->open_partitions--;
	}
	blkdev_put_whole(whole);
}

struct block_device *blkdev_get_no_open(dev_t dev, bool autoload)
{
	struct block_device *bdev;
	struct inode *inode;

	inode = ilookup(blockdev_superblock, dev);
	if (!inode && autoload && IS_ENABLED(CONFIG_BLOCK_LEGACY_AUTOLOAD)) {
		blk_request_module(dev);
		inode = ilookup(blockdev_superblock, dev);
		if (inode)
			pr_warn_ratelimited(
"block device autoloading is deprecated and will be removed.\n");
	}
	if (!inode)
		return NULL;

	/* switch from the inode reference to a device mode one: */
	bdev = &BDEV_I(inode)->bdev;
	if (!kobject_get_unless_zero(&bdev->bd_device.kobj))
		bdev = NULL;
	iput(inode);
	return bdev;
}

void blkdev_put_no_open(struct block_device *bdev)
{
	put_device(&bdev->bd_device);
}

static bool bdev_writes_blocked(struct block_device *bdev)
{
	return bdev->bd_writers < 0;
}

static void bdev_block_writes(struct block_device *bdev)
{
	bdev->bd_writers--;
}

static void bdev_unblock_writes(struct block_device *bdev)
{
	bdev->bd_writers++;
}

static bool bdev_may_open(struct block_device *bdev, blk_mode_t mode)
{
	if (bdev_allow_write_mounted)
		return true;
	/* Writes blocked? */
	if (mode & BLK_OPEN_WRITE && bdev_writes_blocked(bdev))
		return false;
	if (mode & BLK_OPEN_RESTRICT_WRITES && bdev->bd_writers > 0)
		return false;
	return true;
}

static void bdev_claim_write_access(struct block_device *bdev, blk_mode_t mode)
{
	if (bdev_allow_write_mounted)
		return;

	/* Claim exclusive or shared write access. */
	if (mode & BLK_OPEN_RESTRICT_WRITES)
		bdev_block_writes(bdev);
	else if (mode & BLK_OPEN_WRITE)
		bdev->bd_writers++;
}

static inline bool bdev_unclaimed(const struct file *bdev_file)
{
	return bdev_file->private_data == BDEV_I(bdev_file->f_mapping->host);
}

static void bdev_yield_write_access(struct file *bdev_file)
{
	struct block_device *bdev;

	if (bdev_allow_write_mounted)
		return;

	if (bdev_unclaimed(bdev_file))
		return;

	bdev = file_bdev(bdev_file);

	if (bdev_file->f_mode & FMODE_WRITE_RESTRICTED)
		bdev_unblock_writes(bdev);
	else if (bdev_file->f_mode & FMODE_WRITE)
		bdev->bd_writers--;
}

/**
 * bdev_open - open a block device
 * @bdev: block device to open
 * @mode: open mode (BLK_OPEN_*)
 * @holder: exclusive holder identifier
 * @hops: holder operations
 * @bdev_file: file for the block device
 *
 * Open the block device. If @holder is not %NULL, the block device is opened
 * with exclusive access.  Exclusive opens may nest for the same @holder.
 *
 * CONTEXT:
 * Might sleep.
 *
 * RETURNS:
 * zero on success, -errno on failure.
 */
int bdev_open(struct block_device *bdev, blk_mode_t mode, void *holder,
	      const struct blk_holder_ops *hops, struct file *bdev_file)
{
	bool unblock_events = true;
	struct gendisk *disk = bdev->bd_disk;
	int ret;

	if (holder) {
		mode |= BLK_OPEN_EXCL;
		ret = bd_prepare_to_claim(bdev, holder, hops);
		if (ret)
			return ret;
	} else {
		if (WARN_ON_ONCE(mode & BLK_OPEN_EXCL))
			return -EIO;
	}

	disk_block_events(disk);

	mutex_lock(&disk->open_mutex);
	ret = -ENXIO;
	if (!disk_live(disk))
		goto abort_claiming;
	if (!try_module_get(disk->fops->owner))
		goto abort_claiming;
	ret = -EBUSY;
	if (!bdev_may_open(bdev, mode))
		goto put_module;
	if (bdev_is_partition(bdev))
		ret = blkdev_get_part(bdev, mode);
	else
		ret = blkdev_get_whole(bdev, mode);
	if (ret)
		goto put_module;
	bdev_claim_write_access(bdev, mode);
	if (holder) {
		bd_finish_claiming(bdev, holder, hops);

		/*
		 * Block event polling for write claims if requested.  Any write
		 * holder makes the write_holder state stick until all are
		 * released.  This is good enough and tracking individual
		 * writeable reference is too fragile given the way @mode is
		 * used in blkdev_get/put().
		 */
		if ((mode & BLK_OPEN_WRITE) &&
		    !bdev_test_flag(bdev, BD_WRITE_HOLDER) &&
		    (disk->event_flags & DISK_EVENT_FLAG_BLOCK_ON_EXCL_WRITE)) {
			bdev_set_flag(bdev, BD_WRITE_HOLDER);
			unblock_events = false;
		}
	}
	mutex_unlock(&disk->open_mutex);

	if (unblock_events)
		disk_unblock_events(disk);

	bdev_file->f_flags |= O_LARGEFILE;
	bdev_file->f_mode |= FMODE_CAN_ODIRECT;
	if (bdev_nowait(bdev))
		bdev_file->f_mode |= FMODE_NOWAIT;
	if (mode & BLK_OPEN_RESTRICT_WRITES)
		bdev_file->f_mode |= FMODE_WRITE_RESTRICTED;
	bdev_file->f_mapping = bdev->bd_mapping;
	bdev_file->f_wb_err = filemap_sample_wb_err(bdev_file->f_mapping);
	bdev_file->private_data = holder;

	return 0;
put_module:
	module_put(disk->fops->owner);
abort_claiming:
	if (holder)
		bd_abort_claiming(bdev, holder);
	mutex_unlock(&disk->open_mutex);
	disk_unblock_events(disk);
	return ret;
}

/*
 * If BLK_OPEN_WRITE_IOCTL is set then this is a historical quirk
 * associated with the floppy driver where it has allowed ioctls if the
 * file was opened for writing, but does not allow reads or writes.
 * Make sure that this quirk is reflected in @f_flags.
 *
 * It can also happen if a block device is opened as O_RDWR | O_WRONLY.
 */
static unsigned blk_to_file_flags(blk_mode_t mode)
{
	unsigned int flags = 0;

	if ((mode & (BLK_OPEN_READ | BLK_OPEN_WRITE)) ==
	    (BLK_OPEN_READ | BLK_OPEN_WRITE))
		flags |= O_RDWR;
	else if (mode & BLK_OPEN_WRITE_IOCTL)
		flags |= O_RDWR | O_WRONLY;
	else if (mode & BLK_OPEN_WRITE)
		flags |= O_WRONLY;
	else if (mode & BLK_OPEN_READ)
		flags |= O_RDONLY; /* homeopathic, because O_RDONLY is 0 */
	else
		WARN_ON_ONCE(true);

	if (mode & BLK_OPEN_NDELAY)
		flags |= O_NDELAY;

	return flags;
}

struct file *bdev_file_open_by_dev(dev_t dev, blk_mode_t mode, void *holder,
				   const struct blk_holder_ops *hops)
{
	struct file *bdev_file;
	struct block_device *bdev;
	unsigned int flags;
	int ret;

	ret = bdev_permission(dev, mode, holder);
	if (ret)
		return ERR_PTR(ret);

	bdev = blkdev_get_no_open(dev, true);
	if (!bdev)
		return ERR_PTR(-ENXIO);

	flags = blk_to_file_flags(mode);
	bdev_file = alloc_file_pseudo_noaccount(BD_INODE(bdev),
			blockdev_mnt, "", flags | O_LARGEFILE, &def_blk_fops);
	if (IS_ERR(bdev_file)) {
		blkdev_put_no_open(bdev);
		return bdev_file;
	}
	ihold(BD_INODE(bdev));

	ret = bdev_open(bdev, mode, holder, hops, bdev_file);
	if (ret) {
		/* We failed to open the block device. Let ->release() know. */
		bdev_file->private_data = ERR_PTR(ret);
		fput(bdev_file);
		return ERR_PTR(ret);
	}
	return bdev_file;
}
EXPORT_SYMBOL(bdev_file_open_by_dev);

struct file *bdev_file_open_by_path(const char *path, blk_mode_t mode,
				    void *holder,
				    const struct blk_holder_ops *hops)
{
	struct file *file;
	dev_t dev;
	int error;

	error = lookup_bdev(path, &dev);
	if (error)
		return ERR_PTR(error);

	file = bdev_file_open_by_dev(dev, mode, holder, hops);
	if (!IS_ERR(file) && (mode & BLK_OPEN_WRITE)) {
		if (bdev_read_only(file_bdev(file))) {
			fput(file);
			file = ERR_PTR(-EACCES);
		}
	}

	return file;
}
EXPORT_SYMBOL(bdev_file_open_by_path);

static inline void bd_yield_claim(struct file *bdev_file)
{
	struct block_device *bdev = file_bdev(bdev_file);
	void *holder = bdev_file->private_data;

	lockdep_assert_held(&bdev->bd_disk->open_mutex);

	if (WARN_ON_ONCE(IS_ERR_OR_NULL(holder)))
		return;

	if (!bdev_unclaimed(bdev_file))
		bd_end_claim(bdev, holder);
}

void bdev_release(struct file *bdev_file)
{
	struct block_device *bdev = file_bdev(bdev_file);
	void *holder = bdev_file->private_data;
	struct gendisk *disk = bdev->bd_disk;

	/* We failed to open that block device. */
	if (IS_ERR(holder))
		goto put_no_open;

	/*
	 * Sync early if it looks like we're the last one.  If someone else
	 * opens the block device between now and the decrement of bd_openers
	 * then we did a sync that we didn't need to, but that's not the end
	 * of the world and we want to avoid long (could be several minute)
	 * syncs while holding the mutex.
	 */
	if (atomic_read(&bdev->bd_openers) == 1)
		sync_blockdev(bdev);

	mutex_lock(&disk->open_mutex);
	bdev_yield_write_access(bdev_file);

	if (holder)
		bd_yield_claim(bdev_file);

	/*
	 * Trigger event checking and tell drivers to flush MEDIA_CHANGE
	 * event.  This is to ensure detection of media removal commanded
	 * from userland - e.g. eject(1).
	 */
	disk_flush_events(disk, DISK_EVENT_MEDIA_CHANGE);

	if (bdev_is_partition(bdev))
		blkdev_put_part(bdev);
	else
		blkdev_put_whole(bdev);
	mutex_unlock(&disk->open_mutex);

	module_put(disk->fops->owner);
put_no_open:
	blkdev_put_no_open(bdev);
}

/**
 * bdev_fput - yield claim to the block device and put the file
 * @bdev_file: open block device
 *
 * Yield claim on the block device and put the file. Ensure that the
 * block device can be reclaimed before the file is closed which is a
 * deferred operation.
 */
void bdev_fput(struct file *bdev_file)
{
	if (WARN_ON_ONCE(bdev_file->f_op != &def_blk_fops))
		return;

	if (bdev_file->private_data) {
		struct block_device *bdev = file_bdev(bdev_file);
		struct gendisk *disk = bdev->bd_disk;

		mutex_lock(&disk->open_mutex);
		bdev_yield_write_access(bdev_file);
		bd_yield_claim(bdev_file);
		/*
		 * Tell release we already gave up our hold on the
		 * device and if write restrictions are available that
		 * we already gave up write access to the device.
		 */
		bdev_file->private_data = BDEV_I(bdev_file->f_mapping->host);
		mutex_unlock(&disk->open_mutex);
	}

	fput(bdev_file);
}
EXPORT_SYMBOL(bdev_fput);

/**
 * lookup_bdev() - Look up a struct block_device by name.
 * @pathname: Name of the block device in the filesystem.
 * @dev: Pointer to the block device's dev_t, if found.
 *
 * Lookup the block device's dev_t at @pathname in the current
 * namespace if possible and return it in @dev.
 *
 * Context: May sleep.
 * Return: 0 if succeeded, negative errno otherwise.
 */
int lookup_bdev(const char *pathname, dev_t *dev)
{
	struct inode *inode;
	struct path path;
	int error;

	if (!pathname || !*pathname)
		return -EINVAL;

	error = kern_path(pathname, LOOKUP_FOLLOW, &path);
	if (error)
		return error;

	inode = d_backing_inode(path.dentry);
	error = -ENOTBLK;
	if (!S_ISBLK(inode->i_mode))
		goto out_path_put;
	error = -EACCES;
	if (!may_open_dev(&path))
		goto out_path_put;

	*dev = inode->i_rdev;
	error = 0;
out_path_put:
	path_put(&path);
	return error;
}
EXPORT_SYMBOL(lookup_bdev);

/**
 * bdev_mark_dead - mark a block device as dead
 * @bdev: block device to operate on
 * @surprise: indicate a surprise removal
 *
 * Tell the file system that this devices or media is dead.  If @surprise is set
 * to %true the device or media is already gone, if not we are preparing for an
 * orderly removal.
 *
 * This calls into the file system, which then typicall syncs out all dirty data
 * and writes back inodes and then invalidates any cached data in the inodes on
 * the file system.  In addition we also invalidate the block device mapping.
 */
void bdev_mark_dead(struct block_device *bdev, bool surprise)
{
	mutex_lock(&bdev->bd_holder_lock);
	if (bdev->bd_holder_ops && bdev->bd_holder_ops->mark_dead)
		bdev->bd_holder_ops->mark_dead(bdev, surprise);
	else {
		mutex_unlock(&bdev->bd_holder_lock);
		sync_blockdev(bdev);
	}

	invalidate_bdev(bdev);
}
/*
 * New drivers should not use this directly.  There are some drivers however
 * that needs this for historical reasons. For example, the DASD driver has
 * historically had a shutdown to offline mode that doesn't actually remove the
 * gendisk that otherwise looks a lot like a safe device removal.
 */
EXPORT_SYMBOL_GPL(bdev_mark_dead);

void sync_bdevs(bool wait)
{
	struct inode *inode, *old_inode = NULL;

	spin_lock(&blockdev_superblock->s_inode_list_lock);
	list_for_each_entry(inode, &blockdev_superblock->s_inodes, i_sb_list) {
		struct address_space *mapping = inode->i_mapping;
		struct block_device *bdev;

		spin_lock(&inode->i_lock);
		if (inode_state_read(inode) & (I_FREEING | I_WILL_FREE | I_NEW) ||
		    mapping->nrpages == 0) {
			spin_unlock(&inode->i_lock);
			continue;
		}
		__iget(inode);
		spin_unlock(&inode->i_lock);
		spin_unlock(&blockdev_superblock->s_inode_list_lock);
		/*
		 * We hold a reference to 'inode' so it couldn't have been
		 * removed from s_inodes list while we dropped the
		 * s_inode_list_lock  We cannot iput the inode now as we can
		 * be holding the last reference and we cannot iput it under
		 * s_inode_list_lock. So we keep the reference and iput it
		 * later.
		 */
		iput(old_inode);
		old_inode = inode;
		bdev = I_BDEV(inode);

		mutex_lock(&bdev->bd_disk->open_mutex);
		if (!atomic_read(&bdev->bd_openers)) {
			; /* skip */
		} else if (wait) {
			/*
			 * We keep the error status of individual mapping so
			 * that applications can catch the writeback error using
			 * fsync(2). See filemap_fdatawait_keep_errors() for
			 * details.
			 */
			filemap_fdatawait_keep_errors(inode->i_mapping);
		} else {
			filemap_fdatawrite(inode->i_mapping);
		}
		mutex_unlock(&bdev->bd_disk->open_mutex);

		spin_lock(&blockdev_superblock->s_inode_list_lock);
	}
	spin_unlock(&blockdev_superblock->s_inode_list_lock);
	iput(old_inode);
}

/*
 * Handle STATX_{DIOALIGN, WRITE_ATOMIC} for block devices.
 */
void bdev_statx(const struct path *path, struct kstat *stat, u32 request_mask)
{
	struct block_device *bdev;

	/*
	 * Note that d_backing_inode() returns the block device node inode, not
	 * the block device's internal inode.  Therefore it is *not* valid to
	 * use I_BDEV() here; the block device has to be looked up by i_rdev
	 * instead.
	 */
	bdev = blkdev_get_no_open(d_backing_inode(path->dentry)->i_rdev, false);
	if (!bdev)
		return;

	if (request_mask & STATX_DIOALIGN) {
		stat->dio_mem_align = bdev_dma_alignment(bdev) + 1;
		stat->dio_offset_align = bdev_logical_block_size(bdev);
		stat->result_mask |= STATX_DIOALIGN;
	}

	if (request_mask & STATX_WRITE_ATOMIC && bdev_can_atomic_write(bdev)) {
		struct request_queue *bd_queue = bdev->bd_queue;

		generic_fill_statx_atomic_writes(stat,
			queue_atomic_write_unit_min_bytes(bd_queue),
			queue_atomic_write_unit_max_bytes(bd_queue),
			0);
	}

	stat->blksize = bdev_io_min(bdev);

	blkdev_put_no_open(bdev);
}

bool disk_live(struct gendisk *disk)
{
	return !inode_unhashed(BD_INODE(disk->part0));
}
EXPORT_SYMBOL_GPL(disk_live);

unsigned int block_size(struct block_device *bdev)
{
	return 1 << BD_INODE(bdev)->i_blkbits;
}
EXPORT_SYMBOL_GPL(block_size);

static int __init setup_bdev_allow_write_mounted(char *str)
{
	if (kstrtobool(str, &bdev_allow_write_mounted))
		pr_warn("Invalid option string for bdev_allow_write_mounted:"
			" '%s'\n", str);
	return 1;
}
__setup("bdev_allow_write_mounted=", setup_bdev_allow_write_mounted);
]

gemini-3-flash-preview

							input: 67513
							output: 20
							thoughts: 3439
						

**Analysis of Buffer Head LRU Invalidation Race**

Alright, let's break this down. We've got a potential race condition in the buffer head (bh) LRU invalidation process. Specifically, the concern is around `invalidate_bdev` and related functions in the block device handling. The core issue revolves around `has_bh_in_lru`, which checks if a buffer head is present in a per-CPU LRU, and `__invalidate_bh_lrus`, which clears these LRUs during unmount/device removal.

`invalidate_bdev` aims to invalidate buffer heads associated with a block device. It calls `invalidate_bh_lrus()` to ensure no BHs are pinned by LRUs before invalidating the mapping pages. This is critical in the `kill_bdev` path as well when aggressively unmapping pages. The problem arises if `has_bh_in_lru` incorrectly returns `false` due to a race. This could lead to a missed CPU during invalidation, potentially leaving some pages "stuck" because their buffer heads remain pinned in the LRU. This path gets hit from `ext4_put_super`, which is called during unmount, meaning that we could potentially leak pages.

My initial thoughts are, is this truly a problem? `invalidate_mapping_pages` is often best-effort. However, `truncate_inode_pages` is much more forceful. The key is to understand WHEN this race could happen.  `has_bh_in_lru` checks the entire LRU, not just the LRU for the bdev we are unmounting. This is important to understand.

Here's my thought process. If CPU 0 has a buffer head from `loop4` in its LRU, and CPU 1 is unmounting `loop4`. CPU 1 calls `invalidate_bh_lrus()`.  `has_bh_in_lru(0, NULL)` is called. If it returns false incorrectly (but really the right thing), CPU 0's LRU is not cleared. The buffer head from `loop4` stays in CPU 0's LRU, pinning the page, and `invalidate_mapping_pages` might fail to free that page.

However, further down the stack it is actually an LRU drain process (`invalidate_bh_lrus_cpu` called by `lru_add_drain_per_cpu`). If that is running on CPU 0, it *should* already be clearing the LRU. Therefore, the race is likely to be benign because `has_bh_in_lru` is, in reality, probably *correct*. In this specific context.

The only "harm" is if it's reading garbage, but `has_bh_in_lru` doesn't dereference, it just checks for non-NULL. Also, since pointers are written atomically, this is not an issue.

The KCSAN report flagged a "write of 8 bytes". Torn writes are something KCSAN would flag.

Thinking about other scenarios, what is *actually* written?

If `b->bhs[i]` is read, then `__invalidate_bh_lrus` clears it and calls `brelse(bh)`. `brelse` might free the buffer head, which is not what happens. But `has_bh_in_lru` doesn't dereference `bh`, it *just* checks if it's non-NULL.

```c
	for (i = 0; i < BH_LRU_SIZE; i++) {
		if (b->bhs[i])
			return true;
	}
```

So, it's just checking the pointer value.

If `has_bh_in_lru` returns true, `invalidate_bh_lrus` will schedule `invalidate_bh_lru` on CPU 0. CPU 0 will then call `__invalidate_bh_lrus` again. This is safe. The race is on the pointer itself. Since it's a simple pointer, we care about NULL vs. not, we are okay with false positives (extra IPI) and "false" negatives (because it's being cleared anyway). This looks benign.

I should verify if `READ_ONCE` should be used. The compiler loop could optimize it, but here it's simple. Many similar kernel checks *are* done without `READ_ONCE` if they are optimization checks where the pointer is word-sized and aligned.  However, KCSAN is designed to flag these situations, so it is a good thing to revisit.

Looking closely at the code:
```c
bool has_bh_in_lru(int cpu, void *dummy)

map[File:mm/swap.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
 *  linux/mm/swap.c
 *
 *  Copyright (C) 1991, 1992, 1993, 1994  Linus Torvalds
 */

/*
 * This file contains the default values for the operation of the
 * Linux VM subsystem. Fine-tuning documentation can be found in
 * Documentation/admin-guide/sysctl/vm.rst.
 * Started 18.12.91
 * Swap aging added 23.2.95, Stephen Tweedie.
 * Buffermem limits added 12.3.98, Rik van Riel.
 */

#include <linux/mm.h>
#include <linux/sched.h>
#include <linux/kernel_stat.h>
#include <linux/swap.h>
#include <linux/mman.h>
#include <linux/pagemap.h>
#include <linux/pagevec.h>
#include <linux/init.h>
#include <linux/export.h>
#include <linux/mm_inline.h>
#include <linux/percpu_counter.h>
#include <linux/memremap.h>
#include <linux/percpu.h>
#include <linux/cpu.h>
#include <linux/notifier.h>
#include <linux/backing-dev.h>
#include <linux/memcontrol.h>
#include <linux/gfp.h>
#include <linux/uio.h>
#include <linux/hugetlb.h>
#include <linux/page_idle.h>
#include <linux/local_lock.h>
#include <linux/buffer_head.h>

#include "internal.h"

#define CREATE_TRACE_POINTS
#include <trace/events/pagemap.h>

/* How many pages do we try to swap or page in/out together? As a power of 2 */
int page_cluster;
static const int page_cluster_max = 31;

struct cpu_fbatches {
	/*
	 * The following folio batches are grouped together because they are protected
	 * by disabling preemption (and interrupts remain enabled).
	 */
	local_lock_t lock;
	struct folio_batch lru_add;
	struct folio_batch lru_deactivate_file;
	struct folio_batch lru_deactivate;
	struct folio_batch lru_lazyfree;
#ifdef CONFIG_SMP
	struct folio_batch lru_activate;
#endif
	/* Protecting the following batches which require disabling interrupts */
	local_lock_t lock_irq;
	struct folio_batch lru_move_tail;
};

static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = {
	.lock = INIT_LOCAL_LOCK(lock),
	.lock_irq = INIT_LOCAL_LOCK(lock_irq),
};

static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp,
		unsigned long *flagsp)
{
	if (folio_test_lru(folio)) {
		folio_lruvec_relock_irqsave(folio, lruvecp, flagsp);
		lruvec_del_folio(*lruvecp, folio);
		__folio_clear_lru_flags(folio);
	}
}

/*
 * This path almost never happens for VM activity - pages are normally freed
 * in batches.  But it gets used by networking - and for compound pages.
 */
static void page_cache_release(struct folio *folio)
{
	struct lruvec *lruvec = NULL;
	unsigned long flags;

	__page_cache_release(folio, &lruvec, &flags);
	if (lruvec)
		unlock_page_lruvec_irqrestore(lruvec, flags);
}

void __folio_put(struct folio *folio)
{
	if (unlikely(folio_is_zone_device(folio))) {
		free_zone_device_folio(folio);
		return;
	}

	if (folio_test_hugetlb(folio)) {
		free_huge_folio(folio);
		return;
	}

	page_cache_release(folio);
	folio_unqueue_deferred_split(folio);
	mem_cgroup_uncharge(folio);
	free_frozen_pages(&folio->page, folio_order(folio));
}
EXPORT_SYMBOL(__folio_put);

typedef void (*move_fn_t)(struct lruvec *lruvec, struct folio *folio);

static void lru_add(struct lruvec *lruvec, struct folio *folio)
{
	int was_unevictable = folio_test_clear_unevictable(folio);
	long nr_pages = folio_nr_pages(folio);

	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);

	/*
	 * Is an smp_mb__after_atomic() still required here, before
	 * folio_evictable() tests the mlocked flag, to rule out the possibility
	 * of stranding an evictable folio on an unevictable LRU?  I think
	 * not, because __munlock_folio() only clears the mlocked flag
	 * while the LRU lock is held.
	 *
	 * (That is not true of __page_cache_release(), and not necessarily
	 * true of folios_put(): but those only clear the mlocked flag after
	 * folio_put_testzero() has excluded any other users of the folio.)
	 */
	if (folio_evictable(folio)) {
		if (was_unevictable)
			__count_vm_events(UNEVICTABLE_PGRESCUED, nr_pages);
	} else {
		folio_clear_active(folio);
		folio_set_unevictable(folio);
		/*
		 * folio->mlock_count = !!folio_test_mlocked(folio)?
		 * But that leaves __mlock_folio() in doubt whether another
		 * actor has already counted the mlock or not.  Err on the
		 * safe side, underestimate, let page reclaim fix it, rather
		 * than leaving a page on the unevictable LRU indefinitely.
		 */
		folio->mlock_count = 0;
		if (!was_unevictable)
			__count_vm_events(UNEVICTABLE_PGCULLED, nr_pages);
	}

	lruvec_add_folio(lruvec, folio);
	trace_mm_lru_insertion(folio);
}

static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn)
{
	int i;
	struct lruvec *lruvec = NULL;
	unsigned long flags = 0;

	for (i = 0; i < folio_batch_count(fbatch); i++) {
		struct folio *folio = fbatch->folios[i];

		/* block memcg migration while the folio moves between lru */
		if (move_fn != lru_add && !folio_test_clear_lru(folio))
			continue;

		folio_lruvec_relock_irqsave(folio, &lruvec, &flags);
		move_fn(lruvec, folio);

		folio_set_lru(folio);
	}

	if (lruvec)
		unlock_page_lruvec_irqrestore(lruvec, flags);
	folios_put(fbatch);
}

static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch,
		struct folio *folio, move_fn_t move_fn, bool disable_irq)
{
	unsigned long flags;

	folio_get(folio);

	if (disable_irq)
		local_lock_irqsave(&cpu_fbatches.lock_irq, flags);
	else
		local_lock(&cpu_fbatches.lock);

	if (!folio_batch_add(this_cpu_ptr(fbatch), folio) ||
			!folio_may_be_lru_cached(folio) || lru_cache_disabled())
		folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn);

	if (disable_irq)
		local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags);
	else
		local_unlock(&cpu_fbatches.lock);
}

#define folio_batch_add_and_move(folio, op)		\
	__folio_batch_add_and_move(			\
		&cpu_fbatches.op,			\
		folio,					\
		op,					\
		offsetof(struct cpu_fbatches, op) >=	\
		offsetof(struct cpu_fbatches, lock_irq)	\
	)

static void lru_move_tail(struct lruvec *lruvec, struct folio *folio)
{
	if (folio_test_unevictable(folio))
		return;

	lruvec_del_folio(lruvec, folio);
	folio_clear_active(folio);
	lruvec_add_folio_tail(lruvec, folio);
	__count_vm_events(PGROTATED, folio_nr_pages(folio));
}

/*
 * Writeback is about to end against a folio which has been marked for
 * immediate reclaim.  If it still appears to be reclaimable, move it
 * to the tail of the inactive list.
 *
 * folio_rotate_reclaimable() must disable IRQs, to prevent nasty races.
 */
void folio_rotate_reclaimable(struct folio *folio)
{
	if (folio_test_locked(folio) || folio_test_dirty(folio) ||
	    folio_test_unevictable(folio) || !folio_test_lru(folio))
		return;

	folio_batch_add_and_move(folio, lru_move_tail);
}

void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file,
		unsigned int nr_io, unsigned int nr_rotated)
		__releases(lruvec->lru_lock)
{
	unsigned long cost;

	/*
	 * Reflect the relative cost of incurring IO and spending CPU
	 * time on rotations. This doesn't attempt to make a precise
	 * comparison, it just says: if reloads are about comparable
	 * between the LRU lists, or rotations are overwhelmingly
	 * different between them, adjust scan balance for CPU work.
	 */
	cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated;
	if (!cost) {
		spin_unlock_irq(&lruvec->lru_lock);
		return;
	}

	for (;;) {
		unsigned long lrusize;

		/* Record cost event */
		if (file)
			lruvec->file_cost += cost;
		else
			lruvec->anon_cost += cost;

		/*
		 * Decay previous events
		 *
		 * Because workloads change over time (and to avoid
		 * overflow) we keep these statistics as a floating
		 * average, which ends up weighing recent refaults
		 * more than old ones.
		 */
		lrusize = lruvec_page_state(lruvec, NR_INACTIVE_ANON) +
			  lruvec_page_state(lruvec, NR_ACTIVE_ANON) +
			  lruvec_page_state(lruvec, NR_INACTIVE_FILE) +
			  lruvec_page_state(lruvec, NR_ACTIVE_FILE);

		if (lruvec->file_cost + lruvec->anon_cost > lrusize / 4) {
			lruvec->file_cost /= 2;
			lruvec->anon_cost /= 2;
		}

		spin_unlock_irq(&lruvec->lru_lock);
		lruvec = parent_lruvec(lruvec);
		if (!lruvec)
			break;
		spin_lock_irq(&lruvec->lru_lock);
	}
}

void lru_note_cost_refault(struct folio *folio)
{
	struct lruvec *lruvec;

	lruvec = folio_lruvec_lock_irq(folio);
	lru_note_cost_unlock_irq(lruvec, folio_is_file_lru(folio),
				folio_nr_pages(folio), 0);
}

static void lru_activate(struct lruvec *lruvec, struct folio *folio)
{
	long nr_pages = folio_nr_pages(folio);

	if (folio_test_active(folio) || folio_test_unevictable(folio))
		return;


	lruvec_del_folio(lruvec, folio);
	folio_set_active(folio);
	lruvec_add_folio(lruvec, folio);
	trace_mm_lru_activate(folio);

	__count_vm_events(PGACTIVATE, nr_pages);
	count_memcg_events(lruvec_memcg(lruvec), PGACTIVATE, nr_pages);
}

#ifdef CONFIG_SMP
static void folio_activate_drain(int cpu)
{
	struct folio_batch *fbatch = &per_cpu(cpu_fbatches.lru_activate, cpu);

	if (folio_batch_count(fbatch))
		folio_batch_move_lru(fbatch, lru_activate);
}

void folio_activate(struct folio *folio)
{
	if (folio_test_active(folio) || folio_test_unevictable(folio) ||
	    !folio_test_lru(folio))
		return;

	folio_batch_add_and_move(folio, lru_activate);
}

#else
static inline void folio_activate_drain(int cpu)
{
}

void folio_activate(struct folio *folio)
{
	struct lruvec *lruvec;

	if (!folio_test_clear_lru(folio))
		return;

	lruvec = folio_lruvec_lock_irq(folio);
	lru_activate(lruvec, folio);
	unlock_page_lruvec_irq(lruvec);
	folio_set_lru(folio);
}
#endif

static void __lru_cache_activate_folio(struct folio *folio)
{
	struct folio_batch *fbatch;
	int i;

	local_lock(&cpu_fbatches.lock);
	fbatch = this_cpu_ptr(&cpu_fbatches.lru_add);

	/*
	 * Search backwards on the optimistic assumption that the folio being
	 * activated has just been added to this batch. Note that only
	 * the local batch is examined as a !LRU folio could be in the
	 * process of being released, reclaimed, migrated or on a remote
	 * batch that is currently being drained. Furthermore, marking
	 * a remote batch's folio active potentially hits a race where
	 * a folio is marked active just after it is added to the inactive
	 * list causing accounting errors and BUG_ON checks to trigger.
	 */
	for (i = folio_batch_count(fbatch) - 1; i >= 0; i--) {
		struct folio *batch_folio = fbatch->folios[i];

		if (batch_folio == folio) {
			folio_set_active(folio);
			break;
		}
	}

	local_unlock(&cpu_fbatches.lock);
}

#ifdef CONFIG_LRU_GEN

static void lru_gen_inc_refs(struct folio *folio)
{
	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);

	if (folio_test_unevictable(folio))
		return;

	/* see the comment on LRU_REFS_FLAGS */
	if (!folio_test_referenced(folio)) {
		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
		return;
	}

	do {
		if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
			if (!folio_test_workingset(folio))
				folio_set_workingset(folio);
			return;
		}

		new_flags = old_flags + BIT(LRU_REFS_PGOFF);
	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
}

static bool lru_gen_clear_refs(struct folio *folio)
{
	struct lru_gen_folio *lrugen;
	int gen = folio_lru_gen(folio);
	int type = folio_is_file_lru(folio);

	if (gen < 0)
		return true;

	set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS | BIT(PG_workingset), 0);

	lrugen = &folio_lruvec(folio)->lrugen;
	/* whether can do without shuffling under the LRU lock */
	return gen == lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type]));
}

#else /* !CONFIG_LRU_GEN */

static void lru_gen_inc_refs(struct folio *folio)
{
}

static bool lru_gen_clear_refs(struct folio *folio)
{
	return false;
}

#endif /* CONFIG_LRU_GEN */

/**
 * folio_mark_accessed - Mark a folio as having seen activity.
 * @folio: The folio to mark.
 *
 * This function will perform one of the following transitions:
 *
 * * inactive,unreferenced	->	inactive,referenced
 * * inactive,referenced	->	active,unreferenced
 * * active,unreferenced	->	active,referenced
 *
 * When a newly allocated folio is not yet visible, so safe for non-atomic ops,
 * __folio_set_referenced() may be substituted for folio_mark_accessed().
 */
void folio_mark_accessed(struct folio *folio)
{
	if (folio_test_dropbehind(folio))
		return;
	if (lru_gen_enabled()) {
		lru_gen_inc_refs(folio);
		return;
	}

	if (!folio_test_referenced(folio)) {
		folio_set_referenced(folio);
	} else if (folio_test_unevictable(folio)) {
		/*
		 * Unevictable pages are on the "LRU_UNEVICTABLE" list. But,
		 * this list is never rotated or maintained, so marking an
		 * unevictable page accessed has no effect.
		 */
	} else if (!folio_test_active(folio)) {
		/*
		 * If the folio is on the LRU, queue it for activation via
		 * cpu_fbatches.lru_activate. Otherwise, assume the folio is in a
		 * folio_batch, mark it active and it'll be moved to the active
		 * LRU on the next drain.
		 */
		if (folio_test_lru(folio))
			folio_activate(folio);
		else
			__lru_cache_activate_folio(folio);
		folio_clear_referenced(folio);
		workingset_activation(folio);
	}
	if (folio_test_idle(folio))
		folio_clear_idle(folio);
}
EXPORT_SYMBOL(folio_mark_accessed);

/**
 * folio_add_lru - Add a folio to an LRU list.
 * @folio: The folio to be added to the LRU.
 *
 * Queue the folio for addition to the LRU. The decision on whether
 * to add the page to the [in]active [file|anon] list is deferred until the
 * folio_batch is drained. This gives a chance for the caller of folio_add_lru()
 * have the folio added to the active list using folio_mark_accessed().
 */
void folio_add_lru(struct folio *folio)
{
	VM_BUG_ON_FOLIO(folio_test_active(folio) &&
			folio_test_unevictable(folio), folio);
	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);

	/* see the comment in lru_gen_folio_seq() */
	if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
	    lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))
		folio_set_active(folio);

	folio_batch_add_and_move(folio, lru_add);
}
EXPORT_SYMBOL(folio_add_lru);

/**
 * folio_add_lru_vma() - Add a folio to the appropriate LRU list for this VMA.
 * @folio: The folio to be added to the LRU.
 * @vma: VMA in which the folio is mapped.
 *
 * If the VMA is mlocked, @folio is added to the unevictable list.
 * Otherwise, it is treated the same way as folio_add_lru().
 */
void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma)
{
	VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);

	if (unlikely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) == VM_LOCKED))
		mlock_new_folio(folio);
	else
		folio_add_lru(folio);
}

/*
 * If the folio cannot be invalidated, it is moved to the
 * inactive list to speed up its reclaim.  It is moved to the
 * head of the list, rather than the tail, to give the flusher
 * threads some time to write it out, as this is much more
 * effective than the single-page writeout from reclaim.
 *
 * If the folio isn't mapped and dirty/writeback, the folio
 * could be reclaimed asap using the reclaim flag.
 *
 * 1. active, mapped folio -> none
 * 2. active, dirty/writeback folio -> inactive, head, reclaim
 * 3. inactive, mapped folio -> none
 * 4. inactive, dirty/writeback folio -> inactive, head, reclaim
 * 5. inactive, clean -> inactive, tail
 * 6. Others -> none
 *
 * In 4, it moves to the head of the inactive list so the folio is
 * written out by flusher threads as this is much more efficient
 * than the single-page writeout from reclaim.
 */
static void lru_deactivate_file(struct lruvec *lruvec, struct folio *folio)
{
	bool active = folio_test_active(folio) || lru_gen_enabled();
	long nr_pages = folio_nr_pages(folio);

	if (folio_test_unevictable(folio))
		return;

	/* Some processes are using the folio */
	if (folio_mapped(folio))
		return;

	lruvec_del_folio(lruvec, folio);
	folio_clear_active(folio);
	folio_clear_referenced(folio);

	if (folio_test_writeback(folio) || folio_test_dirty(folio)) {
		/*
		 * Setting the reclaim flag could race with
		 * folio_end_writeback() and confuse readahead.  But the
		 * race window is _really_ small and  it's not a critical
		 * problem.
		 */
		lruvec_add_folio(lruvec, folio);
		folio_set_reclaim(folio);
	} else {
		/*
		 * The folio's writeback ended while it was in the batch.
		 * We move that folio to the tail of the inactive list.
		 */
		lruvec_add_folio_tail(lruvec, folio);
		__count_vm_events(PGROTATED, nr_pages);
	}

	if (active) {
		__count_vm_events(PGDEACTIVATE, nr_pages);
		count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE,
				     nr_pages);
	}
}

static void lru_deactivate(struct lruvec *lruvec, struct folio *folio)
{
	long nr_pages = folio_nr_pages(folio);

	if (folio_test_unevictable(folio) || !(folio_test_active(folio) || lru_gen_enabled()))
		return;

	lruvec_del_folio(lruvec, folio);
	folio_clear_active(folio);
	folio_clear_referenced(folio);
	lruvec_add_folio(lruvec, folio);

	__count_vm_events(PGDEACTIVATE, nr_pages);
	count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_pages);
}

static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
{
	long nr_pages = folio_nr_pages(folio);

	if (!folio_test_anon(folio) || !folio_test_swapbacked(folio) ||
	    folio_test_swapcache(folio) || folio_test_unevictable(folio))
		return;

	lruvec_del_folio(lruvec, folio);
	folio_clear_active(folio);
	if (lru_gen_enabled())
		lru_gen_clear_refs(folio);
	else
		folio_clear_referenced(folio);
	/*
	 * Lazyfree folios are clean anonymous folios.  They have
	 * the swapbacked flag cleared, to distinguish them from normal
	 * anonymous folios
	 */
	folio_clear_swapbacked(folio);
	lruvec_add_folio(lruvec, folio);

	__count_vm_events(PGLAZYFREE, nr_pages);
	count_memcg_events(lruvec_memcg(lruvec), PGLAZYFREE, nr_pages);
}

/*
 * Drain pages out of the cpu's folio_batch.
 * Either "cpu" is the current CPU, and preemption has already been
 * disabled; or "cpu" is being hot-unplugged, and is already dead.
 */
void lru_add_drain_cpu(int cpu)
{
	struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);
	struct folio_batch *fbatch = &fbatches->lru_add;

	if (folio_batch_count(fbatch))
		folio_batch_move_lru(fbatch, lru_add);

	fbatch = &fbatches->lru_move_tail;
	/* Disabling interrupts below acts as a compiler barrier. */
	if (data_race(folio_batch_count(fbatch))) {
		unsigned long flags;

		/* No harm done if a racing interrupt already did this */
		local_lock_irqsave(&cpu_fbatches.lock_irq, flags);
		folio_batch_move_lru(fbatch, lru_move_tail);
		local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags);
	}

	fbatch = &fbatches->lru_deactivate_file;
	if (folio_batch_count(fbatch))
		folio_batch_move_lru(fbatch, lru_deactivate_file);

	fbatch = &fbatches->lru_deactivate;
	if (folio_batch_count(fbatch))
		folio_batch_move_lru(fbatch, lru_deactivate);

	fbatch = &fbatches->lru_lazyfree;
	if (folio_batch_count(fbatch))
		folio_batch_move_lru(fbatch, lru_lazyfree);

	folio_activate_drain(cpu);
}

/**
 * deactivate_file_folio() - Deactivate a file folio.
 * @folio: Folio to deactivate.
 *
 * This function hints to the VM that @folio is a good reclaim candidate,
 * for example if its invalidation fails due to the folio being dirty
 * or under writeback.
 *
 * Context: Caller holds a reference on the folio.
 */
void deactivate_file_folio(struct folio *folio)
{
	/* Deactivating an unevictable folio will not accelerate reclaim */
	if (folio_test_unevictable(folio) || !folio_test_lru(folio))
		return;

	if (lru_gen_enabled() && lru_gen_clear_refs(folio))
		return;

	folio_batch_add_and_move(folio, lru_deactivate_file);
}

/*
 * folio_deactivate - deactivate a folio
 * @folio: folio to deactivate
 *
 * folio_deactivate() moves @folio to the inactive list if @folio was on the
 * active list and was not unevictable. This is done to accelerate the
 * reclaim of @folio.
 */
void folio_deactivate(struct folio *folio)
{
	if (folio_test_unevictable(folio) || !folio_test_lru(folio))
		return;

	if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio))
		return;

	folio_batch_add_and_move(folio, lru_deactivate);
}

/**
 * folio_mark_lazyfree - make an anon folio lazyfree
 * @folio: folio to deactivate
 *
 * folio_mark_lazyfree() moves @folio to the inactive file list.
 * This is done to accelerate the reclaim of @folio.
 */
void folio_mark_lazyfree(struct folio *folio)
{
	if (!folio_test_anon(folio) || !folio_test_swapbacked(folio) ||
	    !folio_test_lru(folio) ||
	    folio_test_swapcache(folio) || folio_test_unevictable(folio))
		return;

	folio_batch_add_and_move(folio, lru_lazyfree);
}

void lru_add_drain(void)
{
	local_lock(&cpu_fbatches.lock);
	lru_add_drain_cpu(smp_processor_id());
	local_unlock(&cpu_fbatches.lock);
	mlock_drain_local();
}

/*
 * It's called from per-cpu workqueue context in SMP case so
 * lru_add_drain_cpu and invalidate_bh_lrus_cpu should run on
 * the same cpu. It shouldn't be a problem in !SMP case since
 * the core is only one and the locks will disable preemption.
 */
static void lru_add_and_bh_lrus_drain(void)
{
	local_lock(&cpu_fbatches.lock);
	lru_add_drain_cpu(smp_processor_id());
	local_unlock(&cpu_fbatches.lock);
	invalidate_bh_lrus_cpu();
	mlock_drain_local();
}

void lru_add_drain_cpu_zone(struct zone *zone)
{
	local_lock(&cpu_fbatches.lock);
	lru_add_drain_cpu(smp_processor_id());
	drain_local_pages(zone);
	local_unlock(&cpu_fbatches.lock);
	mlock_drain_local();
}

#ifdef CONFIG_SMP

static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work);

static void lru_add_drain_per_cpu(struct work_struct *dummy)
{
	lru_add_and_bh_lrus_drain();
}

static bool cpu_needs_drain(unsigned int cpu)
{
	struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu);

	/* Check these in order of likelihood that they're not zero */
	return folio_batch_count(&fbatches->lru_add) ||
		folio_batch_count(&fbatches->lru_move_tail) ||
		folio_batch_count(&fbatches->lru_deactivate_file) ||
		folio_batch_count(&fbatches->lru_deactivate) ||
		folio_batch_count(&fbatches->lru_lazyfree) ||
		folio_batch_count(&fbatches->lru_activate) ||
		need_mlock_drain(cpu) ||
		has_bh_in_lru(cpu, NULL);
}

/*
 * Doesn't need any cpu hotplug locking because we do rely on per-cpu
 * kworkers being shut down before our page_alloc_cpu_dead callback is
 * executed on the offlined cpu.
 * Calling this function with cpu hotplug locks held can actually lead
 * to obscure indirect dependencies via WQ context.
 */
static inline void __lru_add_drain_all(bool force_all_cpus)
{
	/*
	 * lru_drain_gen - Global pages generation number
	 *
	 * (A) Definition: global lru_drain_gen = x implies that all generations
	 *     0 < n <= x are already *scheduled* for draining.
	 *
	 * This is an optimization for the highly-contended use case where a
	 * user space workload keeps constantly generating a flow of pages for
	 * each CPU.
	 */
	static unsigned int lru_drain_gen;
	static struct cpumask has_work;
	static DEFINE_MUTEX(lock);
	unsigned cpu, this_gen;

	/*
	 * Make sure nobody triggers this path before mm_percpu_wq is fully
	 * initialized.
	 */
	if (WARN_ON(!mm_percpu_wq))
		return;

	/*
	 * Guarantee folio_batch counter stores visible by this CPU
	 * are visible to other CPUs before loading the current drain
	 * generation.
	 */
	smp_mb();

	/*
	 * (B) Locally cache global LRU draining generation number
	 *
	 * The read barrier ensures that the counter is loaded before the mutex
	 * is taken. It pairs with smp_mb() inside the mutex critical section
	 * at (D).
	 */
	this_gen = smp_load_acquire(&lru_drain_gen);

	/* It helps everyone if we do our own local drain immediately. */
	lru_add_drain();

	mutex_lock(&lock);

	/*
	 * (C) Exit the draining operation if a newer generation, from another
	 * lru_add_drain_all(), was already scheduled for draining. Check (A).
	 */
	if (unlikely(this_gen != lru_drain_gen && !force_all_cpus))
		goto done;

	/*
	 * (D) Increment global generation number
	 *
	 * Pairs with smp_load_acquire() at (B), outside of the critical
	 * section. Use a full memory barrier to guarantee that the
	 * new global drain generation number is stored before loading
	 * folio_batch counters.
	 *
	 * This pairing must be done here, before the for_each_online_cpu loop
	 * below which drains the page vectors.
	 *
	 * Let x, y, and z represent some system CPU numbers, where x < y < z.
	 * Assume CPU #z is in the middle of the for_each_online_cpu loop
	 * below and has already reached CPU #y's per-cpu data. CPU #x comes
	 * along, adds some pages to its per-cpu vectors, then calls
	 * lru_add_drain_all().
	 *
	 * If the paired barrier is done at any later step, e.g. after the
	 * loop, CPU #x will just exit at (C) and miss flushing out all of its
	 * added pages.
	 */
	WRITE_ONCE(lru_drain_gen, lru_drain_gen + 1);
	smp_mb();

	cpumask_clear(&has_work);
	for_each_online_cpu(cpu) {
		struct work_struct *work = &per_cpu(lru_add_drain_work, cpu);

		if (cpu_needs_drain(cpu)) {
			INIT_WORK(work, lru_add_drain_per_cpu);
			queue_work_on(cpu, mm_percpu_wq, work);
			__cpumask_set_cpu(cpu, &has_work);
		}
	}

	for_each_cpu(cpu, &has_work)
		flush_work(&per_cpu(lru_add_drain_work, cpu));

done:
	mutex_unlock(&lock);
}

void lru_add_drain_all(void)
{
	__lru_add_drain_all(false);
}
#else
void lru_add_drain_all(void)
{
	lru_add_drain();
}
#endif /* CONFIG_SMP */

atomic_t lru_disable_count = ATOMIC_INIT(0);

/*
 * lru_cache_disable() needs to be called before we start compiling
 * a list of folios to be migrated using folio_isolate_lru().
 * It drains folios on LRU cache and then disable on all cpus until
 * lru_cache_enable is called.
 *
 * Must be paired with a call to lru_cache_enable().
 */
void lru_cache_disable(void)
{
	atomic_inc(&lru_disable_count);
	/*
	 * Readers of lru_disable_count are protected by either disabling
	 * preemption or rcu_read_lock:
	 *
	 * preempt_disable, local_irq_disable  [bh_lru_lock()]
	 * rcu_read_lock		       [rt_spin_lock CONFIG_PREEMPT_RT]
	 * preempt_disable		       [local_lock !CONFIG_PREEMPT_RT]
	 *
	 * Since v5.1 kernel, synchronize_rcu() is guaranteed to wait on
	 * preempt_disable() regions of code. So any CPU which sees
	 * lru_disable_count = 0 will have exited the critical
	 * section when synchronize_rcu() returns.
	 */
	synchronize_rcu_expedited();
#ifdef CONFIG_SMP
	__lru_add_drain_all(true);
#else
	lru_add_and_bh_lrus_drain();
#endif
}

/**
 * folios_put_refs - Reduce the reference count on a batch of folios.
 * @folios: The folios.
 * @refs: The number of refs to subtract from each folio.
 *
 * Like folio_put(), but for a batch of folios.  This is more efficient
 * than writing the loop yourself as it will optimise the locks which need
 * to be taken if the folios are freed.  The folios batch is returned
 * empty and ready to be reused for another batch; there is no need
 * to reinitialise it.  If @refs is NULL, we subtract one from each
 * folio refcount.
 *
 * Context: May be called in process or interrupt context, but not in NMI
 * context.  May be called while holding a spinlock.
 */
void folios_put_refs(struct folio_batch *folios, unsigned int *refs)
{
	int i, j;
	struct lruvec *lruvec = NULL;
	unsigned long flags = 0;

	for (i = 0, j = 0; i < folios->nr; i++) {
		struct folio *folio = folios->folios[i];
		unsigned int nr_refs = refs ? refs[i] : 1;

		if (is_huge_zero_folio(folio))
			continue;

		if (folio_is_zone_device(folio)) {
			if (lruvec) {
				unlock_page_lruvec_irqrestore(lruvec, flags);
				lruvec = NULL;
			}
			if (folio_ref_sub_and_test(folio, nr_refs))
				free_zone_device_folio(folio);
			continue;
		}

		if (!folio_ref_sub_and_test(folio, nr_refs))
			continue;

		/* hugetlb has its own memcg */
		if (folio_test_hugetlb(folio)) {
			if (lruvec) {
				unlock_page_lruvec_irqrestore(lruvec, flags);
				lruvec = NULL;
			}
			free_huge_folio(folio);
			continue;
		}
		folio_unqueue_deferred_split(folio);
		__page_cache_release(folio, &lruvec, &flags);

		if (j != i)
			folios->folios[j] = folio;
		j++;
	}
	if (lruvec)
		unlock_page_lruvec_irqrestore(lruvec, flags);
	if (!j) {
		folio_batch_reinit(folios);
		return;
	}

	folios->nr = j;
	mem_cgroup_uncharge_folios(folios);
	free_unref_folios(folios);
}
EXPORT_SYMBOL(folios_put_refs);

/**
 * release_pages - batched put_page()
 * @arg: array of pages to release
 * @nr: number of pages
 *
 * Decrement the reference count on all the pages in @arg.  If it
 * fell to zero, remove the page from the LRU and free it.
 *
 * Note that the argument can be an array of pages, encoded pages,
 * or folio pointers. We ignore any encoded bits, and turn any of
 * them into just a folio that gets free'd.
 */
void release_pages(release_pages_arg arg, int nr)
{
	struct folio_batch fbatch;
	int refs[PAGEVEC_SIZE];
	struct encoded_page **encoded = arg.encoded_pages;
	int i;

	folio_batch_init(&fbatch);
	for (i = 0; i < nr; i++) {
		/* Turn any of the argument types into a folio */
		struct folio *folio = page_folio(encoded_page_ptr(encoded[i]));

		/* Is our next entry actually "nr_pages" -> "nr_refs" ? */
		refs[fbatch.nr] = 1;
		if (unlikely(encoded_page_flags(encoded[i]) &
			     ENCODED_PAGE_BIT_NR_PAGES_NEXT))
			refs[fbatch.nr] = encoded_nr_pages(encoded[++i]);

		if (folio_batch_add(&fbatch, folio) > 0)
			continue;
		folios_put_refs(&fbatch, refs);
	}

	if (fbatch.nr)
		folios_put_refs(&fbatch, refs);
}
EXPORT_SYMBOL(release_pages);

/*
 * The folios which we're about to release may be in the deferred lru-addition
 * queues.  That would prevent them from really being freed right now.  That's
 * OK from a correctness point of view but is inefficient - those folios may be
 * cache-warm and we want to give them back to the page allocator ASAP.
 *
 * So __folio_batch_release() will drain those queues here.
 * folio_batch_move_lru() calls folios_put() directly to avoid
 * mutual recursion.
 */
void __folio_batch_release(struct folio_batch *fbatch)
{
	if (!fbatch->percpu_pvec_drained) {
		lru_add_drain();
		fbatch->percpu_pvec_drained = true;
	}
	folios_put(fbatch);
}
EXPORT_SYMBOL(__folio_batch_release);

/**
 * folio_batch_remove_exceptionals() - Prune non-folios from a batch.
 * @fbatch: The batch to prune
 *
 * find_get_entries() fills a batch with both folios and shadow/swap/DAX
 * entries.  This function prunes all the non-folio entries from @fbatch
 * without leaving holes, so that it can be passed on to folio-only batch
 * operations.
 */
void folio_batch_remove_exceptionals(struct folio_batch *fbatch)
{
	unsigned int i, j;

	for (i = 0, j = 0; i < folio_batch_count(fbatch); i++) {
		struct folio *folio = fbatch->folios[i];
		if (!xa_is_value(folio))
			fbatch->folios[j++] = folio;
	}
	fbatch->nr = j;
}

static const struct ctl_table swap_sysctl_table[] = {
	{
		.procname	= "page-cluster",
		.data		= &page_cluster,
		.maxlen		= sizeof(int),
		.mode		= 0644,
		.proc_handler	= proc_dointvec_minmax,
		.extra1		= SYSCTL_ZERO,
		.extra2		= (void *)&page_cluster_max,
	}
};

/*
 * Perform any setup for the swap system
 */
void __init swap_setup(void)
{
	unsigned long megs = PAGES_TO_MB(totalram_pages());

	/* Use a smaller cluster for small-memory machines */
	if (megs < 16)
		page_cluster = 2;
	else
		page_cluster = 3;
	/*
	 * Right now other parts of the system means that we
	 * _really_ don't want to cluster much more
	 */

	register_sysctl_init("vm", swap_sysctl_table);
}
]

gemini-3-flash-preview

							input: 83023
							output: 570
							thoughts: 1666
						

**Benign Data Race in `has_bh_in_lru` - Analysis**

Okay, so I'm digging into this `mm/swap.c` code, specifically the `cpu_needs_drain` function. It calls `has_bh_in_lru`, which seems relevant for deciding whether to queue a work item on a CPU. My initial concern was that an incorrect `false` return from `has_bh_in_lru` could lead to missed drain operations for the buffer head LRU.

Looking deeper, `has_bh_in_lru` only incorrectly returns `false` if it races with either `__invalidate_bh_lrus` (which is already draining the LRU) or `bh_lru_install` and `lookup_bh_lru`. If it races with the latter, we might miss a drain. However, `lru_add_drain_all` is *often* called to ensure a complete drain, so missing it once isn't necessarily catastrophic, unless it's a critical path like unmount.

Ah, but for unmount, we call `invalidate_bh_lrus()`, which calls `on_each_cpu_cond(has_bh_in_lru, ...)` *with* `wait=1`. This is where it gets more interesting.  `invalidate_bh_lrus()` itself uses `has_bh_in_lru` as a condition. If it misses a CPU here, that's more problematic.

Now, the reported race is specifically with `__invalidate_bh_lrus`, which is called from `invalidate_bh_lrus_cpu`, which is itself called from the work item queued by `lru_add_drain_all`, which is named `lru_add_drain_per_cpu`.

Here's how I've rationalized this: If `lru_add_drain_all` is running on CPU 0 and CPU 1 calls `invalidate_bh_lrus`, the `has_bh_in_lru(0)` call on CPU 1 is racing with the `__invalidate_bh_lrus` call on CPU 0. If `has_bh_in_lru` returns `false`, it's *because* CPU 0 is already in the process of clearing its LRU. Thus, `invalidate_bh_lrus` doesn't need to send another IPI to CPU 0.

So it seems the race is benign: It's a race between two independent drain/invalidate mechanisms that are ultimately trying to achieve the same result. It's a race on a simple pointer field used as an optimization check; the field values are either valid pointers or `NULL`. The worst case scenario is an extra IPI, or a skipped IPI that wasn't actually needed. I'm reasonably confident this is fine.

But, I should *still* check if `has_bh_in_lru` should use `READ_ONCE`/`WRITE_ONCE` to avoid KCSAN warnings. While the race *is* benign from a code functionality perspective, documenting the race is always good practice. Although my task right now is to determine the impact on the functionality, not a code style fix.

Okay, further checking. `invalidate_bh_lrus` is called in `ext4_put_super` (during unmount). So, we have this:

*   **Task 3409 (CPU 0)**: `worker_thread` -> `process_scheduled_works` -> `lru_add_drain_per_cpu` -> `invalidate_bh_lrus_cpu` -> `__invalidate_bh_lrus` (Write).
*   **Task 3315 (CPU 1)**: `ext4_put_super` -> `invalidate_bdev` -> `invalidate_bh_lrus` -> `on_each_cpu_cond` -> `has_bh_in_lru` (Read).

Both are attempting to clear the LRUs.

I want to make *absolutely* certain this is benign, so I'm doing a final check: Could `has_bh_in_lru` be harmful in any other way? No. The pointer it reads is a simple pointer, not something more complicated, and `BH_LRU_SIZE` is also reasonable (16).

Now, the detailed analysis:  The `brelse` call happens *before* writing `NULL` in `__invalidate_bh_lrus`.  If `has_bh_in_lru` reads before the write, it *could* see a non-NULL and return `true`, causing an IPI.  The IPI handler (`


The data race occurs between `has_bh_in_lru()` and `__invalidate_bh_lrus()`. 

1.  **`has_bh_in_lru(cpu, ...)`** is used as a condition check in `on_each_cpu_cond()` (called by `invalidate_bh_lrus()`) and in `cpu_needs_drain()` (called by `lru_add_drain_all()`). It iterates through a CPU's buffer head LRU and returns `true` if any entry is non-NULL. This is an optimization to avoid sending Inter-Processor Interrupts (IPIs) to CPUs that have empty LRUs.
2.  **`__invalidate_bh_lrus(b)`** is called to clear a CPU's LRU, setting all entries to `NULL`. It is called either via an IPI handler (`invalidate_bh_lru`) or directly on the local CPU (`invalidate_bh_lrus_cpu`).

The reported race shows CPU 1 reading an LRU entry of CPU 0 while CPU 0 is clearing it. 

**Analysis of the race:**
- **Atomicity:** The access is on a pointer field (`struct buffer_head *`). On the involved 64-bit architecture (x86_64), aligned 8-byte loads and stores are atomic. CPU 1 will see either the old valid pointer or `NULL`.
- **Correctness:** 
    - If `has_bh_in_lru` sees a non-NULL pointer, it returns `true`, and an IPI is sent to CPU 0 to clear its LRU. If CPU 0 was already clearing its LRU, the IPI handler will simply find an empty LRU and do nothing. This is a harmless extra IPI.
    - If `has_bh_in_lru` sees `NULL` for all entries because it raced with the clearing process, it returns `false`, and no IPI is sent. This is also correct because the LRU is already being emptied or is already empty.
- **Safety:** `has_bh_in_lru` only checks if the pointer is non-zero; it does not dereference the pointer. Thus, even if the pointer was to a buffer head that was just released, no invalid memory access occurs.

The race is a classic example of a benign data race on an optimization check where the outcome of the race does not affect the correctness of the system, only potentially causing a minor performance difference (an extra IPI).

map[Benign:true Confident:true]

map[Benign:true Confident:true]

gemini-3-flash-preview

							input: 86300
							output: 670
							thoughts: 117
						

**Analyzing the Benign Nature of a Data Race**

Okay, so I've looked into this data race and I'm comfortable with why it's harmless. The core of it seems to be between `has_bh_in_lru` and `__invalidate_bh_lrus`. I already understand the code in question. It's an optimization, really, this check with `has_bh_in_lru`. The crucial point here is the size and alignment of the pointers involved. They are word-sized and properly aligned, which, of course, means they're subject to atomic read/write operations.

The worst-case scenario from this race? I can see it: we're either generating an extraneous IPI, which is wasteful but not catastrophic, or we're skipping an IPI which was ultimately unnecessary in the first place. Either way, there is no chance for any kind of memory corruption or the dreaded crash. It boils down to a potential performance hiccup, nothing more. A slightly less optimal IPI delivery. So, it is benign.