syzbot

map[DenialOfService:true Explanation:The bug is a critical security vulnerability in the Intel IOMMU driver that
leads to a Use-After-Free (UAF) of physical pages, allowing an unprivileged user
to achieve arbitrary physical memory read/write and full system compromise.

### Root Cause Analysis
The vulnerability lies in the error handling path of `intel_iommu_map_pages()`
in `drivers/iommu/intel/iommu.c`.

When a device driver maps a scatterlist for DMA (e.g., during a
`DRM_IOCTL_PRIME_FD_TO_HANDLE` ioctl), the IOMMU core calls
`iommu_map_nosync()`, which in turn calls the IOMMU driver's `map_pages`
callback (`intel_iommu_map_pages()`).

`intel_iommu_map_pages()` calls `intel_iommu_map()` (which wraps
`__domain_mapping()`) to map the pages. If `__domain_mapping()` fails in the
middle of mapping a large range (e.g., due to a `GFP_ATOMIC` memory allocation
failure in `pfn_to_dma_pte()` under memory pressure), it returns an error like
`-ENOMEM`.

However, `intel_iommu_map_pages()` has a flaw where it **does not update the
`mapped` output parameter** if an error is returned:
```c
	ret = intel_iommu_map(domain, iova, paddr, size, prot, gfp);
	if (!ret && mapped)
		*mapped = size; // Only updated on success!

	return ret;
```
Because `mapped` remains `0`, the caller `iommu_map_nosync()` assumes that 0
bytes were mapped during this specific call. Consequently, its error unrolling
path unmaps 0 bytes for this segment, **leaking the PTEs that were successfully
set by `__domain_mapping()` before the failure occurred**.

### Exploitation Mechanism
1. **Triggering the Leak**: An unprivileged user allocates a large DMA buffer
(e.g., via `/dev/dma_heap/system`) and maps it to a device (e.g., a GPU). By
inducing memory pressure, the user can cause the `GFP_ATOMIC` allocation in
`pfn_to_dma_pte()` to fail midway through the mapping. The mapping syscall
fails, but the first few pages remain mapped in the IOMMU page table.
2. **Freeing the Physical Pages**: The user closes the DMA buffer, returning the
physical pages to the system allocator (where they might be reused for sensitive
kernel data or page cache). However, the IOMMU still has valid PTEs pointing to
them. The IOVA is also freed back to the IOVA allocator.
3. **Reusing the IOVA**: The user allocates a new DMA buffer and maps it. The
IOVA allocator is highly likely to reuse the recently freed IOVA.
4. **Bypassing the Warning**: When `__domain_mapping()` is called for the new
buffer, it encounters the stale PTEs. It prints the warning seen in the bug
report (`ERROR: DMA PTE for vPFN ... already set`) and triggers a `WARN_ON(1)`.
Crucially, because `try_cmpxchg64_local()` fails, **it does not overwrite the
existing PTE**. The function simply continues and returns success.
5. **Arbitrary Memory Access**: The device is now configured to use the reused
IOVA, but the IOMMU translates this IOVA to the *old, freed* physical pages. The
user can instruct the device to perform DMA reads/writes to this IOVA, achieving
arbitrary physical memory read/write.

### Impact
- **Exploitable**: Yes. It provides a reliable primitive for arbitrary physical
memory read/write.
- **Accessible From Unprivileged Processes**: Yes. Unprivileged users can
allocate DMA buffers (e.g., via DMA heaps or GPU drivers) and trigger DMA
mappings.
- **Denial of Service**: Yes. The bug can also be used to corrupt memory and
crash the system. Exploitable:true FilesystemTrigger:false NetworkTrigger:false PeripheralTrigger:false RemoteTrigger:false Unprivileged:true UserNamespace:true VMGuestTrigger:false VMHostTrigger:false]

map[SimplifiedCRepro:// autogenerated by syzkaller (https://github.com/google/syzkaller)

#define _GNU_SOURCE 

#include <endian.h>
#include <errno.h>
#include <fcntl.h>
#include <stdarg.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>

#ifndef __NR_seccomp
#define __NR_seccomp 317
#endif

static bool write_file(const char* file, const char* what, ...)
{
	char buf[1024];
	va_list args;
	va_start(args, what);
	vsnprintf(buf, sizeof(buf), what, args);
	va_end(args);
	buf[sizeof(buf) - 1] = 0;
	int len = strlen(buf);
	int fd = open(file, O_WRONLY | O_CLOEXEC);
	if (fd == -1)
		return false;
	if (write(fd, buf, len) != len) {
		int err = errno;
		close(fd);
		errno = err;
		return false;
	}
	close(fd);
	return true;
}

static long syz_open_dev(volatile long a0, volatile long a1, volatile long a2)
{
	if (a0 == 0xc || a0 == 0xb) {
		char buf[128];
		sprintf(buf, "/dev/%s/%d:%d", a0 == 0xc ? "char" : "block", (uint8_t)a1, (uint8_t)a2);
		return open(buf, O_RDWR, 0);
	} else {
		unsigned long nb = a1;
		char buf[1024];
		char* hash;
		strncpy(buf, (char*)a0, sizeof(buf) - 1);
		buf[sizeof(buf) - 1] = 0;
		while ((hash = strchr(buf, '#'))) {
			*hash = '0' + (char)(nb % 10);
			nb /= 10;
		}
		return open(buf, a2 & ~O_CREAT, 0);
	}
}

static int inject_fault(int nth)
{
	int fd;
	fd = open("/proc/thread-self/fail-nth", O_RDWR);
	if (fd == -1)
	exit(1);
	char buf[16];
	sprintf(buf, "%d", nth);
	if (write(fd, buf, strlen(buf)) != (ssize_t)strlen(buf))
	exit(1);
	return fd;
}

static const char* setup_fault()
{
	int fd = open("/proc/self/make-it-fail", O_WRONLY);
	if (fd == -1)
		return "CONFIG_FAULT_INJECTION is not enabled";
	close(fd);
	fd = open("/proc/thread-self/fail-nth", O_WRONLY);
	if (fd == -1)
		return "kernel does not have systematic fault injection support";
	close(fd);
	static struct {
		const char* file;
		const char* val;
		bool fatal;
	} files[] = {
	    {"/sys/kernel/debug/failslab/ignore-gfp-wait", "N", true},
	    {"/sys/kernel/debug/fail_futex/ignore-private", "N", false},
	    {"/sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem", "N", false},
	    {"/sys/kernel/debug/fail_page_alloc/ignore-gfp-wait", "N", false},
	    {"/sys/kernel/debug/fail_page_alloc/min-order", "0", false},
	};
	unsigned i;
	for (i = 0; i < sizeof(files) / sizeof(files[0]); i++) {
		if (!write_file(files[i].file, files[i].val)) {
			if (files[i].fatal)
				return "failed to write fault injection file";
		}
	}
	return NULL;
}

uint64_t r[4] = {0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff};

int main(void)
{
		syscall(__NR_mmap, /*addr=*/0x1ffffffff000ul, /*len=*/0x1000ul, /*prot=*/0ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
	syscall(__NR_mmap, /*addr=*/0x200000000000ul, /*len=*/0x1000000ul, /*prot=PROT_WRITE|PROT_READ|PROT_EXEC*/7ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
	syscall(__NR_mmap, /*addr=*/0x200001000000ul, /*len=*/0x1000ul, /*prot=*/0ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
	const char* reason;
	(void)reason;
	if ((reason = setup_fault()))
		printf("the reproducer may not work as expected: fault injection setup failed: %s\n", reason);
				intptr_t res = 0;
	if (write(1, "executing program\n", sizeof("executing program\n") - 1)) {}
//  seccomp$SECCOMP_SET_MODE_FILTER_LISTENER arguments: [
//    op: const = 0x1 (8 bytes)
//    flags: seccomp_flags_listener = 0x0 (8 bytes)
//    arg: ptr[in, sock_fprog] {
//      sock_fprog {
//        len: len = 0x1 (2 bytes)
//        pad = 0x0 (6 bytes)
//        filter: ptr[in, array[sock_filter]] {
//          array[sock_filter] {
//            sock_filter {
//              code: int16 = 0x6 (2 bytes)
//              jt: int8 = 0x0 (1 bytes)
//              jf: int8 = 0x0 (1 bytes)
//              k: int32 = 0x7fff0006 (4 bytes)
//            }
//          }
//        }
//      }
//    }
//  ]
//  returns fd_seccomp
*(uint16_t*)0x2000000000c0 = 1;
*(uint64_t*)0x2000000000c8 = 0x200000000000;
*(uint16_t*)0x200000000000 = 6;
*(uint8_t*)0x200000000002 = 0;
*(uint8_t*)0x200000000003 = 0;
*(uint32_t*)0x200000000004 = 0x7fff0006;
	res = syscall(__NR_seccomp, /*op=*/1ul, /*flags=*/0ul, /*arg=*/0x2000000000c0ul);
	if (res != -1)
		r[0] = res;
//  openat$dma_heap arguments: [
//    fd: const = 0xffffffffffffff9c (8 bytes)
//    file: ptr[in, buffer] {
//      buffer: {2f 64 65 76 2f 64 6d 61 5f 68 65 61 70 2f 73 79 73 74 65 6d 00} (length 0x15)
//    }
//    flags: open_flags = 0x0 (4 bytes)
//    mode: const = 0x0 (2 bytes)
//  ]
//  returns fd_dma_heap
memcpy((void*)0x200000000000, "/dev/dma_heap/system\000", 21);
	res = syscall(__NR_openat, /*fd=*/0xffffffffffffff9cul, /*file=*/0x200000000000ul, /*flags=*/0, /*mode=*/0);
	if (res != -1)
		r[1] = res;
//  ioctl$DMA_HEAP_IOCTL_ALLOC arguments: [
//    fd: fd_dma_heap (resource)
//    cmd: const = 0xc0184800 (4 bytes)
//    arg: ptr[inout, dma_heap_allocation_data] {
//      dma_heap_allocation_data {
//        len: int64 = 0x1020004 (8 bytes)
//        fd: fd (resource)
//        fd_flags: dma_open_flags = 0x80000 (4 bytes)
//        heap_flags: const = 0x0 (8 bytes)
//      }
//    }
//  ]
*(uint64_t*)0x200000000100 = 0x1020004;
*(uint32_t*)0x200000000108 = r[0];
*(uint32_t*)0x20000000010c = 0x80000;
*(uint64_t*)0x200000000110 = 0;
	res = syscall(__NR_ioctl, /*fd=*/r[1], /*cmd=*/0xc0184800, /*arg=*/0x200000000100ul);
	if (res != -1)
r[2] = *(uint32_t*)0x200000000108;
//  syz_open_dev$dri arguments: [
//    dev: ptr[in, buffer] {
//      buffer: {2f 64 65 76 2f 64 72 69 2f 63 61 72 64 23 00} (length 0xf)
//    }
//    id: intptr = 0x48 (8 bytes)
//    flags: open_flags = 0x80000 (8 bytes)
//  ]
//  returns fd_dri
memcpy((void*)0x200000000180, "/dev/dri/card#\000", 15);
	res = -1;
res = syz_open_dev(/*dev=*/0x200000000180, /*id=*/0x48, /*flags=O_CLOEXEC*/0x80000);
	if (res != -1)
		r[3] = res;
//  ioctl$DRM_IOCTL_PRIME_FD_TO_HANDLE arguments: [
//    fd: fd_dri (resource)
//    cmd: const = 0xc00c642e (4 bytes)
//    arg: ptr[inout, drm_prime_handle$DRM_IOCTL_PRIME_FD_TO_HANDLE] {
//      drm_prime_handle$DRM_IOCTL_PRIME_FD_TO_HANDLE {
//        handle: drm_gem_handle (resource)
//        flags: const = 0x0 (4 bytes)
//        fd: fd_dri (resource)
//      }
//    }
//  ]
*(uint32_t*)0x2000000000c4 = 0;
*(uint32_t*)0x2000000000c8 = r[2];
	inject_fault(10);
	syscall(__NR_ioctl, /*fd=*/r[3], /*cmd=*/0xc00c642e, /*arg=*/0x2000000000c0ul);
	return 0;
}
]

map[KernelSrc:/app/workdir/cache/src/884ad8089d06fce59808d20d654d657f773bc146]

map[KernelObj:/app/workdir/cache/build/6097f467bd006650a608457d35109086ad2b6649]

map[Index:codesearch-index]

gemini-3.1-pro-preview

map[DenialOfService:true Exploitable:true FilesystemTrigger:false NetworkTrigger:false PeripheralTrigger:false RemoteTrigger:false Unprivileged:true UserNamespace:true VMGuestTrigger:false VMHostTrigger:false]

You are an experienced Linux kernel security engineer. Your task is to analyze given kernel bug report
and determine its security impact based on the following dimensions.

Use the provided tools to examine the source code, check for capability checks (e.g., capable(), ns_capable()),
and understand the nature of the bug. Analyze the given kernel build and configuration.
You can check the kernel config by grepping ".config" file; you can check kernel cmdline by greeping
".config" file for "CONFIG_CMDLINE=". Assume sysctl parameters have default values.
But analyze for the corresponding production build w/o debugging tools enabled (like KASAN, KMSAN, UBSAN).

Don't make assumptions; verify them with source code access. Try different strategies when analyzing the bug:
 - think of ways in which the vulnerable code is unreachable
 - or the other way around: try to come up with different ideas of how an unprivileged user can reach the bug
If still unsure err on the side of the bug being non-exploitable/not-accessible.
In the final reply, provide a reasoning for your assessment.

Analysis dimensions:

* Exploitable:
Determine if the bug can result in memory corruption or elevated privileges.
Memory safety issues are almost always exploitable (KASAN or UBSAN reports for use-after-free, out-of-bounds;
refcounting issues, corrupted lists, etc). When kernel is crashing on a completly wild pointer access
(e.g. user-space address, or non-canonical address, but not on NULL or address corresponding to KASAN shadow
for NULL address), including both data accesses and control tranfers, that's also usually implies possibility
of exploitation. Such reports usually say "unable to handle kernel paging request".
Uses of uninitialized values detected by KMSAN may be exploitable b/c attacker frequently can affect uninit
values with spraying techniques. However, for these exploitabability depends on how exactly the uninit value
is used in the code, and what it affects.

Think of what happens after the bug is triggered. Some bugs cause kernel panic and halt execution,
they are harder to exploit. For example, BUG reports halts the kernel. However, WARNING reports don't halt
execution in production builds. Debug bug detection tools (like KASAN, KMSAN, KCSAN, UBSAN) are also not enabled
in production builds, so attacker can freely exploit these bugs w/o being detected by these tools.
If you see an integer overflow, think how the overflowed value used later (if it's used as allocation size,
or an array index). If you see an out-of-bounds read, think if it's followed by an out-of-bounds write as well.
Some KCSAN data-races may be exploitable by skilled attackers as well. Think what data structures got corrupted
as the result of data races and how. However, note that kernel has lots of "benign" data races that don't lead
to any runtime misbehavior at all.

* Denial Of Service:
Determine if the bug can result in denial-of-service. Most bugs can, since they cause system crash,
hangs, deadlocks, or resource leaks. This is mostly applicable to WARNING bugs that won't cause system crash
in production. For these think what will be consequences of the violation of the kernel assumptions flagged
by the WARNING. In some cases the unexpected condition is also properly handled by the normal control flow
(e.g. with "if (WARN_ON(...))"), these won't cause denial-of-service. If the condition is not handled,
then it may or may not cause denial-of-service.

* Accessible From Unprivileged Processes:
Determine if the bug can be reached from a typical (non-root) user process that does NOT have any special capabilities
(like CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_NET_RAW, CAP_PERFMON) or access to device nodes restricted to root.
Assume that unprivileged_bpf_disabled=1, that is eBPF loading is not accessible. However, cBPF (classical BPF)
is still accessible to non-root processes.
Assume that user namespaces are not accessible, that is, the process cannot get the mentioned capabilities even
within a new user namespace (checked by ns_capable() function in the kernel sources).

* Accessible From User Namespaces:
Determine if the bug can be reached within a user-namespace where the process has all capabilities
(including CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_NET_RAW, CAP_PERFMON). Such capabilities are checked with ns_capable()
function in the kernel sources.

* VM Guest Trigger:
Determine if the bug can be triggered from the context of a typical KVM guest (e.g., set up by a QEMU VMM).
Consider accesses to standard Linux host paravirtualized features (virtio-blk, virtio-net, etc.),
and handling of VM exits in the KVM code.

* VM Host Trigger in The Confidetial Computing Context:
Determine if the bug can be triggered in a confidential computing guest kernel from the context of a KVM host.
Consider access to standard Linux guest paravirtualized features (virtio-blk, virtio-net, etc.).

* Ethernet Network Trigger:
Determine if the bug can be triggered by processing ingress network Ethernet traffic, either directly (network stack)
or via drivers exposed to network data.

* Other Remote Trigger:
Determine if the bug can be triggered by processing remote traffic other than Ethernet (Wifi, Bluetooth, NFC, etc).

* Peripheral Trigger:
Determine if the bug can be triggered via an untrusted peripheral device that can be physically plugged
into a system, such as a USB device or a niche hardware driver handling external hardware inputs.
This is particularly important for mobile and desktop environments where users can plug in unknown devices.

* Malicious Filesystem Trigger:
Determine if the bug can be triggered by the kernel mounting and parsing a malicious filesystem image.
This is highly critical for Desktop and Mobile environments where external media or downloaded images
might be auto-mounted.

Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

The kernel bug report is:

DMAR: ERROR: DMA PTE for vPFN 0xbe300 already set (to 55400003 not 51800003)
------------[ cut here ]------------
WARNING: CPU: 0 PID: 6199 at drivers/iommu/intel/iommu.c:1679 __domain_mapping drivers/iommu/intel/iommu.c:1679 [inline]
WARNING: CPU: 0 PID: 6199 at drivers/iommu/intel/iommu.c:1679 intel_iommu_map drivers/iommu/intel/iommu.c:3593 [inline]
WARNING: CPU: 0 PID: 6199 at drivers/iommu/intel/iommu.c:1679 intel_iommu_map_pages+0xaa7/0x1520 drivers/iommu/intel/iommu.c:3612
Modules linked in:
CPU: 0 UID: 0 PID: 6199 Comm: syz.2.40 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014
RIP: 0010:__domain_mapping drivers/iommu/intel/iommu.c:1679 [inline]
RIP: 0010:intel_iommu_map drivers/iommu/intel/iommu.c:3593 [inline]
RIP: 0010:intel_iommu_map_pages+0xaa7/0x1520 drivers/iommu/intel/iommu.c:3612
Code: ba 26 fc 8b 2d 1a e5 be 09 31 ff 89 ee e8 b1 06 48 fc 85 ed 74 0e e8 68 0b 48 fc 83 ed 01 89 2d ff e4 be 09 e8 5a 0b 48 fc 90 <0f> 0b 90 e9 da fa ff ff e8 4c 0b 48 fc 4c 8b 7c 24 40 48 8d 43 ff
RSP: 0018:ffffc900038575e0 EFLAGS: 00010293
RAX: 0000000000000000 RBX: ffff88803b24e800 RCX: ffffffff8573920f
RDX: ffff88802577c880 RSI: ffffffff85739226 RDI: 0000000000000005
RBP: 0000000000000004 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000005 R11: 0000000000000000 R12: 00000000000ffb00
R13: 0000000000000001 R14: 0000000051800003 R15: 0000000000000002
FS:  0000555564edf500(0000) GS:ffff8880d66b2000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005555884bf808 CR3: 000000005203d000 CR4: 0000000000352ef0
Call Trace:
 <TASK>
 iommu_map_nosync+0x337/0x700 drivers/iommu/iommu.c:2505
 iommu_map_sg+0x1c1/0x9d0 drivers/iommu/iommu.c:2677
 iommu_dma_map_sg+0x88c/0xde0 drivers/iommu/dma-iommu.c:1483
 __dma_map_sg_attrs+0x293/0x590 kernel/dma/mapping.c:216
 dma_map_sgtable+0x78/0x100 kernel/dma/mapping.c:294
 system_heap_map_dma_buf+0x66/0xf0 drivers/dma-buf/heaps/system_heap.c:124
 dma_buf_map_attachment+0x15e/0x5f0 drivers/dma-buf/dma-buf.c:1126
 dma_buf_map_attachment_unlocked+0x9e/0x150 drivers/dma-buf/dma-buf.c:1196
 drm_gem_prime_import_dev drivers/gpu/drm/drm_prime.c:999 [inline]
 drm_gem_prime_import_dev+0x166/0x440 drivers/gpu/drm/drm_prime.c:971
 virtgpu_gem_prime_import+0x16c/0x800 drivers/gpu/drm/virtio/virtgpu_prime.c:316
 drm_gem_prime_fd_to_handle+0x1a6/0x5f0 drivers/gpu/drm/drm_prime.c:317
 drm_prime_fd_to_handle_ioctl+0xd6/0x110 drivers/gpu/drm/drm_prime.c:375
 drm_ioctl_kernel+0x1f1/0x3e0 drivers/gpu/drm/drm_ioctl.c:796
 drm_ioctl+0x5c9/0xc30 drivers/gpu/drm/drm_ioctl.c:893
 vfs_ioctl fs/ioctl.c:51 [inline]
 __do_sys_ioctl fs/ioctl.c:598 [inline]
 __se_sys_ioctl fs/ioctl.c:584 [inline]
 __x64_sys_ioctl+0x18e/0x210 fs/ioctl.c:584
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0xcd/0x4e0 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f377a58ec29
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffdfa8b0f78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f377a7d5fa0 RCX: 00007f377a58ec29
RDX: 00002000000000c0 RSI: 00000000c00c642e RDI: 0000000000000005
RBP: 00007ffdfa8b0fd0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002
R13: 00007f377a7d5fa0 R14: 00007f377a7d5fa0 R15: 0000000000000003
 </TASK>




It is reproducible with the followint program.
Keep in mind that it may lack the precise threading, sandboxing, and some arguments of a working reproducer.
But it should give an idea of the involved syscalls.

// autogenerated by syzkaller (https://github.com/google/syzkaller)

#define _GNU_SOURCE 

#include <endian.h>
#include <errno.h>
#include <fcntl.h>
#include <stdarg.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>

#ifndef __NR_seccomp
#define __NR_seccomp 317
#endif

static bool write_file(const char* file, const char* what, ...)
{
	char buf[1024];
	va_list args;
	va_start(args, what);
	vsnprintf(buf, sizeof(buf), what, args);
	va_end(args);
	buf[sizeof(buf) - 1] = 0;
	int len = strlen(buf);
	int fd = open(file, O_WRONLY | O_CLOEXEC);
	if (fd == -1)
		return false;
	if (write(fd, buf, len) != len) {
		int err = errno;
		close(fd);
		errno = err;
		return false;
	}
	close(fd);
	return true;
}

static long syz_open_dev(volatile long a0, volatile long a1, volatile long a2)
{
	if (a0 == 0xc || a0 == 0xb) {
		char buf[128];
		sprintf(buf, "/dev/%s/%d:%d", a0 == 0xc ? "char" : "block", (uint8_t)a1, (uint8_t)a2);
		return open(buf, O_RDWR, 0);
	} else {
		unsigned long nb = a1;
		char buf[1024];
		char* hash;
		strncpy(buf, (char*)a0, sizeof(buf) - 1);
		buf[sizeof(buf) - 1] = 0;
		while ((hash = strchr(buf, '#'))) {
			*hash = '0' + (char)(nb % 10);
			nb /= 10;
		}
		return open(buf, a2 & ~O_CREAT, 0);
	}
}

static int inject_fault(int nth)
{
	int fd;
	fd = open("/proc/thread-self/fail-nth", O_RDWR);
	if (fd == -1)
	exit(1);
	char buf[16];
	sprintf(buf, "%d", nth);
	if (write(fd, buf, strlen(buf)) != (ssize_t)strlen(buf))
	exit(1);
	return fd;
}

static const char* setup_fault()
{
	int fd = open("/proc/self/make-it-fail", O_WRONLY);
	if (fd == -1)
		return "CONFIG_FAULT_INJECTION is not enabled";
	close(fd);
	fd = open("/proc/thread-self/fail-nth", O_WRONLY);
	if (fd == -1)
		return "kernel does not have systematic fault injection support";
	close(fd);
	static struct {
		const char* file;
		const char* val;
		bool fatal;
	} files[] = {
	    {"/sys/kernel/debug/failslab/ignore-gfp-wait", "N", true},
	    {"/sys/kernel/debug/fail_futex/ignore-private", "N", false},
	    {"/sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem", "N", false},
	    {"/sys/kernel/debug/fail_page_alloc/ignore-gfp-wait", "N", false},
	    {"/sys/kernel/debug/fail_page_alloc/min-order", "0", false},
	};
	unsigned i;
	for (i = 0; i < sizeof(files) / sizeof(files[0]); i++) {
		if (!write_file(files[i].file, files[i].val)) {
			if (files[i].fatal)
				return "failed to write fault injection file";
		}
	}
	return NULL;
}

uint64_t r[4] = {0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff, 0xffffffffffffffff};

int main(void)
{
		syscall(__NR_mmap, /*addr=*/0x1ffffffff000ul, /*len=*/0x1000ul, /*prot=*/0ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
	syscall(__NR_mmap, /*addr=*/0x200000000000ul, /*len=*/0x1000000ul, /*prot=PROT_WRITE|PROT_READ|PROT_EXEC*/7ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
	syscall(__NR_mmap, /*addr=*/0x200001000000ul, /*len=*/0x1000ul, /*prot=*/0ul, /*flags=MAP_FIXED|MAP_ANONYMOUS|MAP_PRIVATE*/0x32ul, /*fd=*/(intptr_t)-1, /*offset=*/0ul);
	const char* reason;
	(void)reason;
	if ((reason = setup_fault()))
		printf("the reproducer may not work as expected: fault injection setup failed: %s\n", reason);
				intptr_t res = 0;
	if (write(1, "executing program\n", sizeof("executing program\n") - 1)) {}
//  seccomp$SECCOMP_SET_MODE_FILTER_LISTENER arguments: [
//    op: const = 0x1 (8 bytes)
//    flags: seccomp_flags_listener = 0x0 (8 bytes)
//    arg: ptr[in, sock_fprog] {
//      sock_fprog {
//        len: len = 0x1 (2 bytes)
//        pad = 0x0 (6 bytes)
//        filter: ptr[in, array[sock_filter]] {
//          array[sock_filter] {
//            sock_filter {
//              code: int16 = 0x6 (2 bytes)
//              jt: int8 = 0x0 (1 bytes)
//              jf: int8 = 0x0 (1 bytes)
//              k: int32 = 0x7fff0006 (4 bytes)
//            }
//          }
//        }
//      }
//    }
//  ]
//  returns fd_seccomp
*(uint16_t*)0x2000000000c0 = 1;
*(uint64_t*)0x2000000000c8 = 0x200000000000;
*(uint16_t*)0x200000000000 = 6;
*(uint8_t*)0x200000000002 = 0;
*(uint8_t*)0x200000000003 = 0;
*(uint32_t*)0x200000000004 = 0x7fff0006;
	res = syscall(__NR_seccomp, /*op=*/1ul, /*flags=*/0ul, /*arg=*/0x2000000000c0ul);
	if (res != -1)
		r[0] = res;
//  openat$dma_heap arguments: [
//    fd: const = 0xffffffffffffff9c (8 bytes)
//    file: ptr[in, buffer] {
//      buffer: {2f 64 65 76 2f 64 6d 61 5f 68 65 61 70 2f 73 79 73 74 65 6d 00} (length 0x15)
//    }
//    flags: open_flags = 0x0 (4 bytes)
//    mode: const = 0x0 (2 bytes)
//  ]
//  returns fd_dma_heap
memcpy((void*)0x200000000000, "/dev/dma_heap/system\000", 21);
	res = syscall(__NR_openat, /*fd=*/0xffffffffffffff9cul, /*file=*/0x200000000000ul, /*flags=*/0, /*mode=*/0);
	if (res != -1)
		r[1] = res;
//  ioctl$DMA_HEAP_IOCTL_ALLOC arguments: [
//    fd: fd_dma_heap (resource)
//    cmd: const = 0xc0184800 (4 bytes)
//    arg: ptr[inout, dma_heap_allocation_data] {
//      dma_heap_allocation_data {
//        len: int64 = 0x1020004 (8 bytes)
//        fd: fd (resource)
//        fd_flags: dma_open_flags = 0x80000 (4 bytes)
//        heap_flags: const = 0x0 (8 bytes)
//      }
//    }
//  ]
*(uint64_t*)0x200000000100 = 0x1020004;
*(uint32_t*)0x200000000108 = r[0];
*(uint32_t*)0x20000000010c = 0x80000;
*(uint64_t*)0x200000000110 = 0;
	res = syscall(__NR_ioctl, /*fd=*/r[1], /*cmd=*/0xc0184800, /*arg=*/0x200000000100ul);
	if (res != -1)
r[2] = *(uint32_t*)0x200000000108;
//  syz_open_dev$dri arguments: [
//    dev: ptr[in, buffer] {
//      buffer: {2f 64 65 76 2f 64 72 69 2f 63 61 72 64 23 00} (length 0xf)
//    }
//    id: intptr = 0x48 (8 bytes)
//    flags: open_flags = 0x80000 (8 bytes)
//  ]
//  returns fd_dri
memcpy((void*)0x200000000180, "/dev/dri/card#\000", 15);
	res = -1;
res = syz_open_dev(/*dev=*/0x200000000180, /*id=*/0x48, /*flags=O_CLOEXEC*/0x80000);
	if (res != -1)
		r[3] = res;
//  ioctl$DRM_IOCTL_PRIME_FD_TO_HANDLE arguments: [
//    fd: fd_dri (resource)
//    cmd: const = 0xc00c642e (4 bytes)
//    arg: ptr[inout, drm_prime_handle$DRM_IOCTL_PRIME_FD_TO_HANDLE] {
//      drm_prime_handle$DRM_IOCTL_PRIME_FD_TO_HANDLE {
//        handle: drm_gem_handle (resource)
//        flags: const = 0x0 (4 bytes)
//        fd: fd_dri (resource)
//      }
//    }
//  ]
*(uint32_t*)0x2000000000c4 = 0;
*(uint32_t*)0x2000000000c8 = r[2];
	inject_fault(10);
	syscall(__NR_ioctl, /*fd=*/r[3], /*cmd=*/0xc00c642e, /*arg=*/0x2000000000c0ul);
	return 0;
}

The bug is a critical security vulnerability in the Intel IOMMU driver that leads to a Use-After-Free (UAF) of physical pages, allowing an unprivileged user to achieve arbitrary physical memory read/write and full system compromise.

### Root Cause Analysis
The vulnerability lies in the error handling path of `intel_iommu_map_pages()` in `drivers/iommu/intel/iommu.c`. 

When a device driver maps a scatterlist for DMA (e.g., during a `DRM_IOCTL_PRIME_FD_TO_HANDLE` ioctl), the IOMMU core calls `iommu_map_nosync()`, which in turn calls the IOMMU driver's `map_pages` callback (`intel_iommu_map_pages()`). 

`intel_iommu_map_pages()` calls `intel_iommu_map()` (which wraps `__domain_mapping()`) to map the pages. If `__domain_mapping()` fails in the middle of mapping a large range (e.g., due to a `GFP_ATOMIC` memory allocation failure in `pfn_to_dma_pte()` under memory pressure), it returns an error like `-ENOMEM`. 

However, `intel_iommu_map_pages()` has a flaw where it **does not update the `mapped` output parameter** if an error is returned:
```c
	ret = intel_iommu_map(domain, iova, paddr, size, prot, gfp);
	if (!ret && mapped)
		*mapped = size; // Only updated on success!

	return ret;
```
Because `mapped` remains `0`, the caller `iommu_map_nosync()` assumes that 0 bytes were mapped during this specific call. Consequently, its error unrolling path unmaps 0 bytes for this segment, **leaking the PTEs that were successfully set by `__domain_mapping()` before the failure occurred**.

### Exploitation Mechanism
1. **Triggering the Leak**: An unprivileged user allocates a large DMA buffer (e.g., via `/dev/dma_heap/system`) and maps it to a device (e.g., a GPU). By inducing memory pressure, the user can cause the `GFP_ATOMIC` allocation in `pfn_to_dma_pte()` to fail midway through the mapping. The mapping syscall fails, but the first few pages remain mapped in the IOMMU page table.
2. **Freeing the Physical Pages**: The user closes the DMA buffer, returning the physical pages to the system allocator (where they might be reused for sensitive kernel data or page cache). However, the IOMMU still has valid PTEs pointing to them. The IOVA is also freed back to the IOVA allocator.
3. **Reusing the IOVA**: The user allocates a new DMA buffer and maps it. The IOVA allocator is highly likely to reuse the recently freed IOVA.
4. **Bypassing the Warning**: When `__domain_mapping()` is called for the new buffer, it encounters the stale PTEs. It prints the warning seen in the bug report (`ERROR: DMA PTE for vPFN ... already set`) and triggers a `WARN_ON(1)`. Crucially, because `try_cmpxchg64_local()` fails, **it does not overwrite the existing PTE**. The function simply continues and returns success.
5. **Arbitrary Memory Access**: The device is now configured to use the reused IOVA, but the IOMMU translates this IOVA to the *old, freed* physical pages. The user can instruct the device to perform DMA reads/writes to this IOVA, achieving arbitrary physical memory read/write.

### Impact
- **Exploitable**: Yes. It provides a reliable primitive for arbitrary physical memory read/write.
- **Accessible From Unprivileged Processes**: Yes. Unprivileged users can allocate DMA buffers (e.g., via DMA heaps or GPU drivers) and trigger DMA mappings.
- **Denial of Service**: Yes. The bug can also be used to corrupt memory and crash the system.

gemini-3.1-pro-preview

							input: 9986
							output: 18
							thoughts: 944
						

**Analyzing a DMAR PTE Collision in Intel IOMMU**

Okay, so I've got this bug report, and it's screaming a DMAR error: "DMA PTE for vPFN 0xbe300 already set." It looks like an Intel IOMMU issue, specifically a collision, because the PTE has been set to the wrong value. The kernel stack trace tells me the problem's hitting in `__domain_mapping` within `drivers/iommu/intel/iommu.c`, which is usually a bad sign, particularly when it comes to memory management. The fact that the call stack includes calls to `iommu_map_nosync`, `iommu_map_sg`, and ultimately `dma_buf_map_attachment` points to DMA buffer mapping and IOMMU interactions.

The call trace starts with the `dma_buf_map_attachment`, and flows up through the DRM prime code, specifically `drm_gem_prime_import_dev`. This suggests the problem occurs when importing a DMA buffer via PRIME, likely related to the virtual GPU. The reproducer steps clarify the issue: the steps involve seccomp, opening a system DMA heap, DMA allocation, opening the card, injecting a fault (interesting!), and then calling `DRM_IOCTL_PRIME_FD_TO_HANDLE`. Injecting a fault after heap allocation suggests we may be looking at a race condition or an edge case related to fault handling within the IOMMU translation logic.

My immediate thought is to zero in on that `__domain_mapping` function in `drivers/iommu/intel/iommu.c`. The fact that it's an inline function is good, it allows us to examine the context of the call. I need to understand what's happening just before line 1679. I want to see how this particular virtual page frame number is getting mapped, and what could cause it to already be set to an incorrect value. I'll need to scrutinize the locking and synchronization within this function and the related IOMMU mapping functions to determine if there's a race, or perhaps a double-free, or a similar error. The fault injection step is crucial here. I bet it is trying to work around some memory management in the iommu and it is falling foul of a race. This all looks like a fun one.

map[Expression:__domain_mapping]

map[Output:drivers/iommu/intel/iommu.c=1603=static int
drivers/iommu/intel/iommu.c:1604:__domain_mapping(struct dmar_domain *domain, unsigned long iov_pfn,
drivers/iommu/intel/iommu.c-1605-		 unsigned long phys_pfn, unsigned long nr_pages, int prot,
--
drivers/iommu/intel/iommu.c=3561=static int intel_iommu_map(struct iommu_domain *domain,
--
drivers/iommu/intel/iommu.c-3592-	size = aligned_nrpages(hpa, size);
drivers/iommu/intel/iommu.c:3593:	return __domain_mapping(dmar_domain, iova >> VTD_PAGE_SHIFT,
drivers/iommu/intel/iommu.c-3594-				hpa >> VTD_PAGE_SHIFT, size, prot, gfp);
]

gemini-3.1-pro-preview

							input: 11723
							output: 17
							thoughts: 31
						

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Analyzing Intel IOMMU Initialization Code (iommu.c)**

Okay, time to dive into some code. I need to get a handle on what's happening with the Intel IOMMU initialization, specifically within the `drivers/iommu/intel/iommu.c` file. I'm focusing on lines 1600 to 1700. Let's see... I'm expecting to find some crucial aspects of the boot-up or setup phase. My initial thoughts are that this section is most likely dealing with one or more of these core functionalities:

*   **Hardware Detection and Enumeration:** Perhaps this section is involved in identifying the specific Intel IOMMU hardware present on the system. It could be probing the PCI bus or memory-mapped configuration space to determine the IOMMU's features, capabilities, and registers. I'm looking for evidence of register access and decoding of the returned values.

*   **Page Table Setup/Initialization:** This is a key area. The IOMMU relies on page tables to perform address translation. I'm anticipating that this region contains the code to set up the initial page table structures. That likely includes allocating memory for these tables, zeroing them, and populating them with initial mappings – possibly identity mappings or mappings related to the kernel's early boot stage. I'm particularly interested in seeing how they're handling the root entry (or the top-level table)

*   **Interrupt Handling Configuration:** Another possibility is that this code deals with IOMMU interrupt management. It might be setting up interrupt routing or configuration for IOMMU-related error interrupts or event notifications. This could involve configuring interrupt controllers like the MSI-X, or potentially writing to registers related to interrupt masking or enabling.

*   **DMA Remapping Setup** This could contain crucial logic for enabling and configuring DMA remapping to allow devices to perform I/O to memory. I'm looking for the code responsible for configuring the default or initial IOMMU settings. This could include things like configuring the default domain, enabling the IOMMU, or setting up any initial access control lists.

Essentially, I want to understand what initial steps are being taken before any devices start using the IOMMU. Let's start reading and see what's actually there. I'll need to pay close attention to the register accesses, function calls, and the control flow to piece together the exact functionality of this part of the initialization. I'll expect some pointer dereferencing and bitwise operations to understand what's being set.

map[Expression:WARN_ON]

map[Output:
Full output is too long, showing 500 out of 130658 lines.
Use more precise expression if possible.

[Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst=78=lock-acquisition and lock-release functions::
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-104-   25
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:105:   26 WARN_ON(r1 == 0 && r2 == 0 && r3 == 0);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-106-
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:107:The ``WARN_ON()`` is evaluated at "the end of time",
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-108-after all changes have propagated throughout the system.
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst=109=Without the ``smp_mb__after_unlock_lock()`` provided by the
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:110:acquisition functions, this ``WARN_ON()`` could trigger, for example
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-111-on PowerPC.
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst=112=The ``smp_mb__after_unlock_lock()`` invocations prevent this
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:113:``WARN_ON()`` from triggering.
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-114-
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=755=example illustrates this:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-765-       7     r2 = READ_ONCE(x);
 Documentation/RCU/Design/Requirements/Requirements.rst:766:       8     WARN_ON(!r2); /* BUG!!! */
 Documentation/RCU/Design/Requirements/Requirements.rst-767-       9   }
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=779=If the thread0() function's rcu_read_lock() excluded the
 Documentation/RCU/Design/Requirements/Requirements.rst:780:thread1() function's update, the WARN_ON() could never fire. But
 Documentation/RCU/Design/Requirements/Requirements.rst-781-the fact is that rcu_read_lock() does not exclude much of anything
 Documentation/RCU/Design/Requirements/Requirements.rst=782=aside from subsequent grace periods, of which thread1() has none, so
 Documentation/RCU/Design/Requirements/Requirements.rst:783:the WARN_ON() can and does fire.
 Documentation/RCU/Design/Requirements/Requirements.rst-784-
 --
 Documentation/RCU/lockdep.rst=25=aren't certain (for example, if CONFIG_DEBUG_LOCK_ALLOC is not set).
 Documentation/RCU/lockdep.rst:26:This prevents things like WARN_ON(!rcu_read_lock_held()) from giving false
 Documentation/RCU/lockdep.rst-27-positives when lockdep is disabled.
 --
 Documentation/admin-guide/clearing-warn-once.rst:1:Clearing WARN_ONCE
 Documentation/admin-guide/clearing-warn-once.rst-2-------------------
 Documentation/admin-guide/clearing-warn-once.rst-3-
 Documentation/admin-guide/clearing-warn-once.rst:4:WARN_ONCE / WARN_ON_ONCE / printk_once only emit a message once.
 Documentation/admin-guide/clearing-warn-once.rst-5-
 --
 Documentation/admin-guide/kernel-parameters.txt-3921-			this means that duplicate request_module() calls will
 Documentation/admin-guide/kernel-parameters.txt:3922:			trigger a WARN_ON() instead of a pr_warn(). Note that
 Documentation/admin-guide/kernel-parameters.txt:3923:			if MODULE_DEBUG_AUTOLOAD_DUPS_TRACE is set, WARN_ON()s
 Documentation/admin-guide/kernel-parameters.txt-3924-			will always be issued and this option does nothing.
 --
 Documentation/admin-guide/kernel-parameters.txt-5525-			This wake_up() will be accompanied by a
 Documentation/admin-guide/kernel-parameters.txt:5526:			WARN_ONCE() splat and an ftrace_dump().
 Documentation/admin-guide/kernel-parameters.txt-5527-
 --
 Documentation/dev-tools/checkpatch.rst=272=API usage
 --
 Documentation/dev-tools/checkpatch.rst-286-    BUG() or BUG_ON() should be avoided totally.
 Documentation/dev-tools/checkpatch.rst:287:    Use WARN() and WARN_ON() instead, and handle the "impossible"
 Documentation/dev-tools/checkpatch.rst-288-    error condition as gracefully as possible.
 --
 Documentation/driver-api/media/v4l2-controls.rst=814=There can be only one notify function per control handler. Any attempt
 Documentation/driver-api/media/v4l2-controls.rst:815:to set another notify function will cause a WARN_ON.
 Documentation/driver-api/media/v4l2-controls.rst-816-
 --
 Documentation/driver-api/mtdnand.rst=272=unregisters the partitions in the MTD layer.
 --
 Documentation/driver-api/mtdnand.rst-279-        /* Unregister device */
 Documentation/driver-api/mtdnand.rst:280:        WARN_ON(mtd_device_unregister(board_mtd));
 Documentation/driver-api/mtdnand.rst-281-        /* Release resources */
 --
 Documentation/filesystems/porting.rst=818=just get rid of it.  Synchronous work (e.g. the stuff that can't
 Documentation/filesystems/porting.rst:819:be done from an RCU callback, or any WARN_ON() where we want the
 Documentation/filesystems/porting.rst-820-stack trace) *might* be movable to ->evict_inode(); however,
 --
 Documentation/gpu/introduction.rst=35=Instead locking should be check at runtime using e.g.
 Documentation/gpu/introduction.rst:36:``WARN_ON(!mutex_is_locked(...));``. Since it's much easier to ignore
 Documentation/gpu/introduction.rst-37-documentation than runtime noise this provides more value. And on top of
 --
 Documentation/gpu/todo.rst=153=interfaces to fix these issues:
 --
 Documentation/gpu/todo.rst-161-  Except for some driver code this is done. This task should be finished by
 Documentation/gpu/todo.rst:162:  adding WARN_ON(!drm_drv_uses_atomic_modeset) in drm_modeset_lock_all().
 Documentation/gpu/todo.rst-163-
 --
 Documentation/gpu/todo.rst=478=double-calls prepare/enable/disable/unprepare. Eventually that should probably
 Documentation/gpu/todo.rst:479:be turned into a WARN_ON() or somehow made louder.
 Documentation/gpu/todo.rst-480-
 --
 Documentation/leds/leds-class.rst=147=led_classdev_notify_brightness_hw_changed on a classdev not registered with
 Documentation/leds/leds-class.rst:148:the LED_BRIGHT_HW_CHANGED flag is a bug and will trigger a WARN_ON.
 Documentation/leds/leds-class.rst-149-
 --
 Documentation/locking/lockdep-design.rst=340=Exceeding this number will trigger the following lockdep warning::
 Documentation/locking/lockdep-design.rst-341-
 Documentation/locking/lockdep-design.rst:342:	(DEBUG_LOCKS_WARN_ON(id >= MAX_LOCKDEP_KEYS))
 Documentation/locking/lockdep-design.rst-343-
 --
 Documentation/memory-barriers.txt=1558=of optimizations:
 --
 Documentation/memory-barriers.txt-1732-     interrupt handler returns with interrupts enabled, you will get a
 Documentation/memory-barriers.txt:1733:     WARN_ONCE() splat.)
 Documentation/memory-barriers.txt-1734-
 --
 Documentation/process/coding-style.rst=1225=BUG_ON(), or VM_BUG_ON(). Instead, use a WARN*() variant, preferably
 Documentation/process/coding-style.rst:1226:WARN_ON_ONCE(), and possibly with recovery code. Recovery code is not
 Documentation/process/coding-style.rst-1227-required if there is no reasonable way to at least partially recover.
 --
 Documentation/process/coding-style.rst=1231=good justification.
 Documentation/process/coding-style.rst-1232-
 Documentation/process/coding-style.rst:1233:Use WARN_ON_ONCE() rather than WARN() or WARN_ON()
 Documentation/process/coding-style.rst-1234-**************************************************
 Documentation/process/coding-style.rst-1235-
 Documentation/process/coding-style.rst:1236:WARN_ON_ONCE() is generally preferred over WARN() or WARN_ON(), because it
 Documentation/process/coding-style.rst-1237-is common for a given warning condition, if it occurs at all, to occur
 --
 Documentation/process/debugging/userspace_debugging_guide.rst=210=Kernel panic analysis tools
 --
 Documentation/process/debugging/userspace_debugging_guide.rst-243-     ae8:	f9418015 	ldr	x21, [x0, #768]
 Documentation/process/debugging/userspace_debugging_guide.rst:244:        if (WARN_ON(!desc))
 Documentation/process/debugging/userspace_debugging_guide.rst-245-     aec:	b4000654 	cbz	x20, bb4 <rkvdec_device_run+0xec>
 --
 Documentation/process/deprecated.rst=32=BUG() and BUG_ON()
 Documentation/process/deprecated.rst-33-------------------
 Documentation/process/deprecated.rst:34:Use WARN() and WARN_ON() instead, and handle the "impossible"
 Documentation/process/deprecated.rst-35-error condition as gracefully as possible. While the BUG()-family
 --
 Documentation/sound/kernel-api/writing-an-alsa-driver.rst=4040=When no debug flag is set, this macro is ignored.
 --
 Documentation/sound/kernel-api/writing-an-alsa-driver.rst-4045-:c:func:`snd_BUG_ON()` macro is similar with
 Documentation/sound/kernel-api/writing-an-alsa-driver.rst:4046::c:func:`WARN_ON()` macro. For example, snd_BUG_ON(!pointer); or
 Documentation/sound/kernel-api/writing-an-alsa-driver.rst-4047-it can be used as the condition, if (snd_BUG_ON(non_zero_is_bug))
 --
 Documentation/trace/boottime-trace.rst=43=kernel.traceoff_on_warning
 Documentation/trace/boottime-trace.rst:44:   Stop tracing if WARN_ON() occurs.
 Documentation/trace/boottime-trace.rst-45-
 --
 Documentation/translations/it_IT/locking/lockdep-design.rst=342=seguente avviso::
 Documentation/translations/it_IT/locking/lockdep-design.rst-343-
 Documentation/translations/it_IT/locking/lockdep-design.rst:344:	(DEBUG_LOCKS_WARN_ON(id >= MAX_LOCKDEP_KEYS))
 Documentation/translations/it_IT/locking/lockdep-design.rst-345-
 --
 Documentation/translations/it_IT/process/deprecated.rst=37=BUG() e BUG_ON()
 Documentation/translations/it_IT/process/deprecated.rst-38-----------------
 Documentation/translations/it_IT/process/deprecated.rst:39:Al loro posto usate WARN() e WARN_ON() per gestire le
 Documentation/translations/it_IT/process/deprecated.rst-40-condizioni "impossibili" e gestitele come se fosse possibile farlo.
 --
 Documentation/translations/ko_KR/memory-barriers.txt=1535=READ_ONCE() 와 WRITE_ONCE() 함수는 싱글 쓰레드 코드에서는 문제 없지만 동시성이
 --
 Documentation/translations/ko_KR/memory-barriers.txt-1700-     중첩된 인터럽트는 보통 잘 일어나지 않음도 기억해 두세요, 실제로, 어떤
 Documentation/translations/ko_KR/memory-barriers.txt:1701:     인터럽트 핸들러가 인터럽트가 활성화된 채로 리턴하면 WARN_ONCE() 가
 Documentation/translations/ko_KR/memory-barriers.txt-1702-     실행됩니다.)
 --
 Documentation/translations/sp_SP/memory-barriers.txt=1622=algunos ejemplos de tal tipo de optimizaciones:
 --
 Documentation/translations/sp_SP/memory-barriers.txt-1803-     hecho, si un controlador de interrupciones regresa con interrupciones
 Documentation/translations/sp_SP/memory-barriers.txt:1804:     habilitadas, obtendrá un WARN_ONCE().)
 Documentation/translations/sp_SP/memory-barriers.txt-1805-
 --
 Documentation/translations/sp_SP/process/coding-style.rst=1247=BUG(), BUG_ON() o VM_BUG_ON(). En su lugar, use una variante WARN*(),
 Documentation/translations/sp_SP/process/coding-style.rst:1248:preferiblemente WARN_ON_ONCE(), y posiblemente con código de recuperación.
 Documentation/translations/sp_SP/process/coding-style.rst-1249-El código de recuperación no es requerido si no hay una forma razonable de
 --
 Documentation/translations/sp_SP/process/coding-style.rst=1254=aún pueden usar BUG(), pero necesitan una buena justificación.
 Documentation/translations/sp_SP/process/coding-style.rst-1255-
 Documentation/translations/sp_SP/process/coding-style.rst:1256:Use WARN_ON_ONCE() en lugar de WARN() o WARN_ON()
 Documentation/translations/sp_SP/process/coding-style.rst-1257-*************************************************
 Documentation/translations/sp_SP/process/coding-style.rst-1258-
 Documentation/translations/sp_SP/process/coding-style.rst:1259:Generalmente, se prefiere WARN_ON_ONCE() a WARN() o WARN_ON(), porque es
 Documentation/translations/sp_SP/process/coding-style.rst-1260-común que una condición de advertencia dada, si ocurre, ocurra varias
 --
 Documentation/translations/sp_SP/process/deprecated.rst=35=BUG() y BUG_ON()
 Documentation/translations/sp_SP/process/deprecated.rst-36-----------------
 Documentation/translations/sp_SP/process/deprecated.rst:37:Use WARN() y WARN_ON() en su lugar, y gestione las condiciones de error
 Documentation/translations/sp_SP/process/deprecated.rst-38-"imposibles" tan elegantemente como se pueda. Mientras que la familia de
 --
 Documentation/translations/zh_CN/admin-guide/clearing-warn-once.rst:1:清除 WARN_ONCE
 Documentation/translations/zh_CN/admin-guide/clearing-warn-once.rst-2---------------
 Documentation/translations/zh_CN/admin-guide/clearing-warn-once.rst-3-
 Documentation/translations/zh_CN/admin-guide/clearing-warn-once.rst:4:WARN_ONCE / WARN_ON_ONCE / printk_once 仅仅打印一次消息.
 Documentation/translations/zh_CN/admin-guide/clearing-warn-once.rst-5-
 --
 Documentation/translations/zh_TW/admin-guide/clearing-warn-once.rst-6-
 Documentation/translations/zh_TW/admin-guide/clearing-warn-once.rst:7:清除 WARN_ONCE
 Documentation/translations/zh_TW/admin-guide/clearing-warn-once.rst-8---------------
 Documentation/translations/zh_TW/admin-guide/clearing-warn-once.rst-9-
 Documentation/translations/zh_TW/admin-guide/clearing-warn-once.rst:10:WARN_ONCE / WARN_ON_ONCE / printk_once 僅僅打印一次消息.
 Documentation/translations/zh_TW/admin-guide/clearing-warn-once.rst-11-
 --
 arch/alpha/kernel/perf_event.c=552=static void alpha_pmu_start(struct perf_event *event, int flags)
 --
 arch/alpha/kernel/perf_event.c-556-
 arch/alpha/kernel/perf_event.c:557:	if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED)))
 arch/alpha/kernel/perf_event.c-558-		return;
 --
 arch/alpha/kernel/perf_event.c-560-	if (flags & PERF_EF_RELOAD) {
 arch/alpha/kernel/perf_event.c:561:		WARN_ON_ONCE(!(hwc->state & PERF_HES_UPTODATE));
 arch/alpha/kernel/perf_event.c-562-		alpha_perf_event_set_period(event, hwc, hwc->idx);
 --
 arch/arc/kernel/perf_event.c=451=static void arc_pmu_start(struct perf_event *event, int flags)
 --
 arch/arc/kernel/perf_event.c-455-
 arch/arc/kernel/perf_event.c:456:	if (WARN_ON_ONCE(idx == -1))
 arch/arc/kernel/perf_event.c-457-		return;
 --
 arch/arc/kernel/perf_event.c-459-	if (flags & PERF_EF_RELOAD)
 arch/arc/kernel/perf_event.c:460:		WARN_ON_ONCE(!(hwc->state & PERF_HES_UPTODATE));
 arch/arc/kernel/perf_event.c-461-
 --
 arch/arc/kernel/perf_event.c=562=static irqreturn_t arc_pmu_intr(int irq, void *dev)
 --
 arch/arc/kernel/perf_event.c-597-
 arch/arc/kernel/perf_event.c:598:		WARN_ON_ONCE(hwc->idx != idx);
 arch/arc/kernel/perf_event.c-599-
 --
 arch/arc/kernel/perf_event.c=721=static int arc_pmu_device_probe(struct platform_device *pdev)
 --
 arch/arc/kernel/perf_event.c-742-	BUILD_BUG_ON(ARC_PERF_MAX_COUNTERS > 32);
 arch/arc/kernel/perf_event.c:743:	if (WARN_ON(pct_bcr.c > ARC_PERF_MAX_COUNTERS))
 arch/arc/kernel/perf_event.c-744-		return -EINVAL;
 --
 arch/arc/kernel/process.c=46=SYSCALL_DEFINE3(arc_usr_cmpxchg, int __user *, uaddr, int, expected, int, new)
 --
 arch/arc/kernel/process.c-57-	 */
 arch/arc/kernel/process.c:58:	WARN_ON_ONCE(IS_ENABLED(CONFIG_SMP));
 arch/arc/kernel/process.c-59-
 --
 arch/arc/kernel/unwind.c=244=static void init_unwind_hdr(struct unwind_table *table,
 --
 arch/arc/kernel/unwind.c-333-	}
 arch/arc/kernel/unwind.c:334:	WARN_ON(n != header->fde_count);
 arch/arc/kernel/unwind.c-335-
 --
 arch/arm/common/bL_switcher.c=417=static int bL_switcher_halve_cpus(void)
 --
 arch/arm/common/bL_switcher.c-431-		}
 arch/arm/common/bL_switcher.c:432:		if (WARN_ON(cpu >= MAX_CPUS_PER_CLUSTER))
 arch/arm/common/bL_switcher.c-433-			return -EINVAL;
 --
 arch/arm/common/mcpm_entry.c=239=void mcpm_cpu_power_down(void)
 --
 arch/arm/common/mcpm_entry.c-248-	pr_debug("%s: cpu %u cluster %u\n", __func__, cpu, cluster);
 arch/arm/common/mcpm_entry.c:249:	if (WARN_ON_ONCE(!platform_ops))
 arch/arm/common/mcpm_entry.c-250-	       return;
 --
 arch/arm/common/mcpm_entry.c=308=int mcpm_wait_for_cpu_powerdown(unsigned int cpu, unsigned int cluster)
 --
 arch/arm/common/mcpm_entry.c-311-
 arch/arm/common/mcpm_entry.c:312:	if (WARN_ON_ONCE(!platform_ops || !platform_ops->wait_for_powerdown))
 arch/arm/common/mcpm_entry.c-313-		return -EUNATCH;
 --
 arch/arm/common/mcpm_entry.c=323=void mcpm_cpu_suspend(void)
 arch/arm/common/mcpm_entry.c-324-{
 arch/arm/common/mcpm_entry.c:325:	if (WARN_ON_ONCE(!platform_ops))
 arch/arm/common/mcpm_entry.c-326-		return;
 --
 arch/arm/configs/sama7_defconfig=232=CONFIG_CMA_ALIGNMENT=9
 arch/arm/configs/sama7_defconfig-233-# CONFIG_DEBUG_MISC is not set
 arch/arm/configs/sama7_defconfig:234:# CONFIG_SECTION_MISMATCH_WARN_ONLY is not set
 arch/arm/configs/sama7_defconfig-235-CONFIG_DEBUG_FS=y
 --
 arch/arm/include/asm/arch_gicv3.h=243=static inline void gic_pmr_mask_irqs(void)
 --
 arch/arm/include/asm/arch_gicv3.h-245-	/* Should not get called. */
 arch/arm/include/asm/arch_gicv3.h:246:	WARN_ON_ONCE(true);
 arch/arm/include/asm/arch_gicv3.h-247-}
 --
 arch/arm/include/asm/arch_gicv3.h=249=static inline void gic_arch_enable_irqs(void)
 --
 arch/arm/include/asm/arch_gicv3.h-251-	/* Should not get called. */
 arch/arm/include/asm/arch_gicv3.h:252:	WARN_ON_ONCE(true);
 arch/arm/include/asm/arch_gicv3.h-253-}
 --
 arch/arm/include/asm/cputype.h=118=struct proc_info_list *lookup_processor(u32 midr);
 --
 arch/arm/include/asm/cputype.h-152-	({								\
 arch/arm/include/asm/cputype.h:153:		WARN_ON_ONCE(1);					\
 arch/arm/include/asm/cputype.h-154-		0;							\
 --
 arch/arm/include/asm/cputype.h=157=static inline unsigned int __attribute_const__ read_cpuid_ext(unsigned offset)
 --
 arch/arm/include/asm/cputype.h-169-	({								\
 arch/arm/include/asm/cputype.h:170:		WARN_ON_ONCE(1);					\
 arch/arm/include/asm/cputype.h-171-		0;							\
 --
 arch/arm/include/asm/proc-fns.h=116=static inline void init_proc_vtable(const struct processor *p)
 --
 arch/arm/include/asm/proc-fns.h-119-	*cpu_vtable[cpu] = *p;
 arch/arm/include/asm/proc-fns.h:120:	WARN_ON_ONCE(cpu_vtable[cpu]->dcache_clean_area !=
 arch/arm/include/asm/proc-fns.h-121-		     cpu_vtable[0]->dcache_clean_area);
 arch/arm/include/asm/proc-fns.h:122:	WARN_ON_ONCE(cpu_vtable[cpu]->set_pte_ext !=
 arch/arm/include/asm/proc-fns.h-123-		     cpu_vtable[0]->set_pte_ext);
 --
 arch/arm/kernel/atags_proc.c=33=static int __init init_atags_procfs(void)
 --
 arch/arm/kernel/atags_proc.c-54-
 arch/arm/kernel/atags_proc.c:55:	WARN_ON(tag->hdr.tag != ATAG_NONE);
 arch/arm/kernel/atags_proc.c-56-
 --
 arch/arm/kernel/cacheinfo.c=34=static inline int cache_line_size_cp15(void)
 --
 arch/arm/kernel/cacheinfo.c-42-		return cwg ? 4 << cwg : ARCH_DMA_MINALIGN;
 arch/arm/kernel/cacheinfo.c:43:	} else if (WARN_ON_ONCE(format != CTR_FORMAT_ARMV6)) {
 arch/arm/kernel/cacheinfo.c-44-		return ARCH_DMA_MINALIGN;
 --
 arch/arm/kernel/hibernate.c=34=void notrace save_processor_state(void)
 arch/arm/kernel/hibernate.c-35-{
 arch/arm/kernel/hibernate.c:36:	WARN_ON(num_online_cpus() != 1);
 arch/arm/kernel/hibernate.c-37-	local_fiq_disable();
 --
 arch/arm/kernel/hw_breakpoint.c=1174=static int __init arch_hw_breakpoint_init(void)
 --
 arch/arm/kernel/hw_breakpoint.c-1223-	unregister_undef_hook(&debug_reg_hook);
 arch/arm/kernel/hw_breakpoint.c:1224:	if (WARN_ON(ret < 0) || !cpumask_empty(&debug_err_mask)) {
 arch/arm/kernel/hw_breakpoint.c-1225-		core_num_brps = 0;
 --
 arch/arm/kernel/insn.c=6=static unsigned long __arm_gen_branch_thumb2(unsigned long pc,
 --
 arch/arm/kernel/insn.c-15-	if (offset < -16777216 || offset > 16777214) {
 arch/arm/kernel/insn.c:16:		WARN_ON_ONCE(warn);
 arch/arm/kernel/insn.c-17-		return 0;
 --
 arch/arm/kernel/insn.c=37=static unsigned long __arm_gen_branch_arm(unsigned long pc, unsigned long addr,
 --
 arch/arm/kernel/insn.c-47-	if (unlikely(offset < -33554432 || offset > 33554428)) {
 arch/arm/kernel/insn.c:48:		WARN_ON_ONCE(warn);
 arch/arm/kernel/insn.c-49-		return 0;
 --
 arch/arm/kernel/irq.c=54=static void __init init_irq_stacks(void)
 --
 arch/arm/kernel/irq.c-67-
 arch/arm/kernel/irq.c:68:		if (WARN_ON(!stack))
 arch/arm/kernel/irq.c-69-			break;
 --
 arch/arm/kernel/perf_regs.c=11=u64 perf_reg_value(struct pt_regs *regs, int idx)
 arch/arm/kernel/perf_regs.c-12-{
 arch/arm/kernel/perf_regs.c:13:	if (WARN_ON_ONCE((u32)idx >= PERF_REG_ARM_MAX))
 arch/arm/kernel/perf_regs.c-14-		return 0;
 --
 arch/arm/kernel/smp.c=238=static void ipi_teardown(int cpu)
 --
 arch/arm/kernel/smp.c-241-
 arch/arm/kernel/smp.c:242:	if (WARN_ON_ONCE(!ipi_irq_base))
 arch/arm/kernel/smp.c-243-		return;
 --
 arch/arm/kernel/smp.c=713=static void ipi_setup(int cpu)
 --
 arch/arm/kernel/smp.c-716-
 arch/arm/kernel/smp.c:717:	if (WARN_ON_ONCE(!ipi_irq_base))
 arch/arm/kernel/smp.c-718-		return;
 --
 arch/arm/kernel/smp.c=724=void __init set_smp_ipi_range(int ipi_base, int n)
 --
 arch/arm/kernel/smp.c-727-
 arch/arm/kernel/smp.c:728:	WARN_ON(n < MAX_IPI);
 arch/arm/kernel/smp.c-729-	nr_ipi = min(n, MAX_IPI);
 --
 arch/arm/kernel/smp.c-735-					 "IPI", &irq_stat);
 arch/arm/kernel/smp.c:736:		WARN_ON(err);
 arch/arm/kernel/smp.c-737-
 --
 arch/arm/kernel/suspend.c=117=static int cpu_suspend_alloc_sp(void)
 --
 arch/arm/kernel/suspend.c-122-
 arch/arm/kernel/suspend.c:123:	if (WARN_ON(!ctx_ptr))
 arch/arm/kernel/suspend.c-124-		return -ENOMEM;
 --
 arch/arm/kernel/traps.c=906=static int __init allocate_overflow_stacks(void)
 --
 arch/arm/kernel/traps.c-912-		stack = (u8 *)__get_free_page(GFP_KERNEL);
 arch/arm/kernel/traps.c:913:		if (WARN_ON(!stack))
 arch/arm/kernel/traps.c-914-			return -ENOMEM;
 --
 arch/arm/mach-artpec/board-artpec6.c=45=static void artpec6_l2c310_write_sec(unsigned long val, unsigned reg)
 --
 arch/arm/mach-artpec/board-artpec6.c-51-
 arch/arm/mach-artpec/board-artpec6.c:52:	WARN_ON(res.a0);
 arch/arm/mach-artpec/board-artpec6.c-53-}
 --
 arch/arm/mach-at91/sam_secure.c=17=struct arm_smccc_res sam_smccc_call(u32 fn, u32 arg0, u32 arg1)
 --
 arch/arm/mach-at91/sam_secure.c-20-
 arch/arm/mach-at91/sam_secure.c:21:	if (WARN_ON(!optee_available))
 arch/arm/mach-at91/sam_secure.c-22-		return res;
 --
 arch/arm/mach-bcm/bcm63xx_pmb.c=124=int bcm63xx_pmb_power_on_cpu(struct device_node *dn)
 --
 arch/arm/mach-bcm/bcm63xx_pmb.c-136-	/* We would not know how to enable a third and greater CPU */
 arch/arm/mach-bcm/bcm63xx_pmb.c:137:	WARN_ON(cpu > 1);
 arch/arm/mach-bcm/bcm63xx_pmb.c-138-
 --
 arch/arm/mach-davinci/mux.c=31=int davinci_cfg_reg(const unsigned long index)
 --
 arch/arm/mach-davinci/mux.c-39-
 arch/arm/mach-davinci/mux.c:40:	if (WARN_ON(!soc_info->pinmux_pins))
 arch/arm/mach-davinci/mux.c-41-		return -ENODEV;
 --
 arch/arm/mach-davinci/mux.c-44-		pinmux_base = ioremap(soc_info->pinmux_base, SZ_4K);
 arch/arm/mach-davinci/mux.c:45:		if (WARN_ON(!pinmux_base))
 arch/arm/mach-davinci/mux.c-46-			return -ENOMEM;
 --
 arch/arm/mach-davinci/sram.c=49=static int __init sram_init(void)
 --
 arch/arm/mach-davinci/sram.c-72-
 arch/arm/mach-davinci/sram.c:73:	WARN_ON(status < 0);
 arch/arm/mach-davinci/sram.c-74-	return status;
 --
 arch/arm/mach-exynos/firmware.c=159=static void exynos_l2_write_sec(unsigned long val, unsigned reg)
 --
 arch/arm/mach-exynos/firmware.c-184-	default:
 arch/arm/mach-exynos/firmware.c:185:		WARN_ONCE(1, "%s: ignoring write to reg 0x%x\n", __func__, reg);
 arch/arm/mach-exynos/firmware.c-186-	}
 --
 arch/arm/mach-exynos/suspend.c=451=static void exynos5420_prepare_pm_resume(void)
 --
 arch/arm/mach-exynos/suspend.c-458-	if (IS_ENABLED(CONFIG_EXYNOS_MCPM))
 arch/arm/mach-exynos/suspend.c:459:		WARN_ON(mcpm_cpu_powered_up());
 arch/arm/mach-exynos/suspend.c-460-
 --
 arch/arm/mach-exynos/suspend.c=661=void __init exynos_pm_init(void)
 --
 arch/arm/mach-exynos/suspend.c-672-
 arch/arm/mach-exynos/suspend.c:673:	if (WARN_ON(!of_property_read_bool(np, "interrupt-controller"))) {
 arch/arm/mach-exynos/suspend.c-674-		pr_warn("Outdated DT detected, suspend/resume will NOT work\n");
 --
 arch/arm/mach-footbridge/dc21285.c=261=int __init dc21285_setup(int nr, struct pci_sys_data *sys)
 --
 arch/arm/mach-footbridge/dc21285.c-291-#define dc21285_request_irq(_a, _b, _c, _d, _e) \
 arch/arm/mach-footbridge/dc21285.c:292:	WARN_ON(request_irq(_a, _b, _c, _d, _e) < 0)
 arch/arm/mach-footbridge/dc21285.c-293-
 --
 arch/arm/mach-highbank/highbank.c=42=static void highbank_l2c310_write_sec(unsigned long val, unsigned reg)
 --
 arch/arm/mach-highbank/highbank.c-46-	else
 arch/arm/mach-highbank/highbank.c:47:		WARN_ONCE(1, "Highbank L2C310: ignoring write to reg 0x%x\n",
 arch/arm/mach-highbank/highbank.c-48-			  reg);
 --
 arch/arm/mach-highbank/highbank.c=137=static void __init highbank_init(void)
 --
 arch/arm/mach-highbank/highbank.c-143-	sregs_base = of_iomap(np, 0);
 arch/arm/mach-highbank/highbank.c:144:	WARN_ON(!sregs_base);
 arch/arm/mach-highbank/highbank.c-145-
 --
 arch/arm/mach-hisi/platsmp.c=152=static int hip01_boot_secondary(unsigned int cpu, struct task_struct *idle)
 --
 arch/arm/mach-hisi/platsmp.c-162-	node = of_find_compatible_node(NULL, NULL, "hisilicon,hip01-sysctrl");
 arch/arm/mach-hisi/platsmp.c:163:	if (WARN_ON(!node))
 arch/arm/mach-hisi/platsmp.c-164-		return -1;
 --
 arch/arm/mach-imx/anatop.c=94=void __init imx_init_revision_from_anatop(void)
 --
 arch/arm/mach-imx/anatop.c-104-	anatop_base = of_iomap(np, 0);
 arch/arm/mach-imx/anatop.c:105:	WARN_ON(!anatop_base);
 arch/arm/mach-imx/anatop.c-106-	if (of_device_is_compatible(np, "fsl,imx6sl-anatop"))
 --
 arch/arm/mach-imx/anatop.c-140-			of_node_put(src_np);
 arch/arm/mach-imx/anatop.c:141:			WARN_ON(!src_base);
 arch/arm/mach-imx/anatop.c-142-			sbmr2 = readl_relaxed(src_base + SRC_SBMR2);
 --
 arch/arm/mach-imx/avic.c=166=static void __init mxc_init_irq(void __iomem *irqbase)
 --
 arch/arm/mach-imx/avic.c-200-	irq_base = irq_alloc_descs(-1, 0, AVIC_NUM_IRQS, numa_node_id());
 arch/arm/mach-imx/avic.c:201:	WARN_ON(irq_base < 0);
 arch/arm/mach-imx/avic.c-202-
 --
 arch/arm/mach-imx/avic.c-205-					  &irq_domain_simple_ops, NULL);
 arch/arm/mach-imx/avic.c:206:	WARN_ON(!domain);
 arch/arm/mach-imx/avic.c-207-
 --
 arch/arm/mach-imx/cpu-imx5.c=23=static u32 imx5_read_srev_reg(const char *compat)
 --
 arch/arm/mach-imx/cpu-imx5.c-31-	of_node_put(np);
 arch/arm/mach-imx/cpu-imx5.c:32:	WARN_ON(!iim_base);
 arch/arm/mach-imx/cpu-imx5.c-33-
 --
 arch/arm/mach-imx/cpu.c=61=void __init imx_aips_allow_unprivileged_access(
 --
 arch/arm/mach-imx/cpu.c-68-		aips_base_addr = of_iomap(np, 0);
 arch/arm/mach-imx/cpu.c:69:		WARN_ON(!aips_base_addr);
 arch/arm/mach-imx/cpu.c-70-		imx_set_aips(aips_base_addr);
 --
 arch/arm/mach-imx/gpc.c=227=static int __init imx_gpc_init(struct device_node *node,
 --
 arch/arm/mach-imx/gpc.c-244-	gpc_base = of_iomap(node, 0);
 arch/arm/mach-imx/gpc.c:245:	if (WARN_ON(!gpc_base))
 arch/arm/mach-imx/gpc.c-246-	        return -ENOMEM;
 --
 arch/arm/mach-imx/gpc.c=269=void __init imx_gpc_check_dt(void)
 --
 arch/arm/mach-imx/gpc.c-273-	np = of_find_compatible_node(NULL, NULL, "fsl,imx6q-gpc");
 arch/arm/mach-imx/gpc.c:274:	if (WARN_ON(!np))
 arch/arm/mach-imx/gpc.c-275-		return;
 arch/arm/mach-imx/gpc.c-276-
 arch/arm/mach-imx/gpc.c:277:	if (WARN_ON(!of_property_read_bool(np, "interrupt-controller"))) {
 arch/arm/mach-imx/gpc.c-278-		pr_warn("Outdated DT detected, suspend/resume will NOT work\n");
 --
 arch/arm/mach-imx/mach-imx51.c=27=static void __init imx51_ipu_mipi_setup(void)
 --
 arch/arm/mach-imx/mach-imx51.c-31-	hsc_addr = ioremap(MX51_MIPI_HSC_BASE, SZ_16K);
 arch/arm/mach-imx/mach-imx51.c:32:	WARN_ON(!hsc_addr);
 arch/arm/mach-imx/mach-imx51.c-33-
 --
 arch/arm/mach-imx/mach-vf610.c=20=static void __init vf610_detect_cpu(void)
 --
 arch/arm/mach-imx/mach-vf610.c-27-	np = of_find_compatible_node(NULL, NULL, "fsl,vf610-mscm-cpucfg");
 arch/arm/mach-imx/mach-vf610.c:28:	if (WARN_ON(!np))
 arch/arm/mach-imx/mach-vf610.c-29-		return;
 --
 arch/arm/mach-imx/mach-vf610.c-33-
]
]

gemini-3.1-pro-preview

							input: 23415
							output: 18
							thoughts: 55
						

Okay, here's my quick take on this.

**Debugging the IOMMU Driver**

Alright, so the breadcrumb trail is pointing me directly to `drivers/iommu/intel/iommu.c`, specifically around line 1679. That's where the bug's likely hiding. Time to dive into the code around that line and see what's going on. Let's see what's happening there!

map[Expression:__domain_mapping]

map[Output:drivers/iommu/intel/iommu.c=1603=static int
drivers/iommu/intel/iommu.c:1604:__domain_mapping(struct dmar_domain *domain, unsigned long iov_pfn,
drivers/iommu/intel/iommu.c-1605-		 unsigned long phys_pfn, unsigned long nr_pages, int prot,
--
drivers/iommu/intel/iommu.c=3561=static int intel_iommu_map(struct iommu_domain *domain,
--
drivers/iommu/intel/iommu.c-3592-	size = aligned_nrpages(hpa, size);
drivers/iommu/intel/iommu.c:3593:	return __domain_mapping(dmar_domain, iova >> VTD_PAGE_SHIFT,
drivers/iommu/intel/iommu.c-3594-				hpa >> VTD_PAGE_SHIFT, size, prot, gfp);
]

gemini-3.1-pro-preview

							input: 23852
							output: 18
							thoughts: 31
						

Here's a first-person summary, tailored for an expert audience, of the thought process described:

**Diving into the Intel IOMMU Code**

Okay, time to get my hands dirty. I need to understand a specific aspect of the Intel IOMMU driver, so let's jump right into the source code. I'm going to focus on `drivers/iommu/intel/iommu.c` and specifically, lines 1600 through 1700. This range should give me a concentrated view of whatever functionality is being implemented there. I'm expecting to find either a particular feature implementation or maybe a crucial data structure manipulation. My initial hypothesis is that it could deal with device assignment or interrupt remapping, but I’ll need to confirm that with the code itself. Let's see what gems are waiting to be uncovered.

map[Expression:__domain_mapping]

map[Output:drivers/iommu/intel/iommu.c=1603=static int
drivers/iommu/intel/iommu.c:1604:__domain_mapping(struct dmar_domain *domain, unsigned long iov_pfn,
drivers/iommu/intel/iommu.c-1605-		 unsigned long phys_pfn, unsigned long nr_pages, int prot,
--
drivers/iommu/intel/iommu.c=3561=static int intel_iommu_map(struct iommu_domain *domain,
--
drivers/iommu/intel/iommu.c-3592-	size = aligned_nrpages(hpa, size);
drivers/iommu/intel/iommu.c:3593:	return __domain_mapping(dmar_domain, iova >> VTD_PAGE_SHIFT,
drivers/iommu/intel/iommu.c-3594-				hpa >> VTD_PAGE_SHIFT, size, prot, gfp);
]

gemini-3.1-pro-preview

							input: 24353
							output: 28
							thoughts: 21
						

Here's a first-person summary, tailored for an expert audience, based on the provided starting point:

**Diving into the Intel IOMMU Code**

Okay, time to get my hands dirty. Instead of relying on a pre-digested summary, I need to see what's actually happening at the source. Let's pull up `drivers/iommu/intel/iommu.c` directly. This is where the rubber meets the road, where the low-level interactions with the hardware are defined. I want to see the precise register mappings, the interrupt handling logic, the DMA remapping table manipulation – the *guts* of the Intel IOMMU implementation. I'm looking for any subtle performance optimizations, potential race conditions that the upstream folks might have missed, or any architectural nuances that could inform a new project I'm considering. It's time to cut through the abstraction and *really* understand what's going on under the hood. Time to see what I can learn, and how I can leverage this knowledge.

map[File:drivers/iommu/intel/iommu.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
 * Copyright © 2006-2014 Intel Corporation.
 *
 * Authors: David Woodhouse <dwmw2@infradead.org>,
 *          Ashok Raj <ashok.raj@intel.com>,
 *          Shaohua Li <shaohua.li@intel.com>,
 *          Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>,
 *          Fenghua Yu <fenghua.yu@intel.com>
 *          Joerg Roedel <jroedel@suse.de>
 */

#define pr_fmt(fmt)     "DMAR: " fmt
#define dev_fmt(fmt)    pr_fmt(fmt)

#include <linux/crash_dump.h>
#include <linux/dma-direct.h>
#include <linux/dmi.h>
#include <linux/memory.h>
#include <linux/pci.h>
#include <linux/pci-ats.h>
#include <linux/spinlock.h>
#include <linux/syscore_ops.h>
#include <linux/tboot.h>
#include <uapi/linux/iommufd.h>

#include "iommu.h"
#include "../dma-iommu.h"
#include "../irq_remapping.h"
#include "../iommu-pages.h"
#include "pasid.h"
#include "perfmon.h"

#define ROOT_SIZE		VTD_PAGE_SIZE
#define CONTEXT_SIZE		VTD_PAGE_SIZE

#define IS_GFX_DEVICE(pdev) pci_is_display(pdev)
#define IS_USB_DEVICE(pdev) ((pdev->class >> 8) == PCI_CLASS_SERIAL_USB)
#define IS_ISA_DEVICE(pdev) ((pdev->class >> 8) == PCI_CLASS_BRIDGE_ISA)
#define IS_AZALIA(pdev) ((pdev)->vendor == 0x8086 && (pdev)->device == 0x3a3e)

#define IOAPIC_RANGE_START	(0xfee00000)
#define IOAPIC_RANGE_END	(0xfeefffff)
#define IOVA_START_ADDR		(0x1000)

#define DEFAULT_DOMAIN_ADDRESS_WIDTH 57

#define __DOMAIN_MAX_PFN(gaw)  ((((uint64_t)1) << ((gaw) - VTD_PAGE_SHIFT)) - 1)
#define __DOMAIN_MAX_ADDR(gaw) ((((uint64_t)1) << (gaw)) - 1)

/* We limit DOMAIN_MAX_PFN to fit in an unsigned long, and DOMAIN_MAX_ADDR
   to match. That way, we can use 'unsigned long' for PFNs with impunity. */
#define DOMAIN_MAX_PFN(gaw)	((unsigned long) min_t(uint64_t, \
				__DOMAIN_MAX_PFN(gaw), (unsigned long)-1))
#define DOMAIN_MAX_ADDR(gaw)	(((uint64_t)__DOMAIN_MAX_PFN(gaw)) << VTD_PAGE_SHIFT)

static void __init check_tylersburg_isoch(void);
static int rwbf_quirk;

#define rwbf_required(iommu)	(rwbf_quirk || cap_rwbf((iommu)->cap))

/*
 * set to 1 to panic kernel if can't successfully enable VT-d
 * (used when kernel is launched w/ TXT)
 */
static int force_on = 0;
static int intel_iommu_tboot_noforce;
static int no_platform_optin;

#define ROOT_ENTRY_NR (VTD_PAGE_SIZE/sizeof(struct root_entry))

/*
 * Take a root_entry and return the Lower Context Table Pointer (LCTP)
 * if marked present.
 */
static phys_addr_t root_entry_lctp(struct root_entry *re)
{
	if (!(re->lo & 1))
		return 0;

	return re->lo & VTD_PAGE_MASK;
}

/*
 * Take a root_entry and return the Upper Context Table Pointer (UCTP)
 * if marked present.
 */
static phys_addr_t root_entry_uctp(struct root_entry *re)
{
	if (!(re->hi & 1))
		return 0;

	return re->hi & VTD_PAGE_MASK;
}

static int device_rid_cmp_key(const void *key, const struct rb_node *node)
{
	struct device_domain_info *info =
		rb_entry(node, struct device_domain_info, node);
	const u16 *rid_lhs = key;

	if (*rid_lhs < PCI_DEVID(info->bus, info->devfn))
		return -1;

	if (*rid_lhs > PCI_DEVID(info->bus, info->devfn))
		return 1;

	return 0;
}

static int device_rid_cmp(struct rb_node *lhs, const struct rb_node *rhs)
{
	struct device_domain_info *info =
		rb_entry(lhs, struct device_domain_info, node);
	u16 key = PCI_DEVID(info->bus, info->devfn);

	return device_rid_cmp_key(&key, rhs);
}

/*
 * Looks up an IOMMU-probed device using its source ID.
 *
 * Returns the pointer to the device if there is a match. Otherwise,
 * returns NULL.
 *
 * Note that this helper doesn't guarantee that the device won't be
 * released by the iommu subsystem after being returned. The caller
 * should use its own synchronization mechanism to avoid the device
 * being released during its use if its possibly the case.
 */
struct device *device_rbtree_find(struct intel_iommu *iommu, u16 rid)
{
	struct device_domain_info *info = NULL;
	struct rb_node *node;
	unsigned long flags;

	spin_lock_irqsave(&iommu->device_rbtree_lock, flags);
	node = rb_find(&rid, &iommu->device_rbtree, device_rid_cmp_key);
	if (node)
		info = rb_entry(node, struct device_domain_info, node);
	spin_unlock_irqrestore(&iommu->device_rbtree_lock, flags);

	return info ? info->dev : NULL;
}

static int device_rbtree_insert(struct intel_iommu *iommu,
				struct device_domain_info *info)
{
	struct rb_node *curr;
	unsigned long flags;

	spin_lock_irqsave(&iommu->device_rbtree_lock, flags);
	curr = rb_find_add(&info->node, &iommu->device_rbtree, device_rid_cmp);
	spin_unlock_irqrestore(&iommu->device_rbtree_lock, flags);
	if (WARN_ON(curr))
		return -EEXIST;

	return 0;
}

static void device_rbtree_remove(struct device_domain_info *info)
{
	struct intel_iommu *iommu = info->iommu;
	unsigned long flags;

	spin_lock_irqsave(&iommu->device_rbtree_lock, flags);
	rb_erase(&info->node, &iommu->device_rbtree);
	spin_unlock_irqrestore(&iommu->device_rbtree_lock, flags);
}

struct dmar_rmrr_unit {
	struct list_head list;		/* list of rmrr units	*/
	struct acpi_dmar_header *hdr;	/* ACPI header		*/
	u64	base_address;		/* reserved base address*/
	u64	end_address;		/* reserved end address */
	struct dmar_dev_scope *devices;	/* target devices */
	int	devices_cnt;		/* target device count */
};

struct dmar_atsr_unit {
	struct list_head list;		/* list of ATSR units */
	struct acpi_dmar_header *hdr;	/* ACPI header */
	struct dmar_dev_scope *devices;	/* target devices */
	int devices_cnt;		/* target device count */
	u8 include_all:1;		/* include all ports */
};

struct dmar_satc_unit {
	struct list_head list;		/* list of SATC units */
	struct acpi_dmar_header *hdr;	/* ACPI header */
	struct dmar_dev_scope *devices;	/* target devices */
	struct intel_iommu *iommu;	/* the corresponding iommu */
	int devices_cnt;		/* target device count */
	u8 atc_required:1;		/* ATS is required */
};

static LIST_HEAD(dmar_atsr_units);
static LIST_HEAD(dmar_rmrr_units);
static LIST_HEAD(dmar_satc_units);

#define for_each_rmrr_units(rmrr) \
	list_for_each_entry(rmrr, &dmar_rmrr_units, list)

static void intel_iommu_domain_free(struct iommu_domain *domain);

int dmar_disabled = !IS_ENABLED(CONFIG_INTEL_IOMMU_DEFAULT_ON);
int intel_iommu_sm = IS_ENABLED(CONFIG_INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON);

int intel_iommu_enabled = 0;
EXPORT_SYMBOL_GPL(intel_iommu_enabled);

static int intel_iommu_superpage = 1;
static int iommu_identity_mapping;
static int iommu_skip_te_disable;
static int disable_igfx_iommu;

#define IDENTMAP_AZALIA		4

const struct iommu_ops intel_iommu_ops;
static const struct iommu_dirty_ops intel_dirty_ops;

static bool translation_pre_enabled(struct intel_iommu *iommu)
{
	return (iommu->flags & VTD_FLAG_TRANS_PRE_ENABLED);
}

static void clear_translation_pre_enabled(struct intel_iommu *iommu)
{
	iommu->flags &= ~VTD_FLAG_TRANS_PRE_ENABLED;
}

static void init_translation_status(struct intel_iommu *iommu)
{
	u32 gsts;

	gsts = readl(iommu->reg + DMAR_GSTS_REG);
	if (gsts & DMA_GSTS_TES)
		iommu->flags |= VTD_FLAG_TRANS_PRE_ENABLED;
}

static int __init intel_iommu_setup(char *str)
{
	if (!str)
		return -EINVAL;

	while (*str) {
		if (!strncmp(str, "on", 2)) {
			dmar_disabled = 0;
			pr_info("IOMMU enabled\n");
		} else if (!strncmp(str, "off", 3)) {
			dmar_disabled = 1;
			no_platform_optin = 1;
			pr_info("IOMMU disabled\n");
		} else if (!strncmp(str, "igfx_off", 8)) {
			disable_igfx_iommu = 1;
			pr_info("Disable GFX device mapping\n");
		} else if (!strncmp(str, "forcedac", 8)) {
			pr_warn("intel_iommu=forcedac deprecated; use iommu.forcedac instead\n");
			iommu_dma_forcedac = true;
		} else if (!strncmp(str, "strict", 6)) {
			pr_warn("intel_iommu=strict deprecated; use iommu.strict=1 instead\n");
			iommu_set_dma_strict();
		} else if (!strncmp(str, "sp_off", 6)) {
			pr_info("Disable supported super page\n");
			intel_iommu_superpage = 0;
		} else if (!strncmp(str, "sm_on", 5)) {
			pr_info("Enable scalable mode if hardware supports\n");
			intel_iommu_sm = 1;
		} else if (!strncmp(str, "sm_off", 6)) {
			pr_info("Scalable mode is disallowed\n");
			intel_iommu_sm = 0;
		} else if (!strncmp(str, "tboot_noforce", 13)) {
			pr_info("Intel-IOMMU: not forcing on after tboot. This could expose security risk for tboot\n");
			intel_iommu_tboot_noforce = 1;
		} else {
			pr_notice("Unknown option - '%s'\n", str);
		}

		str += strcspn(str, ",");
		while (*str == ',')
			str++;
	}

	return 1;
}
__setup("intel_iommu=", intel_iommu_setup);

static int domain_pfn_supported(struct dmar_domain *domain, unsigned long pfn)
{
	int addr_width = agaw_to_width(domain->agaw) - VTD_PAGE_SHIFT;

	return !(addr_width < BITS_PER_LONG && pfn >> addr_width);
}

/*
 * Calculate the Supported Adjusted Guest Address Widths of an IOMMU.
 * Refer to 11.4.2 of the VT-d spec for the encoding of each bit of
 * the returned SAGAW.
 */
static unsigned long __iommu_calculate_sagaw(struct intel_iommu *iommu)
{
	unsigned long fl_sagaw, sl_sagaw;

	fl_sagaw = BIT(2) | (cap_fl5lp_support(iommu->cap) ? BIT(3) : 0);
	sl_sagaw = cap_sagaw(iommu->cap);

	/* Second level only. */
	if (!sm_supported(iommu) || !ecap_flts(iommu->ecap))
		return sl_sagaw;

	/* First level only. */
	if (!ecap_slts(iommu->ecap))
		return fl_sagaw;

	return fl_sagaw & sl_sagaw;
}

static int __iommu_calculate_agaw(struct intel_iommu *iommu, int max_gaw)
{
	unsigned long sagaw;
	int agaw;

	sagaw = __iommu_calculate_sagaw(iommu);
	for (agaw = width_to_agaw(max_gaw); agaw >= 0; agaw--) {
		if (test_bit(agaw, &sagaw))
			break;
	}

	return agaw;
}

/*
 * Calculate max SAGAW for each iommu.
 */
int iommu_calculate_max_sagaw(struct intel_iommu *iommu)
{
	return __iommu_calculate_agaw(iommu, MAX_AGAW_WIDTH);
}

/*
 * calculate agaw for each iommu.
 * "SAGAW" may be different across iommus, use a default agaw, and
 * get a supported less agaw for iommus that don't support the default agaw.
 */
int iommu_calculate_agaw(struct intel_iommu *iommu)
{
	return __iommu_calculate_agaw(iommu, DEFAULT_DOMAIN_ADDRESS_WIDTH);
}

static bool iommu_paging_structure_coherency(struct intel_iommu *iommu)
{
	return sm_supported(iommu) ?
			ecap_smpwc(iommu->ecap) : ecap_coherent(iommu->ecap);
}

/* Return the super pagesize bitmap if supported. */
static unsigned long domain_super_pgsize_bitmap(struct dmar_domain *domain)
{
	unsigned long bitmap = 0;

	/*
	 * 1-level super page supports page size of 2MiB, 2-level super page
	 * supports page size of both 2MiB and 1GiB.
	 */
	if (domain->iommu_superpage == 1)
		bitmap |= SZ_2M;
	else if (domain->iommu_superpage == 2)
		bitmap |= SZ_2M | SZ_1G;

	return bitmap;
}

struct context_entry *iommu_context_addr(struct intel_iommu *iommu, u8 bus,
					 u8 devfn, int alloc)
{
	struct root_entry *root = &iommu->root_entry[bus];
	struct context_entry *context;
	u64 *entry;

	/*
	 * Except that the caller requested to allocate a new entry,
	 * returning a copied context entry makes no sense.
	 */
	if (!alloc && context_copied(iommu, bus, devfn))
		return NULL;

	entry = &root->lo;
	if (sm_supported(iommu)) {
		if (devfn >= 0x80) {
			devfn -= 0x80;
			entry = &root->hi;
		}
		devfn *= 2;
	}
	if (*entry & 1)
		context = phys_to_virt(*entry & VTD_PAGE_MASK);
	else {
		unsigned long phy_addr;
		if (!alloc)
			return NULL;

		context = iommu_alloc_pages_node_sz(iommu->node, GFP_ATOMIC,
						    SZ_4K);
		if (!context)
			return NULL;

		__iommu_flush_cache(iommu, (void *)context, CONTEXT_SIZE);
		phy_addr = virt_to_phys((void *)context);
		*entry = phy_addr | 1;
		__iommu_flush_cache(iommu, entry, sizeof(*entry));
	}
	return &context[devfn];
}

/**
 * is_downstream_to_pci_bridge - test if a device belongs to the PCI
 *				 sub-hierarchy of a candidate PCI-PCI bridge
 * @dev: candidate PCI device belonging to @bridge PCI sub-hierarchy
 * @bridge: the candidate PCI-PCI bridge
 *
 * Return: true if @dev belongs to @bridge PCI sub-hierarchy, else false.
 */
static bool
is_downstream_to_pci_bridge(struct device *dev, struct device *bridge)
{
	struct pci_dev *pdev, *pbridge;

	if (!dev_is_pci(dev) || !dev_is_pci(bridge))
		return false;

	pdev = to_pci_dev(dev);
	pbridge = to_pci_dev(bridge);

	if (pbridge->subordinate &&
	    pbridge->subordinate->number <= pdev->bus->number &&
	    pbridge->subordinate->busn_res.end >= pdev->bus->number)
		return true;

	return false;
}

static bool quirk_ioat_snb_local_iommu(struct pci_dev *pdev)
{
	struct dmar_drhd_unit *drhd;
	u32 vtbar;
	int rc;

	/* We know that this device on this chipset has its own IOMMU.
	 * If we find it under a different IOMMU, then the BIOS is lying
	 * to us. Hope that the IOMMU for this device is actually
	 * disabled, and it needs no translation...
	 */
	rc = pci_bus_read_config_dword(pdev->bus, PCI_DEVFN(0, 0), 0xb0, &vtbar);
	if (rc) {
		/* "can't" happen */
		dev_info(&pdev->dev, "failed to run vt-d quirk\n");
		return false;
	}
	vtbar &= 0xffff0000;

	/* we know that the this iommu should be at offset 0xa000 from vtbar */
	drhd = dmar_find_matched_drhd_unit(pdev);
	if (!drhd || drhd->reg_base_addr - vtbar != 0xa000) {
		pr_warn_once(FW_BUG "BIOS assigned incorrect VT-d unit for Intel(R) QuickData Technology device\n");
		add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_STILL_OK);
		return true;
	}

	return false;
}

static bool iommu_is_dummy(struct intel_iommu *iommu, struct device *dev)
{
	if (!iommu || iommu->drhd->ignored)
		return true;

	if (dev_is_pci(dev)) {
		struct pci_dev *pdev = to_pci_dev(dev);

		if (pdev->vendor == PCI_VENDOR_ID_INTEL &&
		    pdev->device == PCI_DEVICE_ID_INTEL_IOAT_SNB &&
		    quirk_ioat_snb_local_iommu(pdev))
			return true;
	}

	return false;
}

static struct intel_iommu *device_lookup_iommu(struct device *dev, u8 *bus, u8 *devfn)
{
	struct dmar_drhd_unit *drhd = NULL;
	struct pci_dev *pdev = NULL;
	struct intel_iommu *iommu;
	struct device *tmp;
	u16 segment = 0;
	int i;

	if (!dev)
		return NULL;

	if (dev_is_pci(dev)) {
		struct pci_dev *pf_pdev;

		pdev = pci_real_dma_dev(to_pci_dev(dev));

		/* VFs aren't listed in scope tables; we need to look up
		 * the PF instead to find the IOMMU. */
		pf_pdev = pci_physfn(pdev);
		dev = &pf_pdev->dev;
		segment = pci_domain_nr(pdev->bus);
	} else if (has_acpi_companion(dev))
		dev = &ACPI_COMPANION(dev)->dev;

	rcu_read_lock();
	for_each_iommu(iommu, drhd) {
		if (pdev && segment != drhd->segment)
			continue;

		for_each_active_dev_scope(drhd->devices,
					  drhd->devices_cnt, i, tmp) {
			if (tmp == dev) {
				/* For a VF use its original BDF# not that of the PF
				 * which we used for the IOMMU lookup. Strictly speaking
				 * we could do this for all PCI devices; we only need to
				 * get the BDF# from the scope table for ACPI matches. */
				if (pdev && pdev->is_virtfn)
					goto got_pdev;

				if (bus && devfn) {
					*bus = drhd->devices[i].bus;
					*devfn = drhd->devices[i].devfn;
				}
				goto out;
			}

			if (is_downstream_to_pci_bridge(dev, tmp))
				goto got_pdev;
		}

		if (pdev && drhd->include_all) {
got_pdev:
			if (bus && devfn) {
				*bus = pdev->bus->number;
				*devfn = pdev->devfn;
			}
			goto out;
		}
	}
	iommu = NULL;
out:
	if (iommu_is_dummy(iommu, dev))
		iommu = NULL;

	rcu_read_unlock();

	return iommu;
}

static void domain_flush_cache(struct dmar_domain *domain,
			       void *addr, int size)
{
	if (!domain->iommu_coherency)
		clflush_cache_range(addr, size);
}

static void free_context_table(struct intel_iommu *iommu)
{
	struct context_entry *context;
	int i;

	if (!iommu->root_entry)
		return;

	for (i = 0; i < ROOT_ENTRY_NR; i++) {
		context = iommu_context_addr(iommu, i, 0, 0);
		if (context)
			iommu_free_pages(context);

		if (!sm_supported(iommu))
			continue;

		context = iommu_context_addr(iommu, i, 0x80, 0);
		if (context)
			iommu_free_pages(context);
	}

	iommu_free_pages(iommu->root_entry);
	iommu->root_entry = NULL;
}

#ifdef CONFIG_DMAR_DEBUG
static void pgtable_walk(struct intel_iommu *iommu, unsigned long pfn,
			 u8 bus, u8 devfn, struct dma_pte *parent, int level)
{
	struct dma_pte *pte;
	int offset;

	while (1) {
		offset = pfn_level_offset(pfn, level);
		pte = &parent[offset];

		pr_info("pte level: %d, pte value: 0x%016llx\n", level, pte->val);

		if (!dma_pte_present(pte)) {
			pr_info("page table not present at level %d\n", level - 1);
			break;
		}

		if (level == 1 || dma_pte_superpage(pte))
			break;

		parent = phys_to_virt(dma_pte_addr(pte));
		level--;
	}
}

void dmar_fault_dump_ptes(struct intel_iommu *iommu, u16 source_id,
			  unsigned long long addr, u32 pasid)
{
	struct pasid_dir_entry *dir, *pde;
	struct pasid_entry *entries, *pte;
	struct context_entry *ctx_entry;
	struct root_entry *rt_entry;
	int i, dir_index, index, level;
	u8 devfn = source_id & 0xff;
	u8 bus = source_id >> 8;
	struct dma_pte *pgtable;

	pr_info("Dump %s table entries for IOVA 0x%llx\n", iommu->name, addr);

	/* root entry dump */
	if (!iommu->root_entry) {
		pr_info("root table is not present\n");
		return;
	}
	rt_entry = &iommu->root_entry[bus];

	if (sm_supported(iommu))
		pr_info("scalable mode root entry: hi 0x%016llx, low 0x%016llx\n",
			rt_entry->hi, rt_entry->lo);
	else
		pr_info("root entry: 0x%016llx", rt_entry->lo);

	/* context entry dump */
	ctx_entry = iommu_context_addr(iommu, bus, devfn, 0);
	if (!ctx_entry) {
		pr_info("context table is not present\n");
		return;
	}

	pr_info("context entry: hi 0x%016llx, low 0x%016llx\n",
		ctx_entry->hi, ctx_entry->lo);

	/* legacy mode does not require PASID entries */
	if (!sm_supported(iommu)) {
		if (!context_present(ctx_entry)) {
			pr_info("legacy mode page table is not present\n");
			return;
		}
		level = agaw_to_level(ctx_entry->hi & 7);
		pgtable = phys_to_virt(ctx_entry->lo & VTD_PAGE_MASK);
		goto pgtable_walk;
	}

	if (!context_present(ctx_entry)) {
		pr_info("pasid directory table is not present\n");
		return;
	}

	/* get the pointer to pasid directory entry */
	dir = phys_to_virt(ctx_entry->lo & VTD_PAGE_MASK);

	/* For request-without-pasid, get the pasid from context entry */
	if (intel_iommu_sm && pasid == IOMMU_PASID_INVALID)
		pasid = IOMMU_NO_PASID;

	dir_index = pasid >> PASID_PDE_SHIFT;
	pde = &dir[dir_index];
	pr_info("pasid dir entry: 0x%016llx\n", pde->val);

	/* get the pointer to the pasid table entry */
	entries = get_pasid_table_from_pde(pde);
	if (!entries) {
		pr_info("pasid table is not present\n");
		return;
	}
	index = pasid & PASID_PTE_MASK;
	pte = &entries[index];
	for (i = 0; i < ARRAY_SIZE(pte->val); i++)
		pr_info("pasid table entry[%d]: 0x%016llx\n", i, pte->val[i]);

	if (!pasid_pte_is_present(pte)) {
		pr_info("scalable mode page table is not present\n");
		return;
	}

	if (pasid_pte_get_pgtt(pte) == PASID_ENTRY_PGTT_FL_ONLY) {
		level = pte->val[2] & BIT_ULL(2) ? 5 : 4;
		pgtable = phys_to_virt(pte->val[2] & VTD_PAGE_MASK);
	} else {
		level = agaw_to_level((pte->val[0] >> 2) & 0x7);
		pgtable = phys_to_virt(pte->val[0] & VTD_PAGE_MASK);
	}

pgtable_walk:
	pgtable_walk(iommu, addr >> VTD_PAGE_SHIFT, bus, devfn, pgtable, level);
}
#endif

static struct dma_pte *pfn_to_dma_pte(struct dmar_domain *domain,
				      unsigned long pfn, int *target_level,
				      gfp_t gfp)
{
	struct dma_pte *parent, *pte;
	int level = agaw_to_level(domain->agaw);
	int offset;

	if (!domain_pfn_supported(domain, pfn))
		/* Address beyond IOMMU's addressing capabilities. */
		return NULL;

	parent = domain->pgd;

	while (1) {
		void *tmp_page;

		offset = pfn_level_offset(pfn, level);
		pte = &parent[offset];
		if (!*target_level && (dma_pte_superpage(pte) || !dma_pte_present(pte)))
			break;
		if (level == *target_level)
			break;

		if (!dma_pte_present(pte)) {
			uint64_t pteval, tmp;

			tmp_page = iommu_alloc_pages_node_sz(domain->nid, gfp,
							     SZ_4K);

			if (!tmp_page)
				return NULL;

			domain_flush_cache(domain, tmp_page, VTD_PAGE_SIZE);
			pteval = virt_to_phys(tmp_page) | DMA_PTE_READ |
				 DMA_PTE_WRITE;
			if (domain->use_first_level)
				pteval |= DMA_FL_PTE_US | DMA_FL_PTE_ACCESS;

			tmp = 0ULL;
			if (!try_cmpxchg64(&pte->val, &tmp, pteval))
				/* Someone else set it while we were thinking; use theirs. */
				iommu_free_pages(tmp_page);
			else
				domain_flush_cache(domain, pte, sizeof(*pte));
		}
		if (level == 1)
			break;

		parent = phys_to_virt(dma_pte_addr(pte));
		level--;
	}

	if (!*target_level)
		*target_level = level;

	return pte;
}

/* return address's pte at specific level */
static struct dma_pte *dma_pfn_level_pte(struct dmar_domain *domain,
					 unsigned long pfn,
					 int level, int *large_page)
{
	struct dma_pte *parent, *pte;
	int total = agaw_to_level(domain->agaw);
	int offset;

	parent = domain->pgd;
	while (level <= total) {
		offset = pfn_level_offset(pfn, total);
		pte = &parent[offset];
		if (level == total)
			return pte;

		if (!dma_pte_present(pte)) {
			*large_page = total;
			break;
		}

		if (dma_pte_superpage(pte)) {
			*large_page = total;
			return pte;
		}

		parent = phys_to_virt(dma_pte_addr(pte));
		total--;
	}
	return NULL;
}

/* clear last level pte, a tlb flush should be followed */
static void dma_pte_clear_range(struct dmar_domain *domain,
				unsigned long start_pfn,
				unsigned long last_pfn)
{
	unsigned int large_page;
	struct dma_pte *first_pte, *pte;

	if (WARN_ON(!domain_pfn_supported(domain, last_pfn)) ||
	    WARN_ON(start_pfn > last_pfn))
		return;

	/* we don't need lock here; nobody else touches the iova range */
	do {
		large_page = 1;
		first_pte = pte = dma_pfn_level_pte(domain, start_pfn, 1, &large_page);
		if (!pte) {
			start_pfn = align_to_level(start_pfn + 1, large_page + 1);
			continue;
		}
		do {
			dma_clear_pte(pte);
			start_pfn += lvl_to_nr_pages(large_page);
			pte++;
		} while (start_pfn <= last_pfn && !first_pte_in_page(pte));

		domain_flush_cache(domain, first_pte,
				   (void *)pte - (void *)first_pte);

	} while (start_pfn && start_pfn <= last_pfn);
}

static void dma_pte_free_level(struct dmar_domain *domain, int level,
			       int retain_level, struct dma_pte *pte,
			       unsigned long pfn, unsigned long start_pfn,
			       unsigned long last_pfn)
{
	pfn = max(start_pfn, pfn);
	pte = &pte[pfn_level_offset(pfn, level)];

	do {
		unsigned long level_pfn;
		struct dma_pte *level_pte;

		if (!dma_pte_present(pte) || dma_pte_superpage(pte))
			goto next;

		level_pfn = pfn & level_mask(level);
		level_pte = phys_to_virt(dma_pte_addr(pte));

		if (level > 2) {
			dma_pte_free_level(domain, level - 1, retain_level,
					   level_pte, level_pfn, start_pfn,
					   last_pfn);
		}

		/*
		 * Free the page table if we're below the level we want to
		 * retain and the range covers the entire table.
		 */
		if (level < retain_level && !(start_pfn > level_pfn ||
		      last_pfn < level_pfn + level_size(level) - 1)) {
			dma_clear_pte(pte);
			domain_flush_cache(domain, pte, sizeof(*pte));
			iommu_free_pages(level_pte);
		}
next:
		pfn += level_size(level);
	} while (!first_pte_in_page(++pte) && pfn <= last_pfn);
}

/*
 * clear last level (leaf) ptes and free page table pages below the
 * level we wish to keep intact.
 */
static void dma_pte_free_pagetable(struct dmar_domain *domain,
				   unsigned long start_pfn,
				   unsigned long last_pfn,
				   int retain_level)
{
	dma_pte_clear_range(domain, start_pfn, last_pfn);

	/* We don't need lock here; nobody else touches the iova range */
	dma_pte_free_level(domain, agaw_to_level(domain->agaw), retain_level,
			   domain->pgd, 0, start_pfn, last_pfn);

	/* free pgd */
	if (start_pfn == 0 && last_pfn == DOMAIN_MAX_PFN(domain->gaw)) {
		iommu_free_pages(domain->pgd);
		domain->pgd = NULL;
	}
}

/* When a page at a given level is being unlinked from its parent, we don't
   need to *modify* it at all. All we need to do is make a list of all the
   pages which can be freed just as soon as we've flushed the IOTLB and we
   know the hardware page-walk will no longer touch them.
   The 'pte' argument is the *parent* PTE, pointing to the page that is to
   be freed. */
static void dma_pte_list_pagetables(struct dmar_domain *domain,
				    int level, struct dma_pte *parent_pte,
				    struct iommu_pages_list *freelist)
{
	struct dma_pte *pte = phys_to_virt(dma_pte_addr(parent_pte));

	iommu_pages_list_add(freelist, pte);

	if (level == 1)
		return;

	do {
		if (dma_pte_present(pte) && !dma_pte_superpage(pte))
			dma_pte_list_pagetables(domain, level - 1, pte, freelist);
		pte++;
	} while (!first_pte_in_page(pte));
}

static void dma_pte_clear_level(struct dmar_domain *domain, int level,
				struct dma_pte *pte, unsigned long pfn,
				unsigned long start_pfn, unsigned long last_pfn,
				struct iommu_pages_list *freelist)
{
	struct dma_pte *first_pte = NULL, *last_pte = NULL;

	pfn = max(start_pfn, pfn);
	pte = &pte[pfn_level_offset(pfn, level)];

	do {
		unsigned long level_pfn = pfn & level_mask(level);

		if (!dma_pte_present(pte))
			goto next;

		/* If range covers entire pagetable, free it */
		if (start_pfn <= level_pfn &&
		    last_pfn >= level_pfn + level_size(level) - 1) {
			/* These suborbinate page tables are going away entirely. Don't
			   bother to clear them; we're just going to *free* them. */
			if (level > 1 && !dma_pte_superpage(pte))
				dma_pte_list_pagetables(domain, level - 1, pte, freelist);

			dma_clear_pte(pte);
			if (!first_pte)
				first_pte = pte;
			last_pte = pte;
		} else if (level > 1) {
			/* Recurse down into a level that isn't *entirely* obsolete */
			dma_pte_clear_level(domain, level - 1,
					    phys_to_virt(dma_pte_addr(pte)),
					    level_pfn, start_pfn, last_pfn,
					    freelist);
		}
next:
		pfn = level_pfn + level_size(level);
	} while (!first_pte_in_page(++pte) && pfn <= last_pfn);

	if (first_pte)
		domain_flush_cache(domain, first_pte,
				   (void *)++last_pte - (void *)first_pte);
}

/* We can't just free the pages because the IOMMU may still be walking
   the page tables, and may have cached the intermediate levels. The
   pages can only be freed after the IOTLB flush has been done. */
static void domain_unmap(struct dmar_domain *domain, unsigned long start_pfn,
			 unsigned long last_pfn,
			 struct iommu_pages_list *freelist)
{
	if (WARN_ON(!domain_pfn_supported(domain, last_pfn)) ||
	    WARN_ON(start_pfn > last_pfn))
		return;

	/* we don't need lock here; nobody else touches the iova range */
	dma_pte_clear_level(domain, agaw_to_level(domain->agaw),
			    domain->pgd, 0, start_pfn, last_pfn, freelist);

	/* free pgd */
	if (start_pfn == 0 && last_pfn == DOMAIN_MAX_PFN(domain->gaw)) {
		iommu_pages_list_add(freelist, domain->pgd);
		domain->pgd = NULL;
	}
}

/* iommu handling */
static int iommu_alloc_root_entry(struct intel_iommu *iommu)
{
	struct root_entry *root;

	root = iommu_alloc_pages_node_sz(iommu->node, GFP_ATOMIC, SZ_4K);
	if (!root) {
		pr_err("Allocating root entry for %s failed\n",
			iommu->name);
		return -ENOMEM;
	}

	__iommu_flush_cache(iommu, root, ROOT_SIZE);
	iommu->root_entry = root;

	return 0;
}

static void iommu_set_root_entry(struct intel_iommu *iommu)
{
	u64 addr;
	u32 sts;
	unsigned long flag;

	addr = virt_to_phys(iommu->root_entry);
	if (sm_supported(iommu))
		addr |= DMA_RTADDR_SMT;

	raw_spin_lock_irqsave(&iommu->register_lock, flag);
	dmar_writeq(iommu->reg + DMAR_RTADDR_REG, addr);

	writel(iommu->gcmd | DMA_GCMD_SRTP, iommu->reg + DMAR_GCMD_REG);

	/* Make sure hardware complete it */
	IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
		      readl, (sts & DMA_GSTS_RTPS), sts);

	raw_spin_unlock_irqrestore(&iommu->register_lock, flag);

	/*
	 * Hardware invalidates all DMA remapping hardware translation
	 * caches as part of SRTP flow.
	 */
	if (cap_esrtps(iommu->cap))
		return;

	iommu->flush.flush_context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL);
	if (sm_supported(iommu))
		qi_flush_pasid_cache(iommu, 0, QI_PC_GLOBAL, 0);
	iommu->flush.flush_iotlb(iommu, 0, 0, 0, DMA_TLB_GLOBAL_FLUSH);
}

void iommu_flush_write_buffer(struct intel_iommu *iommu)
{
	u32 val;
	unsigned long flag;

	if (!rwbf_quirk && !cap_rwbf(iommu->cap))
		return;

	raw_spin_lock_irqsave(&iommu->register_lock, flag);
	writel(iommu->gcmd | DMA_GCMD_WBF, iommu->reg + DMAR_GCMD_REG);

	/* Make sure hardware complete it */
	IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
		      readl, (!(val & DMA_GSTS_WBFS)), val);

	raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
}

/* return value determine if we need a write buffer flush */
static void __iommu_flush_context(struct intel_iommu *iommu,
				  u16 did, u16 source_id, u8 function_mask,
				  u64 type)
{
	u64 val = 0;
	unsigned long flag;

	switch (type) {
	case DMA_CCMD_GLOBAL_INVL:
		val = DMA_CCMD_GLOBAL_INVL;
		break;
	case DMA_CCMD_DOMAIN_INVL:
		val = DMA_CCMD_DOMAIN_INVL|DMA_CCMD_DID(did);
		break;
	case DMA_CCMD_DEVICE_INVL:
		val = DMA_CCMD_DEVICE_INVL|DMA_CCMD_DID(did)
			| DMA_CCMD_SID(source_id) | DMA_CCMD_FM(function_mask);
		break;
	default:
		pr_warn("%s: Unexpected context-cache invalidation type 0x%llx\n",
			iommu->name, type);
		return;
	}
	val |= DMA_CCMD_ICC;

	raw_spin_lock_irqsave(&iommu->register_lock, flag);
	dmar_writeq(iommu->reg + DMAR_CCMD_REG, val);

	/* Make sure hardware complete it */
	IOMMU_WAIT_OP(iommu, DMAR_CCMD_REG,
		dmar_readq, (!(val & DMA_CCMD_ICC)), val);

	raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
}

void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
			 unsigned int size_order, u64 type)
{
	int tlb_offset = ecap_iotlb_offset(iommu->ecap);
	u64 val = 0, val_iva = 0;
	unsigned long flag;

	switch (type) {
	case DMA_TLB_GLOBAL_FLUSH:
		/* global flush doesn't need set IVA_REG */
		val = DMA_TLB_GLOBAL_FLUSH|DMA_TLB_IVT;
		break;
	case DMA_TLB_DSI_FLUSH:
		val = DMA_TLB_DSI_FLUSH|DMA_TLB_IVT|DMA_TLB_DID(did);
		break;
	case DMA_TLB_PSI_FLUSH:
		val = DMA_TLB_PSI_FLUSH|DMA_TLB_IVT|DMA_TLB_DID(did);
		/* IH bit is passed in as part of address */
		val_iva = size_order | addr;
		break;
	default:
		pr_warn("%s: Unexpected iotlb invalidation type 0x%llx\n",
			iommu->name, type);
		return;
	}

	if (cap_write_drain(iommu->cap))
		val |= DMA_TLB_WRITE_DRAIN;

	raw_spin_lock_irqsave(&iommu->register_lock, flag);
	/* Note: Only uses first TLB reg currently */
	if (val_iva)
		dmar_writeq(iommu->reg + tlb_offset, val_iva);
	dmar_writeq(iommu->reg + tlb_offset + 8, val);

	/* Make sure hardware complete it */
	IOMMU_WAIT_OP(iommu, tlb_offset + 8,
		dmar_readq, (!(val & DMA_TLB_IVT)), val);

	raw_spin_unlock_irqrestore(&iommu->register_lock, flag);

	/* check IOTLB invalidation granularity */
	if (DMA_TLB_IAIG(val) == 0)
		pr_err("Flush IOTLB failed\n");
	if (DMA_TLB_IAIG(val) != DMA_TLB_IIRG(type))
		pr_debug("TLB flush request %Lx, actual %Lx\n",
			(unsigned long long)DMA_TLB_IIRG(type),
			(unsigned long long)DMA_TLB_IAIG(val));
}

static struct device_domain_info *
domain_lookup_dev_info(struct dmar_domain *domain,
		       struct intel_iommu *iommu, u8 bus, u8 devfn)
{
	struct device_domain_info *info;
	unsigned long flags;

	spin_lock_irqsave(&domain->lock, flags);
	list_for_each_entry(info, &domain->devices, link) {
		if (info->iommu == iommu && info->bus == bus &&
		    info->devfn == devfn) {
			spin_unlock_irqrestore(&domain->lock, flags);
			return info;
		}
	}
	spin_unlock_irqrestore(&domain->lock, flags);

	return NULL;
}

/*
 * The extra devTLB flush quirk impacts those QAT devices with PCI device
 * IDs ranging from 0x4940 to 0x4943. It is exempted from risky_device()
 * check because it applies only to the built-in QAT devices and it doesn't
 * grant additional privileges.
 */
#define BUGGY_QAT_DEVID_MASK 0x4940
static bool dev_needs_extra_dtlb_flush(struct pci_dev *pdev)
{
	if (pdev->vendor != PCI_VENDOR_ID_INTEL)
		return false;

	if ((pdev->device & 0xfffc) != BUGGY_QAT_DEVID_MASK)
		return false;

	return true;
}

static void iommu_enable_pci_ats(struct device_domain_info *info)
{
	struct pci_dev *pdev;

	if (!info->ats_supported)
		return;

	pdev = to_pci_dev(info->dev);
	if (!pci_ats_page_aligned(pdev))
		return;

	if (!pci_enable_ats(pdev, VTD_PAGE_SHIFT))
		info->ats_enabled = 1;
}

static void iommu_disable_pci_ats(struct device_domain_info *info)
{
	if (!info->ats_enabled)
		return;

	pci_disable_ats(to_pci_dev(info->dev));
	info->ats_enabled = 0;
}

static void iommu_enable_pci_pri(struct device_domain_info *info)
{
	struct pci_dev *pdev;

	if (!info->ats_enabled || !info->pri_supported)
		return;

	pdev = to_pci_dev(info->dev);
	/* PASID is required in PRG Response Message. */
	if (info->pasid_enabled && !pci_prg_resp_pasid_required(pdev))
		return;

	if (pci_reset_pri(pdev))
		return;

	if (!pci_enable_pri(pdev, PRQ_DEPTH))
		info->pri_enabled = 1;
}

static void iommu_disable_pci_pri(struct device_domain_info *info)
{
	if (!info->pri_enabled)
		return;

	if (WARN_ON(info->iopf_refcount))
		iopf_queue_remove_device(info->iommu->iopf_queue, info->dev);

	pci_disable_pri(to_pci_dev(info->dev));
	info->pri_enabled = 0;
}

static void intel_flush_iotlb_all(struct iommu_domain *domain)
{
	cache_tag_flush_all(to_dmar_domain(domain));
}

static void iommu_disable_protect_mem_regions(struct intel_iommu *iommu)
{
	u32 pmen;
	unsigned long flags;

	if (!cap_plmr(iommu->cap) && !cap_phmr(iommu->cap))
		return;

	raw_spin_lock_irqsave(&iommu->register_lock, flags);
	pmen = readl(iommu->reg + DMAR_PMEN_REG);
	pmen &= ~DMA_PMEN_EPM;
	writel(pmen, iommu->reg + DMAR_PMEN_REG);

	/* wait for the protected region status bit to clear */
	IOMMU_WAIT_OP(iommu, DMAR_PMEN_REG,
		readl, !(pmen & DMA_PMEN_PRS), pmen);

	raw_spin_unlock_irqrestore(&iommu->register_lock, flags);
}

static void iommu_enable_translation(struct intel_iommu *iommu)
{
	u32 sts;
	unsigned long flags;

	raw_spin_lock_irqsave(&iommu->register_lock, flags);
	iommu->gcmd |= DMA_GCMD_TE;
	writel(iommu->gcmd, iommu->reg + DMAR_GCMD_REG);

	/* Make sure hardware complete it */
	IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
		      readl, (sts & DMA_GSTS_TES), sts);

	raw_spin_unlock_irqrestore(&iommu->register_lock, flags);
}

static void iommu_disable_translation(struct intel_iommu *iommu)
{
	u32 sts;
	unsigned long flag;

	if (iommu_skip_te_disable && iommu->drhd->gfx_dedicated &&
	    (cap_read_drain(iommu->cap) || cap_write_drain(iommu->cap)))
		return;

	raw_spin_lock_irqsave(&iommu->register_lock, flag);
	iommu->gcmd &= ~DMA_GCMD_TE;
	writel(iommu->gcmd, iommu->reg + DMAR_GCMD_REG);

	/* Make sure hardware complete it */
	IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
		      readl, (!(sts & DMA_GSTS_TES)), sts);

	raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
}

static void disable_dmar_iommu(struct intel_iommu *iommu)
{
	/*
	 * All iommu domains must have been detached from the devices,
	 * hence there should be no domain IDs in use.
	 */
	if (WARN_ON(!ida_is_empty(&iommu->domain_ida)))
		return;

	if (iommu->gcmd & DMA_GCMD_TE)
		iommu_disable_translation(iommu);
}

static void free_dmar_iommu(struct intel_iommu *iommu)
{
	if (iommu->copied_tables) {
		bitmap_free(iommu->copied_tables);
		iommu->copied_tables = NULL;
	}

	/* free context mapping */
	free_context_table(iommu);

	if (ecap_prs(iommu->ecap))
		intel_iommu_finish_prq(iommu);
}

/*
 * Check and return whether first level is used by default for
 * DMA translation.
 */
static bool first_level_by_default(struct intel_iommu *iommu)
{
	/* Only SL is available in legacy mode */
	if (!sm_supported(iommu))
		return false;

	/* Only level (either FL or SL) is available, just use it */
	if (ecap_flts(iommu->ecap) ^ ecap_slts(iommu->ecap))
		return ecap_flts(iommu->ecap);

	return true;
}

int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
{
	struct iommu_domain_info *info, *curr;
	int num, ret = -ENOSPC;

	if (domain->domain.type == IOMMU_DOMAIN_SVA)
		return 0;

	info = kzalloc(sizeof(*info), GFP_KERNEL);
	if (!info)
		return -ENOMEM;

	guard(mutex)(&iommu->did_lock);
	curr = xa_load(&domain->iommu_array, iommu->seq_id);
	if (curr) {
		curr->refcnt++;
		kfree(info);
		return 0;
	}

	num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID,
			      cap_ndoms(iommu->cap) - 1, GFP_KERNEL);
	if (num < 0) {
		pr_err("%s: No free domain ids\n", iommu->name);
		goto err_unlock;
	}

	info->refcnt	= 1;
	info->did	= num;
	info->iommu	= iommu;
	curr = xa_cmpxchg(&domain->iommu_array, iommu->seq_id,
			  NULL, info, GFP_KERNEL);
	if (curr) {
		ret = xa_err(curr) ? : -EBUSY;
		goto err_clear;
	}

	return 0;

err_clear:
	ida_free(&iommu->domain_ida, info->did);
err_unlock:
	kfree(info);
	return ret;
}

void domain_detach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu)
{
	struct iommu_domain_info *info;

	if (domain->domain.type == IOMMU_DOMAIN_SVA)
		return;

	guard(mutex)(&iommu->did_lock);
	info = xa_load(&domain->iommu_array, iommu->seq_id);
	if (--info->refcnt == 0) {
		ida_free(&iommu->domain_ida, info->did);
		xa_erase(&domain->iommu_array, iommu->seq_id);
		kfree(info);
	}
}

/*
 * For kdump cases, old valid entries may be cached due to the
 * in-flight DMA and copied pgtable, but there is no unmapping
 * behaviour for them, thus we need an explicit cache flush for
 * the newly-mapped device. For kdump, at this point, the device
 * is supposed to finish reset at its driver probe stage, so no
 * in-flight DMA will exist, and we don't need to worry anymore
 * hereafter.
 */
static void copied_context_tear_down(struct intel_iommu *iommu,
				     struct context_entry *context,
				     u8 bus, u8 devfn)
{
	u16 did_old;

	if (!context_copied(iommu, bus, devfn))
		return;

	assert_spin_locked(&iommu->lock);

	did_old = context_domain_id(context);
	context_clear_entry(context);

	if (did_old < cap_ndoms(iommu->cap)) {
		iommu->flush.flush_context(iommu, did_old,
					   PCI_DEVID(bus, devfn),
					   DMA_CCMD_MASK_NOBIT,
					   DMA_CCMD_DEVICE_INVL);
		iommu->flush.flush_iotlb(iommu, did_old, 0, 0,
					 DMA_TLB_DSI_FLUSH);
	}

	clear_context_copied(iommu, bus, devfn);
}

/*
 * It's a non-present to present mapping. If hardware doesn't cache
 * non-present entry we only need to flush the write-buffer. If the
 * _does_ cache non-present entries, then it does so in the special
 * domain #0, which we have to flush:
 */
static void context_present_cache_flush(struct intel_iommu *iommu, u16 did,
					u8 bus, u8 devfn)
{
	if (cap_caching_mode(iommu->cap)) {
		iommu->flush.flush_context(iommu, 0,
					   PCI_DEVID(bus, devfn),
					   DMA_CCMD_MASK_NOBIT,
					   DMA_CCMD_DEVICE_INVL);
		iommu->flush.flush_iotlb(iommu, did, 0, 0, DMA_TLB_DSI_FLUSH);
	} else {
		iommu_flush_write_buffer(iommu);
	}
}

static int domain_context_mapping_one(struct dmar_domain *domain,
				      struct intel_iommu *iommu,
				      u8 bus, u8 devfn)
{
	struct device_domain_info *info =
			domain_lookup_dev_info(domain, iommu, bus, devfn);
	u16 did = domain_id_iommu(domain, iommu);
	int translation = CONTEXT_TT_MULTI_LEVEL;
	struct dma_pte *pgd = domain->pgd;
	struct context_entry *context;
	int ret;

	if (WARN_ON(!intel_domain_is_ss_paging(domain)))
		return -EINVAL;

	pr_debug("Set context mapping for %02x:%02x.%d\n",
		bus, PCI_SLOT(devfn), PCI_FUNC(devfn));

	spin_lock(&iommu->lock);
	ret = -ENOMEM;
	context = iommu_context_addr(iommu, bus, devfn, 1);
	if (!context)
		goto out_unlock;

	ret = 0;
	if (context_present(context) && !context_copied(iommu, bus, devfn))
		goto out_unlock;

	copied_context_tear_down(iommu, context, bus, devfn);
	context_clear_entry(context);
	context_set_domain_id(context, did);

	if (info && info->ats_supported)
		translation = CONTEXT_TT_DEV_IOTLB;
	else
		translation = CONTEXT_TT_MULTI_LEVEL;

	context_set_address_root(context, virt_to_phys(pgd));
	context_set_address_width(context, domain->agaw);
	context_set_translation_type(context, translation);
	context_set_fault_enable(context);
	context_set_present(context);
	if (!ecap_coherent(iommu->ecap))
		clflush_cache_range(context, sizeof(*context));
	context_present_cache_flush(iommu, did, bus, devfn);
	ret = 0;

out_unlock:
	spin_unlock(&iommu->lock);

	return ret;
}

static int domain_context_mapping_cb(struct pci_dev *pdev,
				     u16 alias, void *opaque)
{
	struct device_domain_info *info = dev_iommu_priv_get(&pdev->dev);
	struct intel_iommu *iommu = info->iommu;
	struct dmar_domain *domain = opaque;

	return domain_context_mapping_one(domain, iommu,
					  PCI_BUS_NUM(alias), alias & 0xff);
}

static int
domain_context_mapping(struct dmar_domain *domain, struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;
	u8 bus = info->bus, devfn = info->devfn;
	int ret;

	if (!dev_is_pci(dev))
		return domain_context_mapping_one(domain, iommu, bus, devfn);

	ret = pci_for_each_dma_alias(to_pci_dev(dev),
				     domain_context_mapping_cb, domain);
	if (ret)
		return ret;

	iommu_enable_pci_ats(info);

	return 0;
}

/* Return largest possible superpage level for a given mapping */
static int hardware_largepage_caps(struct dmar_domain *domain, unsigned long iov_pfn,
				   unsigned long phy_pfn, unsigned long pages)
{
	int support, level = 1;
	unsigned long pfnmerge;

	support = domain->iommu_superpage;

	/* To use a large page, the virtual *and* physical addresses
	   must be aligned to 2MiB/1GiB/etc. Lower bits set in either
	   of them will mean we have to use smaller pages. So just
	   merge them and check both at once. */
	pfnmerge = iov_pfn | phy_pfn;

	while (support && !(pfnmerge & ~VTD_STRIDE_MASK)) {
		pages >>= VTD_STRIDE_SHIFT;
		if (!pages)
			break;
		pfnmerge >>= VTD_STRIDE_SHIFT;
		level++;
		support--;
	}
	return level;
}

/*
 * Ensure that old small page tables are removed to make room for superpage(s).
 * We're going to add new large pages, so make sure we don't remove their parent
 * tables. The IOTLB/devTLBs should be flushed if any PDE/PTEs are cleared.
 */
static void switch_to_super_page(struct dmar_domain *domain,
				 unsigned long start_pfn,
				 unsigned long end_pfn, int level)
{
	unsigned long lvl_pages = lvl_to_nr_pages(level);
	struct dma_pte *pte = NULL;

	if (WARN_ON(!IS_ALIGNED(start_pfn, lvl_pages) ||
		    !IS_ALIGNED(end_pfn + 1, lvl_pages)))
		return;

	while (start_pfn <= end_pfn) {
		if (!pte)
			pte = pfn_to_dma_pte(domain, start_pfn, &level,
					     GFP_ATOMIC);

		if (dma_pte_present(pte)) {
			dma_pte_free_pagetable(domain, start_pfn,
					       start_pfn + lvl_pages - 1,
					       level + 1);

			cache_tag_flush_range(domain, start_pfn << VTD_PAGE_SHIFT,
					      end_pfn << VTD_PAGE_SHIFT, 0);
		}

		pte++;
		start_pfn += lvl_pages;
		if (first_pte_in_page(pte))
			pte = NULL;
	}
}

static int
__domain_mapping(struct dmar_domain *domain, unsigned long iov_pfn,
		 unsigned long phys_pfn, unsigned long nr_pages, int prot,
		 gfp_t gfp)
{
	struct dma_pte *first_pte = NULL, *pte = NULL;
	unsigned int largepage_lvl = 0;
	unsigned long lvl_pages = 0;
	phys_addr_t pteval;
	u64 attr;

	if (unlikely(!domain_pfn_supported(domain, iov_pfn + nr_pages - 1)))
		return -EINVAL;

	if ((prot & (DMA_PTE_READ|DMA_PTE_WRITE)) == 0)
		return -EINVAL;

	if (!(prot & DMA_PTE_WRITE) && domain->nested_parent) {
		pr_err_ratelimited("Read-only mapping is disallowed on the domain which serves as the parent in a nested configuration, due to HW errata (ERRATA_772415_SPR17)\n");
		return -EINVAL;
	}

	attr = prot & (DMA_PTE_READ | DMA_PTE_WRITE | DMA_PTE_SNP);
	if (domain->use_first_level) {
		attr |= DMA_FL_PTE_PRESENT | DMA_FL_PTE_US | DMA_FL_PTE_ACCESS;
		if (prot & DMA_PTE_WRITE)
			attr |= DMA_FL_PTE_DIRTY;
	}

	domain->has_mappings = true;

	pteval = ((phys_addr_t)phys_pfn << VTD_PAGE_SHIFT) | attr;

	while (nr_pages > 0) {
		uint64_t tmp;

		if (!pte) {
			largepage_lvl = hardware_largepage_caps(domain, iov_pfn,
					phys_pfn, nr_pages);

			pte = pfn_to_dma_pte(domain, iov_pfn, &largepage_lvl,
					     gfp);
			if (!pte)
				return -ENOMEM;
			first_pte = pte;

			lvl_pages = lvl_to_nr_pages(largepage_lvl);

			/* It is large page*/
			if (largepage_lvl > 1) {
				unsigned long end_pfn;
				unsigned long pages_to_remove;

				pteval |= DMA_PTE_LARGE_PAGE;
				pages_to_remove = min_t(unsigned long,
							round_down(nr_pages, lvl_pages),
							nr_pte_to_next_page(pte) * lvl_pages);
				end_pfn = iov_pfn + pages_to_remove - 1;
				switch_to_super_page(domain, iov_pfn, end_pfn, largepage_lvl);
			} else {
				pteval &= ~(uint64_t)DMA_PTE_LARGE_PAGE;
			}

		}
		/* We don't need lock here, nobody else
		 * touches the iova range
		 */
		tmp = 0ULL;
		if (!try_cmpxchg64_local(&pte->val, &tmp, pteval)) {
			static int dumps = 5;
			pr_crit("ERROR: DMA PTE for vPFN 0x%lx already set (to %llx not %llx)\n",
				iov_pfn, tmp, (unsigned long long)pteval);
			if (dumps) {
				dumps--;
				debug_dma_dump_mappings(NULL);
			}
			WARN_ON(1);
		}

		nr_pages -= lvl_pages;
		iov_pfn += lvl_pages;
		phys_pfn += lvl_pages;
		pteval += lvl_pages * VTD_PAGE_SIZE;

		/* If the next PTE would be the first in a new page, then we
		 * need to flush the cache on the entries we've just written.
		 * And then we'll need to recalculate 'pte', so clear it and
		 * let it get set again in the if (!pte) block above.
		 *
		 * If we're done (!nr_pages) we need to flush the cache too.
		 *
		 * Also if we've been setting superpages, we may need to
		 * recalculate 'pte' and switch back to smaller pages for the
		 * end of the mapping, if the trailing size is not enough to
		 * use another superpage (i.e. nr_pages < lvl_pages).
		 */
		pte++;
		if (!nr_pages || first_pte_in_page(pte) ||
		    (largepage_lvl > 1 && nr_pages < lvl_pages)) {
			domain_flush_cache(domain, first_pte,
					   (void *)pte - (void *)first_pte);
			pte = NULL;
		}
	}

	return 0;
}

static void domain_context_clear_one(struct device_domain_info *info, u8 bus, u8 devfn)
{
	struct intel_iommu *iommu = info->iommu;
	struct context_entry *context;
	u16 did;

	spin_lock(&iommu->lock);
	context = iommu_context_addr(iommu, bus, devfn, 0);
	if (!context) {
		spin_unlock(&iommu->lock);
		return;
	}

	did = context_domain_id(context);
	context_clear_entry(context);
	__iommu_flush_cache(iommu, context, sizeof(*context));
	spin_unlock(&iommu->lock);
	intel_context_flush_no_pasid(info, context, did);
}

int __domain_setup_first_level(struct intel_iommu *iommu, struct device *dev,
			       ioasid_t pasid, u16 did, phys_addr_t fsptptr,
			       int flags, struct iommu_domain *old)
{
	if (!old)
		return intel_pasid_setup_first_level(iommu, dev, fsptptr, pasid,
						     did, flags);
	return intel_pasid_replace_first_level(iommu, dev, fsptptr, pasid, did,
					       iommu_domain_did(old, iommu),
					       flags);
}

static int domain_setup_second_level(struct intel_iommu *iommu,
				     struct dmar_domain *domain,
				     struct device *dev, ioasid_t pasid,
				     struct iommu_domain *old)
{
	if (!old)
		return intel_pasid_setup_second_level(iommu, domain,
						      dev, pasid);
	return intel_pasid_replace_second_level(iommu, domain, dev,
						iommu_domain_did(old, iommu),
						pasid);
}

static int domain_setup_passthrough(struct intel_iommu *iommu,
				    struct device *dev, ioasid_t pasid,
				    struct iommu_domain *old)
{
	if (!old)
		return intel_pasid_setup_pass_through(iommu, dev, pasid);
	return intel_pasid_replace_pass_through(iommu, dev,
						iommu_domain_did(old, iommu),
						pasid);
}

static int domain_setup_first_level(struct intel_iommu *iommu,
				    struct dmar_domain *domain,
				    struct device *dev,
				    u32 pasid, struct iommu_domain *old)
{
	struct dma_pte *pgd = domain->pgd;
	int level, flags = 0;

	level = agaw_to_level(domain->agaw);
	if (level != 4 && level != 5)
		return -EINVAL;

	if (level == 5)
		flags |= PASID_FLAG_FL5LP;

	if (domain->force_snooping)
		flags |= PASID_FLAG_PAGE_SNOOP;

	return __domain_setup_first_level(iommu, dev, pasid,
					  domain_id_iommu(domain, iommu),
					  __pa(pgd), flags, old);
}

static int dmar_domain_attach_device(struct dmar_domain *domain,
				     struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;
	unsigned long flags;
	int ret;

	ret = domain_attach_iommu(domain, iommu);
	if (ret)
		return ret;

	info->domain = domain;
	info->domain_attached = true;
	spin_lock_irqsave(&domain->lock, flags);
	list_add(&info->link, &domain->devices);
	spin_unlock_irqrestore(&domain->lock, flags);

	if (dev_is_real_dma_subdevice(dev))
		return 0;

	if (!sm_supported(iommu))
		ret = domain_context_mapping(domain, dev);
	else if (intel_domain_is_fs_paging(domain))
		ret = domain_setup_first_level(iommu, domain, dev,
					       IOMMU_NO_PASID, NULL);
	else if (intel_domain_is_ss_paging(domain))
		ret = domain_setup_second_level(iommu, domain, dev,
						IOMMU_NO_PASID, NULL);
	else if (WARN_ON(true))
		ret = -EINVAL;

	if (ret)
		goto out_block_translation;

	ret = cache_tag_assign_domain(domain, dev, IOMMU_NO_PASID);
	if (ret)
		goto out_block_translation;

	return 0;

out_block_translation:
	device_block_translation(dev);
	return ret;
}

/**
 * device_rmrr_is_relaxable - Test whether the RMRR of this device
 * is relaxable (ie. is allowed to be not enforced under some conditions)
 * @dev: device handle
 *
 * We assume that PCI USB devices with RMRRs have them largely
 * for historical reasons and that the RMRR space is not actively used post
 * boot.  This exclusion may change if vendors begin to abuse it.
 *
 * The same exception is made for graphics devices, with the requirement that
 * any use of the RMRR regions will be torn down before assigning the device
 * to a guest.
 *
 * Return: true if the RMRR is relaxable, false otherwise
 */
static bool device_rmrr_is_relaxable(struct device *dev)
{
	struct pci_dev *pdev;

	if (!dev_is_pci(dev))
		return false;

	pdev = to_pci_dev(dev);
	if (IS_USB_DEVICE(pdev) || IS_GFX_DEVICE(pdev))
		return true;
	else
		return false;
}

static int device_def_domain_type(struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;

	/*
	 * Hardware does not support the passthrough translation mode.
	 * Always use a dynamaic mapping domain.
	 */
	if (!ecap_pass_through(iommu->ecap))
		return IOMMU_DOMAIN_DMA;

	if (dev_is_pci(dev)) {
		struct pci_dev *pdev = to_pci_dev(dev);

		if ((iommu_identity_mapping & IDENTMAP_AZALIA) && IS_AZALIA(pdev))
			return IOMMU_DOMAIN_IDENTITY;
	}

	return 0;
}

static void intel_iommu_init_qi(struct intel_iommu *iommu)
{
	/*
	 * Start from the sane iommu hardware state.
	 * If the queued invalidation is already initialized by us
	 * (for example, while enabling interrupt-remapping) then
	 * we got the things already rolling from a sane state.
	 */
	if (!iommu->qi) {
		/*
		 * Clear any previous faults.
		 */
		dmar_fault(-1, iommu);
		/*
		 * Disable queued invalidation if supported and already enabled
		 * before OS handover.
		 */
		dmar_disable_qi(iommu);
	}

	if (dmar_enable_qi(iommu)) {
		/*
		 * Queued Invalidate not enabled, use Register Based Invalidate
		 */
		iommu->flush.flush_context = __iommu_flush_context;
		iommu->flush.flush_iotlb = __iommu_flush_iotlb;
		pr_info("%s: Using Register based invalidation\n",
			iommu->name);
	} else {
		iommu->flush.flush_context = qi_flush_context;
		iommu->flush.flush_iotlb = qi_flush_iotlb;
		pr_info("%s: Using Queued invalidation\n", iommu->name);
	}
}

static int copy_context_table(struct intel_iommu *iommu,
			      struct root_entry *old_re,
			      struct context_entry **tbl,
			      int bus, bool ext)
{
	int tbl_idx, pos = 0, idx, devfn, ret = 0, did;
	struct context_entry *new_ce = NULL, ce;
	struct context_entry *old_ce = NULL;
	struct root_entry re;
	phys_addr_t old_ce_phys;

	tbl_idx = ext ? bus * 2 : bus;
	memcpy(&re, old_re, sizeof(re));

	for (devfn = 0; devfn < 256; devfn++) {
		/* First calculate the correct index */
		idx = (ext ? devfn * 2 : devfn) % 256;

		if (idx == 0) {
			/* First save what we may have and clean up */
			if (new_ce) {
				tbl[tbl_idx] = new_ce;
				__iommu_flush_cache(iommu, new_ce,
						    VTD_PAGE_SIZE);
				pos = 1;
			}

			if (old_ce)
				memunmap(old_ce);

			ret = 0;
			if (devfn < 0x80)
				old_ce_phys = root_entry_lctp(&re);
			else
				old_ce_phys = root_entry_uctp(&re);

			if (!old_ce_phys) {
				if (ext && devfn == 0) {
					/* No LCTP, try UCTP */
					devfn = 0x7f;
					continue;
				} else {
					goto out;
				}
			}

			ret = -ENOMEM;
			old_ce = memremap(old_ce_phys, PAGE_SIZE,
					MEMREMAP_WB);
			if (!old_ce)
				goto out;

			new_ce = iommu_alloc_pages_node_sz(iommu->node,
							   GFP_KERNEL, SZ_4K);
			if (!new_ce)
				goto out_unmap;

			ret = 0;
		}

		/* Now copy the context entry */
		memcpy(&ce, old_ce + idx, sizeof(ce));

		if (!context_present(&ce))
			continue;

		did = context_domain_id(&ce);
		if (did >= 0 && did < cap_ndoms(iommu->cap))
			ida_alloc_range(&iommu->domain_ida, did, did, GFP_KERNEL);

		set_context_copied(iommu, bus, devfn);
		new_ce[idx] = ce;
	}

	tbl[tbl_idx + pos] = new_ce;

	__iommu_flush_cache(iommu, new_ce, VTD_PAGE_SIZE);

out_unmap:
	memunmap(old_ce);

out:
	return ret;
}

static int copy_translation_tables(struct intel_iommu *iommu)
{
	struct context_entry **ctxt_tbls;
	struct root_entry *old_rt;
	phys_addr_t old_rt_phys;
	int ctxt_table_entries;
	u64 rtaddr_reg;
	int bus, ret;
	bool new_ext, ext;

	rtaddr_reg = dmar_readq(iommu->reg + DMAR_RTADDR_REG);
	ext        = !!(rtaddr_reg & DMA_RTADDR_SMT);
	new_ext    = !!sm_supported(iommu);

	/*
	 * The RTT bit can only be changed when translation is disabled,
	 * but disabling translation means to open a window for data
	 * corruption. So bail out and don't copy anything if we would
	 * have to change the bit.
	 */
	if (new_ext != ext)
		return -EINVAL;

	iommu->copied_tables = bitmap_zalloc(BIT_ULL(16), GFP_KERNEL);
	if (!iommu->copied_tables)
		return -ENOMEM;

	old_rt_phys = rtaddr_reg & VTD_PAGE_MASK;
	if (!old_rt_phys)
		return -EINVAL;

	old_rt = memremap(old_rt_phys, PAGE_SIZE, MEMREMAP_WB);
	if (!old_rt)
		return -ENOMEM;

	/* This is too big for the stack - allocate it from slab */
	ctxt_table_entries = ext ? 512 : 256;
	ret = -ENOMEM;
	ctxt_tbls = kcalloc(ctxt_table_entries, sizeof(void *), GFP_KERNEL);
	if (!ctxt_tbls)
		goto out_unmap;

	for (bus = 0; bus < 256; bus++) {
		ret = copy_context_table(iommu, &old_rt[bus],
					 ctxt_tbls, bus, ext);
		if (ret) {
			pr_err("%s: Failed to copy context table for bus %d\n",
				iommu->name, bus);
			continue;
		}
	}

	spin_lock(&iommu->lock);

	/* Context tables are copied, now write them to the root_entry table */
	for (bus = 0; bus < 256; bus++) {
		int idx = ext ? bus * 2 : bus;
		u64 val;

		if (ctxt_tbls[idx]) {
			val = virt_to_phys(ctxt_tbls[idx]) | 1;
			iommu->root_entry[bus].lo = val;
		}

		if (!ext || !ctxt_tbls[idx + 1])
			continue;

		val = virt_to_phys(ctxt_tbls[idx + 1]) | 1;
		iommu->root_entry[bus].hi = val;
	}

	spin_unlock(&iommu->lock);

	kfree(ctxt_tbls);

	__iommu_flush_cache(iommu, iommu->root_entry, PAGE_SIZE);

	ret = 0;

out_unmap:
	memunmap(old_rt);

	return ret;
}

static int __init init_dmars(void)
{
	struct dmar_drhd_unit *drhd;
	struct intel_iommu *iommu;
	int ret;

	for_each_iommu(iommu, drhd) {
		if (drhd->ignored) {
			iommu_disable_translation(iommu);
			continue;
		}

		/*
		 * Find the max pasid size of all IOMMU's in the system.
		 * We need to ensure the system pasid table is no bigger
		 * than the smallest supported.
		 */
		if (pasid_supported(iommu)) {
			u32 temp = 2 << ecap_pss(iommu->ecap);

			intel_pasid_max_id = min_t(u32, temp,
						   intel_pasid_max_id);
		}

		intel_iommu_init_qi(iommu);
		init_translation_status(iommu);

		if (translation_pre_enabled(iommu) && !is_kdump_kernel()) {
			iommu_disable_translation(iommu);
			clear_translation_pre_enabled(iommu);
			pr_warn("Translation was enabled for %s but we are not in kdump mode\n",
				iommu->name);
		}

		/*
		 * TBD:
		 * we could share the same root & context tables
		 * among all IOMMU's. Need to Split it later.
		 */
		ret = iommu_alloc_root_entry(iommu);
		if (ret)
			goto free_iommu;

		if (translation_pre_enabled(iommu)) {
			pr_info("Translation already enabled - trying to copy translation structures\n");

			ret = copy_translation_tables(iommu);
			if (ret) {
				/*
				 * We found the IOMMU with translation
				 * enabled - but failed to copy over the
				 * old root-entry table. Try to proceed
				 * by disabling translation now and
				 * allocating a clean root-entry table.
				 * This might cause DMAR faults, but
				 * probably the dump will still succeed.
				 */
				pr_err("Failed to copy translation tables from previous kernel for %s\n",
				       iommu->name);
				iommu_disable_translation(iommu);
				clear_translation_pre_enabled(iommu);
			} else {
				pr_info("Copied translation tables from previous kernel for %s\n",
					iommu->name);
			}
		}

		intel_svm_check(iommu);
	}

	/*
	 * Now that qi is enabled on all iommus, set the root entry and flush
	 * caches. This is required on some Intel X58 chipsets, otherwise the
	 * flush_context function will loop forever and the boot hangs.
	 */
	for_each_active_iommu(iommu, drhd) {
		iommu_flush_write_buffer(iommu);
		iommu_set_root_entry(iommu);
	}

	check_tylersburg_isoch();

	/*
	 * for each drhd
	 *   enable fault log
	 *   global invalidate context cache
	 *   global invalidate iotlb
	 *   enable translation
	 */
	for_each_iommu(iommu, drhd) {
		if (drhd->ignored) {
			/*
			 * we always have to disable PMRs or DMA may fail on
			 * this device
			 */
			if (force_on)
				iommu_disable_protect_mem_regions(iommu);
			continue;
		}

		iommu_flush_write_buffer(iommu);

		if (ecap_prs(iommu->ecap)) {
			/*
			 * Call dmar_alloc_hwirq() with dmar_global_lock held,
			 * could cause possible lock race condition.
			 */
			up_write(&dmar_global_lock);
			ret = intel_iommu_enable_prq(iommu);
			down_write(&dmar_global_lock);
			if (ret)
				goto free_iommu;
		}

		ret = dmar_set_interrupt(iommu);
		if (ret)
			goto free_iommu;
	}

	return 0;

free_iommu:
	for_each_active_iommu(iommu, drhd) {
		disable_dmar_iommu(iommu);
		free_dmar_iommu(iommu);
	}

	return ret;
}

static void __init init_no_remapping_devices(void)
{
	struct dmar_drhd_unit *drhd;
	struct device *dev;
	int i;

	for_each_drhd_unit(drhd) {
		if (!drhd->include_all) {
			for_each_active_dev_scope(drhd->devices,
						  drhd->devices_cnt, i, dev)
				break;
			/* ignore DMAR unit if no devices exist */
			if (i == drhd->devices_cnt)
				drhd->ignored = 1;
		}
	}

	for_each_active_drhd_unit(drhd) {
		if (drhd->include_all)
			continue;

		for_each_active_dev_scope(drhd->devices,
					  drhd->devices_cnt, i, dev)
			if (!dev_is_pci(dev) || !IS_GFX_DEVICE(to_pci_dev(dev)))
				break;
		if (i < drhd->devices_cnt)
			continue;

		/* This IOMMU has *only* gfx devices. Either bypass it or
		   set the gfx_mapped flag, as appropriate */
		drhd->gfx_dedicated = 1;
		if (disable_igfx_iommu)
			drhd->ignored = 1;
	}
}

#ifdef CONFIG_SUSPEND
static int init_iommu_hw(void)
{
	struct dmar_drhd_unit *drhd;
	struct intel_iommu *iommu = NULL;
	int ret;

	for_each_active_iommu(iommu, drhd) {
		if (iommu->qi) {
			ret = dmar_reenable_qi(iommu);
			if (ret)
				return ret;
		}
	}

	for_each_iommu(iommu, drhd) {
		if (drhd->ignored) {
			/*
			 * we always have to disable PMRs or DMA may fail on
			 * this device
			 */
			if (force_on)
				iommu_disable_protect_mem_regions(iommu);
			continue;
		}

		iommu_flush_write_buffer(iommu);
		iommu_set_root_entry(iommu);
		iommu_enable_translation(iommu);
		iommu_disable_protect_mem_regions(iommu);
	}

	return 0;
}

static void iommu_flush_all(void)
{
	struct dmar_drhd_unit *drhd;
	struct intel_iommu *iommu;

	for_each_active_iommu(iommu, drhd) {
		iommu->flush.flush_context(iommu, 0, 0, 0,
					   DMA_CCMD_GLOBAL_INVL);
		iommu->flush.flush_iotlb(iommu, 0, 0, 0,
					 DMA_TLB_GLOBAL_FLUSH);
	}
}

static int iommu_suspend(void)
{
	struct dmar_drhd_unit *drhd;
	struct intel_iommu *iommu = NULL;
	unsigned long flag;

	iommu_flush_all();

	for_each_active_iommu(iommu, drhd) {
		iommu_disable_translation(iommu);

		raw_spin_lock_irqsave(&iommu->register_lock, flag);

		iommu->iommu_state[SR_DMAR_FECTL_REG] =
			readl(iommu->reg + DMAR_FECTL_REG);
		iommu->iommu_state[SR_DMAR_FEDATA_REG] =
			readl(iommu->reg + DMAR_FEDATA_REG);
		iommu->iommu_state[SR_DMAR_FEADDR_REG] =
			readl(iommu->reg + DMAR_FEADDR_REG);
		iommu->iommu_state[SR_DMAR_FEUADDR_REG] =
			readl(iommu->reg + DMAR_FEUADDR_REG);

		raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
	}
	return 0;
}

static void iommu_resume(void)
{
	struct dmar_drhd_unit *drhd;
	struct intel_iommu *iommu = NULL;
	unsigned long flag;

	if (init_iommu_hw()) {
		if (force_on)
			panic("tboot: IOMMU setup failed, DMAR can not resume!\n");
		else
			WARN(1, "IOMMU setup failed, DMAR can not resume!\n");
		return;
	}

	for_each_active_iommu(iommu, drhd) {

		raw_spin_lock_irqsave(&iommu->register_lock, flag);

		writel(iommu->iommu_state[SR_DMAR_FECTL_REG],
			iommu->reg + DMAR_FECTL_REG);
		writel(iommu->iommu_state[SR_DMAR_FEDATA_REG],
			iommu->reg + DMAR_FEDATA_REG);
		writel(iommu->iommu_state[SR_DMAR_FEADDR_REG],
			iommu->reg + DMAR_FEADDR_REG);
		writel(iommu->iommu_state[SR_DMAR_FEUADDR_REG],
			iommu->reg + DMAR_FEUADDR_REG);

		raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
	}
}

static struct syscore_ops iommu_syscore_ops = {
	.resume		= iommu_resume,
	.suspend	= iommu_suspend,
};

static void __init init_iommu_pm_ops(void)
{
	register_syscore_ops(&iommu_syscore_ops);
}

#else
static inline void init_iommu_pm_ops(void) {}
#endif	/* CONFIG_PM */

static int __init rmrr_sanity_check(struct acpi_dmar_reserved_memory *rmrr)
{
	if (!IS_ALIGNED(rmrr->base_address, PAGE_SIZE) ||
	    !IS_ALIGNED(rmrr->end_address + 1, PAGE_SIZE) ||
	    rmrr->end_address <= rmrr->base_address ||
	    arch_rmrr_sanity_check(rmrr))
		return -EINVAL;

	return 0;
}

int __init dmar_parse_one_rmrr(struct acpi_dmar_header *header, void *arg)
{
	struct acpi_dmar_reserved_memory *rmrr;
	struct dmar_rmrr_unit *rmrru;

	rmrr = (struct acpi_dmar_reserved_memory *)header;
	if (rmrr_sanity_check(rmrr)) {
		pr_warn(FW_BUG
			   "Your BIOS is broken; bad RMRR [%#018Lx-%#018Lx]\n"
			   "BIOS vendor: %s; Ver: %s; Product Version: %s\n",
			   rmrr->base_address, rmrr->end_address,
			   dmi_get_system_info(DMI_BIOS_VENDOR),
			   dmi_get_system_info(DMI_BIOS_VERSION),
			   dmi_get_system_info(DMI_PRODUCT_VERSION));
		add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_STILL_OK);
	}

	rmrru = kzalloc(sizeof(*rmrru), GFP_KERNEL);
	if (!rmrru)
		goto out;

	rmrru->hdr = header;

	rmrru->base_address = rmrr->base_address;
	rmrru->end_address = rmrr->end_address;

	rmrru->devices = dmar_alloc_dev_scope((void *)(rmrr + 1),
				((void *)rmrr) + rmrr->header.length,
				&rmrru->devices_cnt);
	if (rmrru->devices_cnt && rmrru->devices == NULL)
		goto free_rmrru;

	list_add(&rmrru->list, &dmar_rmrr_units);

	return 0;
free_rmrru:
	kfree(rmrru);
out:
	return -ENOMEM;
}

static struct dmar_atsr_unit *dmar_find_atsr(struct acpi_dmar_atsr *atsr)
{
	struct dmar_atsr_unit *atsru;
	struct acpi_dmar_atsr *tmp;

	list_for_each_entry_rcu(atsru, &dmar_atsr_units, list,
				dmar_rcu_check()) {
		tmp = (struct acpi_dmar_atsr *)atsru->hdr;
		if (atsr->segment != tmp->segment)
			continue;
		if (atsr->header.length != tmp->header.length)
			continue;
		if (memcmp(atsr, tmp, atsr->header.length) == 0)
			return atsru;
	}

	return NULL;
}

int dmar_parse_one_atsr(struct acpi_dmar_header *hdr, void *arg)
{
	struct acpi_dmar_atsr *atsr;
	struct dmar_atsr_unit *atsru;

	if (system_state >= SYSTEM_RUNNING && !intel_iommu_enabled)
		return 0;

	atsr = container_of(hdr, struct acpi_dmar_atsr, header);
	atsru = dmar_find_atsr(atsr);
	if (atsru)
		return 0;

	atsru = kzalloc(sizeof(*atsru) + hdr->length, GFP_KERNEL);
	if (!atsru)
		return -ENOMEM;

	/*
	 * If memory is allocated from slab by ACPI _DSM method, we need to
	 * copy the memory content because the memory buffer will be freed
	 * on return.
	 */
	atsru->hdr = (void *)(atsru + 1);
	memcpy(atsru->hdr, hdr, hdr->length);
	atsru->include_all = atsr->flags & 0x1;
	if (!atsru->include_all) {
		atsru->devices = dmar_alloc_dev_scope((void *)(atsr + 1),
				(void *)atsr + atsr->header.length,
				&atsru->devices_cnt);
		if (atsru->devices_cnt && atsru->devices == NULL) {
			kfree(atsru);
			return -ENOMEM;
		}
	}

	list_add_rcu(&atsru->list, &dmar_atsr_units);

	return 0;
}

static void intel_iommu_free_atsr(struct dmar_atsr_unit *atsru)
{
	dmar_free_dev_scope(&atsru->devices, &atsru->devices_cnt);
	kfree(atsru);
}

int dmar_release_one_atsr(struct acpi_dmar_header *hdr, void *arg)
{
	struct acpi_dmar_atsr *atsr;
	struct dmar_atsr_unit *atsru;

	atsr = container_of(hdr, struct acpi_dmar_atsr, header);
	atsru = dmar_find_atsr(atsr);
	if (atsru) {
		list_del_rcu(&atsru->list);
		synchronize_rcu();
		intel_iommu_free_atsr(atsru);
	}

	return 0;
}

int dmar_check_one_atsr(struct acpi_dmar_header *hdr, void *arg)
{
	int i;
	struct device *dev;
	struct acpi_dmar_atsr *atsr;
	struct dmar_atsr_unit *atsru;

	atsr = container_of(hdr, struct acpi_dmar_atsr, header);
	atsru = dmar_find_atsr(atsr);
	if (!atsru)
		return 0;

	if (!atsru->include_all && atsru->devices && atsru->devices_cnt) {
		for_each_active_dev_scope(atsru->devices, atsru->devices_cnt,
					  i, dev)
			return -EBUSY;
	}

	return 0;
}

static struct dmar_satc_unit *dmar_find_satc(struct acpi_dmar_satc *satc)
{
	struct dmar_satc_unit *satcu;
	struct acpi_dmar_satc *tmp;

	list_for_each_entry_rcu(satcu, &dmar_satc_units, list,
				dmar_rcu_check()) {
		tmp = (struct acpi_dmar_satc *)satcu->hdr;
		if (satc->segment != tmp->segment)
			continue;
		if (satc->header.length != tmp->header.length)
			continue;
		if (memcmp(satc, tmp, satc->header.length) == 0)
			return satcu;
	}

	return NULL;
}

int dmar_parse_one_satc(struct acpi_dmar_header *hdr, void *arg)
{
	struct acpi_dmar_satc *satc;
	struct dmar_satc_unit *satcu;

	if (system_state >= SYSTEM_RUNNING && !intel_iommu_enabled)
		return 0;

	satc = container_of(hdr, struct acpi_dmar_satc, header);
	satcu = dmar_find_satc(satc);
	if (satcu)
		return 0;

	satcu = kzalloc(sizeof(*satcu) + hdr->length, GFP_KERNEL);
	if (!satcu)
		return -ENOMEM;

	satcu->hdr = (void *)(satcu + 1);
	memcpy(satcu->hdr, hdr, hdr->length);
	satcu->atc_required = satc->flags & 0x1;
	satcu->devices = dmar_alloc_dev_scope((void *)(satc + 1),
					      (void *)satc + satc->header.length,
					      &satcu->devices_cnt);
	if (satcu->devices_cnt && !satcu->devices) {
		kfree(satcu);
		return -ENOMEM;
	}
	list_add_rcu(&satcu->list, &dmar_satc_units);

	return 0;
}

static int intel_iommu_add(struct dmar_drhd_unit *dmaru)
{
	struct intel_iommu *iommu = dmaru->iommu;
	int ret;

	/*
	 * Disable translation if already enabled prior to OS handover.
	 */
	if (iommu->gcmd & DMA_GCMD_TE)
		iommu_disable_translation(iommu);

	ret = iommu_alloc_root_entry(iommu);
	if (ret)
		goto out;

	intel_svm_check(iommu);

	if (dmaru->ignored) {
		/*
		 * we always have to disable PMRs or DMA may fail on this device
		 */
		if (force_on)
			iommu_disable_protect_mem_regions(iommu);
		return 0;
	}

	intel_iommu_init_qi(iommu);
	iommu_flush_write_buffer(iommu);

	if (ecap_prs(iommu->ecap)) {
		ret = intel_iommu_enable_prq(iommu);
		if (ret)
			goto disable_iommu;
	}

	ret = dmar_set_interrupt(iommu);
	if (ret)
		goto disable_iommu;

	iommu_set_root_entry(iommu);
	iommu_enable_translation(iommu);

	iommu_disable_protect_mem_regions(iommu);
	return 0;

disable_iommu:
	disable_dmar_iommu(iommu);
out:
	free_dmar_iommu(iommu);
	return ret;
}

int dmar_iommu_hotplug(struct dmar_drhd_unit *dmaru, bool insert)
{
	int ret = 0;
	struct intel_iommu *iommu = dmaru->iommu;

	if (!intel_iommu_enabled)
		return 0;
	if (iommu == NULL)
		return -EINVAL;

	if (insert) {
		ret = intel_iommu_add(dmaru);
	} else {
		disable_dmar_iommu(iommu);
		free_dmar_iommu(iommu);
	}

	return ret;
}

static void intel_iommu_free_dmars(void)
{
	struct dmar_rmrr_unit *rmrru, *rmrr_n;
	struct dmar_atsr_unit *atsru, *atsr_n;
	struct dmar_satc_unit *satcu, *satc_n;

	list_for_each_entry_safe(rmrru, rmrr_n, &dmar_rmrr_units, list) {
		list_del(&rmrru->list);
		dmar_free_dev_scope(&rmrru->devices, &rmrru->devices_cnt);
		kfree(rmrru);
	}

	list_for_each_entry_safe(atsru, atsr_n, &dmar_atsr_units, list) {
		list_del(&atsru->list);
		intel_iommu_free_atsr(atsru);
	}
	list_for_each_entry_safe(satcu, satc_n, &dmar_satc_units, list) {
		list_del(&satcu->list);
		dmar_free_dev_scope(&satcu->devices, &satcu->devices_cnt);
		kfree(satcu);
	}
}

static struct dmar_satc_unit *dmar_find_matched_satc_unit(struct pci_dev *dev)
{
	struct dmar_satc_unit *satcu;
	struct acpi_dmar_satc *satc;
	struct device *tmp;
	int i;

	rcu_read_lock();

	list_for_each_entry_rcu(satcu, &dmar_satc_units, list) {
		satc = container_of(satcu->hdr, struct acpi_dmar_satc, header);
		if (satc->segment != pci_domain_nr(dev->bus))
			continue;
		for_each_dev_scope(satcu->devices, satcu->devices_cnt, i, tmp)
			if (to_pci_dev(tmp) == dev)
				goto out;
	}
	satcu = NULL;
out:
	rcu_read_unlock();
	return satcu;
}

static bool dmar_ats_supported(struct pci_dev *dev, struct intel_iommu *iommu)
{
	struct pci_dev *bridge = NULL;
	struct dmar_atsr_unit *atsru;
	struct dmar_satc_unit *satcu;
	struct acpi_dmar_atsr *atsr;
	bool supported = true;
	struct pci_bus *bus;
	struct device *tmp;
	int i;

	dev = pci_physfn(dev);
	satcu = dmar_find_matched_satc_unit(dev);
	if (satcu)
		/*
		 * This device supports ATS as it is in SATC table.
		 * When IOMMU is in legacy mode, enabling ATS is done
		 * automatically by HW for the device that requires
		 * ATS, hence OS should not enable this device ATS
		 * to avoid duplicated TLB invalidation.
		 */
		return !(satcu->atc_required && !sm_supported(iommu));

	for (bus = dev->bus; bus; bus = bus->parent) {
		bridge = bus->self;
		/* If it's an integrated device, allow ATS */
		if (!bridge)
			return true;
		/* Connected via non-PCIe: no ATS */
		if (!pci_is_pcie(bridge) ||
		    pci_pcie_type(bridge) == PCI_EXP_TYPE_PCI_BRIDGE)
			return false;
		/* If we found the root port, look it up in the ATSR */
		if (pci_pcie_type(bridge) == PCI_EXP_TYPE_ROOT_PORT)
			break;
	}

	rcu_read_lock();
	list_for_each_entry_rcu(atsru, &dmar_atsr_units, list) {
		atsr = container_of(atsru->hdr, struct acpi_dmar_atsr, header);
		if (atsr->segment != pci_domain_nr(dev->bus))
			continue;

		for_each_dev_scope(atsru->devices, atsru->devices_cnt, i, tmp)
			if (tmp == &bridge->dev)
				goto out;

		if (atsru->include_all)
			goto out;
	}
	supported = false;
out:
	rcu_read_unlock();

	return supported;
}

int dmar_iommu_notify_scope_dev(struct dmar_pci_notify_info *info)
{
	int ret;
	struct dmar_rmrr_unit *rmrru;
	struct dmar_atsr_unit *atsru;
	struct dmar_satc_unit *satcu;
	struct acpi_dmar_atsr *atsr;
	struct acpi_dmar_reserved_memory *rmrr;
	struct acpi_dmar_satc *satc;

	if (!intel_iommu_enabled && system_state >= SYSTEM_RUNNING)
		return 0;

	list_for_each_entry(rmrru, &dmar_rmrr_units, list) {
		rmrr = container_of(rmrru->hdr,
				    struct acpi_dmar_reserved_memory, header);
		if (info->event == BUS_NOTIFY_ADD_DEVICE) {
			ret = dmar_insert_dev_scope(info, (void *)(rmrr + 1),
				((void *)rmrr) + rmrr->header.length,
				rmrr->segment, rmrru->devices,
				rmrru->devices_cnt);
			if (ret < 0)
				return ret;
		} else if (info->event == BUS_NOTIFY_REMOVED_DEVICE) {
			dmar_remove_dev_scope(info, rmrr->segment,
				rmrru->devices, rmrru->devices_cnt);
		}
	}

	list_for_each_entry(atsru, &dmar_atsr_units, list) {
		if (atsru->include_all)
			continue;

		atsr = container_of(atsru->hdr, struct acpi_dmar_atsr, header);
		if (info->event == BUS_NOTIFY_ADD_DEVICE) {
			ret = dmar_insert_dev_scope(info, (void *)(atsr + 1),
					(void *)atsr + atsr->header.length,
					atsr->segment, atsru->devices,
					atsru->devices_cnt);
			if (ret > 0)
				break;
			else if (ret < 0)
				return ret;
		} else if (info->event == BUS_NOTIFY_REMOVED_DEVICE) {
			if (dmar_remove_dev_scope(info, atsr->segment,
					atsru->devices, atsru->devices_cnt))
				break;
		}
	}
	list_for_each_entry(satcu, &dmar_satc_units, list) {
		satc = container_of(satcu->hdr, struct acpi_dmar_satc, header);
		if (info->event == BUS_NOTIFY_ADD_DEVICE) {
			ret = dmar_insert_dev_scope(info, (void *)(satc + 1),
					(void *)satc + satc->header.length,
					satc->segment, satcu->devices,
					satcu->devices_cnt);
			if (ret > 0)
				break;
			else if (ret < 0)
				return ret;
		} else if (info->event == BUS_NOTIFY_REMOVED_DEVICE) {
			if (dmar_remove_dev_scope(info, satc->segment,
					satcu->devices, satcu->devices_cnt))
				break;
		}
	}

	return 0;
}

static void intel_disable_iommus(void)
{
	struct intel_iommu *iommu = NULL;
	struct dmar_drhd_unit *drhd;

	for_each_iommu(iommu, drhd)
		iommu_disable_translation(iommu);
}

void intel_iommu_shutdown(void)
{
	struct dmar_drhd_unit *drhd;
	struct intel_iommu *iommu = NULL;

	if (no_iommu || dmar_disabled)
		return;

	/*
	 * All other CPUs were brought down, hotplug interrupts were disabled,
	 * no lock and RCU checking needed anymore
	 */
	list_for_each_entry(drhd, &dmar_drhd_units, list) {
		iommu = drhd->iommu;

		/* Disable PMRs explicitly here. */
		iommu_disable_protect_mem_regions(iommu);

		/* Make sure the IOMMUs are switched off */
		iommu_disable_translation(iommu);
	}
}

static struct intel_iommu *dev_to_intel_iommu(struct device *dev)
{
	struct iommu_device *iommu_dev = dev_to_iommu_device(dev);

	return container_of(iommu_dev, struct intel_iommu, iommu);
}

static ssize_t version_show(struct device *dev,
			    struct device_attribute *attr, char *buf)
{
	struct intel_iommu *iommu = dev_to_intel_iommu(dev);
	u32 ver = readl(iommu->reg + DMAR_VER_REG);
	return sysfs_emit(buf, "%d:%d\n",
			  DMAR_VER_MAJOR(ver), DMAR_VER_MINOR(ver));
}
static DEVICE_ATTR_RO(version);

static ssize_t address_show(struct device *dev,
			    struct device_attribute *attr, char *buf)
{
	struct intel_iommu *iommu = dev_to_intel_iommu(dev);
	return sysfs_emit(buf, "%llx\n", iommu->reg_phys);
}
static DEVICE_ATTR_RO(address);

static ssize_t cap_show(struct device *dev,
			struct device_attribute *attr, char *buf)
{
	struct intel_iommu *iommu = dev_to_intel_iommu(dev);
	return sysfs_emit(buf, "%llx\n", iommu->cap);
}
static DEVICE_ATTR_RO(cap);

static ssize_t ecap_show(struct device *dev,
			 struct device_attribute *attr, char *buf)
{
	struct intel_iommu *iommu = dev_to_intel_iommu(dev);
	return sysfs_emit(buf, "%llx\n", iommu->ecap);
}
static DEVICE_ATTR_RO(ecap);

static ssize_t domains_supported_show(struct device *dev,
				      struct device_attribute *attr, char *buf)
{
	struct intel_iommu *iommu = dev_to_intel_iommu(dev);
	return sysfs_emit(buf, "%ld\n", cap_ndoms(iommu->cap));
}
static DEVICE_ATTR_RO(domains_supported);

static ssize_t domains_used_show(struct device *dev,
				 struct device_attribute *attr, char *buf)
{
	struct intel_iommu *iommu = dev_to_intel_iommu(dev);
	unsigned int count = 0;
	int id;

	for (id = 0; id < cap_ndoms(iommu->cap); id++)
		if (ida_exists(&iommu->domain_ida, id))
			count++;

	return sysfs_emit(buf, "%d\n", count);
}
static DEVICE_ATTR_RO(domains_used);

static struct attribute *intel_iommu_attrs[] = {
	&dev_attr_version.attr,
	&dev_attr_address.attr,
	&dev_attr_cap.attr,
	&dev_attr_ecap.attr,
	&dev_attr_domains_supported.attr,
	&dev_attr_domains_used.attr,
	NULL,
};

static struct attribute_group intel_iommu_group = {
	.name = "intel-iommu",
	.attrs = intel_iommu_attrs,
};

const struct attribute_group *intel_iommu_groups[] = {
	&intel_iommu_group,
	NULL,
};

static bool has_external_pci(void)
{
	struct pci_dev *pdev = NULL;

	for_each_pci_dev(pdev)
		if (pdev->external_facing) {
			pci_dev_put(pdev);
			return true;
		}

	return false;
}

static int __init platform_optin_force_iommu(void)
{
	if (!dmar_platform_optin() || no_platform_optin || !has_external_pci())
		return 0;

	if (no_iommu || dmar_disabled)
		pr_info("Intel-IOMMU force enabled due to platform opt in\n");

	/*
	 * If Intel-IOMMU is disabled by default, we will apply identity
	 * map for all devices except those marked as being untrusted.
	 */
	if (dmar_disabled)
		iommu_set_default_passthrough(false);

	dmar_disabled = 0;
	no_iommu = 0;

	return 1;
}

static int __init probe_acpi_namespace_devices(void)
{
	struct dmar_drhd_unit *drhd;
	/* To avoid a -Wunused-but-set-variable warning. */
	struct intel_iommu *iommu __maybe_unused;
	struct device *dev;
	int i, ret = 0;

	for_each_active_iommu(iommu, drhd) {
		for_each_active_dev_scope(drhd->devices,
					  drhd->devices_cnt, i, dev) {
			struct acpi_device_physical_node *pn;
			struct acpi_device *adev;

			if (dev->bus != &acpi_bus_type)
				continue;

			up_read(&dmar_global_lock);
			adev = to_acpi_device(dev);
			mutex_lock(&adev->physical_node_lock);
			list_for_each_entry(pn,
					    &adev->physical_node_list, node) {
				ret = iommu_probe_device(pn->dev);
				if (ret)
					break;
			}
			mutex_unlock(&adev->physical_node_lock);
			down_read(&dmar_global_lock);

			if (ret)
				return ret;
		}
	}

	return 0;
}

static __init int tboot_force_iommu(void)
{
	if (!tboot_enabled())
		return 0;

	if (no_iommu || dmar_disabled)
		pr_warn("Forcing Intel-IOMMU to enabled\n");

	dmar_disabled = 0;
	no_iommu = 0;

	return 1;
}

int __init intel_iommu_init(void)
{
	int ret = -ENODEV;
	struct dmar_drhd_unit *drhd;
	struct intel_iommu *iommu;

	/*
	 * Intel IOMMU is required for a TXT/tboot launch or platform
	 * opt in, so enforce that.
	 */
	force_on = (!intel_iommu_tboot_noforce && tboot_force_iommu()) ||
		    platform_optin_force_iommu();

	down_write(&dmar_global_lock);
	if (dmar_table_init()) {
		if (force_on)
			panic("tboot: Failed to initialize DMAR table\n");
		goto out_free_dmar;
	}

	if (dmar_dev_scope_init() < 0) {
		if (force_on)
			panic("tboot: Failed to initialize DMAR device scope\n");
		goto out_free_dmar;
	}

	up_write(&dmar_global_lock);

	/*
	 * The bus notifier takes the dmar_global_lock, so lockdep will
	 * complain later when we register it under the lock.
	 */
	dmar_register_bus_notifier();

	down_write(&dmar_global_lock);

	if (!no_iommu)
		intel_iommu_debugfs_init();

	if (no_iommu || dmar_disabled) {
		/*
		 * We exit the function here to ensure IOMMU's remapping and
		 * mempool aren't setup, which means that the IOMMU's PMRs
		 * won't be disabled via the call to init_dmars(). So disable
		 * it explicitly here. The PMRs were setup by tboot prior to
		 * calling SENTER, but the kernel is expected to reset/tear
		 * down the PMRs.
		 */
		if (intel_iommu_tboot_noforce) {
			for_each_iommu(iommu, drhd)
				iommu_disable_protect_mem_regions(iommu);
		}

		/*
		 * Make sure the IOMMUs are switched off, even when we
		 * boot into a kexec kernel and the previous kernel left
		 * them enabled
		 */
		intel_disable_iommus();
		goto out_free_dmar;
	}

	if (list_empty(&dmar_rmrr_units))
		pr_info("No RMRR found\n");

	if (list_empty(&dmar_atsr_units))
		pr_info("No ATSR found\n");

	if (list_empty(&dmar_satc_units))
		pr_info("No SATC found\n");

	init_no_remapping_devices();

	ret = init_dmars();
	if (ret) {
		if (force_on)
			panic("tboot: Failed to initialize DMARs\n");
		pr_err("Initialization failed\n");
		goto out_free_dmar;
	}
	up_write(&dmar_global_lock);

	init_iommu_pm_ops();

	down_read(&dmar_global_lock);
	for_each_active_iommu(iommu, drhd) {
		/*
		 * The flush queue implementation does not perform
		 * page-selective invalidations that are required for efficient
		 * TLB flushes in virtual environments.  The benefit of batching
		 * is likely to be much lower than the overhead of synchronizing
		 * the virtual and physical IOMMU page-tables.
		 */
		if (cap_caching_mode(iommu->cap) &&
		    !first_level_by_default(iommu)) {
			pr_info_once("IOMMU batching disallowed due to virtualization\n");
			iommu_set_dma_strict();
		}
		iommu_device_sysfs_add(&iommu->iommu, NULL,
				       intel_iommu_groups,
				       "%s", iommu->name);
		/*
		 * The iommu device probe is protected by the iommu_probe_device_lock.
		 * Release the dmar_global_lock before entering the device probe path
		 * to avoid unnecessary lock order splat.
		 */
		up_read(&dmar_global_lock);
		iommu_device_register(&iommu->iommu, &intel_iommu_ops, NULL);
		down_read(&dmar_global_lock);

		iommu_pmu_register(iommu);
	}

	if (probe_acpi_namespace_devices())
		pr_warn("ACPI name space devices didn't probe correctly\n");

	/* Finally, we enable the DMA remapping hardware. */
	for_each_iommu(iommu, drhd) {
		if (!drhd->ignored && !translation_pre_enabled(iommu))
			iommu_enable_translation(iommu);

		iommu_disable_protect_mem_regions(iommu);
	}
	up_read(&dmar_global_lock);

	pr_info("Intel(R) Virtualization Technology for Directed I/O\n");

	intel_iommu_enabled = 1;

	return 0;

out_free_dmar:
	intel_iommu_free_dmars();
	up_write(&dmar_global_lock);
	return ret;
}

static int domain_context_clear_one_cb(struct pci_dev *pdev, u16 alias, void *opaque)
{
	struct device_domain_info *info = opaque;

	domain_context_clear_one(info, PCI_BUS_NUM(alias), alias & 0xff);
	return 0;
}

/*
 * NB - intel-iommu lacks any sort of reference counting for the users of
 * dependent devices.  If multiple endpoints have intersecting dependent
 * devices, unbinding the driver from any one of them will possibly leave
 * the others unable to operate.
 */
static void domain_context_clear(struct device_domain_info *info)
{
	if (!dev_is_pci(info->dev)) {
		domain_context_clear_one(info, info->bus, info->devfn);
		return;
	}

	pci_for_each_dma_alias(to_pci_dev(info->dev),
			       &domain_context_clear_one_cb, info);
	iommu_disable_pci_ats(info);
}

/*
 * Clear the page table pointer in context or pasid table entries so that
 * all DMA requests without PASID from the device are blocked. If the page
 * table has been set, clean up the data structures.
 */
void device_block_translation(struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;
	unsigned long flags;

	/* Device in DMA blocking state. Noting to do. */
	if (!info->domain_attached)
		return;

	if (info->domain)
		cache_tag_unassign_domain(info->domain, dev, IOMMU_NO_PASID);

	if (!dev_is_real_dma_subdevice(dev)) {
		if (sm_supported(iommu))
			intel_pasid_tear_down_entry(iommu, dev,
						    IOMMU_NO_PASID, false);
		else
			domain_context_clear(info);
	}

	/* Device now in DMA blocking state. */
	info->domain_attached = false;

	if (!info->domain)
		return;

	spin_lock_irqsave(&info->domain->lock, flags);
	list_del(&info->link);
	spin_unlock_irqrestore(&info->domain->lock, flags);

	domain_detach_iommu(info->domain, iommu);
	info->domain = NULL;
}

static int blocking_domain_attach_dev(struct iommu_domain *domain,
				      struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);

	iopf_for_domain_remove(info->domain ? &info->domain->domain : NULL, dev);
	device_block_translation(dev);
	return 0;
}

static int blocking_domain_set_dev_pasid(struct iommu_domain *domain,
					 struct device *dev, ioasid_t pasid,
					 struct iommu_domain *old);

static struct iommu_domain blocking_domain = {
	.type = IOMMU_DOMAIN_BLOCKED,
	.ops = &(const struct iommu_domain_ops) {
		.attach_dev	= blocking_domain_attach_dev,
		.set_dev_pasid	= blocking_domain_set_dev_pasid,
	}
};

static int iommu_superpage_capability(struct intel_iommu *iommu, bool first_stage)
{
	if (!intel_iommu_superpage)
		return 0;

	if (first_stage)
		return cap_fl1gp_support(iommu->cap) ? 2 : 1;

	return fls(cap_super_page_val(iommu->cap));
}

static struct dmar_domain *paging_domain_alloc(struct device *dev, bool first_stage)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;
	struct dmar_domain *domain;
	int addr_width;

	domain = kzalloc(sizeof(*domain), GFP_KERNEL);
	if (!domain)
		return ERR_PTR(-ENOMEM);

	INIT_LIST_HEAD(&domain->devices);
	INIT_LIST_HEAD(&domain->dev_pasids);
	INIT_LIST_HEAD(&domain->cache_tags);
	spin_lock_init(&domain->lock);
	spin_lock_init(&domain->cache_lock);
	xa_init(&domain->iommu_array);
	INIT_LIST_HEAD(&domain->s1_domains);
	spin_lock_init(&domain->s1_lock);

	domain->nid = dev_to_node(dev);
	domain->use_first_level = first_stage;

	domain->domain.type = IOMMU_DOMAIN_UNMANAGED;

	/* calculate the address width */
	addr_width = agaw_to_width(iommu->agaw);
	if (addr_width > cap_mgaw(iommu->cap))
		addr_width = cap_mgaw(iommu->cap);
	domain->gaw = addr_width;
	domain->agaw = iommu->agaw;
	domain->max_addr = __DOMAIN_MAX_ADDR(addr_width);

	/* iommu memory access coherency */
	domain->iommu_coherency = iommu_paging_structure_coherency(iommu);

	/* pagesize bitmap */
	domain->domain.pgsize_bitmap = SZ_4K;
	domain->iommu_superpage = iommu_superpage_capability(iommu, first_stage);
	domain->domain.pgsize_bitmap |= domain_super_pgsize_bitmap(domain);

	/*
	 * IOVA aperture: First-level translation restricts the input-address
	 * to a canonical address (i.e., address bits 63:N have the same value
	 * as address bit [N-1], where N is 48-bits with 4-level paging and
	 * 57-bits with 5-level paging). Hence, skip bit [N-1].
	 */
	domain->domain.geometry.force_aperture = true;
	domain->domain.geometry.aperture_start = 0;
	if (first_stage)
		domain->domain.geometry.aperture_end = __DOMAIN_MAX_ADDR(domain->gaw - 1);
	else
		domain->domain.geometry.aperture_end = __DOMAIN_MAX_ADDR(domain->gaw);

	/* always allocate the top pgd */
	domain->pgd = iommu_alloc_pages_node_sz(domain->nid, GFP_KERNEL, SZ_4K);
	if (!domain->pgd) {
		kfree(domain);
		return ERR_PTR(-ENOMEM);
	}
	domain_flush_cache(domain, domain->pgd, PAGE_SIZE);

	return domain;
}

static struct iommu_domain *
intel_iommu_domain_alloc_first_stage(struct device *dev,
				     struct intel_iommu *iommu, u32 flags)
{
	struct dmar_domain *dmar_domain;

	if (flags & ~IOMMU_HWPT_ALLOC_PASID)
		return ERR_PTR(-EOPNOTSUPP);

	/* Only SL is available in legacy mode */
	if (!sm_supported(iommu) || !ecap_flts(iommu->ecap))
		return ERR_PTR(-EOPNOTSUPP);

	dmar_domain = paging_domain_alloc(dev, true);
	if (IS_ERR(dmar_domain))
		return ERR_CAST(dmar_domain);

	dmar_domain->domain.ops = &intel_fs_paging_domain_ops;
	/*
	 * iotlb sync for map is only needed for legacy implementations that
	 * explicitly require flushing internal write buffers to ensure memory
	 * coherence.
	 */
	if (rwbf_required(iommu))
		dmar_domain->iotlb_sync_map = true;

	return &dmar_domain->domain;
}

static struct iommu_domain *
intel_iommu_domain_alloc_second_stage(struct device *dev,
				      struct intel_iommu *iommu, u32 flags)
{
	struct dmar_domain *dmar_domain;

	if (flags &
	    (~(IOMMU_HWPT_ALLOC_NEST_PARENT | IOMMU_HWPT_ALLOC_DIRTY_TRACKING |
	       IOMMU_HWPT_ALLOC_PASID)))
		return ERR_PTR(-EOPNOTSUPP);

	if (((flags & IOMMU_HWPT_ALLOC_NEST_PARENT) &&
	     !nested_supported(iommu)) ||
	    ((flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING) &&
	     !ssads_supported(iommu)))
		return ERR_PTR(-EOPNOTSUPP);

	/* Legacy mode always supports second stage */
	if (sm_supported(iommu) && !ecap_slts(iommu->ecap))
		return ERR_PTR(-EOPNOTSUPP);

	dmar_domain = paging_domain_alloc(dev, false);
	if (IS_ERR(dmar_domain))
		return ERR_CAST(dmar_domain);

	dmar_domain->domain.ops = &intel_ss_paging_domain_ops;
	dmar_domain->nested_parent = flags & IOMMU_HWPT_ALLOC_NEST_PARENT;

	if (flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING)
		dmar_domain->domain.dirty_ops = &intel_dirty_ops;

	/*
	 * Besides the internal write buffer flush, the caching mode used for
	 * legacy nested translation (which utilizes shadowing page tables)
	 * also requires iotlb sync on map.
	 */
	if (rwbf_required(iommu) || cap_caching_mode(iommu->cap))
		dmar_domain->iotlb_sync_map = true;

	return &dmar_domain->domain;
}

static struct iommu_domain *
intel_iommu_domain_alloc_paging_flags(struct device *dev, u32 flags,
				      const struct iommu_user_data *user_data)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;
	struct iommu_domain *domain;

	if (user_data)
		return ERR_PTR(-EOPNOTSUPP);

	/* Prefer first stage if possible by default. */
	domain = intel_iommu_domain_alloc_first_stage(dev, iommu, flags);
	if (domain != ERR_PTR(-EOPNOTSUPP))
		return domain;
	return intel_iommu_domain_alloc_second_stage(dev, iommu, flags);
}

static void intel_iommu_domain_free(struct iommu_domain *domain)
{
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);

	if (WARN_ON(dmar_domain->nested_parent &&
		    !list_empty(&dmar_domain->s1_domains)))
		return;

	if (WARN_ON(!list_empty(&dmar_domain->devices)))
		return;

	if (dmar_domain->pgd) {
		struct iommu_pages_list freelist =
			IOMMU_PAGES_LIST_INIT(freelist);

		domain_unmap(dmar_domain, 0, DOMAIN_MAX_PFN(dmar_domain->gaw),
			     &freelist);
		iommu_put_pages_list(&freelist);
	}

	kfree(dmar_domain->qi_batch);
	kfree(dmar_domain);
}

static int paging_domain_compatible_first_stage(struct dmar_domain *dmar_domain,
						struct intel_iommu *iommu)
{
	if (WARN_ON(dmar_domain->domain.dirty_ops ||
		    dmar_domain->nested_parent))
		return -EINVAL;

	/* Only SL is available in legacy mode */
	if (!sm_supported(iommu) || !ecap_flts(iommu->ecap))
		return -EINVAL;

	/* Same page size support */
	if (!cap_fl1gp_support(iommu->cap) &&
	    (dmar_domain->domain.pgsize_bitmap & SZ_1G))
		return -EINVAL;

	/* iotlb sync on map requirement */
	if ((rwbf_required(iommu)) && !dmar_domain->iotlb_sync_map)
		return -EINVAL;

	return 0;
}

static int
paging_domain_compatible_second_stage(struct dmar_domain *dmar_domain,
				      struct intel_iommu *iommu)
{
	unsigned int sslps = cap_super_page_val(iommu->cap);

	if (dmar_domain->domain.dirty_ops && !ssads_supported(iommu))
		return -EINVAL;
	if (dmar_domain->nested_parent && !nested_supported(iommu))
		return -EINVAL;

	/* Legacy mode always supports second stage */
	if (sm_supported(iommu) && !ecap_slts(iommu->ecap))
		return -EINVAL;

	/* Same page size support */
	if (!(sslps & BIT(0)) && (dmar_domain->domain.pgsize_bitmap & SZ_2M))
		return -EINVAL;
	if (!(sslps & BIT(1)) && (dmar_domain->domain.pgsize_bitmap & SZ_1G))
		return -EINVAL;

	/* iotlb sync on map requirement */
	if ((rwbf_required(iommu) || cap_caching_mode(iommu->cap)) &&
	    !dmar_domain->iotlb_sync_map)
		return -EINVAL;

	return 0;
}

int paging_domain_compatible(struct iommu_domain *domain, struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
	struct intel_iommu *iommu = info->iommu;
	int ret = -EINVAL;
	int addr_width;

	if (intel_domain_is_fs_paging(dmar_domain))
		ret = paging_domain_compatible_first_stage(dmar_domain, iommu);
	else if (intel_domain_is_ss_paging(dmar_domain))
		ret = paging_domain_compatible_second_stage(dmar_domain, iommu);
	else if (WARN_ON(true))
		ret = -EINVAL;
	if (ret)
		return ret;

	/*
	 * FIXME this is locked wrong, it needs to be under the
	 * dmar_domain->lock
	 */
	if (dmar_domain->force_snooping && !ecap_sc_support(iommu->ecap))
		return -EINVAL;

	if (dmar_domain->iommu_coherency !=
			iommu_paging_structure_coherency(iommu))
		return -EINVAL;


	/* check if this iommu agaw is sufficient for max mapped address */
	addr_width = agaw_to_width(iommu->agaw);
	if (addr_width > cap_mgaw(iommu->cap))
		addr_width = cap_mgaw(iommu->cap);

	if (dmar_domain->gaw > addr_width || dmar_domain->agaw > iommu->agaw)
		return -EINVAL;

	if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev) &&
	    context_copied(iommu, info->bus, info->devfn))
		return intel_pasid_setup_sm_context(dev);

	return 0;
}

static int intel_iommu_attach_device(struct iommu_domain *domain,
				     struct device *dev)
{
	int ret;

	device_block_translation(dev);

	ret = paging_domain_compatible(domain, dev);
	if (ret)
		return ret;

	ret = iopf_for_domain_set(domain, dev);
	if (ret)
		return ret;

	ret = dmar_domain_attach_device(to_dmar_domain(domain), dev);
	if (ret)
		iopf_for_domain_remove(domain, dev);

	return ret;
}

static int intel_iommu_map(struct iommu_domain *domain,
			   unsigned long iova, phys_addr_t hpa,
			   size_t size, int iommu_prot, gfp_t gfp)
{
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
	u64 max_addr;
	int prot = 0;

	if (iommu_prot & IOMMU_READ)
		prot |= DMA_PTE_READ;
	if (iommu_prot & IOMMU_WRITE)
		prot |= DMA_PTE_WRITE;
	if (dmar_domain->set_pte_snp)
		prot |= DMA_PTE_SNP;

	max_addr = iova + size;
	if (dmar_domain->max_addr < max_addr) {
		u64 end;

		/* check if minimum agaw is sufficient for mapped address */
		end = __DOMAIN_MAX_ADDR(dmar_domain->gaw) + 1;
		if (end < max_addr) {
			pr_err("%s: iommu width (%d) is not "
			       "sufficient for the mapped address (%llx)\n",
			       __func__, dmar_domain->gaw, max_addr);
			return -EFAULT;
		}
		dmar_domain->max_addr = max_addr;
	}
	/* Round up size to next multiple of PAGE_SIZE, if it and
	   the low bits of hpa would take us onto the next page */
	size = aligned_nrpages(hpa, size);
	return __domain_mapping(dmar_domain, iova >> VTD_PAGE_SHIFT,
				hpa >> VTD_PAGE_SHIFT, size, prot, gfp);
}

static int intel_iommu_map_pages(struct iommu_domain *domain,
				 unsigned long iova, phys_addr_t paddr,
				 size_t pgsize, size_t pgcount,
				 int prot, gfp_t gfp, size_t *mapped)
{
	unsigned long pgshift = __ffs(pgsize);
	size_t size = pgcount << pgshift;
	int ret;

	if (pgsize != SZ_4K && pgsize != SZ_2M && pgsize != SZ_1G)
		return -EINVAL;

	if (!IS_ALIGNED(iova | paddr, pgsize))
		return -EINVAL;

	ret = intel_iommu_map(domain, iova, paddr, size, prot, gfp);
	if (!ret && mapped)
		*mapped = size;

	return ret;
}

static size_t intel_iommu_unmap(struct iommu_domain *domain,
				unsigned long iova, size_t size,
				struct iommu_iotlb_gather *gather)
{
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
	unsigned long start_pfn, last_pfn;
	int level = 0;

	/* Cope with horrid API which requires us to unmap more than the
	   size argument if it happens to be a large-page mapping. */
	if (unlikely(!pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT,
				     &level, GFP_ATOMIC)))
		return 0;

	if (size < VTD_PAGE_SIZE << level_to_offset_bits(level))
		size = VTD_PAGE_SIZE << level_to_offset_bits(level);

	start_pfn = iova >> VTD_PAGE_SHIFT;
	last_pfn = (iova + size - 1) >> VTD_PAGE_SHIFT;

	domain_unmap(dmar_domain, start_pfn, last_pfn, &gather->freelist);

	if (dmar_domain->max_addr == iova + size)
		dmar_domain->max_addr = iova;

	/*
	 * We do not use page-selective IOTLB invalidation in flush queue,
	 * so there is no need to track page and sync iotlb.
	 */
	if (!iommu_iotlb_gather_queued(gather))
		iommu_iotlb_gather_add_page(domain, gather, iova, size);

	return size;
}

static size_t intel_iommu_unmap_pages(struct iommu_domain *domain,
				      unsigned long iova,
				      size_t pgsize, size_t pgcount,
				      struct iommu_iotlb_gather *gather)
{
	unsigned long pgshift = __ffs(pgsize);
	size_t size = pgcount << pgshift;

	return intel_iommu_unmap(domain, iova, size, gather);
}

static void intel_iommu_tlb_sync(struct iommu_domain *domain,
				 struct iommu_iotlb_gather *gather)
{
	cache_tag_flush_range(to_dmar_domain(domain), gather->start,
			      gather->end,
			      iommu_pages_list_empty(&gather->freelist));
	iommu_put_pages_list(&gather->freelist);
}

static phys_addr_t intel_iommu_iova_to_phys(struct iommu_domain *domain,
					    dma_addr_t iova)
{
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
	struct dma_pte *pte;
	int level = 0;
	u64 phys = 0;

	pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level,
			     GFP_ATOMIC);
	if (pte && dma_pte_present(pte))
		phys = dma_pte_addr(pte) +
			(iova & (BIT_MASK(level_to_offset_bits(level) +
						VTD_PAGE_SHIFT) - 1));

	return phys;
}

static bool domain_support_force_snooping(struct dmar_domain *domain)
{
	struct device_domain_info *info;
	bool support = true;

	assert_spin_locked(&domain->lock);
	list_for_each_entry(info, &domain->devices, link) {
		if (!ecap_sc_support(info->iommu->ecap)) {
			support = false;
			break;
		}
	}

	return support;
}

static bool intel_iommu_enforce_cache_coherency_fs(struct iommu_domain *domain)
{
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
	struct device_domain_info *info;

	guard(spinlock_irqsave)(&dmar_domain->lock);

	if (dmar_domain->force_snooping)
		return true;

	if (!domain_support_force_snooping(dmar_domain))
		return false;

	dmar_domain->force_snooping = true;
	list_for_each_entry(info, &dmar_domain->devices, link)
		intel_pasid_setup_page_snoop_control(info->iommu, info->dev,
						     IOMMU_NO_PASID);
	return true;
}

static bool intel_iommu_enforce_cache_coherency_ss(struct iommu_domain *domain)
{
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);

	guard(spinlock_irqsave)(&dmar_domain->lock);
	if (!domain_support_force_snooping(dmar_domain) ||
	    dmar_domain->has_mappings)
		return false;

	/*
	 * Second level page table supports per-PTE snoop control. The
	 * iommu_map() interface will handle this by setting SNP bit.
	 */
	dmar_domain->set_pte_snp = true;
	dmar_domain->force_snooping = true;
	return true;
}

static bool intel_iommu_capable(struct device *dev, enum iommu_cap cap)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);

	switch (cap) {
	case IOMMU_CAP_CACHE_COHERENCY:
	case IOMMU_CAP_DEFERRED_FLUSH:
		return true;
	case IOMMU_CAP_PRE_BOOT_PROTECTION:
		return dmar_platform_optin();
	case IOMMU_CAP_ENFORCE_CACHE_COHERENCY:
		return ecap_sc_support(info->iommu->ecap);
	case IOMMU_CAP_DIRTY_TRACKING:
		return ssads_supported(info->iommu);
	default:
		return false;
	}
}

static struct iommu_device *intel_iommu_probe_device(struct device *dev)
{
	struct pci_dev *pdev = dev_is_pci(dev) ? to_pci_dev(dev) : NULL;
	struct device_domain_info *info;
	struct intel_iommu *iommu;
	u8 bus, devfn;
	int ret;

	iommu = device_lookup_iommu(dev, &bus, &devfn);
	if (!iommu || !iommu->iommu.ops)
		return ERR_PTR(-ENODEV);

	info = kzalloc(sizeof(*info), GFP_KERNEL);
	if (!info)
		return ERR_PTR(-ENOMEM);

	if (dev_is_real_dma_subdevice(dev)) {
		info->bus = pdev->bus->number;
		info->devfn = pdev->devfn;
		info->segment = pci_domain_nr(pdev->bus);
	} else {
		info->bus = bus;
		info->devfn = devfn;
		info->segment = iommu->segment;
	}

	info->dev = dev;
	info->iommu = iommu;
	if (dev_is_pci(dev)) {
		if (ecap_dev_iotlb_support(iommu->ecap) &&
		    pci_ats_supported(pdev) &&
		    dmar_ats_supported(pdev, iommu)) {
			info->ats_supported = 1;
			info->dtlb_extra_inval = dev_needs_extra_dtlb_flush(pdev);

			/*
			 * For IOMMU that supports device IOTLB throttling
			 * (DIT), we assign PFSID to the invalidation desc
			 * of a VF such that IOMMU HW can gauge queue depth
			 * at PF level. If DIT is not set, PFSID will be
			 * treated as reserved, which should be set to 0.
			 */
			if (ecap_dit(iommu->ecap))
				info->pfsid = pci_dev_id(pci_physfn(pdev));
			info->ats_qdep = pci_ats_queue_depth(pdev);
		}
		if (sm_supported(iommu)) {
			if (pasid_supported(iommu)) {
				int features = pci_pasid_features(pdev);

				if (features >= 0)
					info->pasid_supported = features | 1;
			}

			if (info->ats_supported && ecap_prs(iommu->ecap) &&
			    pci_pri_supported(pdev))
				info->pri_supported = 1;
		}
	}

	dev_iommu_priv_set(dev, info);
	if (pdev && pci_ats_supported(pdev)) {
		pci_prepare_ats(pdev, VTD_PAGE_SHIFT);
		ret = device_rbtree_insert(iommu, info);
		if (ret)
			goto free;
	}

	if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) {
		ret = intel_pasid_alloc_table(dev);
		if (ret) {
			dev_err(dev, "PASID table allocation failed\n");
			goto clear_rbtree;
		}

		if (!context_copied(iommu, info->bus, info->devfn)) {
			ret = intel_pasid_setup_sm_context(dev);
			if (ret)
				goto free_table;
		}
	}

	intel_iommu_debugfs_create_dev(info);

	return &iommu->iommu;
free_table:
	intel_pasid_free_table(dev);
clear_rbtree:
	device_rbtree_remove(info);
free:
	kfree(info);

	return ERR_PTR(ret);
}

static void intel_iommu_probe_finalize(struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;

	/*
	 * The PCIe spec, in its wisdom, declares that the behaviour of the
	 * device is undefined if you enable PASID support after ATS support.
	 * So always enable PASID support on devices which have it, even if
	 * we can't yet know if we're ever going to use it.
	 */
	if (info->pasid_supported &&
	    !pci_enable_pasid(to_pci_dev(dev), info->pasid_supported & ~1))
		info->pasid_enabled = 1;

	if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) {
		iommu_enable_pci_ats(info);
		/* Assign a DEVTLB cache tag to the default domain. */
		if (info->ats_enabled && info->domain) {
			u16 did = domain_id_iommu(info->domain, iommu);

			if (cache_tag_assign(info->domain, did, dev,
					     IOMMU_NO_PASID, CACHE_TAG_DEVTLB))
				iommu_disable_pci_ats(info);
		}
	}
	iommu_enable_pci_pri(info);
}

static void intel_iommu_release_device(struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;

	iommu_disable_pci_pri(info);
	iommu_disable_pci_ats(info);

	if (info->pasid_enabled) {
		pci_disable_pasid(to_pci_dev(dev));
		info->pasid_enabled = 0;
	}

	mutex_lock(&iommu->iopf_lock);
	if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
		device_rbtree_remove(info);
	mutex_unlock(&iommu->iopf_lock);

	if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev) &&
	    !context_copied(iommu, info->bus, info->devfn))
		intel_pasid_teardown_sm_context(dev);

	intel_pasid_free_table(dev);
	intel_iommu_debugfs_remove_dev(info);
	kfree(info);
}

static void intel_iommu_get_resv_regions(struct device *device,
					 struct list_head *head)
{
	int prot = DMA_PTE_READ | DMA_PTE_WRITE;
	struct iommu_resv_region *reg;
	struct dmar_rmrr_unit *rmrr;
	struct device *i_dev;
	int i;

	rcu_read_lock();
	for_each_rmrr_units(rmrr) {
		for_each_active_dev_scope(rmrr->devices, rmrr->devices_cnt,
					  i, i_dev) {
			struct iommu_resv_region *resv;
			enum iommu_resv_type type;
			size_t length;

			if (i_dev != device &&
			    !is_downstream_to_pci_bridge(device, i_dev))
				continue;

			length = rmrr->end_address - rmrr->base_address + 1;

			type = device_rmrr_is_relaxable(device) ?
				IOMMU_RESV_DIRECT_RELAXABLE : IOMMU_RESV_DIRECT;

			resv = iommu_alloc_resv_region(rmrr->base_address,
						       length, prot, type,
						       GFP_ATOMIC);
			if (!resv)
				break;

			list_add_tail(&resv->list, head);
		}
	}
	rcu_read_unlock();

#ifdef CONFIG_INTEL_IOMMU_FLOPPY_WA
	if (dev_is_pci(device)) {
		struct pci_dev *pdev = to_pci_dev(device);

		if ((pdev->class >> 8) == PCI_CLASS_BRIDGE_ISA) {
			reg = iommu_alloc_resv_region(0, 1UL << 24, prot,
					IOMMU_RESV_DIRECT_RELAXABLE,
					GFP_KERNEL);
			if (reg)
				list_add_tail(&reg->list, head);
		}
	}
#endif /* CONFIG_INTEL_IOMMU_FLOPPY_WA */

	reg = iommu_alloc_resv_region(IOAPIC_RANGE_START,
				      IOAPIC_RANGE_END - IOAPIC_RANGE_START + 1,
				      0, IOMMU_RESV_MSI, GFP_KERNEL);
	if (!reg)
		return;
	list_add_tail(&reg->list, head);
}

static struct iommu_group *intel_iommu_device_group(struct device *dev)
{
	if (dev_is_pci(dev))
		return pci_device_group(dev);
	return generic_device_group(dev);
}

int intel_iommu_enable_iopf(struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;
	int ret;

	if (!info->pri_enabled)
		return -ENODEV;

	/* pri_enabled is protected by the group mutex. */
	iommu_group_mutex_assert(dev);
	if (info->iopf_refcount) {
		info->iopf_refcount++;
		return 0;
	}

	ret = iopf_queue_add_device(iommu->iopf_queue, dev);
	if (ret)
		return ret;

	info->iopf_refcount = 1;

	return 0;
}

void intel_iommu_disable_iopf(struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;

	if (WARN_ON(!info->pri_enabled || !info->iopf_refcount))
		return;

	iommu_group_mutex_assert(dev);
	if (--info->iopf_refcount)
		return;

	iopf_queue_remove_device(iommu->iopf_queue, dev);
}

static bool intel_iommu_is_attach_deferred(struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);

	return translation_pre_enabled(info->iommu) && !info->domain;
}

/*
 * Check that the device does not live on an external facing PCI port that is
 * marked as untrusted. Such devices should not be able to apply quirks and
 * thus not be able to bypass the IOMMU restrictions.
 */
static bool risky_device(struct pci_dev *pdev)
{
	if (pdev->untrusted) {
		pci_info(pdev,
			 "Skipping IOMMU quirk for dev [%04X:%04X] on untrusted PCI link\n",
			 pdev->vendor, pdev->device);
		pci_info(pdev, "Please check with your BIOS/Platform vendor about this\n");
		return true;
	}
	return false;
}

static int intel_iommu_iotlb_sync_map(struct iommu_domain *domain,
				      unsigned long iova, size_t size)
{
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);

	if (dmar_domain->iotlb_sync_map)
		cache_tag_flush_range_np(dmar_domain, iova, iova + size - 1);

	return 0;
}

void domain_remove_dev_pasid(struct iommu_domain *domain,
			     struct device *dev, ioasid_t pasid)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct dev_pasid_info *curr, *dev_pasid = NULL;
	struct intel_iommu *iommu = info->iommu;
	struct dmar_domain *dmar_domain;
	unsigned long flags;

	if (!domain)
		return;

	/* Identity domain has no meta data for pasid. */
	if (domain->type == IOMMU_DOMAIN_IDENTITY)
		return;

	dmar_domain = to_dmar_domain(domain);
	spin_lock_irqsave(&dmar_domain->lock, flags);
	list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) {
		if (curr->dev == dev && curr->pasid == pasid) {
			list_del(&curr->link_domain);
			dev_pasid = curr;
			break;
		}
	}
	spin_unlock_irqrestore(&dmar_domain->lock, flags);

	cache_tag_unassign_domain(dmar_domain, dev, pasid);
	domain_detach_iommu(dmar_domain, iommu);
	if (!WARN_ON_ONCE(!dev_pasid)) {
		intel_iommu_debugfs_remove_dev_pasid(dev_pasid);
		kfree(dev_pasid);
	}
}

static int blocking_domain_set_dev_pasid(struct iommu_domain *domain,
					 struct device *dev, ioasid_t pasid,
					 struct iommu_domain *old)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);

	intel_pasid_tear_down_entry(info->iommu, dev, pasid, false);
	iopf_for_domain_remove(old, dev);
	domain_remove_dev_pasid(old, dev, pasid);

	return 0;
}

struct dev_pasid_info *
domain_add_dev_pasid(struct iommu_domain *domain,
		     struct device *dev, ioasid_t pasid)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
	struct intel_iommu *iommu = info->iommu;
	struct dev_pasid_info *dev_pasid;
	unsigned long flags;
	int ret;

	dev_pasid = kzalloc(sizeof(*dev_pasid), GFP_KERNEL);
	if (!dev_pasid)
		return ERR_PTR(-ENOMEM);

	ret = domain_attach_iommu(dmar_domain, iommu);
	if (ret)
		goto out_free;

	ret = cache_tag_assign_domain(dmar_domain, dev, pasid);
	if (ret)
		goto out_detach_iommu;

	dev_pasid->dev = dev;
	dev_pasid->pasid = pasid;
	spin_lock_irqsave(&dmar_domain->lock, flags);
	list_add(&dev_pasid->link_domain, &dmar_domain->dev_pasids);
	spin_unlock_irqrestore(&dmar_domain->lock, flags);

	return dev_pasid;
out_detach_iommu:
	domain_detach_iommu(dmar_domain, iommu);
out_free:
	kfree(dev_pasid);
	return ERR_PTR(ret);
}

static int intel_iommu_set_dev_pasid(struct iommu_domain *domain,
				     struct device *dev, ioasid_t pasid,
				     struct iommu_domain *old)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
	struct intel_iommu *iommu = info->iommu;
	struct dev_pasid_info *dev_pasid;
	int ret;

	if (WARN_ON_ONCE(!(domain->type & __IOMMU_DOMAIN_PAGING)))
		return -EINVAL;

	if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev))
		return -EOPNOTSUPP;

	if (domain->dirty_ops)
		return -EINVAL;

	if (context_copied(iommu, info->bus, info->devfn))
		return -EBUSY;

	ret = paging_domain_compatible(domain, dev);
	if (ret)
		return ret;

	dev_pasid = domain_add_dev_pasid(domain, dev, pasid);
	if (IS_ERR(dev_pasid))
		return PTR_ERR(dev_pasid);

	ret = iopf_for_domain_replace(domain, old, dev);
	if (ret)
		goto out_remove_dev_pasid;

	if (intel_domain_is_fs_paging(dmar_domain))
		ret = domain_setup_first_level(iommu, dmar_domain,
					       dev, pasid, old);
	else if (intel_domain_is_ss_paging(dmar_domain))
		ret = domain_setup_second_level(iommu, dmar_domain,
						dev, pasid, old);
	else if (WARN_ON(true))
		ret = -EINVAL;

	if (ret)
		goto out_unwind_iopf;

	domain_remove_dev_pasid(old, dev, pasid);

	intel_iommu_debugfs_create_dev_pasid(dev_pasid);

	return 0;

out_unwind_iopf:
	iopf_for_domain_replace(old, domain, dev);
out_remove_dev_pasid:
	domain_remove_dev_pasid(domain, dev, pasid);
	return ret;
}

static void *intel_iommu_hw_info(struct device *dev, u32 *length,
				 enum iommu_hw_info_type *type)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;
	struct iommu_hw_info_vtd *vtd;

	if (*type != IOMMU_HW_INFO_TYPE_DEFAULT &&
	    *type != IOMMU_HW_INFO_TYPE_INTEL_VTD)
		return ERR_PTR(-EOPNOTSUPP);

	vtd = kzalloc(sizeof(*vtd), GFP_KERNEL);
	if (!vtd)
		return ERR_PTR(-ENOMEM);

	vtd->flags = IOMMU_HW_INFO_VTD_ERRATA_772415_SPR17;
	vtd->cap_reg = iommu->cap;
	vtd->ecap_reg = iommu->ecap;
	*length = sizeof(*vtd);
	*type = IOMMU_HW_INFO_TYPE_INTEL_VTD;
	return vtd;
}

/*
 * Set dirty tracking for the device list of a domain. The caller must
 * hold the domain->lock when calling it.
 */
static int device_set_dirty_tracking(struct list_head *devices, bool enable)
{
	struct device_domain_info *info;
	int ret = 0;

	list_for_each_entry(info, devices, link) {
		ret = intel_pasid_setup_dirty_tracking(info->iommu, info->dev,
						       IOMMU_NO_PASID, enable);
		if (ret)
			break;
	}

	return ret;
}

static int parent_domain_set_dirty_tracking(struct dmar_domain *domain,
					    bool enable)
{
	struct dmar_domain *s1_domain;
	unsigned long flags;
	int ret;

	spin_lock(&domain->s1_lock);
	list_for_each_entry(s1_domain, &domain->s1_domains, s2_link) {
		spin_lock_irqsave(&s1_domain->lock, flags);
		ret = device_set_dirty_tracking(&s1_domain->devices, enable);
		spin_unlock_irqrestore(&s1_domain->lock, flags);
		if (ret)
			goto err_unwind;
	}
	spin_unlock(&domain->s1_lock);
	return 0;

err_unwind:
	list_for_each_entry(s1_domain, &domain->s1_domains, s2_link) {
		spin_lock_irqsave(&s1_domain->lock, flags);
		device_set_dirty_tracking(&s1_domain->devices,
					  domain->dirty_tracking);
		spin_unlock_irqrestore(&s1_domain->lock, flags);
	}
	spin_unlock(&domain->s1_lock);
	return ret;
}

static int intel_iommu_set_dirty_tracking(struct iommu_domain *domain,
					  bool enable)
{
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
	int ret;

	spin_lock(&dmar_domain->lock);
	if (dmar_domain->dirty_tracking == enable)
		goto out_unlock;

	ret = device_set_dirty_tracking(&dmar_domain->devices, enable);
	if (ret)
		goto err_unwind;

	if (dmar_domain->nested_parent) {
		ret = parent_domain_set_dirty_tracking(dmar_domain, enable);
		if (ret)
			goto err_unwind;
	}

	dmar_domain->dirty_tracking = enable;
out_unlock:
	spin_unlock(&dmar_domain->lock);

	return 0;

err_unwind:
	device_set_dirty_tracking(&dmar_domain->devices,
				  dmar_domain->dirty_tracking);
	spin_unlock(&dmar_domain->lock);
	return ret;
}

static int intel_iommu_read_and_clear_dirty(struct iommu_domain *domain,
					    unsigned long iova, size_t size,
					    unsigned long flags,
					    struct iommu_dirty_bitmap *dirty)
{
	struct dmar_domain *dmar_domain = to_dmar_domain(domain);
	unsigned long end = iova + size - 1;
	unsigned long pgsize;

	/*
	 * IOMMUFD core calls into a dirty tracking disabled domain without an
	 * IOVA bitmap set in order to clean dirty bits in all PTEs that might
	 * have occurred when we stopped dirty tracking. This ensures that we
	 * never inherit dirtied bits from a previous cycle.
	 */
	if (!dmar_domain->dirty_tracking && dirty->bitmap)
		return -EINVAL;

	do {
		struct dma_pte *pte;
		int lvl = 0;

		pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &lvl,
				     GFP_ATOMIC);
		pgsize = level_size(lvl) << VTD_PAGE_SHIFT;
		if (!pte || !dma_pte_present(pte)) {
			iova += pgsize;
			continue;
		}

		if (dma_sl_pte_test_and_clear_dirty(pte, flags))
			iommu_dirty_bitmap_record(dirty, iova, pgsize);
		iova += pgsize;
	} while (iova < end);

	return 0;
}

static const struct iommu_dirty_ops intel_dirty_ops = {
	.set_dirty_tracking = intel_iommu_set_dirty_tracking,
	.read_and_clear_dirty = intel_iommu_read_and_clear_dirty,
};

static int context_setup_pass_through(struct device *dev, u8 bus, u8 devfn)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;
	struct context_entry *context;

	spin_lock(&iommu->lock);
	context = iommu_context_addr(iommu, bus, devfn, 1);
	if (!context) {
		spin_unlock(&iommu->lock);
		return -ENOMEM;
	}

	if (context_present(context) && !context_copied(iommu, bus, devfn)) {
		spin_unlock(&iommu->lock);
		return 0;
	}

	copied_context_tear_down(iommu, context, bus, devfn);
	context_clear_entry(context);
	context_set_domain_id(context, FLPT_DEFAULT_DID);

	/*
	 * In pass through mode, AW must be programmed to indicate the largest
	 * AGAW value supported by hardware. And ASR is ignored by hardware.
	 */
	context_set_address_width(context, iommu->msagaw);
	context_set_translation_type(context, CONTEXT_TT_PASS_THROUGH);
	context_set_fault_enable(context);
	context_set_present(context);
	if (!ecap_coherent(iommu->ecap))
		clflush_cache_range(context, sizeof(*context));
	context_present_cache_flush(iommu, FLPT_DEFAULT_DID, bus, devfn);
	spin_unlock(&iommu->lock);

	return 0;
}

static int context_setup_pass_through_cb(struct pci_dev *pdev, u16 alias, void *data)
{
	struct device *dev = data;

	return context_setup_pass_through(dev, PCI_BUS_NUM(alias), alias & 0xff);
}

static int device_setup_pass_through(struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);

	if (!dev_is_pci(dev))
		return context_setup_pass_through(dev, info->bus, info->devfn);

	return pci_for_each_dma_alias(to_pci_dev(dev),
				      context_setup_pass_through_cb, dev);
}

static int identity_domain_attach_dev(struct iommu_domain *domain, struct device *dev)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;
	int ret;

	device_block_translation(dev);

	if (dev_is_real_dma_subdevice(dev))
		return 0;

	/*
	 * No PRI support with the global identity domain. No need to enable or
	 * disable PRI in this path as the iommu has been put in the blocking
	 * state.
	 */
	if (sm_supported(iommu))
		ret = intel_pasid_setup_pass_through(iommu, dev, IOMMU_NO_PASID);
	else
		ret = device_setup_pass_through(dev);

	if (!ret)
		info->domain_attached = true;

	return ret;
}

static int identity_domain_set_dev_pasid(struct iommu_domain *domain,
					 struct device *dev, ioasid_t pasid,
					 struct iommu_domain *old)
{
	struct device_domain_info *info = dev_iommu_priv_get(dev);
	struct intel_iommu *iommu = info->iommu;
	int ret;

	if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev))
		return -EOPNOTSUPP;

	ret = iopf_for_domain_replace(domain, old, dev);
	if (ret)
		return ret;

	ret = domain_setup_passthrough(iommu, dev, pasid, old);
	if (ret) {
		iopf_for_domain_replace(old, domain, dev);
		return ret;
	}

	domain_remove_dev_pasid(old, dev, pasid);
	return 0;
}

static struct iommu_domain identity_domain = {
	.type = IOMMU_DOMAIN_IDENTITY,
	.ops = &(const struct iommu_domain_ops) {
		.attach_dev	= identity_domain_attach_dev,
		.set_dev_pasid	= identity_domain_set_dev_pasid,
	},
};

const struct iommu_domain_ops intel_fs_paging_domain_ops = {
	.attach_dev = intel_iommu_attach_device,
	.set_dev_pasid = intel_iommu_set_dev_pasid,
	.map_pages = intel_iommu_map_pages,
	.unmap_pages = intel_iommu_unmap_pages,
	.iotlb_sync_map = intel_iommu_iotlb_sync_map,
	.flush_iotlb_all = intel_flush_iotlb_all,
	.iotlb_sync = intel_iommu_tlb_sync,
	.iova_to_phys = intel_iommu_iova_to_phys,
	.free = intel_iommu_domain_free,
	.enforce_cache_coherency = intel_iommu_enforce_cache_coherency_fs,
};

const struct iommu_domain_ops intel_ss_paging_domain_ops = {
	.attach_dev = intel_iommu_attach_device,
	.set_dev_pasid = intel_iommu_set_dev_pasid,
	.map_pages = intel_iommu_map_pages,
	.unmap_pages = intel_iommu_unmap_pages,
	.iotlb_sync_map = intel_iommu_iotlb_sync_map,
	.flush_iotlb_all = intel_flush_iotlb_all,
	.iotlb_sync = intel_iommu_tlb_sync,
	.iova_to_phys = intel_iommu_iova_to_phys,
	.free = intel_iommu_domain_free,
	.enforce_cache_coherency = intel_iommu_enforce_cache_coherency_ss,
};

const struct iommu_ops intel_iommu_ops = {
	.blocked_domain		= &blocking_domain,
	.release_domain		= &blocking_domain,
	.identity_domain	= &identity_domain,
	.capable		= intel_iommu_capable,
	.hw_info		= intel_iommu_hw_info,
	.domain_alloc_paging_flags = intel_iommu_domain_alloc_paging_flags,
	.domain_alloc_sva	= intel_svm_domain_alloc,
	.domain_alloc_nested	= intel_iommu_domain_alloc_nested,
	.probe_device		= intel_iommu_probe_device,
	.probe_finalize		= intel_iommu_probe_finalize,
	.release_device		= intel_iommu_release_device,
	.get_resv_regions	= intel_iommu_get_resv_regions,
	.device_group		= intel_iommu_device_group,
	.is_attach_deferred	= intel_iommu_is_attach_deferred,
	.def_domain_type	= device_def_domain_type,
	.page_response		= intel_iommu_page_response,
};

static void quirk_iommu_igfx(struct pci_dev *dev)
{
	if (risky_device(dev))
		return;

	pci_info(dev, "Disabling IOMMU for graphics on this chipset\n");
	disable_igfx_iommu = 1;
}

/* G4x/GM45 integrated gfx dmar support is totally busted. */
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2a40, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e00, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e10, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e20, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e30, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e40, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e90, quirk_iommu_igfx);

/* QM57/QS57 integrated gfx malfunctions with dmar */
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x0044, quirk_iommu_igfx);

/* Broadwell igfx malfunctions with dmar */
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x1606, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x160B, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x160E, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x1602, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x160A, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x160D, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x1616, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x161B, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x161E, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x1612, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x161A, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x161D, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x1626, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x162B, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x162E, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x1622, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x162A, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x162D, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x1636, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x163B, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x163E, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x1632, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x163A, quirk_iommu_igfx);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x163D, quirk_iommu_igfx);

static void quirk_iommu_rwbf(struct pci_dev *dev)
{
	if (risky_device(dev))
		return;

	/*
	 * Mobile 4 Series Chipset neglects to set RWBF capability,
	 * but needs it. Same seems to hold for the desktop versions.
	 */
	pci_info(dev, "Forcing write-buffer flush capability\n");
	rwbf_quirk = 1;
}

DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2a40, quirk_iommu_rwbf);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e00, quirk_iommu_rwbf);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e10, quirk_iommu_rwbf);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e20, quirk_iommu_rwbf);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e30, quirk_iommu_rwbf);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e40, quirk_iommu_rwbf);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x2e90, quirk_iommu_rwbf);

#define GGC 0x52
#define GGC_MEMORY_SIZE_MASK	(0xf << 8)
#define GGC_MEMORY_SIZE_NONE	(0x0 << 8)
#define GGC_MEMORY_SIZE_1M	(0x1 << 8)
#define GGC_MEMORY_SIZE_2M	(0x3 << 8)
#define GGC_MEMORY_VT_ENABLED	(0x8 << 8)
#define GGC_MEMORY_SIZE_2M_VT	(0x9 << 8)
#define GGC_MEMORY_SIZE_3M_VT	(0xa << 8)
#define GGC_MEMORY_SIZE_4M_VT	(0xb << 8)

static void quirk_calpella_no_shadow_gtt(struct pci_dev *dev)
{
	unsigned short ggc;

	if (risky_device(dev))
		return;

	if (pci_read_config_word(dev, GGC, &ggc))
		return;

	if (!(ggc & GGC_MEMORY_VT_ENABLED)) {
		pci_info(dev, "BIOS has allocated no shadow GTT; disabling IOMMU for graphics\n");
		disable_igfx_iommu = 1;
	} else if (!disable_igfx_iommu) {
		/* we have to ensure the gfx device is idle before we flush */
		pci_info(dev, "Disabling batched IOTLB flush on Ironlake\n");
		iommu_set_dma_strict();
	}
}
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x0040, quirk_calpella_no_shadow_gtt);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x0062, quirk_calpella_no_shadow_gtt);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x006a, quirk_calpella_no_shadow_gtt);

static void quirk_igfx_skip_te_disable(struct pci_dev *dev)
{
	unsigned short ver;

	if (!IS_GFX_DEVICE(dev))
		return;

	ver = (dev->device >> 8) & 0xff;
	if (ver != 0x45 && ver != 0x46 && ver != 0x4c &&
	    ver != 0x4e && ver != 0x8a && ver != 0x98 &&
	    ver != 0x9a && ver != 0xa7 && ver != 0x7d)
		return;

	if (risky_device(dev))
		return;

	pci_info(dev, "Skip IOMMU disabling for graphics\n");
	iommu_skip_te_disable = 1;
}
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, PCI_ANY_ID, quirk_igfx_skip_te_disable);

/* On Tylersburg chipsets, some BIOSes have been known to enable the
   ISOCH DMAR unit for the Azalia sound device, but not give it any
   TLB entries, which causes it to deadlock. Check for that.  We do
   this in a function called from init_dmars(), instead of in a PCI
   quirk, because we don't want to print the obnoxious "BIOS broken"
   message if VT-d is actually disabled.
*/
static void __init check_tylersburg_isoch(void)
{
	struct pci_dev *pdev;
	uint32_t vtisochctrl;

	/* If there's no Azalia in the system anyway, forget it. */
	pdev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x3a3e, NULL);
	if (!pdev)
		return;

	if (risky_device(pdev)) {
		pci_dev_put(pdev);
		return;
	}

	pci_dev_put(pdev);

	/* System Management Registers. Might be hidden, in which case
	   we can't do the sanity check. But that's OK, because the
	   known-broken BIOSes _don't_ actually hide it, so far. */
	pdev = pci_get_device(PCI_VENDOR_ID_INTEL, 0x342e, NULL);
	if (!pdev)
		return;

	if (risky_device(pdev)) {
		pci_dev_put(pdev);
		return;
	}

	if (pci_read_config_dword(pdev, 0x188, &vtisochctrl)) {
		pci_dev_put(pdev);
		return;
	}

	pci_dev_put(pdev);

	/* If Azalia DMA is routed to the non-isoch DMAR unit, fine. */
	if (vtisochctrl & 1)
		return;

	/* Drop all bits other than the number of TLB entries */
	vtisochctrl &= 0x1c;

	/* If we have the recommended number of TLB entries (16), fine. */
	if (vtisochctrl == 0x10)
		return;

	/* Zero TLB entries? You get to ride the short bus to school. */
	if (!vtisochctrl) {
		WARN(1, "Your BIOS is broken; DMA routed to ISOCH DMAR unit but no TLB space.\n"
		     "BIOS vendor: %s; Ver: %s; Product Version: %s\n",
		     dmi_get_system_info(DMI_BIOS_VENDOR),
		     dmi_get_system_info(DMI_BIOS_VERSION),
		     dmi_get_system_info(DMI_PRODUCT_VERSION));
		iommu_identity_mapping |= IDENTMAP_AZALIA;
		return;
	}

	pr_warn("Recommended TLB entries for ISOCH unit is 16; your BIOS set %d\n",
	       vtisochctrl);
}

/*
 * Here we deal with a device TLB defect where device may inadvertently issue ATS
 * invalidation completion before posted writes initiated with translated address
 * that utilized translations matching the invalidation address range, violating
 * the invalidation completion ordering.
 * Therefore, any use cases that cannot guarantee DMA is stopped before unmap is
 * vulnerable to this defect. In other words, any dTLB invalidation initiated not
 * under the control of the trusted/privileged host device driver must use this
 * quirk.
 * Device TLBs are invalidated under the following six conditions:
 * 1. Device driver does DMA API unmap IOVA
 * 2. Device driver unbind a PASID from a process, sva_unbind_device()
 * 3. PASID is torn down, after PASID cache is flushed. e.g. process
 *    exit_mmap() due to crash
 * 4. Under SVA usage, called by mmu_notifier.invalidate_range() where
 *    VM has to free pages that were unmapped
 * 5. Userspace driver unmaps a DMA buffer
 * 6. Cache invalidation in vSVA usage (upcoming)
 *
 * For #1 and #2, device drivers are responsible for stopping DMA traffic
 * before unmap/unbind. For #3, iommu driver gets mmu_notifier to
 * invalidate TLB the same way as normal user unmap which will use this quirk.
 * The dTLB invalidation after PASID cache flush does not need this quirk.
 *
 * As a reminder, #6 will *NEED* this quirk as we enable nested translation.
 */
void quirk_extra_dev_tlb_flush(struct device_domain_info *info,
			       unsigned long address, unsigned long mask,
			       u32 pasid, u16 qdep)
{
	u16 sid;

	if (likely(!info->dtlb_extra_inval))
		return;

	sid = PCI_DEVID(info->bus, info->devfn);
	if (pasid == IOMMU_NO_PASID) {
		qi_flush_dev_iotlb(info->iommu, sid, info->pfsid,
				   qdep, address, mask);
	} else {
		qi_flush_dev_iotlb_pasid(info->iommu, sid, info->pfsid,
					 pasid, qdep, address, mask);
	}
}

#define ecmd_get_status_code(res)	(((res) & 0xff) >> 1)

/*
 * Function to submit a command to the enhanced command interface. The
 * valid enhanced command descriptions are defined in Table 47 of the
 * VT-d spec. The VT-d hardware implementation may support some but not
 * all commands, which can be determined by checking the Enhanced
 * Command Capability Register.
 *
 * Return values:
 *  - 0: Command successful without any error;
 *  - Negative: software error value;
 *  - Nonzero positive: failure status code defined in Table 48.
 */
int ecmd_submit_sync(struct intel_iommu *iommu, u8 ecmd, u64 oa, u64 ob)
{
	unsigned long flags;
	u64 res;
	int ret;

	if (!cap_ecmds(iommu->cap))
		return -ENODEV;

	raw_spin_lock_irqsave(&iommu->register_lock, flags);

	res = dmar_readq(iommu->reg + DMAR_ECRSP_REG);
	if (res & DMA_ECMD_ECRSP_IP) {
		ret = -EBUSY;
		goto err;
	}

	/*
	 * Unconditionally write the operand B, because
	 * - There is no side effect if an ecmd doesn't require an
	 *   operand B, but we set the register to some value.
	 * - It's not invoked in any critical path. The extra MMIO
	 *   write doesn't bring any performance concerns.
	 */
	dmar_writeq(iommu->reg + DMAR_ECEO_REG, ob);
	dmar_writeq(iommu->reg + DMAR_ECMD_REG, ecmd | (oa << DMA_ECMD_OA_SHIFT));

	IOMMU_WAIT_OP(iommu, DMAR_ECRSP_REG, dmar_readq,
		      !(res & DMA_ECMD_ECRSP_IP), res);

	if (res & DMA_ECMD_ECRSP_IP) {
		ret = -ETIMEDOUT;
		goto err;
	}

	ret = ecmd_get_status_code(res);
err:
	raw_spin_unlock_irqrestore(&iommu->register_lock, flags);

	return ret;
}
]

gemini-3.1-pro-preview

							input: 75481
							output: 21
							thoughts: 3174
						

**Okay, let's break this down. The core issue is a double mapping, specifically a PTE being set twice. The `WARN_ON(1)` in `__domain_mapping` at `drivers/iommu/intel/iommu.c:1679` is the smoking gun.** The call trace helps clarify the context: the mapping is triggered by a `DRM_IOCTL_PRIME_FD_TO_HANDLE` ioctl within virtio-gpu, ultimately leading to a dma-buf being imported from a system dma_heap.  The key part is that, the flow goes through `virtgpu_gem_prime_import` -> `system_heap_map_dma_buf` -> `dma_map_sgtable` -> `__dma_map_sg_attrs` -> `iommu_dma_map_sg` -> `iommu_map_sg` -> `iommu_map_nosync` -> `intel_iommu_map_pages` -> `intel_iommu_map` -> `__domain_mapping`. This is a pretty standard dma-buf import sequence.

The reproducer provides the critical clue. It *injects a fault* with `inject_fault(10);` right before the `DRM_IOCTL_PRIME_FD_TO_HANDLE` ioctl is called.  This strongly suggests the double mapping occurs due to an error path that fails to properly clean up after a previous mapping attempt.  The fault injection is very likely corrupting something, likely the IOMMU state. This could be a race condition, where one part of the mapping is successful, and then due to the fault injection, an error occurs, but some state is left around and a second mapping is attempted. The `try_cmpxchg64_local` is the check, but it's only catching the fact that a prior mapping is already there.

Therefore, the fix *might* be in ensuring proper cleanup in error paths related to the IOMMU mapping code when a fault is injected in `DRM_IOCTL_PRIME_FD_TO_HANDLE` function. The key is to examine the error handling in the chain of calls, especially in `system_heap_map_dma_buf`, `dma_map_sgtable` and `__dma_map_sg_attrs` since that's where the mapping happens.  Also, it's possible the injection could occur anywhere in the system, even outside the DMA path, and corruption happens.  In the end, though, the solution is in fixing any missing cleanup steps after potential errors with the IOMMU mapping, and ensuring no state lingers around.

map[Expression:iommu_map_sg]

map[Output:arch/arm/mm/dma-mapping.c=1185=static int __map_sg_chunk(struct device *dev, struct scatterlist *sg,
--
arch/arm/mm/dma-mapping.c-1228-/**
arch/arm/mm/dma-mapping.c:1229: * arm_iommu_map_sg - map a set of SG buffers for streaming mode DMA
arch/arm/mm/dma-mapping.c-1230- * @dev: valid struct device pointer
--
arch/arm/mm/dma-mapping.c-1239- */
arch/arm/mm/dma-mapping.c:1240:static int arm_iommu_map_sg(struct device *dev, struct scatterlist *sg,
arch/arm/mm/dma-mapping.c-1241-		int nents, enum dma_data_direction dir, unsigned long attrs)
--
arch/arm/mm/dma-mapping.c=1513=static const struct dma_map_ops iommu_ops = {
--
arch/arm/mm/dma-mapping.c-1523-
arch/arm/mm/dma-mapping.c:1524:	.map_sg			= arm_iommu_map_sg,
arch/arm/mm/dma-mapping.c-1525-	.unmap_sg		= arm_iommu_unmap_sg,
--
arch/powerpc/include/asm/iommu.h=254=static inline int dma_iommu_dma_supported(struct device *dev, u64 mask)
--
arch/powerpc/include/asm/iommu.h-260-
arch/powerpc/include/asm/iommu.h:261:extern int ppc_iommu_map_sg(struct device *dev, struct iommu_table *tbl,
arch/powerpc/include/asm/iommu.h-262-			    struct scatterlist *sglist, int nelems,
--
arch/powerpc/kernel/dma-iommu.c=109=static void dma_iommu_unmap_page(struct device *dev, dma_addr_t dma_handle,
--
arch/powerpc/kernel/dma-iommu.c-117-
arch/powerpc/kernel/dma-iommu.c:118:static int dma_iommu_map_sg(struct device *dev, struct scatterlist *sglist,
arch/powerpc/kernel/dma-iommu.c-119-			    int nelems, enum dma_data_direction direction,
--
arch/powerpc/kernel/dma-iommu.c-121-{
arch/powerpc/kernel/dma-iommu.c:122:	return ppc_iommu_map_sg(dev, get_iommu_table_base(dev), sglist, nelems,
arch/powerpc/kernel/dma-iommu.c-123-				dma_get_mask(dev), direction, attrs);
--
arch/powerpc/kernel/dma-iommu.c=208=const struct dma_map_ops dma_iommu_ops = {
--
arch/powerpc/kernel/dma-iommu.c-210-	.free			= dma_iommu_free_coherent,
arch/powerpc/kernel/dma-iommu.c:211:	.map_sg			= dma_iommu_map_sg,
arch/powerpc/kernel/dma-iommu.c-212-	.unmap_sg		= dma_iommu_unmap_sg,
--
arch/powerpc/kernel/iommu.c=459=static void iommu_free(struct iommu_table *tbl, dma_addr_t dma_addr,
--
arch/powerpc/kernel/iommu.c-471-
arch/powerpc/kernel/iommu.c:472:int ppc_iommu_map_sg(struct device *dev, struct iommu_table *tbl,
arch/powerpc/kernel/iommu.c-473-		     struct scatterlist *sglist, int nelems,
--
arch/powerpc/platforms/pseries/vio.c=539=static void vio_dma_iommu_unmap_page(struct device *dev, dma_addr_t dma_handle,
--
arch/powerpc/platforms/pseries/vio.c-550-
arch/powerpc/platforms/pseries/vio.c:551:static int vio_dma_iommu_map_sg(struct device *dev, struct scatterlist *sglist,
arch/powerpc/platforms/pseries/vio.c-552-                                int nelems, enum dma_data_direction direction,
--
arch/powerpc/platforms/pseries/vio.c-566-		goto out_fail;
arch/powerpc/platforms/pseries/vio.c:567:	ret = ppc_iommu_map_sg(dev, tbl, sglist, nelems, dma_get_mask(dev),
arch/powerpc/platforms/pseries/vio.c-568-			direction, attrs);
--
arch/powerpc/platforms/pseries/vio.c=603=static const struct dma_map_ops vio_dma_mapping_ops = {
--
arch/powerpc/platforms/pseries/vio.c-605-	.free              = vio_dma_iommu_free_coherent,
arch/powerpc/platforms/pseries/vio.c:606:	.map_sg            = vio_dma_iommu_map_sg,
arch/powerpc/platforms/pseries/vio.c-607-	.unmap_sg          = vio_dma_iommu_unmap_sg,
--
arch/sparc/mm/iommu.c=242=static dma_addr_t sbus_iommu_map_page_pflush(struct device *dev,
--
arch/sparc/mm/iommu.c-248-
arch/sparc/mm/iommu.c:249:static int __sbus_iommu_map_sg(struct device *dev, struct scatterlist *sgl,
arch/sparc/mm/iommu.c-250-		int nents, enum dma_data_direction dir, unsigned long attrs,
--
arch/sparc/mm/iommu.c-266-
arch/sparc/mm/iommu.c:267:static int sbus_iommu_map_sg_gflush(struct device *dev, struct scatterlist *sgl,
arch/sparc/mm/iommu.c-268-		int nents, enum dma_data_direction dir, unsigned long attrs)
--
arch/sparc/mm/iommu.c-270-	flush_page_for_dma(0);
arch/sparc/mm/iommu.c:271:	return __sbus_iommu_map_sg(dev, sgl, nents, dir, attrs, false);
arch/sparc/mm/iommu.c-272-}
arch/sparc/mm/iommu.c-273-
arch/sparc/mm/iommu.c:274:static int sbus_iommu_map_sg_pflush(struct device *dev, struct scatterlist *sgl,
arch/sparc/mm/iommu.c-275-		int nents, enum dma_data_direction dir, unsigned long attrs)
arch/sparc/mm/iommu.c-276-{
arch/sparc/mm/iommu.c:277:	return __sbus_iommu_map_sg(dev, sgl, nents, dir, attrs, true);
arch/sparc/mm/iommu.c-278-}
--
arch/sparc/mm/iommu.c=424=static const struct dma_map_ops sbus_iommu_dma_gflush_ops = {
--
arch/sparc/mm/iommu.c-430-	.unmap_page		= sbus_iommu_unmap_page,
arch/sparc/mm/iommu.c:431:	.map_sg			= sbus_iommu_map_sg_gflush,
arch/sparc/mm/iommu.c-432-	.unmap_sg		= sbus_iommu_unmap_sg,
--
arch/sparc/mm/iommu.c=435=static const struct dma_map_ops sbus_iommu_dma_pflush_ops = {
--
arch/sparc/mm/iommu.c-441-	.unmap_page		= sbus_iommu_unmap_page,
arch/sparc/mm/iommu.c:442:	.map_sg			= sbus_iommu_map_sg_pflush,
arch/sparc/mm/iommu.c-443-	.unmap_sg		= sbus_iommu_unmap_sg,
--
drivers/gpu/drm/msm/msm_iommu.c=670=static int msm_iommu_map(struct msm_mmu *mmu, uint64_t iova,
--
drivers/gpu/drm/msm/msm_iommu.c-682-
drivers/gpu/drm/msm/msm_iommu.c:683:	ret = iommu_map_sgtable(iommu->domain, iova, sgt, prot);
drivers/gpu/drm/msm/msm_iommu.c-684-	WARN_ON(!ret);
--
drivers/gpu/drm/rockchip/rockchip_drm_gem.c=21=static int rockchip_gem_iommu_map(struct rockchip_gem_object *rk_obj)
--
drivers/gpu/drm/rockchip/rockchip_drm_gem.c-40-
drivers/gpu/drm/rockchip/rockchip_drm_gem.c:41:	ret = iommu_map_sgtable(private->domain, rk_obj->dma_addr, rk_obj->sgt,
drivers/gpu/drm/rockchip/rockchip_drm_gem.c-42-				prot);
--
drivers/gpu/drm/rockchip/rockchip_drm_gem.c=452=static int
drivers/gpu/drm/rockchip/rockchip_drm_gem.c:453:rockchip_gem_iommu_map_sg(struct drm_device *drm,
drivers/gpu/drm/rockchip/rockchip_drm_gem.c-454-			  struct dma_buf_attachment *attach,
--
drivers/gpu/drm/rockchip/rockchip_drm_gem.c=484=rockchip_gem_prime_import_sg_table(struct drm_device *drm,
--
drivers/gpu/drm/rockchip/rockchip_drm_gem.c-496-	if (private->domain)
drivers/gpu/drm/rockchip/rockchip_drm_gem.c:497:		ret = rockchip_gem_iommu_map_sg(drm, attach, sg, rk_obj);
drivers/gpu/drm/rockchip/rockchip_drm_gem.c-498-	else
--
drivers/gpu/drm/tegra/gem.c=234=static int tegra_bo_iommu_map(struct tegra_drm *tegra, struct tegra_bo *bo)
--
drivers/gpu/drm/tegra/gem.c-257-
drivers/gpu/drm/tegra/gem.c:258:	bo->size = iommu_map_sgtable(tegra->domain, bo->iova, bo->sgt, prot);
drivers/gpu/drm/tegra/gem.c-259-	if (!bo->size) {
--
drivers/gpu/host1x/job.c=141=static unsigned int pin_job(struct host1x *host, struct host1x_job *job)
--
drivers/gpu/host1x/job.c-250-
drivers/gpu/host1x/job.c:251:			err = iommu_map_sgtable(host->domain, iova_dma_addr(&host->iova, alloc),
drivers/gpu/host1x/job.c-252-						map->sgt, IOMMU_READ);
--
drivers/iommu/dma-iommu.c=924=static struct page **__iommu_dma_alloc_noncontiguous(struct device *dev,
--
drivers/iommu/dma-iommu.c-979-
drivers/iommu/dma-iommu.c:980:	ret = iommu_map_sg(domain, iova, sgt->sgl, sgt->orig_nents, ioprot,
drivers/iommu/dma-iommu.c-981-			   gfp);
--
drivers/iommu/dma-iommu.c=1378=int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
--
drivers/iommu/dma-iommu.c-1427-			/*
drivers/iommu/dma-iommu.c:1428:			 * iommu_map_sg() will skip this segment as it is marked
drivers/iommu/dma-iommu.c-1429-			 * as a bus address, __finalise_sg() will copy the dma
--
drivers/iommu/dma-iommu.c-1482-	 */
drivers/iommu/dma-iommu.c:1483:	ret = iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC);
drivers/iommu/dma-iommu.c-1484-	if (ret < 0 || ret < iova_len)
--
drivers/iommu/iommu.c=2662=EXPORT_SYMBOL_GPL(iommu_unmap_fast);
drivers/iommu/iommu.c-2663-
drivers/iommu/iommu.c:2664:ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
drivers/iommu/iommu.c-2665-		     struct scatterlist *sg, unsigned int nents, int prot,
--
drivers/iommu/iommu.c-2712-}
drivers/iommu/iommu.c:2713:EXPORT_SYMBOL_GPL(iommu_map_sg);
drivers/iommu/iommu.c-2714-
--
drivers/media/platform/nvidia/tegra-vde/iommu.c=19=int tegra_vde_iommu_map(struct tegra_vde *vde,
--
drivers/media/platform/nvidia/tegra-vde/iommu.c-38-
drivers/media/platform/nvidia/tegra-vde/iommu.c:39:	size = iommu_map_sgtable(vde->domain, addr, sgt,
drivers/media/platform/nvidia/tegra-vde/iommu.c-40-				 IOMMU_READ | IOMMU_WRITE);
--
drivers/staging/media/ipu3/ipu3-mmu.c=251=int imgu_mmu_map(struct imgu_mmu_info *info, unsigned long iova,
--
drivers/staging/media/ipu3/ipu3-mmu.c-295- *
drivers/staging/media/ipu3/ipu3-mmu.c:296: * The function has been adapted from default_iommu_map_sg() in
drivers/staging/media/ipu3/ipu3-mmu.c-297- * drivers/iommu/iommu.c .
--
include/linux/iommu.h=921=extern size_t iommu_unmap_fast(struct iommu_domain *domain,
--
include/linux/iommu.h-923-			       struct iommu_iotlb_gather *iotlb_gather);
include/linux/iommu.h:924:extern ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
include/linux/iommu.h-925-			    struct scatterlist *sg, unsigned int nents,
--
include/linux/iommu.h=1246=static inline size_t iommu_unmap_fast(struct iommu_domain *domain,
--
include/linux/iommu.h-1252-
include/linux/iommu.h:1253:static inline ssize_t iommu_map_sg(struct iommu_domain *domain,
include/linux/iommu.h-1254-				   unsigned long iova, struct scatterlist *sg,
--
include/linux/iommu.h=1527=static inline void iommu_group_mutex_assert(struct device *dev)
--
include/linux/iommu.h-1532-/**
include/linux/iommu.h:1533: * iommu_map_sgtable - Map the given buffer to the IOMMU domain
include/linux/iommu.h-1534- * @domain:	The IOMMU domain to perform the mapping
--
include/linux/iommu.h-1541- */
include/linux/iommu.h:1542:static inline ssize_t iommu_map_sgtable(struct iommu_domain *domain,
include/linux/iommu.h-1543-			unsigned long iova, struct sg_table *sgt, int prot)
include/linux/iommu.h-1544-{
include/linux/iommu.h:1545:	return iommu_map_sg(domain, iova, sgt->sgl, sgt->orig_nents, prot,
include/linux/iommu.h-1546-			    GFP_KERNEL);
]

gemini-3.1-pro-preview

							input: 83951
							output: 26
							thoughts: 47
						

Okay, here's my take on those thoughts, tailored for an expert audience:

**Analyzing a Fault Injection Scenario**

Alright, so the fault injection attempt is biting the dust right smack in the middle of an `iommu_map_sg` call. That points the finger directly at the Scatter Gather mapping stage within the IOMMU translation process. My immediate reaction is to zero in on the `iommu_map_sg` function itself. I need to understand what's happening during the mapping, where it might be vulnerable, and the exact point of the fault. The fact that it's failing *within* the mapping process strongly suggests a problem related to the hardware page table walks, the allocation of IOMMU-specific resources, or a subtle timing issue during the translation setup. Time to open up `drivers/iommu/iommu.c` and dig into that function. The architecture likely has its quirks when it comes to SG list management, and that's the place to start.

map[File:drivers/iommu/iommu.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
 * Copyright (C) 2007-2008 Advanced Micro Devices, Inc.
 * Author: Joerg Roedel <jroedel@suse.de>
 */

#define pr_fmt(fmt)    "iommu: " fmt

#include <linux/amba/bus.h>
#include <linux/device.h>
#include <linux/kernel.h>
#include <linux/bits.h>
#include <linux/bug.h>
#include <linux/types.h>
#include <linux/init.h>
#include <linux/export.h>
#include <linux/slab.h>
#include <linux/errno.h>
#include <linux/host1x_context_bus.h>
#include <linux/iommu.h>
#include <linux/iommufd.h>
#include <linux/idr.h>
#include <linux/err.h>
#include <linux/pci.h>
#include <linux/pci-ats.h>
#include <linux/bitops.h>
#include <linux/platform_device.h>
#include <linux/property.h>
#include <linux/fsl/mc.h>
#include <linux/module.h>
#include <linux/cc_platform.h>
#include <linux/cdx/cdx_bus.h>
#include <trace/events/iommu.h>
#include <linux/sched/mm.h>
#include <linux/msi.h>
#include <uapi/linux/iommufd.h>

#include "dma-iommu.h"
#include "iommu-priv.h"

static struct kset *iommu_group_kset;
static DEFINE_IDA(iommu_group_ida);
static DEFINE_IDA(iommu_global_pasid_ida);

static unsigned int iommu_def_domain_type __read_mostly;
static bool iommu_dma_strict __read_mostly = IS_ENABLED(CONFIG_IOMMU_DEFAULT_DMA_STRICT);
static u32 iommu_cmd_line __read_mostly;

/* Tags used with xa_tag_pointer() in group->pasid_array */
enum { IOMMU_PASID_ARRAY_DOMAIN = 0, IOMMU_PASID_ARRAY_HANDLE = 1 };

struct iommu_group {
	struct kobject kobj;
	struct kobject *devices_kobj;
	struct list_head devices;
	struct xarray pasid_array;
	struct mutex mutex;
	void *iommu_data;
	void (*iommu_data_release)(void *iommu_data);
	char *name;
	int id;
	struct iommu_domain *default_domain;
	struct iommu_domain *blocking_domain;
	struct iommu_domain *domain;
	struct list_head entry;
	unsigned int owner_cnt;
	void *owner;
};

struct group_device {
	struct list_head list;
	struct device *dev;
	char *name;
};

/* Iterate over each struct group_device in a struct iommu_group */
#define for_each_group_device(group, pos) \
	list_for_each_entry(pos, &(group)->devices, list)

struct iommu_group_attribute {
	struct attribute attr;
	ssize_t (*show)(struct iommu_group *group, char *buf);
	ssize_t (*store)(struct iommu_group *group,
			 const char *buf, size_t count);
};

static const char * const iommu_group_resv_type_string[] = {
	[IOMMU_RESV_DIRECT]			= "direct",
	[IOMMU_RESV_DIRECT_RELAXABLE]		= "direct-relaxable",
	[IOMMU_RESV_RESERVED]			= "reserved",
	[IOMMU_RESV_MSI]			= "msi",
	[IOMMU_RESV_SW_MSI]			= "msi",
};

#define IOMMU_CMD_LINE_DMA_API		BIT(0)
#define IOMMU_CMD_LINE_STRICT		BIT(1)

static int bus_iommu_probe(const struct bus_type *bus);
static int iommu_bus_notifier(struct notifier_block *nb,
			      unsigned long action, void *data);
static void iommu_release_device(struct device *dev);
static int __iommu_attach_device(struct iommu_domain *domain,
				 struct device *dev);
static int __iommu_attach_group(struct iommu_domain *domain,
				struct iommu_group *group);
static struct iommu_domain *__iommu_paging_domain_alloc_flags(struct device *dev,
						       unsigned int type,
						       unsigned int flags);

enum {
	IOMMU_SET_DOMAIN_MUST_SUCCEED = 1 << 0,
};

static int __iommu_device_set_domain(struct iommu_group *group,
				     struct device *dev,
				     struct iommu_domain *new_domain,
				     unsigned int flags);
static int __iommu_group_set_domain_internal(struct iommu_group *group,
					     struct iommu_domain *new_domain,
					     unsigned int flags);
static int __iommu_group_set_domain(struct iommu_group *group,
				    struct iommu_domain *new_domain)
{
	return __iommu_group_set_domain_internal(group, new_domain, 0);
}
static void __iommu_group_set_domain_nofail(struct iommu_group *group,
					    struct iommu_domain *new_domain)
{
	WARN_ON(__iommu_group_set_domain_internal(
		group, new_domain, IOMMU_SET_DOMAIN_MUST_SUCCEED));
}

static int iommu_setup_default_domain(struct iommu_group *group,
				      int target_type);
static int iommu_create_device_direct_mappings(struct iommu_domain *domain,
					       struct device *dev);
static ssize_t iommu_group_store_type(struct iommu_group *group,
				      const char *buf, size_t count);
static struct group_device *iommu_group_alloc_device(struct iommu_group *group,
						     struct device *dev);
static void __iommu_group_free_device(struct iommu_group *group,
				      struct group_device *grp_dev);
static void iommu_domain_init(struct iommu_domain *domain, unsigned int type,
			      const struct iommu_ops *ops);

#define IOMMU_GROUP_ATTR(_name, _mode, _show, _store)		\
struct iommu_group_attribute iommu_group_attr_##_name =		\
	__ATTR(_name, _mode, _show, _store)

#define to_iommu_group_attr(_attr)	\
	container_of(_attr, struct iommu_group_attribute, attr)
#define to_iommu_group(_kobj)		\
	container_of(_kobj, struct iommu_group, kobj)

static LIST_HEAD(iommu_device_list);
static DEFINE_SPINLOCK(iommu_device_lock);

static const struct bus_type * const iommu_buses[] = {
	&platform_bus_type,
#ifdef CONFIG_PCI
	&pci_bus_type,
#endif
#ifdef CONFIG_ARM_AMBA
	&amba_bustype,
#endif
#ifdef CONFIG_FSL_MC_BUS
	&fsl_mc_bus_type,
#endif
#ifdef CONFIG_TEGRA_HOST1X_CONTEXT_BUS
	&host1x_context_device_bus_type,
#endif
#ifdef CONFIG_CDX_BUS
	&cdx_bus_type,
#endif
};

/*
 * Use a function instead of an array here because the domain-type is a
 * bit-field, so an array would waste memory.
 */
static const char *iommu_domain_type_str(unsigned int t)
{
	switch (t) {
	case IOMMU_DOMAIN_BLOCKED:
		return "Blocked";
	case IOMMU_DOMAIN_IDENTITY:
		return "Passthrough";
	case IOMMU_DOMAIN_UNMANAGED:
		return "Unmanaged";
	case IOMMU_DOMAIN_DMA:
	case IOMMU_DOMAIN_DMA_FQ:
		return "Translated";
	case IOMMU_DOMAIN_PLATFORM:
		return "Platform";
	default:
		return "Unknown";
	}
}

static int __init iommu_subsys_init(void)
{
	struct notifier_block *nb;

	if (!(iommu_cmd_line & IOMMU_CMD_LINE_DMA_API)) {
		if (IS_ENABLED(CONFIG_IOMMU_DEFAULT_PASSTHROUGH))
			iommu_set_default_passthrough(false);
		else
			iommu_set_default_translated(false);

		if (iommu_default_passthrough() && cc_platform_has(CC_ATTR_MEM_ENCRYPT)) {
			pr_info("Memory encryption detected - Disabling default IOMMU Passthrough\n");
			iommu_set_default_translated(false);
		}
	}

	if (!iommu_default_passthrough() && !iommu_dma_strict)
		iommu_def_domain_type = IOMMU_DOMAIN_DMA_FQ;

	pr_info("Default domain type: %s%s\n",
		iommu_domain_type_str(iommu_def_domain_type),
		(iommu_cmd_line & IOMMU_CMD_LINE_DMA_API) ?
			" (set via kernel command line)" : "");

	if (!iommu_default_passthrough())
		pr_info("DMA domain TLB invalidation policy: %s mode%s\n",
			iommu_dma_strict ? "strict" : "lazy",
			(iommu_cmd_line & IOMMU_CMD_LINE_STRICT) ?
				" (set via kernel command line)" : "");

	nb = kcalloc(ARRAY_SIZE(iommu_buses), sizeof(*nb), GFP_KERNEL);
	if (!nb)
		return -ENOMEM;

	for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++) {
		nb[i].notifier_call = iommu_bus_notifier;
		bus_register_notifier(iommu_buses[i], &nb[i]);
	}

	return 0;
}
subsys_initcall(iommu_subsys_init);

static int remove_iommu_group(struct device *dev, void *data)
{
	if (dev->iommu && dev->iommu->iommu_dev == data)
		iommu_release_device(dev);

	return 0;
}

/**
 * iommu_device_register() - Register an IOMMU hardware instance
 * @iommu: IOMMU handle for the instance
 * @ops:   IOMMU ops to associate with the instance
 * @hwdev: (optional) actual instance device, used for fwnode lookup
 *
 * Return: 0 on success, or an error.
 */
int iommu_device_register(struct iommu_device *iommu,
			  const struct iommu_ops *ops, struct device *hwdev)
{
	int err = 0;

	/* We need to be able to take module references appropriately */
	if (WARN_ON(is_module_address((unsigned long)ops) && !ops->owner))
		return -EINVAL;

	iommu->ops = ops;
	if (hwdev)
		iommu->fwnode = dev_fwnode(hwdev);

	spin_lock(&iommu_device_lock);
	list_add_tail(&iommu->list, &iommu_device_list);
	spin_unlock(&iommu_device_lock);

	for (int i = 0; i < ARRAY_SIZE(iommu_buses) && !err; i++)
		err = bus_iommu_probe(iommu_buses[i]);
	if (err)
		iommu_device_unregister(iommu);
	else
		WRITE_ONCE(iommu->ready, true);
	return err;
}
EXPORT_SYMBOL_GPL(iommu_device_register);

void iommu_device_unregister(struct iommu_device *iommu)
{
	for (int i = 0; i < ARRAY_SIZE(iommu_buses); i++)
		bus_for_each_dev(iommu_buses[i], NULL, iommu, remove_iommu_group);

	spin_lock(&iommu_device_lock);
	list_del(&iommu->list);
	spin_unlock(&iommu_device_lock);

	/* Pairs with the alloc in generic_single_device_group() */
	iommu_group_put(iommu->singleton_group);
	iommu->singleton_group = NULL;
}
EXPORT_SYMBOL_GPL(iommu_device_unregister);

#if IS_ENABLED(CONFIG_IOMMUFD_TEST)
void iommu_device_unregister_bus(struct iommu_device *iommu,
				 const struct bus_type *bus,
				 struct notifier_block *nb)
{
	bus_unregister_notifier(bus, nb);
	iommu_device_unregister(iommu);
}
EXPORT_SYMBOL_GPL(iommu_device_unregister_bus);

/*
 * Register an iommu driver against a single bus. This is only used by iommufd
 * selftest to create a mock iommu driver. The caller must provide
 * some memory to hold a notifier_block.
 */
int iommu_device_register_bus(struct iommu_device *iommu,
			      const struct iommu_ops *ops,
			      const struct bus_type *bus,
			      struct notifier_block *nb)
{
	int err;

	iommu->ops = ops;
	nb->notifier_call = iommu_bus_notifier;
	err = bus_register_notifier(bus, nb);
	if (err)
		return err;

	spin_lock(&iommu_device_lock);
	list_add_tail(&iommu->list, &iommu_device_list);
	spin_unlock(&iommu_device_lock);

	err = bus_iommu_probe(bus);
	if (err) {
		iommu_device_unregister_bus(iommu, bus, nb);
		return err;
	}
	return 0;
}
EXPORT_SYMBOL_GPL(iommu_device_register_bus);
#endif

static struct dev_iommu *dev_iommu_get(struct device *dev)
{
	struct dev_iommu *param = dev->iommu;

	lockdep_assert_held(&iommu_probe_device_lock);

	if (param)
		return param;

	param = kzalloc(sizeof(*param), GFP_KERNEL);
	if (!param)
		return NULL;

	mutex_init(&param->lock);
	dev->iommu = param;
	return param;
}

void dev_iommu_free(struct device *dev)
{
	struct dev_iommu *param = dev->iommu;

	dev->iommu = NULL;
	if (param->fwspec) {
		fwnode_handle_put(param->fwspec->iommu_fwnode);
		kfree(param->fwspec);
	}
	kfree(param);
}

/*
 * Internal equivalent of device_iommu_mapped() for when we care that a device
 * actually has API ops, and don't want false positives from VFIO-only groups.
 */
static bool dev_has_iommu(struct device *dev)
{
	return dev->iommu && dev->iommu->iommu_dev;
}

static u32 dev_iommu_get_max_pasids(struct device *dev)
{
	u32 max_pasids = 0, bits = 0;
	int ret;

	if (dev_is_pci(dev)) {
		ret = pci_max_pasids(to_pci_dev(dev));
		if (ret > 0)
			max_pasids = ret;
	} else {
		ret = device_property_read_u32(dev, "pasid-num-bits", &bits);
		if (!ret)
			max_pasids = 1UL << bits;
	}

	return min_t(u32, max_pasids, dev->iommu->iommu_dev->max_pasids);
}

void dev_iommu_priv_set(struct device *dev, void *priv)
{
	/* FSL_PAMU does something weird */
	if (!IS_ENABLED(CONFIG_FSL_PAMU))
		lockdep_assert_held(&iommu_probe_device_lock);
	dev->iommu->priv = priv;
}
EXPORT_SYMBOL_GPL(dev_iommu_priv_set);

/*
 * Init the dev->iommu and dev->iommu_group in the struct device and get the
 * driver probed
 */
static int iommu_init_device(struct device *dev)
{
	const struct iommu_ops *ops;
	struct iommu_device *iommu_dev;
	struct iommu_group *group;
	int ret;

	if (!dev_iommu_get(dev))
		return -ENOMEM;
	/*
	 * For FDT-based systems and ACPI IORT/VIOT, the common firmware parsing
	 * is buried in the bus dma_configure path. Properly unpicking that is
	 * still a big job, so for now just invoke the whole thing. The device
	 * already having a driver bound means dma_configure has already run and
	 * found no IOMMU to wait for, so there's no point calling it again.
	 */
	if (!dev->iommu->fwspec && !dev->driver && dev->bus->dma_configure) {
		mutex_unlock(&iommu_probe_device_lock);
		dev->bus->dma_configure(dev);
		mutex_lock(&iommu_probe_device_lock);
		/* If another instance finished the job for us, skip it */
		if (!dev->iommu || dev->iommu_group)
			return -ENODEV;
	}
	/*
	 * At this point, relevant devices either now have a fwspec which will
	 * match ops registered with a non-NULL fwnode, or we can reasonably
	 * assume that only one of Intel, AMD, s390, PAMU or legacy SMMUv2 can
	 * be present, and that any of their registered instances has suitable
	 * ops for probing, and thus cheekily co-opt the same mechanism.
	 */
	ops = iommu_fwspec_ops(dev->iommu->fwspec);
	if (!ops) {
		ret = -ENODEV;
		goto err_free;
	}

	if (!try_module_get(ops->owner)) {
		ret = -EINVAL;
		goto err_free;
	}

	iommu_dev = ops->probe_device(dev);
	if (IS_ERR(iommu_dev)) {
		ret = PTR_ERR(iommu_dev);
		goto err_module_put;
	}
	dev->iommu->iommu_dev = iommu_dev;

	ret = iommu_device_link(iommu_dev, dev);
	if (ret)
		goto err_release;

	group = ops->device_group(dev);
	if (WARN_ON_ONCE(group == NULL))
		group = ERR_PTR(-EINVAL);
	if (IS_ERR(group)) {
		ret = PTR_ERR(group);
		goto err_unlink;
	}
	dev->iommu_group = group;

	dev->iommu->max_pasids = dev_iommu_get_max_pasids(dev);
	if (ops->is_attach_deferred)
		dev->iommu->attach_deferred = ops->is_attach_deferred(dev);
	return 0;

err_unlink:
	iommu_device_unlink(iommu_dev, dev);
err_release:
	if (ops->release_device)
		ops->release_device(dev);
err_module_put:
	module_put(ops->owner);
err_free:
	dev->iommu->iommu_dev = NULL;
	dev_iommu_free(dev);
	return ret;
}

static void iommu_deinit_device(struct device *dev)
{
	struct iommu_group *group = dev->iommu_group;
	const struct iommu_ops *ops = dev_iommu_ops(dev);

	lockdep_assert_held(&group->mutex);

	iommu_device_unlink(dev->iommu->iommu_dev, dev);

	/*
	 * release_device() must stop using any attached domain on the device.
	 * If there are still other devices in the group, they are not affected
	 * by this callback.
	 *
	 * If the iommu driver provides release_domain, the core code ensures
	 * that domain is attached prior to calling release_device. Drivers can
	 * use this to enforce a translation on the idle iommu. Typically, the
	 * global static blocked_domain is a good choice.
	 *
	 * Otherwise, the iommu driver must set the device to either an identity
	 * or a blocking translation in release_device() and stop using any
	 * domain pointer, as it is going to be freed.
	 *
	 * Regardless, if a delayed attach never occurred, then the release
	 * should still avoid touching any hardware configuration either.
	 */
	if (!dev->iommu->attach_deferred && ops->release_domain)
		ops->release_domain->ops->attach_dev(ops->release_domain, dev);

	if (ops->release_device)
		ops->release_device(dev);

	/*
	 * If this is the last driver to use the group then we must free the
	 * domains before we do the module_put().
	 */
	if (list_empty(&group->devices)) {
		if (group->default_domain) {
			iommu_domain_free(group->default_domain);
			group->default_domain = NULL;
		}
		if (group->blocking_domain) {
			iommu_domain_free(group->blocking_domain);
			group->blocking_domain = NULL;
		}
		group->domain = NULL;
	}

	/* Caller must put iommu_group */
	dev->iommu_group = NULL;
	module_put(ops->owner);
	dev_iommu_free(dev);
#ifdef CONFIG_IOMMU_DMA
	dev->dma_iommu = false;
#endif
}

static struct iommu_domain *pasid_array_entry_to_domain(void *entry)
{
	if (xa_pointer_tag(entry) == IOMMU_PASID_ARRAY_DOMAIN)
		return xa_untag_pointer(entry);
	return ((struct iommu_attach_handle *)xa_untag_pointer(entry))->domain;
}

DEFINE_MUTEX(iommu_probe_device_lock);

static int __iommu_probe_device(struct device *dev, struct list_head *group_list)
{
	struct iommu_group *group;
	struct group_device *gdev;
	int ret;

	/*
	 * Serialise to avoid races between IOMMU drivers registering in
	 * parallel and/or the "replay" calls from ACPI/OF code via client
	 * driver probe. Once the latter have been cleaned up we should
	 * probably be able to use device_lock() here to minimise the scope,
	 * but for now enforcing a simple global ordering is fine.
	 */
	lockdep_assert_held(&iommu_probe_device_lock);

	/* Device is probed already if in a group */
	if (dev->iommu_group)
		return 0;

	ret = iommu_init_device(dev);
	if (ret)
		return ret;
	/*
	 * And if we do now see any replay calls, they would indicate someone
	 * misusing the dma_configure path outside bus code.
	 */
	if (dev->driver)
		dev_WARN(dev, "late IOMMU probe at driver bind, something fishy here!\n");

	group = dev->iommu_group;
	gdev = iommu_group_alloc_device(group, dev);
	mutex_lock(&group->mutex);
	if (IS_ERR(gdev)) {
		ret = PTR_ERR(gdev);
		goto err_put_group;
	}

	/*
	 * The gdev must be in the list before calling
	 * iommu_setup_default_domain()
	 */
	list_add_tail(&gdev->list, &group->devices);
	WARN_ON(group->default_domain && !group->domain);
	if (group->default_domain)
		iommu_create_device_direct_mappings(group->default_domain, dev);
	if (group->domain) {
		ret = __iommu_device_set_domain(group, dev, group->domain, 0);
		if (ret)
			goto err_remove_gdev;
	} else if (!group->default_domain && !group_list) {
		ret = iommu_setup_default_domain(group, 0);
		if (ret)
			goto err_remove_gdev;
	} else if (!group->default_domain) {
		/*
		 * With a group_list argument we defer the default_domain setup
		 * to the caller by providing a de-duplicated list of groups
		 * that need further setup.
		 */
		if (list_empty(&group->entry))
			list_add_tail(&group->entry, group_list);
	}

	if (group->default_domain)
		iommu_setup_dma_ops(dev);

	mutex_unlock(&group->mutex);

	return 0;

err_remove_gdev:
	list_del(&gdev->list);
	__iommu_group_free_device(group, gdev);
err_put_group:
	iommu_deinit_device(dev);
	mutex_unlock(&group->mutex);
	iommu_group_put(group);

	return ret;
}

int iommu_probe_device(struct device *dev)
{
	const struct iommu_ops *ops;
	int ret;

	mutex_lock(&iommu_probe_device_lock);
	ret = __iommu_probe_device(dev, NULL);
	mutex_unlock(&iommu_probe_device_lock);
	if (ret)
		return ret;

	ops = dev_iommu_ops(dev);
	if (ops->probe_finalize)
		ops->probe_finalize(dev);

	return 0;
}

static void __iommu_group_free_device(struct iommu_group *group,
				      struct group_device *grp_dev)
{
	struct device *dev = grp_dev->dev;

	sysfs_remove_link(group->devices_kobj, grp_dev->name);
	sysfs_remove_link(&dev->kobj, "iommu_group");

	trace_remove_device_from_group(group->id, dev);

	/*
	 * If the group has become empty then ownership must have been
	 * released, and the current domain must be set back to NULL or
	 * the default domain.
	 */
	if (list_empty(&group->devices))
		WARN_ON(group->owner_cnt ||
			group->domain != group->default_domain);

	kfree(grp_dev->name);
	kfree(grp_dev);
}

/* Remove the iommu_group from the struct device. */
static void __iommu_group_remove_device(struct device *dev)
{
	struct iommu_group *group = dev->iommu_group;
	struct group_device *device;

	mutex_lock(&group->mutex);
	for_each_group_device(group, device) {
		if (device->dev != dev)
			continue;

		list_del(&device->list);
		__iommu_group_free_device(group, device);
		if (dev_has_iommu(dev))
			iommu_deinit_device(dev);
		else
			dev->iommu_group = NULL;
		break;
	}
	mutex_unlock(&group->mutex);

	/*
	 * Pairs with the get in iommu_init_device() or
	 * iommu_group_add_device()
	 */
	iommu_group_put(group);
}

static void iommu_release_device(struct device *dev)
{
	struct iommu_group *group = dev->iommu_group;

	if (group)
		__iommu_group_remove_device(dev);

	/* Free any fwspec if no iommu_driver was ever attached */
	if (dev->iommu)
		dev_iommu_free(dev);
}

static int __init iommu_set_def_domain_type(char *str)
{
	bool pt;
	int ret;

	ret = kstrtobool(str, &pt);
	if (ret)
		return ret;

	if (pt)
		iommu_set_default_passthrough(true);
	else
		iommu_set_default_translated(true);

	return 0;
}
early_param("iommu.passthrough", iommu_set_def_domain_type);

static int __init iommu_dma_setup(char *str)
{
	int ret = kstrtobool(str, &iommu_dma_strict);

	if (!ret)
		iommu_cmd_line |= IOMMU_CMD_LINE_STRICT;
	return ret;
}
early_param("iommu.strict", iommu_dma_setup);

void iommu_set_dma_strict(void)
{
	iommu_dma_strict = true;
	if (iommu_def_domain_type == IOMMU_DOMAIN_DMA_FQ)
		iommu_def_domain_type = IOMMU_DOMAIN_DMA;
}

static ssize_t iommu_group_attr_show(struct kobject *kobj,
				     struct attribute *__attr, char *buf)
{
	struct iommu_group_attribute *attr = to_iommu_group_attr(__attr);
	struct iommu_group *group = to_iommu_group(kobj);
	ssize_t ret = -EIO;

	if (attr->show)
		ret = attr->show(group, buf);
	return ret;
}

static ssize_t iommu_group_attr_store(struct kobject *kobj,
				      struct attribute *__attr,
				      const char *buf, size_t count)
{
	struct iommu_group_attribute *attr = to_iommu_group_attr(__attr);
	struct iommu_group *group = to_iommu_group(kobj);
	ssize_t ret = -EIO;

	if (attr->store)
		ret = attr->store(group, buf, count);
	return ret;
}

static const struct sysfs_ops iommu_group_sysfs_ops = {
	.show = iommu_group_attr_show,
	.store = iommu_group_attr_store,
};

static int iommu_group_create_file(struct iommu_group *group,
				   struct iommu_group_attribute *attr)
{
	return sysfs_create_file(&group->kobj, &attr->attr);
}

static void iommu_group_remove_file(struct iommu_group *group,
				    struct iommu_group_attribute *attr)
{
	sysfs_remove_file(&group->kobj, &attr->attr);
}

static ssize_t iommu_group_show_name(struct iommu_group *group, char *buf)
{
	return sysfs_emit(buf, "%s\n", group->name);
}

/**
 * iommu_insert_resv_region - Insert a new region in the
 * list of reserved regions.
 * @new: new region to insert
 * @regions: list of regions
 *
 * Elements are sorted by start address and overlapping segments
 * of the same type are merged.
 */
static int iommu_insert_resv_region(struct iommu_resv_region *new,
				    struct list_head *regions)
{
	struct iommu_resv_region *iter, *tmp, *nr, *top;
	LIST_HEAD(stack);

	nr = iommu_alloc_resv_region(new->start, new->length,
				     new->prot, new->type, GFP_KERNEL);
	if (!nr)
		return -ENOMEM;

	/* First add the new element based on start address sorting */
	list_for_each_entry(iter, regions, list) {
		if (nr->start < iter->start ||
		    (nr->start == iter->start && nr->type <= iter->type))
			break;
	}
	list_add_tail(&nr->list, &iter->list);

	/* Merge overlapping segments of type nr->type in @regions, if any */
	list_for_each_entry_safe(iter, tmp, regions, list) {
		phys_addr_t top_end, iter_end = iter->start + iter->length - 1;

		/* no merge needed on elements of different types than @new */
		if (iter->type != new->type) {
			list_move_tail(&iter->list, &stack);
			continue;
		}

		/* look for the last stack element of same type as @iter */
		list_for_each_entry_reverse(top, &stack, list)
			if (top->type == iter->type)
				goto check_overlap;

		list_move_tail(&iter->list, &stack);
		continue;

check_overlap:
		top_end = top->start + top->length - 1;

		if (iter->start > top_end + 1) {
			list_move_tail(&iter->list, &stack);
		} else {
			top->length = max(top_end, iter_end) - top->start + 1;
			list_del(&iter->list);
			kfree(iter);
		}
	}
	list_splice(&stack, regions);
	return 0;
}

static int
iommu_insert_device_resv_regions(struct list_head *dev_resv_regions,
				 struct list_head *group_resv_regions)
{
	struct iommu_resv_region *entry;
	int ret = 0;

	list_for_each_entry(entry, dev_resv_regions, list) {
		ret = iommu_insert_resv_region(entry, group_resv_regions);
		if (ret)
			break;
	}
	return ret;
}

int iommu_get_group_resv_regions(struct iommu_group *group,
				 struct list_head *head)
{
	struct group_device *device;
	int ret = 0;

	mutex_lock(&group->mutex);
	for_each_group_device(group, device) {
		struct list_head dev_resv_regions;

		/*
		 * Non-API groups still expose reserved_regions in sysfs,
		 * so filter out calls that get here that way.
		 */
		if (!dev_has_iommu(device->dev))
			break;

		INIT_LIST_HEAD(&dev_resv_regions);
		iommu_get_resv_regions(device->dev, &dev_resv_regions);
		ret = iommu_insert_device_resv_regions(&dev_resv_regions, head);
		iommu_put_resv_regions(device->dev, &dev_resv_regions);
		if (ret)
			break;
	}
	mutex_unlock(&group->mutex);
	return ret;
}
EXPORT_SYMBOL_GPL(iommu_get_group_resv_regions);

static ssize_t iommu_group_show_resv_regions(struct iommu_group *group,
					     char *buf)
{
	struct iommu_resv_region *region, *next;
	struct list_head group_resv_regions;
	int offset = 0;

	INIT_LIST_HEAD(&group_resv_regions);
	iommu_get_group_resv_regions(group, &group_resv_regions);

	list_for_each_entry_safe(region, next, &group_resv_regions, list) {
		offset += sysfs_emit_at(buf, offset, "0x%016llx 0x%016llx %s\n",
					(long long)region->start,
					(long long)(region->start +
						    region->length - 1),
					iommu_group_resv_type_string[region->type]);
		kfree(region);
	}

	return offset;
}

static ssize_t iommu_group_show_type(struct iommu_group *group,
				     char *buf)
{
	char *type = "unknown";

	mutex_lock(&group->mutex);
	if (group->default_domain) {
		switch (group->default_domain->type) {
		case IOMMU_DOMAIN_BLOCKED:
			type = "blocked";
			break;
		case IOMMU_DOMAIN_IDENTITY:
			type = "identity";
			break;
		case IOMMU_DOMAIN_UNMANAGED:
			type = "unmanaged";
			break;
		case IOMMU_DOMAIN_DMA:
			type = "DMA";
			break;
		case IOMMU_DOMAIN_DMA_FQ:
			type = "DMA-FQ";
			break;
		}
	}
	mutex_unlock(&group->mutex);

	return sysfs_emit(buf, "%s\n", type);
}

static IOMMU_GROUP_ATTR(name, S_IRUGO, iommu_group_show_name, NULL);

static IOMMU_GROUP_ATTR(reserved_regions, 0444,
			iommu_group_show_resv_regions, NULL);

static IOMMU_GROUP_ATTR(type, 0644, iommu_group_show_type,
			iommu_group_store_type);

static void iommu_group_release(struct kobject *kobj)
{
	struct iommu_group *group = to_iommu_group(kobj);

	pr_debug("Releasing group %d\n", group->id);

	if (group->iommu_data_release)
		group->iommu_data_release(group->iommu_data);

	ida_free(&iommu_group_ida, group->id);

	/* Domains are free'd by iommu_deinit_device() */
	WARN_ON(group->default_domain);
	WARN_ON(group->blocking_domain);

	kfree(group->name);
	kfree(group);
}

static const struct kobj_type iommu_group_ktype = {
	.sysfs_ops = &iommu_group_sysfs_ops,
	.release = iommu_group_release,
};

/**
 * iommu_group_alloc - Allocate a new group
 *
 * This function is called by an iommu driver to allocate a new iommu
 * group.  The iommu group represents the minimum granularity of the iommu.
 * Upon successful return, the caller holds a reference to the supplied
 * group in order to hold the group until devices are added.  Use
 * iommu_group_put() to release this extra reference count, allowing the
 * group to be automatically reclaimed once it has no devices or external
 * references.
 */
struct iommu_group *iommu_group_alloc(void)
{
	struct iommu_group *group;
	int ret;

	group = kzalloc(sizeof(*group), GFP_KERNEL);
	if (!group)
		return ERR_PTR(-ENOMEM);

	group->kobj.kset = iommu_group_kset;
	mutex_init(&group->mutex);
	INIT_LIST_HEAD(&group->devices);
	INIT_LIST_HEAD(&group->entry);
	xa_init(&group->pasid_array);

	ret = ida_alloc(&iommu_group_ida, GFP_KERNEL);
	if (ret < 0) {
		kfree(group);
		return ERR_PTR(ret);
	}
	group->id = ret;

	ret = kobject_init_and_add(&group->kobj, &iommu_group_ktype,
				   NULL, "%d", group->id);
	if (ret) {
		kobject_put(&group->kobj);
		return ERR_PTR(ret);
	}

	group->devices_kobj = kobject_create_and_add("devices", &group->kobj);
	if (!group->devices_kobj) {
		kobject_put(&group->kobj); /* triggers .release & free */
		return ERR_PTR(-ENOMEM);
	}

	/*
	 * The devices_kobj holds a reference on the group kobject, so
	 * as long as that exists so will the group.  We can therefore
	 * use the devices_kobj for reference counting.
	 */
	kobject_put(&group->kobj);

	ret = iommu_group_create_file(group,
				      &iommu_group_attr_reserved_regions);
	if (ret) {
		kobject_put(group->devices_kobj);
		return ERR_PTR(ret);
	}

	ret = iommu_group_create_file(group, &iommu_group_attr_type);
	if (ret) {
		kobject_put(group->devices_kobj);
		return ERR_PTR(ret);
	}

	pr_debug("Allocated group %d\n", group->id);

	return group;
}
EXPORT_SYMBOL_GPL(iommu_group_alloc);

/**
 * iommu_group_get_iommudata - retrieve iommu_data registered for a group
 * @group: the group
 *
 * iommu drivers can store data in the group for use when doing iommu
 * operations.  This function provides a way to retrieve it.  Caller
 * should hold a group reference.
 */
void *iommu_group_get_iommudata(struct iommu_group *group)
{
	return group->iommu_data;
}
EXPORT_SYMBOL_GPL(iommu_group_get_iommudata);

/**
 * iommu_group_set_iommudata - set iommu_data for a group
 * @group: the group
 * @iommu_data: new data
 * @release: release function for iommu_data
 *
 * iommu drivers can store data in the group for use when doing iommu
 * operations.  This function provides a way to set the data after
 * the group has been allocated.  Caller should hold a group reference.
 */
void iommu_group_set_iommudata(struct iommu_group *group, void *iommu_data,
			       void (*release)(void *iommu_data))
{
	group->iommu_data = iommu_data;
	group->iommu_data_release = release;
}
EXPORT_SYMBOL_GPL(iommu_group_set_iommudata);

/**
 * iommu_group_set_name - set name for a group
 * @group: the group
 * @name: name
 *
 * Allow iommu driver to set a name for a group.  When set it will
 * appear in a name attribute file under the group in sysfs.
 */
int iommu_group_set_name(struct iommu_group *group, const char *name)
{
	int ret;

	if (group->name) {
		iommu_group_remove_file(group, &iommu_group_attr_name);
		kfree(group->name);
		group->name = NULL;
		if (!name)
			return 0;
	}

	group->name = kstrdup(name, GFP_KERNEL);
	if (!group->name)
		return -ENOMEM;

	ret = iommu_group_create_file(group, &iommu_group_attr_name);
	if (ret) {
		kfree(group->name);
		group->name = NULL;
		return ret;
	}

	return 0;
}
EXPORT_SYMBOL_GPL(iommu_group_set_name);

static int iommu_create_device_direct_mappings(struct iommu_domain *domain,
					       struct device *dev)
{
	struct iommu_resv_region *entry;
	struct list_head mappings;
	unsigned long pg_size;
	int ret = 0;

	pg_size = domain->pgsize_bitmap ? 1UL << __ffs(domain->pgsize_bitmap) : 0;
	INIT_LIST_HEAD(&mappings);

	if (WARN_ON_ONCE(iommu_is_dma_domain(domain) && !pg_size))
		return -EINVAL;

	iommu_get_resv_regions(dev, &mappings);

	/* We need to consider overlapping regions for different devices */
	list_for_each_entry(entry, &mappings, list) {
		dma_addr_t start, end, addr;
		size_t map_size = 0;

		if (entry->type == IOMMU_RESV_DIRECT)
			dev->iommu->require_direct = 1;

		if ((entry->type != IOMMU_RESV_DIRECT &&
		     entry->type != IOMMU_RESV_DIRECT_RELAXABLE) ||
		    !iommu_is_dma_domain(domain))
			continue;

		start = ALIGN(entry->start, pg_size);
		end   = ALIGN(entry->start + entry->length, pg_size);

		for (addr = start; addr <= end; addr += pg_size) {
			phys_addr_t phys_addr;

			if (addr == end)
				goto map_end;

			phys_addr = iommu_iova_to_phys(domain, addr);
			if (!phys_addr) {
				map_size += pg_size;
				continue;
			}

map_end:
			if (map_size) {
				ret = iommu_map(domain, addr - map_size,
						addr - map_size, map_size,
						entry->prot, GFP_KERNEL);
				if (ret)
					goto out;
				map_size = 0;
			}
		}

	}
out:
	iommu_put_resv_regions(dev, &mappings);

	return ret;
}

/* This is undone by __iommu_group_free_device() */
static struct group_device *iommu_group_alloc_device(struct iommu_group *group,
						     struct device *dev)
{
	int ret, i = 0;
	struct group_device *device;

	device = kzalloc(sizeof(*device), GFP_KERNEL);
	if (!device)
		return ERR_PTR(-ENOMEM);

	device->dev = dev;

	ret = sysfs_create_link(&dev->kobj, &group->kobj, "iommu_group");
	if (ret)
		goto err_free_device;

	device->name = kasprintf(GFP_KERNEL, "%s", kobject_name(&dev->kobj));
rename:
	if (!device->name) {
		ret = -ENOMEM;
		goto err_remove_link;
	}

	ret = sysfs_create_link_nowarn(group->devices_kobj,
				       &dev->kobj, device->name);
	if (ret) {
		if (ret == -EEXIST && i >= 0) {
			/*
			 * Account for the slim chance of collision
			 * and append an instance to the name.
			 */
			kfree(device->name);
			device->name = kasprintf(GFP_KERNEL, "%s.%d",
						 kobject_name(&dev->kobj), i++);
			goto rename;
		}
		goto err_free_name;
	}

	trace_add_device_to_group(group->id, dev);

	dev_info(dev, "Adding to iommu group %d\n", group->id);

	return device;

err_free_name:
	kfree(device->name);
err_remove_link:
	sysfs_remove_link(&dev->kobj, "iommu_group");
err_free_device:
	kfree(device);
	dev_err(dev, "Failed to add to iommu group %d: %d\n", group->id, ret);
	return ERR_PTR(ret);
}

/**
 * iommu_group_add_device - add a device to an iommu group
 * @group: the group into which to add the device (reference should be held)
 * @dev: the device
 *
 * This function is called by an iommu driver to add a device into a
 * group.  Adding a device increments the group reference count.
 */
int iommu_group_add_device(struct iommu_group *group, struct device *dev)
{
	struct group_device *gdev;

	gdev = iommu_group_alloc_device(group, dev);
	if (IS_ERR(gdev))
		return PTR_ERR(gdev);

	iommu_group_ref_get(group);
	dev->iommu_group = group;

	mutex_lock(&group->mutex);
	list_add_tail(&gdev->list, &group->devices);
	mutex_unlock(&group->mutex);
	return 0;
}
EXPORT_SYMBOL_GPL(iommu_group_add_device);

/**
 * iommu_group_remove_device - remove a device from it's current group
 * @dev: device to be removed
 *
 * This function is called by an iommu driver to remove the device from
 * it's current group.  This decrements the iommu group reference count.
 */
void iommu_group_remove_device(struct device *dev)
{
	struct iommu_group *group = dev->iommu_group;

	if (!group)
		return;

	dev_info(dev, "Removing from iommu group %d\n", group->id);

	__iommu_group_remove_device(dev);
}
EXPORT_SYMBOL_GPL(iommu_group_remove_device);

#if IS_ENABLED(CONFIG_LOCKDEP) && IS_ENABLED(CONFIG_IOMMU_API)
/**
 * iommu_group_mutex_assert - Check device group mutex lock
 * @dev: the device that has group param set
 *
 * This function is called by an iommu driver to check whether it holds
 * group mutex lock for the given device or not.
 *
 * Note that this function must be called after device group param is set.
 */
void iommu_group_mutex_assert(struct device *dev)
{
	struct iommu_group *group = dev->iommu_group;

	lockdep_assert_held(&group->mutex);
}
EXPORT_SYMBOL_GPL(iommu_group_mutex_assert);
#endif

static struct device *iommu_group_first_dev(struct iommu_group *group)
{
	lockdep_assert_held(&group->mutex);
	return list_first_entry(&group->devices, struct group_device, list)->dev;
}

/**
 * iommu_group_for_each_dev - iterate over each device in the group
 * @group: the group
 * @data: caller opaque data to be passed to callback function
 * @fn: caller supplied callback function
 *
 * This function is called by group users to iterate over group devices.
 * Callers should hold a reference count to the group during callback.
 * The group->mutex is held across callbacks, which will block calls to
 * iommu_group_add/remove_device.
 */
int iommu_group_for_each_dev(struct iommu_group *group, void *data,
			     int (*fn)(struct device *, void *))
{
	struct group_device *device;
	int ret = 0;

	mutex_lock(&group->mutex);
	for_each_group_device(group, device) {
		ret = fn(device->dev, data);
		if (ret)
			break;
	}
	mutex_unlock(&group->mutex);

	return ret;
}
EXPORT_SYMBOL_GPL(iommu_group_for_each_dev);

/**
 * iommu_group_get - Return the group for a device and increment reference
 * @dev: get the group that this device belongs to
 *
 * This function is called by iommu drivers and users to get the group
 * for the specified device.  If found, the group is returned and the group
 * reference in incremented, else NULL.
 */
struct iommu_group *iommu_group_get(struct device *dev)
{
	struct iommu_group *group = dev->iommu_group;

	if (group)
		kobject_get(group->devices_kobj);

	return group;
}
EXPORT_SYMBOL_GPL(iommu_group_get);

/**
 * iommu_group_ref_get - Increment reference on a group
 * @group: the group to use, must not be NULL
 *
 * This function is called by iommu drivers to take additional references on an
 * existing group.  Returns the given group for convenience.
 */
struct iommu_group *iommu_group_ref_get(struct iommu_group *group)
{
	kobject_get(group->devices_kobj);
	return group;
}
EXPORT_SYMBOL_GPL(iommu_group_ref_get);

/**
 * iommu_group_put - Decrement group reference
 * @group: the group to use
 *
 * This function is called by iommu drivers and users to release the
 * iommu group.  Once the reference count is zero, the group is released.
 */
void iommu_group_put(struct iommu_group *group)
{
	if (group)
		kobject_put(group->devices_kobj);
}
EXPORT_SYMBOL_GPL(iommu_group_put);

/**
 * iommu_group_id - Return ID for a group
 * @group: the group to ID
 *
 * Return the unique ID for the group matching the sysfs group number.
 */
int iommu_group_id(struct iommu_group *group)
{
	return group->id;
}
EXPORT_SYMBOL_GPL(iommu_group_id);

static struct iommu_group *get_pci_alias_group(struct pci_dev *pdev,
					       unsigned long *devfns);

/*
 * To consider a PCI device isolated, we require ACS to support Source
 * Validation, Request Redirection, Completer Redirection, and Upstream
 * Forwarding.  This effectively means that devices cannot spoof their
 * requester ID, requests and completions cannot be redirected, and all
 * transactions are forwarded upstream, even as it passes through a
 * bridge where the target device is downstream.
 */
#define REQ_ACS_FLAGS   (PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF)

/*
 * For multifunction devices which are not isolated from each other, find
 * all the other non-isolated functions and look for existing groups.  For
 * each function, we also need to look for aliases to or from other devices
 * that may already have a group.
 */
static struct iommu_group *get_pci_function_alias_group(struct pci_dev *pdev,
							unsigned long *devfns)
{
	struct pci_dev *tmp = NULL;
	struct iommu_group *group;

	if (!pdev->multifunction || pci_acs_enabled(pdev, REQ_ACS_FLAGS))
		return NULL;

	for_each_pci_dev(tmp) {
		if (tmp == pdev || tmp->bus != pdev->bus ||
		    PCI_SLOT(tmp->devfn) != PCI_SLOT(pdev->devfn) ||
		    pci_acs_enabled(tmp, REQ_ACS_FLAGS))
			continue;

		group = get_pci_alias_group(tmp, devfns);
		if (group) {
			pci_dev_put(tmp);
			return group;
		}
	}

	return NULL;
}

/*
 * Look for aliases to or from the given device for existing groups. DMA
 * aliases are only supported on the same bus, therefore the search
 * space is quite small (especially since we're really only looking at pcie
 * device, and therefore only expect multiple slots on the root complex or
 * downstream switch ports).  It's conceivable though that a pair of
 * multifunction devices could have aliases between them that would cause a
 * loop.  To prevent this, we use a bitmap to track where we've been.
 */
static struct iommu_group *get_pci_alias_group(struct pci_dev *pdev,
					       unsigned long *devfns)
{
	struct pci_dev *tmp = NULL;
	struct iommu_group *group;

	if (test_and_set_bit(pdev->devfn & 0xff, devfns))
		return NULL;

	group = iommu_group_get(&pdev->dev);
	if (group)
		return group;

	for_each_pci_dev(tmp) {
		if (tmp == pdev || tmp->bus != pdev->bus)
			continue;

		/* We alias them or they alias us */
		if (pci_devs_are_dma_aliases(pdev, tmp)) {
			group = get_pci_alias_group(tmp, devfns);
			if (group) {
				pci_dev_put(tmp);
				return group;
			}

			group = get_pci_function_alias_group(tmp, devfns);
			if (group) {
				pci_dev_put(tmp);
				return group;
			}
		}
	}

	return NULL;
}

struct group_for_pci_data {
	struct pci_dev *pdev;
	struct iommu_group *group;
};

/*
 * DMA alias iterator callback, return the last seen device.  Stop and return
 * the IOMMU group if we find one along the way.
 */
static int get_pci_alias_or_group(struct pci_dev *pdev, u16 alias, void *opaque)
{
	struct group_for_pci_data *data = opaque;

	data->pdev = pdev;
	data->group = iommu_group_get(&pdev->dev);

	return data->group != NULL;
}

/*
 * Generic device_group call-back function. It just allocates one
 * iommu-group per device.
 */
struct iommu_group *generic_device_group(struct device *dev)
{
	return iommu_group_alloc();
}
EXPORT_SYMBOL_GPL(generic_device_group);

/*
 * Generic device_group call-back function. It just allocates one
 * iommu-group per iommu driver instance shared by every device
 * probed by that iommu driver.
 */
struct iommu_group *generic_single_device_group(struct device *dev)
{
	struct iommu_device *iommu = dev->iommu->iommu_dev;

	if (!iommu->singleton_group) {
		struct iommu_group *group;

		group = iommu_group_alloc();
		if (IS_ERR(group))
			return group;
		iommu->singleton_group = group;
	}
	return iommu_group_ref_get(iommu->singleton_group);
}
EXPORT_SYMBOL_GPL(generic_single_device_group);

/*
 * Use standard PCI bus topology, isolation features, and DMA alias quirks
 * to find or create an IOMMU group for a device.
 */
struct iommu_group *pci_device_group(struct device *dev)
{
	struct pci_dev *pdev = to_pci_dev(dev);
	struct group_for_pci_data data;
	struct pci_bus *bus;
	struct iommu_group *group = NULL;
	u64 devfns[4] = { 0 };

	if (WARN_ON(!dev_is_pci(dev)))
		return ERR_PTR(-EINVAL);

	/*
	 * Find the upstream DMA alias for the device.  A device must not
	 * be aliased due to topology in order to have its own IOMMU group.
	 * If we find an alias along the way that already belongs to a
	 * group, use it.
	 */
	if (pci_for_each_dma_alias(pdev, get_pci_alias_or_group, &data))
		return data.group;

	pdev = data.pdev;

	/*
	 * Continue upstream from the point of minimum IOMMU granularity
	 * due to aliases to the point where devices are protected from
	 * peer-to-peer DMA by PCI ACS.  Again, if we find an existing
	 * group, use it.
	 */
	for (bus = pdev->bus; !pci_is_root_bus(bus); bus = bus->parent) {
		if (!bus->self)
			continue;

		if (pci_acs_path_enabled(bus->self, NULL, REQ_ACS_FLAGS))
			break;

		pdev = bus->self;

		group = iommu_group_get(&pdev->dev);
		if (group)
			return group;
	}

	/*
	 * Look for existing groups on device aliases.  If we alias another
	 * device or another device aliases us, use the same group.
	 */
	group = get_pci_alias_group(pdev, (unsigned long *)devfns);
	if (group)
		return group;

	/*
	 * Look for existing groups on non-isolated functions on the same
	 * slot and aliases of those funcions, if any.  No need to clear
	 * the search bitmap, the tested devfns are still valid.
	 */
	group = get_pci_function_alias_group(pdev, (unsigned long *)devfns);
	if (group)
		return group;

	/* No shared group found, allocate new */
	return iommu_group_alloc();
}
EXPORT_SYMBOL_GPL(pci_device_group);

/* Get the IOMMU group for device on fsl-mc bus */
struct iommu_group *fsl_mc_device_group(struct device *dev)
{
	struct device *cont_dev = fsl_mc_cont_dev(dev);
	struct iommu_group *group;

	group = iommu_group_get(cont_dev);
	if (!group)
		group = iommu_group_alloc();
	return group;
}
EXPORT_SYMBOL_GPL(fsl_mc_device_group);

static struct iommu_domain *__iommu_alloc_identity_domain(struct device *dev)
{
	const struct iommu_ops *ops = dev_iommu_ops(dev);
	struct iommu_domain *domain;

	if (ops->identity_domain)
		return ops->identity_domain;

	if (ops->domain_alloc_identity) {
		domain = ops->domain_alloc_identity(dev);
		if (IS_ERR(domain))
			return domain;
	} else {
		return ERR_PTR(-EOPNOTSUPP);
	}

	iommu_domain_init(domain, IOMMU_DOMAIN_IDENTITY, ops);
	return domain;
}

static struct iommu_domain *
__iommu_group_alloc_default_domain(struct iommu_group *group, int req_type)
{
	struct device *dev = iommu_group_first_dev(group);
	struct iommu_domain *dom;

	if (group->default_domain && group->default_domain->type == req_type)
		return group->default_domain;

	/*
	 * When allocating the DMA API domain assume that the driver is going to
	 * use PASID and make sure the RID's domain is PASID compatible.
	 */
	if (req_type & __IOMMU_DOMAIN_PAGING) {
		dom = __iommu_paging_domain_alloc_flags(dev, req_type,
			   dev->iommu->max_pasids ? IOMMU_HWPT_ALLOC_PASID : 0);

		/*
		 * If driver does not support PASID feature then
		 * try to allocate non-PASID domain
		 */
		if (PTR_ERR(dom) == -EOPNOTSUPP)
			dom = __iommu_paging_domain_alloc_flags(dev, req_type, 0);

		return dom;
	}

	if (req_type == IOMMU_DOMAIN_IDENTITY)
		return __iommu_alloc_identity_domain(dev);

	return ERR_PTR(-EINVAL);
}

/*
 * req_type of 0 means "auto" which means to select a domain based on
 * iommu_def_domain_type or what the driver actually supports.
 */
static struct iommu_domain *
iommu_group_alloc_default_domain(struct iommu_group *group, int req_type)
{
	const struct iommu_ops *ops = dev_iommu_ops(iommu_group_first_dev(group));
	struct iommu_domain *dom;

	lockdep_assert_held(&group->mutex);

	/*
	 * Allow legacy drivers to specify the domain that will be the default
	 * domain. This should always be either an IDENTITY/BLOCKED/PLATFORM
	 * domain. Do not use in new drivers.
	 */
	if (ops->default_domain) {
		if (req_type != ops->default_domain->type)
			return ERR_PTR(-EINVAL);
		return ops->default_domain;
	}

	if (req_type)
		return __iommu_group_alloc_default_domain(group, req_type);

	/* The driver gave no guidance on what type to use, try the default */
	dom = __iommu_group_alloc_default_domain(group, iommu_def_domain_type);
	if (!IS_ERR(dom))
		return dom;

	/* Otherwise IDENTITY and DMA_FQ defaults will try DMA */
	if (iommu_def_domain_type == IOMMU_DOMAIN_DMA)
		return ERR_PTR(-EINVAL);
	dom = __iommu_group_alloc_default_domain(group, IOMMU_DOMAIN_DMA);
	if (IS_ERR(dom))
		return dom;

	pr_warn("Failed to allocate default IOMMU domain of type %u for group %s - Falling back to IOMMU_DOMAIN_DMA",
		iommu_def_domain_type, group->name);
	return dom;
}

struct iommu_domain *iommu_group_default_domain(struct iommu_group *group)
{
	return group->default_domain;
}

static int probe_iommu_group(struct device *dev, void *data)
{
	struct list_head *group_list = data;
	int ret;

	mutex_lock(&iommu_probe_device_lock);
	ret = __iommu_probe_device(dev, group_list);
	mutex_unlock(&iommu_probe_device_lock);
	if (ret == -ENODEV)
		ret = 0;

	return ret;
}

static int iommu_bus_notifier(struct notifier_block *nb,
			      unsigned long action, void *data)
{
	struct device *dev = data;

	if (action == BUS_NOTIFY_ADD_DEVICE) {
		int ret;

		ret = iommu_probe_device(dev);
		return (ret) ? NOTIFY_DONE : NOTIFY_OK;
	} else if (action == BUS_NOTIFY_REMOVED_DEVICE) {
		iommu_release_device(dev);
		return NOTIFY_OK;
	}

	return 0;
}

/*
 * Combine the driver's chosen def_domain_type across all the devices in a
 * group. Drivers must give a consistent result.
 */
static int iommu_get_def_domain_type(struct iommu_group *group,
				     struct device *dev, int cur_type)
{
	const struct iommu_ops *ops = dev_iommu_ops(dev);
	int type;

	if (ops->default_domain) {
		/*
		 * Drivers that declare a global static default_domain will
		 * always choose that.
		 */
		type = ops->default_domain->type;
	} else {
		if (ops->def_domain_type)
			type = ops->def_domain_type(dev);
		else
			return cur_type;
	}
	if (!type || cur_type == type)
		return cur_type;
	if (!cur_type)
		return type;

	dev_err_ratelimited(
		dev,
		"IOMMU driver error, requesting conflicting def_domain_type, %s and %s, for devices in group %u.\n",
		iommu_domain_type_str(cur_type), iommu_domain_type_str(type),
		group->id);

	/*
	 * Try to recover, drivers are allowed to force IDENTITY or DMA, IDENTITY
	 * takes precedence.
	 */
	if (type == IOMMU_DOMAIN_IDENTITY)
		return type;
	return cur_type;
}

/*
 * A target_type of 0 will select the best domain type. 0 can be returned in
 * this case meaning the global default should be used.
 */
static int iommu_get_default_domain_type(struct iommu_group *group,
					 int target_type)
{
	struct device *untrusted = NULL;
	struct group_device *gdev;
	int driver_type = 0;

	lockdep_assert_held(&group->mutex);

	/*
	 * ARM32 drivers supporting CONFIG_ARM_DMA_USE_IOMMU can declare an
	 * identity_domain and it will automatically become their default
	 * domain. Later on ARM_DMA_USE_IOMMU will install its UNMANAGED domain.
	 * Override the selection to IDENTITY.
	 */
	if (IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU)) {
		static_assert(!(IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU) &&
				IS_ENABLED(CONFIG_IOMMU_DMA)));
		driver_type = IOMMU_DOMAIN_IDENTITY;
	}

	for_each_group_device(group, gdev) {
		driver_type = iommu_get_def_domain_type(group, gdev->dev,
							driver_type);

		if (dev_is_pci(gdev->dev) && to_pci_dev(gdev->dev)->untrusted) {
			/*
			 * No ARM32 using systems will set untrusted, it cannot
			 * work.
			 */
			if (WARN_ON(IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU)))
				return -1;
			untrusted = gdev->dev;
		}
	}

	/*
	 * If the common dma ops are not selected in kconfig then we cannot use
	 * IOMMU_DOMAIN_DMA at all. Force IDENTITY if nothing else has been
	 * selected.
	 */
	if (!IS_ENABLED(CONFIG_IOMMU_DMA)) {
		if (WARN_ON(driver_type == IOMMU_DOMAIN_DMA))
			return -1;
		if (!driver_type)
			driver_type = IOMMU_DOMAIN_IDENTITY;
	}

	if (untrusted) {
		if (driver_type && driver_type != IOMMU_DOMAIN_DMA) {
			dev_err_ratelimited(
				untrusted,
				"Device is not trusted, but driver is overriding group %u to %s, refusing to probe.\n",
				group->id, iommu_domain_type_str(driver_type));
			return -1;
		}
		driver_type = IOMMU_DOMAIN_DMA;
	}

	if (target_type) {
		if (driver_type && target_type != driver_type)
			return -1;
		return target_type;
	}
	return driver_type;
}

static void iommu_group_do_probe_finalize(struct device *dev)
{
	const struct iommu_ops *ops = dev_iommu_ops(dev);

	if (ops->probe_finalize)
		ops->probe_finalize(dev);
}

static int bus_iommu_probe(const struct bus_type *bus)
{
	struct iommu_group *group, *next;
	LIST_HEAD(group_list);
	int ret;

	ret = bus_for_each_dev(bus, NULL, &group_list, probe_iommu_group);
	if (ret)
		return ret;

	list_for_each_entry_safe(group, next, &group_list, entry) {
		struct group_device *gdev;

		mutex_lock(&group->mutex);

		/* Remove item from the list */
		list_del_init(&group->entry);

		/*
		 * We go to the trouble of deferred default domain creation so
		 * that the cross-group default domain type and the setup of the
		 * IOMMU_RESV_DIRECT will work correctly in non-hotpug scenarios.
		 */
		ret = iommu_setup_default_domain(group, 0);
		if (ret) {
			mutex_unlock(&group->mutex);
			return ret;
		}
		for_each_group_device(group, gdev)
			iommu_setup_dma_ops(gdev->dev);
		mutex_unlock(&group->mutex);

		/*
		 * FIXME: Mis-locked because the ops->probe_finalize() call-back
		 * of some IOMMU drivers calls arm_iommu_attach_device() which
		 * in-turn might call back into IOMMU core code, where it tries
		 * to take group->mutex, resulting in a deadlock.
		 */
		for_each_group_device(group, gdev)
			iommu_group_do_probe_finalize(gdev->dev);
	}

	return 0;
}

/**
 * device_iommu_capable() - check for a general IOMMU capability
 * @dev: device to which the capability would be relevant, if available
 * @cap: IOMMU capability
 *
 * Return: true if an IOMMU is present and supports the given capability
 * for the given device, otherwise false.
 */
bool device_iommu_capable(struct device *dev, enum iommu_cap cap)
{
	const struct iommu_ops *ops;

	if (!dev_has_iommu(dev))
		return false;

	ops = dev_iommu_ops(dev);
	if (!ops->capable)
		return false;

	return ops->capable(dev, cap);
}
EXPORT_SYMBOL_GPL(device_iommu_capable);

/**
 * iommu_group_has_isolated_msi() - Compute msi_device_has_isolated_msi()
 *       for a group
 * @group: Group to query
 *
 * IOMMU groups should not have differing values of
 * msi_device_has_isolated_msi() for devices in a group. However nothing
 * directly prevents this, so ensure mistakes don't result in isolation failures
 * by checking that all the devices are the same.
 */
bool iommu_group_has_isolated_msi(struct iommu_group *group)
{
	struct group_device *group_dev;
	bool ret = true;

	mutex_lock(&group->mutex);
	for_each_group_device(group, group_dev)
		ret &= msi_device_has_isolated_msi(group_dev->dev);
	mutex_unlock(&group->mutex);
	return ret;
}
EXPORT_SYMBOL_GPL(iommu_group_has_isolated_msi);

/**
 * iommu_set_fault_handler() - set a fault handler for an iommu domain
 * @domain: iommu domain
 * @handler: fault handler
 * @token: user data, will be passed back to the fault handler
 *
 * This function should be used by IOMMU users which want to be notified
 * whenever an IOMMU fault happens.
 *
 * The fault handler itself should return 0 on success, and an appropriate
 * error code otherwise.
 */
void iommu_set_fault_handler(struct iommu_domain *domain,
					iommu_fault_handler_t handler,
					void *token)
{
	if (WARN_ON(!domain || domain->cookie_type != IOMMU_COOKIE_NONE))
		return;

	domain->cookie_type = IOMMU_COOKIE_FAULT_HANDLER;
	domain->handler = handler;
	domain->handler_token = token;
}
EXPORT_SYMBOL_GPL(iommu_set_fault_handler);

static void iommu_domain_init(struct iommu_domain *domain, unsigned int type,
			      const struct iommu_ops *ops)
{
	domain->type = type;
	domain->owner = ops;
	if (!domain->ops)
		domain->ops = ops->default_domain_ops;
}

static struct iommu_domain *
__iommu_paging_domain_alloc_flags(struct device *dev, unsigned int type,
				  unsigned int flags)
{
	const struct iommu_ops *ops;
	struct iommu_domain *domain;

	if (!dev_has_iommu(dev))
		return ERR_PTR(-ENODEV);

	ops = dev_iommu_ops(dev);

	if (ops->domain_alloc_paging && !flags)
		domain = ops->domain_alloc_paging(dev);
	else if (ops->domain_alloc_paging_flags)
		domain = ops->domain_alloc_paging_flags(dev, flags, NULL);
#if IS_ENABLED(CONFIG_FSL_PAMU)
	else if (ops->domain_alloc && !flags)
		domain = ops->domain_alloc(IOMMU_DOMAIN_UNMANAGED);
#endif
	else
		return ERR_PTR(-EOPNOTSUPP);

	if (IS_ERR(domain))
		return domain;
	if (!domain)
		return ERR_PTR(-ENOMEM);

	iommu_domain_init(domain, type, ops);
	return domain;
}

/**
 * iommu_paging_domain_alloc_flags() - Allocate a paging domain
 * @dev: device for which the domain is allocated
 * @flags: Bitmap of iommufd_hwpt_alloc_flags
 *
 * Allocate a paging domain which will be managed by a kernel driver. Return
 * allocated domain if successful, or an ERR pointer for failure.
 */
struct iommu_domain *iommu_paging_domain_alloc_flags(struct device *dev,
						     unsigned int flags)
{
	return __iommu_paging_domain_alloc_flags(dev,
					 IOMMU_DOMAIN_UNMANAGED, flags);
}
EXPORT_SYMBOL_GPL(iommu_paging_domain_alloc_flags);

void iommu_domain_free(struct iommu_domain *domain)
{
	switch (domain->cookie_type) {
	case IOMMU_COOKIE_DMA_IOVA:
		iommu_put_dma_cookie(domain);
		break;
	case IOMMU_COOKIE_DMA_MSI:
		iommu_put_msi_cookie(domain);
		break;
	case IOMMU_COOKIE_SVA:
		mmdrop(domain->mm);
		break;
	default:
		break;
	}
	if (domain->ops->free)
		domain->ops->free(domain);
}
EXPORT_SYMBOL_GPL(iommu_domain_free);

/*
 * Put the group's domain back to the appropriate core-owned domain - either the
 * standard kernel-mode DMA configuration or an all-DMA-blocked domain.
 */
static void __iommu_group_set_core_domain(struct iommu_group *group)
{
	struct iommu_domain *new_domain;

	if (group->owner)
		new_domain = group->blocking_domain;
	else
		new_domain = group->default_domain;

	__iommu_group_set_domain_nofail(group, new_domain);
}

static int __iommu_attach_device(struct iommu_domain *domain,
				 struct device *dev)
{
	int ret;

	if (unlikely(domain->ops->attach_dev == NULL))
		return -ENODEV;

	ret = domain->ops->attach_dev(domain, dev);
	if (ret)
		return ret;
	dev->iommu->attach_deferred = 0;
	trace_attach_device_to_domain(dev);
	return 0;
}

/**
 * iommu_attach_device - Attach an IOMMU domain to a device
 * @domain: IOMMU domain to attach
 * @dev: Device that will be attached
 *
 * Returns 0 on success and error code on failure
 *
 * Note that EINVAL can be treated as a soft failure, indicating
 * that certain configuration of the domain is incompatible with
 * the device. In this case attaching a different domain to the
 * device may succeed.
 */
int iommu_attach_device(struct iommu_domain *domain, struct device *dev)
{
	/* Caller must be a probed driver on dev */
	struct iommu_group *group = dev->iommu_group;
	int ret;

	if (!group)
		return -ENODEV;

	/*
	 * Lock the group to make sure the device-count doesn't
	 * change while we are attaching
	 */
	mutex_lock(&group->mutex);
	ret = -EINVAL;
	if (list_count_nodes(&group->devices) != 1)
		goto out_unlock;

	ret = __iommu_attach_group(domain, group);

out_unlock:
	mutex_unlock(&group->mutex);
	return ret;
}
EXPORT_SYMBOL_GPL(iommu_attach_device);

int iommu_deferred_attach(struct device *dev, struct iommu_domain *domain)
{
	if (dev->iommu && dev->iommu->attach_deferred)
		return __iommu_attach_device(domain, dev);

	return 0;
}

void iommu_detach_device(struct iommu_domain *domain, struct device *dev)
{
	/* Caller must be a probed driver on dev */
	struct iommu_group *group = dev->iommu_group;

	if (!group)
		return;

	mutex_lock(&group->mutex);
	if (WARN_ON(domain != group->domain) ||
	    WARN_ON(list_count_nodes(&group->devices) != 1))
		goto out_unlock;
	__iommu_group_set_core_domain(group);

out_unlock:
	mutex_unlock(&group->mutex);
}
EXPORT_SYMBOL_GPL(iommu_detach_device);

struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
{
	/* Caller must be a probed driver on dev */
	struct iommu_group *group = dev->iommu_group;

	if (!group)
		return NULL;

	return group->domain;
}
EXPORT_SYMBOL_GPL(iommu_get_domain_for_dev);

/*
 * For IOMMU_DOMAIN_DMA implementations which already provide their own
 * guarantees that the group and its default domain are valid and correct.
 */
struct iommu_domain *iommu_get_dma_domain(struct device *dev)
{
	return dev->iommu_group->default_domain;
}

static void *iommu_make_pasid_array_entry(struct iommu_domain *domain,
					  struct iommu_attach_handle *handle)
{
	if (handle) {
		handle->domain = domain;
		return xa_tag_pointer(handle, IOMMU_PASID_ARRAY_HANDLE);
	}

	return xa_tag_pointer(domain, IOMMU_PASID_ARRAY_DOMAIN);
}

static bool domain_iommu_ops_compatible(const struct iommu_ops *ops,
					struct iommu_domain *domain)
{
	if (domain->owner == ops)
		return true;

	/* For static domains, owner isn't set. */
	if (domain == ops->blocked_domain || domain == ops->identity_domain)
		return true;

	return false;
}

static int __iommu_attach_group(struct iommu_domain *domain,
				struct iommu_group *group)
{
	struct device *dev;

	if (group->domain && group->domain != group->default_domain &&
	    group->domain != group->blocking_domain)
		return -EBUSY;

	dev = iommu_group_first_dev(group);
	if (!dev_has_iommu(dev) ||
	    !domain_iommu_ops_compatible(dev_iommu_ops(dev), domain))
		return -EINVAL;

	return __iommu_group_set_domain(group, domain);
}

/**
 * iommu_attach_group - Attach an IOMMU domain to an IOMMU group
 * @domain: IOMMU domain to attach
 * @group: IOMMU group that will be attached
 *
 * Returns 0 on success and error code on failure
 *
 * Note that EINVAL can be treated as a soft failure, indicating
 * that certain configuration of the domain is incompatible with
 * the group. In this case attaching a different domain to the
 * group may succeed.
 */
int iommu_attach_group(struct iommu_domain *domain, struct iommu_group *group)
{
	int ret;

	mutex_lock(&group->mutex);
	ret = __iommu_attach_group(domain, group);
	mutex_unlock(&group->mutex);

	return ret;
}
EXPORT_SYMBOL_GPL(iommu_attach_group);

static int __iommu_device_set_domain(struct iommu_group *group,
				     struct device *dev,
				     struct iommu_domain *new_domain,
				     unsigned int flags)
{
	int ret;

	/*
	 * If the device requires IOMMU_RESV_DIRECT then we cannot allow
	 * the blocking domain to be attached as it does not contain the
	 * required 1:1 mapping. This test effectively excludes the device
	 * being used with iommu_group_claim_dma_owner() which will block
	 * vfio and iommufd as well.
	 */
	if (dev->iommu->require_direct &&
	    (new_domain->type == IOMMU_DOMAIN_BLOCKED ||
	     new_domain == group->blocking_domain)) {
		dev_warn(dev,
			 "Firmware has requested this device have a 1:1 IOMMU mapping, rejecting configuring the device without a 1:1 mapping. Contact your platform vendor.\n");
		return -EINVAL;
	}

	if (dev->iommu->attach_deferred) {
		if (new_domain == group->default_domain)
			return 0;
		dev->iommu->attach_deferred = 0;
	}

	ret = __iommu_attach_device(new_domain, dev);
	if (ret) {
		/*
		 * If we have a blocking domain then try to attach that in hopes
		 * of avoiding a UAF. Modern drivers should implement blocking
		 * domains as global statics that cannot fail.
		 */
		if ((flags & IOMMU_SET_DOMAIN_MUST_SUCCEED) &&
		    group->blocking_domain &&
		    group->blocking_domain != new_domain)
			__iommu_attach_device(group->blocking_domain, dev);
		return ret;
	}
	return 0;
}

/*
 * If 0 is returned the group's domain is new_domain. If an error is returned
 * then the group's domain will be set back to the existing domain unless
 * IOMMU_SET_DOMAIN_MUST_SUCCEED, otherwise an error is returned and the group's
 * domains is left inconsistent. This is a driver bug to fail attach with a
 * previously good domain. We try to avoid a kernel UAF because of this.
 *
 * IOMMU groups are really the natural working unit of the IOMMU, but the IOMMU
 * API works on domains and devices.  Bridge that gap by iterating over the
 * devices in a group.  Ideally we'd have a single device which represents the
 * requestor ID of the group, but we also allow IOMMU drivers to create policy
 * defined minimum sets, where the physical hardware may be able to distiguish
 * members, but we wish to group them at a higher level (ex. untrusted
 * multi-function PCI devices).  Thus we attach each device.
 */
static int __iommu_group_set_domain_internal(struct iommu_group *group,
					     struct iommu_domain *new_domain,
					     unsigned int flags)
{
	struct group_device *last_gdev;
	struct group_device *gdev;
	int result;
	int ret;

	lockdep_assert_held(&group->mutex);

	if (group->domain == new_domain)
		return 0;

	if (WARN_ON(!new_domain))
		return -EINVAL;

	/*
	 * Changing the domain is done by calling attach_dev() on the new
	 * domain. This switch does not have to be atomic and DMA can be
	 * discarded during the transition. DMA must only be able to access
	 * either new_domain or group->domain, never something else.
	 */
	result = 0;
	for_each_group_device(group, gdev) {
		ret = __iommu_device_set_domain(group, gdev->dev, new_domain,
						flags);
		if (ret) {
			result = ret;
			/*
			 * Keep trying the other devices in the group. If a
			 * driver fails attach to an otherwise good domain, and
			 * does not support blocking domains, it should at least
			 * drop its reference on the current domain so we don't
			 * UAF.
			 */
			if (flags & IOMMU_SET_DOMAIN_MUST_SUCCEED)
				continue;
			goto err_revert;
		}
	}
	group->domain = new_domain;
	return result;

err_revert:
	/*
	 * This is called in error unwind paths. A well behaved driver should
	 * always allow us to attach to a domain that was already attached.
	 */
	last_gdev = gdev;
	for_each_group_device(group, gdev) {
		/*
		 * A NULL domain can happen only for first probe, in which case
		 * we leave group->domain as NULL and let release clean
		 * everything up.
		 */
		if (group->domain)
			WARN_ON(__iommu_device_set_domain(
				group, gdev->dev, group->domain,
				IOMMU_SET_DOMAIN_MUST_SUCCEED));
		if (gdev == last_gdev)
			break;
	}
	return ret;
}

void iommu_detach_group(struct iommu_domain *domain, struct iommu_group *group)
{
	mutex_lock(&group->mutex);
	__iommu_group_set_core_domain(group);
	mutex_unlock(&group->mutex);
}
EXPORT_SYMBOL_GPL(iommu_detach_group);

phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova)
{
	if (domain->type == IOMMU_DOMAIN_IDENTITY)
		return iova;

	if (domain->type == IOMMU_DOMAIN_BLOCKED)
		return 0;

	return domain->ops->iova_to_phys(domain, iova);
}
EXPORT_SYMBOL_GPL(iommu_iova_to_phys);

static size_t iommu_pgsize(struct iommu_domain *domain, unsigned long iova,
			   phys_addr_t paddr, size_t size, size_t *count)
{
	unsigned int pgsize_idx, pgsize_idx_next;
	unsigned long pgsizes;
	size_t offset, pgsize, pgsize_next;
	size_t offset_end;
	unsigned long addr_merge = paddr | iova;

	/* Page sizes supported by the hardware and small enough for @size */
	pgsizes = domain->pgsize_bitmap & GENMASK(__fls(size), 0);

	/* Constrain the page sizes further based on the maximum alignment */
	if (likely(addr_merge))
		pgsizes &= GENMASK(__ffs(addr_merge), 0);

	/* Make sure we have at least one suitable page size */
	BUG_ON(!pgsizes);

	/* Pick the biggest page size remaining */
	pgsize_idx = __fls(pgsizes);
	pgsize = BIT(pgsize_idx);
	if (!count)
		return pgsize;

	/* Find the next biggest support page size, if it exists */
	pgsizes = domain->pgsize_bitmap & ~GENMASK(pgsize_idx, 0);
	if (!pgsizes)
		goto out_set_count;

	pgsize_idx_next = __ffs(pgsizes);
	pgsize_next = BIT(pgsize_idx_next);

	/*
	 * There's no point trying a bigger page size unless the virtual
	 * and physical addresses are similarly offset within the larger page.
	 */
	if ((iova ^ paddr) & (pgsize_next - 1))
		goto out_set_count;

	/* Calculate the offset to the next page size alignment boundary */
	offset = pgsize_next - (addr_merge & (pgsize_next - 1));

	/*
	 * If size is big enough to accommodate the larger page, reduce
	 * the number of smaller pages.
	 */
	if (!check_add_overflow(offset, pgsize_next, &offset_end) &&
	    offset_end <= size)
		size = offset;

out_set_count:
	*count = size >> pgsize_idx;
	return pgsize;
}

int iommu_map_nosync(struct iommu_domain *domain, unsigned long iova,
		phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
{
	const struct iommu_domain_ops *ops = domain->ops;
	unsigned long orig_iova = iova;
	unsigned int min_pagesz;
	size_t orig_size = size;
	phys_addr_t orig_paddr = paddr;
	int ret = 0;

	might_sleep_if(gfpflags_allow_blocking(gfp));

	if (unlikely(!(domain->type & __IOMMU_DOMAIN_PAGING)))
		return -EINVAL;

	if (WARN_ON(!ops->map_pages || domain->pgsize_bitmap == 0UL))
		return -ENODEV;

	/* Discourage passing strange GFP flags */
	if (WARN_ON_ONCE(gfp & (__GFP_COMP | __GFP_DMA | __GFP_DMA32 |
				__GFP_HIGHMEM)))
		return -EINVAL;

	/* find out the minimum page size supported */
	min_pagesz = 1 << __ffs(domain->pgsize_bitmap);

	/*
	 * both the virtual address and the physical one, as well as
	 * the size of the mapping, must be aligned (at least) to the
	 * size of the smallest page supported by the hardware
	 */
	if (!IS_ALIGNED(iova | paddr | size, min_pagesz)) {
		pr_err("unaligned: iova 0x%lx pa %pa size 0x%zx min_pagesz 0x%x\n",
		       iova, &paddr, size, min_pagesz);
		return -EINVAL;
	}

	pr_debug("map: iova 0x%lx pa %pa size 0x%zx\n", iova, &paddr, size);

	while (size) {
		size_t pgsize, count, mapped = 0;

		pgsize = iommu_pgsize(domain, iova, paddr, size, &count);

		pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx count %zu\n",
			 iova, &paddr, pgsize, count);
		ret = ops->map_pages(domain, iova, paddr, pgsize, count, prot,
				     gfp, &mapped);
		/*
		 * Some pages may have been mapped, even if an error occurred,
		 * so we should account for those so they can be unmapped.
		 */
		size -= mapped;

		if (ret)
			break;

		iova += mapped;
		paddr += mapped;
	}

	/* unroll mapping in case something went wrong */
	if (ret)
		iommu_unmap(domain, orig_iova, orig_size - size);
	else
		trace_map(orig_iova, orig_paddr, orig_size);

	return ret;
}

int iommu_sync_map(struct iommu_domain *domain, unsigned long iova, size_t size)
{
	const struct iommu_domain_ops *ops = domain->ops;

	if (!ops->iotlb_sync_map)
		return 0;
	return ops->iotlb_sync_map(domain, iova, size);
}

int iommu_map(struct iommu_domain *domain, unsigned long iova,
	      phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
{
	int ret;

	ret = iommu_map_nosync(domain, iova, paddr, size, prot, gfp);
	if (ret)
		return ret;

	ret = iommu_sync_map(domain, iova, size);
	if (ret)
		iommu_unmap(domain, iova, size);

	return ret;
}
EXPORT_SYMBOL_GPL(iommu_map);

static size_t __iommu_unmap(struct iommu_domain *domain,
			    unsigned long iova, size_t size,
			    struct iommu_iotlb_gather *iotlb_gather)
{
	const struct iommu_domain_ops *ops = domain->ops;
	size_t unmapped_page, unmapped = 0;
	unsigned long orig_iova = iova;
	unsigned int min_pagesz;

	if (unlikely(!(domain->type & __IOMMU_DOMAIN_PAGING)))
		return 0;

	if (WARN_ON(!ops->unmap_pages || domain->pgsize_bitmap == 0UL))
		return 0;

	/* find out the minimum page size supported */
	min_pagesz = 1 << __ffs(domain->pgsize_bitmap);

	/*
	 * The virtual address, as well as the size of the mapping, must be
	 * aligned (at least) to the size of the smallest page supported
	 * by the hardware
	 */
	if (!IS_ALIGNED(iova | size, min_pagesz)) {
		pr_err("unaligned: iova 0x%lx size 0x%zx min_pagesz 0x%x\n",
		       iova, size, min_pagesz);
		return 0;
	}

	pr_debug("unmap this: iova 0x%lx size 0x%zx\n", iova, size);

	/*
	 * Keep iterating until we either unmap 'size' bytes (or more)
	 * or we hit an area that isn't mapped.
	 */
	while (unmapped < size) {
		size_t pgsize, count;

		pgsize = iommu_pgsize(domain, iova, iova, size - unmapped, &count);
		unmapped_page = ops->unmap_pages(domain, iova, pgsize, count, iotlb_gather);
		if (!unmapped_page)
			break;

		pr_debug("unmapped: iova 0x%lx size 0x%zx\n",
			 iova, unmapped_page);

		iova += unmapped_page;
		unmapped += unmapped_page;
	}

	trace_unmap(orig_iova, size, unmapped);
	return unmapped;
}

/**
 * iommu_unmap() - Remove mappings from a range of IOVA
 * @domain: Domain to manipulate
 * @iova: IO virtual address to start
 * @size: Length of the range starting from @iova
 *
 * iommu_unmap() will remove a translation created by iommu_map(). It cannot
 * subdivide a mapping created by iommu_map(), so it should be called with IOVA
 * ranges that match what was passed to iommu_map(). The range can aggregate
 * contiguous iommu_map() calls so long as no individual range is split.
 *
 * Returns: Number of bytes of IOVA unmapped. iova + res will be the point
 * unmapping stopped.
 */
size_t iommu_unmap(struct iommu_domain *domain,
		   unsigned long iova, size_t size)
{
	struct iommu_iotlb_gather iotlb_gather;
	size_t ret;

	iommu_iotlb_gather_init(&iotlb_gather);
	ret = __iommu_unmap(domain, iova, size, &iotlb_gather);
	iommu_iotlb_sync(domain, &iotlb_gather);

	return ret;
}
EXPORT_SYMBOL_GPL(iommu_unmap);

/**
 * iommu_unmap_fast() - Remove mappings from a range of IOVA without IOTLB sync
 * @domain: Domain to manipulate
 * @iova: IO virtual address to start
 * @size: Length of the range starting from @iova
 * @iotlb_gather: range information for a pending IOTLB flush
 *
 * iommu_unmap_fast() will remove a translation created by iommu_map().
 * It can't subdivide a mapping created by iommu_map(), so it should be
 * called with IOVA ranges that match what was passed to iommu_map(). The
 * range can aggregate contiguous iommu_map() calls so long as no individual
 * range is split.
 *
 * Basically iommu_unmap_fast() is the same as iommu_unmap() but for callers
 * which manage the IOTLB flushing externally to perform a batched sync.
 *
 * Returns: Number of bytes of IOVA unmapped. iova + res will be the point
 * unmapping stopped.
 */
size_t iommu_unmap_fast(struct iommu_domain *domain,
			unsigned long iova, size_t size,
			struct iommu_iotlb_gather *iotlb_gather)
{
	return __iommu_unmap(domain, iova, size, iotlb_gather);
}
EXPORT_SYMBOL_GPL(iommu_unmap_fast);

ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
		     struct scatterlist *sg, unsigned int nents, int prot,
		     gfp_t gfp)
{
	size_t len = 0, mapped = 0;
	phys_addr_t start;
	unsigned int i = 0;
	int ret;

	while (i <= nents) {
		phys_addr_t s_phys = sg_phys(sg);

		if (len && s_phys != start + len) {
			ret = iommu_map_nosync(domain, iova + mapped, start,
					len, prot, gfp);
			if (ret)
				goto out_err;

			mapped += len;
			len = 0;
		}

		if (sg_dma_is_bus_address(sg))
			goto next;

		if (len) {
			len += sg->length;
		} else {
			len = sg->length;
			start = s_phys;
		}

next:
		if (++i < nents)
			sg = sg_next(sg);
	}

	ret = iommu_sync_map(domain, iova, mapped);
	if (ret)
		goto out_err;

	return mapped;

out_err:
	/* undo mappings already done */
	iommu_unmap(domain, iova, mapped);

	return ret;
}
EXPORT_SYMBOL_GPL(iommu_map_sg);

/**
 * report_iommu_fault() - report about an IOMMU fault to the IOMMU framework
 * @domain: the iommu domain where the fault has happened
 * @dev: the device where the fault has happened
 * @iova: the faulting address
 * @flags: mmu fault flags (e.g. IOMMU_FAULT_READ/IOMMU_FAULT_WRITE/...)
 *
 * This function should be called by the low-level IOMMU implementations
 * whenever IOMMU faults happen, to allow high-level users, that are
 * interested in such events, to know about them.
 *
 * This event may be useful for several possible use cases:
 * - mere logging of the event
 * - dynamic TLB/PTE loading
 * - if restarting of the faulting device is required
 *
 * Returns 0 on success and an appropriate error code otherwise (if dynamic
 * PTE/TLB loading will one day be supported, implementations will be able
 * to tell whether it succeeded or not according to this return value).
 *
 * Specifically, -ENOSYS is returned if a fault handler isn't installed
 * (though fault handlers can also return -ENOSYS, in case they want to
 * elicit the default behavior of the IOMMU drivers).
 */
int report_iommu_fault(struct iommu_domain *domain, struct device *dev,
		       unsigned long iova, int flags)
{
	int ret = -ENOSYS;

	/*
	 * if upper layers showed interest and installed a fault handler,
	 * invoke it.
	 */
	if (domain->cookie_type == IOMMU_COOKIE_FAULT_HANDLER &&
	    domain->handler)
		ret = domain->handler(domain, dev, iova, flags,
						domain->handler_token);

	trace_io_page_fault(dev, iova, flags);
	return ret;
}
EXPORT_SYMBOL_GPL(report_iommu_fault);

static int __init iommu_init(void)
{
	iommu_group_kset = kset_create_and_add("iommu_groups",
					       NULL, kernel_kobj);
	BUG_ON(!iommu_group_kset);

	iommu_debugfs_setup();

	return 0;
}
core_initcall(iommu_init);

int iommu_set_pgtable_quirks(struct iommu_domain *domain,
		unsigned long quirk)
{
	if (domain->type != IOMMU_DOMAIN_UNMANAGED)
		return -EINVAL;
	if (!domain->ops->set_pgtable_quirks)
		return -EINVAL;
	return domain->ops->set_pgtable_quirks(domain, quirk);
}
EXPORT_SYMBOL_GPL(iommu_set_pgtable_quirks);

/**
 * iommu_get_resv_regions - get reserved regions
 * @dev: device for which to get reserved regions
 * @list: reserved region list for device
 *
 * This returns a list of reserved IOVA regions specific to this device.
 * A domain user should not map IOVA in these ranges.
 */
void iommu_get_resv_regions(struct device *dev, struct list_head *list)
{
	const struct iommu_ops *ops = dev_iommu_ops(dev);

	if (ops->get_resv_regions)
		ops->get_resv_regions(dev, list);
}
EXPORT_SYMBOL_GPL(iommu_get_resv_regions);

/**
 * iommu_put_resv_regions - release reserved regions
 * @dev: device for which to free reserved regions
 * @list: reserved region list for device
 *
 * This releases a reserved region list acquired by iommu_get_resv_regions().
 */
void iommu_put_resv_regions(struct device *dev, struct list_head *list)
{
	struct iommu_resv_region *entry, *next;

	list_for_each_entry_safe(entry, next, list, list) {
		if (entry->free)
			entry->free(dev, entry);
		else
			kfree(entry);
	}
}
EXPORT_SYMBOL(iommu_put_resv_regions);

struct iommu_resv_region *iommu_alloc_resv_region(phys_addr_t start,
						  size_t length, int prot,
						  enum iommu_resv_type type,
						  gfp_t gfp)
{
	struct iommu_resv_region *region;

	region = kzalloc(sizeof(*region), gfp);
	if (!region)
		return NULL;

	INIT_LIST_HEAD(&region->list);
	region->start = start;
	region->length = length;
	region->prot = prot;
	region->type = type;
	return region;
}
EXPORT_SYMBOL_GPL(iommu_alloc_resv_region);

void iommu_set_default_passthrough(bool cmd_line)
{
	if (cmd_line)
		iommu_cmd_line |= IOMMU_CMD_LINE_DMA_API;
	iommu_def_domain_type = IOMMU_DOMAIN_IDENTITY;
}

void iommu_set_default_translated(bool cmd_line)
{
	if (cmd_line)
		iommu_cmd_line |= IOMMU_CMD_LINE_DMA_API;
	iommu_def_domain_type = IOMMU_DOMAIN_DMA;
}

bool iommu_default_passthrough(void)
{
	return iommu_def_domain_type == IOMMU_DOMAIN_IDENTITY;
}
EXPORT_SYMBOL_GPL(iommu_default_passthrough);

static const struct iommu_device *iommu_from_fwnode(const struct fwnode_handle *fwnode)
{
	const struct iommu_device *iommu, *ret = NULL;

	spin_lock(&iommu_device_lock);
	list_for_each_entry(iommu, &iommu_device_list, list)
		if (iommu->fwnode == fwnode) {
			ret = iommu;
			break;
		}
	spin_unlock(&iommu_device_lock);
	return ret;
}

const struct iommu_ops *iommu_ops_from_fwnode(const struct fwnode_handle *fwnode)
{
	const struct iommu_device *iommu = iommu_from_fwnode(fwnode);

	return iommu ? iommu->ops : NULL;
}

int iommu_fwspec_init(struct device *dev, struct fwnode_handle *iommu_fwnode)
{
	const struct iommu_device *iommu = iommu_from_fwnode(iommu_fwnode);
	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);

	if (!iommu)
		return driver_deferred_probe_check_state(dev);
	if (!dev->iommu && !READ_ONCE(iommu->ready))
		return -EPROBE_DEFER;

	if (fwspec)
		return iommu->ops == iommu_fwspec_ops(fwspec) ? 0 : -EINVAL;

	if (!dev_iommu_get(dev))
		return -ENOMEM;

	/* Preallocate for the overwhelmingly common case of 1 ID */
	fwspec = kzalloc(struct_size(fwspec, ids, 1), GFP_KERNEL);
	if (!fwspec)
		return -ENOMEM;

	fwnode_handle_get(iommu_fwnode);
	fwspec->iommu_fwnode = iommu_fwnode;
	dev_iommu_fwspec_set(dev, fwspec);
	return 0;
}
EXPORT_SYMBOL_GPL(iommu_fwspec_init);

void iommu_fwspec_free(struct device *dev)
{
	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);

	if (fwspec) {
		fwnode_handle_put(fwspec->iommu_fwnode);
		kfree(fwspec);
		dev_iommu_fwspec_set(dev, NULL);
	}
}

int iommu_fwspec_add_ids(struct device *dev, const u32 *ids, int num_ids)
{
	struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
	int i, new_num;

	if (!fwspec)
		return -EINVAL;

	new_num = fwspec->num_ids + num_ids;
	if (new_num > 1) {
		fwspec = krealloc(fwspec, struct_size(fwspec, ids, new_num),
				  GFP_KERNEL);
		if (!fwspec)
			return -ENOMEM;

		dev_iommu_fwspec_set(dev, fwspec);
	}

	for (i = 0; i < num_ids; i++)
		fwspec->ids[fwspec->num_ids + i] = ids[i];

	fwspec->num_ids = new_num;
	return 0;
}
EXPORT_SYMBOL_GPL(iommu_fwspec_add_ids);

/**
 * iommu_setup_default_domain - Set the default_domain for the group
 * @group: Group to change
 * @target_type: Domain type to set as the default_domain
 *
 * Allocate a default domain and set it as the current domain on the group. If
 * the group already has a default domain it will be changed to the target_type.
 * When target_type is 0 the default domain is selected based on driver and
 * system preferences.
 */
static int iommu_setup_default_domain(struct iommu_group *group,
				      int target_type)
{
	struct iommu_domain *old_dom = group->default_domain;
	struct group_device *gdev;
	struct iommu_domain *dom;
	bool direct_failed;
	int req_type;
	int ret;

	lockdep_assert_held(&group->mutex);

	req_type = iommu_get_default_domain_type(group, target_type);
	if (req_type < 0)
		return -EINVAL;

	dom = iommu_group_alloc_default_domain(group, req_type);
	if (IS_ERR(dom))
		return PTR_ERR(dom);

	if (group->default_domain == dom)
		return 0;

	if (iommu_is_dma_domain(dom)) {
		ret = iommu_get_dma_cookie(dom);
		if (ret) {
			iommu_domain_free(dom);
			return ret;
		}
	}

	/*
	 * IOMMU_RESV_DIRECT and IOMMU_RESV_DIRECT_RELAXABLE regions must be
	 * mapped before their device is attached, in order to guarantee
	 * continuity with any FW activity
	 */
	direct_failed = false;
	for_each_group_device(group, gdev) {
		if (iommu_create_device_direct_mappings(dom, gdev->dev)) {
			direct_failed = true;
			dev_warn_once(
				gdev->dev->iommu->iommu_dev->dev,
				"IOMMU driver was not able to establish FW requested direct mapping.");
		}
	}

	/* We must set default_domain early for __iommu_device_set_domain */
	group->default_domain = dom;
	if (!group->domain) {
		/*
		 * Drivers are not allowed to fail the first domain attach.
		 * The only way to recover from this is to fail attaching the
		 * iommu driver and call ops->release_device. Put the domain
		 * in group->default_domain so it is freed after.
		 */
		ret = __iommu_group_set_domain_internal(
			group, dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
		if (WARN_ON(ret))
			goto out_free_old;
	} else {
		ret = __iommu_group_set_domain(group, dom);
		if (ret)
			goto err_restore_def_domain;
	}

	/*
	 * Drivers are supposed to allow mappings to be installed in a domain
	 * before device attachment, but some don't. Hack around this defect by
	 * trying again after attaching. If this happens it means the device
	 * will not continuously have the IOMMU_RESV_DIRECT map.
	 */
	if (direct_failed) {
		for_each_group_device(group, gdev) {
			ret = iommu_create_device_direct_mappings(dom, gdev->dev);
			if (ret)
				goto err_restore_domain;
		}
	}

out_free_old:
	if (old_dom)
		iommu_domain_free(old_dom);
	return ret;

err_restore_domain:
	if (old_dom)
		__iommu_group_set_domain_internal(
			group, old_dom, IOMMU_SET_DOMAIN_MUST_SUCCEED);
err_restore_def_domain:
	if (old_dom) {
		iommu_domain_free(dom);
		group->default_domain = old_dom;
	}
	return ret;
}

/*
 * Changing the default domain through sysfs requires the users to unbind the
 * drivers from the devices in the iommu group, except for a DMA -> DMA-FQ
 * transition. Return failure if this isn't met.
 *
 * We need to consider the race between this and the device release path.
 * group->mutex is used here to guarantee that the device release path
 * will not be entered at the same time.
 */
static ssize_t iommu_group_store_type(struct iommu_group *group,
				      const char *buf, size_t count)
{
	struct group_device *gdev;
	int ret, req_type;

	if (!capable(CAP_SYS_ADMIN) || !capable(CAP_SYS_RAWIO))
		return -EACCES;

	if (WARN_ON(!group) || !group->default_domain)
		return -EINVAL;

	if (sysfs_streq(buf, "identity"))
		req_type = IOMMU_DOMAIN_IDENTITY;
	else if (sysfs_streq(buf, "DMA"))
		req_type = IOMMU_DOMAIN_DMA;
	else if (sysfs_streq(buf, "DMA-FQ"))
		req_type = IOMMU_DOMAIN_DMA_FQ;
	else if (sysfs_streq(buf, "auto"))
		req_type = 0;
	else
		return -EINVAL;

	mutex_lock(&group->mutex);
	/* We can bring up a flush queue without tearing down the domain. */
	if (req_type == IOMMU_DOMAIN_DMA_FQ &&
	    group->default_domain->type == IOMMU_DOMAIN_DMA) {
		ret = iommu_dma_init_fq(group->default_domain);
		if (ret)
			goto out_unlock;

		group->default_domain->type = IOMMU_DOMAIN_DMA_FQ;
		ret = count;
		goto out_unlock;
	}

	/* Otherwise, ensure that device exists and no driver is bound. */
	if (list_empty(&group->devices) || group->owner_cnt) {
		ret = -EPERM;
		goto out_unlock;
	}

	ret = iommu_setup_default_domain(group, req_type);
	if (ret)
		goto out_unlock;

	/* Make sure dma_ops is appropriatley set */
	for_each_group_device(group, gdev)
		iommu_setup_dma_ops(gdev->dev);

out_unlock:
	mutex_unlock(&group->mutex);
	return ret ?: count;
}

/**
 * iommu_device_use_default_domain() - Device driver wants to handle device
 *                                     DMA through the kernel DMA API.
 * @dev: The device.
 *
 * The device driver about to bind @dev wants to do DMA through the kernel
 * DMA API. Return 0 if it is allowed, otherwise an error.
 */
int iommu_device_use_default_domain(struct device *dev)
{
	/* Caller is the driver core during the pre-probe path */
	struct iommu_group *group = dev->iommu_group;
	int ret = 0;

	if (!group)
		return 0;

	mutex_lock(&group->mutex);
	/* We may race against bus_iommu_probe() finalising groups here */
	if (!group->default_domain) {
		ret = -EPROBE_DEFER;
		goto unlock_out;
	}
	if (group->owner_cnt) {
		if (group->domain != group->default_domain || group->owner ||
		    !xa_empty(&group->pasid_array)) {
			ret = -EBUSY;
			goto unlock_out;
		}
	}

	group->owner_cnt++;

unlock_out:
	mutex_unlock(&group->mutex);
	return ret;
}

/**
 * iommu_device_unuse_default_domain() - Device driver stops handling device
 *                                       DMA through the kernel DMA API.
 * @dev: The device.
 *
 * The device driver doesn't want to do DMA through kernel DMA API anymore.
 * It must be called after iommu_device_use_default_domain().
 */
void iommu_device_unuse_default_domain(struct device *dev)
{
	/* Caller is the driver core during the post-probe path */
	struct iommu_group *group = dev->iommu_group;

	if (!group)
		return;

	mutex_lock(&group->mutex);
	if (!WARN_ON(!group->owner_cnt || !xa_empty(&group->pasid_array)))
		group->owner_cnt--;

	mutex_unlock(&group->mutex);
}

static int __iommu_group_alloc_blocking_domain(struct iommu_group *group)
{
	struct device *dev = iommu_group_first_dev(group);
	const struct iommu_ops *ops = dev_iommu_ops(dev);
	struct iommu_domain *domain;

	if (group->blocking_domain)
		return 0;

	if (ops->blocked_domain) {
		group->blocking_domain = ops->blocked_domain;
		return 0;
	}

	/*
	 * For drivers that do not yet understand IOMMU_DOMAIN_BLOCKED create an
	 * empty PAGING domain instead.
	 */
	domain = iommu_paging_domain_alloc(dev);
	if (IS_ERR(domain))
		return PTR_ERR(domain);
	group->blocking_domain = domain;
	return 0;
}

static int __iommu_take_dma_ownership(struct iommu_group *group, void *owner)
{
	int ret;

	if ((group->domain && group->domain != group->default_domain) ||
	    !xa_empty(&group->pasid_array))
		return -EBUSY;

	ret = __iommu_group_alloc_blocking_domain(group);
	if (ret)
		return ret;
	ret = __iommu_group_set_domain(group, group->blocking_domain);
	if (ret)
		return ret;

	group->owner = owner;
	group->owner_cnt++;
	return 0;
}

/**
 * iommu_group_claim_dma_owner() - Set DMA ownership of a group
 * @group: The group.
 * @owner: Caller specified pointer. Used for exclusive ownership.
 *
 * This is to support backward compatibility for vfio which manages the dma
 * ownership in iommu_group level. New invocations on this interface should be
 * prohibited. Only a single owner may exist for a group.
 */
int iommu_group_claim_dma_owner(struct iommu_group *group, void *owner)
{
	int ret = 0;

	if (WARN_ON(!owner))
		return -EINVAL;

	mutex_lock(&group->mutex);
	if (group->owner_cnt) {
		ret = -EPERM;
		goto unlock_out;
	}

	ret = __iommu_take_dma_ownership(group, owner);
unlock_out:
	mutex_unlock(&group->mutex);

	return ret;
}
EXPORT_SYMBOL_GPL(iommu_group_claim_dma_owner);

/**
 * iommu_device_claim_dma_owner() - Set DMA ownership of a device
 * @dev: The device.
 * @owner: Caller specified pointer. Used for exclusive ownership.
 *
 * Claim the DMA ownership of a device. Multiple devices in the same group may
 * concurrently claim ownership if they present the same owner value. Returns 0
 * on success and error code on failure
 */
int iommu_device_claim_dma_owner(struct device *dev, void *owner)
{
	/* Caller must be a probed driver on dev */
	struct iommu_group *group = dev->iommu_group;
	int ret = 0;

	if (WARN_ON(!owner))
		return -EINVAL;

	if (!group)
		return -ENODEV;

	mutex_lock(&group->mutex);
	if (group->owner_cnt) {
		if (group->owner != owner) {
			ret = -EPERM;
			goto unlock_out;
		}
		group->owner_cnt++;
		goto unlock_out;
	}

	ret = __iommu_take_dma_ownership(group, owner);
unlock_out:
	mutex_unlock(&group->mutex);
	return ret;
}
EXPORT_SYMBOL_GPL(iommu_device_claim_dma_owner);

static void __iommu_release_dma_ownership(struct iommu_group *group)
{
	if (WARN_ON(!group->owner_cnt || !group->owner ||
		    !xa_empty(&group->pasid_array)))
		return;

	group->owner_cnt = 0;
	group->owner = NULL;
	__iommu_group_set_domain_nofail(group, group->default_domain);
}

/**
 * iommu_group_release_dma_owner() - Release DMA ownership of a group
 * @group: The group
 *
 * Release the DMA ownership claimed by iommu_group_claim_dma_owner().
 */
void iommu_group_release_dma_owner(struct iommu_group *group)
{
	mutex_lock(&group->mutex);
	__iommu_release_dma_ownership(group);
	mutex_unlock(&group->mutex);
}
EXPORT_SYMBOL_GPL(iommu_group_release_dma_owner);

/**
 * iommu_device_release_dma_owner() - Release DMA ownership of a device
 * @dev: The device.
 *
 * Release the DMA ownership claimed by iommu_device_claim_dma_owner().
 */
void iommu_device_release_dma_owner(struct device *dev)
{
	/* Caller must be a probed driver on dev */
	struct iommu_group *group = dev->iommu_group;

	mutex_lock(&group->mutex);
	if (group->owner_cnt > 1)
		group->owner_cnt--;
	else
		__iommu_release_dma_ownership(group);
	mutex_unlock(&group->mutex);
}
EXPORT_SYMBOL_GPL(iommu_device_release_dma_owner);

/**
 * iommu_group_dma_owner_claimed() - Query group dma ownership status
 * @group: The group.
 *
 * This provides status query on a given group. It is racy and only for
 * non-binding status reporting.
 */
bool iommu_group_dma_owner_claimed(struct iommu_group *group)
{
	unsigned int user;

	mutex_lock(&group->mutex);
	user = group->owner_cnt;
	mutex_unlock(&group->mutex);

	return user;
}
EXPORT_SYMBOL_GPL(iommu_group_dma_owner_claimed);

static void iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid,
				   struct iommu_domain *domain)
{
	const struct iommu_ops *ops = dev_iommu_ops(dev);
	struct iommu_domain *blocked_domain = ops->blocked_domain;

	WARN_ON(blocked_domain->ops->set_dev_pasid(blocked_domain,
						   dev, pasid, domain));
}

static int __iommu_set_group_pasid(struct iommu_domain *domain,
				   struct iommu_group *group, ioasid_t pasid,
				   struct iommu_domain *old)
{
	struct group_device *device, *last_gdev;
	int ret;

	for_each_group_device(group, device) {
		if (device->dev->iommu->max_pasids > 0) {
			ret = domain->ops->set_dev_pasid(domain, device->dev,
							 pasid, old);
			if (ret)
				goto err_revert;
		}
	}

	return 0;

err_revert:
	last_gdev = device;
	for_each_group_device(group, device) {
		if (device == last_gdev)
			break;
		if (device->dev->iommu->max_pasids > 0) {
			/*
			 * If no old domain, undo the succeeded devices/pasid.
			 * Otherwise, rollback the succeeded devices/pasid to
			 * the old domain. And it is a driver bug to fail
			 * attaching with a previously good domain.
			 */
			if (!old ||
			    WARN_ON(old->ops->set_dev_pasid(old, device->dev,
							    pasid, domain)))
				iommu_remove_dev_pasid(device->dev, pasid, domain);
		}
	}
	return ret;
}

static void __iommu_remove_group_pasid(struct iommu_group *group,
				       ioasid_t pasid,
				       struct iommu_domain *domain)
{
	struct group_device *device;

	for_each_group_device(group, device) {
		if (device->dev->iommu->max_pasids > 0)
			iommu_remove_dev_pasid(device->dev, pasid, domain);
	}
}

/*
 * iommu_attach_device_pasid() - Attach a domain to pasid of device
 * @domain: the iommu domain.
 * @dev: the attached device.
 * @pasid: the pasid of the device.
 * @handle: the attach handle.
 *
 * Caller should always provide a new handle to avoid race with the paths
 * that have lockless reference to handle if it intends to pass a valid handle.
 *
 * Return: 0 on success, or an error.
 */
int iommu_attach_device_pasid(struct iommu_domain *domain,
			      struct device *dev, ioasid_t pasid,
			      struct iommu_attach_handle *handle)
{
	/* Caller must be a probed driver on dev */
	struct iommu_group *group = dev->iommu_group;
	struct group_device *device;
	const struct iommu_ops *ops;
	void *entry;
	int ret;

	if (!group)
		return -ENODEV;

	ops = dev_iommu_ops(dev);

	if (!domain->ops->set_dev_pasid ||
	    !ops->blocked_domain ||
	    !ops->blocked_domain->ops->set_dev_pasid)
		return -EOPNOTSUPP;

	if (!domain_iommu_ops_compatible(ops, domain) ||
	    pasid == IOMMU_NO_PASID)
		return -EINVAL;

	mutex_lock(&group->mutex);
	for_each_group_device(group, device) {
		/*
		 * Skip PASID validation for devices without PASID support
		 * (max_pasids = 0). These devices cannot issue transactions
		 * with PASID, so they don't affect group's PASID usage.
		 */
		if ((device->dev->iommu->max_pasids > 0) &&
		    (pasid >= device->dev->iommu->max_pasids)) {
			ret = -EINVAL;
			goto out_unlock;
		}
	}

	entry = iommu_make_pasid_array_entry(domain, handle);

	/*
	 * Entry present is a failure case. Use xa_insert() instead of
	 * xa_reserve().
	 */
	ret = xa_insert(&group->pasid_array, pasid, XA_ZERO_ENTRY, GFP_KERNEL);
	if (ret)
		goto out_unlock;

	ret = __iommu_set_group_pasid(domain, group, pasid, NULL);
	if (ret) {
		xa_release(&group->pasid_array, pasid);
		goto out_unlock;
	}

	/*
	 * The xa_insert() above reserved the memory, and the group->mutex is
	 * held, this cannot fail. The new domain cannot be visible until the
	 * operation succeeds as we cannot tolerate PRIs becoming concurrently
	 * queued and then failing attach.
	 */
	WARN_ON(xa_is_err(xa_store(&group->pasid_array,
				   pasid, entry, GFP_KERNEL)));

out_unlock:
	mutex_unlock(&group->mutex);
	return ret;
}
EXPORT_SYMBOL_GPL(iommu_attach_device_pasid);

/**
 * iommu_replace_device_pasid - Replace the domain that a specific pasid
 *                              of the device is attached to
 * @domain: the new iommu domain
 * @dev: the attached device.
 * @pasid: the pasid of the device.
 * @handle: the attach handle.
 *
 * This API allows the pasid to switch domains. The @pasid should have been
 * attached. Otherwise, this fails. The pasid will keep the old configuration
 * if replacement failed.
 *
 * Caller should always provide a new handle to avoid race with the paths
 * that have lockless reference to handle if it intends to pass a valid handle.
 *
 * Return 0 on success, or an error.
 */
int iommu_replace_device_pasid(struct iommu_domain *domain,
			       struct device *dev, ioasid_t pasid,
			       struct iommu_attach_handle *handle)
{
	/* Caller must be a probed driver on dev */
	struct iommu_group *group = dev->iommu_group;
	struct iommu_attach_handle *entry;
	struct iommu_domain *curr_domain;
	void *curr;
	int ret;

	if (!group)
		return -ENODEV;

	if (!domain->ops->set_dev_pasid)
		return -EOPNOTSUPP;

	if (!domain_iommu_ops_compatible(dev_iommu_ops(dev), domain) ||
	    pasid == IOMMU_NO_PASID || !handle)
		return -EINVAL;

	mutex_lock(&group->mutex);
	entry = iommu_make_pasid_array_entry(domain, handle);
	curr = xa_cmpxchg(&group->pasid_array, pasid, NULL,
			  XA_ZERO_ENTRY, GFP_KERNEL);
	if (xa_is_err(curr)) {
		ret = xa_err(curr);
		goto out_unlock;
	}

	/*
	 * No domain (with or without handle) attached, hence not
	 * a replace case.
	 */
	if (!curr) {
		xa_release(&group->pasid_array, pasid);
		ret = -EINVAL;
		goto out_unlock;
	}

	/*
	 * Reusing handle is problematic as there are paths that refers
	 * the handle without lock. To avoid race, reject the callers that
	 * attempt it.
	 */
	if (curr == entry) {
		WARN_ON(1);
		ret = -EINVAL;
		goto out_unlock;
	}

	curr_domain = pasid_array_entry_to_domain(curr);
	ret = 0;

	if (curr_domain != domain) {
		ret = __iommu_set_group_pasid(domain, group,
					      pasid, curr_domain);
		if (ret)
			goto out_unlock;
	}

	/*
	 * The above xa_cmpxchg() reserved the memory, and the
	 * group->mutex is held, this cannot fail.
	 */
	WARN_ON(xa_is_err(xa_store(&group->pasid_array,
				   pasid, entry, GFP_KERNEL)));

out_unlock:
	mutex_unlock(&group->mutex);
	return ret;
}
EXPORT_SYMBOL_NS_GPL(iommu_replace_device_pasid, "IOMMUFD_INTERNAL");

/*
 * iommu_detach_device_pasid() - Detach the domain from pasid of device
 * @domain: the iommu domain.
 * @dev: the attached device.
 * @pasid: the pasid of the device.
 *
 * The @domain must have been attached to @pasid of the @dev with
 * iommu_attach_device_pasid().
 */
void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev,
			       ioasid_t pasid)
{
	/* Caller must be a probed driver on dev */
	struct iommu_group *group = dev->iommu_group;

	mutex_lock(&group->mutex);
	__iommu_remove_group_pasid(group, pasid, domain);
	xa_erase(&group->pasid_array, pasid);
	mutex_unlock(&group->mutex);
}
EXPORT_SYMBOL_GPL(iommu_detach_device_pasid);

ioasid_t iommu_alloc_global_pasid(struct device *dev)
{
	int ret;

	/* max_pasids == 0 means that the device does not support PASID */
	if (!dev->iommu->max_pasids)
		return IOMMU_PASID_INVALID;

	/*
	 * max_pasids is set up by vendor driver based on number of PASID bits
	 * supported but the IDA allocation is inclusive.
	 */
	ret = ida_alloc_range(&iommu_global_pasid_ida, IOMMU_FIRST_GLOBAL_PASID,
			      dev->iommu->max_pasids - 1, GFP_KERNEL);
	return ret < 0 ? IOMMU_PASID_INVALID : ret;
}
EXPORT_SYMBOL_GPL(iommu_alloc_global_pasid);

void iommu_free_global_pasid(ioasid_t pasid)
{
	if (WARN_ON(pasid == IOMMU_PASID_INVALID))
		return;

	ida_free(&iommu_global_pasid_ida, pasid);
}
EXPORT_SYMBOL_GPL(iommu_free_global_pasid);

/**
 * iommu_attach_handle_get - Return the attach handle
 * @group: the iommu group that domain was attached to
 * @pasid: the pasid within the group
 * @type: matched domain type, 0 for any match
 *
 * Return handle or ERR_PTR(-ENOENT) on none, ERR_PTR(-EBUSY) on mismatch.
 *
 * Return the attach handle to the caller. The life cycle of an iommu attach
 * handle is from the time when the domain is attached to the time when the
 * domain is detached. Callers are required to synchronize the call of
 * iommu_attach_handle_get() with domain attachment and detachment. The attach
 * handle can only be used during its life cycle.
 */
struct iommu_attach_handle *
iommu_attach_handle_get(struct iommu_group *group, ioasid_t pasid, unsigned int type)
{
	struct iommu_attach_handle *handle;
	void *entry;

	xa_lock(&group->pasid_array);
	entry = xa_load(&group->pasid_array, pasid);
	if (!entry || xa_pointer_tag(entry) != IOMMU_PASID_ARRAY_HANDLE) {
		handle = ERR_PTR(-ENOENT);
	} else {
		handle = xa_untag_pointer(entry);
		if (type && handle->domain->type != type)
			handle = ERR_PTR(-EBUSY);
	}
	xa_unlock(&group->pasid_array);

	return handle;
}
EXPORT_SYMBOL_NS_GPL(iommu_attach_handle_get, "IOMMUFD_INTERNAL");

/**
 * iommu_attach_group_handle - Attach an IOMMU domain to an IOMMU group
 * @domain: IOMMU domain to attach
 * @group: IOMMU group that will be attached
 * @handle: attach handle
 *
 * Returns 0 on success and error code on failure.
 *
 * This is a variant of iommu_attach_group(). It allows the caller to provide
 * an attach handle and use it when the domain is attached. This is currently
 * used by IOMMUFD to deliver the I/O page faults.
 *
 * Caller should always provide a new handle to avoid race with the paths
 * that have lockless reference to handle.
 */
int iommu_attach_group_handle(struct iommu_domain *domain,
			      struct iommu_group *group,
			      struct iommu_attach_handle *handle)
{
	void *entry;
	int ret;

	if (!handle)
		return -EINVAL;

	mutex_lock(&group->mutex);
	entry = iommu_make_pasid_array_entry(domain, handle);
	ret = xa_insert(&group->pasid_array,
			IOMMU_NO_PASID, XA_ZERO_ENTRY, GFP_KERNEL);
	if (ret)
		goto out_unlock;

	ret = __iommu_attach_group(domain, group);
	if (ret) {
		xa_release(&group->pasid_array, IOMMU_NO_PASID);
		goto out_unlock;
	}

	/*
	 * The xa_insert() above reserved the memory, and the group->mutex is
	 * held, this cannot fail. The new domain cannot be visible until the
	 * operation succeeds as we cannot tolerate PRIs becoming concurrently
	 * queued and then failing attach.
	 */
	WARN_ON(xa_is_err(xa_store(&group->pasid_array,
				   IOMMU_NO_PASID, entry, GFP_KERNEL)));

out_unlock:
	mutex_unlock(&group->mutex);
	return ret;
}
EXPORT_SYMBOL_NS_GPL(iommu_attach_group_handle, "IOMMUFD_INTERNAL");

/**
 * iommu_detach_group_handle - Detach an IOMMU domain from an IOMMU group
 * @domain: IOMMU domain to attach
 * @group: IOMMU group that will be attached
 *
 * Detach the specified IOMMU domain from the specified IOMMU group.
 * It must be used in conjunction with iommu_attach_group_handle().
 */
void iommu_detach_group_handle(struct iommu_domain *domain,
			       struct iommu_group *group)
{
	mutex_lock(&group->mutex);
	__iommu_group_set_core_domain(group);
	xa_erase(&group->pasid_array, IOMMU_NO_PASID);
	mutex_unlock(&group->mutex);
}
EXPORT_SYMBOL_NS_GPL(iommu_detach_group_handle, "IOMMUFD_INTERNAL");

/**
 * iommu_replace_group_handle - replace the domain that a group is attached to
 * @group: IOMMU group that will be attached to the new domain
 * @new_domain: new IOMMU domain to replace with
 * @handle: attach handle
 *
 * This API allows the group to switch domains without being forced to go to
 * the blocking domain in-between. It allows the caller to provide an attach
 * handle for the new domain and use it when the domain is attached.
 *
 * If the currently attached domain is a core domain (e.g. a default_domain),
 * it will act just like the iommu_attach_group_handle().
 *
 * Caller should always provide a new handle to avoid race with the paths
 * that have lockless reference to handle.
 */
int iommu_replace_group_handle(struct iommu_group *group,
			       struct iommu_domain *new_domain,
			       struct iommu_attach_handle *handle)
{
	void *curr, *entry;
	int ret;

	if (!new_domain || !handle)
		return -EINVAL;

	mutex_lock(&group->mutex);
	entry = iommu_make_pasid_array_entry(new_domain, handle);
	ret = xa_reserve(&group->pasid_array, IOMMU_NO_PASID, GFP_KERNEL);
	if (ret)
		goto err_unlock;

	ret = __iommu_group_set_domain(group, new_domain);
	if (ret)
		goto err_release;

	curr = xa_store(&group->pasid_array, IOMMU_NO_PASID, entry, GFP_KERNEL);
	WARN_ON(xa_is_err(curr));

	mutex_unlock(&group->mutex);

	return 0;
err_release:
	xa_release(&group->pasid_array, IOMMU_NO_PASID);
err_unlock:
	mutex_unlock(&group->mutex);
	return ret;
}
EXPORT_SYMBOL_NS_GPL(iommu_replace_group_handle, "IOMMUFD_INTERNAL");

#if IS_ENABLED(CONFIG_IRQ_MSI_IOMMU)
/**
 * iommu_dma_prepare_msi() - Map the MSI page in the IOMMU domain
 * @desc: MSI descriptor, will store the MSI page
 * @msi_addr: MSI target address to be mapped
 *
 * The implementation of sw_msi() should take msi_addr and map it to
 * an IOVA in the domain and call msi_desc_set_iommu_msi_iova() with the
 * mapping information.
 *
 * Return: 0 on success or negative error code if the mapping failed.
 */
int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr)
{
	struct device *dev = msi_desc_to_dev(desc);
	struct iommu_group *group = dev->iommu_group;
	int ret = 0;

	if (!group)
		return 0;

	mutex_lock(&group->mutex);
	/* An IDENTITY domain must pass through */
	if (group->domain && group->domain->type != IOMMU_DOMAIN_IDENTITY) {
		switch (group->domain->cookie_type) {
		case IOMMU_COOKIE_DMA_MSI:
		case IOMMU_COOKIE_DMA_IOVA:
			ret = iommu_dma_sw_msi(group->domain, desc, msi_addr);
			break;
		case IOMMU_COOKIE_IOMMUFD:
			ret = iommufd_sw_msi(group->domain, desc, msi_addr);
			break;
		default:
			ret = -EOPNOTSUPP;
			break;
		}
	}
	mutex_unlock(&group->mutex);
	return ret;
}
#endif /* CONFIG_IRQ_MSI_IOMMU */
]

gemini-3.1-pro-preview

							input: 121454
							output: 23
							thoughts: 4361
						

**Vulnerability Discovered: IOMMU Page Table Leak Leading to Arbitrary Memory Read/Write**

Okay, here's what I've found. The core issue lies within the IOMMU mapping process during DMA buffer import, specifically triggered during the `DRM_IOCTL_PRIME_FD_TO_HANDLE` call which results in  `virtgpu_gem_prime_import` and subsequent calls down to the IOMMU level. The fault injection mechanism highlights a critical race condition within the IOMMU's mapping and unmapping routines. During `iommu_map_nosync`, when `intel_iommu_map_pages` fails to map a contiguous block of pages (due to out-of-memory or forced failures) `intel_iommu_map`'s `__domain_mapping` function fails because of a `pfn_to_dma_pte` allocation failure (e.g. `ENOMEM`), it *doesn't* unmap what it has already mapped in the loop before returning. This causes a partial mapping where some pages are successfully mapped into the IOMMU page table, while others are not, leading to a "leak" when the operation returns an error. This is because the caller (e.g., `iommu_map_nosync`) is given an incorrect `mapped = 0` value, and attempts to unmap 0 bytes.

Critically, when the same IOVA is reused later, a `WARN_ON(1)` is triggered due to the already present PTE. The existing PTE is not overwritten, so the IOMMU continues, thinking the mapping is correct. However, because pages *were* previously mapped, but are now orphaned, they are not unmapped and can be re-allocated to a different process after the original DMA buffer is freed. When a new DMA buffer is mapped at that IOVA, the *old* (now freed) physical pages are still in the IOMMU page table.  The device will then DMA into the old (and now possibly re-allocated) physical pages instead of the new intended pages! The user, controlling the DMA operations, can read or write these physical pages, leading to a use-after-free scenario and arbitrary memory access.

The initial fault injection is merely a convenient trigger, as `GFP_ATOMIC` allocations during the page mapping process are also prone to failure under memory pressure. Furthermore, `/dev/dri/card0` and `/dev/dma_heap/system` are often accessible to unprivileged users, making this vulnerability accessible without special privileges (though fault injection is still useful for testing/reproducing).  The leakage is caused by the incorrect error handling within the nested IOMMU routines which result in leaving "leaked" PTEs in the IOMMU page table, pointing to freed physical pages.

map[Expression:amd_iommu_map_pages]

map[Output:drivers/iommu/amd/iommu.c=2736=static int amd_iommu_iotlb_sync_map(struct iommu_domain *dom,
--
drivers/iommu/amd/iommu.c-2746-
drivers/iommu/amd/iommu.c:2747:static int amd_iommu_map_pages(struct iommu_domain *dom, unsigned long iova,
drivers/iommu/amd/iommu.c-2748-			       phys_addr_t paddr, size_t pgsize, size_t pgcount,
--
drivers/iommu/amd/iommu.c=3041=const struct iommu_ops amd_iommu_ops = {
--
drivers/iommu/amd/iommu.c-3056-		.attach_dev	= amd_iommu_attach_device,
drivers/iommu/amd/iommu.c:3057:		.map_pages	= amd_iommu_map_pages,
drivers/iommu/amd/iommu.c-3058-		.unmap_pages	= amd_iommu_unmap_pages,
]

gemini-3.1-pro-preview

							input: 126759
							output: 51
							thoughts: 15
						

Here's my attempt at summarizing the thought process, tailored for an expert audience and written in the first person:

**Diving into `amd_iommu_map_pages`**

Okay, so we need to take a look at `amd_iommu_map_pages`. That function is crucial for mapping physical pages, and therefore absolutely central to the IOMMU's functionality, especially when it comes to DMA. I need to refresh my understanding of how it handles the address translation and protection. I'll need to see how it interacts with the page tables, what kind of locking is employed, and how it deals with the potential for page faults during the mapping process. Is it using any prefetching or optimization for large mappings? Are there any obvious areas for potential bottlenecks or race conditions? Time to crack open the code and start digging. I'll focus on the data structures it manipulates, specifically the page tables and any internal state related to the mapping entries. I'll be looking for anything out of the ordinary, and hoping to spot some interesting tidbits. I also want to consider how this function would behave in concurrent environments, since IOMMU operations may involve multiple CPUs. Let's see if this function is as robust as I expect it to be.

requested entity does not exist

map[ContextFile:drivers/iommu/amd/iommu.c IncludeLines:true Name:amd_iommu_map_pages]

map[SourceCode: SourceFile:]

gemini-3.1-pro-preview

							input: 127112
							output: 23
							thoughts: 27
						

Here's a summary of my thought process, as requested:

**Diving into the AMD IOMMU Code**

Okay, so we're talking AMD IOMMU, and I need to get a handle on something specific. My mental map is guiding me towards `drivers/iommu/amd/iommu.c`. The core functionality must be around that file. Let's start with a look around line 2747. I'm expecting to find some crucial logic there, probably related to a specific configuration option, interrupt handling, or maybe a DMA operation sequence. I'm not entirely sure *what* I'll find yet, but I'm betting that area of the code holds the key to answering the question at hand. Time to pull up the source and dig in. I will likely be looking at the context surrounding that line as well - function definitions, variable declarations, and surrounding control flow. My years of experience should allow me to pick out the important bits quickly.

map[Expression:amd_iommu_map_pages]

map[Output:drivers/iommu/amd/iommu.c=2736=static int amd_iommu_iotlb_sync_map(struct iommu_domain *dom,
--
drivers/iommu/amd/iommu.c-2746-
drivers/iommu/amd/iommu.c:2747:static int amd_iommu_map_pages(struct iommu_domain *dom, unsigned long iova,
drivers/iommu/amd/iommu.c-2748-			       phys_addr_t paddr, size_t pgsize, size_t pgcount,
--
drivers/iommu/amd/iommu.c=3041=const struct iommu_ops amd_iommu_ops = {
--
drivers/iommu/amd/iommu.c-3056-		.attach_dev	= amd_iommu_attach_device,
drivers/iommu/amd/iommu.c:3057:		.map_pages	= amd_iommu_map_pages,
drivers/iommu/amd/iommu.c-3058-		.unmap_pages	= amd_iommu_unmap_pages,
]

gemini-3.1-pro-preview

							input: 127691
							output: 25
							thoughts: 31
						

Here's my analysis of the provided text, rendered in the first person, assuming an expert understanding:

**Deep Dive into AMD IOMMU Code (drivers/iommu/amd/iommu.c:2740-2800)**

Okay, let's get down to brass tacks. I need to take a look at the AMD IOMMU driver code, specifically within `drivers/iommu/amd/iommu.c`, and I'm targeting lines 2740 through 2800. This is probably where some critical interaction or specific functionality resides. Just the line range already tells me this is an area of significant implementation. I'm hoping to understand a particular piece of logic, maybe related to device assignment, interrupt handling, or even a nuanced aspect of the translation process. Based on my experience, I'm expecting to find a conditional block or a loop that is doing something quite granular. I may look for error handling, or perhaps a section where memory barriers are carefully placed. I'll need to pay close attention to the data structures involved, the variables being manipulated, and the overall control flow. This specific chunk of code is likely critical to performance or stability. Let's see what gems are hidden in there!

map[Expression:static int amd_iommu_map_pages]

map[Output:drivers/iommu/amd/iommu.c=2736=static int amd_iommu_iotlb_sync_map(struct iommu_domain *dom,
--
drivers/iommu/amd/iommu.c-2746-
drivers/iommu/amd/iommu.c:2747:static int amd_iommu_map_pages(struct iommu_domain *dom, unsigned long iova,
drivers/iommu/amd/iommu.c-2748-			       phys_addr_t paddr, size_t pgsize, size_t pgcount,
]

gemini-3.1-pro-preview

							input: 128182
							output: 30
							thoughts: 13
						

Okay, here's my attempt at summarizing the thought process, as requested, aimed at an expert audience and in the first person:

**Analyzing the Function with Grepper**

Right, let's dive in. My immediate thought is to use `grepper` to get a quick and dirty understanding of this function. Given my expertise, I don't want to get bogged down in manual scrolling or searching through the source code. `grepper` is the perfect tool for a rapid overview. I want to quickly identify its key components, any obvious data flow, and perhaps any external dependencies. My goal isn't necessarily a line-by-line understanding at this stage, but rather a high-level comprehension of the function's purpose and how it operates within the larger system.

map[Expression:amd_iommu_map_pages\(struct iommu_domain]

map[Output:drivers/iommu/amd/iommu.c=2736=static int amd_iommu_iotlb_sync_map(struct iommu_domain *dom,
--
drivers/iommu/amd/iommu.c-2746-
drivers/iommu/amd/iommu.c:2747:static int amd_iommu_map_pages(struct iommu_domain *dom, unsigned long iova,
drivers/iommu/amd/iommu.c-2748-			       phys_addr_t paddr, size_t pgsize, size_t pgcount,
]

gemini-3.1-pro-preview

							input: 128553
							output: 28
							thoughts: 25
						

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Diving into the AMD IOMMU Driver**

Alright, let's pull up `drivers/iommu/amd/iommu.c` and take a look. I'm especially interested in... well, everything, but specifically how AMD handles IOMMU initialization and page table management. I want to quickly refresh my memory on the basic data structures, like the `struct amd_iommu` and the associated DMA remapping structures. I'm keen to trace the interrupt handling flow, because I suspect some interesting corner cases there, especially around nested virtualization. I'll need to confirm how they deal with the various platform quirks that always seem to creep in with AMD's IOMMUs. Specifically, I'm thinking about the implications of the different interrupt remapping implementations and how the driver deals with them. I'll be looking for any hints on their SMMU emulation, if it's there. I'll be checking to see what DMA-API interfaces they use and ensure there's no subtle corruption happening. From a security perspective, I want to double-check their address space isolation implementation, especially with respect to the root of trust, and how it's handled. Then I want to check for any potential race conditions that might have snuck in during DMA operations.

map[File:drivers/iommu/amd/iommu.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
 * Copyright (C) 2007-2010 Advanced Micro Devices, Inc.
 * Author: Joerg Roedel <jroedel@suse.de>
 *         Leo Duran <leo.duran@amd.com>
 */

#define pr_fmt(fmt)     "AMD-Vi: " fmt
#define dev_fmt(fmt)    pr_fmt(fmt)

#include <linux/ratelimit.h>
#include <linux/pci.h>
#include <linux/acpi.h>
#include <linux/pci-ats.h>
#include <linux/bitmap.h>
#include <linux/slab.h>
#include <linux/debugfs.h>
#include <linux/scatterlist.h>
#include <linux/dma-map-ops.h>
#include <linux/dma-direct.h>
#include <linux/idr.h>
#include <linux/iommu-helper.h>
#include <linux/delay.h>
#include <linux/amd-iommu.h>
#include <linux/notifier.h>
#include <linux/export.h>
#include <linux/irq.h>
#include <linux/irqchip/irq-msi-lib.h>
#include <linux/msi.h>
#include <linux/irqdomain.h>
#include <linux/percpu.h>
#include <linux/io-pgtable.h>
#include <linux/cc_platform.h>
#include <asm/irq_remapping.h>
#include <asm/io_apic.h>
#include <asm/apic.h>
#include <asm/hw_irq.h>
#include <asm/proto.h>
#include <asm/iommu.h>
#include <asm/gart.h>
#include <asm/dma.h>
#include <uapi/linux/iommufd.h>

#include "amd_iommu.h"
#include "../dma-iommu.h"
#include "../irq_remapping.h"
#include "../iommu-pages.h"

#define CMD_SET_TYPE(cmd, t) ((cmd)->data[1] |= ((t) << 28))

/* Reserved IOVA ranges */
#define MSI_RANGE_START		(0xfee00000)
#define MSI_RANGE_END		(0xfeefffff)
#define HT_RANGE_START		(0xfd00000000ULL)
#define HT_RANGE_END		(0xffffffffffULL)

LIST_HEAD(ioapic_map);
LIST_HEAD(hpet_map);
LIST_HEAD(acpihid_map);

const struct iommu_ops amd_iommu_ops;
static const struct iommu_dirty_ops amd_dirty_ops;

int amd_iommu_max_glx_val = -1;

/*
 * AMD IOMMU allows up to 2^16 different protection domains. This is a bitmap
 * to know which ones are already in use.
 */
DEFINE_IDA(pdom_ids);

static int amd_iommu_attach_device(struct iommu_domain *dom,
				   struct device *dev);

static void set_dte_entry(struct amd_iommu *iommu,
			  struct iommu_dev_data *dev_data);

static void iommu_flush_dte_sync(struct amd_iommu *iommu, u16 devid);

static struct iommu_dev_data *find_dev_data(struct amd_iommu *iommu, u16 devid);

/****************************************************************************
 *
 * Helper functions
 *
 ****************************************************************************/

static __always_inline void amd_iommu_atomic128_set(__int128 *ptr, __int128 val)
{
	/*
	 * Note:
	 * We use arch_cmpxchg128_local() because:
	 * - Need cmpxchg16b instruction mainly for 128-bit store to DTE
	 *   (not necessary for cmpxchg since this function is already
	 *   protected by a spin_lock for this DTE).
	 * - Neither need LOCK_PREFIX nor try loop because of the spin_lock.
	 */
	arch_cmpxchg128_local(ptr, *ptr, val);
}

static void write_dte_upper128(struct dev_table_entry *ptr, struct dev_table_entry *new)
{
	struct dev_table_entry old;

	old.data128[1] = ptr->data128[1];
	/*
	 * Preserve DTE_DATA2_INTR_MASK. This needs to be
	 * done here since it requires to be inside
	 * spin_lock(&dev_data->dte_lock) context.
	 */
	new->data[2] &= ~DTE_DATA2_INTR_MASK;
	new->data[2] |= old.data[2] & DTE_DATA2_INTR_MASK;

	amd_iommu_atomic128_set(&ptr->data128[1], new->data128[1]);
}

static void write_dte_lower128(struct dev_table_entry *ptr, struct dev_table_entry *new)
{
	amd_iommu_atomic128_set(&ptr->data128[0], new->data128[0]);
}

/*
 * Note:
 * IOMMU reads the entire Device Table entry in a single 256-bit transaction
 * but the driver is programming DTE using 2 128-bit cmpxchg. So, the driver
 * need to ensure the following:
 *   - DTE[V|GV] bit is being written last when setting.
 *   - DTE[V|GV] bit is being written first when clearing.
 *
 * This function is used only by code, which updates DMA translation part of the DTE.
 * So, only consider control bits related to DMA when updating the entry.
 */
static void update_dte256(struct amd_iommu *iommu, struct iommu_dev_data *dev_data,
			  struct dev_table_entry *new)
{
	unsigned long flags;
	struct dev_table_entry *dev_table = get_dev_table(iommu);
	struct dev_table_entry *ptr = &dev_table[dev_data->devid];

	spin_lock_irqsave(&dev_data->dte_lock, flags);

	if (!(ptr->data[0] & DTE_FLAG_V)) {
		/* Existing DTE is not valid. */
		write_dte_upper128(ptr, new);
		write_dte_lower128(ptr, new);
		iommu_flush_dte_sync(iommu, dev_data->devid);
	} else if (!(new->data[0] & DTE_FLAG_V)) {
		/* Existing DTE is valid. New DTE is not valid.  */
		write_dte_lower128(ptr, new);
		write_dte_upper128(ptr, new);
		iommu_flush_dte_sync(iommu, dev_data->devid);
	} else if (!FIELD_GET(DTE_FLAG_GV, ptr->data[0])) {
		/*
		 * Both DTEs are valid.
		 * Existing DTE has no guest page table.
		 */
		write_dte_upper128(ptr, new);
		write_dte_lower128(ptr, new);
		iommu_flush_dte_sync(iommu, dev_data->devid);
	} else if (!FIELD_GET(DTE_FLAG_GV, new->data[0])) {
		/*
		 * Both DTEs are valid.
		 * Existing DTE has guest page table,
		 * new DTE has no guest page table,
		 */
		write_dte_lower128(ptr, new);
		write_dte_upper128(ptr, new);
		iommu_flush_dte_sync(iommu, dev_data->devid);
	} else if (FIELD_GET(DTE_GPT_LEVEL_MASK, ptr->data[2]) !=
		   FIELD_GET(DTE_GPT_LEVEL_MASK, new->data[2])) {
		/*
		 * Both DTEs are valid and have guest page table,
		 * but have different number of levels. So, we need
		 * to upadte both upper and lower 128-bit value, which
		 * require disabling and flushing.
		 */
		struct dev_table_entry clear = {};

		/* First disable DTE */
		write_dte_lower128(ptr, &clear);
		iommu_flush_dte_sync(iommu, dev_data->devid);

		/* Then update DTE */
		write_dte_upper128(ptr, new);
		write_dte_lower128(ptr, new);
		iommu_flush_dte_sync(iommu, dev_data->devid);
	} else {
		/*
		 * Both DTEs are valid and have guest page table,
		 * and same number of levels. We just need to only
		 * update the lower 128-bit. So no need to disable DTE.
		 */
		write_dte_lower128(ptr, new);
	}

	spin_unlock_irqrestore(&dev_data->dte_lock, flags);
}

static void get_dte256(struct amd_iommu *iommu, struct iommu_dev_data *dev_data,
		      struct dev_table_entry *dte)
{
	unsigned long flags;
	struct dev_table_entry *ptr;
	struct dev_table_entry *dev_table = get_dev_table(iommu);

	ptr = &dev_table[dev_data->devid];

	spin_lock_irqsave(&dev_data->dte_lock, flags);
	dte->data128[0] = ptr->data128[0];
	dte->data128[1] = ptr->data128[1];
	spin_unlock_irqrestore(&dev_data->dte_lock, flags);
}

static inline bool pdom_is_v2_pgtbl_mode(struct protection_domain *pdom)
{
	return (pdom && (pdom->pd_mode == PD_MODE_V2));
}

static inline bool pdom_is_in_pt_mode(struct protection_domain *pdom)
{
	return (pdom->domain.type == IOMMU_DOMAIN_IDENTITY);
}

/*
 * We cannot support PASID w/ existing v1 page table in the same domain
 * since it will be nested. However, existing domain w/ v2 page table
 * or passthrough mode can be used for PASID.
 */
static inline bool pdom_is_sva_capable(struct protection_domain *pdom)
{
	return pdom_is_v2_pgtbl_mode(pdom) || pdom_is_in_pt_mode(pdom);
}

static inline int get_acpihid_device_id(struct device *dev,
					struct acpihid_map_entry **entry)
{
	struct acpi_device *adev = ACPI_COMPANION(dev);
	struct acpihid_map_entry *p, *p1 = NULL;
	int hid_count = 0;
	bool fw_bug;

	if (!adev)
		return -ENODEV;

	list_for_each_entry(p, &acpihid_map, list) {
		if (acpi_dev_hid_uid_match(adev, p->hid,
					   p->uid[0] ? p->uid : NULL)) {
			p1 = p;
			fw_bug = false;
			hid_count = 1;
			break;
		}

		/*
		 * Count HID matches w/o UID, raise FW_BUG but allow exactly one match
		 */
		if (acpi_dev_hid_match(adev, p->hid)) {
			p1 = p;
			hid_count++;
			fw_bug = true;
		}
	}

	if (!p1)
		return -EINVAL;
	if (fw_bug)
		dev_err_once(dev, FW_BUG "No ACPI device matched UID, but %d device%s matched HID.\n",
			     hid_count, hid_count > 1 ? "s" : "");
	if (hid_count > 1)
		return -EINVAL;
	if (entry)
		*entry = p1;

	return p1->devid;
}

static inline int get_device_sbdf_id(struct device *dev)
{
	int sbdf;

	if (dev_is_pci(dev))
		sbdf = get_pci_sbdf_id(to_pci_dev(dev));
	else
		sbdf = get_acpihid_device_id(dev, NULL);

	return sbdf;
}

struct dev_table_entry *get_dev_table(struct amd_iommu *iommu)
{
	struct dev_table_entry *dev_table;
	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;

	BUG_ON(pci_seg == NULL);
	dev_table = pci_seg->dev_table;
	BUG_ON(dev_table == NULL);

	return dev_table;
}

static inline u16 get_device_segment(struct device *dev)
{
	u16 seg;

	if (dev_is_pci(dev)) {
		struct pci_dev *pdev = to_pci_dev(dev);

		seg = pci_domain_nr(pdev->bus);
	} else {
		u32 devid = get_acpihid_device_id(dev, NULL);

		seg = PCI_SBDF_TO_SEGID(devid);
	}

	return seg;
}

/* Writes the specific IOMMU for a device into the PCI segment rlookup table */
void amd_iommu_set_rlookup_table(struct amd_iommu *iommu, u16 devid)
{
	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;

	pci_seg->rlookup_table[devid] = iommu;
}

static struct amd_iommu *__rlookup_amd_iommu(u16 seg, u16 devid)
{
	struct amd_iommu_pci_seg *pci_seg;

	for_each_pci_segment(pci_seg) {
		if (pci_seg->id == seg)
			return pci_seg->rlookup_table[devid];
	}
	return NULL;
}

static struct amd_iommu *rlookup_amd_iommu(struct device *dev)
{
	u16 seg = get_device_segment(dev);
	int devid = get_device_sbdf_id(dev);

	if (devid < 0)
		return NULL;
	return __rlookup_amd_iommu(seg, PCI_SBDF_TO_DEVID(devid));
}

static struct iommu_dev_data *alloc_dev_data(struct amd_iommu *iommu, u16 devid)
{
	struct iommu_dev_data *dev_data;
	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;

	dev_data = kzalloc(sizeof(*dev_data), GFP_KERNEL);
	if (!dev_data)
		return NULL;

	mutex_init(&dev_data->mutex);
	spin_lock_init(&dev_data->dte_lock);
	dev_data->devid = devid;
	ratelimit_default_init(&dev_data->rs);

	llist_add(&dev_data->dev_data_list, &pci_seg->dev_data_list);
	return dev_data;
}

struct iommu_dev_data *search_dev_data(struct amd_iommu *iommu, u16 devid)
{
	struct iommu_dev_data *dev_data;
	struct llist_node *node;
	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;

	if (llist_empty(&pci_seg->dev_data_list))
		return NULL;

	node = pci_seg->dev_data_list.first;
	llist_for_each_entry(dev_data, node, dev_data_list) {
		if (dev_data->devid == devid)
			return dev_data;
	}

	return NULL;
}

static int clone_alias(struct pci_dev *pdev, u16 alias, void *data)
{
	struct dev_table_entry new;
	struct amd_iommu *iommu;
	struct iommu_dev_data *dev_data, *alias_data;
	u16 devid = pci_dev_id(pdev);
	int ret = 0;

	if (devid == alias)
		return 0;

	iommu = rlookup_amd_iommu(&pdev->dev);
	if (!iommu)
		return 0;

	/* Copy the data from pdev */
	dev_data = dev_iommu_priv_get(&pdev->dev);
	if (!dev_data) {
		pr_err("%s : Failed to get dev_data for 0x%x\n", __func__, devid);
		ret = -EINVAL;
		goto out;
	}
	get_dte256(iommu, dev_data, &new);

	/* Setup alias */
	alias_data = find_dev_data(iommu, alias);
	if (!alias_data) {
		pr_err("%s : Failed to get alias dev_data for 0x%x\n", __func__, alias);
		ret = -EINVAL;
		goto out;
	}
	update_dte256(iommu, alias_data, &new);

	amd_iommu_set_rlookup_table(iommu, alias);
out:
	return ret;
}

static void clone_aliases(struct amd_iommu *iommu, struct device *dev)
{
	struct pci_dev *pdev;

	if (!dev_is_pci(dev))
		return;
	pdev = to_pci_dev(dev);

	/*
	 * The IVRS alias stored in the alias table may not be
	 * part of the PCI DMA aliases if it's bus differs
	 * from the original device.
	 */
	clone_alias(pdev, iommu->pci_seg->alias_table[pci_dev_id(pdev)], NULL);

	pci_for_each_dma_alias(pdev, clone_alias, NULL);
}

static void setup_aliases(struct amd_iommu *iommu, struct device *dev)
{
	struct pci_dev *pdev = to_pci_dev(dev);
	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;
	u16 ivrs_alias;

	/* For ACPI HID devices, there are no aliases */
	if (!dev_is_pci(dev))
		return;

	/*
	 * Add the IVRS alias to the pci aliases if it is on the same
	 * bus. The IVRS table may know about a quirk that we don't.
	 */
	ivrs_alias = pci_seg->alias_table[pci_dev_id(pdev)];
	if (ivrs_alias != pci_dev_id(pdev) &&
	    PCI_BUS_NUM(ivrs_alias) == pdev->bus->number)
		pci_add_dma_alias(pdev, ivrs_alias & 0xff, 1);

	clone_aliases(iommu, dev);
}

static struct iommu_dev_data *find_dev_data(struct amd_iommu *iommu, u16 devid)
{
	struct iommu_dev_data *dev_data;

	dev_data = search_dev_data(iommu, devid);

	if (dev_data == NULL) {
		dev_data = alloc_dev_data(iommu, devid);
		if (!dev_data)
			return NULL;

		if (translation_pre_enabled(iommu))
			dev_data->defer_attach = true;
	}

	return dev_data;
}

/*
* Find or create an IOMMU group for a acpihid device.
*/
static struct iommu_group *acpihid_device_group(struct device *dev)
{
	struct acpihid_map_entry *p, *entry = NULL;
	int devid;

	devid = get_acpihid_device_id(dev, &entry);
	if (devid < 0)
		return ERR_PTR(devid);

	list_for_each_entry(p, &acpihid_map, list) {
		if ((devid == p->devid) && p->group)
			entry->group = p->group;
	}

	if (!entry->group)
		entry->group = generic_device_group(dev);
	else
		iommu_group_ref_get(entry->group);

	return entry->group;
}

static inline bool pdev_pasid_supported(struct iommu_dev_data *dev_data)
{
	return (dev_data->flags & AMD_IOMMU_DEVICE_FLAG_PASID_SUP);
}

static u32 pdev_get_caps(struct pci_dev *pdev)
{
	int features;
	u32 flags = 0;

	if (pci_ats_supported(pdev))
		flags |= AMD_IOMMU_DEVICE_FLAG_ATS_SUP;

	if (pci_pri_supported(pdev))
		flags |= AMD_IOMMU_DEVICE_FLAG_PRI_SUP;

	features = pci_pasid_features(pdev);
	if (features >= 0) {
		flags |= AMD_IOMMU_DEVICE_FLAG_PASID_SUP;

		if (features & PCI_PASID_CAP_EXEC)
			flags |= AMD_IOMMU_DEVICE_FLAG_EXEC_SUP;

		if (features & PCI_PASID_CAP_PRIV)
			flags |= AMD_IOMMU_DEVICE_FLAG_PRIV_SUP;
	}

	return flags;
}

static inline int pdev_enable_cap_ats(struct pci_dev *pdev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
	int ret = -EINVAL;

	if (dev_data->ats_enabled)
		return 0;

	if (amd_iommu_iotlb_sup &&
	    (dev_data->flags & AMD_IOMMU_DEVICE_FLAG_ATS_SUP)) {
		ret = pci_enable_ats(pdev, PAGE_SHIFT);
		if (!ret) {
			dev_data->ats_enabled = 1;
			dev_data->ats_qdep    = pci_ats_queue_depth(pdev);
		}
	}

	return ret;
}

static inline void pdev_disable_cap_ats(struct pci_dev *pdev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);

	if (dev_data->ats_enabled) {
		pci_disable_ats(pdev);
		dev_data->ats_enabled = 0;
	}
}

static inline int pdev_enable_cap_pri(struct pci_dev *pdev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
	int ret = -EINVAL;

	if (dev_data->pri_enabled)
		return 0;

	if (!dev_data->ats_enabled)
		return 0;

	if (dev_data->flags & AMD_IOMMU_DEVICE_FLAG_PRI_SUP) {
		/*
		 * First reset the PRI state of the device.
		 * FIXME: Hardcode number of outstanding requests for now
		 */
		if (!pci_reset_pri(pdev) && !pci_enable_pri(pdev, 32)) {
			dev_data->pri_enabled = 1;
			dev_data->pri_tlp     = pci_prg_resp_pasid_required(pdev);

			ret = 0;
		}
	}

	return ret;
}

static inline void pdev_disable_cap_pri(struct pci_dev *pdev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);

	if (dev_data->pri_enabled) {
		pci_disable_pri(pdev);
		dev_data->pri_enabled = 0;
	}
}

static inline int pdev_enable_cap_pasid(struct pci_dev *pdev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);
	int ret = -EINVAL;

	if (dev_data->pasid_enabled)
		return 0;

	if (dev_data->flags & AMD_IOMMU_DEVICE_FLAG_PASID_SUP) {
		/* Only allow access to user-accessible pages */
		ret = pci_enable_pasid(pdev, 0);
		if (!ret)
			dev_data->pasid_enabled = 1;
	}

	return ret;
}

static inline void pdev_disable_cap_pasid(struct pci_dev *pdev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(&pdev->dev);

	if (dev_data->pasid_enabled) {
		pci_disable_pasid(pdev);
		dev_data->pasid_enabled = 0;
	}
}

static void pdev_enable_caps(struct pci_dev *pdev)
{
	pdev_enable_cap_pasid(pdev);
	pdev_enable_cap_ats(pdev);
	pdev_enable_cap_pri(pdev);
}

static void pdev_disable_caps(struct pci_dev *pdev)
{
	pdev_disable_cap_ats(pdev);
	pdev_disable_cap_pasid(pdev);
	pdev_disable_cap_pri(pdev);
}

/*
 * This function checks if the driver got a valid device from the caller to
 * avoid dereferencing invalid pointers.
 */
static bool check_device(struct device *dev)
{
	struct amd_iommu_pci_seg *pci_seg;
	struct amd_iommu *iommu;
	int devid, sbdf;

	if (!dev)
		return false;

	sbdf = get_device_sbdf_id(dev);
	if (sbdf < 0)
		return false;
	devid = PCI_SBDF_TO_DEVID(sbdf);

	iommu = rlookup_amd_iommu(dev);
	if (!iommu)
		return false;

	/* Out of our scope? */
	pci_seg = iommu->pci_seg;
	if (devid > pci_seg->last_bdf)
		return false;

	return true;
}

static int iommu_init_device(struct amd_iommu *iommu, struct device *dev)
{
	struct iommu_dev_data *dev_data;
	int devid, sbdf;

	if (dev_iommu_priv_get(dev))
		return 0;

	sbdf = get_device_sbdf_id(dev);
	if (sbdf < 0)
		return sbdf;

	devid = PCI_SBDF_TO_DEVID(sbdf);
	dev_data = find_dev_data(iommu, devid);
	if (!dev_data)
		return -ENOMEM;

	dev_data->dev = dev;

	/*
	 * The dev_iommu_priv_set() needes to be called before setup_aliases.
	 * Otherwise, subsequent call to dev_iommu_priv_get() will fail.
	 */
	dev_iommu_priv_set(dev, dev_data);
	setup_aliases(iommu, dev);

	/*
	 * By default we use passthrough mode for IOMMUv2 capable device.
	 * But if amd_iommu=force_isolation is set (e.g. to debug DMA to
	 * invalid address), we ignore the capability for the device so
	 * it'll be forced to go into translation mode.
	 */
	if ((iommu_default_passthrough() || !amd_iommu_force_isolation) &&
	    dev_is_pci(dev) && amd_iommu_gt_ppr_supported()) {
		dev_data->flags = pdev_get_caps(to_pci_dev(dev));
	}

	return 0;
}

static void iommu_ignore_device(struct amd_iommu *iommu, struct device *dev)
{
	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;
	struct dev_table_entry *dev_table = get_dev_table(iommu);
	int devid, sbdf;

	sbdf = get_device_sbdf_id(dev);
	if (sbdf < 0)
		return;

	devid = PCI_SBDF_TO_DEVID(sbdf);
	pci_seg->rlookup_table[devid] = NULL;
	memset(&dev_table[devid], 0, sizeof(struct dev_table_entry));

	setup_aliases(iommu, dev);
}


/****************************************************************************
 *
 * Interrupt handling functions
 *
 ****************************************************************************/

static void dump_dte_entry(struct amd_iommu *iommu, u16 devid)
{
	int i;
	struct dev_table_entry dte;
	struct iommu_dev_data *dev_data = find_dev_data(iommu, devid);

	get_dte256(iommu, dev_data, &dte);

	for (i = 0; i < 4; ++i)
		pr_err("DTE[%d]: %016llx\n", i, dte.data[i]);
}

static void dump_command(unsigned long phys_addr)
{
	struct iommu_cmd *cmd = iommu_phys_to_virt(phys_addr);
	int i;

	for (i = 0; i < 4; ++i)
		pr_err("CMD[%d]: %08x\n", i, cmd->data[i]);
}

static void amd_iommu_report_rmp_hw_error(struct amd_iommu *iommu, volatile u32 *event)
{
	struct iommu_dev_data *dev_data = NULL;
	int devid, vmg_tag, flags;
	struct pci_dev *pdev;
	u64 spa;

	devid   = (event[0] >> EVENT_DEVID_SHIFT) & EVENT_DEVID_MASK;
	vmg_tag = (event[1]) & 0xFFFF;
	flags   = (event[1] >> EVENT_FLAGS_SHIFT) & EVENT_FLAGS_MASK;
	spa     = ((u64)event[3] << 32) | (event[2] & 0xFFFFFFF8);

	pdev = pci_get_domain_bus_and_slot(iommu->pci_seg->id, PCI_BUS_NUM(devid),
					   devid & 0xff);
	if (pdev)
		dev_data = dev_iommu_priv_get(&pdev->dev);

	if (dev_data) {
		if (__ratelimit(&dev_data->rs)) {
			pci_err(pdev, "Event logged [RMP_HW_ERROR vmg_tag=0x%04x, spa=0x%llx, flags=0x%04x]\n",
				vmg_tag, spa, flags);
		}
	} else {
		pr_err_ratelimited("Event logged [RMP_HW_ERROR device=%04x:%02x:%02x.%x, vmg_tag=0x%04x, spa=0x%llx, flags=0x%04x]\n",
			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
			vmg_tag, spa, flags);
	}

	if (pdev)
		pci_dev_put(pdev);
}

static void amd_iommu_report_rmp_fault(struct amd_iommu *iommu, volatile u32 *event)
{
	struct iommu_dev_data *dev_data = NULL;
	int devid, flags_rmp, vmg_tag, flags;
	struct pci_dev *pdev;
	u64 gpa;

	devid     = (event[0] >> EVENT_DEVID_SHIFT) & EVENT_DEVID_MASK;
	flags_rmp = (event[0] >> EVENT_FLAGS_SHIFT) & 0xFF;
	vmg_tag   = (event[1]) & 0xFFFF;
	flags     = (event[1] >> EVENT_FLAGS_SHIFT) & EVENT_FLAGS_MASK;
	gpa       = ((u64)event[3] << 32) | event[2];

	pdev = pci_get_domain_bus_and_slot(iommu->pci_seg->id, PCI_BUS_NUM(devid),
					   devid & 0xff);
	if (pdev)
		dev_data = dev_iommu_priv_get(&pdev->dev);

	if (dev_data) {
		if (__ratelimit(&dev_data->rs)) {
			pci_err(pdev, "Event logged [RMP_PAGE_FAULT vmg_tag=0x%04x, gpa=0x%llx, flags_rmp=0x%04x, flags=0x%04x]\n",
				vmg_tag, gpa, flags_rmp, flags);
		}
	} else {
		pr_err_ratelimited("Event logged [RMP_PAGE_FAULT device=%04x:%02x:%02x.%x, vmg_tag=0x%04x, gpa=0x%llx, flags_rmp=0x%04x, flags=0x%04x]\n",
			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
			vmg_tag, gpa, flags_rmp, flags);
	}

	if (pdev)
		pci_dev_put(pdev);
}

#define IS_IOMMU_MEM_TRANSACTION(flags)		\
	(((flags) & EVENT_FLAG_I) == 0)

#define IS_WRITE_REQUEST(flags)			\
	((flags) & EVENT_FLAG_RW)

static void amd_iommu_report_page_fault(struct amd_iommu *iommu,
					u16 devid, u16 domain_id,
					u64 address, int flags)
{
	struct iommu_dev_data *dev_data = NULL;
	struct pci_dev *pdev;

	pdev = pci_get_domain_bus_and_slot(iommu->pci_seg->id, PCI_BUS_NUM(devid),
					   devid & 0xff);
	if (pdev)
		dev_data = dev_iommu_priv_get(&pdev->dev);

	if (dev_data) {
		/*
		 * If this is a DMA fault (for which the I(nterrupt)
		 * bit will be unset), allow report_iommu_fault() to
		 * prevent logging it.
		 */
		if (IS_IOMMU_MEM_TRANSACTION(flags)) {
			/* Device not attached to domain properly */
			if (dev_data->domain == NULL) {
				pr_err_ratelimited("Event logged [Device not attached to domain properly]\n");
				pr_err_ratelimited("  device=%04x:%02x:%02x.%x domain=0x%04x\n",
						   iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid),
						   PCI_FUNC(devid), domain_id);
				goto out;
			}

			if (!report_iommu_fault(&dev_data->domain->domain,
						&pdev->dev, address,
						IS_WRITE_REQUEST(flags) ?
							IOMMU_FAULT_WRITE :
							IOMMU_FAULT_READ))
				goto out;
		}

		if (__ratelimit(&dev_data->rs)) {
			pci_err(pdev, "Event logged [IO_PAGE_FAULT domain=0x%04x address=0x%llx flags=0x%04x]\n",
				domain_id, address, flags);
		}
	} else {
		pr_err_ratelimited("Event logged [IO_PAGE_FAULT device=%04x:%02x:%02x.%x domain=0x%04x address=0x%llx flags=0x%04x]\n",
			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
			domain_id, address, flags);
	}

out:
	if (pdev)
		pci_dev_put(pdev);
}

static void iommu_print_event(struct amd_iommu *iommu, void *__evt)
{
	struct device *dev = iommu->iommu.dev;
	int type, devid, flags, tag;
	volatile u32 *event = __evt;
	int count = 0;
	u64 address, ctrl;
	u32 pasid;

retry:
	type    = (event[1] >> EVENT_TYPE_SHIFT)  & EVENT_TYPE_MASK;
	devid   = (event[0] >> EVENT_DEVID_SHIFT) & EVENT_DEVID_MASK;
	pasid   = (event[0] & EVENT_DOMID_MASK_HI) |
		  (event[1] & EVENT_DOMID_MASK_LO);
	flags   = (event[1] >> EVENT_FLAGS_SHIFT) & EVENT_FLAGS_MASK;
	address = (u64)(((u64)event[3]) << 32) | event[2];
	ctrl    = readq(iommu->mmio_base + MMIO_CONTROL_OFFSET);

	if (type == 0) {
		/* Did we hit the erratum? */
		if (++count == LOOP_TIMEOUT) {
			pr_err("No event written to event log\n");
			return;
		}
		udelay(1);
		goto retry;
	}

	if (type == EVENT_TYPE_IO_FAULT) {
		amd_iommu_report_page_fault(iommu, devid, pasid, address, flags);
		return;
	}

	switch (type) {
	case EVENT_TYPE_ILL_DEV:
		dev_err(dev, "Event logged [ILLEGAL_DEV_TABLE_ENTRY device=%04x:%02x:%02x.%x pasid=0x%05x address=0x%llx flags=0x%04x]\n",
			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
			pasid, address, flags);
		dev_err(dev, "Control Reg : 0x%llx\n", ctrl);
		dump_dte_entry(iommu, devid);
		break;
	case EVENT_TYPE_DEV_TAB_ERR:
		dev_err(dev, "Event logged [DEV_TAB_HARDWARE_ERROR device=%04x:%02x:%02x.%x "
			"address=0x%llx flags=0x%04x]\n",
			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
			address, flags);
		break;
	case EVENT_TYPE_PAGE_TAB_ERR:
		dev_err(dev, "Event logged [PAGE_TAB_HARDWARE_ERROR device=%04x:%02x:%02x.%x pasid=0x%04x address=0x%llx flags=0x%04x]\n",
			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
			pasid, address, flags);
		break;
	case EVENT_TYPE_ILL_CMD:
		dev_err(dev, "Event logged [ILLEGAL_COMMAND_ERROR address=0x%llx]\n", address);
		dump_command(address);
		break;
	case EVENT_TYPE_CMD_HARD_ERR:
		dev_err(dev, "Event logged [COMMAND_HARDWARE_ERROR address=0x%llx flags=0x%04x]\n",
			address, flags);
		break;
	case EVENT_TYPE_IOTLB_INV_TO:
		dev_err(dev, "Event logged [IOTLB_INV_TIMEOUT device=%04x:%02x:%02x.%x address=0x%llx]\n",
			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
			address);
		break;
	case EVENT_TYPE_INV_DEV_REQ:
		dev_err(dev, "Event logged [INVALID_DEVICE_REQUEST device=%04x:%02x:%02x.%x pasid=0x%05x address=0x%llx flags=0x%04x]\n",
			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
			pasid, address, flags);
		break;
	case EVENT_TYPE_RMP_FAULT:
		amd_iommu_report_rmp_fault(iommu, event);
		break;
	case EVENT_TYPE_RMP_HW_ERR:
		amd_iommu_report_rmp_hw_error(iommu, event);
		break;
	case EVENT_TYPE_INV_PPR_REQ:
		pasid = PPR_PASID(*((u64 *)__evt));
		tag = event[1] & 0x03FF;
		dev_err(dev, "Event logged [INVALID_PPR_REQUEST device=%04x:%02x:%02x.%x pasid=0x%05x address=0x%llx flags=0x%04x tag=0x%03x]\n",
			iommu->pci_seg->id, PCI_BUS_NUM(devid), PCI_SLOT(devid), PCI_FUNC(devid),
			pasid, address, flags, tag);
		break;
	default:
		dev_err(dev, "Event logged [UNKNOWN event[0]=0x%08x event[1]=0x%08x event[2]=0x%08x event[3]=0x%08x\n",
			event[0], event[1], event[2], event[3]);
	}

	/*
	 * To detect the hardware errata 732 we need to clear the
	 * entry back to zero. This issue does not exist on SNP
	 * enabled system. Also this buffer is not writeable on
	 * SNP enabled system.
	 */
	if (!amd_iommu_snp_en)
		memset(__evt, 0, 4 * sizeof(u32));
}

static void iommu_poll_events(struct amd_iommu *iommu)
{
	u32 head, tail;

	head = readl(iommu->mmio_base + MMIO_EVT_HEAD_OFFSET);
	tail = readl(iommu->mmio_base + MMIO_EVT_TAIL_OFFSET);

	while (head != tail) {
		iommu_print_event(iommu, iommu->evt_buf + head);

		/* Update head pointer of hardware ring-buffer */
		head = (head + EVENT_ENTRY_SIZE) % EVT_BUFFER_SIZE;
		writel(head, iommu->mmio_base + MMIO_EVT_HEAD_OFFSET);
	}

}

#ifdef CONFIG_IRQ_REMAP
static int (*iommu_ga_log_notifier)(u32);

int amd_iommu_register_ga_log_notifier(int (*notifier)(u32))
{
	iommu_ga_log_notifier = notifier;

	/*
	 * Ensure all in-flight IRQ handlers run to completion before returning
	 * to the caller, e.g. to ensure module code isn't unloaded while it's
	 * being executed in the IRQ handler.
	 */
	if (!notifier)
		synchronize_rcu();

	return 0;
}
EXPORT_SYMBOL(amd_iommu_register_ga_log_notifier);

static void iommu_poll_ga_log(struct amd_iommu *iommu)
{
	u32 head, tail;

	if (iommu->ga_log == NULL)
		return;

	head = readl(iommu->mmio_base + MMIO_GA_HEAD_OFFSET);
	tail = readl(iommu->mmio_base + MMIO_GA_TAIL_OFFSET);

	while (head != tail) {
		volatile u64 *raw;
		u64 log_entry;

		raw = (u64 *)(iommu->ga_log + head);

		/* Avoid memcpy function-call overhead */
		log_entry = *raw;

		/* Update head pointer of hardware ring-buffer */
		head = (head + GA_ENTRY_SIZE) % GA_LOG_SIZE;
		writel(head, iommu->mmio_base + MMIO_GA_HEAD_OFFSET);

		/* Handle GA entry */
		switch (GA_REQ_TYPE(log_entry)) {
		case GA_GUEST_NR:
			if (!iommu_ga_log_notifier)
				break;

			pr_debug("%s: devid=%#x, ga_tag=%#x\n",
				 __func__, GA_DEVID(log_entry),
				 GA_TAG(log_entry));

			if (iommu_ga_log_notifier(GA_TAG(log_entry)) != 0)
				pr_err("GA log notifier failed.\n");
			break;
		default:
			break;
		}
	}
}

static void
amd_iommu_set_pci_msi_domain(struct device *dev, struct amd_iommu *iommu)
{
	if (!irq_remapping_enabled || !dev_is_pci(dev) ||
	    !pci_dev_has_default_msi_parent_domain(to_pci_dev(dev)))
		return;

	dev_set_msi_domain(dev, iommu->ir_domain);
}

#else /* CONFIG_IRQ_REMAP */
static inline void
amd_iommu_set_pci_msi_domain(struct device *dev, struct amd_iommu *iommu) { }
#endif /* !CONFIG_IRQ_REMAP */

static void amd_iommu_handle_irq(void *data, const char *evt_type,
				 u32 int_mask, u32 overflow_mask,
				 void (*int_handler)(struct amd_iommu *),
				 void (*overflow_handler)(struct amd_iommu *))
{
	struct amd_iommu *iommu = (struct amd_iommu *) data;
	u32 status = readl(iommu->mmio_base + MMIO_STATUS_OFFSET);
	u32 mask = int_mask | overflow_mask;

	while (status & mask) {
		/* Enable interrupt sources again */
		writel(mask, iommu->mmio_base + MMIO_STATUS_OFFSET);

		if (int_handler) {
			pr_devel("Processing IOMMU (ivhd%d) %s Log\n",
				 iommu->index, evt_type);
			int_handler(iommu);
		}

		if ((status & overflow_mask) && overflow_handler)
			overflow_handler(iommu);

		/*
		 * Hardware bug: ERBT1312
		 * When re-enabling interrupt (by writing 1
		 * to clear the bit), the hardware might also try to set
		 * the interrupt bit in the event status register.
		 * In this scenario, the bit will be set, and disable
		 * subsequent interrupts.
		 *
		 * Workaround: The IOMMU driver should read back the
		 * status register and check if the interrupt bits are cleared.
		 * If not, driver will need to go through the interrupt handler
		 * again and re-clear the bits
		 */
		status = readl(iommu->mmio_base + MMIO_STATUS_OFFSET);
	}
}

irqreturn_t amd_iommu_int_thread_evtlog(int irq, void *data)
{
	amd_iommu_handle_irq(data, "Evt", MMIO_STATUS_EVT_INT_MASK,
			     MMIO_STATUS_EVT_OVERFLOW_MASK,
			     iommu_poll_events, amd_iommu_restart_event_logging);

	return IRQ_HANDLED;
}

irqreturn_t amd_iommu_int_thread_pprlog(int irq, void *data)
{
	amd_iommu_handle_irq(data, "PPR", MMIO_STATUS_PPR_INT_MASK,
			     MMIO_STATUS_PPR_OVERFLOW_MASK,
			     amd_iommu_poll_ppr_log, amd_iommu_restart_ppr_log);

	return IRQ_HANDLED;
}

irqreturn_t amd_iommu_int_thread_galog(int irq, void *data)
{
#ifdef CONFIG_IRQ_REMAP
	amd_iommu_handle_irq(data, "GA", MMIO_STATUS_GALOG_INT_MASK,
			     MMIO_STATUS_GALOG_OVERFLOW_MASK,
			     iommu_poll_ga_log, amd_iommu_restart_ga_log);
#endif

	return IRQ_HANDLED;
}

irqreturn_t amd_iommu_int_thread(int irq, void *data)
{
	amd_iommu_int_thread_evtlog(irq, data);
	amd_iommu_int_thread_pprlog(irq, data);
	amd_iommu_int_thread_galog(irq, data);

	return IRQ_HANDLED;
}

irqreturn_t amd_iommu_int_handler(int irq, void *data)
{
	return IRQ_WAKE_THREAD;
}

/****************************************************************************
 *
 * IOMMU command queuing functions
 *
 ****************************************************************************/

static int wait_on_sem(struct amd_iommu *iommu, u64 data)
{
	int i = 0;

	while (*iommu->cmd_sem != data && i < LOOP_TIMEOUT) {
		udelay(1);
		i += 1;
	}

	if (i == LOOP_TIMEOUT) {
		pr_alert("Completion-Wait loop timed out\n");
		return -EIO;
	}

	return 0;
}

static void copy_cmd_to_buffer(struct amd_iommu *iommu,
			       struct iommu_cmd *cmd)
{
	u8 *target;
	u32 tail;

	/* Copy command to buffer */
	tail = iommu->cmd_buf_tail;
	target = iommu->cmd_buf + tail;
	memcpy(target, cmd, sizeof(*cmd));

	tail = (tail + sizeof(*cmd)) % CMD_BUFFER_SIZE;
	iommu->cmd_buf_tail = tail;

	/* Tell the IOMMU about it */
	writel(tail, iommu->mmio_base + MMIO_CMD_TAIL_OFFSET);
}

static void build_completion_wait(struct iommu_cmd *cmd,
				  struct amd_iommu *iommu,
				  u64 data)
{
	u64 paddr = iommu_virt_to_phys((void *)iommu->cmd_sem);

	memset(cmd, 0, sizeof(*cmd));
	cmd->data[0] = lower_32_bits(paddr) | CMD_COMPL_WAIT_STORE_MASK;
	cmd->data[1] = upper_32_bits(paddr);
	cmd->data[2] = lower_32_bits(data);
	cmd->data[3] = upper_32_bits(data);
	CMD_SET_TYPE(cmd, CMD_COMPL_WAIT);
}

static void build_inv_dte(struct iommu_cmd *cmd, u16 devid)
{
	memset(cmd, 0, sizeof(*cmd));
	cmd->data[0] = devid;
	CMD_SET_TYPE(cmd, CMD_INV_DEV_ENTRY);
}

/*
 * Builds an invalidation address which is suitable for one page or multiple
 * pages. Sets the size bit (S) as needed is more than one page is flushed.
 */
static inline u64 build_inv_address(u64 address, size_t size)
{
	u64 pages, end, msb_diff;

	pages = iommu_num_pages(address, size, PAGE_SIZE);

	if (pages == 1)
		return address & PAGE_MASK;

	end = address + size - 1;

	/*
	 * msb_diff would hold the index of the most significant bit that
	 * flipped between the start and end.
	 */
	msb_diff = fls64(end ^ address) - 1;

	/*
	 * Bits 63:52 are sign extended. If for some reason bit 51 is different
	 * between the start and the end, invalidate everything.
	 */
	if (unlikely(msb_diff > 51)) {
		address = CMD_INV_IOMMU_ALL_PAGES_ADDRESS;
	} else {
		/*
		 * The msb-bit must be clear on the address. Just set all the
		 * lower bits.
		 */
		address |= (1ull << msb_diff) - 1;
	}

	/* Clear bits 11:0 */
	address &= PAGE_MASK;

	/* Set the size bit - we flush more than one 4kb page */
	return address | CMD_INV_IOMMU_PAGES_SIZE_MASK;
}

static void build_inv_iommu_pages(struct iommu_cmd *cmd, u64 address,
				  size_t size, u16 domid,
				  ioasid_t pasid, bool gn)
{
	u64 inv_address = build_inv_address(address, size);

	memset(cmd, 0, sizeof(*cmd));

	cmd->data[1] |= domid;
	cmd->data[2]  = lower_32_bits(inv_address);
	cmd->data[3]  = upper_32_bits(inv_address);
	/* PDE bit - we want to flush everything, not only the PTEs */
	cmd->data[2] |= CMD_INV_IOMMU_PAGES_PDE_MASK;
	if (gn) {
		cmd->data[0] |= pasid;
		cmd->data[2] |= CMD_INV_IOMMU_PAGES_GN_MASK;
	}
	CMD_SET_TYPE(cmd, CMD_INV_IOMMU_PAGES);
}

static void build_inv_iotlb_pages(struct iommu_cmd *cmd, u16 devid, int qdep,
				  u64 address, size_t size,
				  ioasid_t pasid, bool gn)
{
	u64 inv_address = build_inv_address(address, size);

	memset(cmd, 0, sizeof(*cmd));

	cmd->data[0]  = devid;
	cmd->data[0] |= (qdep & 0xff) << 24;
	cmd->data[1]  = devid;
	cmd->data[2]  = lower_32_bits(inv_address);
	cmd->data[3]  = upper_32_bits(inv_address);
	if (gn) {
		cmd->data[0] |= ((pasid >> 8) & 0xff) << 16;
		cmd->data[1] |= (pasid & 0xff) << 16;
		cmd->data[2] |= CMD_INV_IOMMU_PAGES_GN_MASK;
	}

	CMD_SET_TYPE(cmd, CMD_INV_IOTLB_PAGES);
}

static void build_complete_ppr(struct iommu_cmd *cmd, u16 devid, u32 pasid,
			       int status, int tag, u8 gn)
{
	memset(cmd, 0, sizeof(*cmd));

	cmd->data[0]  = devid;
	if (gn) {
		cmd->data[1]  = pasid;
		cmd->data[2]  = CMD_INV_IOMMU_PAGES_GN_MASK;
	}
	cmd->data[3]  = tag & 0x1ff;
	cmd->data[3] |= (status & PPR_STATUS_MASK) << PPR_STATUS_SHIFT;

	CMD_SET_TYPE(cmd, CMD_COMPLETE_PPR);
}

static void build_inv_all(struct iommu_cmd *cmd)
{
	memset(cmd, 0, sizeof(*cmd));
	CMD_SET_TYPE(cmd, CMD_INV_ALL);
}

static void build_inv_irt(struct iommu_cmd *cmd, u16 devid)
{
	memset(cmd, 0, sizeof(*cmd));
	cmd->data[0] = devid;
	CMD_SET_TYPE(cmd, CMD_INV_IRT);
}

/*
 * Writes the command to the IOMMUs command buffer and informs the
 * hardware about the new command.
 */
static int __iommu_queue_command_sync(struct amd_iommu *iommu,
				      struct iommu_cmd *cmd,
				      bool sync)
{
	unsigned int count = 0;
	u32 left, next_tail;

	next_tail = (iommu->cmd_buf_tail + sizeof(*cmd)) % CMD_BUFFER_SIZE;
again:
	left      = (iommu->cmd_buf_head - next_tail) % CMD_BUFFER_SIZE;

	if (left <= 0x20) {
		/* Skip udelay() the first time around */
		if (count++) {
			if (count == LOOP_TIMEOUT) {
				pr_err("Command buffer timeout\n");
				return -EIO;
			}

			udelay(1);
		}

		/* Update head and recheck remaining space */
		iommu->cmd_buf_head = readl(iommu->mmio_base +
					    MMIO_CMD_HEAD_OFFSET);

		goto again;
	}

	copy_cmd_to_buffer(iommu, cmd);

	/* Do we need to make sure all commands are processed? */
	iommu->need_sync = sync;

	return 0;
}

static int iommu_queue_command_sync(struct amd_iommu *iommu,
				    struct iommu_cmd *cmd,
				    bool sync)
{
	unsigned long flags;
	int ret;

	raw_spin_lock_irqsave(&iommu->lock, flags);
	ret = __iommu_queue_command_sync(iommu, cmd, sync);
	raw_spin_unlock_irqrestore(&iommu->lock, flags);

	return ret;
}

static int iommu_queue_command(struct amd_iommu *iommu, struct iommu_cmd *cmd)
{
	return iommu_queue_command_sync(iommu, cmd, true);
}

/*
 * This function queues a completion wait command into the command
 * buffer of an IOMMU
 */
static int iommu_completion_wait(struct amd_iommu *iommu)
{
	struct iommu_cmd cmd;
	unsigned long flags;
	int ret;
	u64 data;

	if (!iommu->need_sync)
		return 0;

	data = atomic64_inc_return(&iommu->cmd_sem_val);
	build_completion_wait(&cmd, iommu, data);

	raw_spin_lock_irqsave(&iommu->lock, flags);

	ret = __iommu_queue_command_sync(iommu, &cmd, false);
	if (ret)
		goto out_unlock;

	ret = wait_on_sem(iommu, data);

out_unlock:
	raw_spin_unlock_irqrestore(&iommu->lock, flags);

	return ret;
}

static void domain_flush_complete(struct protection_domain *domain)
{
	struct pdom_iommu_info *pdom_iommu_info;
	unsigned long i;

	lockdep_assert_held(&domain->lock);

	/*
	 * Devices of this domain are behind this IOMMU
	 * We need to wait for completion of all commands.
	 */
	 xa_for_each(&domain->iommu_array, i, pdom_iommu_info)
		iommu_completion_wait(pdom_iommu_info->iommu);
}

static int iommu_flush_dte(struct amd_iommu *iommu, u16 devid)
{
	struct iommu_cmd cmd;

	build_inv_dte(&cmd, devid);

	return iommu_queue_command(iommu, &cmd);
}

static void iommu_flush_dte_sync(struct amd_iommu *iommu, u16 devid)
{
	int ret;

	ret = iommu_flush_dte(iommu, devid);
	if (!ret)
		iommu_completion_wait(iommu);
}

static void amd_iommu_flush_dte_all(struct amd_iommu *iommu)
{
	u32 devid;
	u16 last_bdf = iommu->pci_seg->last_bdf;

	for (devid = 0; devid <= last_bdf; ++devid)
		iommu_flush_dte(iommu, devid);

	iommu_completion_wait(iommu);
}

/*
 * This function uses heavy locking and may disable irqs for some time. But
 * this is no issue because it is only called during resume.
 */
static void amd_iommu_flush_tlb_all(struct amd_iommu *iommu)
{
	u32 dom_id;
	u16 last_bdf = iommu->pci_seg->last_bdf;

	for (dom_id = 0; dom_id <= last_bdf; ++dom_id) {
		struct iommu_cmd cmd;
		build_inv_iommu_pages(&cmd, 0, CMD_INV_IOMMU_ALL_PAGES_ADDRESS,
				      dom_id, IOMMU_NO_PASID, false);
		iommu_queue_command(iommu, &cmd);
	}

	iommu_completion_wait(iommu);
}

static void amd_iommu_flush_tlb_domid(struct amd_iommu *iommu, u32 dom_id)
{
	struct iommu_cmd cmd;

	build_inv_iommu_pages(&cmd, 0, CMD_INV_IOMMU_ALL_PAGES_ADDRESS,
			      dom_id, IOMMU_NO_PASID, false);
	iommu_queue_command(iommu, &cmd);

	iommu_completion_wait(iommu);
}

static void amd_iommu_flush_all(struct amd_iommu *iommu)
{
	struct iommu_cmd cmd;

	build_inv_all(&cmd);

	iommu_queue_command(iommu, &cmd);
	iommu_completion_wait(iommu);
}

static void iommu_flush_irt(struct amd_iommu *iommu, u16 devid)
{
	struct iommu_cmd cmd;

	build_inv_irt(&cmd, devid);

	iommu_queue_command(iommu, &cmd);
}

static void amd_iommu_flush_irt_all(struct amd_iommu *iommu)
{
	u32 devid;
	u16 last_bdf = iommu->pci_seg->last_bdf;

	if (iommu->irtcachedis_enabled)
		return;

	for (devid = 0; devid <= last_bdf; devid++)
		iommu_flush_irt(iommu, devid);

	iommu_completion_wait(iommu);
}

void amd_iommu_flush_all_caches(struct amd_iommu *iommu)
{
	if (check_feature(FEATURE_IA)) {
		amd_iommu_flush_all(iommu);
	} else {
		amd_iommu_flush_dte_all(iommu);
		amd_iommu_flush_irt_all(iommu);
		amd_iommu_flush_tlb_all(iommu);
	}
}

/*
 * Command send function for flushing on-device TLB
 */
static int device_flush_iotlb(struct iommu_dev_data *dev_data, u64 address,
			      size_t size, ioasid_t pasid, bool gn)
{
	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
	struct iommu_cmd cmd;
	int qdep = dev_data->ats_qdep;

	build_inv_iotlb_pages(&cmd, dev_data->devid, qdep, address,
			      size, pasid, gn);

	return iommu_queue_command(iommu, &cmd);
}

static int device_flush_dte_alias(struct pci_dev *pdev, u16 alias, void *data)
{
	struct amd_iommu *iommu = data;

	return iommu_flush_dte(iommu, alias);
}

/*
 * Command send function for invalidating a device table entry
 */
static int device_flush_dte(struct iommu_dev_data *dev_data)
{
	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
	struct pci_dev *pdev = NULL;
	struct amd_iommu_pci_seg *pci_seg;
	u16 alias;
	int ret;

	if (dev_is_pci(dev_data->dev))
		pdev = to_pci_dev(dev_data->dev);

	if (pdev)
		ret = pci_for_each_dma_alias(pdev,
					     device_flush_dte_alias, iommu);
	else
		ret = iommu_flush_dte(iommu, dev_data->devid);
	if (ret)
		return ret;

	pci_seg = iommu->pci_seg;
	alias = pci_seg->alias_table[dev_data->devid];
	if (alias != dev_data->devid) {
		ret = iommu_flush_dte(iommu, alias);
		if (ret)
			return ret;
	}

	if (dev_data->ats_enabled) {
		/* Invalidate the entire contents of an IOTLB */
		ret = device_flush_iotlb(dev_data, 0, ~0UL,
					 IOMMU_NO_PASID, false);
	}

	return ret;
}

static int domain_flush_pages_v2(struct protection_domain *pdom,
				 u64 address, size_t size)
{
	struct iommu_dev_data *dev_data;
	struct iommu_cmd cmd;
	int ret = 0;

	lockdep_assert_held(&pdom->lock);
	list_for_each_entry(dev_data, &pdom->dev_list, list) {
		struct amd_iommu *iommu = get_amd_iommu_from_dev(dev_data->dev);
		u16 domid = dev_data->gcr3_info.domid;

		build_inv_iommu_pages(&cmd, address, size,
				      domid, IOMMU_NO_PASID, true);

		ret |= iommu_queue_command(iommu, &cmd);
	}

	return ret;
}

static int domain_flush_pages_v1(struct protection_domain *pdom,
				 u64 address, size_t size)
{
	struct pdom_iommu_info *pdom_iommu_info;
	struct iommu_cmd cmd;
	int ret = 0;
	unsigned long i;

	lockdep_assert_held(&pdom->lock);

	build_inv_iommu_pages(&cmd, address, size,
			      pdom->id, IOMMU_NO_PASID, false);

	xa_for_each(&pdom->iommu_array, i, pdom_iommu_info) {
		/*
		 * Devices of this domain are behind this IOMMU
		 * We need a TLB flush
		 */
		ret |= iommu_queue_command(pdom_iommu_info->iommu, &cmd);
	}

	return ret;
}

/*
 * TLB invalidation function which is called from the mapping functions.
 * It flushes range of PTEs of the domain.
 */
static void __domain_flush_pages(struct protection_domain *domain,
				 u64 address, size_t size)
{
	struct iommu_dev_data *dev_data;
	int ret = 0;
	ioasid_t pasid = IOMMU_NO_PASID;
	bool gn = false;

	lockdep_assert_held(&domain->lock);

	if (pdom_is_v2_pgtbl_mode(domain)) {
		gn = true;
		ret = domain_flush_pages_v2(domain, address, size);
	} else {
		ret = domain_flush_pages_v1(domain, address, size);
	}

	list_for_each_entry(dev_data, &domain->dev_list, list) {

		if (!dev_data->ats_enabled)
			continue;

		ret |= device_flush_iotlb(dev_data, address, size, pasid, gn);
	}

	WARN_ON(ret);
}

void amd_iommu_domain_flush_pages(struct protection_domain *domain,
				  u64 address, size_t size)
{
	lockdep_assert_held(&domain->lock);

	if (likely(!amd_iommu_np_cache)) {
		__domain_flush_pages(domain, address, size);

		/* Wait until IOMMU TLB and all device IOTLB flushes are complete */
		domain_flush_complete(domain);

		return;
	}

	/*
	 * When NpCache is on, we infer that we run in a VM and use a vIOMMU.
	 * In such setups it is best to avoid flushes of ranges which are not
	 * naturally aligned, since it would lead to flushes of unmodified
	 * PTEs. Such flushes would require the hypervisor to do more work than
	 * necessary. Therefore, perform repeated flushes of aligned ranges
	 * until you cover the range. Each iteration flushes the smaller
	 * between the natural alignment of the address that we flush and the
	 * greatest naturally aligned region that fits in the range.
	 */
	while (size != 0) {
		int addr_alignment = __ffs(address);
		int size_alignment = __fls(size);
		int min_alignment;
		size_t flush_size;

		/*
		 * size is always non-zero, but address might be zero, causing
		 * addr_alignment to be negative. As the casting of the
		 * argument in __ffs(address) to long might trim the high bits
		 * of the address on x86-32, cast to long when doing the check.
		 */
		if (likely((unsigned long)address != 0))
			min_alignment = min(addr_alignment, size_alignment);
		else
			min_alignment = size_alignment;

		flush_size = 1ul << min_alignment;

		__domain_flush_pages(domain, address, flush_size);
		address += flush_size;
		size -= flush_size;
	}

	/* Wait until IOMMU TLB and all device IOTLB flushes are complete */
	domain_flush_complete(domain);
}

/* Flush the whole IO/TLB for a given protection domain - including PDE */
static void amd_iommu_domain_flush_all(struct protection_domain *domain)
{
	amd_iommu_domain_flush_pages(domain, 0,
				     CMD_INV_IOMMU_ALL_PAGES_ADDRESS);
}

void amd_iommu_dev_flush_pasid_pages(struct iommu_dev_data *dev_data,
				     ioasid_t pasid, u64 address, size_t size)
{
	struct iommu_cmd cmd;
	struct amd_iommu *iommu = get_amd_iommu_from_dev(dev_data->dev);

	build_inv_iommu_pages(&cmd, address, size,
			      dev_data->gcr3_info.domid, pasid, true);
	iommu_queue_command(iommu, &cmd);

	if (dev_data->ats_enabled)
		device_flush_iotlb(dev_data, address, size, pasid, true);

	iommu_completion_wait(iommu);
}

static void dev_flush_pasid_all(struct iommu_dev_data *dev_data,
				ioasid_t pasid)
{
	amd_iommu_dev_flush_pasid_pages(dev_data, pasid, 0,
					CMD_INV_IOMMU_ALL_PAGES_ADDRESS);
}

/* Flush the not present cache if it exists */
static void domain_flush_np_cache(struct protection_domain *domain,
		dma_addr_t iova, size_t size)
{
	if (unlikely(amd_iommu_np_cache)) {
		unsigned long flags;

		spin_lock_irqsave(&domain->lock, flags);
		amd_iommu_domain_flush_pages(domain, iova, size);
		spin_unlock_irqrestore(&domain->lock, flags);
	}
}


/*
 * This function flushes the DTEs for all devices in domain
 */
void amd_iommu_update_and_flush_device_table(struct protection_domain *domain)
{
	struct iommu_dev_data *dev_data;

	lockdep_assert_held(&domain->lock);

	list_for_each_entry(dev_data, &domain->dev_list, list) {
		struct amd_iommu *iommu = rlookup_amd_iommu(dev_data->dev);

		set_dte_entry(iommu, dev_data);
		clone_aliases(iommu, dev_data->dev);
	}

	list_for_each_entry(dev_data, &domain->dev_list, list)
		device_flush_dte(dev_data);

	domain_flush_complete(domain);
}

int amd_iommu_complete_ppr(struct device *dev, u32 pasid, int status, int tag)
{
	struct iommu_dev_data *dev_data;
	struct amd_iommu *iommu;
	struct iommu_cmd cmd;

	dev_data = dev_iommu_priv_get(dev);
	iommu    = get_amd_iommu_from_dev(dev);

	build_complete_ppr(&cmd, dev_data->devid, pasid, status,
			   tag, dev_data->pri_tlp);

	return iommu_queue_command(iommu, &cmd);
}

/****************************************************************************
 *
 * The next functions belong to the domain allocation. A domain is
 * allocated for every IOMMU as the default domain. If device isolation
 * is enabled, every device get its own domain. The most important thing
 * about domains is the page table mapping the DMA address space they
 * contain.
 *
 ****************************************************************************/

static int pdom_id_alloc(void)
{
	return ida_alloc_range(&pdom_ids, 1, MAX_DOMAIN_ID - 1, GFP_ATOMIC);
}

static void pdom_id_free(int id)
{
	ida_free(&pdom_ids, id);
}

static void free_gcr3_tbl_level1(u64 *tbl)
{
	u64 *ptr;
	int i;

	for (i = 0; i < 512; ++i) {
		if (!(tbl[i] & GCR3_VALID))
			continue;

		ptr = iommu_phys_to_virt(tbl[i] & PAGE_MASK);

		iommu_free_pages(ptr);
	}
}

static void free_gcr3_tbl_level2(u64 *tbl)
{
	u64 *ptr;
	int i;

	for (i = 0; i < 512; ++i) {
		if (!(tbl[i] & GCR3_VALID))
			continue;

		ptr = iommu_phys_to_virt(tbl[i] & PAGE_MASK);

		free_gcr3_tbl_level1(ptr);
	}
}

static void free_gcr3_table(struct gcr3_tbl_info *gcr3_info)
{
	if (gcr3_info->glx == 2)
		free_gcr3_tbl_level2(gcr3_info->gcr3_tbl);
	else if (gcr3_info->glx == 1)
		free_gcr3_tbl_level1(gcr3_info->gcr3_tbl);
	else
		WARN_ON_ONCE(gcr3_info->glx != 0);

	gcr3_info->glx = 0;

	/* Free per device domain ID */
	pdom_id_free(gcr3_info->domid);

	iommu_free_pages(gcr3_info->gcr3_tbl);
	gcr3_info->gcr3_tbl = NULL;
}

/*
 * Number of GCR3 table levels required. Level must be 4-Kbyte
 * page and can contain up to 512 entries.
 */
static int get_gcr3_levels(int pasids)
{
	int levels;

	if (pasids == -1)
		return amd_iommu_max_glx_val;

	levels = get_count_order(pasids);

	return levels ? (DIV_ROUND_UP(levels, 9) - 1) : levels;
}

static int setup_gcr3_table(struct gcr3_tbl_info *gcr3_info,
			    struct amd_iommu *iommu, int pasids)
{
	int levels = get_gcr3_levels(pasids);
	int nid = iommu ? dev_to_node(&iommu->dev->dev) : NUMA_NO_NODE;
	int domid;

	if (levels > amd_iommu_max_glx_val)
		return -EINVAL;

	if (gcr3_info->gcr3_tbl)
		return -EBUSY;

	/* Allocate per device domain ID */
	domid = pdom_id_alloc();
	if (domid <= 0)
		return -ENOSPC;
	gcr3_info->domid = domid;

	gcr3_info->gcr3_tbl = iommu_alloc_pages_node_sz(nid, GFP_ATOMIC, SZ_4K);
	if (gcr3_info->gcr3_tbl == NULL) {
		pdom_id_free(domid);
		return -ENOMEM;
	}

	gcr3_info->glx = levels;

	return 0;
}

static u64 *__get_gcr3_pte(struct gcr3_tbl_info *gcr3_info,
			   ioasid_t pasid, bool alloc)
{
	int index;
	u64 *pte;
	u64 *root = gcr3_info->gcr3_tbl;
	int level = gcr3_info->glx;

	while (true) {

		index = (pasid >> (9 * level)) & 0x1ff;
		pte   = &root[index];

		if (level == 0)
			break;

		if (!(*pte & GCR3_VALID)) {
			if (!alloc)
				return NULL;

			root = (void *)get_zeroed_page(GFP_ATOMIC);
			if (root == NULL)
				return NULL;

			*pte = iommu_virt_to_phys(root) | GCR3_VALID;
		}

		root = iommu_phys_to_virt(*pte & PAGE_MASK);

		level -= 1;
	}

	return pte;
}

static int update_gcr3(struct iommu_dev_data *dev_data,
		       ioasid_t pasid, unsigned long gcr3, bool set)
{
	struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
	u64 *pte;

	pte = __get_gcr3_pte(gcr3_info, pasid, true);
	if (pte == NULL)
		return -ENOMEM;

	if (set)
		*pte = (gcr3 & PAGE_MASK) | GCR3_VALID;
	else
		*pte = 0;

	dev_flush_pasid_all(dev_data, pasid);
	return 0;
}

int amd_iommu_set_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid,
		       unsigned long gcr3)
{
	struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
	int ret;

	iommu_group_mutex_assert(dev_data->dev);

	ret = update_gcr3(dev_data, pasid, gcr3, true);
	if (ret)
		return ret;

	gcr3_info->pasid_cnt++;
	return ret;
}

int amd_iommu_clear_gcr3(struct iommu_dev_data *dev_data, ioasid_t pasid)
{
	struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
	int ret;

	iommu_group_mutex_assert(dev_data->dev);

	ret = update_gcr3(dev_data, pasid, 0, false);
	if (ret)
		return ret;

	gcr3_info->pasid_cnt--;
	return ret;
}

static void make_clear_dte(struct iommu_dev_data *dev_data, struct dev_table_entry *ptr,
			   struct dev_table_entry *new)
{
	/* All existing DTE must have V bit set */
	new->data128[0] = DTE_FLAG_V;
	new->data128[1] = 0;
}

/*
 * Note:
 * The old value for GCR3 table and GPT have been cleared from caller.
 */
static void set_dte_gcr3_table(struct amd_iommu *iommu,
			       struct iommu_dev_data *dev_data,
			       struct dev_table_entry *target)
{
	struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
	u64 gcr3;

	if (!gcr3_info->gcr3_tbl)
		return;

	pr_debug("%s: devid=%#x, glx=%#x, gcr3_tbl=%#llx\n",
		 __func__, dev_data->devid, gcr3_info->glx,
		 (unsigned long long)gcr3_info->gcr3_tbl);

	gcr3 = iommu_virt_to_phys(gcr3_info->gcr3_tbl);

	target->data[0] |= DTE_FLAG_GV |
			   FIELD_PREP(DTE_GLX, gcr3_info->glx) |
			   FIELD_PREP(DTE_GCR3_14_12, gcr3 >> 12);
	if (pdom_is_v2_pgtbl_mode(dev_data->domain))
		target->data[0] |= DTE_FLAG_GIOV;

	target->data[1] |= FIELD_PREP(DTE_GCR3_30_15, gcr3 >> 15) |
			   FIELD_PREP(DTE_GCR3_51_31, gcr3 >> 31);

	/* Guest page table can only support 4 and 5 levels  */
	if (amd_iommu_gpt_level == PAGE_MODE_5_LEVEL)
		target->data[2] |= FIELD_PREP(DTE_GPT_LEVEL_MASK, GUEST_PGTABLE_5_LEVEL);
	else
		target->data[2] |= FIELD_PREP(DTE_GPT_LEVEL_MASK, GUEST_PGTABLE_4_LEVEL);
}

static void set_dte_entry(struct amd_iommu *iommu,
			  struct iommu_dev_data *dev_data)
{
	u16 domid;
	u32 old_domid;
	struct dev_table_entry *initial_dte;
	struct dev_table_entry new = {};
	struct protection_domain *domain = dev_data->domain;
	struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;
	struct dev_table_entry *dte = &get_dev_table(iommu)[dev_data->devid];

	if (gcr3_info && gcr3_info->gcr3_tbl)
		domid = dev_data->gcr3_info.domid;
	else
		domid = domain->id;

	make_clear_dte(dev_data, dte, &new);

	if (domain->iop.mode != PAGE_MODE_NONE)
		new.data[0] |= iommu_virt_to_phys(domain->iop.root);

	new.data[0] |= (domain->iop.mode & DEV_ENTRY_MODE_MASK)
		    << DEV_ENTRY_MODE_SHIFT;

	new.data[0] |= DTE_FLAG_IR | DTE_FLAG_IW;

	/*
	 * When SNP is enabled, we can only support TV=1 with non-zero domain ID.
	 * This is prevented by the SNP-enable and IOMMU_DOMAIN_IDENTITY check in
	 * do_iommu_domain_alloc().
	 */
	WARN_ON(amd_iommu_snp_en && (domid == 0));
	new.data[0] |= DTE_FLAG_TV;

	if (dev_data->ppr)
		new.data[0] |= 1ULL << DEV_ENTRY_PPR;

	if (domain->dirty_tracking)
		new.data[0] |= DTE_FLAG_HAD;

	if (dev_data->ats_enabled)
		new.data[1] |= DTE_FLAG_IOTLB;

	old_domid = READ_ONCE(dte->data[1]) & DEV_DOMID_MASK;
	new.data[1] |= domid;

	/*
	 * Restore cached persistent DTE bits, which can be set by information
	 * in IVRS table. See set_dev_entry_from_acpi().
	 */
	initial_dte = amd_iommu_get_ivhd_dte_flags(iommu->pci_seg->id, dev_data->devid);
	if (initial_dte) {
		new.data128[0] |= initial_dte->data128[0];
		new.data128[1] |= initial_dte->data128[1];
	}

	set_dte_gcr3_table(iommu, dev_data, &new);

	update_dte256(iommu, dev_data, &new);

	/*
	 * A kdump kernel might be replacing a domain ID that was copied from
	 * the previous kernel--if so, it needs to flush the translation cache
	 * entries for the old domain ID that is being overwritten
	 */
	if (old_domid) {
		amd_iommu_flush_tlb_domid(iommu, old_domid);
	}
}

/*
 * Clear DMA-remap related flags to block all DMA (blockeded domain)
 */
static void clear_dte_entry(struct amd_iommu *iommu, struct iommu_dev_data *dev_data)
{
	struct dev_table_entry new = {};
	struct dev_table_entry *dte = &get_dev_table(iommu)[dev_data->devid];

	make_clear_dte(dev_data, dte, &new);
	update_dte256(iommu, dev_data, &new);
}

/* Update and flush DTE for the given device */
static void dev_update_dte(struct iommu_dev_data *dev_data, bool set)
{
	struct amd_iommu *iommu = get_amd_iommu_from_dev(dev_data->dev);

	if (set)
		set_dte_entry(iommu, dev_data);
	else
		clear_dte_entry(iommu, dev_data);

	clone_aliases(iommu, dev_data->dev);
	device_flush_dte(dev_data);
	iommu_completion_wait(iommu);
}

/*
 * If domain is SVA capable then initialize GCR3 table. Also if domain is
 * in v2 page table mode then update GCR3[0].
 */
static int init_gcr3_table(struct iommu_dev_data *dev_data,
			   struct protection_domain *pdom)
{
	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
	int max_pasids = dev_data->max_pasids;
	int ret = 0;

	 /*
	  * If domain is in pt mode then setup GCR3 table only if device
	  * is PASID capable
	  */
	if (pdom_is_in_pt_mode(pdom) && !pdev_pasid_supported(dev_data))
		return ret;

	/*
	 * By default, setup GCR3 table to support MAX PASIDs
	 * supported by the device/IOMMU.
	 */
	ret = setup_gcr3_table(&dev_data->gcr3_info, iommu,
			       max_pasids > 0 ?  max_pasids : 1);
	if (ret)
		return ret;

	/* Setup GCR3[0] only if domain is setup with v2 page table mode */
	if (!pdom_is_v2_pgtbl_mode(pdom))
		return ret;

	ret = update_gcr3(dev_data, 0, iommu_virt_to_phys(pdom->iop.pgd), true);
	if (ret)
		free_gcr3_table(&dev_data->gcr3_info);

	return ret;
}

static void destroy_gcr3_table(struct iommu_dev_data *dev_data,
			       struct protection_domain *pdom)
{
	struct gcr3_tbl_info *gcr3_info = &dev_data->gcr3_info;

	if (pdom_is_v2_pgtbl_mode(pdom))
		update_gcr3(dev_data, 0, 0, false);

	if (gcr3_info->gcr3_tbl == NULL)
		return;

	free_gcr3_table(gcr3_info);
}

static int pdom_attach_iommu(struct amd_iommu *iommu,
			     struct protection_domain *pdom)
{
	struct pdom_iommu_info *pdom_iommu_info, *curr;
	unsigned long flags;
	int ret = 0;

	spin_lock_irqsave(&pdom->lock, flags);

	pdom_iommu_info = xa_load(&pdom->iommu_array, iommu->index);
	if (pdom_iommu_info) {
		pdom_iommu_info->refcnt++;
		goto out_unlock;
	}

	pdom_iommu_info = kzalloc(sizeof(*pdom_iommu_info), GFP_ATOMIC);
	if (!pdom_iommu_info) {
		ret = -ENOMEM;
		goto out_unlock;
	}

	pdom_iommu_info->iommu = iommu;
	pdom_iommu_info->refcnt = 1;

	curr = xa_cmpxchg(&pdom->iommu_array, iommu->index,
			  NULL, pdom_iommu_info, GFP_ATOMIC);
	if (curr) {
		kfree(pdom_iommu_info);
		ret = -ENOSPC;
		goto out_unlock;
	}

out_unlock:
	spin_unlock_irqrestore(&pdom->lock, flags);
	return ret;
}

static void pdom_detach_iommu(struct amd_iommu *iommu,
			      struct protection_domain *pdom)
{
	struct pdom_iommu_info *pdom_iommu_info;
	unsigned long flags;

	spin_lock_irqsave(&pdom->lock, flags);

	pdom_iommu_info = xa_load(&pdom->iommu_array, iommu->index);
	if (!pdom_iommu_info) {
		spin_unlock_irqrestore(&pdom->lock, flags);
		return;
	}

	pdom_iommu_info->refcnt--;
	if (pdom_iommu_info->refcnt == 0) {
		xa_erase(&pdom->iommu_array, iommu->index);
		kfree(pdom_iommu_info);
	}

	spin_unlock_irqrestore(&pdom->lock, flags);
}

/*
 * If a device is not yet associated with a domain, this function makes the
 * device visible in the domain
 */
static int attach_device(struct device *dev,
			 struct protection_domain *domain)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
	struct pci_dev *pdev;
	unsigned long flags;
	int ret = 0;

	mutex_lock(&dev_data->mutex);

	if (dev_data->domain != NULL) {
		ret = -EBUSY;
		goto out;
	}

	/* Do reference counting */
	ret = pdom_attach_iommu(iommu, domain);
	if (ret)
		goto out;

	/* Setup GCR3 table */
	if (pdom_is_sva_capable(domain)) {
		ret = init_gcr3_table(dev_data, domain);
		if (ret) {
			pdom_detach_iommu(iommu, domain);
			goto out;
		}
	}

	pdev = dev_is_pci(dev_data->dev) ? to_pci_dev(dev_data->dev) : NULL;
	if (pdev && pdom_is_sva_capable(domain)) {
		pdev_enable_caps(pdev);

		/*
		 * Device can continue to function even if IOPF
		 * enablement failed. Hence in error path just
		 * disable device PRI support.
		 */
		if (amd_iommu_iopf_add_device(iommu, dev_data))
			pdev_disable_cap_pri(pdev);
	} else if (pdev) {
		pdev_enable_cap_ats(pdev);
	}

	/* Update data structures */
	dev_data->domain = domain;
	spin_lock_irqsave(&domain->lock, flags);
	list_add(&dev_data->list, &domain->dev_list);
	spin_unlock_irqrestore(&domain->lock, flags);

	/* Update device table */
	dev_update_dte(dev_data, true);

out:
	mutex_unlock(&dev_data->mutex);

	return ret;
}

/*
 * Removes a device from a protection domain (with devtable_lock held)
 */
static void detach_device(struct device *dev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
	struct amd_iommu *iommu = get_amd_iommu_from_dev_data(dev_data);
	struct protection_domain *domain = dev_data->domain;
	unsigned long flags;

	mutex_lock(&dev_data->mutex);

	/*
	 * First check if the device is still attached. It might already
	 * be detached from its domain because the generic
	 * iommu_detach_group code detached it and we try again here in
	 * our alias handling.
	 */
	if (WARN_ON(!dev_data->domain))
		goto out;

	/* Remove IOPF handler */
	if (dev_data->ppr) {
		iopf_queue_flush_dev(dev);
		amd_iommu_iopf_remove_device(iommu, dev_data);
	}

	if (dev_is_pci(dev))
		pdev_disable_caps(to_pci_dev(dev));

	/* Clear DTE and flush the entry */
	dev_update_dte(dev_data, false);

	/* Flush IOTLB and wait for the flushes to finish */
	spin_lock_irqsave(&domain->lock, flags);
	amd_iommu_domain_flush_all(domain);
	list_del(&dev_data->list);
	spin_unlock_irqrestore(&domain->lock, flags);

	/* Clear GCR3 table */
	if (pdom_is_sva_capable(domain))
		destroy_gcr3_table(dev_data, domain);

	/* Update data structures */
	dev_data->domain = NULL;

	/* decrease reference counters - needs to happen after the flushes */
	pdom_detach_iommu(iommu, domain);

out:
	mutex_unlock(&dev_data->mutex);
}

static struct iommu_device *amd_iommu_probe_device(struct device *dev)
{
	struct iommu_device *iommu_dev;
	struct amd_iommu *iommu;
	struct iommu_dev_data *dev_data;
	int ret;

	if (!check_device(dev))
		return ERR_PTR(-ENODEV);

	iommu = rlookup_amd_iommu(dev);
	if (!iommu)
		return ERR_PTR(-ENODEV);

	/* Not registered yet? */
	if (!iommu->iommu.ops)
		return ERR_PTR(-ENODEV);

	if (dev_iommu_priv_get(dev))
		return &iommu->iommu;

	ret = iommu_init_device(iommu, dev);
	if (ret) {
		dev_err(dev, "Failed to initialize - trying to proceed anyway\n");
		iommu_dev = ERR_PTR(ret);
		iommu_ignore_device(iommu, dev);
		goto out_err;
	}

	amd_iommu_set_pci_msi_domain(dev, iommu);
	iommu_dev = &iommu->iommu;

	/*
	 * If IOMMU and device supports PASID then it will contain max
	 * supported PASIDs, else it will be zero.
	 */
	dev_data = dev_iommu_priv_get(dev);
	if (amd_iommu_pasid_supported() && dev_is_pci(dev) &&
	    pdev_pasid_supported(dev_data)) {
		dev_data->max_pasids = min_t(u32, iommu->iommu.max_pasids,
					     pci_max_pasids(to_pci_dev(dev)));
	}

	if (amd_iommu_pgtable == PD_MODE_NONE) {
		pr_warn_once("%s: DMA translation not supported by iommu.\n",
			     __func__);
		iommu_dev = ERR_PTR(-ENODEV);
		goto out_err;
	}

out_err:

	iommu_completion_wait(iommu);

	if (FEATURE_NUM_INT_REMAP_SUP_2K(amd_iommu_efr2))
		dev_data->max_irqs = MAX_IRQS_PER_TABLE_2K;
	else
		dev_data->max_irqs = MAX_IRQS_PER_TABLE_512;

	if (dev_is_pci(dev))
		pci_prepare_ats(to_pci_dev(dev), PAGE_SHIFT);

	return iommu_dev;
}

static void amd_iommu_release_device(struct device *dev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);

	WARN_ON(dev_data->domain);

	/*
	 * We keep dev_data around for unplugged devices and reuse it when the
	 * device is re-plugged - not doing so would introduce a ton of races.
	 */
}

static struct iommu_group *amd_iommu_device_group(struct device *dev)
{
	if (dev_is_pci(dev))
		return pci_device_group(dev);

	return acpihid_device_group(dev);
}

/*****************************************************************************
 *
 * The following functions belong to the exported interface of AMD IOMMU
 *
 * This interface allows access to lower level functions of the IOMMU
 * like protection domain handling and assignement of devices to domains
 * which is not possible with the dma_ops interface.
 *
 *****************************************************************************/

static void protection_domain_init(struct protection_domain *domain)
{
	spin_lock_init(&domain->lock);
	INIT_LIST_HEAD(&domain->dev_list);
	INIT_LIST_HEAD(&domain->dev_data_list);
	xa_init(&domain->iommu_array);
}

struct protection_domain *protection_domain_alloc(void)
{
	struct protection_domain *domain;
	int domid;

	domain = kzalloc(sizeof(*domain), GFP_KERNEL);
	if (!domain)
		return NULL;

	domid = pdom_id_alloc();
	if (domid <= 0) {
		kfree(domain);
		return NULL;
	}
	domain->id = domid;

	protection_domain_init(domain);

	return domain;
}

static int pdom_setup_pgtable(struct protection_domain *domain,
			      struct device *dev)
{
	struct io_pgtable_ops *pgtbl_ops;
	enum io_pgtable_fmt fmt;

	switch (domain->pd_mode) {
	case PD_MODE_V1:
		fmt = AMD_IOMMU_V1;
		break;
	case PD_MODE_V2:
		fmt = AMD_IOMMU_V2;
		break;
	case PD_MODE_NONE:
		WARN_ON_ONCE(1);
		return -EPERM;
	}

	domain->iop.pgtbl.cfg.amd.nid = dev_to_node(dev);
	pgtbl_ops = alloc_io_pgtable_ops(fmt, &domain->iop.pgtbl.cfg, domain);
	if (!pgtbl_ops)
		return -ENOMEM;

	return 0;
}

static inline u64 dma_max_address(enum protection_domain_mode pgtable)
{
	if (pgtable == PD_MODE_V1)
		return PM_LEVEL_SIZE(amd_iommu_hpt_level);

	/*
	 * V2 with 4/5 level page table. Note that "2.2.6.5 AMD64 4-Kbyte Page
	 * Translation" shows that the V2 table sign extends the top of the
	 * address space creating a reserved region in the middle of the
	 * translation, just like the CPU does. Further Vasant says the docs are
	 * incomplete and this only applies to non-zero PASIDs. If the AMDv2
	 * page table is assigned to the 0 PASID then there is no sign extension
	 * check.
	 *
	 * Since the IOMMU must have a fixed geometry, and the core code does
	 * not understand sign extended addressing, we have to chop off the high
	 * bit to get consistent behavior with attachments of the domain to any
	 * PASID.
	 */
	return ((1ULL << (PM_LEVEL_SHIFT(amd_iommu_gpt_level) - 1)) - 1);
}

static bool amd_iommu_hd_support(struct amd_iommu *iommu)
{
	if (amd_iommu_hatdis)
		return false;

	return iommu && (iommu->features & FEATURE_HDSUP);
}

static struct iommu_domain *
do_iommu_domain_alloc(struct device *dev, u32 flags,
		      enum protection_domain_mode pgtable)
{
	bool dirty_tracking = flags & IOMMU_HWPT_ALLOC_DIRTY_TRACKING;
	struct amd_iommu *iommu = get_amd_iommu_from_dev(dev);
	struct protection_domain *domain;
	int ret;

	domain = protection_domain_alloc();
	if (!domain)
		return ERR_PTR(-ENOMEM);

	domain->pd_mode = pgtable;
	ret = pdom_setup_pgtable(domain, dev);
	if (ret) {
		pdom_id_free(domain->id);
		kfree(domain);
		return ERR_PTR(ret);
	}

	domain->domain.geometry.aperture_start = 0;
	domain->domain.geometry.aperture_end   = dma_max_address(pgtable);
	domain->domain.geometry.force_aperture = true;
	domain->domain.pgsize_bitmap = domain->iop.pgtbl.cfg.pgsize_bitmap;

	domain->domain.type = IOMMU_DOMAIN_UNMANAGED;
	domain->domain.ops = iommu->iommu.ops->default_domain_ops;

	if (dirty_tracking)
		domain->domain.dirty_ops = &amd_dirty_ops;

	return &domain->domain;
}

static struct iommu_domain *
amd_iommu_domain_alloc_paging_flags(struct device *dev, u32 flags,
				    const struct iommu_user_data *user_data)

{
	struct amd_iommu *iommu = get_amd_iommu_from_dev(dev);
	const u32 supported_flags = IOMMU_HWPT_ALLOC_DIRTY_TRACKING |
						IOMMU_HWPT_ALLOC_PASID;

	if ((flags & ~supported_flags) || user_data)
		return ERR_PTR(-EOPNOTSUPP);

	switch (flags & supported_flags) {
	case IOMMU_HWPT_ALLOC_DIRTY_TRACKING:
		/* Allocate domain with v1 page table for dirty tracking */
		if (!amd_iommu_hd_support(iommu))
			break;
		return do_iommu_domain_alloc(dev, flags, PD_MODE_V1);
	case IOMMU_HWPT_ALLOC_PASID:
		/* Allocate domain with v2 page table if IOMMU supports PASID. */
		if (!amd_iommu_pasid_supported())
			break;
		return do_iommu_domain_alloc(dev, flags, PD_MODE_V2);
	case 0:
		/* If nothing specific is required use the kernel commandline default */
		return do_iommu_domain_alloc(dev, 0, amd_iommu_pgtable);
	default:
		break;
	}
	return ERR_PTR(-EOPNOTSUPP);
}

void amd_iommu_domain_free(struct iommu_domain *dom)
{
	struct protection_domain *domain = to_pdomain(dom);

	WARN_ON(!list_empty(&domain->dev_list));
	if (domain->domain.type & __IOMMU_DOMAIN_PAGING)
		free_io_pgtable_ops(&domain->iop.pgtbl.ops);
	pdom_id_free(domain->id);
	kfree(domain);
}

static int blocked_domain_attach_device(struct iommu_domain *domain,
					struct device *dev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);

	if (dev_data->domain)
		detach_device(dev);

	/* Clear DTE and flush the entry */
	mutex_lock(&dev_data->mutex);
	dev_update_dte(dev_data, false);
	mutex_unlock(&dev_data->mutex);

	return 0;
}

static int blocked_domain_set_dev_pasid(struct iommu_domain *domain,
					struct device *dev, ioasid_t pasid,
					struct iommu_domain *old)
{
	amd_iommu_remove_dev_pasid(dev, pasid, old);
	return 0;
}

static struct iommu_domain blocked_domain = {
	.type = IOMMU_DOMAIN_BLOCKED,
	.ops = &(const struct iommu_domain_ops) {
		.attach_dev     = blocked_domain_attach_device,
		.set_dev_pasid  = blocked_domain_set_dev_pasid,
	}
};

static struct protection_domain identity_domain;

static const struct iommu_domain_ops identity_domain_ops = {
	.attach_dev = amd_iommu_attach_device,
};

void amd_iommu_init_identity_domain(void)
{
	struct iommu_domain *domain = &identity_domain.domain;

	domain->type = IOMMU_DOMAIN_IDENTITY;
	domain->ops = &identity_domain_ops;
	domain->owner = &amd_iommu_ops;

	identity_domain.id = pdom_id_alloc();

	protection_domain_init(&identity_domain);
}

/* Same as blocked domain except it supports only ops->attach_dev() */
static struct iommu_domain release_domain = {
	.type = IOMMU_DOMAIN_BLOCKED,
	.ops = &(const struct iommu_domain_ops) {
		.attach_dev     = blocked_domain_attach_device,
	}
};

static int amd_iommu_attach_device(struct iommu_domain *dom,
				   struct device *dev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);
	struct protection_domain *domain = to_pdomain(dom);
	struct amd_iommu *iommu = get_amd_iommu_from_dev(dev);
	int ret;

	/*
	 * Skip attach device to domain if new domain is same as
	 * devices current domain
	 */
	if (dev_data->domain == domain)
		return 0;

	dev_data->defer_attach = false;

	/*
	 * Restrict to devices with compatible IOMMU hardware support
	 * when enforcement of dirty tracking is enabled.
	 */
	if (dom->dirty_ops && !amd_iommu_hd_support(iommu))
		return -EINVAL;

	if (dev_data->domain)
		detach_device(dev);

	ret = attach_device(dev, domain);

#ifdef CONFIG_IRQ_REMAP
	if (AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir)) {
		if (dom->type == IOMMU_DOMAIN_UNMANAGED)
			dev_data->use_vapic = 1;
		else
			dev_data->use_vapic = 0;
	}
#endif

	return ret;
}

static int amd_iommu_iotlb_sync_map(struct iommu_domain *dom,
				    unsigned long iova, size_t size)
{
	struct protection_domain *domain = to_pdomain(dom);
	struct io_pgtable_ops *ops = &domain->iop.pgtbl.ops;

	if (ops->map_pages)
		domain_flush_np_cache(domain, iova, size);
	return 0;
}

static int amd_iommu_map_pages(struct iommu_domain *dom, unsigned long iova,
			       phys_addr_t paddr, size_t pgsize, size_t pgcount,
			       int iommu_prot, gfp_t gfp, size_t *mapped)
{
	struct protection_domain *domain = to_pdomain(dom);
	struct io_pgtable_ops *ops = &domain->iop.pgtbl.ops;
	int prot = 0;
	int ret = -EINVAL;

	if ((domain->pd_mode == PD_MODE_V1) &&
	    (domain->iop.mode == PAGE_MODE_NONE))
		return -EINVAL;

	if (iommu_prot & IOMMU_READ)
		prot |= IOMMU_PROT_IR;
	if (iommu_prot & IOMMU_WRITE)
		prot |= IOMMU_PROT_IW;

	if (ops->map_pages) {
		ret = ops->map_pages(ops, iova, paddr, pgsize,
				     pgcount, prot, gfp, mapped);
	}

	return ret;
}

static void amd_iommu_iotlb_gather_add_page(struct iommu_domain *domain,
					    struct iommu_iotlb_gather *gather,
					    unsigned long iova, size_t size)
{
	/*
	 * AMD's IOMMU can flush as many pages as necessary in a single flush.
	 * Unless we run in a virtual machine, which can be inferred according
	 * to whether "non-present cache" is on, it is probably best to prefer
	 * (potentially) too extensive TLB flushing (i.e., more misses) over
	 * mutliple TLB flushes (i.e., more flushes). For virtual machines the
	 * hypervisor needs to synchronize the host IOMMU PTEs with those of
	 * the guest, and the trade-off is different: unnecessary TLB flushes
	 * should be avoided.
	 */
	if (amd_iommu_np_cache &&
	    iommu_iotlb_gather_is_disjoint(gather, iova, size))
		iommu_iotlb_sync(domain, gather);

	iommu_iotlb_gather_add_range(gather, iova, size);
}

static size_t amd_iommu_unmap_pages(struct iommu_domain *dom, unsigned long iova,
				    size_t pgsize, size_t pgcount,
				    struct iommu_iotlb_gather *gather)
{
	struct protection_domain *domain = to_pdomain(dom);
	struct io_pgtable_ops *ops = &domain->iop.pgtbl.ops;
	size_t r;

	if ((domain->pd_mode == PD_MODE_V1) &&
	    (domain->iop.mode == PAGE_MODE_NONE))
		return 0;

	r = (ops->unmap_pages) ? ops->unmap_pages(ops, iova, pgsize, pgcount, NULL) : 0;

	if (r)
		amd_iommu_iotlb_gather_add_page(dom, gather, iova, r);

	return r;
}

static phys_addr_t amd_iommu_iova_to_phys(struct iommu_domain *dom,
					  dma_addr_t iova)
{
	struct protection_domain *domain = to_pdomain(dom);
	struct io_pgtable_ops *ops = &domain->iop.pgtbl.ops;

	return ops->iova_to_phys(ops, iova);
}

static bool amd_iommu_capable(struct device *dev, enum iommu_cap cap)
{
	switch (cap) {
	case IOMMU_CAP_CACHE_COHERENCY:
		return true;
	case IOMMU_CAP_NOEXEC:
		return false;
	case IOMMU_CAP_PRE_BOOT_PROTECTION:
		return amdr_ivrs_remap_support;
	case IOMMU_CAP_ENFORCE_CACHE_COHERENCY:
		return true;
	case IOMMU_CAP_DEFERRED_FLUSH:
		return true;
	case IOMMU_CAP_DIRTY_TRACKING: {
		struct amd_iommu *iommu = get_amd_iommu_from_dev(dev);

		return amd_iommu_hd_support(iommu);
	}
	default:
		break;
	}

	return false;
}

static int amd_iommu_set_dirty_tracking(struct iommu_domain *domain,
					bool enable)
{
	struct protection_domain *pdomain = to_pdomain(domain);
	struct dev_table_entry *dte;
	struct iommu_dev_data *dev_data;
	bool domain_flush = false;
	struct amd_iommu *iommu;
	unsigned long flags;
	u64 new;

	spin_lock_irqsave(&pdomain->lock, flags);
	if (!(pdomain->dirty_tracking ^ enable)) {
		spin_unlock_irqrestore(&pdomain->lock, flags);
		return 0;
	}

	list_for_each_entry(dev_data, &pdomain->dev_list, list) {
		spin_lock(&dev_data->dte_lock);
		iommu = get_amd_iommu_from_dev_data(dev_data);
		dte = &get_dev_table(iommu)[dev_data->devid];
		new = dte->data[0];
		new = (enable ? new | DTE_FLAG_HAD : new & ~DTE_FLAG_HAD);
		dte->data[0] = new;
		spin_unlock(&dev_data->dte_lock);

		/* Flush device DTE */
		device_flush_dte(dev_data);
		domain_flush = true;
	}

	/* Flush IOTLB to mark IOPTE dirty on the next translation(s) */
	if (domain_flush)
		amd_iommu_domain_flush_all(pdomain);

	pdomain->dirty_tracking = enable;
	spin_unlock_irqrestore(&pdomain->lock, flags);

	return 0;
}

static int amd_iommu_read_and_clear_dirty(struct iommu_domain *domain,
					  unsigned long iova, size_t size,
					  unsigned long flags,
					  struct iommu_dirty_bitmap *dirty)
{
	struct protection_domain *pdomain = to_pdomain(domain);
	struct io_pgtable_ops *ops = &pdomain->iop.pgtbl.ops;
	unsigned long lflags;

	if (!ops || !ops->read_and_clear_dirty)
		return -EOPNOTSUPP;

	spin_lock_irqsave(&pdomain->lock, lflags);
	if (!pdomain->dirty_tracking && dirty->bitmap) {
		spin_unlock_irqrestore(&pdomain->lock, lflags);
		return -EINVAL;
	}
	spin_unlock_irqrestore(&pdomain->lock, lflags);

	return ops->read_and_clear_dirty(ops, iova, size, flags, dirty);
}

static void amd_iommu_get_resv_regions(struct device *dev,
				       struct list_head *head)
{
	struct iommu_resv_region *region;
	struct unity_map_entry *entry;
	struct amd_iommu *iommu;
	struct amd_iommu_pci_seg *pci_seg;
	int devid, sbdf;

	sbdf = get_device_sbdf_id(dev);
	if (sbdf < 0)
		return;

	devid = PCI_SBDF_TO_DEVID(sbdf);
	iommu = get_amd_iommu_from_dev(dev);
	pci_seg = iommu->pci_seg;

	list_for_each_entry(entry, &pci_seg->unity_map, list) {
		int type, prot = 0;
		size_t length;

		if (devid < entry->devid_start || devid > entry->devid_end)
			continue;

		type   = IOMMU_RESV_DIRECT;
		length = entry->address_end - entry->address_start;
		if (entry->prot & IOMMU_PROT_IR)
			prot |= IOMMU_READ;
		if (entry->prot & IOMMU_PROT_IW)
			prot |= IOMMU_WRITE;
		if (entry->prot & IOMMU_UNITY_MAP_FLAG_EXCL_RANGE)
			/* Exclusion range */
			type = IOMMU_RESV_RESERVED;

		region = iommu_alloc_resv_region(entry->address_start,
						 length, prot, type,
						 GFP_KERNEL);
		if (!region) {
			dev_err(dev, "Out of memory allocating dm-regions\n");
			return;
		}
		list_add_tail(&region->list, head);
	}

	region = iommu_alloc_resv_region(MSI_RANGE_START,
					 MSI_RANGE_END - MSI_RANGE_START + 1,
					 0, IOMMU_RESV_MSI, GFP_KERNEL);
	if (!region)
		return;
	list_add_tail(&region->list, head);

	if (amd_iommu_ht_range_ignore())
		return;

	region = iommu_alloc_resv_region(HT_RANGE_START,
					 HT_RANGE_END - HT_RANGE_START + 1,
					 0, IOMMU_RESV_RESERVED, GFP_KERNEL);
	if (!region)
		return;
	list_add_tail(&region->list, head);
}

static bool amd_iommu_is_attach_deferred(struct device *dev)
{
	struct iommu_dev_data *dev_data = dev_iommu_priv_get(dev);

	return dev_data->defer_attach;
}

static void amd_iommu_flush_iotlb_all(struct iommu_domain *domain)
{
	struct protection_domain *dom = to_pdomain(domain);
	unsigned long flags;

	spin_lock_irqsave(&dom->lock, flags);
	amd_iommu_domain_flush_all(dom);
	spin_unlock_irqrestore(&dom->lock, flags);
}

static void amd_iommu_iotlb_sync(struct iommu_domain *domain,
				 struct iommu_iotlb_gather *gather)
{
	struct protection_domain *dom = to_pdomain(domain);
	unsigned long flags;

	spin_lock_irqsave(&dom->lock, flags);
	amd_iommu_domain_flush_pages(dom, gather->start,
				     gather->end - gather->start + 1);
	spin_unlock_irqrestore(&dom->lock, flags);
}

static int amd_iommu_def_domain_type(struct device *dev)
{
	struct iommu_dev_data *dev_data;

	dev_data = dev_iommu_priv_get(dev);
	if (!dev_data)
		return 0;

	/* Always use DMA domain for untrusted device */
	if (dev_is_pci(dev) && to_pci_dev(dev)->untrusted)
		return IOMMU_DOMAIN_DMA;

	/*
	 * Do not identity map IOMMUv2 capable devices when:
	 *  - memory encryption is active, because some of those devices
	 *    (AMD GPUs) don't have the encryption bit in their DMA-mask
	 *    and require remapping.
	 *  - SNP is enabled, because it prohibits DTE[Mode]=0.
	 */
	if (pdev_pasid_supported(dev_data) &&
	    !cc_platform_has(CC_ATTR_MEM_ENCRYPT) &&
	    !amd_iommu_snp_en) {
		return IOMMU_DOMAIN_IDENTITY;
	}

	return 0;
}

static bool amd_iommu_enforce_cache_coherency(struct iommu_domain *domain)
{
	/* IOMMU_PTE_FC is always set */
	return true;
}

static const struct iommu_dirty_ops amd_dirty_ops = {
	.set_dirty_tracking = amd_iommu_set_dirty_tracking,
	.read_and_clear_dirty = amd_iommu_read_and_clear_dirty,
};

const struct iommu_ops amd_iommu_ops = {
	.capable = amd_iommu_capable,
	.blocked_domain = &blocked_domain,
	.release_domain = &release_domain,
	.identity_domain = &identity_domain.domain,
	.domain_alloc_paging_flags = amd_iommu_domain_alloc_paging_flags,
	.domain_alloc_sva = amd_iommu_domain_alloc_sva,
	.probe_device = amd_iommu_probe_device,
	.release_device = amd_iommu_release_device,
	.device_group = amd_iommu_device_group,
	.get_resv_regions = amd_iommu_get_resv_regions,
	.is_attach_deferred = amd_iommu_is_attach_deferred,
	.def_domain_type = amd_iommu_def_domain_type,
	.page_response = amd_iommu_page_response,
	.default_domain_ops = &(const struct iommu_domain_ops) {
		.attach_dev	= amd_iommu_attach_device,
		.map_pages	= amd_iommu_map_pages,
		.unmap_pages	= amd_iommu_unmap_pages,
		.iotlb_sync_map	= amd_iommu_iotlb_sync_map,
		.iova_to_phys	= amd_iommu_iova_to_phys,
		.flush_iotlb_all = amd_iommu_flush_iotlb_all,
		.iotlb_sync	= amd_iommu_iotlb_sync,
		.free		= amd_iommu_domain_free,
		.enforce_cache_coherency = amd_iommu_enforce_cache_coherency,
	}
};

#ifdef CONFIG_IRQ_REMAP

/*****************************************************************************
 *
 * Interrupt Remapping Implementation
 *
 *****************************************************************************/

static struct irq_chip amd_ir_chip;
static DEFINE_SPINLOCK(iommu_table_lock);

static void iommu_flush_irt_and_complete(struct amd_iommu *iommu, u16 devid)
{
	int ret;
	u64 data;
	unsigned long flags;
	struct iommu_cmd cmd, cmd2;

	if (iommu->irtcachedis_enabled)
		return;

	build_inv_irt(&cmd, devid);
	data = atomic64_inc_return(&iommu->cmd_sem_val);
	build_completion_wait(&cmd2, iommu, data);

	raw_spin_lock_irqsave(&iommu->lock, flags);
	ret = __iommu_queue_command_sync(iommu, &cmd, true);
	if (ret)
		goto out;
	ret = __iommu_queue_command_sync(iommu, &cmd2, false);
	if (ret)
		goto out;
	wait_on_sem(iommu, data);
out:
	raw_spin_unlock_irqrestore(&iommu->lock, flags);
}

static inline u8 iommu_get_int_tablen(struct iommu_dev_data *dev_data)
{
	if (dev_data && dev_data->max_irqs == MAX_IRQS_PER_TABLE_2K)
		return DTE_INTTABLEN_2K;
	return DTE_INTTABLEN_512;
}

static void set_dte_irq_entry(struct amd_iommu *iommu, u16 devid,
			      struct irq_remap_table *table)
{
	u64 new;
	struct dev_table_entry *dte = &get_dev_table(iommu)[devid];
	struct iommu_dev_data *dev_data = search_dev_data(iommu, devid);

	if (dev_data)
		spin_lock(&dev_data->dte_lock);

	new = READ_ONCE(dte->data[2]);
	new &= ~DTE_IRQ_PHYS_ADDR_MASK;
	new |= iommu_virt_to_phys(table->table);
	new |= DTE_IRQ_REMAP_INTCTL;
	new |= iommu_get_int_tablen(dev_data);
	new |= DTE_IRQ_REMAP_ENABLE;
	WRITE_ONCE(dte->data[2], new);

	if (dev_data)
		spin_unlock(&dev_data->dte_lock);
}

static struct irq_remap_table *get_irq_table(struct amd_iommu *iommu, u16 devid)
{
	struct irq_remap_table *table;
	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;

	if (WARN_ONCE(!pci_seg->rlookup_table[devid],
		      "%s: no iommu for devid %x:%x\n",
		      __func__, pci_seg->id, devid))
		return NULL;

	table = pci_seg->irq_lookup_table[devid];
	if (WARN_ONCE(!table, "%s: no table for devid %x:%x\n",
		      __func__, pci_seg->id, devid))
		return NULL;

	return table;
}

static struct irq_remap_table *__alloc_irq_table(int nid, size_t size)
{
	struct irq_remap_table *table;

	table = kzalloc(sizeof(*table), GFP_KERNEL);
	if (!table)
		return NULL;

	table->table = iommu_alloc_pages_node_sz(
		nid, GFP_KERNEL, max(DTE_INTTAB_ALIGNMENT, size));
	if (!table->table) {
		kfree(table);
		return NULL;
	}
	raw_spin_lock_init(&table->lock);

	return table;
}

static void set_remap_table_entry(struct amd_iommu *iommu, u16 devid,
				  struct irq_remap_table *table)
{
	struct amd_iommu_pci_seg *pci_seg = iommu->pci_seg;

	pci_seg->irq_lookup_table[devid] = table;
	set_dte_irq_entry(iommu, devid, table);
	iommu_flush_dte(iommu, devid);
}

static int set_remap_table_entry_alias(struct pci_dev *pdev, u16 alias,
				       void *data)
{
	struct irq_remap_table *table = data;
	struct amd_iommu_pci_seg *pci_seg;
	struct amd_iommu *iommu = rlookup_amd_iommu(&pdev->dev);

	if (!iommu)
		return -EINVAL;

	pci_seg = iommu->pci_seg;
	pci_seg->irq_lookup_table[alias] = table;
	set_dte_irq_entry(iommu, alias, table);
	iommu_flush_dte(pci_seg->rlookup_table[alias], alias);

	return 0;
}

static inline size_t get_irq_table_size(unsigned int max_irqs)
{
	if (!AMD_IOMMU_GUEST_IR_GA(amd_iommu_guest_ir))
		return max_irqs * sizeof(u32);

	return max_irqs * (sizeof(u64) * 2);
}

static struct irq_remap_table *alloc_irq_table(struct amd_iommu *iommu,
					       u16 devid, struct pci_dev *pdev,
					       unsigned int max_irqs)
{
	struct irq_remap_table *table = NULL;
	struct irq_remap_table *new_table = NULL;
	struct amd_iommu_pci_seg *pci_seg;
	unsigned long flags;
	int nid = iommu && iommu->dev ? dev_to_node(&iommu->dev->dev) : NUMA_NO_NODE;
	u16 alias;

	spin_lock_irqsave(&iommu_table_lock, flags);

	pci_seg = iommu->pci_seg;
	table = pci_seg->irq_lookup_table[devid];
	if (table)
		goto out_unlock;

	alias = pci_seg->alias_table[devid];
	table = pci_seg->irq_lookup_table[alias];
	if (table) {
		set_remap_table_entry(iommu, devid, table);
		goto out_wait;
	}
	spin_unlock_irqrestore(&iommu_table_lock, flags);

	/* Nothing there yet, allocate new irq remapping table */
	new_table = __alloc_irq_table(nid, get_irq_table_size(max_irqs));
	if (!new_table)
		return NULL;

	spin_lock_irqsave(&iommu_table_lock, flags);

	table = pci_seg->irq_lookup_table[devid];
	if (table)
		goto out_unlock;

	table = pci_seg->irq_lookup_table[alias];
	if (table) {
		set_remap_table_entry(iommu, devid, table);
		goto out_wait;
	}

	table = new_table;
	new_table = NULL;

	if (pdev)
		pci_for_each_dma_alias(pdev, set_remap_table_entry_alias,
				       table);
	else
		set_remap_table_entry(iommu, devid, table);

	if (devid != alias)
		set_remap_table_entry(iommu, alias, table);

out_wait:
	iommu_completion_wait(iommu);

out_unlock:
	spin_unlock_irqrestore(&iommu_table_lock, flags);

	if (new_table) {
		iommu_free_pages(new_table->table);
		kfree(new_table);
	}
	return table;
}

static int alloc_irq_index(struct amd_iommu *iommu, u16 devid, int count,
			   bool align, struct pci_dev *pdev,
			   unsigned long max_irqs)
{
	struct irq_remap_table *table;
	int index, c, alignment = 1;
	unsigned long flags;

	table = alloc_irq_table(iommu, devid, pdev, max_irqs);
	if (!table)
		return -ENODEV;

	if (align)
		alignment = roundup_pow_of_two(count);

	raw_spin_lock_irqsave(&table->lock, flags);

	/* Scan table for free entries */
	for (index = ALIGN(table->min_index, alignment), c = 0;
	     index < max_irqs;) {
		if (!iommu->irte_ops->is_allocated(table, index)) {
			c += 1;
		} else {
			c     = 0;
			index = ALIGN(index + 1, alignment);
			continue;
		}

		if (c == count)	{
			for (; c != 0; --c)
				iommu->irte_ops->set_allocated(table, index - c + 1);

			index -= count - 1;
			goto out;
		}

		index++;
	}

	index = -ENOSPC;

out:
	raw_spin_unlock_irqrestore(&table->lock, flags);

	return index;
}

static int __modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
			    struct irte_ga *irte)
{
	struct irq_remap_table *table;
	struct irte_ga *entry;
	unsigned long flags;
	u128 old;

	table = get_irq_table(iommu, devid);
	if (!table)
		return -ENOMEM;

	raw_spin_lock_irqsave(&table->lock, flags);

	entry = (struct irte_ga *)table->table;
	entry = &entry[index];

	/*
	 * We use cmpxchg16 to atomically update the 128-bit IRTE,
	 * and it cannot be updated by the hardware or other processors
	 * behind us, so the return value of cmpxchg16 should be the
	 * same as the old value.
	 */
	old = entry->irte;
	WARN_ON(!try_cmpxchg128(&entry->irte, &old, irte->irte));

	raw_spin_unlock_irqrestore(&table->lock, flags);

	return 0;
}

static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index,
			  struct irte_ga *irte)
{
	bool ret;

	ret = __modify_irte_ga(iommu, devid, index, irte);
	if (ret)
		return ret;

	iommu_flush_irt_and_complete(iommu, devid);

	return 0;
}

static int modify_irte(struct amd_iommu *iommu,
		       u16 devid, int index, union irte *irte)
{
	struct irq_remap_table *table;
	unsigned long flags;

	table = get_irq_table(iommu, devid);
	if (!table)
		return -ENOMEM;

	raw_spin_lock_irqsave(&table->lock, flags);
	table->table[index] = irte->val;
	raw_spin_unlock_irqrestore(&table->lock, flags);

	iommu_flush_irt_and_complete(iommu, devid);

	return 0;
}

static void free_irte(struct amd_iommu *iommu, u16 devid, int index)
{
	struct irq_remap_table *table;
	unsigned long flags;

	table = get_irq_table(iommu, devid);
	if (!table)
		return;

	raw_spin_lock_irqsave(&table->lock, flags);
	iommu->irte_ops->clear_allocated(table, index);
	raw_spin_unlock_irqrestore(&table->lock, flags);

	iommu_flush_irt_and_complete(iommu, devid);
}

static void irte_prepare(void *entry,
			 u32 delivery_mode, bool dest_mode,
			 u8 vector, u32 dest_apicid, int devid)
{
	union irte *irte = (union irte *) entry;

	irte->val                = 0;
	irte->fields.vector      = vector;
	irte->fields.int_type    = delivery_mode;
	irte->fields.destination = dest_apicid;
	irte->fields.dm          = dest_mode;
	irte->fields.valid       = 1;
}

static void irte_ga_prepare(void *entry,
			    u32 delivery_mode, bool dest_mode,
			    u8 vector, u32 dest_apicid, int devid)
{
	struct irte_ga *irte = (struct irte_ga *) entry;

	irte->lo.val                      = 0;
	irte->hi.val                      = 0;
	irte->lo.fields_remap.int_type    = delivery_mode;
	irte->lo.fields_remap.dm          = dest_mode;
	irte->hi.fields.vector            = vector;
	irte->lo.fields_remap.destination = APICID_TO_IRTE_DEST_LO(dest_apicid);
	irte->hi.fields.destination       = APICID_TO_IRTE_DEST_HI(dest_apicid);
	irte->lo.fields_remap.valid       = 1;
}

static void irte_activate(struct amd_iommu *iommu, void *entry, u16 devid, u16 index)
{
	union irte *irte = (union irte *) entry;

	irte->fields.valid = 1;
	modify_irte(iommu, devid, index, irte);
}

static void irte_ga_activate(struct amd_iommu *iommu, void *entry, u16 devid, u16 index)
{
	struct irte_ga *irte = (struct irte_ga *) entry;

	irte->lo.fields_remap.valid = 1;
	modify_irte_ga(iommu, devid, index, irte);
}

static void irte_deactivate(struct amd_iommu *iommu, void *entry, u16 devid, u16 index)
{
	union irte *irte = (union irte *) entry;

	irte->fields.valid = 0;
	modify_irte(iommu, devid, index, irte);
}

static void irte_ga_deactivate(struct amd_iommu *iommu, void *entry, u16 devid, u16 index)
{
	struct irte_ga *irte = (struct irte_ga *) entry;

	irte->lo.fields_remap.valid = 0;
	modify_irte_ga(iommu, devid, index, irte);
}

static void irte_set_affinity(struct amd_iommu *iommu, void *entry, u16 devid, u16 index,
			      u8 vector, u32 dest_apicid)
{
	union irte *irte = (union irte *) entry;

	irte->fields.vector = vector;
	irte->fields.destination = dest_apicid;
	modify_irte(iommu, devid, index, irte);
}

static void irte_ga_set_affinity(struct amd_iommu *iommu, void *entry, u16 devid, u16 index,
				 u8 vector, u32 dest_apicid)
{
	struct irte_ga *irte = (struct irte_ga *) entry;

	if (!irte->lo.fields_remap.guest_mode) {
		irte->hi.fields.vector = vector;
		irte->lo.fields_remap.destination =
					APICID_TO_IRTE_DEST_LO(dest_apicid);
		irte->hi.fields.destination =
					APICID_TO_IRTE_DEST_HI(dest_apicid);
		modify_irte_ga(iommu, devid, index, irte);
	}
}

#define IRTE_ALLOCATED (~1U)
static void irte_set_allocated(struct irq_remap_table *table, int index)
{
	table->table[index] = IRTE_ALLOCATED;
}

static void irte_ga_set_allocated(struct irq_remap_table *table, int index)
{
	struct irte_ga *ptr = (struct irte_ga *)table->table;
	struct irte_ga *irte = &ptr[index];

	memset(&irte->lo.val, 0, sizeof(u64));
	memset(&irte->hi.val, 0, sizeof(u64));
	irte->hi.fields.vector = 0xff;
}

static bool irte_is_allocated(struct irq_remap_table *table, int index)
{
	union irte *ptr = (union irte *)table->table;
	union irte *irte = &ptr[index];

	return irte->val != 0;
}

static bool irte_ga_is_allocated(struct irq_remap_table *table, int index)
{
	struct irte_ga *ptr = (struct irte_ga *)table->table;
	struct irte_ga *irte = &ptr[index];

	return irte->hi.fields.vector != 0;
}

static void irte_clear_allocated(struct irq_remap_table *table, int index)
{
	table->table[index] = 0;
}

static void irte_ga_clear_allocated(struct irq_remap_table *table, int index)
{
	struct irte_ga *ptr = (struct irte_ga *)table->table;
	struct irte_ga *irte = &ptr[index];

	memset(&irte->lo.val, 0, sizeof(u64));
	memset(&irte->hi.val, 0, sizeof(u64));
}

static int get_devid(struct irq_alloc_info *info)
{
	switch (info->type) {
	case X86_IRQ_ALLOC_TYPE_IOAPIC:
		return get_ioapic_devid(info->devid);
	case X86_IRQ_ALLOC_TYPE_HPET:
		return get_hpet_devid(info->devid);
	case X86_IRQ_ALLOC_TYPE_PCI_MSI:
	case X86_IRQ_ALLOC_TYPE_PCI_MSIX:
		return get_device_sbdf_id(msi_desc_to_dev(info->desc));
	default:
		WARN_ON_ONCE(1);
		return -1;
	}
}

struct irq_remap_ops amd_iommu_irq_ops = {
	.prepare		= amd_iommu_prepare,
	.enable			= amd_iommu_enable,
	.disable		= amd_iommu_disable,
	.reenable		= amd_iommu_reenable,
	.enable_faulting	= amd_iommu_enable_faulting,
};

static void fill_msi_msg(struct msi_msg *msg, u32 index)
{
	msg->data = index;
	msg->address_lo = 0;
	msg->arch_addr_lo.base_address = X86_MSI_BASE_ADDRESS_LOW;
	/*
	 * The struct msi_msg.dest_mode_logical is used to set the DM bit
	 * in MSI Message Address Register. For device w/ 2K int-remap support,
	 * this is bit must be set to 1 regardless of the actual destination
	 * mode, which is signified by the IRTE[DM].
	 */
	if (FEATURE_NUM_INT_REMAP_SUP_2K(amd_iommu_efr2))
		msg->arch_addr_lo.dest_mode_logical = true;
	msg->address_hi = X86_MSI_BASE_ADDRESS_HIGH;
}

static void irq_remapping_prepare_irte(struct amd_ir_data *data,
				       struct irq_cfg *irq_cfg,
				       struct irq_alloc_info *info,
				       int devid, int index, int sub_handle)
{
	struct irq_2_irte *irte_info = &data->irq_2_irte;
	struct amd_iommu *iommu = data->iommu;

	if (!iommu)
		return;

	data->irq_2_irte.devid = devid;
	data->irq_2_irte.index = index + sub_handle;
	iommu->irte_ops->prepare(data->entry, APIC_DELIVERY_MODE_FIXED,
				 apic->dest_mode_logical, irq_cfg->vector,
				 irq_cfg->dest_apicid, devid);

	switch (info->type) {
	case X86_IRQ_ALLOC_TYPE_IOAPIC:
	case X86_IRQ_ALLOC_TYPE_HPET:
	case X86_IRQ_ALLOC_TYPE_PCI_MSI:
	case X86_IRQ_ALLOC_TYPE_PCI_MSIX:
		fill_msi_msg(&data->msi_entry, irte_info->index);
		break;

	default:
		BUG_ON(1);
		break;
	}
}

struct amd_irte_ops irte_32_ops = {
	.prepare = irte_prepare,
	.activate = irte_activate,
	.deactivate = irte_deactivate,
	.set_affinity = irte_set_affinity,
	.set_allocated = irte_set_allocated,
	.is_allocated = irte_is_allocated,
	.clear_allocated = irte_clear_allocated,
};

struct amd_irte_ops irte_128_ops = {
	.prepare = irte_ga_prepare,
	.activate = irte_ga_activate,
	.deactivate = irte_ga_deactivate,
	.set_affinity = irte_ga_set_affinity,
	.set_allocated = irte_ga_set_allocated,
	.is_allocated = irte_ga_is_allocated,
	.clear_allocated = irte_ga_clear_allocated,
};

static int irq_remapping_alloc(struct irq_domain *domain, unsigned int virq,
			       unsigned int nr_irqs, void *arg)
{
	struct irq_alloc_info *info = arg;
	struct irq_data *irq_data;
	struct amd_ir_data *data = NULL;
	struct amd_iommu *iommu;
	struct irq_cfg *cfg;
	struct iommu_dev_data *dev_data;
	unsigned long max_irqs;
	int i, ret, devid, seg, sbdf;
	int index;

	if (!info)
		return -EINVAL;
	if (nr_irqs > 1 && info->type != X86_IRQ_ALLOC_TYPE_PCI_MSI)
		return -EINVAL;

	sbdf = get_devid(info);
	if (sbdf < 0)
		return -EINVAL;

	seg = PCI_SBDF_TO_SEGID(sbdf);
	devid = PCI_SBDF_TO_DEVID(sbdf);
	iommu = __rlookup_amd_iommu(seg, devid);
	if (!iommu)
		return -EINVAL;

	dev_data = search_dev_data(iommu, devid);
	max_irqs = dev_data ? dev_data->max_irqs : MAX_IRQS_PER_TABLE_512;

	ret = irq_domain_alloc_irqs_parent(domain, virq, nr_irqs, arg);
	if (ret < 0)
		return ret;

	if (info->type == X86_IRQ_ALLOC_TYPE_IOAPIC) {
		struct irq_remap_table *table;

		table = alloc_irq_table(iommu, devid, NULL, max_irqs);
		if (table) {
			if (!table->min_index) {
				/*
				 * Keep the first 32 indexes free for IOAPIC
				 * interrupts.
				 */
				table->min_index = 32;
				for (i = 0; i < 32; ++i)
					iommu->irte_ops->set_allocated(table, i);
			}
			WARN_ON(table->min_index != 32);
			index = info->ioapic.pin;
		} else {
			index = -ENOMEM;
		}
	} else if (info->type == X86_IRQ_ALLOC_TYPE_PCI_MSI ||
		   info->type == X86_IRQ_ALLOC_TYPE_PCI_MSIX) {
		bool align = (info->type == X86_IRQ_ALLOC_TYPE_PCI_MSI);

		index = alloc_irq_index(iommu, devid, nr_irqs, align,
					msi_desc_to_pci_dev(info->desc),
					max_irqs);
	} else {
		index = alloc_irq_index(iommu, devid, nr_irqs, false, NULL,
					max_irqs);
	}

	if (index < 0) {
		pr_warn("Failed to allocate IRTE\n");
		ret = index;
		goto out_free_parent;
	}

	for (i = 0; i < nr_irqs; i++) {
		irq_data = irq_domain_get_irq_data(domain, virq + i);
		cfg = irq_data ? irqd_cfg(irq_data) : NULL;
		if (!cfg) {
			ret = -EINVAL;
			goto out_free_data;
		}

		ret = -ENOMEM;
		data = kzalloc(sizeof(*data), GFP_KERNEL);
		if (!data)
			goto out_free_data;

		if (!AMD_IOMMU_GUEST_IR_GA(amd_iommu_guest_ir))
			data->entry = kzalloc(sizeof(union irte), GFP_KERNEL);
		else
			data->entry = kzalloc(sizeof(struct irte_ga),
						     GFP_KERNEL);
		if (!data->entry) {
			kfree(data);
			goto out_free_data;
		}

		data->iommu = iommu;
		irq_data->hwirq = (devid << 16) + i;
		irq_data->chip_data = data;
		irq_data->chip = &amd_ir_chip;
		irq_remapping_prepare_irte(data, cfg, info, devid, index, i);
	}

	return 0;

out_free_data:
	for (i--; i >= 0; i--) {
		irq_data = irq_domain_get_irq_data(domain, virq + i);
		if (irq_data)
			kfree(irq_data->chip_data);
	}
	for (i = 0; i < nr_irqs; i++)
		free_irte(iommu, devid, index + i);
out_free_parent:
	irq_domain_free_irqs_common(domain, virq, nr_irqs);
	return ret;
}

static void irq_remapping_free(struct irq_domain *domain, unsigned int virq,
			       unsigned int nr_irqs)
{
	struct irq_2_irte *irte_info;
	struct irq_data *irq_data;
	struct amd_ir_data *data;
	int i;

	for (i = 0; i < nr_irqs; i++) {
		irq_data = irq_domain_get_irq_data(domain, virq  + i);
		if (irq_data && irq_data->chip_data) {
			data = irq_data->chip_data;
			irte_info = &data->irq_2_irte;
			free_irte(data->iommu, irte_info->devid, irte_info->index);
			kfree(data->entry);
			kfree(data);
		}
	}
	irq_domain_free_irqs_common(domain, virq, nr_irqs);
}

static void amd_ir_update_irte(struct irq_data *irqd, struct amd_iommu *iommu,
			       struct amd_ir_data *ir_data,
			       struct irq_2_irte *irte_info,
			       struct irq_cfg *cfg);

static int irq_remapping_activate(struct irq_domain *domain,
				  struct irq_data *irq_data, bool reserve)
{
	struct amd_ir_data *data = irq_data->chip_data;
	struct irq_2_irte *irte_info = &data->irq_2_irte;
	struct amd_iommu *iommu = data->iommu;
	struct irq_cfg *cfg = irqd_cfg(irq_data);

	if (!iommu)
		return 0;

	iommu->irte_ops->activate(iommu, data->entry, irte_info->devid,
				  irte_info->index);
	amd_ir_update_irte(irq_data, iommu, data, irte_info, cfg);
	return 0;
}

static void irq_remapping_deactivate(struct irq_domain *domain,
				     struct irq_data *irq_data)
{
	struct amd_ir_data *data = irq_data->chip_data;
	struct irq_2_irte *irte_info = &data->irq_2_irte;
	struct amd_iommu *iommu = data->iommu;

	if (iommu)
		iommu->irte_ops->deactivate(iommu, data->entry, irte_info->devid,
					    irte_info->index);
}

static int irq_remapping_select(struct irq_domain *d, struct irq_fwspec *fwspec,
				enum irq_domain_bus_token bus_token)
{
	struct amd_iommu *iommu;
	int devid = -1;

	if (!amd_iommu_irq_remap)
		return 0;

	if (x86_fwspec_is_ioapic(fwspec))
		devid = get_ioapic_devid(fwspec->param[0]);
	else if (x86_fwspec_is_hpet(fwspec))
		devid = get_hpet_devid(fwspec->param[0]);

	if (devid < 0)
		return 0;
	iommu = __rlookup_amd_iommu((devid >> 16), (devid & 0xffff));

	return iommu && iommu->ir_domain == d;
}

static const struct irq_domain_ops amd_ir_domain_ops = {
	.select = irq_remapping_select,
	.alloc = irq_remapping_alloc,
	.free = irq_remapping_free,
	.activate = irq_remapping_activate,
	.deactivate = irq_remapping_deactivate,
};

static void __amd_iommu_update_ga(struct irte_ga *entry, int cpu,
				  bool ga_log_intr)
{
	if (cpu >= 0) {
		entry->lo.fields_vapic.destination =
					APICID_TO_IRTE_DEST_LO(cpu);
		entry->hi.fields.destination =
					APICID_TO_IRTE_DEST_HI(cpu);
		entry->lo.fields_vapic.is_run = true;
		entry->lo.fields_vapic.ga_log_intr = false;
	} else {
		entry->lo.fields_vapic.is_run = false;
		entry->lo.fields_vapic.ga_log_intr = ga_log_intr;
	}
}

/*
 * Update the pCPU information for an IRTE that is configured to post IRQs to
 * a vCPU, without issuing an IOMMU invalidation for the IRTE.
 *
 * If the vCPU is associated with a pCPU (@cpu >= 0), configure the Destination
 * with the pCPU's APIC ID, set IsRun, and clear GALogIntr.  If the vCPU isn't
 * associated with a pCPU (@cpu < 0), clear IsRun and set/clear GALogIntr based
 * on input from the caller (e.g. KVM only requests GALogIntr when the vCPU is
 * blocking and requires a notification wake event).  I.e. treat vCPUs that are
 * associated with a pCPU as running.  This API is intended to be used when a
 * vCPU is scheduled in/out (or stops running for any reason), to do a fast
 * update of IsRun, GALogIntr, and (conditionally) Destination.
 *
 * Per the IOMMU spec, the Destination, IsRun, and GATag fields are not cached
 * and thus don't require an invalidation to ensure the IOMMU consumes fresh
 * information.
 */
int amd_iommu_update_ga(void *data, int cpu, bool ga_log_intr)
{
	struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
	struct irte_ga *entry = (struct irte_ga *) ir_data->entry;

	if (WARN_ON_ONCE(!AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir)))
		return -EINVAL;

	if (!entry || !entry->lo.fields_vapic.guest_mode)
		return 0;

	if (!ir_data->iommu)
		return -ENODEV;

	__amd_iommu_update_ga(entry, cpu, ga_log_intr);

	return __modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
				ir_data->irq_2_irte.index, entry);
}
EXPORT_SYMBOL(amd_iommu_update_ga);

int amd_iommu_activate_guest_mode(void *data, int cpu, bool ga_log_intr)
{
	struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
	struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
	u64 valid;

	if (WARN_ON_ONCE(!AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir)))
		return -EINVAL;

	if (!entry)
		return 0;

	valid = entry->lo.fields_vapic.valid;

	entry->lo.val = 0;
	entry->hi.val = 0;

	entry->lo.fields_vapic.valid       = valid;
	entry->lo.fields_vapic.guest_mode  = 1;
	entry->hi.fields.ga_root_ptr       = ir_data->ga_root_ptr;
	entry->hi.fields.vector            = ir_data->ga_vector;
	entry->lo.fields_vapic.ga_tag      = ir_data->ga_tag;

	__amd_iommu_update_ga(entry, cpu, ga_log_intr);

	return modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
			      ir_data->irq_2_irte.index, entry);
}
EXPORT_SYMBOL(amd_iommu_activate_guest_mode);

int amd_iommu_deactivate_guest_mode(void *data)
{
	struct amd_ir_data *ir_data = (struct amd_ir_data *)data;
	struct irte_ga *entry = (struct irte_ga *) ir_data->entry;
	struct irq_cfg *cfg = ir_data->cfg;
	u64 valid;

	if (WARN_ON_ONCE(!AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir)))
		return -EINVAL;

	if (!entry || !entry->lo.fields_vapic.guest_mode)
		return 0;

	valid = entry->lo.fields_remap.valid;

	entry->lo.val = 0;
	entry->hi.val = 0;

	entry->lo.fields_remap.valid       = valid;
	entry->lo.fields_remap.dm          = apic->dest_mode_logical;
	entry->lo.fields_remap.int_type    = APIC_DELIVERY_MODE_FIXED;
	entry->hi.fields.vector            = cfg->vector;
	entry->lo.fields_remap.destination =
				APICID_TO_IRTE_DEST_LO(cfg->dest_apicid);
	entry->hi.fields.destination =
				APICID_TO_IRTE_DEST_HI(cfg->dest_apicid);

	return modify_irte_ga(ir_data->iommu, ir_data->irq_2_irte.devid,
			      ir_data->irq_2_irte.index, entry);
}
EXPORT_SYMBOL(amd_iommu_deactivate_guest_mode);

static int amd_ir_set_vcpu_affinity(struct irq_data *data, void *info)
{
	int ret;
	struct amd_iommu_pi_data *pi_data = info;
	struct amd_ir_data *ir_data = data->chip_data;
	struct irq_2_irte *irte_info = &ir_data->irq_2_irte;
	struct iommu_dev_data *dev_data;

	if (WARN_ON_ONCE(!AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir)))
		return -EINVAL;

	if (ir_data->iommu == NULL)
		return -EINVAL;

	dev_data = search_dev_data(ir_data->iommu, irte_info->devid);

	/* Note:
	 * This device has never been set up for guest mode.
	 * we should not modify the IRTE
	 */
	if (!dev_data || !dev_data->use_vapic)
		return -EINVAL;

	ir_data->cfg = irqd_cfg(data);

	if (pi_data) {
		pi_data->ir_data = ir_data;

		ir_data->ga_root_ptr = (pi_data->vapic_addr >> 12);
		ir_data->ga_vector = pi_data->vector;
		ir_data->ga_tag = pi_data->ga_tag;
		if (pi_data->is_guest_mode)
			ret = amd_iommu_activate_guest_mode(ir_data, pi_data->cpu,
							    pi_data->ga_log_intr);
		else
			ret = amd_iommu_deactivate_guest_mode(ir_data);
	} else {
		ret = amd_iommu_deactivate_guest_mode(ir_data);
	}

	return ret;
}


static void amd_ir_update_irte(struct irq_data *irqd, struct amd_iommu *iommu,
			       struct amd_ir_data *ir_data,
			       struct irq_2_irte *irte_info,
			       struct irq_cfg *cfg)
{

	/*
	 * Atomically updates the IRTE with the new destination, vector
	 * and flushes the interrupt entry cache.
	 */
	iommu->irte_ops->set_affinity(iommu, ir_data->entry, irte_info->devid,
				      irte_info->index, cfg->vector,
				      cfg->dest_apicid);
}

static int amd_ir_set_affinity(struct irq_data *data,
			       const struct cpumask *mask, bool force)
{
	struct amd_ir_data *ir_data = data->chip_data;
	struct irq_2_irte *irte_info = &ir_data->irq_2_irte;
	struct irq_cfg *cfg = irqd_cfg(data);
	struct irq_data *parent = data->parent_data;
	struct amd_iommu *iommu = ir_data->iommu;
	int ret;

	if (!iommu)
		return -ENODEV;

	ret = parent->chip->irq_set_affinity(parent, mask, force);
	if (ret < 0 || ret == IRQ_SET_MASK_OK_DONE)
		return ret;

	amd_ir_update_irte(data, iommu, ir_data, irte_info, cfg);
	/*
	 * After this point, all the interrupts will start arriving
	 * at the new destination. So, time to cleanup the previous
	 * vector allocation.
	 */
	vector_schedule_cleanup(cfg);

	return IRQ_SET_MASK_OK_DONE;
}

static void ir_compose_msi_msg(struct irq_data *irq_data, struct msi_msg *msg)
{
	struct amd_ir_data *ir_data = irq_data->chip_data;

	*msg = ir_data->msi_entry;
}

static struct irq_chip amd_ir_chip = {
	.name			= "AMD-IR",
	.irq_ack		= apic_ack_irq,
	.irq_set_affinity	= amd_ir_set_affinity,
	.irq_set_vcpu_affinity	= amd_ir_set_vcpu_affinity,
	.irq_compose_msi_msg	= ir_compose_msi_msg,
};

static const struct msi_parent_ops amdvi_msi_parent_ops = {
	.supported_flags	= X86_VECTOR_MSI_FLAGS_SUPPORTED | MSI_FLAG_MULTI_PCI_MSI,
	.bus_select_token	= DOMAIN_BUS_AMDVI,
	.bus_select_mask	= MATCH_PCI_MSI,
	.prefix			= "IR-",
	.init_dev_msi_info	= msi_parent_init_dev_msi_info,
};

int amd_iommu_create_irq_domain(struct amd_iommu *iommu)
{
	struct irq_domain_info info = {
		.fwnode		= irq_domain_alloc_named_id_fwnode("AMD-IR", iommu->index),
		.ops		= &amd_ir_domain_ops,
		.domain_flags	= IRQ_DOMAIN_FLAG_ISOLATED_MSI,
		.host_data	= iommu,
		.parent		= arch_get_ir_parent_domain(),
	};

	if (!info.fwnode)
		return -ENOMEM;

	iommu->ir_domain = msi_create_parent_irq_domain(&info, &amdvi_msi_parent_ops);
	if (!iommu->ir_domain) {
		irq_domain_free_fwnode(info.fwnode);
		return -ENOMEM;
	}
	return 0;
}
#endif
]

gemini-3.1-pro-preview

							input: 172206
							output: 23
							thoughts: 447
						

**Analyzing the AMD IOMMU Page Mapping Function**

Okay, so I'm looking at this `amd_iommu_map_pages` function, and it seems pretty straightforward. The core of it, from what I'm seeing, is that it takes a domain, an IOVA (Input/Output Virtual Address), a physical address, page size, count, protection flags, and memory allocation flags as input, and tries to map those pages.

The important part is it calls `ops->map_pages`. Which, in turn calls either `v1_map_pages` or `v2_map_pages`. The actual page table manipulation is delegated to that function through the `io_pgtable_ops` structure associated with the domain. I should probably check `io_pgtable_ops->map_pages` in `drivers/iommu/io-pgtable.c` or, since this can vary by platform, maybe even `drivers/iommu/io-pgtable-arm.c`.

A quick mental note – I need to be careful with `*mapped`. Even if the `ops->map_pages` function fails, it might still update the number of pages mapped, so I should remember that.

Let me take a closer look at `arm_lpae_map_pages` in `drivers/iommu/io-pgtable-arm.c`, since that's another important possibility for the implementation, based on the ARM architecture.

map[Expression:arm_lpae_map_pages]

map[Output:drivers/iommu/io-pgtable-arm.c=481=static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data,
--
drivers/iommu/io-pgtable-arm.c-550-
drivers/iommu/io-pgtable-arm.c:551:static int arm_lpae_map_pages(struct io_pgtable_ops *ops, unsigned long iova,
drivers/iommu/io-pgtable-arm.c-552-			      phys_addr_t paddr, size_t pgsize, size_t pgcount,
--
drivers/iommu/io-pgtable-arm.c=919=arm_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg)
--
drivers/iommu/io-pgtable-arm.c-949-	data->iop.ops = (struct io_pgtable_ops) {
drivers/iommu/io-pgtable-arm.c:950:		.map_pages	= arm_lpae_map_pages,
drivers/iommu/io-pgtable-arm.c-951-		.unmap_pages	= arm_lpae_unmap_pages,
]

gemini-3.1-pro-preview

							input: 173327
							output: 52
							thoughts: 15
						

Okay, let's dive into this. Here's how I'd approach summarizing the "arm\_lpae\_map\_pages" function, reflecting the kind of thought process I'd have as someone deep in systems programming:

**Deconstructing `arm_lpae_map_pages`**

Alright, the task is to understand and summarize `arm_lpae_map_pages`. My initial instinct is to treat this as a page mapping function, specifically for the ARM Large Physical Address Extension (LPAE) architecture, which means we're dealing with 64-bit addressing on ARMv8 or later. The name itself is fairly descriptive, so the function's core purpose is clear: to map physical pages to virtual addresses.

I'd immediately think about the low-level details of the memory management unit (MMU) on the target ARM processor. What level of paging are we talking about? Is this a straightforward 4KB page size, or something larger, potentially using multi-level page tables? LPAE strongly suggests multi-level, and that means I need to be thinking in terms of Translation Table Base Registers (TTBRs), Page Table Entries (PTEs), and potentially Instruction TLB (ITLB) and Data TLB (DTLB) flushes.

Now, I'm expecting this function to handle allocating virtual address space. Does it handle reservation of the virtual region? How does it interact with the kernel's virtual memory management, e.g., using `vm_alloc` or similar? What data structures are involved in representing the mappings – are we using a custom page table, or is it integrated with the kernel's existing structures?

I would be particularly interested in error handling. What happens if the physical pages aren't available, or the virtual address space is exhausted? Are we talking about returning an error code, or possibly panicking the system if a critical mapping fails? And, are there any mechanisms to deal with concurrent access to the mapping structures, especially if there's any form of shared memory involved? I'd expect locks or atomic operations to ensure consistency.

The choice of `arm_lpae_map_pages` over other possible names is interesting. Is there a specific reason to use this naming? Are we working with a specialized or limited operating system that has custom page table setup?

requested entity does not exist

map[ContextFile:drivers/iommu/io-pgtable-arm.c IncludeLines:true Name:arm_lpae_map_pages]

map[SourceCode: SourceFile:]

gemini-3.1-pro-preview

							input: 173909
							output: 41
							thoughts: 19
						

Okay, here's my summarized thought process, presented as if I were thinking it:

**Analyzing `arm_lpae_map_pages` with grepper**

Alright, time to get to work. I need to understand this `arm_lpae_map_pages` function more deeply. Given my expertise, the first logical step is to quickly dive into its implementation and usage. My intuition screams "use grepper!" It's the perfect tool for quickly navigating and understanding the function's context within the broader codebase. I need to see exactly where it's called, what data it operates on, and how its return values are handled. With grepper, I'm expecting to immediately gain a critical overview. I anticipate it will help me identify potential areas of interest for a more in-depth code review. I'm hoping to quickly identify the function's purpose within the larger system architecture. Let's get started.

map[Expression:static int arm_lpae_map_pages\(struct io_pgtable_ops \*ops, unsigned long iova,]

map[Output:drivers/iommu/io-pgtable-arm.c=481=static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data,
--
drivers/iommu/io-pgtable-arm.c-550-
drivers/iommu/io-pgtable-arm.c:551:static int arm_lpae_map_pages(struct io_pgtable_ops *ops, unsigned long iova,
drivers/iommu/io-pgtable-arm.c-552-			      phys_addr_t paddr, size_t pgsize, size_t pgcount,
]

gemini-3.1-pro-preview

							input: 174344
							output: 41
							thoughts: 30
						

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Analyzing the ARM I/O Page Table Code**

Okay, time to dive into `drivers/iommu/io-pgtable-arm.c`, specifically lines 550 to 600. I need to refresh my memory on the intricacies of the ARM I/O page table implementation. I remember that the ARM architecture has several IOMMU implementations, and understanding this particular chunk of code is key to grasp the specifics. I'll be looking for how they are handling the translation process and any potential optimizations within this context. I'm especially keen to see how they're handling faulting and any potential performance bottlenecks that could arise from virtual address translation. I will scrutinize the code for any potential concurrency issues within the translation process. Let me get to it and see what I can find.

map[Expression:static int arm_lpae_map_pages\(struct io_pgtable_ops \*ops, unsigned long iova,]

map[Output:drivers/iommu/io-pgtable-arm.c=481=static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data,
--
drivers/iommu/io-pgtable-arm.c-550-
drivers/iommu/io-pgtable-arm.c:551:static int arm_lpae_map_pages(struct io_pgtable_ops *ops, unsigned long iova,
drivers/iommu/io-pgtable-arm.c-552-			      phys_addr_t paddr, size_t pgsize, size_t pgcount,
]

gemini-3.1-pro-preview

							input: 174780
							output: 29
							thoughts: 58
						

Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:

**Initial Investigation & Refinement**

Alright, I'm thinking about how to approach this. My first instinct is to reach for `grepper`. I'd immediately lean towards a command line approach, specifically something with a context search. I want to see a reasonable amount of surrounding code, maybe `-A 50` or something comparable to get a good handle on where something's being used and in what context. Wait, no, hold on. I’m pretty sure `grepper` (or at least the version/implementation I'm most familiar with) doesn't natively support that `-A` argument. Scratch that idea. I need to pivot. Okay, let's take a closer look at the codebase. Looks like I'll be reading `drivers/iommu/io-pgtable-arm.c` and digging into the core implementation directly. I'll need to use `read-file` to access and analyze the contents.

map[File:drivers/iommu/io-pgtable-arm.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
 * CPU-agnostic ARM page table allocator.
 *
 * Copyright (C) 2014 ARM Limited
 *
 * Author: Will Deacon <will.deacon@arm.com>
 */

#define pr_fmt(fmt)	"arm-lpae io-pgtable: " fmt

#include <linux/atomic.h>
#include <linux/bitops.h>
#include <linux/io-pgtable.h>
#include <linux/kernel.h>
#include <linux/device/faux.h>
#include <linux/sizes.h>
#include <linux/slab.h>
#include <linux/types.h>
#include <linux/dma-mapping.h>

#include <asm/barrier.h>

#include "io-pgtable-arm.h"
#include "iommu-pages.h"

#define ARM_LPAE_MAX_ADDR_BITS		52
#define ARM_LPAE_S2_MAX_CONCAT_PAGES	16
#define ARM_LPAE_MAX_LEVELS		4

/* Struct accessors */
#define io_pgtable_to_data(x)						\
	container_of((x), struct arm_lpae_io_pgtable, iop)

#define io_pgtable_ops_to_data(x)					\
	io_pgtable_to_data(io_pgtable_ops_to_pgtable(x))

/*
 * Calculate the right shift amount to get to the portion describing level l
 * in a virtual address mapped by the pagetable in d.
 */
#define ARM_LPAE_LVL_SHIFT(l,d)						\
	(((ARM_LPAE_MAX_LEVELS - (l)) * (d)->bits_per_level) +		\
	ilog2(sizeof(arm_lpae_iopte)))

#define ARM_LPAE_GRANULE(d)						\
	(sizeof(arm_lpae_iopte) << (d)->bits_per_level)
#define ARM_LPAE_PGD_SIZE(d)						\
	(sizeof(arm_lpae_iopte) << (d)->pgd_bits)

#define ARM_LPAE_PTES_PER_TABLE(d)					\
	(ARM_LPAE_GRANULE(d) >> ilog2(sizeof(arm_lpae_iopte)))

/*
 * Calculate the index at level l used to map virtual address a using the
 * pagetable in d.
 */
#define ARM_LPAE_PGD_IDX(l,d)						\
	((l) == (d)->start_level ? (d)->pgd_bits - (d)->bits_per_level : 0)

#define ARM_LPAE_LVL_IDX(a,l,d)						\
	(((u64)(a) >> ARM_LPAE_LVL_SHIFT(l,d)) &			\
	 ((1 << ((d)->bits_per_level + ARM_LPAE_PGD_IDX(l,d))) - 1))

/* Calculate the block/page mapping size at level l for pagetable in d. */
#define ARM_LPAE_BLOCK_SIZE(l,d)	(1ULL << ARM_LPAE_LVL_SHIFT(l,d))

/* Page table bits */
#define ARM_LPAE_PTE_TYPE_SHIFT		0
#define ARM_LPAE_PTE_TYPE_MASK		0x3

#define ARM_LPAE_PTE_TYPE_BLOCK		1
#define ARM_LPAE_PTE_TYPE_TABLE		3
#define ARM_LPAE_PTE_TYPE_PAGE		3

#define ARM_LPAE_PTE_ADDR_MASK		GENMASK_ULL(47,12)

#define ARM_LPAE_PTE_NSTABLE		(((arm_lpae_iopte)1) << 63)
#define ARM_LPAE_PTE_XN			(((arm_lpae_iopte)3) << 53)
#define ARM_LPAE_PTE_DBM		(((arm_lpae_iopte)1) << 51)
#define ARM_LPAE_PTE_AF			(((arm_lpae_iopte)1) << 10)
#define ARM_LPAE_PTE_SH_NS		(((arm_lpae_iopte)0) << 8)
#define ARM_LPAE_PTE_SH_OS		(((arm_lpae_iopte)2) << 8)
#define ARM_LPAE_PTE_SH_IS		(((arm_lpae_iopte)3) << 8)
#define ARM_LPAE_PTE_NS			(((arm_lpae_iopte)1) << 5)
#define ARM_LPAE_PTE_VALID		(((arm_lpae_iopte)1) << 0)

/* Software bit for solving coherency races */
#define ARM_LPAE_PTE_SW_SYNC		(((arm_lpae_iopte)1) << 55)

/* Stage-1 PTE */
#define ARM_LPAE_PTE_AP_UNPRIV		(((arm_lpae_iopte)1) << 6)
#define ARM_LPAE_PTE_AP_RDONLY_BIT	7
#define ARM_LPAE_PTE_AP_RDONLY		(((arm_lpae_iopte)1) << \
					   ARM_LPAE_PTE_AP_RDONLY_BIT)
#define ARM_LPAE_PTE_AP_WR_CLEAN_MASK	(ARM_LPAE_PTE_AP_RDONLY | \
					 ARM_LPAE_PTE_DBM)
#define ARM_LPAE_PTE_ATTRINDX_SHIFT	2
#define ARM_LPAE_PTE_nG			(((arm_lpae_iopte)1) << 11)

/* Stage-2 PTE */
#define ARM_LPAE_PTE_HAP_FAULT		(((arm_lpae_iopte)0) << 6)
#define ARM_LPAE_PTE_HAP_READ		(((arm_lpae_iopte)1) << 6)
#define ARM_LPAE_PTE_HAP_WRITE		(((arm_lpae_iopte)2) << 6)
/*
 * For !FWB these code to:
 *  1111 = Normal outer write back cachable / Inner Write Back Cachable
 *         Permit S1 to override
 *  0101 = Normal Non-cachable / Inner Non-cachable
 *  0001 = Device / Device-nGnRE
 * For S2FWB these code:
 *  0110 Force Normal Write Back
 *  0101 Normal* is forced Normal-NC, Device unchanged
 *  0001 Force Device-nGnRE
 */
#define ARM_LPAE_PTE_MEMATTR_FWB_WB	(((arm_lpae_iopte)0x6) << 2)
#define ARM_LPAE_PTE_MEMATTR_OIWB	(((arm_lpae_iopte)0xf) << 2)
#define ARM_LPAE_PTE_MEMATTR_NC		(((arm_lpae_iopte)0x5) << 2)
#define ARM_LPAE_PTE_MEMATTR_DEV	(((arm_lpae_iopte)0x1) << 2)

/* Register bits */
#define ARM_LPAE_VTCR_SL0_MASK		0x3

#define ARM_LPAE_TCR_T0SZ_SHIFT		0

#define ARM_LPAE_VTCR_PS_SHIFT		16
#define ARM_LPAE_VTCR_PS_MASK		0x7

#define ARM_LPAE_MAIR_ATTR_SHIFT(n)	((n) << 3)
#define ARM_LPAE_MAIR_ATTR_MASK		0xff
#define ARM_LPAE_MAIR_ATTR_DEVICE	0x04
#define ARM_LPAE_MAIR_ATTR_NC		0x44
#define ARM_LPAE_MAIR_ATTR_INC_OWBRWA	0xf4
#define ARM_LPAE_MAIR_ATTR_WBRWA	0xff
#define ARM_LPAE_MAIR_ATTR_IDX_NC	0
#define ARM_LPAE_MAIR_ATTR_IDX_CACHE	1
#define ARM_LPAE_MAIR_ATTR_IDX_DEV	2
#define ARM_LPAE_MAIR_ATTR_IDX_INC_OCACHE	3

#define ARM_MALI_LPAE_TTBR_ADRMODE_TABLE (3u << 0)
#define ARM_MALI_LPAE_TTBR_READ_INNER	BIT(2)
#define ARM_MALI_LPAE_TTBR_SHARE_OUTER	BIT(4)

#define ARM_MALI_LPAE_MEMATTR_IMP_DEF	0x88ULL
#define ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC 0x8DULL

/* IOPTE accessors */
#define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d))

#define iopte_type(pte)					\
	(((pte) >> ARM_LPAE_PTE_TYPE_SHIFT) & ARM_LPAE_PTE_TYPE_MASK)

#define iopte_writeable_dirty(pte)				\
	(((pte) & ARM_LPAE_PTE_AP_WR_CLEAN_MASK) == ARM_LPAE_PTE_DBM)

#define iopte_set_writeable_clean(ptep)				\
	set_bit(ARM_LPAE_PTE_AP_RDONLY_BIT, (unsigned long *)(ptep))

struct arm_lpae_io_pgtable {
	struct io_pgtable	iop;

	int			pgd_bits;
	int			start_level;
	int			bits_per_level;

	void			*pgd;
};

typedef u64 arm_lpae_iopte;

static inline bool iopte_leaf(arm_lpae_iopte pte, int lvl,
			      enum io_pgtable_fmt fmt)
{
	if (lvl == (ARM_LPAE_MAX_LEVELS - 1) && fmt != ARM_MALI_LPAE)
		return iopte_type(pte) == ARM_LPAE_PTE_TYPE_PAGE;

	return iopte_type(pte) == ARM_LPAE_PTE_TYPE_BLOCK;
}

static inline bool iopte_table(arm_lpae_iopte pte, int lvl)
{
	if (lvl == (ARM_LPAE_MAX_LEVELS - 1))
		return false;
	return iopte_type(pte) == ARM_LPAE_PTE_TYPE_TABLE;
}

static arm_lpae_iopte paddr_to_iopte(phys_addr_t paddr,
				     struct arm_lpae_io_pgtable *data)
{
	arm_lpae_iopte pte = paddr;

	/* Of the bits which overlap, either 51:48 or 15:12 are always RES0 */
	return (pte | (pte >> (48 - 12))) & ARM_LPAE_PTE_ADDR_MASK;
}

static phys_addr_t iopte_to_paddr(arm_lpae_iopte pte,
				  struct arm_lpae_io_pgtable *data)
{
	u64 paddr = pte & ARM_LPAE_PTE_ADDR_MASK;

	if (ARM_LPAE_GRANULE(data) < SZ_64K)
		return paddr;

	/* Rotate the packed high-order bits back to the top */
	return (paddr | (paddr << (48 - 12))) & (ARM_LPAE_PTE_ADDR_MASK << 4);
}

/*
 * Convert an index returned by ARM_LPAE_PGD_IDX(), which can point into
 * a concatenated PGD, into the maximum number of entries that can be
 * mapped in the same table page.
 */
static inline int arm_lpae_max_entries(int i, struct arm_lpae_io_pgtable *data)
{
	int ptes_per_table = ARM_LPAE_PTES_PER_TABLE(data);

	return ptes_per_table - (i & (ptes_per_table - 1));
}

/*
 * Check if concatenated PGDs are mandatory according to Arm DDI0487 (K.a)
 * 1) R_DXBSH: For 16KB, and 48-bit input size, use level 1 instead of 0.
 * 2) R_SRKBC: After de-ciphering the table for PA size and valid initial lookup
 *   a) 40 bits PA size with 4K: use level 1 instead of level 0 (2 tables for ias = oas)
 *   b) 40 bits PA size with 16K: use level 2 instead of level 1 (16 tables for ias = oas)
 *   c) 42 bits PA size with 4K: use level 1 instead of level 0 (8 tables for ias = oas)
 *   d) 48 bits PA size with 16K: use level 1 instead of level 0 (2 tables for ias = oas)
 */
static inline bool arm_lpae_concat_mandatory(struct io_pgtable_cfg *cfg,
					     struct arm_lpae_io_pgtable *data)
{
	unsigned int ias = cfg->ias;
	unsigned int oas = cfg->oas;

	/* Covers 1 and 2.d */
	if ((ARM_LPAE_GRANULE(data) == SZ_16K) && (data->start_level == 0))
		return (oas == 48) || (ias == 48);

	/* Covers 2.a and 2.c */
	if ((ARM_LPAE_GRANULE(data) == SZ_4K) && (data->start_level == 0))
		return (oas == 40) || (oas == 42);

	/* Case 2.b */
	return (ARM_LPAE_GRANULE(data) == SZ_16K) &&
	       (data->start_level == 1) && (oas == 40);
}

static dma_addr_t __arm_lpae_dma_addr(void *pages)
{
	return (dma_addr_t)virt_to_phys(pages);
}

static void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp,
				    struct io_pgtable_cfg *cfg,
				    void *cookie)
{
	struct device *dev = cfg->iommu_dev;
	size_t alloc_size;
	dma_addr_t dma;
	void *pages;

	/*
	 * For very small starting-level translation tables the HW requires a
	 * minimum alignment of at least 64 to cover all cases.
	 */
	alloc_size = max(size, 64);
	if (cfg->alloc)
		pages = cfg->alloc(cookie, alloc_size, gfp);
	else
		pages = iommu_alloc_pages_node_sz(dev_to_node(dev), gfp,
						  alloc_size);

	if (!pages)
		return NULL;

	if (!cfg->coherent_walk) {
		dma = dma_map_single(dev, pages, size, DMA_TO_DEVICE);
		if (dma_mapping_error(dev, dma))
			goto out_free;
		/*
		 * We depend on the IOMMU being able to work with any physical
		 * address directly, so if the DMA layer suggests otherwise by
		 * translating or truncating them, that bodes very badly...
		 */
		if (dma != virt_to_phys(pages))
			goto out_unmap;
	}

	return pages;

out_unmap:
	dev_err(dev, "Cannot accommodate DMA translation for IOMMU page tables\n");
	dma_unmap_single(dev, dma, size, DMA_TO_DEVICE);

out_free:
	if (cfg->free)
		cfg->free(cookie, pages, size);
	else
		iommu_free_pages(pages);

	return NULL;
}

static void __arm_lpae_free_pages(void *pages, size_t size,
				  struct io_pgtable_cfg *cfg,
				  void *cookie)
{
	if (!cfg->coherent_walk)
		dma_unmap_single(cfg->iommu_dev, __arm_lpae_dma_addr(pages),
				 size, DMA_TO_DEVICE);

	if (cfg->free)
		cfg->free(cookie, pages, size);
	else
		iommu_free_pages(pages);
}

static void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries,
				struct io_pgtable_cfg *cfg)
{
	dma_sync_single_for_device(cfg->iommu_dev, __arm_lpae_dma_addr(ptep),
				   sizeof(*ptep) * num_entries, DMA_TO_DEVICE);
}

static void __arm_lpae_clear_pte(arm_lpae_iopte *ptep, struct io_pgtable_cfg *cfg, int num_entries)
{
	for (int i = 0; i < num_entries; i++)
		ptep[i] = 0;

	if (!cfg->coherent_walk && num_entries)
		__arm_lpae_sync_pte(ptep, num_entries, cfg);
}

static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data,
			       struct iommu_iotlb_gather *gather,
			       unsigned long iova, size_t size, size_t pgcount,
			       int lvl, arm_lpae_iopte *ptep);

static void __arm_lpae_init_pte(struct arm_lpae_io_pgtable *data,
				phys_addr_t paddr, arm_lpae_iopte prot,
				int lvl, int num_entries, arm_lpae_iopte *ptep)
{
	arm_lpae_iopte pte = prot;
	struct io_pgtable_cfg *cfg = &data->iop.cfg;
	size_t sz = ARM_LPAE_BLOCK_SIZE(lvl, data);
	int i;

	if (data->iop.fmt != ARM_MALI_LPAE && lvl == ARM_LPAE_MAX_LEVELS - 1)
		pte |= ARM_LPAE_PTE_TYPE_PAGE;
	else
		pte |= ARM_LPAE_PTE_TYPE_BLOCK;

	for (i = 0; i < num_entries; i++)
		ptep[i] = pte | paddr_to_iopte(paddr + i * sz, data);

	if (!cfg->coherent_walk)
		__arm_lpae_sync_pte(ptep, num_entries, cfg);
}

static int arm_lpae_init_pte(struct arm_lpae_io_pgtable *data,
			     unsigned long iova, phys_addr_t paddr,
			     arm_lpae_iopte prot, int lvl, int num_entries,
			     arm_lpae_iopte *ptep)
{
	int i;

	for (i = 0; i < num_entries; i++)
		if (iopte_leaf(ptep[i], lvl, data->iop.fmt)) {
			/* We require an unmap first */
			WARN_ON(!(data->iop.cfg.quirks & IO_PGTABLE_QUIRK_NO_WARN));
			return -EEXIST;
		} else if (iopte_type(ptep[i]) == ARM_LPAE_PTE_TYPE_TABLE) {
			/*
			 * We need to unmap and free the old table before
			 * overwriting it with a block entry.
			 */
			arm_lpae_iopte *tblp;
			size_t sz = ARM_LPAE_BLOCK_SIZE(lvl, data);

			tblp = ptep - ARM_LPAE_LVL_IDX(iova, lvl, data);
			if (__arm_lpae_unmap(data, NULL, iova + i * sz, sz, 1,
					     lvl, tblp) != sz) {
				WARN_ON(1);
				return -EINVAL;
			}
		}

	__arm_lpae_init_pte(data, paddr, prot, lvl, num_entries, ptep);
	return 0;
}

static arm_lpae_iopte arm_lpae_install_table(arm_lpae_iopte *table,
					     arm_lpae_iopte *ptep,
					     arm_lpae_iopte curr,
					     struct arm_lpae_io_pgtable *data)
{
	arm_lpae_iopte old, new;
	struct io_pgtable_cfg *cfg = &data->iop.cfg;

	new = paddr_to_iopte(__pa(table), data) | ARM_LPAE_PTE_TYPE_TABLE;
	if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_NS)
		new |= ARM_LPAE_PTE_NSTABLE;

	/*
	 * Ensure the table itself is visible before its PTE can be.
	 * Whilst we could get away with cmpxchg64_release below, this
	 * doesn't have any ordering semantics when !CONFIG_SMP.
	 */
	dma_wmb();

	old = cmpxchg64_relaxed(ptep, curr, new);

	if (cfg->coherent_walk || (old & ARM_LPAE_PTE_SW_SYNC))
		return old;

	/* Even if it's not ours, there's no point waiting; just kick it */
	__arm_lpae_sync_pte(ptep, 1, cfg);
	if (old == curr)
		WRITE_ONCE(*ptep, new | ARM_LPAE_PTE_SW_SYNC);

	return old;
}

static int __arm_lpae_map(struct arm_lpae_io_pgtable *data, unsigned long iova,
			  phys_addr_t paddr, size_t size, size_t pgcount,
			  arm_lpae_iopte prot, int lvl, arm_lpae_iopte *ptep,
			  gfp_t gfp, size_t *mapped)
{
	arm_lpae_iopte *cptep, pte;
	size_t block_size = ARM_LPAE_BLOCK_SIZE(lvl, data);
	size_t tblsz = ARM_LPAE_GRANULE(data);
	struct io_pgtable_cfg *cfg = &data->iop.cfg;
	int ret = 0, num_entries, max_entries, map_idx_start;

	/* Find our entry at the current level */
	map_idx_start = ARM_LPAE_LVL_IDX(iova, lvl, data);
	ptep += map_idx_start;

	/* If we can install a leaf entry at this level, then do so */
	if (size == block_size) {
		max_entries = arm_lpae_max_entries(map_idx_start, data);
		num_entries = min_t(int, pgcount, max_entries);
		ret = arm_lpae_init_pte(data, iova, paddr, prot, lvl, num_entries, ptep);
		if (!ret)
			*mapped += num_entries * size;

		return ret;
	}

	/* We can't allocate tables at the final level */
	if (WARN_ON(lvl >= ARM_LPAE_MAX_LEVELS - 1))
		return -EINVAL;

	/* Grab a pointer to the next level */
	pte = READ_ONCE(*ptep);
	if (!pte) {
		cptep = __arm_lpae_alloc_pages(tblsz, gfp, cfg, data->iop.cookie);
		if (!cptep)
			return -ENOMEM;

		pte = arm_lpae_install_table(cptep, ptep, 0, data);
		if (pte)
			__arm_lpae_free_pages(cptep, tblsz, cfg, data->iop.cookie);
	} else if (!cfg->coherent_walk && !(pte & ARM_LPAE_PTE_SW_SYNC)) {
		__arm_lpae_sync_pte(ptep, 1, cfg);
	}

	if (pte && !iopte_leaf(pte, lvl, data->iop.fmt)) {
		cptep = iopte_deref(pte, data);
	} else if (pte) {
		/* We require an unmap first */
		WARN_ON(!(cfg->quirks & IO_PGTABLE_QUIRK_NO_WARN));
		return -EEXIST;
	}

	/* Rinse, repeat */
	return __arm_lpae_map(data, iova, paddr, size, pgcount, prot, lvl + 1,
			      cptep, gfp, mapped);
}

static arm_lpae_iopte arm_lpae_prot_to_pte(struct arm_lpae_io_pgtable *data,
					   int prot)
{
	arm_lpae_iopte pte;

	if (data->iop.fmt == ARM_64_LPAE_S1 ||
	    data->iop.fmt == ARM_32_LPAE_S1) {
		pte = ARM_LPAE_PTE_nG;
		if (!(prot & IOMMU_WRITE) && (prot & IOMMU_READ))
			pte |= ARM_LPAE_PTE_AP_RDONLY;
		else if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_ARM_HD)
			pte |= ARM_LPAE_PTE_DBM;
		if (!(prot & IOMMU_PRIV))
			pte |= ARM_LPAE_PTE_AP_UNPRIV;
	} else {
		pte = ARM_LPAE_PTE_HAP_FAULT;
		if (prot & IOMMU_READ)
			pte |= ARM_LPAE_PTE_HAP_READ;
		if (prot & IOMMU_WRITE)
			pte |= ARM_LPAE_PTE_HAP_WRITE;
	}

	/*
	 * Note that this logic is structured to accommodate Mali LPAE
	 * having stage-1-like attributes but stage-2-like permissions.
	 */
	if (data->iop.fmt == ARM_64_LPAE_S2 ||
	    data->iop.fmt == ARM_32_LPAE_S2) {
		if (prot & IOMMU_MMIO) {
			pte |= ARM_LPAE_PTE_MEMATTR_DEV;
		} else if (prot & IOMMU_CACHE) {
			if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_ARM_S2FWB)
				pte |= ARM_LPAE_PTE_MEMATTR_FWB_WB;
			else
				pte |= ARM_LPAE_PTE_MEMATTR_OIWB;
		} else {
			pte |= ARM_LPAE_PTE_MEMATTR_NC;
		}
	} else {
		if (prot & IOMMU_MMIO)
			pte |= (ARM_LPAE_MAIR_ATTR_IDX_DEV
				<< ARM_LPAE_PTE_ATTRINDX_SHIFT);
		else if (prot & IOMMU_CACHE)
			pte |= (ARM_LPAE_MAIR_ATTR_IDX_CACHE
				<< ARM_LPAE_PTE_ATTRINDX_SHIFT);
	}

	/*
	 * Also Mali has its own notions of shareability wherein its Inner
	 * domain covers the cores within the GPU, and its Outer domain is
	 * "outside the GPU" (i.e. either the Inner or System domain in CPU
	 * terms, depending on coherency).
	 */
	if (prot & IOMMU_CACHE && data->iop.fmt != ARM_MALI_LPAE)
		pte |= ARM_LPAE_PTE_SH_IS;
	else
		pte |= ARM_LPAE_PTE_SH_OS;

	if (prot & IOMMU_NOEXEC)
		pte |= ARM_LPAE_PTE_XN;

	if (data->iop.cfg.quirks & IO_PGTABLE_QUIRK_ARM_NS)
		pte |= ARM_LPAE_PTE_NS;

	if (data->iop.fmt != ARM_MALI_LPAE)
		pte |= ARM_LPAE_PTE_AF;

	return pte;
}

static int arm_lpae_map_pages(struct io_pgtable_ops *ops, unsigned long iova,
			      phys_addr_t paddr, size_t pgsize, size_t pgcount,
			      int iommu_prot, gfp_t gfp, size_t *mapped)
{
	struct arm_lpae_io_pgtable *data = io_pgtable_ops_to_data(ops);
	struct io_pgtable_cfg *cfg = &data->iop.cfg;
	arm_lpae_iopte *ptep = data->pgd;
	int ret, lvl = data->start_level;
	arm_lpae_iopte prot;
	long iaext = (s64)iova >> cfg->ias;

	if (WARN_ON(!pgsize || (pgsize & cfg->pgsize_bitmap) != pgsize))
		return -EINVAL;

	if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_TTBR1)
		iaext = ~iaext;
	if (WARN_ON(iaext || paddr >> cfg->oas))
		return -ERANGE;

	if (!(iommu_prot & (IOMMU_READ | IOMMU_WRITE)))
		return -EINVAL;

	prot = arm_lpae_prot_to_pte(data, iommu_prot);
	ret = __arm_lpae_map(data, iova, paddr, pgsize, pgcount, prot, lvl,
			     ptep, gfp, mapped);
	/*
	 * Synchronise all PTE updates for the new mapping before there's
	 * a chance for anything to kick off a table walk for the new iova.
	 */
	wmb();

	return ret;
}

static void __arm_lpae_free_pgtable(struct arm_lpae_io_pgtable *data, int lvl,
				    arm_lpae_iopte *ptep)
{
	arm_lpae_iopte *start, *end;
	unsigned long table_size;

	if (lvl == data->start_level)
		table_size = ARM_LPAE_PGD_SIZE(data);
	else
		table_size = ARM_LPAE_GRANULE(data);

	start = ptep;

	/* Only leaf entries at the last level */
	if (lvl == ARM_LPAE_MAX_LEVELS - 1)
		end = ptep;
	else
		end = (void *)ptep + table_size;

	while (ptep != end) {
		arm_lpae_iopte pte = *ptep++;

		if (!pte || iopte_leaf(pte, lvl, data->iop.fmt))
			continue;

		__arm_lpae_free_pgtable(data, lvl + 1, iopte_deref(pte, data));
	}

	__arm_lpae_free_pages(start, table_size, &data->iop.cfg, data->iop.cookie);
}

static void arm_lpae_free_pgtable(struct io_pgtable *iop)
{
	struct arm_lpae_io_pgtable *data = io_pgtable_to_data(iop);

	__arm_lpae_free_pgtable(data, data->start_level, data->pgd);
	kfree(data);
}

static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data,
			       struct iommu_iotlb_gather *gather,
			       unsigned long iova, size_t size, size_t pgcount,
			       int lvl, arm_lpae_iopte *ptep)
{
	arm_lpae_iopte pte;
	struct io_pgtable *iop = &data->iop;
	int i = 0, num_entries, max_entries, unmap_idx_start;

	/* Something went horribly wrong and we ran out of page table */
	if (WARN_ON(lvl == ARM_LPAE_MAX_LEVELS))
		return 0;

	unmap_idx_start = ARM_LPAE_LVL_IDX(iova, lvl, data);
	ptep += unmap_idx_start;
	pte = READ_ONCE(*ptep);
	if (!pte) {
		WARN_ON(!(data->iop.cfg.quirks & IO_PGTABLE_QUIRK_NO_WARN));
		return -ENOENT;
	}

	/* If the size matches this level, we're in the right place */
	if (size == ARM_LPAE_BLOCK_SIZE(lvl, data)) {
		max_entries = arm_lpae_max_entries(unmap_idx_start, data);
		num_entries = min_t(int, pgcount, max_entries);

		/* Find and handle non-leaf entries */
		for (i = 0; i < num_entries; i++) {
			pte = READ_ONCE(ptep[i]);
			if (!pte) {
				WARN_ON(!(data->iop.cfg.quirks & IO_PGTABLE_QUIRK_NO_WARN));
				break;
			}

			if (!iopte_leaf(pte, lvl, iop->fmt)) {
				__arm_lpae_clear_pte(&ptep[i], &iop->cfg, 1);

				/* Also flush any partial walks */
				io_pgtable_tlb_flush_walk(iop, iova + i * size, size,
							  ARM_LPAE_GRANULE(data));
				__arm_lpae_free_pgtable(data, lvl + 1, iopte_deref(pte, data));
			}
		}

		/* Clear the remaining entries */
		__arm_lpae_clear_pte(ptep, &iop->cfg, i);

		if (gather && !iommu_iotlb_gather_queued(gather))
			for (int j = 0; j < i; j++)
				io_pgtable_tlb_add_page(iop, gather, iova + j * size, size);

		return i * size;
	} else if (iopte_leaf(pte, lvl, iop->fmt)) {
		WARN_ONCE(true, "Unmap of a partial large IOPTE is not allowed");
		return 0;
	}

	/* Keep on walkin' */
	ptep = iopte_deref(pte, data);
	return __arm_lpae_unmap(data, gather, iova, size, pgcount, lvl + 1, ptep);
}

static size_t arm_lpae_unmap_pages(struct io_pgtable_ops *ops, unsigned long iova,
				   size_t pgsize, size_t pgcount,
				   struct iommu_iotlb_gather *gather)
{
	struct arm_lpae_io_pgtable *data = io_pgtable_ops_to_data(ops);
	struct io_pgtable_cfg *cfg = &data->iop.cfg;
	arm_lpae_iopte *ptep = data->pgd;
	long iaext = (s64)iova >> cfg->ias;

	if (WARN_ON(!pgsize || (pgsize & cfg->pgsize_bitmap) != pgsize || !pgcount))
		return 0;

	if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_TTBR1)
		iaext = ~iaext;
	if (WARN_ON(iaext))
		return 0;

	return __arm_lpae_unmap(data, gather, iova, pgsize, pgcount,
				data->start_level, ptep);
}

struct io_pgtable_walk_data {
	struct io_pgtable		*iop;
	void				*data;
	int (*visit)(struct io_pgtable_walk_data *walk_data, int lvl,
		     arm_lpae_iopte *ptep, size_t size);
	unsigned long			flags;
	u64				addr;
	const u64			end;
};

static int __arm_lpae_iopte_walk(struct arm_lpae_io_pgtable *data,
				 struct io_pgtable_walk_data *walk_data,
				 arm_lpae_iopte *ptep,
				 int lvl);

struct iova_to_phys_data {
	arm_lpae_iopte pte;
	int lvl;
};

static int visit_iova_to_phys(struct io_pgtable_walk_data *walk_data, int lvl,
			      arm_lpae_iopte *ptep, size_t size)
{
	struct iova_to_phys_data *data = walk_data->data;
	data->pte = *ptep;
	data->lvl = lvl;
	return 0;
}

static phys_addr_t arm_lpae_iova_to_phys(struct io_pgtable_ops *ops,
					 unsigned long iova)
{
	struct arm_lpae_io_pgtable *data = io_pgtable_ops_to_data(ops);
	struct iova_to_phys_data d;
	struct io_pgtable_walk_data walk_data = {
		.data = &d,
		.visit = visit_iova_to_phys,
		.addr = iova,
		.end = iova + 1,
	};
	int ret;

	ret = __arm_lpae_iopte_walk(data, &walk_data, data->pgd, data->start_level);
	if (ret)
		return 0;

	iova &= (ARM_LPAE_BLOCK_SIZE(d.lvl, data) - 1);
	return iopte_to_paddr(d.pte, data) | iova;
}

static int visit_pgtable_walk(struct io_pgtable_walk_data *walk_data, int lvl,
			      arm_lpae_iopte *ptep, size_t size)
{
	struct arm_lpae_io_pgtable_walk_data *data = walk_data->data;
	data->ptes[lvl] = *ptep;
	return 0;
}

static int arm_lpae_pgtable_walk(struct io_pgtable_ops *ops, unsigned long iova,
				 void *wd)
{
	struct arm_lpae_io_pgtable *data = io_pgtable_ops_to_data(ops);
	struct io_pgtable_walk_data walk_data = {
		.data = wd,
		.visit = visit_pgtable_walk,
		.addr = iova,
		.end = iova + 1,
	};

	return __arm_lpae_iopte_walk(data, &walk_data, data->pgd, data->start_level);
}

static int io_pgtable_visit(struct arm_lpae_io_pgtable *data,
			    struct io_pgtable_walk_data *walk_data,
			    arm_lpae_iopte *ptep, int lvl)
{
	struct io_pgtable *iop = &data->iop;
	arm_lpae_iopte pte = READ_ONCE(*ptep);

	size_t size = ARM_LPAE_BLOCK_SIZE(lvl, data);
	int ret = walk_data->visit(walk_data, lvl, ptep, size);
	if (ret)
		return ret;

	if (iopte_leaf(pte, lvl, iop->fmt)) {
		walk_data->addr += size;
		return 0;
	}

	if (!iopte_table(pte, lvl)) {
		return -EINVAL;
	}

	ptep = iopte_deref(pte, data);
	return __arm_lpae_iopte_walk(data, walk_data, ptep, lvl + 1);
}

static int __arm_lpae_iopte_walk(struct arm_lpae_io_pgtable *data,
				 struct io_pgtable_walk_data *walk_data,
				 arm_lpae_iopte *ptep,
				 int lvl)
{
	u32 idx;
	int max_entries, ret;

	if (WARN_ON(lvl == ARM_LPAE_MAX_LEVELS))
		return -EINVAL;

	if (lvl == data->start_level)
		max_entries = ARM_LPAE_PGD_SIZE(data) / sizeof(arm_lpae_iopte);
	else
		max_entries = ARM_LPAE_PTES_PER_TABLE(data);

	for (idx = ARM_LPAE_LVL_IDX(walk_data->addr, lvl, data);
	     (idx < max_entries) && (walk_data->addr < walk_data->end); ++idx) {
		ret = io_pgtable_visit(data, walk_data, ptep + idx, lvl);
		if (ret)
			return ret;
	}

	return 0;
}

static int visit_dirty(struct io_pgtable_walk_data *walk_data, int lvl,
		       arm_lpae_iopte *ptep, size_t size)
{
	struct iommu_dirty_bitmap *dirty = walk_data->data;

	if (!iopte_leaf(*ptep, lvl, walk_data->iop->fmt))
		return 0;

	if (iopte_writeable_dirty(*ptep)) {
		iommu_dirty_bitmap_record(dirty, walk_data->addr, size);
		if (!(walk_data->flags & IOMMU_DIRTY_NO_CLEAR))
			iopte_set_writeable_clean(ptep);
	}

	return 0;
}

static int arm_lpae_read_and_clear_dirty(struct io_pgtable_ops *ops,
					 unsigned long iova, size_t size,
					 unsigned long flags,
					 struct iommu_dirty_bitmap *dirty)
{
	struct arm_lpae_io_pgtable *data = io_pgtable_ops_to_data(ops);
	struct io_pgtable_cfg *cfg = &data->iop.cfg;
	struct io_pgtable_walk_data walk_data = {
		.iop = &data->iop,
		.data = dirty,
		.visit = visit_dirty,
		.flags = flags,
		.addr = iova,
		.end = iova + size,
	};
	arm_lpae_iopte *ptep = data->pgd;
	int lvl = data->start_level;

	if (WARN_ON(!size))
		return -EINVAL;
	if (WARN_ON((iova + size - 1) & ~(BIT(cfg->ias) - 1)))
		return -EINVAL;
	if (data->iop.fmt != ARM_64_LPAE_S1)
		return -EINVAL;

	return __arm_lpae_iopte_walk(data, &walk_data, ptep, lvl);
}

static void arm_lpae_restrict_pgsizes(struct io_pgtable_cfg *cfg)
{
	unsigned long granule, page_sizes;
	unsigned int max_addr_bits = 48;

	/*
	 * We need to restrict the supported page sizes to match the
	 * translation regime for a particular granule. Aim to match
	 * the CPU page size if possible, otherwise prefer smaller sizes.
	 * While we're at it, restrict the block sizes to match the
	 * chosen granule.
	 */
	if (cfg->pgsize_bitmap & PAGE_SIZE)
		granule = PAGE_SIZE;
	else if (cfg->pgsize_bitmap & ~PAGE_MASK)
		granule = 1UL << __fls(cfg->pgsize_bitmap & ~PAGE_MASK);
	else if (cfg->pgsize_bitmap & PAGE_MASK)
		granule = 1UL << __ffs(cfg->pgsize_bitmap & PAGE_MASK);
	else
		granule = 0;

	switch (granule) {
	case SZ_4K:
		page_sizes = (SZ_4K | SZ_2M | SZ_1G);
		break;
	case SZ_16K:
		page_sizes = (SZ_16K | SZ_32M);
		break;
	case SZ_64K:
		max_addr_bits = 52;
		page_sizes = (SZ_64K | SZ_512M);
		if (cfg->oas > 48)
			page_sizes |= 1ULL << 42; /* 4TB */
		break;
	default:
		page_sizes = 0;
	}

	cfg->pgsize_bitmap &= page_sizes;
	cfg->ias = min(cfg->ias, max_addr_bits);
	cfg->oas = min(cfg->oas, max_addr_bits);
}

static struct arm_lpae_io_pgtable *
arm_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg)
{
	struct arm_lpae_io_pgtable *data;
	int levels, va_bits, pg_shift;

	arm_lpae_restrict_pgsizes(cfg);

	if (!(cfg->pgsize_bitmap & (SZ_4K | SZ_16K | SZ_64K)))
		return NULL;

	if (cfg->ias > ARM_LPAE_MAX_ADDR_BITS)
		return NULL;

	if (cfg->oas > ARM_LPAE_MAX_ADDR_BITS)
		return NULL;

	data = kmalloc(sizeof(*data), GFP_KERNEL);
	if (!data)
		return NULL;

	pg_shift = __ffs(cfg->pgsize_bitmap);
	data->bits_per_level = pg_shift - ilog2(sizeof(arm_lpae_iopte));

	va_bits = cfg->ias - pg_shift;
	levels = DIV_ROUND_UP(va_bits, data->bits_per_level);
	data->start_level = ARM_LPAE_MAX_LEVELS - levels;

	/* Calculate the actual size of our pgd (without concatenation) */
	data->pgd_bits = va_bits - (data->bits_per_level * (levels - 1));

	data->iop.ops = (struct io_pgtable_ops) {
		.map_pages	= arm_lpae_map_pages,
		.unmap_pages	= arm_lpae_unmap_pages,
		.iova_to_phys	= arm_lpae_iova_to_phys,
		.read_and_clear_dirty = arm_lpae_read_and_clear_dirty,
		.pgtable_walk	= arm_lpae_pgtable_walk,
	};

	return data;
}

static struct io_pgtable *
arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *cfg, void *cookie)
{
	u64 reg;
	struct arm_lpae_io_pgtable *data;
	typeof(&cfg->arm_lpae_s1_cfg.tcr) tcr = &cfg->arm_lpae_s1_cfg.tcr;
	bool tg1;

	if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_NS |
			    IO_PGTABLE_QUIRK_ARM_TTBR1 |
			    IO_PGTABLE_QUIRK_ARM_OUTER_WBWA |
			    IO_PGTABLE_QUIRK_ARM_HD |
			    IO_PGTABLE_QUIRK_NO_WARN))
		return NULL;

	data = arm_lpae_alloc_pgtable(cfg);
	if (!data)
		return NULL;

	/* TCR */
	if (cfg->coherent_walk) {
		tcr->sh = ARM_LPAE_TCR_SH_IS;
		tcr->irgn = ARM_LPAE_TCR_RGN_WBWA;
		tcr->orgn = ARM_LPAE_TCR_RGN_WBWA;
		if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_OUTER_WBWA)
			goto out_free_data;
	} else {
		tcr->sh = ARM_LPAE_TCR_SH_OS;
		tcr->irgn = ARM_LPAE_TCR_RGN_NC;
		if (!(cfg->quirks & IO_PGTABLE_QUIRK_ARM_OUTER_WBWA))
			tcr->orgn = ARM_LPAE_TCR_RGN_NC;
		else
			tcr->orgn = ARM_LPAE_TCR_RGN_WBWA;
	}

	tg1 = cfg->quirks & IO_PGTABLE_QUIRK_ARM_TTBR1;
	switch (ARM_LPAE_GRANULE(data)) {
	case SZ_4K:
		tcr->tg = tg1 ? ARM_LPAE_TCR_TG1_4K : ARM_LPAE_TCR_TG0_4K;
		break;
	case SZ_16K:
		tcr->tg = tg1 ? ARM_LPAE_TCR_TG1_16K : ARM_LPAE_TCR_TG0_16K;
		break;
	case SZ_64K:
		tcr->tg = tg1 ? ARM_LPAE_TCR_TG1_64K : ARM_LPAE_TCR_TG0_64K;
		break;
	}

	switch (cfg->oas) {
	case 32:
		tcr->ips = ARM_LPAE_TCR_PS_32_BIT;
		break;
	case 36:
		tcr->ips = ARM_LPAE_TCR_PS_36_BIT;
		break;
	case 40:
		tcr->ips = ARM_LPAE_TCR_PS_40_BIT;
		break;
	case 42:
		tcr->ips = ARM_LPAE_TCR_PS_42_BIT;
		break;
	case 44:
		tcr->ips = ARM_LPAE_TCR_PS_44_BIT;
		break;
	case 48:
		tcr->ips = ARM_LPAE_TCR_PS_48_BIT;
		break;
	case 52:
		tcr->ips = ARM_LPAE_TCR_PS_52_BIT;
		break;
	default:
		goto out_free_data;
	}

	tcr->tsz = 64ULL - cfg->ias;

	/* MAIRs */
	reg = (ARM_LPAE_MAIR_ATTR_NC
	       << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_NC)) |
	      (ARM_LPAE_MAIR_ATTR_WBRWA
	       << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_CACHE)) |
	      (ARM_LPAE_MAIR_ATTR_DEVICE
	       << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_DEV)) |
	      (ARM_LPAE_MAIR_ATTR_INC_OWBRWA
	       << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_INC_OCACHE));

	cfg->arm_lpae_s1_cfg.mair = reg;

	/* Looking good; allocate a pgd */
	data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data),
					   GFP_KERNEL, cfg, cookie);
	if (!data->pgd)
		goto out_free_data;

	/* Ensure the empty pgd is visible before any actual TTBR write */
	wmb();

	/* TTBR */
	cfg->arm_lpae_s1_cfg.ttbr = virt_to_phys(data->pgd);
	return &data->iop;

out_free_data:
	kfree(data);
	return NULL;
}

static struct io_pgtable *
arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie)
{
	u64 sl;
	struct arm_lpae_io_pgtable *data;
	typeof(&cfg->arm_lpae_s2_cfg.vtcr) vtcr = &cfg->arm_lpae_s2_cfg.vtcr;

	if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_S2FWB |
			    IO_PGTABLE_QUIRK_NO_WARN))
		return NULL;

	data = arm_lpae_alloc_pgtable(cfg);
	if (!data)
		return NULL;

	if (arm_lpae_concat_mandatory(cfg, data)) {
		if (WARN_ON((ARM_LPAE_PGD_SIZE(data) / sizeof(arm_lpae_iopte)) >
			    ARM_LPAE_S2_MAX_CONCAT_PAGES))
			return NULL;
		data->pgd_bits += data->bits_per_level;
		data->start_level++;
	}

	/* VTCR */
	if (cfg->coherent_walk) {
		vtcr->sh = ARM_LPAE_TCR_SH_IS;
		vtcr->irgn = ARM_LPAE_TCR_RGN_WBWA;
		vtcr->orgn = ARM_LPAE_TCR_RGN_WBWA;
	} else {
		vtcr->sh = ARM_LPAE_TCR_SH_OS;
		vtcr->irgn = ARM_LPAE_TCR_RGN_NC;
		vtcr->orgn = ARM_LPAE_TCR_RGN_NC;
	}

	sl = data->start_level;

	switch (ARM_LPAE_GRANULE(data)) {
	case SZ_4K:
		vtcr->tg = ARM_LPAE_TCR_TG0_4K;
		sl++; /* SL0 format is different for 4K granule size */
		break;
	case SZ_16K:
		vtcr->tg = ARM_LPAE_TCR_TG0_16K;
		break;
	case SZ_64K:
		vtcr->tg = ARM_LPAE_TCR_TG0_64K;
		break;
	}

	switch (cfg->oas) {
	case 32:
		vtcr->ps = ARM_LPAE_TCR_PS_32_BIT;
		break;
	case 36:
		vtcr->ps = ARM_LPAE_TCR_PS_36_BIT;
		break;
	case 40:
		vtcr->ps = ARM_LPAE_TCR_PS_40_BIT;
		break;
	case 42:
		vtcr->ps = ARM_LPAE_TCR_PS_42_BIT;
		break;
	case 44:
		vtcr->ps = ARM_LPAE_TCR_PS_44_BIT;
		break;
	case 48:
		vtcr->ps = ARM_LPAE_TCR_PS_48_BIT;
		break;
	case 52:
		vtcr->ps = ARM_LPAE_TCR_PS_52_BIT;
		break;
	default:
		goto out_free_data;
	}

	vtcr->tsz = 64ULL - cfg->ias;
	vtcr->sl = ~sl & ARM_LPAE_VTCR_SL0_MASK;

	/* Allocate pgd pages */
	data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data),
					   GFP_KERNEL, cfg, cookie);
	if (!data->pgd)
		goto out_free_data;

	/* Ensure the empty pgd is visible before any actual TTBR write */
	wmb();

	/* VTTBR */
	cfg->arm_lpae_s2_cfg.vttbr = virt_to_phys(data->pgd);
	return &data->iop;

out_free_data:
	kfree(data);
	return NULL;
}

static struct io_pgtable *
arm_32_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *cfg, void *cookie)
{
	if (cfg->ias > 32 || cfg->oas > 40)
		return NULL;

	cfg->pgsize_bitmap &= (SZ_4K | SZ_2M | SZ_1G);
	return arm_64_lpae_alloc_pgtable_s1(cfg, cookie);
}

static struct io_pgtable *
arm_32_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie)
{
	if (cfg->ias > 40 || cfg->oas > 40)
		return NULL;

	cfg->pgsize_bitmap &= (SZ_4K | SZ_2M | SZ_1G);
	return arm_64_lpae_alloc_pgtable_s2(cfg, cookie);
}

static struct io_pgtable *
arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg, void *cookie)
{
	struct arm_lpae_io_pgtable *data;

	/* No quirks for Mali (hopefully) */
	if (cfg->quirks)
		return NULL;

	if (cfg->ias > 48 || cfg->oas > 40)
		return NULL;

	cfg->pgsize_bitmap &= (SZ_4K | SZ_2M | SZ_1G);

	data = arm_lpae_alloc_pgtable(cfg);
	if (!data)
		return NULL;

	/* Mali seems to need a full 4-level table regardless of IAS */
	if (data->start_level > 0) {
		data->start_level = 0;
		data->pgd_bits = 0;
	}
	/*
	 * MEMATTR: Mali has no actual notion of a non-cacheable type, so the
	 * best we can do is mimic the out-of-tree driver and hope that the
	 * "implementation-defined caching policy" is good enough. Similarly,
	 * we'll use it for the sake of a valid attribute for our 'device'
	 * index, although callers should never request that in practice.
	 */
	cfg->arm_mali_lpae_cfg.memattr =
		(ARM_MALI_LPAE_MEMATTR_IMP_DEF
		 << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_NC)) |
		(ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC
		 << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_CACHE)) |
		(ARM_MALI_LPAE_MEMATTR_IMP_DEF
		 << ARM_LPAE_MAIR_ATTR_SHIFT(ARM_LPAE_MAIR_ATTR_IDX_DEV));

	data->pgd = __arm_lpae_alloc_pages(ARM_LPAE_PGD_SIZE(data), GFP_KERNEL,
					   cfg, cookie);
	if (!data->pgd)
		goto out_free_data;

	/* Ensure the empty pgd is visible before TRANSTAB can be written */
	wmb();

	cfg->arm_mali_lpae_cfg.transtab = virt_to_phys(data->pgd) |
					  ARM_MALI_LPAE_TTBR_READ_INNER |
					  ARM_MALI_LPAE_TTBR_ADRMODE_TABLE;
	if (cfg->coherent_walk)
		cfg->arm_mali_lpae_cfg.transtab |= ARM_MALI_LPAE_TTBR_SHARE_OUTER;

	return &data->iop;

out_free_data:
	kfree(data);
	return NULL;
}

struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s1_init_fns = {
	.caps	= IO_PGTABLE_CAP_CUSTOM_ALLOCATOR,
	.alloc	= arm_64_lpae_alloc_pgtable_s1,
	.free	= arm_lpae_free_pgtable,
};

struct io_pgtable_init_fns io_pgtable_arm_64_lpae_s2_init_fns = {
	.caps	= IO_PGTABLE_CAP_CUSTOM_ALLOCATOR,
	.alloc	= arm_64_lpae_alloc_pgtable_s2,
	.free	= arm_lpae_free_pgtable,
};

struct io_pgtable_init_fns io_pgtable_arm_32_lpae_s1_init_fns = {
	.caps	= IO_PGTABLE_CAP_CUSTOM_ALLOCATOR,
	.alloc	= arm_32_lpae_alloc_pgtable_s1,
	.free	= arm_lpae_free_pgtable,
};

struct io_pgtable_init_fns io_pgtable_arm_32_lpae_s2_init_fns = {
	.caps	= IO_PGTABLE_CAP_CUSTOM_ALLOCATOR,
	.alloc	= arm_32_lpae_alloc_pgtable_s2,
	.free	= arm_lpae_free_pgtable,
};

struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_init_fns = {
	.caps	= IO_PGTABLE_CAP_CUSTOM_ALLOCATOR,
	.alloc	= arm_mali_lpae_alloc_pgtable,
	.free	= arm_lpae_free_pgtable,
};

#ifdef CONFIG_IOMMU_IO_PGTABLE_LPAE_SELFTEST

static struct io_pgtable_cfg *cfg_cookie __initdata;

static void __init dummy_tlb_flush_all(void *cookie)
{
	WARN_ON(cookie != cfg_cookie);
}

static void __init dummy_tlb_flush(unsigned long iova, size_t size,
				   size_t granule, void *cookie)
{
	WARN_ON(cookie != cfg_cookie);
	WARN_ON(!(size & cfg_cookie->pgsize_bitmap));
}

static void __init dummy_tlb_add_page(struct iommu_iotlb_gather *gather,
				      unsigned long iova, size_t granule,
				      void *cookie)
{
	dummy_tlb_flush(iova, granule, granule, cookie);
}

static const struct iommu_flush_ops dummy_tlb_ops __initconst = {
	.tlb_flush_all	= dummy_tlb_flush_all,
	.tlb_flush_walk	= dummy_tlb_flush,
	.tlb_add_page	= dummy_tlb_add_page,
};

static void __init arm_lpae_dump_ops(struct io_pgtable_ops *ops)
{
	struct arm_lpae_io_pgtable *data = io_pgtable_ops_to_data(ops);
	struct io_pgtable_cfg *cfg = &data->iop.cfg;

	pr_err("cfg: pgsize_bitmap 0x%lx, ias %u-bit\n",
		cfg->pgsize_bitmap, cfg->ias);
	pr_err("data: %d levels, 0x%zx pgd_size, %u pg_shift, %u bits_per_level, pgd @ %p\n",
		ARM_LPAE_MAX_LEVELS - data->start_level, ARM_LPAE_PGD_SIZE(data),
		ilog2(ARM_LPAE_GRANULE(data)), data->bits_per_level, data->pgd);
}

#define __FAIL(ops, i)	({						\
		WARN(1, "selftest: test failed for fmt idx %d\n", (i));	\
		arm_lpae_dump_ops(ops);					\
		-EFAULT;						\
})

static int __init arm_lpae_run_tests(struct io_pgtable_cfg *cfg)
{
	static const enum io_pgtable_fmt fmts[] __initconst = {
		ARM_64_LPAE_S1,
		ARM_64_LPAE_S2,
	};

	int i, j;
	unsigned long iova;
	size_t size, mapped;
	struct io_pgtable_ops *ops;

	for (i = 0; i < ARRAY_SIZE(fmts); ++i) {
		cfg_cookie = cfg;
		ops = alloc_io_pgtable_ops(fmts[i], cfg, cfg);
		if (!ops) {
			pr_err("selftest: failed to allocate io pgtable ops\n");
			return -ENOMEM;
		}

		/*
		 * Initial sanity checks.
		 * Empty page tables shouldn't provide any translations.
		 */
		if (ops->iova_to_phys(ops, 42))
			return __FAIL(ops, i);

		if (ops->iova_to_phys(ops, SZ_1G + 42))
			return __FAIL(ops, i);

		if (ops->iova_to_phys(ops, SZ_2G + 42))
			return __FAIL(ops, i);

		/*
		 * Distinct mappings of different granule sizes.
		 */
		iova = 0;
		for_each_set_bit(j, &cfg->pgsize_bitmap, BITS_PER_LONG) {
			size = 1UL << j;

			if (ops->map_pages(ops, iova, iova, size, 1,
					   IOMMU_READ | IOMMU_WRITE |
					   IOMMU_NOEXEC | IOMMU_CACHE,
					   GFP_KERNEL, &mapped))
				return __FAIL(ops, i);

			/* Overlapping mappings */
			if (!ops->map_pages(ops, iova, iova + size, size, 1,
					    IOMMU_READ | IOMMU_NOEXEC,
					    GFP_KERNEL, &mapped))
				return __FAIL(ops, i);

			if (ops->iova_to_phys(ops, iova + 42) != (iova + 42))
				return __FAIL(ops, i);

			iova += SZ_1G;
		}

		/* Full unmap */
		iova = 0;
		for_each_set_bit(j, &cfg->pgsize_bitmap, BITS_PER_LONG) {
			size = 1UL << j;

			if (ops->unmap_pages(ops, iova, size, 1, NULL) != size)
				return __FAIL(ops, i);

			if (ops->iova_to_phys(ops, iova + 42))
				return __FAIL(ops, i);

			/* Remap full block */
			if (ops->map_pages(ops, iova, iova, size, 1,
					   IOMMU_WRITE, GFP_KERNEL, &mapped))
				return __FAIL(ops, i);

			if (ops->iova_to_phys(ops, iova + 42) != (iova + 42))
				return __FAIL(ops, i);

			iova += SZ_1G;
		}

		/*
		 * Map/unmap the last largest supported page of the IAS, this can
		 * trigger corner cases in the concatednated page tables.
		 */
		mapped = 0;
		size = 1UL << __fls(cfg->pgsize_bitmap);
		iova = (1UL << cfg->ias) - size;
		if (ops->map_pages(ops, iova, iova, size, 1,
				   IOMMU_READ | IOMMU_WRITE |
				   IOMMU_NOEXEC | IOMMU_CACHE,
				   GFP_KERNEL, &mapped))
			return __FAIL(ops, i);
		if (mapped != size)
			return __FAIL(ops, i);
		if (ops->unmap_pages(ops, iova, size, 1, NULL) != size)
			return __FAIL(ops, i);

		free_io_pgtable_ops(ops);
	}

	return 0;
}

static int __init arm_lpae_do_selftests(void)
{
	static const unsigned long pgsize[] __initconst = {
		SZ_4K | SZ_2M | SZ_1G,
		SZ_16K | SZ_32M,
		SZ_64K | SZ_512M,
	};

	static const unsigned int address_size[] __initconst = {
		32, 36, 40, 42, 44, 48,
	};

	int i, j, k, pass = 0, fail = 0;
	struct faux_device *dev;
	struct io_pgtable_cfg cfg = {
		.tlb = &dummy_tlb_ops,
		.coherent_walk = true,
		.quirks = IO_PGTABLE_QUIRK_NO_WARN,
	};

	dev = faux_device_create("io-pgtable-test", NULL, 0);
	if (!dev)
		return -ENOMEM;

	cfg.iommu_dev = &dev->dev;

	for (i = 0; i < ARRAY_SIZE(pgsize); ++i) {
		for (j = 0; j < ARRAY_SIZE(address_size); ++j) {
			/* Don't use ias > oas as it is not valid for stage-2. */
			for (k = 0; k <= j; ++k) {
				cfg.pgsize_bitmap = pgsize[i];
				cfg.ias = address_size[k];
				cfg.oas = address_size[j];
				pr_info("selftest: pgsize_bitmap 0x%08lx, IAS %u OAS %u\n",
					pgsize[i], cfg.ias, cfg.oas);
				if (arm_lpae_run_tests(&cfg))
					fail++;
				else
					pass++;
			}
		}
	}

	pr_info("selftest: completed with %d PASS %d FAIL\n", pass, fail);
	faux_device_destroy(dev);

	return fail ? -EFAULT : 0;
}
subsys_initcall(arm_lpae_do_selftests);
#endif
]

gemini-3.1-pro-preview

							input: 192976
							output: 66
							thoughts: 5245
						

**I've identified a critical vulnerability within the Intel IOMMU driver, specifically arising from a failure to properly handle partial mappings during DMA operations.**

Here's my thought process: The function `arm_lpae_map_pages` calls `__arm_lpae_map`, which appears to handle the mapping of physical pages to an IOMMU's address space. It looks like the process is about mapping `num_entries` at the current level, and the `*mapped` value is updated during the mapping. The key functions are `arm_lpae_map_pages` and `__arm_lpae_map`, they seem to handle the page mapping process, but the caller of these functions does not loop, which is interesting. It calls the ops, and updates the `*mapped` value. I observed that the `*mapped` parameter is crucial. If a portion of a mapping succeeds but an error occurs later, the caller, `iommu_map_nosync`, incorrectly assumes that *no* pages were mapped if `ops->map_pages` returns an error, leading to an incorrect `size` calculation and an incorrect unmapping. The function `intel_iommu_map_pages` does not update `*mapped` on error, which is the crux of the issue.

I then followed the execution path into `intel_iommu_map`, which calls `__domain_mapping`. `__domain_mapping` can fail if it runs out of memory, or if the PTE is already set. If a failure occurs mid-mapping (e.g., due to memory exhaustion), only some pages are actually mapped, but the caller isn't informed of this partial success. The IOVA and related physical pages are freed, but the IOMMU page table is still set. The program then breaks, and `orig_size - size` is passed to the unmap function, but because `size` is unchanged, 0 bytes are unmapped, causing an inconsistency between the system's memory management and the IOMMU's view.

This "leak" in the IOMMU page table is exploitable. If the IOVA is re-used, `__domain_mapping` will detect a PTE is already set and `WARN_ON(1)` will be triggered, but critically, it *doesn't* overwrite that PTE! The mapping then "succeeds" silently, but it still points to the old, now potentially re-allocated, physical pages. This allows a malicious actor using DMA to read and write arbitrary physical memory, because the device's DMA will be directed to the old data, potentially sensitive, data.

This is a classic use-after-free issue, because the page table entry is left pointing to a physical page that has been freed and re-allocated.

The exploit is possible by:

1.  Creating memory pressure (e.g. allocate a lot of memory).
2.  Mapping a large DMA buffer (e.g. using `/dev/dma_heap/system` or GPU drivers). The bigger the allocation, the higher the chance of hitting the condition.

The attacker can trigger the bug using `GFP_ATOMIC` and low memory conditions.

The attack involves an unprivileged user, and the impact is full system compromise.

map[DenialOfService:true Exploitable:true FilesystemTrigger:false NetworkTrigger:false PeripheralTrigger:false RemoteTrigger:false Unprivileged:true UserNamespace:true VMGuestTrigger:false VMHostTrigger:false]

map[DenialOfService:true Exploitable:true FilesystemTrigger:false NetworkTrigger:false PeripheralTrigger:false RemoteTrigger:false Unprivileged:true UserNamespace:true VMGuestTrigger:false VMHostTrigger:false]

gemini-3.1-pro-preview

							input: 199052
							output: 898
							thoughts: 587
						

**Vulnerability Deep Dive: A Classic Use-After-Free in the Intel IOMMU**

Okay, the analysis is done, the results are in. Time to concisely articulate this vulnerability and its implications. I've pinpointed the bug in `intel_iommu_map_pages()` within `drivers/iommu/intel/iommu.c`. The core of the issue lies in how the code handles failures within `intel_iommu_map()`, specifically when `__domain_mapping()` encounters a problem during page mapping, often due to an allocation failure – let's say a `GFP_ATOMIC` failure within `pfn_to_dma_pte()`.

The problem is that if `intel_iommu_map()` fails and returns an error code (like `-ENOMEM`), `intel_iommu_map_pages()` doesn't correctly update the `mapped` output parameter. The current logic is:

```c
	ret = intel_iommu_map(domain, iova, paddr, size, prot, gfp);
	if (!ret && mapped)
		*mapped = size;
```

It doesn't update the `mapped` parameter if `ret` is not 0 (meaning there was an error). Consequently, the caller `iommu_map_nosync()` incorrectly assumes that no bytes were mapped during this failed call. This leads to an incorrect unmapping of *zero* bytes in `iommu_map_nosync()`. This is the crux of the leak: the PTEs that were successfully set by `__domain_mapping()` before the failure remain valid, even though the intended mapping was incomplete.

Here’s the breakdown of the impact. When the DMA buffer is later freed, the physical pages are returned to the system, and the IOVA is available for reuse, BUT the IOMMU page table still contains those PTEs, mapping the IOVA to the old, freed physical pages. Then the IOVA is freed, and it can be reallocated. This creates a dangerous condition. If the same IOVA is subsequently re-used for a new DMA mapping, `__domain_mapping()` encounters these "stale" PTEs. It raises a warning (the familiar `ERROR: DMA PTE for vPFN ... already set`) and triggers a `WARN_ON(1)`. Critically, it *doesn't* overwrite the existing PTE because a `try_cmpxchg64_local()` fails. The function continues, and it returns success!

So, we now have a device configured to use a re-used IOVA, but the IOMMU translates this IOVA to the old, now freed physical pages. This sets the stage for an attacker to achieve arbitrary read/write access to physical memory by instructing the device (a GPU, for example) to perform DMA operations to this IOVA. This is textbook Use-After-Free (UAF) of physical pages. An unprivileged user can trigger this. Allocating a large DMA buffer (through `/dev/dma_heap/system` or a GPU driver), creating memory pressure to force that `GFP_ATOMIC` allocation failure during the IOMMU mapping, and then exploiting the stale IOMMU mappings is the game plan for privilege escalation.

map[Explanation:The bug is a critical security vulnerability in the Intel IOMMU driver that
leads to a Use-After-Free (UAF) of physical pages, allowing an unprivileged user
to achieve arbitrary physical memory read/write and full system compromise.

### Root Cause Analysis
The vulnerability lies in the error handling path of `intel_iommu_map_pages()`
in `drivers/iommu/intel/iommu.c`.

When a device driver maps a scatterlist for DMA (e.g., during a
`DRM_IOCTL_PRIME_FD_TO_HANDLE` ioctl), the IOMMU core calls
`iommu_map_nosync()`, which in turn calls the IOMMU driver's `map_pages`
callback (`intel_iommu_map_pages()`).

`intel_iommu_map_pages()` calls `intel_iommu_map()` (which wraps
`__domain_mapping()`) to map the pages. If `__domain_mapping()` fails in the
middle of mapping a large range (e.g., due to a `GFP_ATOMIC` memory allocation
failure in `pfn_to_dma_pte()` under memory pressure), it returns an error like
`-ENOMEM`.

However, `intel_iommu_map_pages()` has a flaw where it **does not update the
`mapped` output parameter** if an error is returned:
```c
	ret = intel_iommu_map(domain, iova, paddr, size, prot, gfp);
	if (!ret && mapped)
		*mapped = size; // Only updated on success!

	return ret;
```
Because `mapped` remains `0`, the caller `iommu_map_nosync()` assumes that 0
bytes were mapped during this specific call. Consequently, its error unrolling
path unmaps 0 bytes for this segment, **leaking the PTEs that were successfully
set by `__domain_mapping()` before the failure occurred**.

### Exploitation Mechanism
1. **Triggering the Leak**: An unprivileged user allocates a large DMA buffer
(e.g., via `/dev/dma_heap/system`) and maps it to a device (e.g., a GPU). By
inducing memory pressure, the user can cause the `GFP_ATOMIC` allocation in
`pfn_to_dma_pte()` to fail midway through the mapping. The mapping syscall
fails, but the first few pages remain mapped in the IOMMU page table.
2. **Freeing the Physical Pages**: The user closes the DMA buffer, returning the
physical pages to the system allocator (where they might be reused for sensitive
kernel data or page cache). However, the IOMMU still has valid PTEs pointing to
them. The IOVA is also freed back to the IOVA allocator.
3. **Reusing the IOVA**: The user allocates a new DMA buffer and maps it. The
IOVA allocator is highly likely to reuse the recently freed IOVA.
4. **Bypassing the Warning**: When `__domain_mapping()` is called for the new
buffer, it encounters the stale PTEs. It prints the warning seen in the bug
report (`ERROR: DMA PTE for vPFN ... already set`) and triggers a `WARN_ON(1)`.
Crucially, because `try_cmpxchg64_local()` fails, **it does not overwrite the
existing PTE**. The function simply continues and returns success.
5. **Arbitrary Memory Access**: The device is now configured to use the reused
IOVA, but the IOMMU translates this IOVA to the *old, freed* physical pages. The
user can instruct the device to perform DMA reads/writes to this IOVA, achieving
arbitrary physical memory read/write.

### Impact
- **Exploitable**: Yes. It provides a reliable primitive for arbitrary physical
memory read/write.
- **Accessible From Unprivileged Processes**: Yes. Unprivileged users can
allocate DMA buffers (e.g., via DMA heaps or GPU drivers) and trigger DMA
mappings.
- **Denial of Service**: Yes. The bug can also be used to corrupt memory and
crash the system.]