Technical Root Cause Analysis

1. Overview & Location

• Subsystem: RDMA / Communication Manager (drivers/infiniband/core/cma.c)
• Functions involved: resolve_prepare_src(), cma_bind_addr(), rdma_bind_addr_dst(), cma_process_remove(), cma_send_device_removal_put()
• Structs & Fields: struct rdma_id_private::state, struct rdma_id_private::device_item, struct cma_device::id_list
• File: drivers/infiniband/core/cma.c

2. Mechanism & Invariant Violation

When an RDMA Communication Manager ID is created via rdma_create_id() (as in rds_ib_conn_path_connect()), its state is initialized to RDMA_CM_IDLE (0).

When resolving an address via rdma_resolve_addr(), it calls resolve_prepare_src():

static int resolve_prepare_src(struct rdma_id_private *id_priv,
			       struct sockaddr *src_addr,
			       const struct sockaddr *dst_addr)
{
	int ret;

	if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_ADDR_QUERY)) {
		/* For a well behaved ULP state will be RDMA_CM_IDLE */
		ret = cma_bind_addr(&id_priv->id, src_addr, dst_addr);
		if (ret)
			return ret;
		if (WARN_ON(!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND,
					   RDMA_CM_ADDR_QUERY)))
			return -EINVAL;
	}
...

The bug is triggered as follows:

1. Since the ID is initially RDMA_CM_IDLE, the first cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_ADDR_QUERY) returns false (0).
2. The function proceeds into the if branch and invokes cma_bind_addr(), which calls rdma_bind_addr_dst().
3. In rdma_bind_addr_dst(), id_priv->state is transitioned from RDMA_CM_IDLE to RDMA_CM_ADDR_BOUND (7).
4. Because a non-wildcard source IP is specified, rdma_bind_addr_dst() calls cma_acquire_dev_by_src_ip(id_priv), which in turn calls cma_attach_to_dev() and adds id_priv->device_item to cma_dev->id_list.
5. Once attached to the device's list, id_priv becomes visible to concurrent device removal handling. If the underlying RDMA device is concurrently removed or unbound (e.g. cma_process_remove()), it iterates over cma_dev->id_list and calls cma_send_device_removal_put(id_priv).
6. Inside cma_send_device_removal_put(), id_priv->lock is acquired and id_priv->state is transitioned to RDMA_CM_DEVICE_REMOVAL (9) (which matches the crash register value R14 = 0x9).
7. When control returns to resolve_prepare_src() and it attempts to execute:

   WARN_ON(!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_ADDR_QUERY))
   

   the comparison id_priv->state == RDMA_CM_ADDR_BOUND evaluates to false because id_priv->state is now RDMA_CM_DEVICE_REMOVAL.
8. The WARN_ON fires due to the false assumption that id_priv->state cannot change between cma_bind_addr() returning 0 and the subsequent cma_comp_exch().

3. Runtime Consequences in Production

• Memory Corruption: None. The state mismatch only causes the exchange to fail, without corrupting pointers, refcounts, or memory.
• Information Leak: None. No kernel memory contents or addresses are exposed or copied to userspace.
• Denial of Service: None. In a production build without panic_on_warn, WARN_ON logs a warning and does not panic the system. The function returns -EINVAL, causing rdma_resolve_addr() to return -EINVAL. Its caller (rds_ib_conn_path_connect()) checks this error, tears down the ID via rdma_destroy_id(), and cleanly drops the connection attempt without hanging or leaking memory.