Bug Analysis & Technical Explanation

Overview and Manifestation

The crash occurs in blk_mq_complete_request_remote() (block/blk-mq.c:1317):

bool blk_mq_complete_request_remote(struct request *rq)
{
	WRITE_ONCE(rq->state, MQ_RQ_COMPLETE);

	/*
	 * For request which hctx has only one ctx mapping,
	 * or a polled request, always complete locally,
	 * it's pointless to redirect the completion.
	 */
	if ((rq->mq_hctx->nr_ctx == 1 &&
	     rq->mq_ctx->cpu == raw_smp_processor_id()) ||
	     rq->cmd_flags & REQ_POLLED)
		return false;
...

In blk_mq_complete_request_remote(), the expression rq->mq_hctx->nr_ctx accesses nr_ctx at offset 0x18e within struct blk_mq_hw_ctx. At the time of the fault, rq (0xffff0000c8ca9000) is a valid pointer to a struct request, but rq->mq_hctx is NULL. Consequently, KASAN intercepts an access to 0x000000000000018e (KASAN shadow address dfff800000000031), triggering a NULL-pointer dereference kernel Oops (ESR = 0x96000005: level 1 translation fault).

Root Cause Analysis

1. Request Lifecycle and Stale Tag Mapping in blk-mq:

   • When a request is queued and started via blk_mq_start_request() (block/blk-mq.c:1375), it sets rq->state = MQ_RQ_IN_FLIGHT and saves the request into the driver tags array: rq->mq_hctx->tags->rqs[rq->tag] = rq;.
   • When a request finishes and is freed via blk_mq_free_request() (block/blk-mq.c:840) $\to$ __blk_mq_free_request() (block/blk-mq.c:828), rq->state transitions to MQ_RQ_IDLE and rq->mq_hctx is explicitly cleared to NULL (rq->mq_hctx = NULL;).
   • However, tags->rqs[tag] is not cleared upon request completion; it retains the pointer to the static request allocation.

2. NVMe Cancellation vs. Generation Counter (genctr):

   • Under normal request completion (nvme_try_complete_req(), drivers/nvme/host/nvme.h:834), the controller increments rq->genctr (rq->genctr++).
   • When an NVMe controller is disabled, nvme_dev_disable() calls nvme_cancel_tagset(&dev->ctrl) $\to$ blk_mq_tagset_busy_iter() with callback nvme_cancel_request() (drivers/nvme/host/core.c:543).
   • nvme_cancel_request() sets nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD and calls blk_mq_complete_request(req) directly. It does not call nvme_try_complete_req() and does not advance rq->genctr. The cancelled requests are completed and freed, leaving rq->mq_hctx = NULL, while rq->genctr remains at the previous generation.

3. Controller Reset Failure and Duplicate nvme_dev_disable():

   • During nvme_reset_work() (drivers/nvme/host/pci.c:3622):

     if (dev->ctrl.ctrl_config & NVME_CC_ENABLE)
         nvme_dev_disable(dev, false);
     nvme_sync_queues(&dev->ctrl);
     
     mutex_lock(&dev->shutdown_lock);
     result = nvme_pci_enable(dev);
     if (result)
         goto out_unlock;
     

   • In the initial nvme_dev_disable(dev, false), IO queues are suspended and nvme_cancel_tagset() terminates and frees all in-flight requests.
   • Next, nvme_pci_enable(dev) re-enables PCI device memory and bus mastering (pci_enable_device_mem(pdev) $\to$ "nvme 0000:00:02.0: enabling device (0201 -> 0203)", pci_set_master(pdev)).
   • If controller initialization fails immediately afterward (e.g., reading NVME_REG_CSTS returns -1, or result = -ENODEV (-19)), nvme_reset_work() branches to out: and prints "nvme nvme0: Disabling device after reset failure: -19".
   • In out:, nvme_dev_disable(dev, true) is called a second time.

4. Reaping CQEs with Idle/Freed Requests:

   • In nvme_dev_disable(dev, true), line 3565 calls nvme_reap_pending_cqes(dev):

     static void nvme_reap_pending_cqes(struct nvme_dev *dev)
     {
         int i;
         for (i = dev->ctrl.queue_count - 1; i > 0; i--) {
             spin_lock(&dev->queues[i].cq_poll_lock);
             nvme_poll_cq(&dev->queues[i], NULL);
             spin_unlock(&dev->queues[i].cq_poll_lock);
         }
     }
     

   • If any completion queue entry is detected pending (nvme_cqe_pending() matching phase), nvme_handle_cqe() is invoked.
   • nvme_handle_cqe() looks up the request using nvme_find_rq(nvme_queue_tagset(nvmeq), command_id).
   • In nvme_find_rq() (drivers/nvme/host/nvme.h:692-712):

     rq = blk_mq_tag_to_rq(tags, tag);
     if (unlikely(!rq))
         return NULL;
     if (unlikely(nvme_genctr_mask(nvme_req(rq)->genctr) != genctr))
         return NULL;
     return rq;
     

   • Because genctr was never incremented during cancellation and nvme_find_rq() does not verify whether rq is currently active (e.g., via blk_mq_request_started(rq) / rq->state == MQ_RQ_IN_FLIGHT), nvme_find_rq() successfully returns the pointer to the freed, idle request.
   • nvme_handle_cqe() proceeds to complete the request via nvme_try_complete_req(), which calls blk_mq_complete_request_remote(rq).
   • blk_mq_complete_request_remote() dereferences rq->mq_hctx to read rq->mq_hctx->nr_ctx, dereferencing NULL at offset 0x18e and crashing the kernel.