Technical Root Cause Analysis
1. Overview and Invariant Violation
The reported lockdep warning detects a circular locking dependency involving fs_reclaim, block layer queue locks, NBD socket transmit locks (&nsock->tx_lock), CPU hotplug lock (cpu_hotplug_lock), and SLAB allocator locks (slab_mutex).
The root cause resides in drivers/block/nbd.c within nbd_genl_reconnect_sock_cb(). The function invokes sk_set_memalloc(sock->sk) while holding the mutex &nsock->tx_lock.
Under the kernel's memory management locking hierarchy, locks held in the block I/O submission/dispatch path of devices that can be written to during direct memory reclaim are child dependencies of fs_reclaim. Calling sk_set_memalloc() under such a lock violates the locking hierarchy because sk_set_memalloc() internally takes cpus_read_lock() (cpu_hotplug_lock), which is an ancestor lock of fs_reclaim.
2. Key Components and Involved Entities
3. Chain of Dependencies Leading to the Circular Deadlock
-
fs_reclaim $\to$ &nsock->tx_lock (Lockdep links #0 through #5):
- When any GFP flag allowing direct reclaim (such as
GFP_KERNEL or GFP_FS) triggers memory reclaim, fs_reclaim is acquired.
- Reclaiming memory can write out dirty pages or swap blocks using block devices, including NBD.
- The block layer request queue and dispatch logic (
blk_mq_dispatch_rq_list $\to$ nbd_queue_rq $\to$ nbd_handle_cmd) acquires the request lock &cmd->lock and then &nsock->tx_lock.
- This establishes the dependency:
fs_reclaim $\to$ ... $\to$ &nsock->tx_lock.
-
&nsock->tx_lock $\to$ cpu_hotplug_lock (Lockdep link #6):
-
cpu_hotplug_lock $\to$ slab_mutex (Lockdep link #7):
-
slab_mutex $\to$ fs_reclaim (Trigger site at #0):
- When creating a slab cache (e.g.
kmem_cache_create()), __kmem_cache_create_args() takes mutex_lock(&slab_mutex).
- Under
slab_mutex, create_cache() calls kmem_cache_zalloc(kmem_cache, GFP_KERNEL).
- The
GFP_KERNEL allocation invokes might_alloc() and attempts to acquire fs_reclaim.
- Since
fs_reclaim already depends on cpu_hotplug_lock via nsock->tx_lock, attempting to acquire fs_reclaim while holding slab_mutex closes the circular cycle:
$$\text{fs_reclaim} \to \text{nsock->tx_lock} \to \text{cpu_hotplug_lock} \to \text{slab_mutex} \to \text{fs_reclaim}$$
4. Remediation
In nbd_genl_reconnect_sock_cb(), sock is a newly provided socket that has not yet been attached to nsock. Calling sk_set_memalloc(sock->sk) and setting socket timeout attributes does not require holding &nsock->tx_lock. Moving sk_set_memalloc(sock->sk) outside of mutex_lock(&nsock->tx_lock) (similar to how nbd_start_device() configures sockets without taking tx_lock) eliminates the dependency between &nsock->tx_lock and cpu_hotplug_lock.