Technical Analysis of Circular Locking Dependency in NBD
1. Functions, Structs, and Files Involved
- Source Files:
- Key Structs & Fields:
struct nbd_device / struct nbd_sock (nsock->tx_lock, nsock->sock)struct socket / struct sock (sk->sk_lock.dep_map, lock class sk_lock-AF_INET-NBD)struct mm_struct (mm->mmap_lock)struct request_queue (q->q_usage_counter, q->elevator_lock)
- Key Functions:
2. Root Cause & Lifecycle Rule Violation
In drivers/block/nbd.c, when an NBD connection is configured (via ioctl(..., NBD_SET_SOCK) or netlink nbd_genl_connect()), nbd_add_socket() looks up a socket descriptor passed by userspace via sockfd_lookup():
sock = nbd_get_socket(nbd, arg, &err);
if (!sock)
return err;
nbd_reclassify_socket(sock);
nbd_reclassify_socket() reinitializes the lockdep map of sock->sk->sk_lock to a custom lock class: sk_lock-AF_INET-NBD (or IPv6 / UNIX equivalents).
In other network block drivers such as nvme-tcp, sockets are created directly in kernel space via sock_create_kern(), so userspace never has access to the underlying socket file descriptor. In NBD, however, the socket is an existing userspace socket passed by file descriptor. The calling process retains that file descriptor in its fdtable, and userspace remains capable of issuing socket operations (sendto(), sendmsg(), write(), etc.) on the descriptorβboth while the NBD device is active and even if nbd_add_socket() subsequently fails and performs sockfd_put(sock).
Because the socket's lockdep class has been modified to sk_lock-AF_INET-NBD, any standard socket I/O performed on that descriptor executes under the sk_lock-AF_INET-NBD lockdep class.
3. Inverted Locking Chain & Deadlock Mechanism
This introduces two conflicting lock acquisition chains involving &mm->mmap_lock and sk_lock-AF_INET-NBD:
-
Chain 1: Userspace sendto() on the NBD socket descriptor (sk_lock-AF_INET-NBD $\to$ &mm->mmap_lock)
- Userspace calls
sendto(fd, user_buf, len).
tcp_sendmsg() acquires lock_sock(sk), which now holds the sk_lock-AF_INET-NBD lock class.tcp_sendmsg_locked() calls skb_copy_to_page_nocache(), which invokes copy_from_iter_full() $\to$ _copy_from_iter() $\to$ __might_fault().- Copying from userspace buffers can trigger a page fault, which requires acquiring
down_read(&mm->mmap_lock).
- Lock dependency established:
sk_lock-AF_INET-NBD $\to$ &mm->mmap_lock.
-
Chain 2: Memory Reclaim & Block I/O Submission (&mm->mmap_lock $\to$ sk_lock-AF_INET-NBD)
- A thread holds
&mm->mmap_lock (e.g., during mmap(), brk(), or handling a page fault).
- The thread triggers a memory allocation (
GFP_KERNEL) under memory pressure, entering direct reclaim (fs_reclaim).
- Memory reclaim flushes dirty pages or swap space backed by an NBD block device (
/dev/nbd*).
- Submitting block I/O enters
nbd_queue_rq() $\to$ nbd_handle_cmd(), which acquires &nsock->tx_lock.
- Under
&nsock->tx_lock (or during error recovery in recv_work() $\to$ nbd_mark_nsock_dead() $\to$ kernel_sock_shutdown() $\to$ inet_shutdown()), the driver calls lock_sock(nsock->sock->sk).
- Lock dependency established:
&mm->mmap_lock $\to$ fs_reclaim $\to$ &nsock->tx_lock $\to$ sk_lock-AF_INET-NBD.
Combining both paths forms an ABBA circular locking dependency:
$$\text{&mm->mmap_lock} \longrightarrow \text{fs_reclaim} \longrightarrow \text{&nsock->tx_lock} \longrightarrow \text{sk_lock-AF_INET-NBD} \longrightarrow \text{&mm->mmap_lock}$$
Lockdep detects this dependency and reports the circular lock warning when __might_fault() is invoked while holding sk_lock-AF_INET-NBD.