Root Cause Analysis: Circular Locking Dependency in Cgroup / Cpuset Migration
1. Involved Functions, Structs, and Locks
- Locks:
cpu_hotplug_lock (percpu_rw_semaphore / cpus_read_lock)cgroup_threadgroup_rwsem (percpu_rw_semaphore)cpuset_mutex (mutex)fs_reclaim (pseudo-lock representing memory allocation / direct reclaim)sk_lock-AF_INET6 (sock::sk_lock)
- Functions & Files:
2. Detailed Mechanism and Lock Inversion
Lockdep detected a circular locking dependency involving cgroup_threadgroup_rwsem, cpuset_mutex, and cpu_hotplug_lock. The circular chain is formed by two conflicting lock dependency paths:
Path A: Migration Path (cpu_hotplug_lock $\rightarrow$ cgroup_threadgroup_rwsem $\rightarrow$ cpuset_mutex)
- During task migration into a cgroup (e.g., writing to
cgroup.procs), cgroup_procs_write_start() calls cgroup_attach_lock().
- To prevent CPU hotplug callbacks from racing with task migrations,
cgroup_attach_lock() (kernel/cgroup/cgroup.c:2544) explicitly acquires cpus_read_lock() (cpu_hotplug_lock) and then write-locks cgroup_threadgroup_rwsem (percpu_down_write(&cgroup_threadgroup_rwsem)).
- Next,
cgroup_migrate_execute() invokes subsystem can_attach callbacks. For the cpuset controller, cpuset_can_attach() (kernel/cgroup/cpuset.c:3226) acquires cpuset_mutex.
- This sequence establishes the lock ordering:
$$\text{cpu_hotplug_lock} \rightarrow \text{cgroup_threadgroup_rwsem} \rightarrow \text{cpuset_mutex}$$
Path B: Sched Domain Rebuild & Memory Reclaim Path (cpuset_mutex $\rightarrow$ fs_reclaim $\rightarrow \dots \rightarrow$ cpu_hotplug_lock)
- In
kernel/cgroup/cpuset.c, rebuild_sched_domains_cpuslocked() acquires cpuset_mutex and calls rebuild_sched_domains_locked().
rebuild_sched_domains_locked() invokes generate_sched_domains(), which calls alloc_sched_domains().alloc_sched_domains() (kernel/sched/topology.c:3284) performs memory allocations using GFP_KERNEL (alloc_cpumask_var(&doms[i], GFP_KERNEL)).- Allocating memory with
GFP_KERNEL while holding cpuset_mutex establishes a dependency on direct memory reclaim:
$$\text{cpuset_mutex} \rightarrow \text{fs_reclaim}$$
- Under direct memory reclaim, the kernel can reclaim dirty pages backed by network block devices (such as NBD), which pushes requests through the block layer (
q->q_usage_counter $\rightarrow$ elevator_lock $\rightarrow$ set->srcu $\rightarrow$ cmd->lock $\rightarrow$ nsock->tx_lock).
- In
drivers/block/nbd.c:319, nbd_mark_nsock_dead() calls kernel_sock_shutdown(), which invokes inet_shutdown() and acquires lock_sock(sk) (sk_lock-AF_INET6) while holding nsock->tx_lock.
- In network tunnel configuration (
net/l2tp/l2tp_core.c:1690), l2tp_tunnel_register() is called with lock_sock(sk) held; it calls setup_udp_tunnel_sock(), which calls udp_tunnel_encap_enable() $\rightarrow$ static_key_slow_inc() $\rightarrow$ cpus_read_lock(), acquiring cpu_hotplug_lock. This completes the link:
$$\text{sk_lock-AF_INET6} \rightarrow \text{cpu_hotplug_lock}$$
3. Summary of the Invariant Violation
The kernel requires that locks protecting memory allocation contexts must not nest outside operations that can acquire locks taken during fs_reclaim. Holding cpuset_mutex across GFP_KERNEL memory allocations in rebuild_sched_domains_locked() creates a dependency on fs_reclaim, which transitively depends on cpu_hotplug_lock. However, cgroup_attach_lock() acquires cpu_hotplug_lock as an outer lock before taking cgroup_threadgroup_rwsem and cpuset_mutex. When these two paths interact, lockdep detects the potential circular dependency, warning of an ABBA deadlock between cpuset_mutex and cpu_hotplug_lock.