Meta’s Kernfs Optimizations Remove Long‑Running Locks in sysfs and cgroupfs

By the end of this article readers will understand the lock‑contention problem that affected kernfs‑based filesystems, the specific changes introduced by Meta engineers, and how those changes alter the behavior of system daemons such as oomd and the telemetry collector that depends on it.

Background: kernfs and Its Role in sysfs and cgroupfs

The Linux kernel exposes many internal objects through virtual filesystems. Two of the most heavily used are sysfs, which provides a view of kernel objects to user space, and cgroupfs, which implements control groups. Both filesystems are built on top of the kernfs subsystem, a generic kernel module that manages the hierarchy of in‑memory nodes and coordinates access through a read‑write semaphore (rwsem) named kernfs_rwsem. Each kernfs root—one for sysfs and one for cgroupfs—holds a single rwsem that protects all create, remove, and rename operations across the entire mount.

Historically, the rwsem implementation permits multiple readers to hold the lock concurrently, but once a writer is queued the semaphore blocks further readers. This design prevents writer starvation but can create a bottleneck when a writer is pending and a large number of readers attempt to traverse the hierarchy. In large‑scale deployments, such as Meta’s fleet of servers, the frequency of cgroup creation, device renames, and other mutating operations can be high enough to cause noticeable stalls.

Observed Bottleneck on Production Servers

Meta engineers observed that certain system daemons were blocked for minutes while waiting on kernfs locks. The most impacted daemons were oomd, the out‑of‑memory decision service, and a telemetry collector that records post‑oomd actions. Both daemons rely on rapid reads of sysfs and cgroupfs entries to make decisions and report state. When a writer—such as a cgroup creation or a device rename—queued the kernfs_rwsem, the entire incoming reader stream entered uninterruptible sleep, effectively halting the daemons.

The root cause was identified in the function kernfs_dop_revalidate(). This function acquires the kernfs_rwsem for read access once per path component during each lookup into a kernfs mount. Because the rwsem does not allow readers to “steal” the lock once a writer is queued, a single pending write operation could serialize all concurrent lookups across the whole system.

Meta’s Patch Set and the Technical Changes

Shakeel Butt authored eighteen patches for Linux 7.4 that address the lock‑contention issue. Four of those patches target the most critical paths and produce measurable performance improvements. The changes can be grouped into two categories: refinement of lock handling and correction of underlying bugs.

First, the patches modify the way kernfs_dop_revalidate() acquires the rwsem. Rather than taking a read lock for each component of a path, the updated code reduces the granularity of lock acquisition, allowing readers to proceed more efficiently when a writer is pending. This adjustment mitigates the “reader parking” effect that previously forced large numbers of readers into uninterruptible sleep.

Second, the patches address several kernfs bugs that manifested under heavy load. While the source does not enumerate each bug, the fixes collectively improve the stability of kernfs operations during rapid create, remove, and rename sequences. By ensuring that internal data structures remain consistent, the patches prevent rare race conditions that could otherwise exacerbate lock contention.

Christian Brauner’s pull request, referenced in the discussion, confirms that the scalability work originated from Meta’s observations. The pull request notes that the per‑root rwsem design means a single writer can block the entire reader stream for both sysfs and cgroupfs, a situation directly addressed by the new lock‑handling logic.

Impact on System Daemons and Administrator Practices

The immediate effect of the patches is a reduction in the latency of read‑heavy daemons that traverse sysfs or cgroupfs. For oomd, faster access to memory‑related cgroup information translates to more responsive out‑of‑memory decisions, reducing the window in which a system might remain in an over‑committed state. The telemetry daemon that records post‑oomd actions also benefits from reduced blocking, allowing it to capture events with higher fidelity.

From an administrative perspective, the kernel improvements diminish the need for workarounds that attempt to serialize cgroup creation or limit device renames during peak load. Administrators can now rely on the kernel to handle high rates of mutating operations without degrading the performance of monitoring or memory‑management services.

Developers building tools that interact heavily with sysfs or cgroupfs—such as container runtimes, resource‑allocation frameworks, or custom monitoring agents—gain a more predictable execution environment. The reduced lock contention means that read‑heavy loops that enumerate cgroup hierarchies will encounter fewer stalls, leading to smoother scaling as the number of containers or cgroups grows.

Practical Considerations for Deployment

The patches were merged into the mainline kernel and are available starting with Linux 7.4. Systems running earlier kernel versions continue to exhibit the original lock‑contention behavior. Administrators seeking the performance benefits should plan kernel upgrades in accordance with their distribution’s release schedule.

Because the changes are confined to the kernfs subsystem, they do not require configuration changes or recompilation of user‑space components. The kernel’s lock‑handling logic is updated transparently, and existing binaries that depend on sysfs or cgroupfs continue to operate without modification.

Monitoring tools that track daemon latency may observe a noticeable drop in wait times for oomd and related services after the upgrade. Administrators can use existing metrics—such as the time spent in uninterruptible sleep (D state) for these processes—to verify the impact of the kernel changes.

Broader Implications for Large‑Scale Linux Deployments

Meta’s contribution highlights a common scalability challenge in massive server farms: kernel subsystems that were designed for modest concurrency can become bottlenecks when scaled to tens or hundreds of thousands of nodes. The kernfs lock‑contention issue is a concrete example of how a single global rwsem can limit parallelism across the entire machine.

The resolution demonstrates the value of instrumenting kernel paths that are traversed frequently by system daemons. By collecting real‑world telemetry from production workloads, engineers can pinpoint hot paths and prioritize patches that yield the greatest return on investment. The collaboration between Meta engineers and the upstream kernel community also underscores the importance of open‑source contributions for addressing enterprise‑scale performance problems.

Future kernel development may consider redesigning other globally scoped locks to reduce contention in similar scenarios. The kernfs experience suggests that per‑instance or finer‑grained locking strategies could provide scalability benefits without sacrificing correctness.

Source: linux-magazine.com