CXL and Its Impact on Memory Management: Key Questions Answered

Compute Express Link (CXL) is a high-speed interconnect standard that aims to revolutionize data center memory architecture by enabling shared memory pools accessible to multiple CPUs. However, as Dan Williams highlighted at the 2026 Linux Storage, Filesystem, Memory Management, and BPF Summit, CXL has been exacerbating memory-management challenges since its introduction in 2021. This Q&A explores how CXL works, the problems it creates, and what the future holds for this technology.

1. What is Compute Express Link (CXL) and what problem does it solve?

CXL is an open-standard interconnect designed to provide low-latency, high-bandwidth communication between CPUs, memory, and accelerators. Its primary goal is to enable memory disaggregation—separating memory from individual servers and pooling it across data centers. This allows multiple CPUs to access a shared memory pool, improving utilization and flexibility. By decoupling memory from compute, CXL helps address the inefficiency of static memory allocation in traditional servers, where memory often sits idle while other nodes face shortages. It also supports memory tiering, where slower, cheaper memory can supplement faster DRAM. However, implementing this technology introduces new complexities in memory management, as the operating system must now handle dynamic memory sharing, coherency, and latency variations across a fabric.

CXL and Its Impact on Memory Management: Key Questions Answered

2. Why did Dan Williams say that CXL has been making memory-management problems worse since 2021?

Dan Williams, a kernel developer, noted that CXL has intensified existing memory-management difficulties. Before CXL, memory was tightly coupled to a single CPU, making allocation and reclaim straightforward. With CXL, memory becomes a shared resource, leading to issues like increased complexity in NUMA (Non-Uniform Memory Access) topology, because access latencies vary depending on whether memory is local, remote within a node, or across a CXL link. The kernel must track multiple memory tiers and decide where to allocate pages, which can degrade performance if not tuned correctly. Additionally, CXL introduces new failure modes—when a shared memory node goes offline, it affects multiple CPUs. The scalability of traditional memory management algorithms also suffers as the number of memory nodes grows. Williams argued that these challenges have compounded since CXL's arrival, requiring significant kernel changes.

3. How does CXL affect memory sharing and coherency in a data center?

CXL supports three protocols: CXL.io for I/O, CXL.cache for caching, and CXL.mem for memory access. The memory access protocol allows a CPU to directly attach to a remote memory pool, but maintaining cache coherency across multiple CPUs sharing the same memory is challenging. Traditional cache coherency protocols (like MESI) assume a small number of processors on a bus; CXL extends this to a fabric, increasing snoop traffic and latency. To mitigate this, CXL uses a device-coherent model where the memory controller manages coherency, but it still adds overhead. In practice, shared memory regions require careful synchronization using locks or atomic operations, which can become bottlenecks. The kernel must also handle NUMA effects—a CPU accessing remote CXL memory experiences higher latency than local DRAM, so the scheduler and memory allocator must account for this to avoid performance degradation.

4. What are the main challenges in integrating CXL with existing operating systems?

Operating systems like Linux were designed for a fixed memory hierarchy with local DRAM. CXL introduces memory hot-plug semantics, where memory can appear or disappear at runtime, requiring dynamic resource management. The memory management subsystem must support multiple tiers (e.g., local DRAM, CXL-attached memory, persistent memory) and decide placement policies. For example, cache coherency across tiers adds complexity. Additionally, memory errors in CXL-attached devices need new handling—a single bit flip can affect multiple VMs. The kernel must also implement efficient migration of pages between tiers based on access patterns, which requires new heuristics and system calls. Virtualization adds another layer: hypervisors must expose CXL memory to guests while maintaining isolation. Dan Williams indicated that these changes are ongoing and that the kernel community is actively developing solutions like memmap and daxctl.

5. How might future developments in CXL improve memory management?

Future versions of CXL (e.g., CXL 3.0) aim to address current limitations by adding features like finer-grained interleaving, improved coherency protocols, and support for composable memory systems. These could reduce latency and simplify sharing. On the software side, Linux is evolving to better handle CXL through mechanisms like numa balancing, automatic tiering, and persistent memory awareness. Research into machine learning–guided page placement might optimize for workload patterns. Additionally, CXL switches will allow complex topologies with many memory nodes, and kernel enhancements like dynamic topology discovery and resource partitioning are expected. However, as Williams noted, these advances will require careful design to avoid worsening existing problems—each new feature adds complexity. The goal is to make CXL memory as transparent and efficient as local DRAM, but that remains a work in progress.

6. What role does the Linux kernel community play in overcoming CXL challenges?

The Linux kernel community is at the forefront of integrating CXL. Developers like Dan Williams contribute patches to the memory management, block layer, and filesystem subsystems to handle CXL devices. Ongoing efforts include the CXL subsystem in the kernel (drivers/cxl), which manages device discovery, hot-plug, and resource allocation. There are also initiatives to extend numa and devdax interfaces, and to handle memory errors through EINJ and MCE handlers. Community discussions at conferences like LSFMM focus on refining these approaches—for instance, balancing the trade-off between performance and flexibility in shared memory pools. The kernel's support for CXL is still maturing, and contributions from hardware vendors and cloud providers are crucial to ensure robust, production-ready solutions. The collaborative model allows rapid iteration, but it also means that changes take time to stabilize, as seen with the ongoing difficulties Williams highlighted.

Tags: