Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale
Files
TR Number
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Disaggregated LLM serving improves resource efficiency by separating prefill and decode across workers, but introduces a new bottleneck: key/value (KV) tensors generated during prefill must be transferred to decode workers, typically through the network stack. CXL memory offers an attractive alternative by enabling multiple hosts to share a large byte-addressable memory pool, allowing KV blocks to be published and consumed without network transfer. Realizing this vision is challenging because current CXL devices provide neither cross-host atomic operations nor full-device cache coherence, and expose only raw shared memory. This thesis presents TraCT, a rack-scale LLM serving system that uses CXL shared memory as both a network-free KV-transfer substrate and a rack-scale prefix-aware KV cache. TraCT overcomes three key challenges imposed by current CXL hardware: inter-node mutual exclusion without hardware atomics, correct visibility on non-coherent shared memory, and portable shared-memory data structures without shared pointers. To this end, it introduces a two-tiered lock, a software-managed coherence discipline, and a shared object directory. Implemented as a general-purpose CXL shared-memory library with a vLLM external KV-caching plugin, TraCT achieves up to 2.6x reduction in average time-to-first-token (TTFT), 3.0x lower P99 latency, and 1.9x higher peak goodput than RDMA-based baselines on a real-world multi-turn conversational workload. These results demonstrate that CXL shared memory is a practical and efficient substrate for KV transfer and prefix caching in rack-scale disaggregated LLM serving.