Mooncake
A KVCache-Centric Disaggregated Architecture for LLM Serving
Trading more storage for less computation—the serving infrastructure behind Kimi, processing 100+ billion tokens daily across thousands of nodes.
Serving LLM chatbots at scale is brutally hard. Each request has two distinct phases with fundamentally different computational characteristics—and traditional systems that couple them together leave massive performance on the table.
Prefill Phase
- •Processes all input tokens in parallel
- •Compute-bound: GPU cores are the bottleneck
- •Generates KVCache for all input tokens
- •Latency metric: TTFT (time to first token)
Decode Phase
- •Generates one token at a time (autoregressive)
- •Memory-bound: KVCache access is the bottleneck
- •Reads KVCache + appends new KV pairs
- •Latency metric: TBT (time between tokens)
The Problem with Coupled Systems
Traditional serving systems (vLLM, TGI) run both phases on the same GPUs. This creates critical inefficiencies:
Long prefills (processing 100k+ token contexts) block time-sensitive decode operations, causing TBT spikes and degraded user experience.
GPU clusters have abundant CPU, DRAM, and SSD resources sitting idle. These could store KVCache but are completely unused.
System prompts and common prefixes are recomputed for every request, wasting compute on redundant work.
Can't independently scale prefill capacity (for long contexts) vs decode capacity (for high concurrency).
The Kimi Challenge
Moonshot AI's Kimi chatbot faces extreme scale: millions of daily users, 200K+ context windows, and exponential traffic growth. Traditional architectures couldn't keep up. Mooncake was built to solve this.
Mooncake's key innovation is disaggregation: separate prefill and decode into independent clusters, connected by a distributed KVCache layer. This enables specialized optimization for each phase while leveraging previously wasted resources.
Disaggregated vs Coupled Architecture
Mooncake's disaggregated design separates prefill (compute-intensive) and decode (memory-bound) into independent clusters. KVCache is stored in a distributed pool using idle CPU DRAM and SSD, enabling cache reuse across requests and independent scaling of each phase.
Prefill Cluster
Dedicated GPUs optimized for compute-intensive parallel processing. Generates KVCache and streams it to the storage layer.
KVCache Pool
Distributed cache using CPU DRAM and SSD across the cluster. Enables prefix reuse and reduces redundant computation.
Decode Cluster
Optimized for memory-bound autoregressive generation. Receives KVCache via high-speed RDMA transfers.
Why Disaggregation Wins
Long-context workloads? Add prefill nodes. High concurrency? Add decode nodes. No longer forced to scale everything together.
Prefill can use chunked pipeline parallelism; decode can maximize batch size. Each cluster runs optimal strategies for its workload.
~75% of cluster DRAM sits idle in coupled systems. Mooncake repurposes it for KVCache storage, dramatically improving efficiency.
Prefill never blocks decode latency. TBT stays consistent even during heavy long-context prefill loads.
Mooncake builds a multi-tier storage hierarchy for KVCache, utilizing resources that traditional systems waste. The key insight: GPU VRAM is precious, but CPU DRAM and SSD are abundant and cheap.
Storage Tiers
Block-Based Storage
KVCache is stored as fixed-size paged blocks with content-addressable hashing:
- •Each block has a hash combining its content + prefix hash
- •Enables efficient prefix matching across requests
- •Automatic deduplication saves storage
Cache Hit Rates
Production traces show significant cache reuse:
Hot-Spot Migration
When certain cache blocks become “hot” (frequently accessed), Mooncake automatically replicates them across multiple nodes. This prevents any single node from becoming a bottleneck and ensures low-latency access for popular prefixes.
At the heart of Mooncake is the Conductor—a global scheduler that makes KVCache-centric routing decisions. Unlike simple load balancers, Conductor considers cache locality, transfer costs, and SLO targets.
KVCache-Centric Scheduling
Cache-Aware
Routes requests to instances with matching prefix cache, avoiding redundant computation.
Load Balanced
Uses SLO satisfaction (not request count) as the load metric for accurate balancing.
TTFT Optimized
Minimizes time-to-first-token by considering queue wait + compute + transfer time.
Scheduling Algorithm
Divide input into blocks, compute hash keys, search for longest matching prefix in cache.
For each prefill instance: queue wait time + execution time + cache transfer time.
Route to instance with lowest estimated TTFT. Reject if no instance can meet SLO.
Proactively replicate hot blocks; trigger transfer if remote cache saves more than transfer costs.
SLO-Based Load Metric
Instead of counting requests (misleading for variable-length inputs), Conductor uses SLO satisfaction rate as the load signal. High TTFT/TBT violations indicate overload—not just high request count.
Cache-Aware vs Random
Random scheduling: requests go to any available instance. Cache-aware scheduling: requests route to instances with matching prefixes. Result: significant TTFT reduction from cache hits.
For very long contexts (100K+ tokens), a single node can't complete prefill fast enough. Mooncake uses Chunked Pipeline Parallelism (CPP)—a novel approach that dramatically reduces cross-node communication compared to traditional sequence parallelism.
Multi-Node Prefill Strategies
Chunked Pipeline Parallelism groups nodes into pipeline stages. Input is split into chunks (>1000 tokens each) that flow through the pipeline. Cross-node communication only occurs at chunk boundaries—dramatically reducing network overhead for long-context prefills.
Sequence Parallelism Problems
- ✗All-reduce after every transformer layer
- ✗32 layers = 32 synchronization points per forward pass
- ✗Network bandwidth becomes the bottleneck
- ✗GPUs spend significant time waiting for communication
CPP Advantages
- ✓Communication only at chunk boundaries
- ✓Chunks >1000 tokens amortize communication cost
- ✓Better GPU utilization (higher MFU)
- ✓Natural pipeline parallelism for overlapping work
How CPP Works
Group N nodes into a pipeline. Partition input into chunks. Each chunk flows through all nodes sequentially:
Chunk 2: Node 1 (layers 1-8) → Node 2 (9-16) → ... (overlapped with Chunk 1)
Chunk 3: ...
While Node 2 processes Chunk 1's layers 9-16, Node 1 can start Chunk 2's layers 1-8.
Mooncake overlaps KVCache loading/storing with computation using layer-wise asynchronous execution. Instead of loading all KVCache upfront, each layer's cache is loaded just before that layer's attention computation.
Async Execution Pattern
Load for layer N+1 overlaps with compute for layer N → hidden latency
Benefits
- •Transfer time hidden behind compute
- •Reduced peak VRAM usage (not all KVCache loaded at once)
- •Effective for long-context scenarios
VRAM Reduction
Traditional: Load all KVCache upfront → O(S×T) VRAM where S=size, T=time. Layer-wise: Only one layer's KVCache in VRAM at a time → O(S) VRAM. Enables serving longer contexts on same hardware.
The Transfer Engine is the foundation of Mooncake's data movement layer. It provides high-speed, topology-aware transfers across heterogeneous memory (VRAM, DRAM, SSD) using RDMA and other optimized protocols.
Supported Protocols
Performance Numbers
40GB transfer (128K token KVCache for LLaMA3-70B):
Key Features
- •Topology-aware: NUMA affinity, path selection
- •Multi-NIC aggregation: Saturate available bandwidth
- •Automatic failover: Retry on alternate paths
- •Batched transfers: Efficient for many small blocks
Under severe overload, Mooncake must reject requests to maintain SLOs for accepted ones. But naive rejection creates oscillation between prefill and decode clusters. Mooncake uses prediction-based early rejection to stabilize load.
The Oscillation Problem
Simple rejection based on current decode load causes anti-phase oscillations:
- 1.Prefill hits capacity → requests queue
- 2.Queued requests complete → flood decode cluster
- 3.Decode overloaded → reject new prefills
- 4.Prefill load drops → decode catches up → accept more prefills
- 5.Cycle repeats → severe underutilization
Prediction-Based Solution
Instead of per-request output prediction (expensive, inaccurate), Mooncake uses system-level prediction:
2. Calculate which decode requests finish by time t
3. Compute average TBT ratio vs threshold
4. Reject if projected TBT exceeds SLO
This smooths the load signal, preventing oscillation while maintaining throughput.
Rejection Policy Comparison
| Policy | Requests Rejected | Improvement |
|---|---|---|
| Baseline (no early) | 4,183 | — |
| Early Rejection | 3,771 | 9.9% |
| Early + Prediction | 3,589 | 14.2% |
Mooncake was evaluated on public datasets, synthetic benchmarks, and real production workloads. The results show massive improvements especially in long-context scenarios.
Headline Results
Public Dataset Results
Simulated Long-Context
Real Workload: 23,000 Requests
Mooncake (10P+10D) vs vLLM-20M on identical production trace:
Mooncake isn't just research—it's the production infrastructure behind Kimi, Moonshot AI's LLM chatbot serving millions of users with 200K+ token context windows.
Production Scale
Hardware Configurations
Handling Growth
Kimi experienced exponential traffic growth. Mooncake enabled:
- •Independent scaling of prefill vs decode capacity
- •Graceful degradation under overload (early rejection)
- •Efficient utilization of existing infrastructure
Explore Mooncake's key innovations interactively.
Disaggregated Architecture
Disaggregated vs Coupled Architecture
Mooncake's disaggregated design separates prefill (compute-intensive) and decode (memory-bound) into independent clusters. KVCache is stored in a distributed pool using idle CPU DRAM and SSD, enabling cache reuse across requests and independent scaling of each phase.
KVCache-Centric Scheduling
KVCache-Centric Scheduling
Cache-Aware
Routes requests to instances with matching prefix cache, avoiding redundant computation.
Load Balanced
Uses SLO satisfaction (not request count) as the load metric for accurate balancing.
TTFT Optimized
Minimizes time-to-first-token by considering queue wait + compute + transfer time.
Multi-Node Prefill Strategies
Multi-Node Prefill Strategies
Chunked Pipeline Parallelism groups nodes into pipeline stages. Input is split into chunks (>1000 tokens each) that flow through the pipeline. Cross-node communication only occurs at chunk boundaries—dramatically reducing network overhead for long-context prefills.
Mooncake's core components are open source and have been integrated into major LLM serving frameworks.
Open Source Components
- ✓Transfer Engine: High-speed RDMA transfer library
- ✓Mooncake Store: Distributed KVCache storage engine
- ✓P2P Store: Checkpoint file distribution
Framework Integrations
- •vLLM v1: KV Connector for PD disaggregation
- •SGLang: HiCache hierarchical storage backend
- •TensorRT LLM: KVCache transfer integration
- •LMDeploy: PD disaggregation backend
Resources
Mooncake: The Complete Picture
By disaggregating LLM serving around KVCache, Mooncake transforms wasted resources into competitive advantage— enabling Kimi to serve millions with 200K+ context while maintaining strict latency SLOs.