Disaggregated LLM Inference - When It Pays Off (2026)

Disaggregated LLM inference runs the two phases of a request, prefill and decode, on separate GPU workers. The projects that ship it warn against treating it as a default. The docs for Dynamo, NVIDIA's distributed inference framework, say, "It is not automatically better." The docs for llm-d, a Kubernetes-native distributed inference framework in the CNCF Sandbox, call it "not a target for all workloads."

Vendor headlines quote gains of up to 7x. Yet a note in the vLLM documentation reads, "Disaggregated prefill DOES NOT improve throughput." If you run your own models, that gap decides whether this project earns a place on your roadmap. A wrong bet costs you GPU spend and on-call load.

"Our reading" marks our inference, not a source's finding. Our reading: the clearest case is long prompts that hurt token latency under load, with a fast link between GPUs.

The Bottom Line

• Disaggregation can steady token-to-token latency, and the price in two tests here is a slower first token.
• Judge it on goodput, meaning throughput that meets your latency targets. A vLLM docs note on a two-instance setup says it doesn't improve throughput.
• Headline figures such as "up to 7x" can mix in new hardware and wider parallelism, and may omit the baseline configuration.
• Confirm chunked prefill and prefix caching are on and tuned, then test single-node disaggregation against that baseline on the same GPU count.
• None of the sources here gives a universal prompt-length cutoff, so your own traffic has to settle the question.

In this guide: how disaggregation works, headline claims, measured results, network and operations costs, a decision ladder, what to try first, and a fair test with costs.

How prefill and decode disaggregation works

Every LLM request runs in two phases with opposite hardware appetites. Prefill processes the whole prompt at once, a parallel matrix-matrix job that saturates the GPU. Decode then generates tokens one at a time, closer to a matrix-vector operation and bound by memory (NVIDIA, DistServe). Prefill also builds the KV cache, the stored attention keys and values that decode reads at each step. For a primer on those phases, see what LLM inference is and how it runs.

On a shared GPU, the phases collide. SGLang's docs describe prefill batches interrupting decode batches. Disaggregation, often shortened to P/D, gives each phase its own workers and transfers the KV cache between them. A 1P1D setup means one prefill worker and one decode worker.

Time to first token (TTFT) includes queuing, prefill and network time. Inter-token latency (ITL) is the average gap between output tokens, and NVIDIA treats it as the same as time per output token (TPOT). Tools differ, so compare results only when definitions match.

Our reading: when p99 ITL, the 99th-percentile value, misses its target, answers stall mid-stream, and that puts user trust and revenue at risk.

Why a vLLM note and a 7x headline can both stand

The vLLM note sits on a page about two vLLM instances, one for prefill and one for decode. The benefits that page claims instead are tuning TTFT and ITL separately and controlling tail ITL. It also calls the feature experimental.

Our reading: the gains reported elsewhere are measured at a fixed latency target, known as a service-level objective (SLO), or they come with wider parallelism and different hardware. So they don't contradict a note about two instances. The same reading covers vLLM's own blog reporting 7.9x, a result that combines P/D with scale and wide expert parallelism (EP). Wide EP spreads a mixture-of-experts (MoE) model across many GPUs. That makes goodput, throughput that meets your SLOs, the honest metric.

What the headline numbers really compare

Of the five claims below, two bundle P/D with other changes, NVIDIA's 7x and vLLM's 7.9x. InferenceX is a third-party benchmark from SemiAnalysis, and FP4 is a 4-bit number format. GB200 NVL72 is a rack with 72 Blackwell GPUs in one NVLink domain. NVLink is NVIDIA's direct GPU-to-GPU interconnect.

Don't average these rows. The preprint "From Inference Engine to Inference Control Plane" by Twinkll Sisodia, which offers no new measurements, warns against averaging ratios across papers.

How one operating point can mislead

A May 2026 InferenceX hardware comparison reports throughput per GPU at matched tok/s/user, with both systems disaggregated. GB200 NVL72 over B200 peaked at 4.39x at 125 tok/s/user. It was 1.15x to 1.20x at 25 to 60 tok/s/user, and 0.97x at 250.

NVIDIA's 7x claim sits near 50 tok/s/user, inside that 25 to 60 range, where GB200 over B200 was only 1.15x to 1.20x. Our reading: hardware alone may not explain the 7x. NVIDIA names B200-based systems but not the baseline configuration or absolute values, so the gap stays unexplained. Dates and settings differ, so this isn't a rebuttal.

Our checklist for any vendor claim:

• What's the baseline?
• Did the hardware change too?
• Is the denominator total GPUs or decode GPUs? InferenceX v2 notes that output-only charts normalize by decode GPUs.
• Are cached tokens counted?
• Which interactivity point was chosen?

What the L40S and MI300X tests measured

Results here and in the next section are self-reported by the project or vendor, so read each as a single data point.

The vLLM disaggregated serving guide shows both sides in one small test. It ran Qwen2.5-7B on two L40S GPUs over PCIe, the general-purpose server bus, with about 8,000 input tokens, 256 output tokens and prefix caching off. The baseline was two data-parallel replicas, which the author says isn't the strongest option. Aggregated p99 ITL rose from 23 ms to 169 ms at 0.4 requests per second, then 263 ms at 2 requests per second. P/D held it between 25 and 52 ms.

The cost in the vLLM guide's test showed up in TTFT. At 0.2 requests per second, it was 2.2 seconds with P/D against 0.7 seconds aggregated, and P/D missed a 2-second target at every rate. About 470 MB of KV cache per prompt took about 1.3 seconds to cross that weak link.

A larger AMD test favored P/D. vLLM's post on MoRI-IO, a KV cache transfer connector, ran Qwen3-235B-A22B-FP8 on eight MI300X GPUs in one node. The targets were TTFT under 1 second and ITL under 50 ms. The write-mode 1P1D setup met both targets on 73 of 100 requests, and read mode met them on 70. Two tensor-parallel replicas of four GPUs (TP4) managed 30, and one eight-GPU instance (TP8) managed 26.

That's a single run on a single node, and the baselines don't meet the test protocol later in this article. Prefix caching was off for 2x TP4 and for P/D and isn't reported for 1x TP8, and chunked prefill isn't reported. The baseline commands shown used a batch budget of 8,192, against 4,096 for P/D.

Requests of 100 meeting both latency targetsQwen3-235B-A22B-FP8, 8x MI300X, TTFT under 1 s and ITL under 50 ms,single run1x TP8262x TP430Read 1P+1D70Write 1P+1D73Source: vLLM MoRI-IO KV connector post (2026-04-07)Source: vLLM MoRI-IO KV connector post, 2026-04-07.

Why the TTFT target can flip the result

Anyscale's MI325X test (June 12, 2026) ran vLLM 0.18.0, and P/D delivered 1.3x to 2.3x the queries per second (QPS) on the same GPUs and SLO. Under a TTFT limit of 300 ms, though, P/D served no traffic while aggregated serving sustained 7.0+ QPS. Under 500 ms, P/D reached 5.0 QPS against 7.0+.

Anyscale's best prefill-to-decode ratios varied by workload. They were 2P:1D for 16K in and 1K out, 1P:3D for 16K in and 4K out, and 1P:2D at an 80% cache hit rate. A mismatch can be worse than aggregated on every metric.

Anyscale's view on short outputs and cache hits is advice, not a measured finding. It says savings don't accumulate enough on short outputs to justify the complexity, and it prefers aggregated when cache hit rates remove contention. Its own data still had P/D ahead at a 140-token output, so the advice rests on complexity.

Papers and project guidance point the same way:

• TaiChi finds aggregation suits tight TTFT with relaxed TPOT, and disaggregation suits the reverse.
• Beyond the Buzz finds P/D works best for prefill-heavy traffic and larger models, with limited benefit for small models and generation-heavy traffic. Larger NVLink domains help.
• DistServe's authors say its advantage may shrink for offline work, and few-GPU setups may be simpler without it.
• llm-d's guidance names medium to large models, 10,000 input and 1,000 output tokens rather than 200 and 200, and sparse MoE with wide EP.

The network bill

The KV cache has to move between workers, and that link can make or break P/D. DistServe put KV transmission under 0.1% of total latency on its setup, and Splitwise put optimized transfer at about 0.8% of end-to-end time. Anyscale saw up to 19x throughput loss on a TCP fallback, and llm-d limits TCP to test and development.

AWS estimates single-digit milliseconds for an 8,000-token KV cache on ml.p5.48xlarge with 3,200 Gbps of Elastic Fabric Adapter (EFA), its network interface for fast inter-node traffic. That's an estimate, not a measurement. AWS also excludes G6, G6e and G7e multi-GPU instances as PCIe-limited.

Our reading: transfer bandwidth is rarely the limit on remote direct memory access (RDMA) fabrics in these sources, but connector design still needs testing. On Qwen3-32B with 1P1D on p5en over EFA, at 8 QPS and 16,000 input tokens, KV push gave about 540 ms TTFT, while pull reached about 1,744 ms.

Connector options depend on your engine. NIXL, an Apache 2.0 transfer library, is adopted by llm-d, TensorRT LLM, SGLang and vLLM. vLLM TorchTPU supports P/D across TPU hosts, but the post gives no performance figures. If you're still choosing an engine, compare vLLM vs SGLang first.

The operations bill

P/D adds to your on-call load. The vLLM guide lists three or four services and a new KV transfer failure mode. Its example proxy is for development only.

Reasoning models add a correctness risk. Bidirectional KV transfer means a decode node returns computed KV to a prefill node on later turns, so prefill computes only new tokens. The vLLM guide warns of wrong output if the template drops thinking traces (issue 43094). That issue was auto-closed as not planned on September 19, 2026, after going stale, and closed doesn't mean fixed.

Our reading: the 1P1D minimum means two replicas, so low load strands GPUs you're paying for. Separate pools also complicate GPU autoscaling for LLM inference, because each phase needs its own scaling signal.

Should you disaggregate your LLM inference?

One override comes first, and it's our reading. If p99 ITL misses its SLO with long prompts at concurrency, TTFT has slack, and the checks in the next section haven't fixed it, test P/D even on a small setup. The vLLM guide's L40S test, a 7B model on two GPUs with about 8,000-token prompts, shows P/D winning on ITL there while losing on TTFT, though its author says the baseline wasn't the strongest. TaiChi also favors disaggregation for strict TPOT with relaxed TTFT.

Otherwise, work through these questions in order, because earlier answers take precedence. None of the sources here gives a universal prompt-length cutoff, so treat them as signals, not thresholds. The order and verdicts are our reading.

The second row narrows your options without ending the walk. If nothing applies, stay aggregated until a target is missed. "Very few GPUs" means fewer than two replicas plus headroom, in our reading. A looser TTFT target changes the first answer. P/D did well at a 1-second target in vLLM's MoRI-IO test and at 1,000 ms in NVIDIA's 38% example.

What to try before disaggregating

Cheaper fixes target the same interference, and in vLLM the first two are already defaults.

Chunked prefill splits long prompts into pieces and schedules them alongside decode steps. In vLLM V1 it's on by default where possible. Confirm it's on, then tune max_num_batched_tokens. A setting of 2,048 favors ITL, and values above 8,192 are recommended for throughput. The vLLM throughput guide covers batching in more depth.

Prefix caching defaults to True in vLLM's cache config. It cuts prefill time, not decode time, so confirm it's on and check whether your prompts share prefixes. After that, weigh cache-aware routing, which sends a request to the worker already holding its prefix KV cache, balanced against load.

In the Sarathi-Serve paper, chunked prefills plus stall-free scheduling gave 2.6x capacity, meaning maximum QPS within latency targets, over 2024 vLLM on Mistral-7B with one A100.

Our recommended order is those checks first, then single-node P/D, then multi-node P/D. Each step adds operational surface, so stop at the first one that meets your SLOs.

How to test disaggregation fairly

This test protocol is ours. Section 15 of the Sisodia preprint supports comparing aggregated, chunked-prefill and P/D setups under the same SLO, with transfer included in TTFT.

1. Give both sides the same GPU count.
2. Keep chunked prefill and prefix caching on in the baseline.
3. Replay your real traffic mix.
4. Report goodput at your actual SLOs.
5. Count KV transfer inside TTFT.
6. Adopt P/D only if the gain exceeds the added operations cost.

What the result costs per compliant request

Goodput converts to money like this:

Cost per SLO-compliant request = GPU-hour rate x GPUs / compliant requests per hour

The example below is hypothetical. Prices are AWS Capacity Blocks list rates per GPU-hour in US regions, H100 at $5.970 and B200 at $14.208. They're reservation rates, not on-demand. Request counts are illustrative.

An illustrative 30% goodput gain on the same GPUs cuts cost per compliant request by about 23%. That $3.67 per 1,000 requests must then beat the engineering and on-call time for the extra services.

B200 costs about 2.38 times H100 per GPU-hour at these rates, so it needs about 2.38 times the compliant requests to break even. For the wider hardware choice, see H100 vs H200 vs Blackwell for LLM inference.

Talk through your numbers with a cloud architect

If your p99 ITL is missing its SLO and you want a second view on the test plan, book a 30-minute call with a cloud architect.

The EaseCloud Team writes about cloud performance, reliability and AI infrastructure. See the team page and about EaseCloud. Sources were retrieved on November 6, 2026, and analysis of ours is labeled as such.