SGLang vs vLLM: Prefix Caching Performance Benchmark 2026

The short answer to SGLang vs. vLLM prefix caching is conditional: SGLang is often the stronger candidate for workloads that reuse long prompt prefixes, while vLLM remains a reliable general-purpose default.

The difference usually appears in time to first token (TTFT) and goodput, not necessarily in raw decode speed. A random-prompt benchmark can make the runtimes look nearly identical. A multi-turn agent that repeatedly sends the same system prompt, tool definitions, conversation history, and retrieved context is a very different workload.

Key takeaways

• SGLang is particularly well suited to repeated-context traffic through RadixAttention and prefix-aware execution.
• vLLM offers broad model and accelerator coverage, mature tooling, continuous batching, and a widely used OpenAI-compatible serving layer.
• Benchmark cold misses, warm hits, and mixed traffic separately. One tokens-per-second number won’t show the real trade-off.

Why Prefix Caching Changes the Runtime Decision

In a conventional serving test, teams tend to focus on output throughput: how many tokens can the cluster generate per second?

That’s useful, but it may not be the first bottleneck in a RAG or agentic application. Long requests often spend substantial time in prefill, when the model processes the input before generating the first output token.

An agent request might contain:

• A 4,000-token system prompt
• 2,000 tokens of tool definitions
• Several thousand tokens of conversation history
• Retrieved documents
• A changing user instruction
• A short generated answer

If the beginning of those requests is identical, prefix caching lets the runtime reuse the corresponding key-value (KV) cache rather than recomputing it. The shared portion avoids most of its prefill work; the changing suffix still has to be processed.

That should reduce prefill latency and TTFT. It doesn’t eliminate queueing, network overhead, suffix processing, decoding, or retrieval time.

SGLang makes this reuse a central part of its execution model through RadixAttention, which stores reusable prompt structure in a radix tree. vLLM also supports prefix caching alongside PagedAttention, continuous batching, chunked prefill, and KV-cache management.

The question isn’t whether either runtime has a cache. Both do. The useful question is how each behaves when caching meets concurrency, cache eviction, long contexts, and mixed request shapes.

A benchmark that repeatedly sends one identical prompt can produce impressive results while measuring a warm-cache laboratory rather than production traffic. That warning applies to both runtimes.

What Public Evidence Can—and Cannot—Tell You

There is no neutral, universal leaderboard showing that SGLang beats vLLM across models and hardware. Runtime version, GPU topology, quantization, prompt distribution, cache policy, and request rate can all change the outcome.

Public benchmark suites are still useful, but mostly for establishing a measurement vocabulary. vLLM’s tooling, for example, reports TTFT, time per output token, inter-token latency, throughput, concurrency, and goodput. SGLang provides its own serving and performance tools.

Those results don’t answer the production question by themselves. A system-level benchmark such as MLPerf can show what a complete hardware and software configuration achieved under a fixed workload. It cannot predict the benefit of RadixAttention for a company’s multi-turn agent traffic.

The practical conclusion is simple: use published results to choose candidate configurations, then run a controlled head-to-head test on your own prompt distribution.

When is SGLang likely to win?

SGLang has the clearest opportunity when:

• Shared prefixes are several thousand tokens long
• A high percentage of input tokens are reused
• Prefill dominates TTFT
• Sessions remain on workers with useful cache locality
• The cache is large enough to avoid constant eviction

The case is weaker when prompts are mostly unique, generated outputs are long enough for decode to dominate, or the workload is spread across models with uneven runtime support.

A 70% cache hit rate also doesn’t imply a 70% latency reduction. Hits save some prefill computation, not every part of the request. Queue time, suffix prefill, decoding, network overhead, and application work remain.

A Benchmark That Answers the Real Question

Run at least three traffic profiles:

1. Cold or random traffic, where little prefix reuse is available.
2. Warm traffic, where requests intentionally share long prefixes.
3. Mixed production traffic, combining cache hits, misses, evictions, and bursts.

Pin the conditions that can otherwise obscure the comparison:

• Model revision and tokenizer
• Runtime release or commit
• CUDA, ROCm, and driver versions
• GPU model, count, and interconnect
• Precision and quantization
• Input and output length distributions
• Request arrival pattern
• Maximum sequence length
• Tensor and pipeline parallel settings
• Sampling and structured-output configuration

A useful test matrix might look like this:

Report p50, p95, and p99 TTFT, not just the mean. Also capture:

• Time per output token
• Inter-token latency
• End-to-end latency
• Queue time
• Input and output token throughput
• GPU utilization
• KV-cache occupancy
• Cache hit tokens and hit rate
• Eviction and preemption counts
• Out-of-memory events
• Goodput

Goodput = requests that meet the defined latency SLO per second

That metric keeps the comparison tied to service quality. A configuration that lowers median TTFT by 30% but misses the p95 target during bursts may be less useful than a slightly slower configuration that serves more compliant requests.

Normalizing the benchmark commands

The benchmark client should be the same for both runtimes whenever possible. Send identical HTTP requests through the same OpenAI-compatible schema, record timestamps at the client, and normalize the server metrics afterward.

For a vLLM deployment, a version-specific command may look like this:

# vLLM example; flags vary by release.
vllm bench serve \
  --model /models/llama \
  --dataset-name random \
  --num-prompts 2000 \
  --random-input-len 4096 \
  --random-output-len 256 \
  --request-rate 16 \
  --max-concurrency 64

SGLang can be tested with its serving benchmark module or the same external load generator pointed at the SGLang OpenAI-compatible endpoint:

# SGLang example; confirm option names for the installed release.
python -m sglang.bench_serving \
  --backend http://127.0.0.1:30000 \
  --dataset-name random \
  --num-prompts 2000 \
  --input-len 4096 \
  --output-len 256 \
  --request-rate 16 \
  --max-concurrency 64

The exact flags differ between releases, so the command syntax is less important than keeping the workload identical. If one runtime’s native benchmark reports server-side TTFT and the other reports client-side latency, use a common load generator and a shared results schema instead of comparing the numbers directly.

Warm the cache deliberately for one test. Clear it, restart the server, or randomize prefixes for another. Label every result as cold, warm, or mixed.

Memory, Eviction, and Session Routing

Prefix caching isn’t free. Retained KV blocks compete with model weights, active sequences, batching headroom, and temporary workspace.

At high concurrency, a common failure pattern looks like this:

1. The cache hit rate rises.
2. Retained KV blocks consume more accelerator memory.
3. Active requests have fewer free blocks.
4. Evictions or preemptions increase.
5. Queue time and p99 TTFT deteriorate.

Plot cache hit rate beside p95 TTFT and KV-cache occupancy. The best operating point is often below the largest possible cache size.

A realistic production example is a support agent with a stable 6,000-token policy and tool prefix, followed by customer-specific history and retrieved documents. Under light load, SGLang may reduce TTFT substantially because the policy and tool prefix remain resident. Under a burst of new tenants, however, those prefixes may compete for memory. If cache occupancy reaches the limit and sessions are routed randomly, hit rates can fall while tail latency climbs. The benchmark should expose that transition rather than report only the best warm-cache result.

Choosing Between vLLM and SGLang

Choose vLLM when you need a broad serving baseline across model families, quantization formats, and accelerator environments. Its OpenAI-compatible API, continuous batching, PagedAttention, benchmark tooling, and established operational practices can reduce integration and rollback risk.

That flexibility matters when an inference team regularly evaluates new Llama, Qwen, Mistral, or multimodal releases. Faster onboarding and predictable model support may outweigh a specialized prefix-cache advantage.

Choose SGLang when repeated context defines the workload:

• Multi-turn agents with long retained histories
• RAG systems with stable instructions and schemas
• Batch jobs generated from the same document or template
• Long-context services where prefill dominates TTFT

Structured output needs its own test. Reusing prompt context does not mean grammar work is automatically reusable. Grammar compilation, schema processing, and constrained decoding have separate costs and may change the comparison. Measure those paths with realistic schemas and request diversity.

Real-World Limitations That Lower Hit Rates

Prefix caching can disappoint when the application changes prompts in small ways. Different whitespace, tokenization, serialization order, timestamps, request IDs, or tool-definition formatting can break an otherwise reusable prefix.

Cache-key normalization deserves explicit attention. Define which fields belong in the reusable prefix and keep volatile values in the suffix. Session routing matters just as much: random load balancing can send each turn to a worker that lacks the previous KV blocks.

Tenant isolation is another constraint. Sharing cached content across tenants may be unacceptable, even when the text appears identical. Per-tenant keys or isolated caches improve safety but reduce reuse and increase memory pressure.

Finally, document every prompt change that invalidates a prefix. A new system instruction, reordered tools, changed retrieval header, or tokenizer revision can turn a warm workload into a cold one overnight.

The practical decision rule is straightforward:

Use SGLang if it delivers a material improvement in SLO-constrained goodput or p95 TTFT on your shared-prefix traffic without unacceptable memory or operational costs. Use vLLM when traffic is mixed, prefixes are often unique, or portability and model coverage matter more.

Don’t choose from a single warm-cache run. Compare cold, warm, and mixed traffic at production concurrency, then run a long soak test to expose cache churn, routing problems, and memory fragmentation.