vLLM vs SGLang Prefix Caching: 2026 Latency Benchmark Results

SGLang can outperform vLLM when an agent repeatedly sends the same long prefix. That advantage isn’t automatic, and it may disappear before the response reaches the user.

Tokenization, queueing, cache eviction, serialization, and network transport can all offset a scheduler-level improvement. The practical comparison between vLLM and SGLang is therefore not a contest over raw generation speed. It’s a measurement exercise: test the complete request path with the cache-hit ratio, prompt structure, and concurrency your production system actually sees.

Key takeaways

• Prefix caching helps most when requests reuse thousands of input tokens.
• Published SGLang figures are useful reference points, not a matched vLLM comparison.
• Random prompts don’t test prefix reuse.
• Choose the runtime that improves end-to-end p95 latency for completed agent turns, not just GPU-side prefill time.

Why prefix caching changes the comparison

An agent often resends the same system prompt, tool definitions, repository instructions, and conversation history on every turn. A support assistant may carry a policy document across a session. A retrieval system may include many of the same passages even when the question changes.

Without caching, the model processes that input again before generating the next token. Prefix caching stores the model’s key-value state for a reusable portion of the prompt. On a hit, the server can skip some prefill work.

That can reduce time to first token (TTFT), especially for long contexts. But a cache hit only removes one part of the request path. The request may still wait in a queue, spend time in tokenization, cross a network boundary, or stall while the client parses streamed output. The cached prefix must also match exactly and remain available on the worker handling the request.

SGLang makes this kind of reuse central to its serving design through radix-tree-style caching, often discussed with RadixAttention. vLLM also supports prefix caching, continuous batching, OpenAI-compatible serving, and mature benchmark tooling. The meaningful comparison is not “caching versus no caching.” It’s how the two runtimes behave under the same workload and cache conditions.

Consider a simple example:

• Request arrives at the client: 0 ms
• Queue delay: 70 ms
• Tokenization: 35 ms
• Cached prefill: 20 ms
• Network and scheduling overhead before the first streamed token: 25 ms

The user sees a TTFT of roughly 150 ms, even though the cached model work takes only 20 ms. If a cache miss adds 180 ms of prefill, TTFT rises to about 310 ms. Caching helps, but it doesn’t make queueing or tokenization irrelevant.

What the published SGLang figures show

The SGLang project’s published serving benchmark reports a workload of 512 requests with maximum concurrency of 40, approximately 1.34 million input tokens, and about 32,100 generated tokens. The figures are available through the SGLang project repository and benchmark materials.

The reported baseline was:

• 219.36 output tokens per second
• 785.93 ms mean TTFT
• 467.95 ms median TTFT
• 868.26 ms p99 ITL

A data-parallel variant reported:

• 194.96 output tokens per second
• 2,413.96 ms mean TTFT
• 1,086.05 ms p99 ITL

Here, ITL means inter-token latency in milliseconds per token. Some tools use TPOT, or time per output token, for essentially the same streaming interval. Pick one term for the report and define it clearly; don’t compare an aggregate token rate with a per-token latency as if they measured the same thing.

These results are useful, but they aren’t a head-to-head vLLM benchmark. The model revision, GPU configuration, tokenizer, prompt distribution, cache state, sampling settings, and request schedule would all need to match before the numbers could establish a winner.

The data-parallel result does make one operational point clear: adding workers can increase latency. Routing overhead, synchronization, queue imbalance, and reduced cache locality may cost more than the additional capacity saves.

The SGLang GitHub issue tracker also contains reports of client-observed ITL diverging sharply from scheduler-side measurements when tokenization or detokenization becomes a bottleneck. One reported case describes p50 ITL improving from roughly 180 ms to 39 ms after tokenizer initialization was bypassed. Treat that as issue-level diagnostic data, not as a universal performance claim.

Measure both sides:

1. Runtime-side scheduler and model execution.
2. Client-observed TTFT, inter-token latency, and completed-request time.

A server trace can look excellent while the application remains slow.

Build a fair vLLM-versus-SGLang test

Hold these variables constant:

• Model revision and quantization
• Tokenizer files
• GPU model and count
• CUDA or ROCm version
• Context-length limits
• Sampling parameters
• Prompt and output distributions
• Concurrency and request rate
• Streaming behavior
• Client implementation and network path

Run three workload profiles.

Interactive agent

Use roughly 128–512 input tokens and 64–256 output tokens at low and medium concurrency. Track TTFT p50 and p95, because users notice the delay before the first response more than a small difference in aggregate throughput.

Long-context retrieval

Use 4K–32K input tokens with short or medium outputs. Test cold-cache requests, warm-cache requests, and a mixed stream that creates realistic eviction pressure.

Multi-turn agent

Keep the system prompt, tool definitions, and most conversation history stable while changing the user request. Include tool calls, retries, structured output, and occasional long turns. The useful unit here is a completed agent turn, not simply output tokens per second.

A simple test matrix is enough to expose most cache behavior:

Don’t use random prompts for cache tests

Random benchmark prompts are useful for load generation, but they generally don’t reproduce prefix reuse. If every request begins with different tokens, the cache will have little opportunity to help.

For a warm test, generate a fixed prefix and append a changing suffix:

Shared prefix:
You are the Acme coding agent.
Tools: search_repo, read_file, run_tests, apply_patch.
Repository map:
[stable repository summary]

Variable suffix:
User request: Fix the failing authentication test in module N.

Send the same shared prefix across all requests while varying only the final user request. For a mixed test, divide traffic among several stable prefixes—for example, 50 repository contexts—and control how often each appears. That produces a measurable hit ratio and lets you test eviction rather than assuming it.

A vLLM benchmark command might look like this:

vllm bench serve \
  --backend openai-chat \
  --model /models/your-model \
  --host 127.0.0.1 \
  --port 8000 \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 256 \
  --num-prompts 512 \
  --max-concurrency 40 \
  --request-rate inf

Flags vary by release, so pin the vLLM version and preserve the full command. For a cache experiment, replace the random dataset with a request harness that sends stable prefixes. Run the equivalent SGLang workload with matching lengths, concurrency, request count, and streaming settings.

Also use an external HTTP client. Record:

request_start
first_byte
first_token
last_token
response_complete

Then calculate:

TTFT = first token − request start
End-to-end latency = response complete − request start
Streaming tail = response complete − first token

Record queue time, cache-hit ratio, tokenizer and detokenizer time, GPU memory, throughput, timeout rate, and errors. Correlate latency with each request’s hit or miss status. A single blended p95 can hide excellent cache-hit performance alongside disastrous misses.

Choosing a runtime

vLLM is a sensible starting point when you need broad model support, a familiar OpenAI-compatible API, predictable serving workflows, or a system that handles varied traffic. Its practical advantage is operational breadth: it gives teams a capable baseline without requiring the workload to be dominated by repeated prefixes.

SGLang deserves a focused evaluation when most requests reuse substantial context—such as long tool schemas, repository state, multi-turn histories, or structured-generation templates. It becomes more attractive when the team can tune and monitor a specialized serving stack.

Don’t use a fixed “50% hit rate” rule without considering prefix length. A 70% hit rate on 300 tokens may matter less than a 30% hit rate on 8,000 tokens. Track the number of reused tokens, cache residency, and eviction rate alongside the percentage of requests marked as hits.

The deployment decision should come from end-to-end evidence:

• Does SGLang reduce p95 TTFT on warm traffic?
• Does that improvement survive mixed traffic and eviction?
• Does client-side tokenization erase the scheduler gain?
• Does routing across workers reduce cache locality?
• Does the runtime improve completed-agent-turn latency, not just tokens per second?

Choose SGLang when it delivers a durable user-visible improvement under those conditions. Choose vLLM when the gain is small, traffic is unpredictable, or simpler operations and broader compatibility carry more value. In either case, test cold, warm, and mixed cache states before putting the serving layer behind real users.