vLLM vs TensorRT-LLM Agents: 2026 Latency and Cost Benchmark

A fast model server can still produce a slow agent.

Consider an agent that makes six model calls, each generating roughly 100 tokens. Between those calls, it waits on a database, a browser, an API, or a code sandbox. The model may decode quickly, yet the user spends most of the time watching the agent wait in queues, rebuild context, and coordinate tools.

For AI agents, the better inference engine is the one that minimizes time and cost per successfully completed task—not the one with the highest isolated tokens-per-second score.

Key takeaways

• TTFT often matters more than peak throughput. Short agent responses make queueing and prefill highly visible.
• Cache reuse can change the economics. Reusing system prompts, tool schemas, and conversation context may save more time than a modest decode improvement.
• Choose from measurements, not reputation. Start with a portable baseline, find the real bottleneck, then consider hardware-specific optimization.

Why tokens per second misleads agent teams

A conventional inference benchmark sends one prompt, generates a long response, and divides output tokens by elapsed time. That model works for batch summarization and offline generation. It doesn't describe a tool-using agent very well.

An agent may make six model calls during one task. Each response might contain only 50 to 200 tokens, followed by a database query, browser action, API request, or code execution step. The user doesn't care that the server eventually reached 400 tokens per second. They care whether the task finished in four seconds or fourteen.

A more useful model is:

Agent Task Time = Queueing + Prefill + Decode + Tool Execution + Network + Orchestration

Across multiple turns:

Total Agent Time = Σ (Queueing + Prefill + Decode + Tool Execution + Network + Orchestration)

This is why a serving engine with excellent decode throughput can still lose on end-to-end performance. If short requests queue behind large batches, repeated context isn't cached, or orchestration adds a network hop to every turn, the agent gets slower while GPU utilization still looks healthy.

For a vLLM versus TensorRT-LLM evaluation, start with TTFT, P95 task latency, cache behavior, and completed-task cost. Tokens per second belongs in the report, but it shouldn't be the headline.

What vLLM and TensorRT-LLM optimize

vLLM is often the practical starting point for teams that need broad model support, quick iteration, and a relatively accessible production path. Continuous batching and paged KV-cache management are useful when requests arrive with different prompt and output lengths. That flexibility fits agent traffic, which is rarely uniform.

TensorRT-LLM becomes more attractive when the deployment is NVIDIA-specific, the model is stable, and the team can invest in engine-specific tuning. Kernel fusion, quantization, speculative decoding, and hardware-aware optimizations can produce excellent results. The trade-off is tighter coupling to NVIDIA GPUs, TensorRT-LLM releases, supported operators, and a more involved build and validation process.

Neither stack wins every workload. A platform that changes models weekly may value portability more than a tuned kernel. A high-volume service with one stable model may gladly accept the additional coupling.

Build a benchmark around completed tasks

A credible comparison needs more than one synthetic prompt. Use at least three workload classes:

Keep the model revision, quantization, tokenizer, sampling parameters, prompt distribution, hardware, and runtime versions identical. Pin the scheduler and cache settings where possible, record the warm-up period, and exclude warm-up traffic from the reported results.

The harness should trace every turn, not just one request. The following is implementation-neutral pseudocode; response timing fields differ across vLLM, TensorRT-LLM, and client libraries.

task_start = clock()

for turn in agent_task:
    request_start = clock()

    trace.mark("request_queued")
    response = model_call(turn.messages)

    trace.record(
        ttft = timestamp_first_token - request_start,
        prefill = timestamp_first_token - timestamp_prefill_start,
        decode = timestamp_complete - timestamp_first_token,
        reused_tokens = response.cache.reused_tokens,
        prefix_cache_hit = response.cache.prefix_hit
    )

    if response.has_tool_call:
        tool_start = clock()
        result = run_tool(response.tool_call)
        trace.record(tool_time = clock() - tool_start)
        turn.messages.append(result)

task_time = clock() - task_start
trace.record(task_time, orchestration_time, network_time)

For long-context tests, report both prefix-cache hit rate and reused-token count. They answer different questions:

• Prefix-cache hit rate tells you how often a request found a reusable prefix.
• Reused-token count tells you how much work was actually avoided.

A request can register a cache hit while reusing only a short system prompt. Another can reuse tens of thousands of conversation tokens. Treating those as equivalent produces misleading results.

The scorecard should include:

• TTFT at P50, P95, and P99
• Time per output token
• End-to-end task completion time
• Queueing, network, and orchestration time
• KV-cache hit rate, reused tokens, and eviction rate
• GPU memory per active request
• Task success and retry rates
• Cost per successfully completed task

Cost per Completed Task = (Model Cost + Tool Cost + Orchestration Cost + Retry Cost) / Successfully Completed Tasks

A system that is 15% cheaper per token but triggers more retries may be the expensive option in production.

How to interpret a benchmark win

Not every faster model call produces a meaningful product improvement.

Suppose TensorRT-LLM improves decode rate by 5% but increases TTFT by 10%. For a long batch completion, that may still be a reasonable trade. For an agent making six short calls, the TTFT regression can dominate the entire task.

Conversely, a 10% TTFT improvement can outweigh a 5% lower decode rate when responses are short and tool calls follow each turn. The right question is not “Which engine is faster?” It is “Which change reduces P95 completed-task time without hurting task success?”

Use the workload’s actual token mix to make that judgment. A useful result should show:

1. The absolute reduction in P95 task time.
2. Whether the improvement holds as concurrency rises.
3. Whether cache reuse or lower queueing caused the gain.
4. The effect on GPU cost and completed-task cost.
5. Any change in retries, failures, or output quality.

A 50-millisecond TTFT improvement may matter greatly in a six-turn interactive agent and barely register in a single long completion. Results should be judged against the task SLO, not an arbitrary percentage.

What public hardware benchmarks can tell you

Public MLPerf results are useful for understanding scale, but they are not direct vLLM-versus-TensorRT-LLM agent benchmarks. They commonly use different hardware counts, model configurations, batch policies, and optimization stacks. The official MLCommons results are available through the MLPerf Inference Datacenter benchmark.

For example, NVIDIA’s published results list the following throughput figures:

See NVIDIA’s MLPerf Inference results and benchmark information for the reported configurations and submission details.

The same-size GPT-OSS comparison suggests roughly 1.19× higher throughput for GB300. The DeepSeek-R1 figures compare different GPU counts, so they are not a per-GPU improvement. Neither result tells you the P95 TTFT of a 32-concurrent agent making sequential tool calls. Use these figures as hardware scale markers, then run a workload that resembles your service.

Find the bottleneck before changing engines

Three bottlenecks appear repeatedly in agent deployments.

Repeated prefill. System instructions, tool definitions, policies, repository context, and prior messages are often sent on every turn. If the serving layer recomputes that material, long-context agents pay the same cost repeatedly. Measure reused tokens and prefill time before assuming you need more decode throughput. Prompt structure and cache policy may deliver a larger gain than a new GPU.

Tool and orchestration overhead. JSON validation, authentication, database calls, browser sessions, sandbox startup, retries, and network hops can leave the GPU idle. If inference accounts for only 30% of task duration, even a theoretical 2× faster model can improve total task time by no more than about 15%, assuming everything else remains fixed. Run independent tools concurrently where dependencies allow.

Tail latency. Average latency hides queueing. At low concurrency, both engines may look excellent. At 128 active agents, the difference between a 600-millisecond P50 and a 2.5-second P95 is the user experience. Test concurrency as a curve, and keep dynamic batching, scheduler settings, and cache limits visible in the report.

python benchmark_agents.py \
  --engine vllm \
  --model your-model \
  --concurrency 1,8,32,128 \
  --turns 6 \
  --prompt-tokens 8192 \
  --output-tokens 128 \
  --report p50,p95,p99,ttft,task_time

Run the same workload with TensorRT-LLM. If the benchmark can't explain why one engine wins—less queueing, faster prefill, better reuse, or faster decode—it isn't yet useful for an architecture decision.

Which stack should you choose?

Choose vLLM when model variety, portability, and rapid iteration matter most. It makes a strong baseline for teams still learning their traffic shape or supporting several open models.

Choose TensorRT-LLM when you run NVIDIA hardware, have a stable model, and can justify quantization, kernel tuning, and release management. It is most compelling when measurements show that model execution—not tools, queues, or orchestration—is the dominant cost.

A sensible rollout has three steps:

1. Establish a portable baseline and capture P95 completed-task latency.
2. Optimize the measured bottleneck: prefix caching, batching, quantization, speculative decoding, tool parallelism, or memory capacity.
3. Move to a more specialized stack only when expected savings exceed migration and maintenance costs.

The best inference engine for a multi-turn agent isn't a permanent label. It's the stack that meets the task-level SLO at the lowest operational cost.

Frequently Asked Questions

Is vLLM or TensorRT-LLM better for agents?

Neither is universally better. vLLM is usually the easier baseline for broad model support and fast iteration. TensorRT-LLM can win on stable NVIDIA deployments after hardware-specific tuning. Compare P95 task time, TTFT, cache reuse, and cost per completed task.

What should a vLLM versus TensorRT-LLM benchmark measure?

Measure TTFT, prefill, decode, queueing, P50/P95/P99 latency, cache reuse, tool time, task success, retries, and completed-task cost. Use identical models, prompts, concurrency, sampling settings, hardware, and runtime versions.

Why is my agent slow when tokens per second is high?

Agents often generate short responses and spend more time waiting on queues, tools, network calls, orchestration, or repeated context prefill. Aggregate decode throughput can hide poor TTFT and tail latency.

How important is KV-cache reuse?

It can be decisive for long prompts and repeated tool schemas. Report both the percentage of requests with a prefix hit and the number of tokens reused. The latter shows how much prefill work the cache actually removed.