vLLM vs TensorRT-LLM: 2026 Production Cost and Latency Benchmark

Is a 10% throughput gain worth changing your serving stack, adding NVIDIA-specific tuning, or carrying another set of upgrade constraints? Sometimes it is. Often, the answer depends less on the framework than on whether the deployment meets its latency SLO with the fewest GPUs.

That is the useful way to read a vLLM vs TensorRT-LLM benchmark. TensorRT-LLM led a March 2026 test of Llama 3.3 70B Instruct in FP8, while vLLM won a separate high-concurrency test of Llama 3.1 8B Instruct in BF16. Neither result makes one server universally faster.

Key takeaways

• TensorRT-LLM is a strong candidate for fixed, busy, NVIDIA-only deployments.
• vLLM is usually the easier starting point for mixed models, changing workloads, and flexible infrastructure.
• Compare p95 and p99 latency, queueing, and effective fleet cost—not just peak output tokens per second.

The short answer

For most teams, start with vLLM. It supports a broad range of models, provides a familiar OpenAI-compatible API, and typically requires less NVIDIA-specific optimization to get a changing workload into production.

TensorRT-LLM deserves a serious evaluation when the model is stable, the fleet is NVIDIA-only, and a 5–15% capacity improvement could remove a GPU or two. Its in-flight batching, quantization support, KV-cache management, CUDA graphs, speculative decoding, and multi-GPU execution can pay off at sustained utilization.

The practical test is simple: does the server meet the required p99 latency at the target traffic level? A service producing 5,000 output tokens per second with unacceptable tail latency is not cheaper than one producing 4,500 while meeting its SLO.

The deployment gap has also narrowed. TensorRT-LLM can load Hugging Face checkpoints and supports a PyTorch-backed serving path through trtllm-serve; it no longer always requires a complicated offline engine-conversion workflow. Still, pin the framework, container, CUDA version, model revision, and quantization implementation you actually test.

The current release references in the supplied comparison are vLLM v0.29.0 and TensorRT-LLM v1.3.0rc26, both dated September 9, 2026. Treat the TensorRT-LLM release candidate as a version under evaluation, not an automatic production recommendation. Check the vLLM releases and TensorRT-LLM releases before reproducing results.

What the H100 benchmarks show

A March 2026 comparison tested one NVIDIA H100 SXM5 with 80 GB of memory using Llama 3.3 70B Instruct in FP8. The reported workload used an average 512-token prompt and 256-token completion, with concurrency from 1 to 100 after a 60-second warm-up.

TensorRT-LLM led by approximately 8% at concurrency 1 and 16% at concurrency 100. It also had lower reported TTFT at every level.

The test summary identifies the GPU, topology, model, precision, prompt and completion sizes, warm-up period, and concurrency. It does not publish enough detail to reproduce the result exactly: the serving flags, CUDA and container versions, quantization calibration or kernel implementation, queueing time, p99 TTFT, and output-token latency are not reported. The 70B run is described as a single 80 GB H100 test, not a multi-GPU deployment.

A separate RunPod/RunInfra comparison tested Llama 3.1 8B Instruct in BF16 on H100 at concurrency 256. It used unique prompts and disabled prefix caching:

Here, vLLM delivered 11% higher output throughput and 22% lower TTFT p50.

The results are not contradictory. The tests changed model size, model version, precision, concurrency, prompt distribution, and cache policy. A small BF16 model under extreme concurrency stresses scheduling and request management differently from a large FP8 model operating close to memory and compute limits.

For interactive services, TTFT needs more scrutiny than a single throughput number. TTFT includes queueing, scheduling, and prompt processing. Output throughput mostly describes decode after requests are already active. Aggressive batching can improve utilization while making a short request wait behind a long prefill.

For chat and agent traffic, measure TTFT p95 and p99, time per output token, and end-to-end latency. For offline generation, sustained output tokens per second may be the better primary metric.

VRAM efficiency is mostly configuration

Weight memory is only the starting point:

The estimate excludes the KV cache, activation buffers, CUDA graph allocations, quantization scales, temporary workspace, communication buffers, and any draft model used for speculative decoding.

A 70B BF16 model needs multiple H100s. A 70B FP8 model may fit on one 80 GB H100, but only with careful limits on context length, active sequences, and runtime memory. Fitting at startup does not guarantee stable service under burst traffic.

Both frameworks use block-based KV-cache management. vLLM’s PagedAttention reduces fragmentation by storing cache blocks instead of reserving one large contiguous region per request. TensorRT-LLM supports dynamic block assignment, prefix matching and reuse, prioritized eviction, secondary-memory offload, and attention variants such as MHA, MQA, and GQA.

The useful questions are operational:

• How many active sequences fit before requests queue?
• What happens to p99 latency near cache capacity?
• How much memory is reserved at startup?
• Does prefix caching match real prompt repetition?
• How do long and short requests interact?
• How much memory goes to CUDA graphs and temporary buffers?

The 70B FP8 comparison measured roughly 71 GB for vLLM and 74 GB for TensorRT-LLM at idle, with the gap narrowing to about 1 GB at high concurrency. That is too close to establish a permanent VRAM winner.

Changing BF16 to FP8, reducing max-model-len from 32K to 8K, or lowering the active sequence limit can have a greater effect than changing frameworks. PagedAttention primarily improves memory management; it does not automatically reduce TTFT or p99 latency.

Cost per million tokens

Using AWS’s supplied H100 reference price of $5.191 per GPU-hour and the RunInfra throughput figures:

Raw cost per 1M output tokens: GPU hourly price ÷ output tokens per second × 1,000,000 ÷ 3,600

These are theoretical saturated costs for output tokens only. They exclude idle capacity, replica count, orchestration, networking, storage, retries, engineering time, and traffic headroom.

A more useful calculation is:

Effective cost per 1M output tokens: number of GPUs × hourly GPU price ÷ sustained throughput at the required SLO × 1,000,000 ÷ 3,600

Replica utilization matters. A two-replica service running at 40% utilization can cost more per token than a single saturated benchmark instance, even if its raw throughput looks attractive. Headroom matters too: a fleet sized to its absolute maximum cannot absorb bursts without queueing or an emergency scale-out.

Benchmark both servers at the actual production SLO. Record TTFT p50, p95, and p99; time per output token; end-to-end latency; queueing delay; steady-state and peak VRAM; KV-cache occupancy; prefix-cache hits and evictions; preemptions; out-of-memory events; and cold-start time.

Use the official tools as a starting point: vllm bench serve and trtllm-bench. Keep the model revision, tokenizer, CUDA version, container image, quantization format, sampling settings, warm-up, and cache state identical. Test low concurrency, normal load, saturation, bursty arrivals, long prompts with short outputs, short prompts with long outputs, repeated system prompts, and unique prompts.

Which server should you choose?

Choose vLLM first when models change frequently, hardware flexibility matters, traffic is bursty, or operational simplicity is worth more than a possible capacity gain.

Evaluate TensorRT-LLM when the model and hardware are fixed, utilization is consistently high, and a measured improvement can reduce the GPU fleet. For long-context models, tune precision and KV-cache limits before assuming a framework change will solve memory pressure.

Make p99 latency the deciding metric for interactive workloads. Use throughput and effective cost per output token for batch workloads. In both cases, run the comparison on your traffic rather than treating a vendor or third-party headline number as a deployment plan.

Frequently Asked Questions

Is TensorRT-LLM faster than vLLM on H100?

Not universally. TensorRT-LLM led the 70B FP8 test by 8–16%, while vLLM led the 8B BF16 test at concurrency 256 by 11% in throughput and 22% in TTFT p50. Model size, precision, prompt length, cache policy, and scheduler settings determine the result.

Which uses less VRAM, vLLM or TensorRT-LLM?

Neither has a consistent advantage. In the 70B FP8 H100 comparison, peak memory was within roughly 4 GB. Precision, context length, active sequences, KV-cache type, CUDA graphs, and temporary buffers often matter more.

What is the vLLM vs TensorRT-LLM cost per million tokens?

In the 8B BF16 H100 test, using a $5.191 hourly H100 price, vLLM cost about $0.27 and TensorRT-LLM about $0.30 per million output tokens. Production cost must include replica utilization, headroom, and the throughput that satisfies the latency SLO.

Which is the best inference server for H100 production?

Use vLLM as the default for flexible or changing workloads. Evaluate TensorRT-LLM for a fixed NVIDIA-only deployment where sustained utilization and GPU savings justify additional tuning and version management.