AWS Trainium3 vs NVIDIA: 2026 Inference Cost Benchmark Data

Trainium3 could lower production LLM inference costs, but a chip’s token price won’t tell you whether migration will pay off. The useful comparison includes Neuron porting, KV-cache performance, p95 latency, retries, utilization, and the engineering cost of changing platforms.

This article presents a benchmark methodology and decision framework. It does not report a completed production benchmark with measured Trainium3-versus-H100 results. The worked numbers below are illustrative so teams can see how to calculate cost per successful task.

Key takeaways

• AWS-reported Trainium3 specifications look attractive for memory-heavy inference, but they aren’t a substitute for workload measurements.
• Unsupported Neuron operators, custom CUDA kernels, quantization changes, and serving-framework differences can erase hardware savings.
• H100 is a more useful comparison than combining H100, H200, Blackwell, and Rubin into one broad NVIDIA category.
• The go/no-go metric should be cost per successful task at an acceptable quality level and p95 latency—not cost per million tokens alone.

What the published specifications tell you

AWS lists Trainium3 with 144 GB of HBM3e, 4.9 TB/s of HBM bandwidth, and up to 2.52 PFLOPs of FP8 compute per chip. AWS also describes a 144-chip Trainium3 UltraServer with 362 PFLOPs of FP8 performance, 20.7 TB of aggregate HBM, and 706 TB/s of aggregate memory bandwidth.

These are announced specifications, not independent benchmark results. The figures come from AWS product and architecture material, including its Trainium product page and Trainium3 announcement. AWS also claims substantially better performance per watt than Trainium2 and higher output-token efficiency on Amazon Bedrock. Those claims describe selected AWS test conditions; they don’t establish how your model will perform.

For a procurement comparison, H100 is a sensible baseline. It is widely available, has mature CUDA and TensorRT-LLM support, and is still common in production fleets. Using one baseline avoids a misleading comparison in which a new Trainium3 system is weighed against several NVIDIA generations with different memory capacities, software stacks, and availability.

NVIDIA lists the H100 SXM with 80 GB of HBM3 and 3.35 TB/s of memory bandwidth. Its official H100 specifications are also vendor-published specifications, not a guarantee of application performance.

The practical question is narrower than “Which chip is faster?”

Can Trainium3 run this model, at this quality and p95 latency, with a lower cost per completed task than an H100 deployment we can actually provision?

Answering that requires a controlled test.

A benchmark design that reflects production

For a chatbot, cost per million tokens is a useful first pass. It’s insufficient for coding agents, document extraction, and tool-using workflows. A malformed JSON response can trigger another request. A tool timeout can repeat an entire reasoning loop. A latency spike can push a workflow past its deadline even when average throughput looks good.

Use a test matrix that includes:

Run the same model revision, tokenizer, prompts, sampling parameters, stop conditions, and output validators on both platforms. Keep speculative decoding disabled for the first comparison. Measure it separately afterward; otherwise, a runtime optimization can obscure the hardware and compiler differences you’re trying to understand.

Report concurrency and context length with every throughput number. “Tokens per second” without those values is nearly meaningless. A large batch may produce impressive throughput while missing an interactive p95 target. Low-concurrency latency may look excellent while the system becomes uneconomical at production load.

A useful record for each run looks like this:

{
  "platform": "Trainium3",
  "model": "target-model-revision",
  "precision": "FP8",
  "input_tokens": 8192,
  "output_tokens": 512,
  "concurrency": 128,
  "ttft_ms": 0,
  "p95_e2e_ms": 0,
  "tokens_per_second": 0,
  "retry_rate": 0,
  "task_success_rate": 0,
  "cost_per_successful_task": 0
}

Include the cost of failure

A production-oriented calculation should include retries and operational overhead:

Cost per successful task = (inference cost + orchestration cost + retry cost + remediation cost) / completed tasks

For most teams, remediation cost is hard to estimate precisely. Start with inference, orchestration, and retry cost, then add a sensitivity analysis for human review or workflow recovery.

Here’s an illustrative example, not a measured Trainium3 result:

• Instance price: $8 per hour
• Sustained useful throughput: 100,000 output tokens per second
• Utilization: 60%
• Average task: 2,000 output tokens
• Retry rate: 5%
• Orchestration overhead: $0.0004 per attempt

At 60% utilization, the instance produces 216 million useful output tokens per hour. The raw compute cost is:

$8 / 216,000,000 = $0.000000037 per output token

A two-thousand-token task therefore costs about $0.000074 in inference. With a 5% retry rate, the expected inference cost becomes $0.0000777. Add orchestration overhead for an expected cost of roughly $0.00050 per successful task, before migration amortization.

If migration and testing cost $300,000 and the service completes 100 million successful tasks per year, migration adds $0.003 per task in the first year. That is far larger than the raw inference charge in this simplified example. At lower volume, migration dominates even more. This is why an apparently cheaper accelerator may not be the cheaper platform.

Use actual instance or reserved-capacity prices for both platforms. Record the pricing model—on-demand, committed use, or reserved capacity—and include idle capacity, host overhead, and replica headroom.

Neuron migration is the main economic variable

A model can compile for Neuron and still perform poorly. Treat “supported” as “tested with this model revision, compiler version, context length, batch size, and serving configuration,” not as a checkbox in documentation.

Inspect the model for:

• custom attention or fused CUDA kernels;
• paged or persistent KV-cache implementations;
• mixture-of-experts routing;
• speculative decoding;
• custom normalization and sampling paths;
• framework extensions tied to vLLM, TensorRT-LLM, or proprietary runtimes.

Representative failure modes include an unsupported fused operation falling back to a slower implementation, graph compilation failing because of dynamic shapes, or quantization changing structured-output accuracy. A model may pass a basic generation test while producing more invalid JSON in production, increasing retries and reducing successful-task throughput.

Build a porting inventory before estimating savings:

Budget for compiler regressions, model upgrades, observability changes, deployment work, and rollback testing—not just the initial port.

A stable service running one well-supported architecture can spread those costs over a large request volume. A research platform changing models every few weeks may never recover them.

Measure KV-cache behavior and tail latency

Long-context inference often exposes differences that short prompts hide. The KV cache grows with sequence length and concurrency, and eviction or recomputation can damage both throughput and latency.

Measure:

• memory used per active sequence;
• maximum active sequences before eviction;
• cache hit rate for shared prefixes;
• TTFT as context grows;
• decode throughput under cache pressure;
• eviction and recomputation frequency;
• p95 and p99 latency at each concurrency level.

Test at 32K and 128K tokens if those lengths appear in production. A 2K-token run may show nearly identical systems; long contexts can produce a different ranking because of memory fragmentation, cache capacity, and attention implementation.

Separate prefill from decode. Retrieval-heavy document processing is often prefill-bound. Interactive chat and agents are more decode-sensitive and frequently pause for tools or external APIs. A platform optimization that improves prefill may have little effect on agent completion time.

The MLPerf Inference benchmark suite provides a useful controlled reference for published accelerator results. It still won’t capture your prompt distribution, prefix reuse, tool calls, output validators, or retry behavior. Treat it as context, not a replacement for private testing.

Set a migration gate before you port

Trainium3 deserves serious consideration when the workload is AWS-native, high volume, and built around a model with a mature Neuron path. It becomes less attractive when the service depends on custom CUDA code, changes models frequently, or requires multi-cloud portability.

Define the gate in advance:

1. No material quality regression on production prompts.
2. p95 and p99 latency remain within the service-level objective.
3. Retry and malformed-output rates do not increase.
4. Cost per successful task falls by a meaningful amount after migration amortization.
5. Required capacity exists in the target AWS region.
6. Rollback to the H100 path has been tested.
7. Future model upgrades don’t depend on unsupported Neuron operators.

The final calculation is straightforward:

Migration value = annual savings − porting cost − operational overhead − reliability penalty

The reliability penalty matters. If a cheaper platform increases failed tool calls or human review, the savings may disappear. Portability also has value: CUDA support gives teams more choices across vendors, cloud providers, frameworks, and contractors.

The right Trainium3 inference cost benchmark is therefore not a single throughput contest. It is a repeatable production test that measures quality, latency, utilization, retries, KV-cache behavior, and total cost per successful task. Only that result can show whether migration is cheaper for your workload.