Terminal-Bench 2.0 Reproducibility: Runtime Benchmark 2026

Two identical coding agents receive different scores. One finishes installing dependencies; the other times out halfway through because its container has a throttled CPU and a strict network policy.

Nothing about the model changed. The benchmark result did.

That scenario is central to reproducing Terminal-Bench 2.0. The benchmark evaluates an agent operating through a particular harness, inside a particular container, with specific limits on CPU, memory, network access, time, and retries. Anthropic reported that infrastructure configuration alone could shift performance by roughly six percentage points. Across the benchmark’s 89 tasks, that represents about five outcomes—enough to overturn a close ranking.

Key takeaways

• Terminal-Bench measures a complete agent system, not a language model in isolation.
• CPU limits, dependency installation, timeouts, network access, and harness behavior can materially change scores.
• Teams should compare first-attempt success, eventual success, runtime, and cost per accepted task—not just leaderboard percentages.

The Hidden Variable Is the Execution Environment

Terminal-Bench tasks look more like real terminal work than ordinary code-generation prompts. An agent may need to explore a repository, install packages, edit files, run commands, interpret failures, and try another approach.

A typical run passes through several layers:

1. The model emits a command or tool call.
2. The harness validates and executes it.
3. The container provides CPU, memory, disk, and network access.
4. Package managers resolve and install dependencies.
5. Tests and subprocesses run under a timeout.
6. The agent inspects the result and decides whether to retry.
7. The evaluator determines whether the task passes.

A failure at any point can appear as a model failure if the benchmark records only a binary result.

Imagine an agent correctly launching a package build, but the container has one throttled CPU. The build exceeds the ten-minute limit, the process is killed, and the task is marked failed. The score doesn’t reveal whether the model misunderstood the task or simply ran out of time.

The result is better understood as:

Model + harness + container + resource policy + timeout + retry policy

The benchmark’s reported infrastructure effect is therefore significant, not cosmetic. A model that leads another by four percentage points may lose that lead when both are run with identical resources and execution rules.

What can change the outcome?

CPU and RAM matter because terminal tasks involve ordinary engineering workloads. Agents compile native packages, index repositories, start services, launch subprocesses, and run test suites. A model can issue the right command and still lose because a child process is killed, a dependency build exhausts memory, or a test suite is throttled into a timeout.

The Terminal-Bench repository documents the benchmark’s task set and execution framework; the infrastructure-sensitivity result is discussed in Anthropic’s published analysis of agent evaluations. See Terminal-Bench and Anthropic’s engineering-agent research for the surrounding methodology.

Terminal-Bench and SWE-bench Test Different Skills

It’s tempting to treat Terminal-Bench and SWE-bench as competing rankings. They measure different slices of coding-agent capability.

SWE-bench asks an agent to modify an existing repository so associated tests pass. That’s valuable for bug fixes and maintenance. It says less about whether an agent can recover from a broken package installation, configure a command-line tool, start a service, or diagnose an environment problem.

Terminal-Bench brings those operational concerns into the evaluation. Its score reflects patch quality, but also command-line fluency, package management, subprocess handling, and recovery after failure.

The reverse is also true. A team building a repository-maintenance agent may learn more from SWE-bench than from a collection of environment-heavy terminal tasks. A platform team automating builds and debugging workflows may find Terminal-Bench more representative.

Test validity complicates the comparison. OpenAI’s audit of SWE-bench Verified reported that at least 59.4% of an audited subset of frequently failed tasks had flawed tests. A failing test doesn’t always mean the patch is wrong, and a passing test doesn’t prove that a change is secure, maintainable, or complete. See OpenAI’s SWE-bench Verified audit.

Public benchmarks also create possible exposure to training data. Issues, patches, and task descriptions may have appeared in model-training corpora. That doesn’t make the benchmarks useless, but it weakens the case for using a public score alone to select a production system. Private tasks should be part of any serious procurement or rollout decision.

Build a Reproducible Harness

Before switching models, run the same model under controlled infrastructure profiles. Two profiles are usually enough to expose whether a result is resource-sensitive:

The exact values depend on the workload. What matters is documenting them and holding them constant.

A container launch might look like this:

docker run --rm \
  --cpus=4 \
  --memory=8g \
  --pids-limit=512 \
  --network=none \
  -v "$PWD/task:/workspace" \
  agent-benchmark:2026-01

The --network=none setting improves isolation, but it can invalidate tasks that need to download packages, reach a package mirror, or communicate with an external service. If you use it, either vendor all dependencies into the image or label network-dependent tasks as unsupported. Otherwise, the benchmark is measuring the network policy more than the agent.

Record the image digest, not just the tag. agent-benchmark:latest is not a reproducibility strategy. Each run should capture:

• model and API version;
• harness commit, system prompt, and tool definitions;
• container image digest;
• CPU, memory, disk, and network policy;
• timeout and retry settings;
• token usage and wall-clock duration;
• exit code, test output, and final patch;
• failure category.

Classify failures as model, harness, or infrastructure. Model failures include misunderstanding the requirement, making an incorrect diagnosis, or repeating a failed strategy despite useful feedback. Harness failures include malformed tool calls, truncated output, lost state, broken retries, or premature termination. Infrastructure failures include out-of-memory kills, unavailable package mirrors, CPU-induced timeouts, and services that cannot start.

The categories aren’t academic. Without them, a team may fine-tune a model when the real fix is adding RAM, extending a timeout, or repairing subprocess handling.

A Worked Comparison

The following is an illustrative report for one model run against the same 50-task private set:

The model did not become better in the production profile. It encountered fewer timeouts and dependency failures, used fewer retries, and finished sooner. A leaderboard that reported only eventual success would show an eight-point difference; a deployment decision also needs the runtime and cost data.

Track first-attempt and eventual success separately. A system that succeeds on its fourth try receives the same binary pass as one that succeeds immediately, but those systems have very different operational costs and supervision requirements.

Measure Cost per Accepted Task

Raw accuracy is a useful headline metric, but it rarely pays the engineering bill.

Cost Formula: Cost per accepted task = (model cost + compute cost + retry cost + review cost) / successfully accepted tasks

“Accepted” matters. A patch that passes a public test but requires a developer to rewrite half the diff shouldn’t count as an unqualified success.

A useful scorecard includes:

• first-attempt success;
• eventual success;
• infrastructure-failure rate;
• median and P95 completion time;
• average retries;
• token consumption;
• human acceptance rate;
• regression and security findings;
• cost per accepted task.

Pricing surprises often appear here. A cheaper model may consume more tokens because it explores aimlessly. A more capable model may cost more per call but finish in one pass. A highly ranked system may also produce larger diffs that take longer to review.

For an internal evaluation, use 20 to 50 representative private tasks. Include bug fixes, dependency upgrades, test writing, API changes, debugging, documentation, command-line operations, and a few greenfield tasks. Mix small and large repositories, and keep some awkward cases that public benchmarks may not capture.

Run repeated trials when the model or harness is stochastic. A one- or two-point difference on a small task set usually isn’t a meaningful capability claim. Report the task count, successful runs, and uncertainty around the result.

A useful report might read:

Agent A completed 34 of 50 tasks on the first attempt and 39 eventually, at $1.82 per accepted task. Agent B completed 31 initially and 40 eventually, at $2.47 per accepted task, with twice the median runtime.

That gives an engineering leader more decision-making information than “Agent B scored 80%.”

Choose Terminal-Bench when shell interaction, environment setup, and recovery are central to the product. Use SWE-bench when repository issue resolution is the target. Use private tasks when the decision involves real money, real code, or a production rollout.

A benchmark records what happened. A reproducible harness helps explain why.

Frequently Asked Questions

Does CPU affect Terminal-Bench scores?

Yes. CPU count and throttling affect compilation, dependency installation, indexing, subprocesses, and test execution. A timeout caused by constrained CPU may be recorded as a model failure even when the command was correct.

How much can infrastructure change results?

Anthropic reported an approximately six-percentage-point performance shift from infrastructure configuration in Terminal-Bench 2.0. Across 89 tasks, that is roughly five task outcomes and can reverse close rankings.

Is SWE-bench more reliable than Terminal-Bench?

Neither is universally more reliable. SWE-bench focuses on repository issue resolution but has documented test-quality concerns. Terminal-Bench captures terminal and environment interaction but is especially sensitive to resource and harness configuration.

How should teams benchmark coding agents fairly?

Freeze the model, harness, container image, CPU, RAM, network policy, timeout, retry policy, and evaluator. Classify every failure, repeat stochastic trials, and compare cost per accepted task alongside pass rate.