Terminal-Bench Infrastructure Requirements: CPU, RAM & Timeouts

A coding agent can understand a task, write a sensible patch—and still fail because the dependency install consumed most of its timeout. By the time the environment is ready, there’s no time left to run the tests.

That’s the problem with comparing agent scores as if they were pure model scores. On terminal-based benchmarks, CPU, memory, disk, network access, package caches, and timeouts all affect the result. Anthropic’s analysis of Terminal-Bench infrastructure found roughly a six-percentage-point spread between resource configurations—not a 6% relative improvement—with infrastructure-related failures reaching up to 6% in some setups.

A four-point gap between two agents can disappear when the environment changes.

Why coding-agent benchmark scores vary

The simple model is:

Benchmark score ≈ model capability

That approximation works for a self-contained code-generation test. It breaks when an agent has to operate a terminal in a fresh environment.

Terminal-Bench 2.0 contains 89 manually curated terminal tasks. Each task has an environment, a human-written solution, and automated tests. The work may include inspecting a repository, installing dependencies, starting services, changing configuration, debugging a failed build, and verifying the final result.

The agent isn’t just generating code. It’s operating a small, temporary production system:

User request
   ↓
Agent planning and tool calls
   ↓
Repository inspection and setup
   ↓
Dependency installation
   ↓
Build, service startup, and tests
   ↓
Error interpretation
   ↓
Patch, retry, and final verification

A failure at any point can produce a failed task. The model may have chosen the right implementation, but the container may run out of memory during compilation. A package registry may be unreachable. A test suite may need 12 minutes when the benchmark allows 10. Two local services may compete for the same port.

Those failures should not all be reported as “the model got the task wrong.”

The distinction matters because published Terminal-Bench 2.0 results show frontier systems scoring below 65%. That’s a useful indication that reliable, long-horizon terminal work remains hard. It doesn’t reveal how much of an individual failure rate came from reasoning and how much came from execution conditions.

Anthropic’s reported six-point spread makes the issue concrete. If one agent scores 59% and another 55%, an infrastructure effect of approximately six percentage points is large enough to erase the apparent model advantage—or even change the ranking.

The practical answer is that these benchmarks measure a compound system. The model is one component; the harness, container, operating limits, package sources, and evaluation policy are the others.

Terminal-Bench infrastructure requirements

There is no universal hardware specification that makes Terminal-Bench fair. A lightweight coding sandbox and a production-like monorepo shouldn’t have identical limits. The important thing is to choose constraints that match the claim being tested, then apply them consistently.

At minimum, publish:

Model and version
Agent harness and commit
System prompt and reasoning setting
Maximum tool calls
Wall-clock timeout
CPU, memory, disk, and process limits
Container image digest and operating system
Network access policy
Package mirror and cache policy
Repository commit
Sampling settings and retry policy
Test command
Token usage and task cost

A container declaration might look like this:

docker run --rm \
  --cpus="4" \
  --memory="16g" \
  --memory-swap="16g" \
  --pids-limit=512 \
  --network=none \
  --tmpfs /tmp:rw,size=4g \
  terminal-bench-task@sha256:IMAGE_DIGEST

This is an example of a disclosed configuration, not a universal baseline. In particular, --network=none can make package installation impossible unless dependencies are preloaded in the image or supplied through a local cache. A network-isolated run and a run with access to a package registry are different evaluations.

Timeouts deserve the same scrutiny. A short limit penalizes large dependency installs, native compilation, slow test suites, multi-service projects, and agents that recover through several attempts. An unlimited timeout, meanwhile, rewards persistence without measuring operational efficiency.

Report more than a single success rate:

Package installation is a particularly common confounder. A warm cache can save several minutes and eliminate transient registry failures. A private mirror may be faster than the public registry. Neither choice is inherently wrong, but a cached run shouldn’t be compared with a cold, network-constrained run without disclosure.

Before evaluating a model, run a calibration phase with known-good scripts or human-authored reference solutions. Confirm that the environment can check out the intended repository, install dependencies, build the project, start required services, execute the test command, and write results where the harness expects them. If the reference solution cannot complete, the environment isn’t ready for model evaluation.

How to reproduce Terminal-Bench results

Reproducibility requires a pinned execution path, not just a pinned model name. A compact controller needs five practical pieces:

1. Task allocator — selects tasks and prevents accidental overlap.
2. Agent runner — records prompts, responses, tool calls, and sampling settings.
3. Container orchestrator — starts a fixed image digest with declared limits.
4. Telemetry and event logger — captures resource use, command output, exit codes, timing, retries, and network events.
5. Test collector and failure classifier — stores raw test results and separates reasoning, tooling, timeout, and infrastructure failures.

Run a representative subset at least three times under identical conditions. Then repeat it with more CPU, more memory, or a longer timeout. Task-level changes are more informative than the aggregate score.

This experiment tells you whether additional resources improve general capability or merely rescue a narrow class of operational failures.

Don’t silently retry failed tasks. If retries are allowed, publish the maximum number, delay between attempts, and whether the agent sees the previous failure output. Recovery-aware evaluation can be useful, but it’s a different benchmark.

The same applies to randomness. Fix sampling settings where possible, record seed behavior, and report task-level variance. One pass per task is weak evidence when tool calls, network state, and model sampling all add noise.

There’s also a separate validity problem: contamination. OpenAI has recommended moving away from SWE-bench Verified because of increasing contamination concerns. That is different from task quality. An audit of SWE-Bench Pro estimated that a substantial portion of tasks appeared broken, while SWE-Bench Pro Verified was created to improve task reliability.

Keep the two risks separate:

Contamination:
The model may have encountered the task or solution before.

Environment validity:
The task, repository, tests, or execution setup may be defective.

A credible evaluation needs controls for both.

Terminal-Bench vs. SWE-bench for real-world coding

Terminal-Bench and SWE-bench measure different kinds of work.

Terminal-Bench is a good fit when environment handling is part of the job. It exposes whether an agent can find the working directory, install dependencies, interpret compiler output, restart a service, and verify its own change.

SWE-bench-style tasks are closer to issue resolution in an existing repository. A passing patch is valuable, but it doesn’t necessarily show that an agent can work reliably in a clean, unfamiliar environment.

FeatureBench is a useful complement for teams evaluating product work rather than bug fixing. Other suites, including Aider Polyglot and DevBench, can add language or developer-assistance coverage, but they shouldn’t turn the report into a catalog of loosely comparable scores.

Public benchmarks are useful for broad comparisons. Private holdouts are better for deployment decisions because they reflect your build system, frameworks, conventions, and review standards.

Also track economics:

Cost per successful task = (model cost + infrastructure cost + review cost + retry cost) / completed tasks

A 60% agent that finishes quickly and produces clean patches may be more useful than a 65% agent that consumes three times the tokens and routinely needs human cleanup.

Frequently Asked Questions

What are the main Terminal-Bench infrastructure requirements?

Use consistent CPU, memory, disk, process limits, timeout, container image, operating system, network policy, package-cache behavior, and test command. There is no single correct allocation. The requirement is to publish the limits and keep them matched across agents.

Does Docker CPU affect coding-agent benchmarks?

Yes. CPU limits change build time, test duration, service startup, and timeout behavior. More CPU doesn’t improve reasoning, but it can prevent an otherwise correct task from failing operationally.

How should teams reproduce and validate results?

Pin the model, harness, repository commit, image digest, sampling settings, resource limits, and timeout. Log every tool call and test result, run calibration tasks first, and repeat noisy tasks. Then validate public results against private tasks drawn from the workflows your team actually runs.

The direct recommendation is simple: use matched, pinned environments; report infrastructure-sensitive metrics alongside success rate; and validate public benchmark results against private tasks before making adoption decisions.