Private Coding Agent Benchmark 2026: Cost & Review Time

Public coding-agent scores help narrow a shortlist, but they’re weak procurement evidence on their own. A private benchmark built from your repository—starting with roughly 30 representative tasks—can show what matters in production: accepted merges, review effort, recovery behavior, regressions, latency, and cost per successful fix.

Key Takeaways

• Public scores are screening tools: SWE-bench Pro and DevBench provide useful capability signals, but task quality, test coverage, and evaluation harnesses can distort rankings.
• The system matters more than the model: Search, context selection, tool access, test execution, and recovery logic can change results as much as the underlying model.
• Start with 30 internal tasks: Treat the sample as a screening benchmark, not a statistically reliable estimate. Measure merge-ready patches, human correction, review burden, regression escapes, retries, and total cost.

Why Public Benchmark Scores Don’t Transfer Cleanly

SWE-bench made coding-agent evaluation more relevant by asking whether an agent can resolve real repository issues rather than merely generate plausible code. It still doesn’t answer the question engineering leaders need answered:

Can this agent safely fix issues in our repository at an acceptable cost?

Public benchmarks provide a standardized issue, repository snapshot, test environment, and evaluation procedure. Production repositories bring different constraints:

• Internal APIs and undocumented conventions
• Legacy build systems and monorepo boundaries
• Flaky or incomplete tests
• Security and permission controls
• Domain-specific correctness requirements
• Review standards that aren’t captured by automated tests

SWE-bench Pro also isn’t a single, unquestionable source of truth. Published audits found fundamental problems in roughly 30% of tasks, while human review identified low-coverage issues in 9.4% of cases. Agent-based review detected only 4.1%, which is a useful warning about relying on automated quality checks alone.

The practical implication is simple: public scores are good for identifying candidates, not for predicting a safe rollout. A 50% result isn’t necessarily twice as meaningful as 25% if the task sets differ in ambiguity, test quality, or acceptable solution paths.

What public benchmarks measure

DevBench offers a useful comparison point, though its figures should be read as results from the cited evaluation rather than universal performance guarantees.

A 43.5% Pass@1 result means the strongest reported system failed more than half the tasks on its first attempt. Pass@1 says little about whether an agent can interpret a failing test, revise an incorrect patch, or recover after misunderstanding the issue.

That recovery behavior often matters more in day-to-day engineering than one-shot accuracy.

Benchmark the Complete Agent System

A coding agent is a runtime, not just a model pointed at a ticket. Its behavior depends on repository search, context selection, shell and editor tools, test execution, failure interpretation, and the logic governing retries.

Issue or ticket
      ↓
Repository discovery and search
      ↓
Context selection and planning
      ↓
Shell, editor, test, and package-manager calls
      ↓
Code modification
      ↓
Test execution and failure interpretation
      ↓
Revision and recovery loop
      ↓
Patch validation and human review

The meaningful comparison is therefore:

Model + prompt + tools + context + runtime + validation

Record the environment for every run:

• Model name and version
• Agent harness and configuration
• System instructions
• Repository commit
• Available tools and permissions
• Time, token, and tool-call budgets
• Test commands
• Recovery-attempt limit
• Input and output token usage
• Human review time
• Final patch disposition

A reproducible task runner might look like this:

git worktree add ../agent-task-014 task-014-base
cd ../agent-task-014

export TASK_ID=014
export TEST_COMMAND="pytest tests/payments/test_refunds.py -q"

agent run \
  --repo . \
  --issue tasks/014.md \
  --max-turns 12 \
  --timeout 1800 \
  --test "$TEST_COMMAND"

The exact command will vary. Isolate each task’s repository state, preserve the original issue, and enforce identical limits for every candidate.

Build a 30-Task Private Benchmark

Thirty carefully selected tasks can expose major differences between agents, but the result is a screening sample, not a statistically reliable estimate of long-term production performance. Expand it later if the shortlist is close or the decision carries high risk.

Historical issues are usually the best source. They have real acceptance criteria and reflect work your team actually handles.

Preserve the original commit and issue wording. If the original work depended on logs, a runbook, or a design note, decide whether that material belongs in the benchmark and provide it consistently to every agent.

Run each task in two modes:

• First attempt only
• Up to three bounded recovery iterations
• Fixed wall-clock and token or tool-call budgets
• No human hints during execution

This separates initial diagnosis from recovery skill. Be careful with flaky tests: a transient failure can make a successful recovery look like an agent failure, while a rerun can make a weak patch appear repaired. Classify infrastructure and test flakiness separately, and report how often retries were caused by the harness rather than the agent.

Measure Merge Value, Not Just Test Passes

A patch can pass visible tests and still be a poor pull request. It may alter unrelated files, miss an edge case, weaken validation, or introduce a security problem.

Accepted-merge rate

Define the denominator clearly:

Accepted-merge rate =
Tasks ultimately accepted after the allowed agent attempts
÷
All tasks started

Count a task once, whether it succeeds on the first attempt or after recovery. This makes the metric useful for workflow decisions. Report first-attempt success separately so recovery isn’t hidden.

“Accepted” should mean merge-ready without substantive human code changes. Formatting, generated files, or a minor comment correction may be exempt; logic changes, missing tests, and reviewer-directed rewrites are not. Set that rule before scoring begins.

Validation and regression rate

Run existing tests plus additional checks where possible:

• Hidden or newly written tests
• Integration tests
• Security scanning
• Backward-compatibility checks
• Performance thresholds
• Production-like fixtures

Track patches that pass the visible suite but fail later validation. A green test result is evidence, not proof.

Human correction and review burden

Human correction ratio =
Lines materially changed by reviewers
÷
Lines changed by the agent

This is useful, but it can mislead. A two-line semantic fix may correct a serious defect in a 200-line patch, while a large mechanical rewrite may need little review. Pair the ratio with reviewer severity:

• No changes
• Cosmetic or formatting changes
• Minor logic correction
• Major rework
• Rejected or abandoned patch

Also record files touched, review rounds, requested revisions, and review minutes.

Recovery rate

Recovery rate =
Failed first attempts later repaired by the agent
÷
Initially failed first attempts

State whether test-harness failures are excluded. Otherwise, flaky tests and broken environments can distort the denominator and make recovery rates nearly meaningless.

Latency and cost

Review burden can reverse the apparent winner:

Agent A looks stronger on raw success. If its patches are broad and difficult to review, Agent B may have lower operating cost and less developer frustration. Record median and p95 completion time, not just averages.

Retries also change cost denominators. If you report “cost per task” using every attempt, a system that retries aggressively can look expensive even when it produces more accepted merges. The primary business measure should be cost per accepted merge, with retry count and cost per attempt reported separately.

Cost Formula: Cost per successful fix = (Model cost + compute + sandbox + retries + human review + remediation) ÷ accepted merges

For example, if 30 tasks consume $84 in model and infrastructure costs, 16 are accepted, and review costs $18 per accepted patch:

($84 + (16 × $18)) ÷ 16 = $23.25 per accepted merge

Include remediation when a failed patch causes rollback work, production debugging, or other measurable expense.

A Practical Evaluation Sequence

Public benchmarks still have real value. Use them to eliminate clearly weak candidates and understand broad capability differences. Then run your own evaluation:

1. Shortlist agents using public benchmark results.
2. Freeze a 30-task private repository benchmark.
3. Run first-attempt and bounded-recovery tests.
4. Score accepted merges, review severity, regressions, latency, and cost.
5. Check security and permission boundaries.
6. Pilot the leading candidate on low-risk production work.
7. Re-measure after real usage.

Set acceptance gates in advance:

• No critical security findings or secret exposure
• No unauthorized network or filesystem access
• Required tests pass
• No unacceptable performance regression
• Patch scope stays controlled
• Required documentation and migration work is included
• A human reviewer considers the change mergeable

Audit the benchmark, too. Tasks need clear acceptance criteria, adequate tests, balanced difficulty, and room for valid solutions beyond the reference patch.

The best choice is rarely the agent with the highest SWE-bench score. It’s the system with the lowest cost per accepted merge while meeting your security, regression, latency, and reviewability thresholds. That may mean one tool for IDE suggestions, another for automated pull requests, and a more capable system for complex incident repair.

Frequently Asked Questions

Does SWE-bench predict production coding-agent performance?

Only partially. It measures repository issue resolution under a standardized setup, not your tests, conventions, domain rules, security controls, or review process. Use it for screening, then validate candidates on private repository tasks.

Are 30 tasks enough?

Thirty tasks are enough for an initial screening benchmark if they cover the work your team actually does. They aren’t enough for a statistically reliable estimate of production performance. Expand to 50–100 tasks once the harness and scoring rules are stable or when candidates are closely matched.

What metric matters most?

Use accepted-merge rate as the primary outcome, counting each task once after the allowed recovery attempts. Pair it with first-attempt success, review severity, regression escapes, recovery rate, latency, and cost per accepted merge. Test pass rate alone can reward narrow patches that satisfy visible tests without meeting the real requirement.