A full SWE-bench Verified run: 394/500 resolved, with attribution limits

Question: What did the integrated model-and-agent system solve across all 500 SWE-bench Verified tasks, and which part of that result can be attributed to an individual component?

Observation: The integrated system resolved 394/500 under the official scoring code. A separate tool A/B put code-graph File Acc@5 4.9 points above grep, while the 27/50→38/50 comparison changed execution environment, candidate count, selection, and completion together.

Decision: Report 78.80% only as an integrated-system result. Make component claims only from component-level controls, and give stopped runs, missing artifacts, and benchmark validity equal prominence.

SWE-bench Verified reconstructs real GitHub issues at their historical repository states and scores generated patches against the official grader tests. The important result here is not one large number. It is a disciplined separation of system performance from causal attribution.

1. Full run: 394/500 belongs to the integrated system

In June 2026, all 500 tasks were run through the swebench 4.1.0 scoring path. The model was Claude Opus 4.8. The agent combined lexical, dense, and code-graph retrieval with Docker reproduction, patching, re-verification, five candidates, and execution-based selection.

Outcome	Tasks	Interpretation
Resolved	394 / 500 (78.80%)	Integrated model-and-scaffold result
Unresolved	106 / 500	Includes five empty patches
Unscored	0 / 500	No omissions in the final aggregate

Repository results were uneven: scikit-learn resolved 30/32 and xarray 20/22, while pylint resolved 4/10. Django contributes 231 tasks, or 46% of the suite, so the aggregate is strongly shaped by repository composition.

The attribution boundary is decisive. There is no Opus 4.8 scaffold-off control over the same 500 tasks. This run cannot divide the 394 successes between the model, retrieval, execution, or selection. I therefore do not call 78.80% a scaffold score or a scaffold improvement.

2. Component evidence: a search-tool-only A/B

The code graph was isolated in a separate localisation experiment. Claude Sonnet 4.6, prompt, agent loop, and task conditions were held fixed; only the search tool exposed to the agent changed between graph and grep. The outcome was not task resolution but File Acc@5: whether an edit-relevant file appeared in the first five results.

Arm	File Acc@5	Scope
grep tool	65.2%	SWE-bench Verified, n=465
code-graph tool	70.1%	Same run, n=465
Difference	+4.9pp	No retained CI or paired test

This is the cleanest component evidence, but not a complete benchmark run. The planned 500-task run stopped at 465, and only File Acc@5 was reconstructed from logs. The 465 raw pairs and @1, @10, and function-level measures were not retained. On the easier SWE-bench Lite the difference shrank to +1.1pp (n=272); on a non-random 102-item LocBench prefix the arms tied. Graph was below grep on the five-item LocBench Security slice, so “no regressions in any slice” is not supportable.

The defensible conclusion is narrow: in this harness, the graph tool improved Verified file localisation by 4.9 points, with small or absent gains on two other datasets. It does not establish the same gain for patch correctness or for code understanding in general.

3. 27/50→38/50 is not a selector-only ablation

Two retained artifacts cover the same 50 submitted task IDs and show the system progressing from 27 to 38 resolved tasks. Several variables changed together.

Retained artifact	Resolved	Completed / empty	Configuration changes
no-Docker graph agent	27/50	35 / 15	Near-single-generation path; no execution selection
Docker exec-select	38/50	50 / 0	Live Docker reproduction, best-of-five, cross-execution selection

The machine-readable files verify the same task IDs. They do not preserve model and prompt flags for both arms. Execution environment, candidate opportunity, selection, and completion rate all changed, so the 22-point difference cannot be attributed to the selector. It is a configuration progression, not a single-variable ablation.

That correction strengthens the record: it distinguishes evidence about the whole architecture, evidence about one component, and a comparison that has not yet been decomposed.

4. Checkpoint design made the 500-task run finish

The first full run was externally killed at task 476. Predictions were checkpointed per repository, and Django was one 231-task unit, so roughly 207 in-memory patches were lost. The aggregate report puts that at about 27 hours and $700 of compute.

The fix was failure recovery rather than model tuning:

• Move checkpoint granularity from repository to instance, persisting each patch immediately.
• Allow a single repository to be split across processes, reducing the replay boundary.
• Pre-pull scoring images and retry transient network failures, separating test execution from network availability.
• Re-run Django in about six hours with zero reported errors.

For long-running agent evaluation, checkpointing is part of measurement validity. If a failure selectively erases one repository, it can change the final estimate instead of merely delaying it.

5. Benchmark-validity boundary

SWE-bench Verified is no longer a stable terminal measure. In February 2026, OpenAI stopped publicly reporting Verified results after auditing 138 hard tasks, judging 59.4% unusable because of flawed tests or underspecified tasks, and finding contamination evidence across the frontier models it examined.

The 78.80% result is therefore a historical integrated-system result against the public suite and official scoring code as used in June 2026. It should not be read as current general software-engineering capability or human-level performance. SWE-bench’s contribution of turning repository issues into executable evaluation remains important; a saturated scalar score should not carry the product thesis.

Two decisions survive:

1. Retain code-graph retrieval as a localisation tool, without extrapolating beyond the measured +4.9pp.
2. Keep integrated resolution as a regression baseline while moving future evaluation toward representative, contamination-resistant tasks and component-level evals.

6. Limitations

• No same-model scaffold-off full-run control separates the causes of 394/500.
• The full prediction and scorer bundle was not found in the retained repository; the aggregate report remains.
• The graph-versus-grep run stopped at n=465, without retained raw pairs or uncertainty estimates.
• Only task IDs are machine-verified as fixed in the 50-task comparison; multiple system variables changed.
• Repository samples and difficulty are unbalanced, and the benchmark itself has documented flaws and contamination risk.
• The $1,800–1,900 cost is an aggregate-report estimate, not a retained call-level ledger.

Reproducibility and evidence boundary

Item	Retained evidence
Measured	June 2026
Full run	docs/archive/verified-500-report.md; 500-task aggregate and repository breakdown
Search-tool A/B	docs/code-graph-eval/benchmark-paper.md; log-reconstructed n=465 aggregate
50-task comparison	coh_nodocker_50.json, coh_execsel_50.json; IDs and outcomes retained, model/prompt flags absent
Harness	src-tauri/src/code_graph/eval_live.rs, eval_bench.rs
Claim status	Full run is an integrated baseline; graph-vs-grep is component evidence; 27→38 is non-causal configuration evidence
Bitwise reproduction	Not possible: full raw predictions, n=465 paired logs, and the historical hosted-model state are incomplete