SWE-bench Verified across all 500 tasks, and what that number cannot claim
The result of running all 500 tasks on the official harness. That run cannot separate the scaffold's contribution from the model's, so the separation was measured in two controlled experiments instead.
- 394 of 500, official harness, no retries
- 78.80%
- model and 50-task set fixed, selection procedure swapped
- 54% → 76%
SWE-bench Verified replays real GitHub issues against the repository state they were filed on, and scores a model’s patch by whether it passes tests it never saw. There is no partial credit, so “nearly right” does not exist.
The measurement
All 500 tasks were run on the official harness. No subset was drawn and no failed task was retried to lift the average. The result is 394 of 500, or 78.80%. The remaining 106 are counted as unresolved, 5 of them because the patch came back empty. Nothing went unscored.
Running the whole set was a decision about comparability rather than cost. A subset score cannot be compared with anyone else’s report unless the subset is published with it, and task difficulty varies enough that which tasks were drawn moves the number.
The distribution across repositories is not flat. scikit-learn at 30/32 and xarray at 20/22 are the highest, pylint at 4/10 the lowest. django alone is 231 tasks, 46% of the set.
What this number does not say
78.80% is not the scaffold’s score. An agent benchmark score is the model’s capability and the scaffold’s capability added together, and no control arm ran the model alone over the same 500 tasks, so this run cannot separate them. The model used is strong on its own, and a large share of the result is the model’s.
Verified is a saturated benchmark. OpenAI stopped reporting it publicly in February 2026, citing an audit that found a large share of flawed test cases in the hard subset, and evidence of training-data contamination in that frontier models reproduce the human gold patch verbatim. This number is therefore a baseline, not a record claim.
Both sentences come first because the honest reading requires them. Whether the scaffold is worth anything can only be established by a controlled experiment, which is why two of them were run separately.
Controlled experiment 1 — what the selection procedure contributes
The model and a fixed 50-task set were held constant, and only the way candidate patches are produced and chosen was changed.
| Condition | Resolve rate |
|---|---|
| Single attempt | 54% |
| Reproduction-based candidate selection | 76% (38/50) |
What changed here is selection, not retrieval. Candidates are generated repeatedly inside one reused worktree, each candidate’s own reproduction command is cross-run against the other candidates, and the ranking comes from execution results. The whole point of the procedure is that the deciding signal is a test run rather than the model’s own opinion of its work.
The experiment only exists because a reproduce, fix, re-verify loop exists first. Without reproducing the failure there is nothing to separate the candidates with.
Controlled experiment 2 — swapping only the tools
Whether the code graph itself earns its keep was isolated a second time. The model, the scaffold and the prompt were all held fixed and only the search tools handed to the agent were swapped. The metric is file localisation — does the file that needs editing appear in the top five — not resolve rate.
| Condition | file@5 |
|---|---|
| Text search | 65.2% |
| Code-graph retrieval | 70.1% (n=465) |
+4.9pp is not a large number. What matters is the consistency of the direction rather than the size. The same A/B on the easier SWE-bench Lite shrinks to +1.1pp, and on a separate file-localisation benchmark the two are level. There is no slice where the graph does worse. The gains sit where the repository is large and the issue does not name the symbol you need.
What makes the result trustworthy
- Before each experiment, the hypothesis, the rejection threshold and the expected shape of failure were recorded in a document together with the commit hash at the time of registration, so it can be verified afterwards that the criteria did not move once the result was in.
- An improvement whose cause could not be explained was not adopted, however good the metric looked.
- Gains that failed to reproduce, a p-value that did not hold on a second independent run, and a commit that was reverted after being compared with an oracle are all on the record. Three of the other write-ups on this site are those records.
What the harness taught
The first full run died to an external SIGKILL at 476 of 500. Predictions were checkpointed per repository, and django is a single 231-task repository, so roughly 207 in-memory patches were lost outright. That was about 27 hours and $700 of compute.
The fix had two parts: drop the checkpoint down to the instance, writing each prediction the moment it exists, and allow one repository to be split across several processes. The django re-run finished in about 6 hours with zero errors. Intermittent network drops that kept failing the scoring batches were handled with a wait-and-retry wrapper and pre-pulled images, so a test run no longer depends on the network.
Getting the checkpoint granularity wrong produces no symptom at all until the most expensive possible moment. Before those 27 hours it did not look like a design defect.
What is not published here
The code graph’s schema, the retrieval seeding and ranking algorithms, and the concrete implementation of the agent execution model are the product’s core assets and are not written down here. What is above goes as far as what was measured and what was held fixed.
Measurement record
| Item | Value |
|---|---|
| When | June 2026; one full run plus one django re-run |
| Scoring harness | The official swebench 4.1.0, on the official image namespace with environment-level caching |
| Architecture | x86_64 canonical prebuilt images (the leaderboard architecture), split into 84 batches |
| Model | anthropic/claude-opus-4.8 via OpenRouter directly |
| Retrieval | Lexical, dense embeddings (3,072-dim) and code-graph Personalized PageRank fused; the same code path the product uses |
| Loop | Reproduce, fix and re-verify inside Docker; five candidates then execution-based selection |
| Patch generation cost | About $1,800–1,900, including the roughly $700 lost to the crash. Scoring is compute only |
| Controlled experiment 1 | Same harness, 50-task subset, model fixed, selection procedure swapped |
| Controlled experiment 2 | file@5 on SWE-bench Verified n=465, tools swapped |
The primary records live in the Consilience repository: docs/archive/verified-500-report.md for the full run, docs/code-graph-eval/benchmark-paper.md for the tool A/B, and docs/details/ultra-mode-implementation-plan.md for the selection procedure. The harnesses are src-tauri/src/code_graph/eval_live.rs and eval_bench.rs, and both controlled experiments ran on them with only the model swapped.
The n=465 in experiment 2 is not 500. That run was stopped just short of completion and per-instance file@5 was recovered from the log rather than from an aggregate file. The @1, @10 and function-level figures were not captured.
What this measures
- Consilience
The world's first general-purpose ontology OS — it builds a knowledge graph from markdown documents automatically, and an agent reasons over that graph