Extracting more does not move retrieval — a null result, and a significance that did not replicate

The knowledge-graph pipeline has a setting for how exhaustively facts are pulled out of prose. The exhaustive setting was turned on everywhere on the strength of an earlier measurement: answer accuracy rising from 60% to 97%.

That measurement came from single-document graphs over a handful of fixtures. With one document there are no other notes to compete with, the retrieval cutoff never binds, and every entity in the graph is reachable. In other words, it has no scale axis. This record puts the axis back in and measures again.

Design

A paired A/B ran against a real 3,002-note vault, with both arms living inside the same production graph. The vault’s graph is imported into an isolated temporary store and only the document graphs of the sampled notes are replaced. Note bodies, the keyword index, the embeddings and the other 2,690 documents are byte-identical between arms, so a difference can only come from that subgraph.

The test questions are derived from the union of both arms. Deriving them from one arm’s graph gives that arm a structural ceiling of 100%. This repository had already been caught by that trap once, so questions come from a third store holding both arms’ output. Neither arm sits its own exam.

A same-arm control ran alongside: two independent extractions with the same prompt. Reading a treatment effect requires knowing how much the number moves when nothing changes, and that had never been measured here.

Result on prose

Metric	Treatment (switched to exhaustive)	Control (same prompt twice)
Gold facts the graph asserts	66.0% → 67.3% (+1.3pp)	69.3% → 66.0% (−3.3pp)
Test	51/49, p=0.92	46/51, p=0.68
Note retrieval recall@10	89.3% → 90.0% (+0.7pp)	91.3% → 92.7% (+1.3pp)

Every treatment number sits inside the control’s noise band. The treatment’s +1.3pp is smaller than the −3.3pp produced by running the same prompt twice, and its +0.7pp on retrieval is smaller than the control’s +1.3pp.

Extraction volume genuinely does rise. Relations per note go from 17.0 to 18.7, a 10% increase, and the control is flat at 17.0 to 17.1, so that increase is a real effect of the prompt. But the extra relations do not change the set of gold facts, and do not convert into retrieval: of the 51 facts the treatment newly found, exactly 1 turned into a retrieval win.

Again on tables, this time with five replicates

With prose returning null, the same experiment was repeated on tables, where the prompt was claimed to matter.

The first finding was that a table-heavy vault does not exist. Surveying all three vaults exhaustively, notes with a table of three or more rows numbered 64 in one vault and 0 and 1 in the others. This is not a sample but the entire population, and the only axis for adding statistical power is replicates, not notes.

A single run showed retrieval recall up 3.3pp with a consistent direction. But the run-to-run spread of one arm against itself was 4.7pp, larger than that effect. So both arms were extracted independently five times each, per-question hit rates were averaged over the five, and a sign test applied. 590 calls, $19.57.

Metric	Baseline	Exhaustive	Δ	Sign test
Relations per note	22.4–23.0	26.4–27.1	+17%	ranges do not overlap
Gold facts the graph asserts	26.7%	32.4%	+5.7pp	p=0.038
Note retrieval recall@10	67.5%	67.6%	+0.1pp	p=0.84

With power behind them the two metrics went opposite ways. The +3.3pp retrieval gain from the single run was noise, and pooling put it back at +0.1pp.

And then the significance did not replicate

The same design ran once more with an answer layer attached: 1,500 answers and 1,500 judgements.

Metric	Run A (retrieval only)	Run B (with answers)
Gold facts asserted, Δ	+5.7pp, p=0.038	+3.9pp, p=0.478
Retrieval recall@10, Δ	+0.1pp, p=0.84	+0.7pp, p=1.0000

Two independent runs of the same design, the same corpus and the same sample size produced p-values of 0.038 and 0.478. The direction is positive both times and the magnitudes are similar, but the significance did not replicate. 0.038 was a borderline value, and it did exactly what borderline values do.

So the first run’s reading, that the extraction gain was established, was too strong. The honest summary is this: the direction consistently favours exhaustive extraction, and at five replicates the significance does not replicate.

Answer accuracy behaves the same way. The +4.4pp has the right sign but p=0.298, and it falls inside the 7.3pp run-to-run spread of the baseline arm against itself.

The one firm conclusion

Only one thing in this experiment never wavered. Extracting more does not move note-level retrieval. Both runs give +0.1pp and +0.7pp, both inside the noise floor.

The reason is that the bottleneck is elsewhere. On prose, retrieval recall runs 23pp above the rate at which the graph holds the gold fact. Even when the graph does not assert the fact, the fact is in the note body and keyword and dense retrieval lift the note anyway. A separate probe found the same thing: of 42 cases where the gold note fell outside the top 10, 39 (93%) were already in the candidate pool and only 3 were genuinely absent. The dominant way retrieval fails on a real vault is not a missing fact but a fact that cannot be lifted.

Two instrument defects

The experiment hit defects in its own instrument twice, and both times only after results existed.

Every extraction call died on an auth error and it looked like a measurement. The production request function goes to an internal proxy rather than the API, and that proxy authenticates with a user token while the eval was sending an API key. The first run had all 48 chunks fail within two seconds and produced a perfectly symmetric “zero relations” in both arms — the most dangerous shape a broken instrument can take. The code that was swallowing the error now preserves and prints the first failure message, and the run aborts outright once the chunk failure rate passes 20%.

There was no same-arm control to begin with. Looking at the treatment alone reads as “exhaustive found 51 more facts”. That reading only holds if extraction is reproducible, and reproducibility had never been measured. Adding the control revealed that the noise floor is larger than the treatment effect.

Decision

The setting did not change. There is no evidence that exhaustive extraction is worse, and it genuinely does extract more. What changed is how the result is cited.

The 60%-to-97% figure is no longer quoted as a vault-scale claim. It is an answer-accuracy number from single-document graphs over a handful of fixtures, with a prompt change mixed together with other changes. Re-running the same prompt comparison against note-level retrieval in a 3,002-note vault gives +0.7pp on prose and +0.1pp on tables.

What this measurement does not show

• A marginal effect, not a global one. Only the subgraphs of 120 documents out of 2,810 (4%) were changed. “What if the whole vault were re-extracted the other way?” is a different question requiring a full re-extraction.
• Production does not process tables through this prompt. The real pipeline classifies tables first, loads them row by row with deterministic code, and cuts them out of the body. This harness sends the whole body down the prompt path, so the table numbers above are an upper bound, not an estimate. In production the difference is smaller.
• The gold instrument under-samples tables structurally. The gold filter requires a subject and predicate to have exactly one distinct object, and financial tables repeat the same metric every quarter, so most of them are eliminated.
• Only fact-lookup questions and note-level retrieval. Multi-hop and synthesis queries are not measured at all.
• One vault, one owner.

Measurement record

Item	Value
When	30 July 2026
Corpus	The Market vault: 3,002 notes on disk, 2,810 documents carrying a path in the graph
Prose sample	120 notes, 525,804 characters, median 3,207, 174 chunks per arm, 150 questions
Table sample	59 notes, 457,985 characters, 3,482 table rows, 106 chunks per arm
Model	claude-sonnet-5
Harness	src-tauri/src/tagging/qa_eval.rs::extraction_arms_vault_ab
Test	Per-question sign test, ties excluded, exact binomial. Replicated runs compare hit rates averaged over five
Total cost	$74.37, the amount OpenRouter actually billed rather than tokens times list price
Failed chunks	1 of 2,240 (0.04%)

The table slice keeps only notes with at least eight table rows and pools five replicates. A single run produced a misreading on this corpus twice, so running once means running the same-arm control first, without exception, to establish the noise floor.

The primary record is docs/details/extraction-arms-vault-scale-2026-07-30.md in the Consilience repository.