Exhaustive extraction added relations, but no retrieval gain was detected

Question: Does extracting relations more exhaustively improve note retrieval in a large vault?

Observation: Relation volume rose 10% on n=120 prose notes and 17% on n=59 table notes at R=5, while recall@10 moved only +0.7 pp on prose and +0.1/+0.7 pp on two table runs. No detectable gain appeared under this harness.

Decision: Keep the setting without declaring a zero global effect, and do not cite an improvement without same-arm controls, replication, preserved failures, and externally validated gold.

1. The question the fixture result could not answer

The original case for exhaustive extraction was answer accuracy rising from 60% to 97% · a few fixtures · single-document graphs · three-run average. It also combined a prompt change with entity and relation cap changes. With one document, there is little retrieval competition and no meaningful ranking cutoff, so that number cannot be generalized to vault-scale retrieval.

The new experiment is not a direct replication of the same outcome. It asks how note recall@10 changes when selected document subgraphs are replaced inside a 2,810-document graph. The result therefore narrows the fixture claim’s scope; it does not show that the original 60%-to-97% answer-accuracy estimate reversed.

2. A marginal intervention with failure preservation

Design element	Condition
Corpus	Market vault · 3,002 notes on disk · 2,810 graph documents with paths · 30 Jul 2026 export
Prose treatment	Replace only 120 sampled-note subgraphs · hold the other 2,690 documents fixed · Sonnet 5
Prose evaluation	n=150 questions · 174 chunks per arm · one run plus an independent same-arm control
Table treatment	59 notes with at least eight table rows · 106 chunks per arm · R=5 independent extractions per arm
Question construction	Seed-sample n=150 questions from the union of every arm and replicate output

The union pool prevents either arm from writing an exam solely from its own output. The same-arm control measures how far extraction moves when the prompt does not change.

The instrument is part of the result. In the first attempt, a bad authentication path caused 48/48 chunks · both arms to fail, while swallowed errors produced a plausible “zero relations versus zero relations” null. src-tauri/src/tagging/qa_eval.rs now preserves the first error and aborts when failed chunks exceed 20% of all chunks. Refusing to publish a symmetric result from a dead instrument is the most reusable part of this evaluation.

3. Observed results: more relations, no detected retrieval gain

The prose run used n=120 notes, n=150 questions, and one extraction per arm.

Prose metric	Concise→comprehensive	Same-arm control	Observation
Relations per note	17.0→18.7 · +10%	17.0→17.1	Volume change exceeds control movement
Subject–predicate occupancy	66.0%→67.3% · +1.3 pp	69.3%→66.0% · −3.3 pp	Inside control movement
Recall@10	89.3%→90.0% · +0.7 pp	91.3%→92.7% · +1.3 pp	Smaller than control movement

The table slice used n=59 notes, n=150 questions, and R=5 per arm in each of two independent runs.

Table metric	Run A · retrieval only	Run B · with answers	Interpretation
Relations per note	concise 22.4–23.0 vs comprehensive 26.4–27.1 · +17%	Same direction; no separate headline retained	Run A’s five-run ranges do not overlap
Subject–predicate occupancy Δ	+5.7 pp · p=.038	+3.9 pp · p=.478	Same direction; threshold significance did not repeat
Recall@10 Δ	+0.1 pp · p=.84	+0.7 pp · p=1.00	Neither run detects a gain
Answer string accuracy Δ	Not measured	+4.4 pp · p=.298 · 1,500 total answers	Inside the baseline arm’s 7.3 pp five-run range

p=.038→.478 is not an effect reversal. Both point estimates, +5.7 pp and +3.9 pp · n=150 questions·R=5 each, are positive; only threshold crossing changed. Likewise, +0.1 pp and +0.7 pp recall · n=150 questions·R=5 each do not prove an exact zero. Without an equivalence margin, confidence interval, or prospective power target, the defensible wording is “no retrieval gain was detected under this harness.”

The “590 calls” previously reported for table run A were 59 notes × two arms × five replicates = 590 note–arm–replicate extraction units, not API calls. The actual count is 106 chunks × two arms × five replicates = 1,060 extraction calls.

4. What the metrics mean and do not mean

“Gold facts the graph asserts” was stronger than the implementation. In qa_eval.rs, answerable ignores the recorded answer object and checks only whether any object exists for the subject–predicate pair. This record therefore calls it subject–predicate occupancy.

The question pool is not human-authored or externally validated truth. It is a self-generated pool formed from the union of relations extracted by every model arm and replicate, then filtered to subject–predicate pairs with one object. An exhaustive arm can add its own extra or mistaken assertion to the exam, and precision was not measured. A +5.7/+3.9 pp occupancy estimate cannot be translated into “more correct facts.”

The table result is not a production estimate either. Production’s run_table_pass classifies entity tables, loads them with deterministic code, and removes them before prose extraction. This harness sends the complete table through the prose prompt, making the observed contrast closer to an upper bound on prompt-path value. The one-object filter also excludes many financial-table facts that repeat the same metric across quarters.

A separate rank probe found 39 of 42 notes outside the top ten already present in the candidate pool, but its golds come from the existing graph. Facts missing from extraction cannot enter that exam. It therefore cannot establish that ranking, rather than missing extraction, is the dominant real-vault cause.

5. Decision, counterfactual, and next test

Observation: the comprehensive prompt increased relation volume on the selected notes in this vault. Decision: the setting stayed in place because there is no evidence that it is worse and no basis for discarding information used by other paths. That decision is not evidence of a retrieval improvement.

Unmeasured counterfactual: re-extracting all 2,810 documents once with each policy. This experiment changed only 120/2,810 document subgraphs · 4.3%; the other 2,690 remained in their existing comprehensive state in both arms. It is also one vault from one owner.

Reopening the claim requires externally validated object-level gold and precision, uncertainty that accounts for question/note clustering, a prespecified equivalence margin, the real run_table_pass, and either full re-extraction or a prospectively powered replicate count. Runs A and B also resampled from 446 versus 445 eligible pooled items, so their p-values cannot be post hoc combined as if they tested the same question sample.

Reproducibility and evidence boundary

Evidence	What is currently verifiable	Boundary
docs/details/extraction-arms-vault-scale-2026-07-30.md	Design, aggregates, and $74.37 total OpenRouter billing across the experiment suite	Market source data and complete raw run logs are not publicly reproducible
src-tauri/src/tagging/qa_eval.rs::extraction_arms_vault_ab	Arm definitions, pooled questions, occupancy implementation, and failure abort	No external truth, precision, or cluster-aware interval
Prose result	n=120 notes · n=150 questions · one run plus same-arm control	Marginal effect; different outcome from the original 60%→97% figure
Table runs A/B	n=59 notes · n=150 questions · R=5 each	Not the production table path; question samples also differ by run
Failed-chunk record	Run A records 1/1,060; run B records 7/1,060	Conflicts with the final summary’s 1/2,240; do not cite a combined total without raw logs
Recall +0.7/+0.1/+0.7 pp	Observed prose/table-A/table-B point estimates	Does not establish zero effect or zero global effect