MuSiQue multi-hop QA — the graph is a precondition, not an improvement
The graph condition and a keyword-only condition, compared on the same question set. The gap widens with the number of hops, and at four hops the control arm scores zero.
- questions with every supporting paragraph retrieved (ALL@10), graph vs keyword-only
- 63.0% vs 23.0%
- of 17 four-hop questions, none succeeded without the graph
- 0.0%
MuSiQue is a public multi-hop QA benchmark built from questions whose answer is in no single document. Two to four documents have to be traversed in order, and the set is constructed so that skipping a step produces a plausible wrong answer.
The control
A claim that the knowledge graph improves retrieval is not established by the graph condition alone. It needs the same question set with the graph taken out.
The three conditions share the question set, the corpus, the prompt and the extraction pipeline, and differ only in the fusion weights. The corpus is the paragraphs of all 100 questions pooled into one vault, with every question asked against the whole of it. This is not the easier setting where retrieval only ranks within a question’s own 20 paragraphs.
The headline metric is ALL@10: the fraction of questions for which every supporting paragraph was retrieved. A two-hop question with one of its two paragraphs retrieved cannot be answered, so what decides answerability on multi-hop questions is this metric, not average recall.
| Condition | ALL@10 |
|---|---|
| Keyword retrieval only | 23.0% |
| Graph removed | 25.0% |
| Knowledge graph | 63.0% |
A paired sign test over per-question R@5 gives p=4.6e-07 against the graph-removed arm and p=2.5e-07 against keyword-only. The comparison is paired because question difficulty varies so widely: comparing two condition means lets the result move with which side happened to draw the hard questions.
Broken out by hop count
The decomposition matters more than the overall average.
| Hops | n | Graph | Graph removed |
|---|---|---|---|
| 2 | 52 | 80.8% | 36.5% |
| 3 | 31 | 54.8% | 19.4% |
| 4 | 17 | 23.5% | 0.0% |
Of 17 four-hop questions, not one had all its supporting paragraphs retrieved without the graph. Not low. Zero.
That is a difference in kind, not degree. With keyword retrieval there is no path to a four-hop answer at all. What points from the first document to the second is a relationship, not surface vocabulary, and a relationship that is not indexed cannot be retrieved.
So the conclusion is not “the graph makes retrieval better”. For some classes of question the graph is not an improvement but a precondition.
The decision this run actually changed
A 45-configuration weight sweep over the same corpus put keyword weight at zero in all six top configurations. That contradicts the measurement on the project’s own vaults head on.
| Corpus | Keyword weight 1.0 | Keyword weight 0 | Effect |
|---|---|---|---|
| Real vault (recall@10) | 95.3% | 88.0% | +7.3pp, helps |
| MuSiQue (R@5) | 61.9% | 77.1% | −15.2pp, hurts |
The same parameter helps on one corpus and hurts on the other by a comparable margin. The mechanism is in how MuSiQue is built: its distractor paragraphs are deliberately chosen to be topically close to the question, which makes them lexical traps, and keyword retrieval promotes precisely those.
The shipped weights did not change. A user’s vault is their own notes, not a set of Wikipedia paragraphs selected to be confusable. Reverting weights validated on two real vaults in order to score better on a benchmark is benchmark overfitting, and the benchmark is the less representative of the two. What this run bought is not a score but a counter-example: the current weights are now known by measurement to be optimal for the two vaults they were tuned on, not universally.
Repeated on real vaults
A public benchmark has a curated corpus and does not resemble real use, so the same comparison was repeated on two real vaults. The metric is recall@10 and the test is an exact paired McNemar.
| Vault | Full fusion | Keyword only | McNemar p |
|---|---|---|---|
| Market (n=150) | 90.7% | 60.0% | 3.5e-13 |
| e2e (n=150) | 76.0% | 37.3% | 6.9e-18 |
In three of the four comparisons the control arm gained zero questions. Pure loss, not a reshuffle. And this question stratum is the one the harness selects for questions that name their subject outright, which is the setting most favourable to keyword retrieval. Beating a control you weakened yourself proves nothing.
A case where sample size reversed the conclusion
Run at the harness default of n=40, the graph-removal comparison gives −5pp on two questions, p=0.25 — indistinguishable from noise. At n=150 the same comparison is −12pp at p=4e-05.
Baseline recall itself was stable across both samples, 90.0% against 90.7%. What moved was the effect estimate, not the level, so this is variance rather than bias — which makes it more dangerous, because the number looks plausible. Had it been run as originally specified at n=40, it would have produced a false negative and mis-prioritised the retrieval roadmap.
Limits
- Do not compare the absolute scores against published numbers. The pool here is 1,730 paragraphs from 100 questions. Published MuSiQue retrieval numbers usually use 1,000 questions and roughly 12,000 paragraphs, and recall falls as the pool grows. This is the easier setting.
- Do not quote the sweep maximum as this system’s score. R@5 of 78% belongs to a keyword-weight-zero configuration that is worse on the real vaults. The shipped configuration scores 62.2%.
- No published baseline was checked against its primary source. An audit in the same period found that roughly a third of external citations failed fact-checking, so no number from memory is quoted here.
- Retrieval only. No reader-stage EM/F1 was produced. That is a different metric.
- Graph construction cost is not in these tables. It is incremental at save time, but it is not zero.
Measurement record
| Item | Value |
|---|---|
| When | 26 July 2026 |
| Dataset | MuSiQue-Ans dev, 100 questions drawn with a fixed seed and stratified by hop count (52 two-hop, 31 three-hop, 17 four-hop) |
| Corpus | 1,730 unique passages, one markdown note each, pooled into a single vault |
| Extraction | The real application pipeline, not a test harness. 1,727 of 1,730 document graphs, 9,499 entities, 63,492 quads, no note missing an embedding |
| Extraction cost | About $45 |
| Harness | eval/musique_prep.py and qa_eval.rs::musique_multihop |
| Real-vault A/B | 26 July 2026, Market and consilience-e2e, n=150, through the shipped path |
The graph-removed arm and the 45-configuration sweep run on the same harness with only its configuration changed. The per-question output lines carry hop count, question id and R@5, and those lines are the input to the sign test.
Stratified sampling is not optional. The dev file is ordered by hop type, so taking the first N yields 100% two-hop questions and silently measures the easiest slice.
The primary records are docs/details/musique-multihop-2026-07-26.md and docs/details/retrieval-ablation-2026-07-26.md in the Consilience repository.
What this measures
- Consilience
The world's first general-purpose ontology OS — it builds a knowledge graph from markdown documents automatically, and an agent reasons over that graph