MuSiQue pooled-corpus ablation: the PPR graph leg and the limits of fixed fusion weights

Question: In a 100-question, 1,730-passage pooled MuSiQue pilot, how much does the PPR graph leg contribute to supporting-passage retrieval, and can fusion weights selected on one corpus transfer to owner vaults?

Observation: Strict support-set ALL@10 was 63% with full fusion, 25% without PPR, and 23% with keyword-only retrieval. The four-hop slice was 23.5% versus 0/17, but the absolute gap did not widen with hop count and reader accuracy was not measured.

Decision: Do not generalise the graph as a “precondition for multi-hop.” Retain PPR as an important retrieval leg in this internal comparison, but do not remove BM25 from the shipped owner-vault configuration to optimise MuSiQue.

MuSiQue is a multi-hop QA benchmark whose questions require evidence across several passages. This study is not a leaderboard attempt. It asks what PPR diffusion contributes within Consilience’s four-leg retriever.

1. One pooled corpus, not twenty passages per question

One hundred questions were sampled from MuSiQue-Ans dev with a fixed, hop-stratified seed. The dev file is ordered by hop type; taking the first 100 would silently select only the easiest two-hop slice. The retained mix is 52 two-hop, 31 three-hop, and 17 four-hop questions.

After deduplication, the questions’ passages form one 1,730-passage vault. Every query searches this entire pool rather than ranking only within its own 20 passages. The production extraction path produced 1,727 document graphs, 9,499 entities, and 63,492 quads, with no missing embeddings. The effect of the three passages without a graph on the gold sets was not separately audited.

Two metrics are kept distinct:

• R@k: the mean fraction of each question’s supporting passages found in the top k.
• ALL@k: the fraction of questions for which every supporting passage is in the top k.

ALL@10 is a strict support-set completeness measure. It is not answerability or final answer accuracy; no reader EM/F1 was measured here.

2. What disappears when the PPR leg is removed

Questions, corpus, extracted state, and evaluation code were fixed while fusion weights changed. The shipped arm uses lexical 0.1, BM25 1.0, PPR 1.0, and semantic 0.05.

Retrieval arm	R@5	R@10	ALL@10
Full fusion	62.2%	83.0%	63.0%
No PPR	51.2%	58.9%	25.0%
Lexical + BM25 only	50.2%	57.5%	23.0%

Relative to no-PPR, full fusion gains 24.1 points of R@10 and 38.0 points of ALL@10. A paired sign test on per-question R@5 found full fusion higher on 29 questions and lower on two (p=4.6e-7, n=100). That p-value belongs to the paired R@5 comparison, not to ALL@10 or the hop-stratified bars.

Nor does this show a graph succeeding without lexical signal. PPR entity seeds originate in lexical and semantic similarity. The measured difference comes from routing those initial signals through entity–document relations and then into note ranking rather than stopping at direct document matching.

3. Hop breakdown: the correct reading of 0/17

Hops	n	Full ALL@10	No PPR	Absolute difference
2	52	80.8%	36.5%	+44.3pp
3	31	54.8%	19.4%	+35.4pp
4	17	23.5%	0.0%	+23.5pp

None of the 17 four-hop questions had its full support set in the top ten without PPR. The scope matters:

• Both arms decline as hop count increases.
• The absolute gap narrows from 44.3 to 35.4 to 23.5 points; it does not widen with hops.
• PPR appears increasingly important in relative terms, but n=17 is small and no per-hop significance tests were run.
• Zero of 17 is an observation under this corpus, retriever, and k=10. It is not proof that graphs are a logical prerequisite for all four-hop retrieval.

The defensible result is that PPR materially improved support-set completeness at every hop level in this pilot, and no no-PPR success was observed in its hardest slice.

4. A higher benchmark score was not a better shipping configuration

With PPR fixed at 1.0, an exploratory 45-configuration sweep over the same pooled corpus reached an observed R@5 maximum of 78.1%. The top six configurations all set BM25=0; the shipped configuration scored 61.9%. This is a grid maximum without multiplicity correction or independent confirmation.

The more useful counterexample came from an owner vault:

Corpus	BM25 1.0	BM25 0	Effect of the same parameter
Market fact lookup	95.3%	88.0%	+7.3pp
MuSiQue R@5	61.9%	77.1%	−15.2pp

MuSiQue distractors are deliberately selected to be topically related to the question and may therefore be unusually difficult for BM25. Personal notes are not adversarially selected in the same way. That mechanism is plausible, but it was not established through a separate distractor-selection intervention.

The shipped weights were therefore not changed to match the benchmark optimum. The experiment rejects universal static weights and motivates a follow-up hypothesis: route fusion by query class and corpus distribution. Such a router has not yet been validated.

5. Transfer boundary on owner vaults

The same shipped retrieval path was ablated on fact-lookup questions from two owner vaults.

Vault and query set	Full fusion	No PPR	Keyword only
Market, n=150	90.7%	78.7%	60.0%
e2e, n=150	76.0%	52.7%	37.3%
e2e cross-language, n=36	63.9%	41.7%	36.1%

These are independent product-side evidence that PPR contributes to recall@10 in those two vaults. Both vaults belong to one owner, however, and the questions are graph-derived single-fact lookups. They do not establish a user-population or general multi-hop effect.

Sample size also changed the decision. On Market, the no-PPR effect was −5pp and p=.25 at n=40, but −12pp and p=4e-5 at n=150. A plausible-looking small run would have produced a false negative, so later retrieval ablations report paired discordances as well as averages.

6. Limits and next experiment

• The 100-question, 1,730-passage pool is smaller than standard public settings; absolute external comparisons are invalid.
• The 45-arm maximum is exploratory and has no independent holdout.
• The study measures evidence retrieval, not reader EM/F1, citation correctness, latency, or graph-build cost.
• The graph-ranking arm still uses lexical/semantic entity seeding and is not lexical-free graph retrieval.
• Three passages failed graph extraction; their gold impact was not decomposed.
• The next valid test is a query-class router evaluated jointly on MuSiQue and owner vaults, rejecting configurations that win on only one distribution.

Reproducibility and evidence boundary

Item	Retained evidence
Measured	26 July 2026
Sample	MuSiQue-Ans dev, 100 fixed-seed stratified questions; 2/3/4 hops = 52/31/17
Corpus	1,730 unique passages through the production extraction path
Harness	eval/musique_prep.py, qa_eval.rs::musique_multihop
Paired test	Per-question R@5; full vs no-PPR 29 wins / 2 losses, p=4.6e-7
Primary records	docs/details/musique-multihop-2026-07-26.md, retrieval-ablation-2026-07-26.md
Claim status	Internal PPR contribution retained; “graph as precondition,” widening hop gap, and external rank claims withdrawn