Post-ship validation: why a judge-source change and tolerant parser were reverted

Question: Did a commit that changed the judge’s source text and the extraction JSON parser outperform the behavior already in production?

Observation: Across 121 relation–verdict pairs from 15 affected, content-deduplicated documents, all three disagreements favored the old source under a model oracle; the tolerant parser recovered content in 0 of 60 direct calls.

Decision: Revert the source change, table-relation exemption, and tolerant parse; retain only the guard that skips an empty judge call.

The two measurements use different units: relation–verdict pairs above and model calls below.

1. Separate the changes, then bound their reach

Commit 7def5456 changed three things at once. It replaced the judge’s table-bearing 6,000-character source clip with a post-table-strip prose chunk, exempted deterministic table relations from judgment, and added a fallback after strict JSON parsing. The first two changes were coupled; the parser had a separate failure surface and was evaluated separately.

The source change can matter only when a note contains a Markdown table and its remaining prose fits one chunk. An exhaustive enumeration of 7,235 Markdown notes across three vaults produced this scope.

Vault	Notes, n	Notes with tables	Gate-eligible notes
Market	4,306	78	47
news	314	0	0
e2e	2,615	1	1
Total	7,235	79	48 (0.66%)

Content deduplication reduced those 48 notes to 15 distinct documents: SEC filings from Apple, Netflix, and NVIDIA, plus one Korean schema note. They were not all corporate filings. Thirty-six of the 48 affected notes already had real pre-change drop records, confirming that this was an exercised path rather than reachability alone.

2. The source A/B moved one variable, but remained a model experiment

For each document, the harness freshly extracted the prose relations once and then held that set fixed across both arms. OLD read the first 6,000 characters of the processed body with tables included; NEW read the single prose chunk after tables were removed. This isolates judge input, but it is not an exact replay of the relations judged in production.

Result	n · unit · condition
Comparisons completed	121 relation–verdict pairs · 15 distinct documents · fixed relation set
Arm disagreements	3 relations · 2.5% of 121 pairs
Oracle matches	OLD 3 / NEW 0 · only the 3 disagreements adjudicated
Failed calls	4 calls · excluded when verdict count differed from input; never padded
Sign test	3–0 · two-sided p=0.25 · not statistically significant

The oracle was not human ground truth. By default it used the same claude-opus-5 model family as the judge, with low effort for the judge and medium effort for the oracle. Each disagreement received up to three independent calls and was scored by majority vote. A 2–1 split was also flagged as contested, but was not excluded from scoring. All three relations in this run were unanimous, so the implementation limitation did not alter the observed result.

The primary record says the rule “NEW must beat OLD; a loss or tie reverts” was written before the run. Repository history does not independently establish that timestamp, so this record does not call it preregistration. It is treated as a conservative adoption rule that places the burden of proof on the proposed change.

3. All three disagreements exposed the same mechanism

Each disagreement was OLD=drop, NEW=keep, and model oracle=unsupported. In a filing about a chief executive transition, the table was not noise around the prose; it was the structured evidence that refuted a stale leadership relation. Removing it gave the judge more complete prose while withholding the role transition, making the judge more permissive.

This does not establish the old judge’s absolute accuracy. The oracle never scored the 118 of 121 pairs on which both arms agreed. The defensible claim is narrower: within this affected scope and relation sample, all three verdicts changed by table removal moved in the harmful direction. With n=3 and p=0.25, the result supports reverting an unproven change, not a universal superiority claim.

4. The tolerant parser did not repair the observed failures

The fallback reads the first complete JSON value and ignores what follows. It can recover a valid payload followed by junk. It can also accept an unrelated leading object with no recognized payload fields, such as {"ok": true}, as a successful extraction containing only defaults. This is narrower than saying every leading object always becomes empty.

The Market vault was traversed at a deterministic stride and yielded 60 usable claude-sonnet-5, low-effort responses from direct /responses calls. Both parsers were applied to the same outputs.

Parser outcome	Calls, n=60	Interpretation
Strict parse succeeded	58	Fallback never consulted
Strict failed → content recovered	0	No observed benefit
Strict failed → empty payload recovered	0	Latent hazard not observed
Both parsers failed	2	Prose without JSON; fallback cannot repair it

Zero recoveries in 60 calls has an approximate 95% upper bound of 5%; it is not proof that production never emits recoverable trailing junk. The harness also called the endpoint directly, while production used an authentication proxy that could affect output shape. Strict parsing was restored because this path showed no benefit and retained a silent-empty risk.

5. A table-relation defect was recorded and deliberately deferred

Forty-two real production drops were re-adjudicated by the same model oracle, then deterministically classified by whether the relation’s object text appeared inside a table region in its note. In that decontaminated analysis, 8 of 8 table-derived drops were wrong; for prose-derived drops, 8 were wrong and 26 were right. The full log showed a much smaller exposure.

Log scope	All dropped relations	Table-derived drops	Affected notes
Market	3,215	9 (0.3%)	2
e2e	1,781	0	0

The 8/8 conditional error rate and the 9/3,215 population exposure have different denominators. A blanket exemption would remove a rare failure while also removing the only check on schema misclassification and disturbing the sparse-relation gate, so no production change followed. The table/prose classifier was a one-off deterministic analysis rather than a fully retained output of the durable harness. Any future expansion of single-chunk judge coverage must remeasure this rate instead of carrying 0.3% forward.

6. Decision and current code state

The source and exemption were reverted in fd8c3513; tolerant parsing was reverted in 80fd3698; both decisions landed together in c516e29f. The current implementation gives the judge a table-bearing 6,000-character source clip, exempts user keep verdicts rather than table relations as a class, and retains the no-op guard when nothing is judgeable. Extraction payloads again use strict serde_json::from_str.

This is not a ruling that the proposed design is always worse. It is a post-ship decision to withdraw a change whose bounded evaluation showed no gain and exposed a credible harm mechanism. The durable outputs are the empty-call guard, a unit test that pins the parser hazard, a rerunnable harness, and explicit conditions for reopening the table issue.

7. Reproducibility and evidence boundary

Evidence	Observation and condition	Boundary	Primary source
Blast-radius census	48 of 7,235 notes; 15 documents after content deduplication	Three local vaults as of 2026-07-26	docs/details/verify-judge-source-2026-07-26.md:26-43
Source A/B	121 relation–verdict pairs; 3 disagreements; OLD 3 / NEW 0	Relations freshly extracted once; model oracle; two-sided p=0.25	src-tauri/src/tagging/eval_extraction_quality.rs:5377-5647
Oracle aggregation	Up to 3 calls per relation; majority scored; 2–1 also flagged	Not human truth; no contested votes occurred in this run	src-tauri/src/tagging/eval_extraction_quality.rs:5550-5582,5624-5644
Parser measurement	60 direct calls: 58 strict successes, 0 content recoveries, 0 empty recoveries, 2 joint failures	Not the production auth proxy; 0/60 has an approximate 5% upper bound	src-tauri/src/tagging/eval_extraction_quality.rs:5765-5848
Current implementation	Table-bearing clip, empty-judge guard, strict JSON parse	Current checkout only; later commits may change it	src-tauri/src/tagging.rs:1771-1824,4447-4472
Table-defect reanalysis	8/8 oracle-scored table drops wrong; 9/3,215 across the full Market log	One-off string classification; two affected notes; no cross-corpus generalization	docs/details/verify-judge-source-2026-07-26.md:148-184