Measuring a shipped commit, then reverting it

The extraction pipeline runs an LLM verify judge that audits extracted relations against the note’s source text and drops the unsupported ones. One commit changed the text that judge reads, exempted relations loaded deterministically from tables, and relaxed the parser on the extraction payload. All three shipped without measurement.

The stated reasoning was sound. The gate that turns the judge on is computed over prose while the judge was reading a different, truncated text, so “one window means the judge sees the source” was false. That observation is correct. It does not follow that the new text produces better verdicts, and that is what nobody had checked.

Blast radius first

The change is a no-op unless a note has a table that the strip cuts and its prose still fits one window. All three vaults were surveyed exhaustively.

Vault	Notes	With a table	Gate-eligible
Market	4,306	78	47
news	314	0	0
e2e	2,615	1	1

48 notes out of 7,235, or 0.66%. Deduplicated by content that is 15 distinct documents, all filings from the same handful of companies. Small enough to audit exhaustively, so no sampling was involved.

Not hypothetical either: 36 of the affected notes already carry real drop verdicts in the pre-change log. The judge really did fire on them.

The experiment

The relation set was held fixed and only the judge’s source text varied. Every relation the two arms disagreed on went to an oracle that reads the complete note untruncated with tables included, which is strictly more information than either arm has, with three independent votes and a 2-1 split reported as contested rather than rounded.

A judge reply whose verdict count does not match its input is discarded as a failed call and never padded. Padding would silently score that arm as “kept everything”.

Result — the new source text is worse

Item	Count
Relation-verdict pairs compared	121
The two arms disagreed on	3
Of those, OLD matched the oracle	3
Of those, NEW matched the oracle	0
Oracle 2-1 contested	0
Failed calls (excluded, never padded)	4

All three disagreements are the same failure, and the mechanism is the reverse of the one assumed.

Relation	Old	New	Oracle
Executive A holds title X	Drop	Keep	unsupported
Executive B transitions to the board	Drop	Keep	unsupported
Executive B leads the company	Drop	Keep	unsupported

Cutting the tables out does not give the judge a cleaner view of the prose. It removes the evidence that refutes a bad triple. On a filing announcing a chief executive’s departure, the table stating the transition is exactly what lets the judge reject “this person leads the company”. Strip it and the judge turns permissive.

More complete prose is worth less than less prose plus its tables.

The statistics, stated honestly

A 3–0 sign test is p=0.25 two-sided. Not significant, and squarely in the underpowered regime. But there is no more power to be had: the affected population is 15 documents and 121 verdict pairs, and they produced three disagreements.

Which is why the decision rule was fixed in writing before the run: “NEW must beat OLD; a loss or a tie reverts.” The burden of proof sits on the change, not on the status quo. NEW went 0 for 3.

What was reverted: the source change, the table exemption that depended on it, and two defects the pair had created. One is a gate regression. The gate constant exists because sparse conversational notes were being over-pruned, but the gate counts relations before the partition, so a note with 50 table rows and 2 prose relations passes the gate and then runs the judge on 2 triples — precisely the regime the constant exists to prevent. The other is a wasted paid call: the exemption could route every relation to exempt, buying a round trip whose reply cannot change anything.

One thing was kept deliberately: a guard that skips the judge call when nothing is judgeable. That path predates the reverted commit and remains reachable, and skipping a request whose reply cannot change any outcome carries no behavioural risk.

The third change — tolerant parsing

The same commit replaced the strict parse on the extraction payload with a fallback, so that a stray token after the object would not fail a whole note and burn one of its retries.

The fallback is a streaming deserializer that reads the first complete JSON value and stops, so strict and tolerant can only diverge in five ways. A closed set, not a matter of degree.

Model output	Strict	Tolerant
Complete payload with trailing junk	fail	recovers it intact	the only gain
Truncated or unbalanced	fail	declines	unchanged
Leading code fence	fail	declines	unchanged
Leading complete object, then the payload	fail	returns an EMPTY payload	silent data loss
Leading scalar, then the payload	fail	declines	unchanged

The fourth row is the one that matters. Every field on the payload struct has a default, so any leading JSON object deserializes into a fully empty payload. The note is then recorded as successfully extracted with zero entities and zero relations, where strict would have failed and retried.

It was measured: 60 calls over a deterministic stride across the vault.

Item	Count
Usable calls	60
Strict parse OK (fallback never consulted)	58
Strict failed, tolerant recovered content	0 — the only case the change wins
Strict failed, tolerant recovered an empty payload	0
Strict failed, tolerant also declined	2

Both real failures were the model answering in prose with no JSON at all. That is the same failure mode the commit cited as its motivation, and the fallback cannot fix it. It was aimed at a failure out of its reach, and the failure it does address did not occur once in 60 calls.

Reverted, but the case-table test was kept. The fallback function is still used in two other places, so “any leading object means an empty payload” remains live knowledge, and the test is the do-not-re-propose record in executable form.

The real defect the experiment exposed, and the decision not to fix it

The same oracle was used to score the real production drops: 42 of them. Taken flat, the judge is right more often than wrong. It should not be taken flat, because the log does not record where a relation came from, so prose extraction and table loading are mixed together.

Provenance can be separated deterministically: a drop is table-derived if its object text appears inside a markdown table region of its own note. That is a string test needing no additional LLM calls.

Producer	Judge was wrong to drop	Judge was right
Table-derived	8	0
Prose	8	26

The judge has never once been right on a deterministic table row. And it is genuinely useful on prose.

But running the same test over the entire log rather than the scored sample reverses the conclusion.

Vault	Distinct drops	Table-derived	Notes affected
Market	3,215	9 (0.3%)	2
e2e	1,781	0	0

Nine relations across the whole recorded history of the vault, in two notes that are the same filing. The reason is the gate: a table big enough to matter splits the note into more than one chunk, the gate fails, and the judge never runs.

Decision: no production change. An exemption would have to reinstate the gate regression that was just removed, would delete the only check on a class where schema misclassification is possible, and would alter 0.3% of drops. The defect is real and its per-event error rate is 8 out of 8, but its total blast radius is nine relations. Recorded rather than fixed.

The reopening condition is recorded with it. The rate is a function of the gate, not of tables, so anything that widens gate coverage pulls far more table notes into the judge’s reach. At that point the classification above has to be re-run rather than assumed to still be 0.3%.

What this does not show

• The oracle is a model, not a human, and it reads the same note the judge does. A fact stated nowhere in the note is invisible to both.
• Relations the two arms treated identically were never scored, so this measures the change, not the judge’s absolute accuracy. 118 of 121 pairs agreed.
• A limit of the oracle is visible in the data: on structurally identical underwriter rows from one table it split. Rows that differ only in the name cannot honestly have different truth values.
• 0 out of 60 on tolerant parsing has a 95% upper bound near 5%. It is not proof the case never happens; the revert follows the rule fixed before the run, that a change with no demonstrated gain does not stay.
• Only one vault has affected notes, so the result does not generalise to a corpus whose tables carry different content.

Measurement record

Item	Value
When	26 July 2026; decontaminated re-analysis 27 July
Harness	tagging/eval_extraction_quality.rs::verify_judge_source_ab and tolerant_parse_rate
Arms	Relation set fixed, only the judge’s input text swapped
Oracle	The full note including tables, three independent votes, 2-1 reported as contested
Decision rule	Fixed in writing before the run: NEW must beat OLD, a tie reverts
Parse measurement	60 calls over a deterministic stride across the vault, sonnet-5

The parse measurement calls the API directly rather than through the proxy. Production goes through an auth proxy, and trailing-token behaviour can differ between endpoints.

No test covers the judge’s source text. Neither the reverted change nor this revert is guarded; the only protection is this record and the harness.

The primary record is docs/details/verify-judge-source-2026-07-26.md in the Consilience repository.