Youngtak Jo
Research
Measurements written up in full: what each was compared against, what the statistic was, and what the result does not show. The null results, the failures to reproduce and the changes that were measured and then reverted are here too.
-
Method
Diagnosing dense-retrieval hubness with a 2010 distance-concentration result, and porting a cross-lingual correction into retrieval
Three intuitive alternatives were tried and all three rejected on measurement. The fix came from another field and cost nothing extra in API spend. Re-measured at 300× the scale it turned out to lose in one place, and the repair is recorded here too.
- 77.5 → 97.0 dense seeding recall@10; keyword-only scored 85.2 on the same set
- $0 added API cost; end-to-end answer accuracy 87% → 97%
-
Method
A defect audit with a refutation round — from 45 candidates to 14 reproducible defects
Suspicions were turned into runnable reproductions, every defect was paired with a control arm, and independent skeptics tried to refute each one; only survivors were recorded as defects. The largest finding was not on the plan, and surfaced only because the audit read a live installation.
- 45 → 14 candidates to reproducible defects; 14 of 14 survived refutation
- 102,020 quads lost silently, found by the audit and now covered by a regression test
-
Benchmark
Two document-parsing benchmarks — first on one, mid-table on the other, and why
A general-purpose VLM with a prompt on top, not a fine-tuned document parser, took first place on DP-Bench table structure. The same pipeline is mid-table on OmniDocBench; the loss is diagnosed, and a scoring artifact is traced down to the 49 documents that produced it.
- 98.11 DP-Bench table structure TEDS-S, first on the official leaderboard
- 79.23 the same pipeline on OmniDocBench Table-TEDS; the leader scores 93.42
-
Method
Extracting more does not move retrieval — a null result, and a significance that did not replicate
A paired A/B asked whether a gain observed on fixtures survives in a 3,002-note vault. It did not, and the one effect that did appear lost its significance on an independent repeat. Both results are on the record, and the citation rule changed as a result.
- +0.1pp retrieval gain bought by 17% more relations (p=0.84, pooled over five runs)
- 0.038 → 0.478 the p-value on two independent runs of the same design and corpus
-
Method
Measuring a shipped commit, then reverting it
One commit that shipped without measurement was measured afterwards. Two of its three changes lost or bought nothing, and all of them were reverted. A separate defect the experiment exposed was left unfixed on purpose: its measured blast radius is nine relations.
- 0 / 3 times the new implementation matched the oracle; reverted under a rule fixed in advance
- 0 / 60 calls the tolerant parser actually recovered; the failure it targeted was out of its reach
-
Method
A default-model swap changed the language documents were written in, and the prompt could not fix it
A user report that documents came out in the wrong language was traced to a default-model swap and reproduced across six cells. Five prompt wordings each fixed one case and broke another, so the decision the model could not make was moved into code.
- 20/20 vs 8/20 documents that kept the source language, before and after the swap (Fisher p=4.5e-5)
- 0 / 5 prompt wordings that passed every cell; moving the decision into code then passed 160 of 160
-
Benchmark
MuSiQue multi-hop QA — the graph is a precondition, not an improvement
The graph condition and a keyword-only condition, compared on the same question set. The gap widens with the number of hops, and at four hops the control arm scores zero.
- 63.0% vs 23.0% questions with every supporting paragraph retrieved (ALL@10), graph vs keyword-only
- 0.0% of 17 four-hop questions, none succeeded without the graph
-
Benchmark
SWE-bench Verified across all 500 tasks, and what that number cannot claim
The result of running all 500 tasks on the official harness. That run cannot separate the scaffold's contribution from the model's, so the separation was measured in two controlled experiments instead.
- 78.80% 394 of 500, official harness, no retries
- 54% → 76% model and 50-task set fixed, selection procedure swapped