A default-model swap changed the language documents were written in, and the prompt could not fix it
A user report that documents came out in the wrong language was traced to a default-model swap and reproduced across six cells. Five prompt wordings each fixed one case and broke another, so the decision the model could not make was moved into code.
- documents that kept the source language, before and after the swap (Fisher p=4.5e-5)
- 20/20 vs 8/20
- prompt wordings that passed every cell; moving the decision into code then passed 160 of 160
- 0 / 5
The user report was one sentence: documents the app writes come out with strange grammar, as if Korean had been translated word for word into English.
A report like that makes you want to fix the prompt. In this case the prompt was not the cause, and after five attempts it was established by measurement that the prompt could not fix it.
The cause was a default-model swap
The build the reporter was running did not contain the language directive at all. It is tagged two days before that directive was merged. Only one change in that window altered model behaviour: two releases earlier the default model had been swapped, and at that point the model selection was session state that reset on every launch. So up to the previous version it was always the old model, and from that version always the new one. The switch was total and silent.
Using the prompt that build actually assembles, the case of a Korean note asked about in English was run at n=20.
| Model | Document kept the source language | Original quotation preserved |
|---|---|---|
| Previous default | 20/20 | 20/20 |
| New default | 8/20 | 8/20 |
Fisher’s exact test gives p=4.5e-5. The flipped cases are sentence-by-sentence literal translation, retranslating even the words inside Korean quotation marks. It reproduces at the reporter’s actual UI language too.
Quotation fabrication follows as a derived effect. Conditioning on the document’s language, quotations survive 24 out of 24 times when the document is in the source’s language and 1 out of 26 times when it is not. An English statement passing through a Korean note came back out as a sentence with the same gist and different words, still carrying quotation marks and an attribution to a named person.
Splitting the diagnosis into cells
Nothing anywhere said what should decide the language. There are three candidates: the language of the material, the language the user wrote in this turn, and the UI language. Every combination where those disagree became a cell, and all of them were measured.
| Cell | Material | User’s turn | UI language | Correct answer |
|---|---|---|---|---|
| A | Korean | English prose | English | English |
| B | English | no prose | Korean | English |
| C | English | Korean prose | Korean | Korean |
| E | Korean | Korean prose | English | Korean |
| F | English | Korean prose | English | Korean |
Scoring used no judge. A deterministic rule classifies a document as Korean when at least 10% of its letters are Hangul.
Cell E is the decisive null. A user writing in Korean gets a Korean document 30 out of 30 times even when the UI is English. So the report is not “I asked in Korean and got English”. The problem comes from entry points with no prose: buttons, slash commands, suggestion cards, a first turn that is only an attachment. A vault that is 99.9% English with a Korean UI is exactly that cell.
The prompt cannot fix it, five times over
A wording has to pass both cell B (no prose, English vault) and cell F (Korean prose, English UI) at once.
| Wording | B | F | Verdict |
|---|---|---|---|
| No section (before) | English 50/50 | Korean 29/29 | but A is a coin flip |
| The shipped directive | Korean 13/50 | Korean 25/30 | regression |
| 1. Documents follow the material | English 30/30 | Korean 0/29 | rejected |
| 2. Use the material when there is no prose | Korean 25/30 | Korean 19/20 | rejected |
| 3. Code names the vault language, in the header | English 30/30 | Korean 11/20 | rejected |
| 4. The same naming, only inside a conditional | English 17/30 | Korean 19/20 | rejected |
It is exactly a trade-off. Naming the vault language prominently fixes the no-prose case and breaks the prose case; confining it to a conditional does the reverse. All five wordings moved along the same axis.
What that says is not that the wording was not good enough. The step the model cannot perform is not choosing a language but deciding whether this turn contains prose. That decision is not the kind of thing to leave to a model.
The fix moves the decision into code
The rule reduces to one line. A document’s body is written in the caller’s UI language. What the agent says follows the language the person wrote in, or the UI language when they wrote no prose.
Which half applies is decided by code, not by the prompt: a function that strips leading slash commands, wiki links, code spans, paths and URLs, and checks whether two or more letters remain. It is biased toward “there is prose”, because a false positive only makes the model read the turn itself, while a false negative overwrites the language the user actually wrote in.
The two competing rules that were rejected are recorded too. “Follow the source material” fixes one cell 30 out of 30 and hands a Korean speaker an English document. “Follow the asker’s turn” passed the test but for the wrong reason: the wording says “in the language the person wrote”, the person wrote English, and the output was Korean. A single line of UI language in the header had beaten the wording, and re-running the same rule with a different UI language gives the opposite result. A rule that passes for a reason other than what it says is not adopted.
After the fix
n between 20 and 30, 160 documents.
| Cell | Correct | Document | Quotation preserved |
|---|---|---|---|
| The report’s condition | Korean | 30/30 | 30/30 |
| Its mirror | Korean | 30/30 | 30/30 |
| No-prose turn | Korean | 20/20 | 20/20 |
| Happy path | Korean | 20/20 | 20/20 |
| Policy cost 1 | English | 20/20 | 20/20 |
| Policy cost 2 | English | 20/20 | 20/20 |
The last two rows are not bugs but the price of this policy. A user with an English UI working in a Korean vault gets English documents. They are not counted as pass or fail; they were measured to be on the record.
Two other things went in alongside. One is a quotation-preservation clause, independent of any language rule, which fixes a pre-existing defect shared by both models. The other is reverting the default model. That is not an accuracy dependency, since the new model also scores 30 out of 30 once the directive exists, but it is the model every user had until the previous version, it got this right without any directive, and it costs half as much per token. Users who explicitly saved a model keep their choice.
One defect created by assembly order
The directive file began with a single newline, and the preceding section ends without one. The result was that the whole directive parsed as a continuation line of the preceding section’s last bullet. The prior work cited in support of the change was comparing exactly those two shapes, isolated instructions against integrated ones, and this shipped in the integrated shape.
The golden-snapshot test that was claimed to be guarding that file did not exist. The actual guard is a different test, and it caught a real regression immediately after being written.
What this measurement does not show
- Korean and English only. Nothing is known about the other five supported languages.
- “Strange grammar” was not scored directly. What was measured is language selection and quotation preservation. Translationese is a subjective judgement and was not scored without a judge; instead every generated document was kept.
- The no-prose turn is synthetic. Real entry points such as suggestion cards and buttons carry their own seed prompts, some of which are generated in the vault’s language.
- These are not production rates. The harness uses a one-turn synthetic history, and prior work measures confusion rising with prompt complexity, so real rates are likely worse.
- Cheaper models fail differently. In the same experiment one model failed to call the file-writing tool at all in 10 or more of 20 runs. The symptom there is not “the writing is strange” but “no file appears”.
Measurement record
| Item | Value |
|---|---|
| When | 30–31 July 2026 |
| Total calls | 1,760, $70.63 |
| Model | claude-opus-5, compared against claude-sonnet-5 |
| Transport | /chat/completions, the path production uses for that model |
| Arms | Every arm is a prompt built by the real assembly function; arms are byte-identical outside the language section |
| Scoring | No judge. A 10% Hangul-share threshold over the document body |
| Sample | n between 20 and 50 per cell; 160 documents in the post-fix verification |
| Harness | A diagnostic test that dumps the assembled prompt verbatim; lang/round10.py runs the cells in batch |
With prompt caching enabled, 91.6% of input tokens were cache hits and the real cost was about 1/2.4 of list price.
The primary record is docs/details/language-directive-document-language-2026-07-31.md in the Consilience repository.
What this measures
- Consilience
The world's first general-purpose ontology OS — it builds a knowledge graph from markdown documents automatically, and an agent reasons over that graph