A default-model swap exposed an undefined document-language policy; a later harness audit narrowed the claim

Question: Did a default-model swap cause the document-language regression, and was it impossible to fix in the prompt?

Observation: A July 2026 synthetic run recorded a model difference, but an August 10 audit found that it omitted the production prompt tail, used the wrong transport, and never exercised the BARE branch.

Decision: Keep the explicit language policy and code boundary, demote the old rates to historical observations, and rerun them with a production-faithful harness.

1. Separating two incidents behind one user report

The report was that an app-written document read like a literal Korean-to-English translation. The verified sequence is more precise:

Date	Verified event	What the evidence can support
25 Jul 2026 · v1.6.2	Default changed from Sonnet 5 to Opus 5	000d86b0 and session-state behavior explain a silent, broad switch
30 Jul 2026 · unshipped	A separate language directive regressed ask_user card language	It reached no release and did not cause the user report
31 Jul 2026 · implementation	Document policy was made explicit and prose/bare classification moved into Rust	The current code and regression tests preserve this boundary
10 Aug 2026 · audit	The July harness was found not to be production-faithful	Production interpretations of its rates are superseded

The default change fits the report’s timing and a synthetic comparison found different model behavior. The reported “strange grammar,” however, was never scored. The July 30 directive regression was also a separate, unshipped defect. They cannot be presented as one causal chain. The primary record makes that distinction in docs/details/language-directive-document-language-2026-07-31.md §§1 and 7.

2. What the July run actually observed

The historical comparison used a one-turn synthetic history with no language directive: an English prose request about a Korean note, followed by document generation.

Model and condition	Kept source language	Preserved source quotation	Current interpretation
Sonnet 5 · n=20 documents · no-directive synthetic prompt	20/20	20/20	Historical harness observation
Opus 5 · n=20 documents · same synthetic prompt	8/20	8/20	Historical observation; exact count needs reconciliation

The recorded comparison gives Fisher p=4.5e-5 · n=40 documents · 20/20 versus 8/20. Current code comments, however, say that Opus translated 9/20 documents under the same condition, implying 11/20 retained rather than the primary record’s 8/20. The raw generations are not in the repository, so the exact 8/20 count and its p-value remain provisional.

Quotations were preserved in 24/24 historical synthetic documents written in the source language and 1/26 historical synthetic documents written in another language. That is a strong association in this sample. The counterfactual claim that changing language caused the quotation mutation was not measured.

3. Prompt trials: separating observations from conclusions

Five policy wordings were compared in cells where material language, turn language, and UI locale disagreed. Representative historical results were:

Historical wording and synthetic condition	B: English material · bare · ko UI	F: English material · Korean prose · en UI
No language section	English 50/50 documents	Korean 29/29 documents
“Follow the material”	English 30/30 documents	Korean 0/29 documents
Code names material language in the header	English 30/30 documents	Korean 11/20 documents
The same name only inside a conditional	English 17/30 documents	Korean 19/20 documents

This supports only the statement that the tested wordings did not jointly pass the selected synthetic cells. It does not show that a prompt could never work. In fact, Opus 5 with an explicit directive passed 30/30 documents under the same old harness.

The later harness audit is more damaging to the original interpretation. Cell B was designed as bare, but the assembly call passed user_message=None, pinning every cell to the PROSE half. These results therefore cannot measure the model’s ability to distinguish prose from a bare trigger.

4. The retained implementation: policy and code boundary

The current policy has two layers:

1. A document body is written in the caller’s UI locale.
2. Conversation and narration follow the person’s prose language, or the UI locale for a bare trigger.

Code does not generate the language decision itself; it decides which half of the directive to include. turn_has_prose removes a leading slash command, wiki links, code spans, paths, and URLs, then checks whether at least two letters remain. build_system_prompt and the turn_has_prose tests in src-tauri/src/agent.rs preserve that boundary. appended_prompt_sections_start_a_new_block also guards the assembly seam that once attached the language section to the preceding list item.

This is not the discovery of an objectively correct language. It is an explicit product policy. An English-UI user working in a Korean vault pays the cost of receiving English documents, and there are no outcome data yet for ja/zh/es/fr/de beyond the measured ko/en pair.

5. The August 10 audit that superseded the interpretation

The follow-up audit compared what the old evaluator assembled with what production actually runs.

Earlier record	Audited condition	Status
“The real assembled production prompt”	Omitted memory, live entity index, DESIGN.md, and /goal clauses appended after build_system_prompt	Retract production interpretation
“Production Anthropic /chat/completions”	Every 30–31 July recorded value came from /responses	Correct the transport record
“Bare cell measured”	user_message=None pinned every cell to PROSE	Retract BARE result
“160/160 after the fix”	n=160 successful documents came from the same incomplete harness	Retain only as historical regression output

The evidence is in the “Assembly (corrected 2026-08-10),” Transport::resolve, and runtime_context notes in src-tauri/src/agent/eval_card_language.rs. The code explicitly says that runtime_context=none reproduces the July setup byte for byte, and that no user with an open workspace receives that setup.

6. Claims retained and questions reopened

Retained: the v1.6.2 default-model history, the absence of an explicit document-language policy, the separation of document and conversational language, the deterministic prose/bare classifier, and the assembly-boundary test are verifiable in history and current code. 20/20 versus 8/20 and 160/160 stay on the record as historical harness observations so the failed evaluation is not erased.

Not currently claimed: that the model swap caused every part of the user report; that prompt engineering was impossible in principle; or that the old numbers estimate a production failure rate or post-fix success rate.

A valid rerun must use the full system prompt after append_runtime_context, the real Anthropic transport and streaming conditions, real button/suggestion-card/attachment entry points, and all seven supported locales. It must retain prompt dumps, request conditions, raw generations, and per-cell classifications so the exact count can be audited.

Reproducibility and evidence boundary

Evidence	What is currently verifiable	Boundary
docs/details/language-directive-document-language-2026-07-31.md	July design, aggregates, and the rationale used at the time	Raw generations and the batch script are absent from the repository
Git 000d86b0, fc294bdd, and v1.6.1–v1.6.3 tags	Order of the default change and later preference persistence	Does not prove the full causal effect on the user report
src-tauri/src/agent.rs, prompts/system/language*.md	Current policy, classifier, and assembly tests	Code tests do not establish model compliance rates
src-tauri/src/agent/eval_card_language.rs	Missing runtime tail, wrong transport, and unmeasured BARE branch	No new production-faithful result has been recorded yet
Public 20/20 versus 8/20 and 160/160 figures	Historical old-harness aggregates	Do not cite as production rates; 8/20 conflicts with current code comments