A defect audit with a refutation round — from 45 candidates to 14 reproducible defects
Suspicions were turned into runnable reproductions, every defect was paired with a control arm, and independent skeptics tried to refute each one; only survivors were recorded as defects. The largest finding was not on the plan, and surfaced only because the audit read a live installation.
- candidates to reproducible defects; 14 of 14 survived refutation
- 45 → 14
- quads lost silently, found by the audit and now covered by a regression test
- 102,020
This was an audit of the file lifecycle, the graph lifecycle and application usability. What an audit report usually produces is a list of suspicions. This one fixed its rules first, specifically in order not to.
The rules
A suspicion is not a defect. Each candidate had to become a runnable reproduction before it counted as one.
Without a control arm, nothing has been proven. Every probe pairs a bug arm with a control arm. A probe with only a bug arm cannot distinguish “this code is designed that way” from “this code is broken”.
Get the baseline green first. Both test suites passed at the start of the audit: 982 frontend tests across 85 files and 1,010 backend tests, zero failures. Only then does a newly failing assertion point at a defect.
State the evidence grade on every conclusion.
| Grade | Meaning |
|---|---|
| Live UI | The real application booted in a browser, with the user’s actions reproduced |
| Live logic | A shipped function called directly from a test, or private logic transcribed verbatim and executed |
| Deductive trace | An exhaustive call-graph survey plus a contrasting sibling path that does have a guard |
| Live forensics | A real installation and a real vault, measured read-only |
The same conclusion carries different weight depending on which grade produced it, and leaving the grade out erases that difference.
The refutation round
45 candidates were triaged down to 14, and all 14 were verified experimentally. Then 14 independent skeptics each took one and tried to refute it.
All 14 survived. But most were downgraded in severity. The refutation round did not delete defects; it cut them down to size, which is what the round is for. An audit assembled without one is a list with confirmation bias still in it, and its priority order cannot be trusted.
What the audit left behind is executable, not prose
| Artefact | Nature |
|---|---|
| 11 backend probes | Passing means the reproduction works. They assert the bug exists, so they fail once it is fixed |
| 10 frontend assertions | Failing means the defect exists. They assert correct behaviour, so they pass once it is fixed |
| 2 e2e specs | Tab lifecycle. Three arms flipped to passing after the fix; two arms still assert the open defects |
| A 6-arm permanent regression spec | Created with the fix. All six arms were confirmed failing beforehand by running them individually |
The assertion direction differs between files. Some probes assert the bug and some assert the correct answer. Promoting one to a regression test without flipping its direction leaves a test that keeps passing after the fix and proves nothing. Each item states which kind it is.
The largest finding was not on the plan
The rewind feature was not in the audit plan. But while reading the database of a real installation, this came out: a single rewind had permanently deleted 102,020 facts across 1,057 documents that the rewind never touched.
The mechanism is this. A rewind compares each document’s graph against a snapshot in order to roll files back, and it reads “present in the list of affected documents, absent from the snapshot” as something to delete. A document first touched by a later turn inside the rewind’s range does not exist in the anchor snapshot, so it is handled as a deletion rather than a rollback.
It could not be undone, because the deletion itself passed the hash gate.
This defect does not appear on fixtures. In a scenario built from one or two documents the affected set and the snapshot agree. It appeared because a live installation was read, which is why live forensics is on the evidence-grade list at all.
A regression test that fails if the fix is reverted shipped with the fix. It asserts directly that a document first touched by a later in-range turn survives.
Something fixable that was deliberately not fixed
The same audit found that checkpoint snapshots over-capture. Measured, not estimated.
| Item | Value |
|---|---|
| Total snapshot store | 145 MiB |
| Of which a rewind can actually read | 0 B, across all five checkpoints |
| Relative to note bodies | 18.7× the size of every note |
| Actually needed for an ordinary edit | Median 7,649 B per document |
| Growth | 11.2 MB per day, with no cap |
Reading zero bytes is structural rather than coincidental. Every recorded checkpoint is a creation, so all the restore has to do is detach that graph, and that needs only an identifier.
Every patch that narrows the capture was nonetheless held. For two reasons. First, narrowing the capture to the turn’s own paths brings back the data loss that had just been fixed, from the other side; narrowing safely requires carrying the snapshot hash per path first, which is a larger change than the fix itself. Second, a test that fails today cannot be written: the function takes an application handle and no harness for that exists in the repository.
Two things were left instead. One is a guard test that passes today and fails the moment the capture is narrowed without per-path attribution. It was confirmed by simulation: removing one line from the anchor snapshot made that document disappear from the output graph entirely. The other is an explicit list of triggers that reopen the decision — the store passing 1 GB, one conversation reaching the checkpoint cap, write-turn frequency rising past a stated level, or a vault appearing under a cloud-sync folder — together with a non-negotiable order of work for when it does.
Measuring the capacity problem first changed the ranking. A different file in the same folder is larger at 319 MB, grows daily rather than per write turn, and grows twice as fast. That one comes first. And the first draft’s framing that this “rides the user’s backup and sync” was false on this machine: no backup target is configured and the vault is not on a cloud path. The claim was checked, corrected and left in the record as a correction.
What this audit does not show
- It proves defects exist, not how often they occur. The probes show a defect reproduces; how frequently it fires in real use needs separate instrumentation.
- One machine, one user. The live-forensics numbers belong to that installation.
- The 31 candidates dropped in triage were not revisited. They were filtered on a severity judgement, and that judgement itself was not validated.
- The refutation round was human. Fourteen people each took one item, and what they collectively missed is unknown.
Measurement record
| Item | Value |
|---|---|
| When | 31 July 2026; the snapshot measurements 1 August |
| Base commit | 96aaf8e6 |
| Baseline suites | 982 frontend tests across 85 files, 1,010 backend tests, zero failures in both |
| Live forensics target | A real installation’s 1.2 GB database and real vaults, read-only |
| Probes | src-tauri/src/audit_probe.rs (11), tests/unit/audit_*.test.ts, tests/e2e/audit_*.spec.ts |
| Fix commit | 1cd80797 for the rewind data loss |
The probe output is deterministic, and each line comes as a bug-arm and control-arm pair. The probe for quads disappearing after a file move, for instance, prints both that the bug arm leaves the new and old graphs at zero and that the control arm leaves the new graph at six. Without both lines side by side there is no way to tell which is the correct behaviour.
The primary record is docs/details/lifecycle-usability-audit-2026-07-31.md in the Consilience repository. It also carries the list of items the refutation round rejected, so the same proposal does not get reviewed twice.
What this measures
- Consilience
The world's first general-purpose ontology OS — it builds a knowledge graph from markdown documents automatically, and an agent reasons over that graph