A single feature brings together requirements, implementation and interface design. Consilience is a general-purpose AI agent that uses a knowledge graph for coding, design and answering questions. Users can find evidence in their materials, revise the relevant code, and create interface mockups and presentations in one workspace. The knowledge graph supplies context for the agent’s work.
The underlying ontology organises concepts and relationships found in documents. To let users build and maintain a document-based ontology without a resident, forward-deployed engineer (FDE), I connected document import, concept and relationship extraction, updates and duplicate resolution. Users can ask the agent to run this process, inspect the sources and proposed changes, and approve proposals that alter meaning.
I started building it independently in November 2025, from the knowledge engine and agent runtime to the editor, design canvas, application releases and subscription billing. The central problem was helping the agent find and use the knowledge a task needs.
A folder of files was only the beginning
Giving an agent access to documents lets it read them. But “Which customer has filed the most claims?” is a different task from finding a file. It requires gathering claim records, recognising the same customer across them, counting the records and joining them to customer information. The documents can all be present while those connections still have to be reconstructed for each question.
I separated the documents from the relationships extracted from them. Ordinary Markdown remains the record a person reads and edits; a separate index holds the facts and their connections. Once a claim’s relationship to a customer is explicit, the next query can follow that relationship or aggregate records. RDF is the format of this knowledge graph, and SPARQL provides its joins and aggregation.
This was also a decision about ownership. A person’s writing should remain usable outside the application. Keeping Markdown at the centre lets people and agents work on the same record and compare derived relationships with the source. Extraction still uses an LLM, so rebuilding from a document does not promise an identical set of relationships every time.
The facts were there, but retrieval missed them
Building a richer graph seemed like a straightforward way to improve retrieval. Tables and catalogues exposed a less convenient result. The required fact was already in the graph, but semantic search filled its highest positions with similar-looking entries and pushed the target down. The information had been captured; the search could not bring it back.
I first tried fixed similarity thresholds to stop those entries crowding the results. A threshold that helped a dense table also blocked useful matches in prose. The neighbourhoods had different densities. I connected the failure to research on hubness in high-dimensional search and adapted CSLS, a correction used in word translation, to the selection of starting points for graph retrieval. A candidate’s score now accounts for how broadly similar it is to its neighbours, as well as how close it is to the question.
In the initial fixtures, recall@10—the retrieval measure at the first ten results—rose from 77.5 to 97.0. Expanding the search space to 100,000 labels exposed another failure. The early results improved, while recall@40 got worse. When the target itself belonged to a dense group, the correction could suppress the very item the search needed.
I added a safeguard that preserves the strongest candidate from the uncorrected similarity ranking. It reduced the damage, but a −4.9 percentage-point regression remained at 30,000 and 60,000 labels. I reported that too. A small successful experiment could not certify the quality of the entire retrieval system. The retrieval and scale study records both the adopted method and the remaining loss.
A memory also needs a way to correct itself
As the graph grew, incorrect connections became more consequential. If “the Company” in unrelated corporate documents is treated as a shared name, separate companies can collapse into one identity. Tightening the rules too far creates the opposite problem: a Korean name and its English equivalent remain disconnected. More connections do not necessarily mean better connections.
Rather than destructively merging entities, I stored reversible links between them. When a user marks two entities as distinct, that decision is retained and prevents them from being automatically joined again. I also separated extracted document assertions from machine-generated inferences. A relationship can be traced to its document, and an incorrect inference can be withdrawn without rewriting the author’s text.
These correction paths became as important as the retrieval algorithm. Memory has to respond when a document is edited, removed or superseded by a later decision. EngramRAG is the architecture that connects this lifecycle to search and the agent’s work.
Misreading a table can undermine everything after it
Document import could not remain an incidental feature. Flattening a PDF table into plain text can lose the relationship between a value and its heading. The conversion records include a comparison table whose check marks disappeared during text extraction. Feeding that result into a graph would let a good search engine confidently retrieve a misread fact.
I built format-specific converters to preserve merged cells, headings and chart data. Parsers read documents with accessible structure; vision models handle scans and complex visual material. A summary does not replace the original document. Detected conversion failures and truncation produce warnings, and the original source is preserved. Support for 22 file formats grew out of this need to protect retrieval before a query ever reaches it.
A separate document-parsing evaluation scored 96.34 TEDS and 98.41 TEDS-S in one locked run on all 200 public DP-Bench documents. These are candidate results for table content and structure, with official listing under review. The document-parsing record gives the evaluation conditions.
Using the retrieved context to do the work
I connected retrieved knowledge to practical tasks. To answer questions, the agent queries or aggregates relationships in the knowledge graph. For coding, it follows definitions and calls to locate the relevant implementation. I extract code structure with tree-sitter and provide tools for editing files and running builds and tests in the terminal.
The same agent handles design. It creates HTML/CSS interface mockups and presentations that render in a canvas, where users can select an element and request changes. The rendered image also returns to the agent for inspection. This lets the user and agent review the visible result and continue refining it together.
To test whether structure helped, I held the model, tasks and agent fixed. Adding the code graph changed success at finding the relevant code from 14/21 to 18/21. In a separate full SWE-bench Verified run, the model and complete execution system together resolved 394 of 500 tasks, or 78.80%. The first comparison investigates the contribution of structure. The second measures the integrated system; its score cannot be credited to the graph alone.
The same agent can use document and code retrieval, but the two graphs are not a single unified store. Connecting a past written decision directly to its current implementation remains a separate evaluation target. The EngramRAG research record distinguishes the implemented paths from the questions still open.
Turning experiments into an application people can keep using
A successful retrieval demo still needs work before it becomes a usable product. I built a 3D graph so people could inspect the relationships and wrote a WebGPU physics engine to handle large graphs. Its measured physics time at 500,000 nodes was 18.3 milliseconds per tick. That is a measurement of the physics computation, not a claim about every operation in the application.
The application ships as signed macOS and Windows releases with automatic updates, and is also available on the web. It includes a seven-language interface, live collaborative editing, administration and four subscription tiers.
The initial question was why a person had to keep explaining the same background to AI. Building Consilience made that question more precise: can the system read the necessary facts faithfully, follow their relationships to the evidence, and let a person correct a bad connection? The project has been the work of bringing those requirements together in one workspace.
