Our family handles tax correspondence, insurance paperwork, housing issues, school matters, medical documents, and legal disputes. That includes reading and writing in German, which is not our first language. Behind this work is an archive of more than 6,600 source files.
Even a short reply can require finding earlier correspondence, gathering supporting documents, and working out what the sender expects. In their research on administrative burden in dealings with government, Moynihan, Herd, and Harvey distinguish the costs of learning what is required, complying with those requirements, and dealing with the psychological strain of the process. The preparation and follow-up around a letter are part of the work I want help with.
I began using ChatGPT and Claude for practical work with family documents: reading letters, discussing what they meant, and drafting replies. A letter would arrive, I would upload it, and the conversation would continue through several rounds of discussion and revision.
This worked well within a single chat. The more interesting problem appeared over time. Correspondence continued, new documents arrived, and the chat became only one part of an ongoing matter. Some conversations grew slow or reached a limit. Moving to a new one meant deciding what to carry forward. The model’s context window held the material for that session; I needed a record that could outlast it.
A summary helped, but it left me choosing what a future conversation might need before I knew the next question. This was lossy summarization: it could preserve the main argument and omit a small qualification. It also could not replace the full documents.
I began treating continuity as an engineering problem. The system needed to preserve the work, find relevant material when needed, and let an agent check how a conclusion had been reached. That investigation gradually became FamilyBoon, or Boon, the system I am building around our family’s documents and context. Context engineering describes this work well: deciding what information to keep, what to load into a session, and how to use it.
My goal is for Boon to take on more of the search, reconstruction, drafting, and follow-up work while asking less of our attention. The time we spend using and maintaining it should be more than repaid by the work it takes off our hands. That should give us the choice to handle more matters or spend the time outside the system.
A summary carries only part of the work.
A conversation is a workspace
The first change was deciding what should survive a conversation.
An incoming letter is part of the record. So is the reply that is eventually sent, followed by the next letter. Their order matters: it shows what prompted a response and what happened afterwards. I think of these files as artifacts, including work still in progress.
The chat is a temporary workspace around that record. Its exact wording does not need to become the permanent memory of the system. A later model can examine the same material again, perhaps with better reasoning or a different question.
An unfinished draft still needs to survive. Closing a chat should not require finishing the entire matter first. If a useful explanation or decision needs to carry forward, it can become a saved note or part of the draft, with enough context to understand it later.
I call this artifact-first design: preserve the artifacts and their sequence, and let conversations work from them. Durable artifacts provide a basis for cross-session memory. The next chat should be able to continue from the documents and saved work without relying on a retelling of the previous chat.
The LongMemEval benchmark tests whether assistants can recall information, combine it across sessions, handle updates, reason about time, and recognise when the history contains no answer. In the ICLR 2025 paper, GPT-4o answered 87% of questions correctly when given only the sessions containing the evidence. When given the full history, roughly 115,000 tokens, it scored 60.6%. Keeping records available and choosing what to load into a session are separate jobs in Boon.
Related work on file-based context in Manus and artifacts between agent sessions at Anthropic addresses similar continuity problems. In Boon, the record follows family matters through successive documents and conversations.
Finding the right material in a mixed archive
Our family already had more than a decade of digital documents. Some were organised into folders; others had accumulated in mixed piles. There were PDFs, emails, Word documents, spreadsheets, scans, and photographs.
I wanted the archive to grow without making every conversation grow with it. The agent needed to find and load relevant material as a question developed. Keeping the original files unchanged was an architectural invariant. Search and extraction could produce new representations; they should not alter the documents themselves.
I built a searchable representation of the archive so the agent could retrieve relevant material and use it to answer a question. In this retrieval-augmented generation (RAG) workflow, each search result also needed a route back to the source.
Consider a scratched car as an illustrative example. The archive might contain photos of the damage, a repair estimate in an email, a scanned invoice, and correspondence with an insurer. A person would connect them through the incident, even if the files used different words.
I experimented with extraction tools and settled on native text extraction where possible, with Apple Vision for optical character recognition (OCR) on scans and photographs of text. Text-based PDFs go through pdftotext; emails and Word documents use their own parsers.
The extracted text goes into a Markdown document sidecar with a reference to the original and its SHA-256 content hash. For plain-text originals, the system can read the file directly without storing a second copy of its contents. Sidecars can be regenerated when extraction improves. The source reference and hash preserve provenance: which file the text came from, and which version of its contents was processed.
Different files, different reading paths.
Every route retains a source pointer and SHA-256 content hash. The original stays unchanged.
For semantic search, the current implementation uses multilingual bge-m3 embeddings, SQLite for storage, and NumPy to calculate cosine similarity. The index is built ahead of time, with incremental indexing as the archive changes. Through the Model Context Protocol (MCP), the agent can use tool calling to search during a conversation.
The precomputed index supports just-in-time retrieval. The agent searches when it needs information, follows the returned references, and can make another query as the question develops. Keeping the index ready avoids extracting and embedding the whole archive for every question. Additional searches and source reads still take time.
Semantic search helps when the question and a document use different language. Exact-match search provides another route for identifiers: the search tool has a separate substring-matching channel for identifier-like queries. File inventories and links between records help extend the search. Finding several relevant passages does not establish that every relevant document has been found.
Applying the same principle inside the chat
Daily use exposed failure modes that began before the model formed its answer. OCR could miss characters in a poor photograph. A flattened table could place an amount beside the wrong label. Handwriting could turn into a plausible word that was never written.
I went through several rounds of improving extraction. That helped, but I did not want the reliability of the whole system to depend on producing a perfect text version of every document. I reconsidered how the agent should use the extracted text.
The artifact-first principle applied here too. I had already made the original documents the basis for continuing work across chats. They also needed to remain the basis for claims made inside a chat.
I assigned different jobs to the representations. The sidecar and index help locate a document. Before using a document in a chat, the agent must open the original and check the content it will rely on. An unreadable passage remains uncertain. This is an operating rule for the agent; it does not guarantee that a model will follow it on every occasion.
In Boon, source grounding includes original-source verification: the agent checks that the original supports the claim. A source link alone does not perform that check.
Gao and colleagues separate answer correctness from citation quality in ALCE, their 2023 benchmark for answers with citations. They check whether the cited passages support the answer’s statements and whether individual citations are relevant. A passage can discuss the right subject without supporting the particular amount, date, or conclusion in the answer. Boon also requires the agent to check the original passage, table, or image, since extraction can lose details or their relationship.
Finding a source is not checking a claim.
01Retrieve
NumPy cosine similarity
02Verify
With this separation of concerns, I could improve extraction and verification independently. Better extraction helps find the right documents. Returning to the original helps catch errors in the text used to find them. Neither removes the need for the other: a document missed by search cannot be rescued by checking only the documents that were found.
The value belongs to a row, not just a search hit.
| Item | Value |
|---|---|
| Item A | 40 |
| Item B | 90 |
The original also has a specific kind of authority. It establishes what its author recorded. The author may still be mistaken, and another document may contradict the claim. Boon therefore needs to preserve attribution and conflicting accounts, rather than turn every statement in a letter into an accepted fact.
Boon makes mistakes. Human work needs checking too. My wife once found three factual errors in a lawyer’s initial court application. In a separate matter, another lawyer asked for our mail to be forwarded to us, but gave the wrong address in the same letter. Boon caught the mismatch and helped us draft a request for correction. The lawyer then corrected the address. With Boon, I can begin working through documents when a question arises, without waiting for an appointment. Specialist judgment still has a role. My goal is to reduce mistakes across the whole process, make them easier to detect, and use what went wrong to improve the system.
Documents need context around them
The car example also shows another limit. A photo of the damage might contain no text at all. OCR cannot describe the scratch, and a text embedding index does not automatically understand the image.
A model can inspect the photo visually. Without context, though, it might describe the car’s colour, a street lamp, and a bush in the background. Those observations may be accurate and still miss why the photo was taken.
The person adding it knows what happened and which part matters. Related correspondence may explain more. That context needs to be stored somewhere, without being presented as text extracted from the photograph.
I had already spent years using Zettelkasten ideas and linked notes in Obsidian. I adapted that approach into cards for people, organisations, and ongoing matters. Here, a matter means a question or situation that continues over time, such as a repair, an application, or a project.
A matter card connects the participants, relevant documents, timeline, current state, and next actions. In the car example, it can connect the damage photos to the estimate and insurance correspondence. The insurer has an organisation card; the people involved have person cards. Those cards can be linked from other matters too. This entity-centric context gives each person or organisation a reusable place in the archive, while matter cards describe the situations connecting them.
The current architecture separates these responsibilities:
| Part | Role |
|---|---|
| Original files | Preserve the documents and images in their original form |
| L1: source access and extraction | Provide source references, hashes, and extracted text where needed |
| L2: people and organisations | Describe participants and their relationships |
| L3: matters | Keep the history, current state, evidence, open questions, and next actions of an ongoing matter |
The cards are indexed as well. An agent can find the matter card and follow its links to the photographs, even when the photographs have no searchable text.
This also makes tentative connections possible. The date and appearance of an older photo might suggest a link to a known incident. The agent can investigate that link using the surrounding records. A plausible match remains a hypothesis until it has been checked.
The matter connects the records.
Why the photo matters: it documents the damage involved in this repair. That context belongs to the matter, not to OCR output.
Original files + L1 source access. Extracted text where appropriate.
Keeping the records useful
Cards add context, but they also create derived state that needs maintenance. A status can become stale. An interpretation can be wrong. Each card needs links back to the material from which it was built. Unlike a search index rebuilt from its inputs, a card can contain context supplied by a person and interpretations assembled from several sources. Updating it requires preserving those distinctions.
When a new document enters Boon, the document ingestion pipeline registers the original, extracts text where appropriate, and updates the relevant cards before refreshing the index. The conversational model does the contextual work of identifying people, relating documents, and updating the cards. The next session can then discover the new material through the same matter and source references.
Processing a document can also enrich the context around it. A letter may supply an organisation’s contact details, clarify a person’s role, or connect an existing participant to another matter. The relevant details are added to the cards with references to their sources. Later conversations can reuse that context instead of extracting the same information again.
A new document changes the context, not the original.
Integrity checks and curation revisit the derived records. Source files remain unchanged throughout.
Alongside intake, integrity checks look for missing files or sidecars, changed source hashes, documents without links to matters, and a search index that has fallen behind. A separate curation process maintains links between people, organisations, and matters. These checks can flag inconsistencies; they cannot establish that every interpretation in a card is correct.
A daily digest uses the matter cards to show what needs attention, what is approaching, and what is waiting for a reply. The aim is to reduce what we have to keep in mind, without turning every stored detail into another task. The update rules, integrity checks, and digest form an operational loop that deserves a separate account.
The working archive consists of ordinary local files, synchronized through Google Drive. We have also created an encrypted backup in Proton Drive and tested restoring it. Text extraction and search run locally.
In the native client, prompts and responses travel directly between the app and the AI provider. Material read by a cloud model still leaves the device, including context used during document intake. The original files and Markdown records remain accessible independently of the model.
Local records. Explicit cloud boundaries.
Folders, linked notes, and document retrieval all contribute to this setup. I use them together with explicit rules for preserving artifacts, updating context, and checking sources. Storing a file or retrieving a relevant passage is only part of the work needed to continue an ongoing matter.
The family determines what is relevant
As the cards developed, another design question emerged: how much of a source belongs in a card?
Take a disagreement about a broken built-in dishwasher. A long rental agreement might contain a relevant clause, alongside many details unrelated to that question. The matter card can include the useful passage and a reference to the full agreement. The agent can return to the surrounding text when needed.
I began applying that selection rule across the system. Context starts with the people in the family, their circumstances, and the matters they are dealing with. Those connections determine which passages to bring into cards, which records to link, and what to revisit.
This is the family-centred principle of Boon. The same source can be relevant to different people or matters for different reasons. A single general summary cannot anticipate all of those uses. Preserving the source allows each use to remain specific without losing access to the rest.
The principle also gives the archive a direction beyond managing paperwork. Family documents describe circumstances and events; books, articles, and conversations with specialists can supply knowledge relevant to them. In the family knowledge base, I organise that material by topic, linked to the people and questions that make it useful. One note can draw on several sources, and one source can contribute to several notes.
Those notes keep source references and explain when the advice or findings apply. Source quality and family relevance remain separate checks: useful-sounding advice can be poorly supported, while a well-supported finding may concern different circumstances. Ideas without adequate sources stay in a research backlog. This creates a route from a family question to a topic and back to the material behind a claim.
A topic connects sources to a family question.
Source references lead back to the material behind a claim.
This is also a foundation for the proactive side I want to develop: using current family context and relevant knowledge to judge when new information deserves attention. How that knowledge should evolve, and when an agent should make a suggestion or wait, are subjects for later articles.
Continuing from the existing work
When the next letter arrives, I can start a fresh conversation and ask Boon to look up the matter. The agent can read its card, follow the correspondence, check the relevant originals, and work from the saved draft. The history no longer has to arrive through my recollection of the previous chat.
A fresh conversation, the same working record.
I still need to make this more reliable and easier to maintain. New features can also create work, so I want to judge each change by the work it saves us after accounting for the effort of maintaining Boon itself. The goal is for either of us to open a matter, understand what changed, and take the next step without spending the evening reconstructing its history.