Where the answers come from

Part 5 of a series from six weeks building AI systems by hand: an operator going beneath the abstractions, on purpose. These posts carry more machinery than executive writing usually does. That’s deliberate.

In week three I asked a system I had built a counting question: how many of Project Orion’s weekly status updates flagged it At Risk. It answered three. Three was correct. I refused to score it a pass.

Part 3 ended on a finding and promised it a post of its own: the quality of an answer is set less by the model than by what the evidence layers feed it. This is that post, told through the corpus builds, where the finding cut deepest. The failures worth writing about were invisible from the answers: the trouble lived underneath, in what was searched for and what came back, and the answer never once told me.

The setting was week three’s build: retrieval-augmented generation, RAG — a corpus embedded so the system can search it by meaning and hand the best passages to the model as evidence. Mine was the internal documentation of a fictional infrastructure company: 450 documents of the messy, shared-drive kind, seeded with lookalike decoys, chunked to 2,616 passages, each embedded as a vector so that a query retrieves its nearest semantic neighbours. The test questions came with answer keys naming the exact documents that hold each answer: the gold documents. The index was wrapped as a tool, search_redwood_docs, that the agent loop from part 3 could call as often as it judged necessary. And one line of my config sets the window everything else must fit through: SEARCH_K = 5, five chunks per search — chosen deliberately, on the reasoning that an agent can always search again. Five, and no more.

The lucky answer

The Orion question needed six documents: six weekly updates, three of which flag At Risk. The search came back with every one of its five slots filled with gold documents. Perfect precision, and arithmetic that cannot work: six documents do not fit through a five-slot window. The agent read the five it had, counted three At-Risk weeks, and answered. It never noticed the series was incomplete. It searched once, and never went back. Nothing in five weekly updates says how many weeks exist.

The sixth week happened to be On Track. A sixth At-Risk week sitting outside the window would have made the same confident “Three” wrong. The process could not have told the difference, because nothing it retrieved said the series went on. So the verdict I logged that evening was lucky: right answer, broken process. My own answer key had been wrong first — I had written “gold: three docs” in my notes when the label listed six. The labeller has to count what the agent must see, not what the answer says.

The query said At Risk, so the three At-Risk weeks ranked highest — those documents are saturated with risk language — and the same mechanism pushed the one upbeat week out of the window. Helpful luck, and luck all the same.

The operating consequence: spot-checking answers cannot catch this class of failure, and spot-checking answers is how most AI deployments get checked. Retrieval needs its own score — which gold documents did the system actually see — with lucky as a verdict distinct from correct. The stakes are structural: the engineering course’s claim, which my sprint kept illustrating, is that some 80% of RAG failures are retrieval failures. Retrieval sets a ceiling the generator cannot rescue.

Four searches, wrong shelf

The hardest question of the sprint had no answer key at all: the normal condition in a real company, where labelled questions are the luxury. Which team received the most follow-up actions across the corpus’s twenty-five incident postmortems? The agent searched four times, retrieved seven documents, and refused to answer. Every one of the seven was a document about the postmortem process: templates, playbooks, ownership guides. Zero of the twenty-five actual postmortems. The first query, verbatim from the trace, shows why:

Embedding search retrieves by proximity: it finds what sounds like the question. That is its power, and this query is its failure surface. Every word comes from the question’s own aggregate register, and no postmortem is written in that register: each one describes a single incident and never says “follow-up actions across”. The quoted phrases are keyword tactics on a semantic engine, which reads the quote marks as just more characters. Nothing had ever told the model what kind of search sat behind its tool, so it used the grammar it knew. The cheapest repair in the whole incident is a sentence in the tool’s description: name the engine, and say queries should read like the passages you want back. On the third search the agent did the sophisticated thing and reused vocabulary from its earlier results. The borrowed words came from the wrong shelf, so the reformulation dug further into the same trap. Bad first results poison the escape route, and retrying is not diversity: different phrasings of the same register land in the same neighbourhood.

Beneath the queries sits a harder truth: perfect search could not have answered this. Ninety-six follow-up items spread across twenty-five documents do not flow through a top-k pipe, however the question is phrased. Aggregation over a corpus is a query — enumerate and tally, in code — not a retrieval. Orion was the small dose of the same disease, six documents nearly fitting through k=5, the failure invisible and the answer lucky; this was the full dose, and the failure undeniable. And the refusal was the discipline holding: the agent had process documents in hand and declined to name a team from them. A worse-behaved system answers confidently here, and nobody ever knows.

Then I tallied the real answer myself, agent-assisted, with the counting rule fixed before I started. A tie, twenty-five each for two teams, and a fragile one: seven items name people rather than teams, and merging one sub-team into its parent, or not, changes the winner. The question sounds like it has one answer. It has one per rulebook, and someone has to own the rulebook.

The answer was in there the whole time, and so was everything else the suite missed that week. A credentials question dropped three of six gold documents: one at rank seven, just outside the k=5 window, two filed under vocabulary nobody asking the question would use. The knowledge existed, in a corpus I controlled.

Your AI doesn’t have a knowledge problem. It has a filing problem, and the filing is the part you own.

Where knowledge should live

Each failure above has a fix, and the prices span from a config edit to a standing engineering programme:

Two rules govern the spend. Raising k is not free depth: more context is noise, not knowledge, and even the brute-force end of that road fails. Models start degrading at 60–70% of their stated context window, long before “just give it everything” runs out of room. And training facts into the model, the fix that surfaces first on most roadmaps, is the most expensive dial on the panel and the wrong one for facts: the model cannot tell you where it learned something, and updating it means owning a model lifecycle forever. I order the fixes cheapest first, and touching the model comes last, usually never. The model is rented intelligence; the corpus is owned knowledge. In most organisations nobody owns it, and the retrieval failures land on whoever trusted the answer.

The artefacts to ask for

Three things I now ask of any system that claims to answer from evidence, mine or a vendor’s. The retrieval score, separate from the answer score: which gold documents did it see, and does the grading have a verdict for lucky. The query log: not which sources the report cites, but what was asked in order to find them. And behind any aggregate number, the rulebook that makes the answer singular, with the name of the person who set it.

What changed for me is the reading order. The answer used to be the artefact. Now it is the last thing I read, because the answer is the one part of the pipeline that cannot tell you when the evidence failed. I start where the answers come from.

COO in energy, previously Twitter and Stripe. I build what I write about.