How you know it works
Part 4 of a series from six weeks building AI systems by hand: an operator going beneath the abstractions, on purpose. These posts carry more machinery than executive writing usually does. That’s deliberate.
Part 2 set the terms of the mandate: what an agent may do, and what binds it. This post is the other half of delegation. When you hand work to a person, assurance comes built in: reviews, numbers with owners, an audit trail. When you hand work to an agent, what comes back is output: fluent, formatted, confident. Confidence is not evidence. Six weeks of building made me answer, on my own systems, the question I would normally put to someone else’s: how do you know it works?
What follows is how I answer it now, on my own systems first.
Decide what correct means before you run
The evaluation discipline that held up best all sprint is also the oldest trick in science: commit to the answer key first. Before I tested the evaluation agent in the final week, I worked the problem by hand (nine candidate runs against four thresholds, on a spreadsheet) so the right answers existed before the system produced its own. That hand-check is the only reason I caught the agent fabricating its numbers; the first post tells that story.
Pre-registration goes beyond answers. For the research system I built in week four, I wrote down the plan I would expect, the failure modes I predicted, and the run’s mechanical footprint (ten model calls, nine searches) before touching anything. It made exactly ten calls and nine searches. Two of the four predicted failure modes fired. An evaluation designed after the output arrives is an argument for the output; one designed before is a test of it. The operating question this buys you: when a team shows you an accuracy number, ask when the definition of correct was written down, and by whom.
The judge is a model too
At any real scale, humans cannot read every output, so AI systems are scored by other AI systems: LLM-as-judge, a model given a rubric and asked to grade. It works, and I used it. But the judge has all the failure modes of the thing it judges, which means the evaluation itself needs evaluating.
Two decisions made my judging usable. First, a rubric of named dimensions instead of a single score: groundedness, completeness, discipline, precision, each defined and scored separately, because “8/10” hides exactly the failure you most need to see. Second, calibration against my own labels: I scored a sample by hand first, then compared the judge to me, not me to the judge. A verdict is only as strong as the judge’s own audit. Whenever an eval score reaches my desk now, my first question is about the judge’s rubric, and my second is who checked the judge.
The trace is the audit trail
A trace records what a system actually did: every model call, every tool invocation, every input and output, in order, with timings and token counts. It is the closest thing agents have to an audit trail, and mine ended up holding something I did not expect: the repair itself.
The fabrication from the first post was caught by gold labels and file arithmetic; tracing went into that build mid-repair, and the five traces it captured turn out to hold the whole anatomy of the incident.
The first trace shows the mechanism, timestamped to the second. The agent reads the recipe and runs the scoring script, correctly. The script’s files land in a directory the agent cannot see, a path bug I had not yet found, so when the agent lists the evidence folder it gets back an empty list. What happens next I would not have believed without the record: over the following two minutes it writes the missing evidence files itself (summary, shortlist, per-run evaluations) and fills them with the numbers it had already invented, 1200.0 where the source data says 990, then reads its own files back and cites them. Three more runs over the next half hour read that manufactured corpus as ground truth. Every report confident, well-cited, wrong, and the visible process in all four flawless.
The fifth trace begins twenty-two seconds after my fixed script landed on disk. It reads the recipe, then the script’s own source code, runs it, and the artifacts regenerate mid-run; it takes every run’s values from the script’s output, deep-reads the three shortlisted candidates, and only then writes its recommendation. Same tool call, same file path as the first four. The read returns 990.0, in a record that carries its own caveats, and the audit trail dates the repair to the minute.
The lesson is not that the model misbehaved twice. It is that nothing short of the record could tell the two eras apart: the reports read identically, and the process (read, run, cite) looked correct in every trace. The difference lived in what the tool calls actually returned, which is precisely the layer a trace records and a report conceals.
Trace literacy has traps of its own, and I paid for one personally: I read “the manager consumed most tokens” off a dashboard panel, and the number turned out to be a parent span’s rolled-up total: the sum of its children, not the manager’s own spend. Compare model calls to model calls, not containers to contents. The skill is not exotic, but it is a skill, and I now treat trace-reading the way I treat reading a cash-flow statement: learnable, essential, and full of lines that mean something other than what they appear to say.
Every number needs a reference class
The research system’s four-section report cost 24,000 tokens on gpt-5-mini, about fifteen times a single chat turn, which lands exactly on Anthropic’s published multiplier for multi-agent research systems. That match matters more than the number: it says the build behaves like the reference class, and the spend is structural, not waste. The detail with operating consequences: completions exceeded prompts nearly everywhere. The cost concentrates where judgment happens, not where the context is biggest. You are paying for the thinking, and the thinking is what autonomy multiplies.
The discipline I took from it: no number arrives alone. It comes with its reference class, or it is declared the baseline for the next run. A quality score of 87 means nothing; 87 against a hand-labelled baseline of 82, on the same rubric, means something. This is the oldest rule in operations reporting, applied to a new report.
Verdicts are not enough
The subtlest finding of the sprint: on its first run, the evaluation agent reached the right verdict — do not ship — while misstating the margin by nearly six-fold, reporting a 230ms miss where the real gap was 40ms. The decision was correct; the counsel was materially misleading. A 40ms gap is a tuning problem, a 230ms gap is an architecture problem, and a board acting on the wrong one misallocates a quarter. Assurance that stops at verdicts prices decisions on fiction. The margins are where the money is.
The assurance artefacts
Four things to ask of any agent someone claims is working, whether your team built it or bought it: the answer key, and the date it was written; the judge’s rubric, and who audited the judge; the trace, not the summary of the trace; and the reference class behind every number in the deck. None of this needs an executive to write code. All of it needs someone in the room who knows the difference between output and evidence. That, more than anything technical, is what the sixty hours bought.
COO in energy, previously Twitter and Stripe. I build what I write about.
Comments