Built like an instrument

Part 6 of a series from six weeks building AI systems by hand: an operator going beneath the abstractions, on purpose. These posts carry more machinery than executive writing usually does. That’s deliberate.

In week one I built a classifier; in week two I scored it against 21 test questions, and it returned 21 out of 21 on the first attempt. I didn’t record the result. I went looking for the bug. The score survived the checks, but the reflex was the real lesson: a perfect score is a hypothesis until it has been shown able to fail — the rule part 4 later stated as doctrine started here, as a suspicion.

Four weeks later, that reflex runs as a scheduled job inside a system I shipped. The first post opened with a question — which parts of these systems are genuinely hard, and which are just sold that way — and this one closes it with the capstone: Archimedes, an AI analyst covering the top public and private AI companies. Ninety-seven tracked: 48 public names on audited SEC filings, 49 private ones on graded disclosures. A site anyone can read, a chat grounded in its own data, an MCP endpoint other agents can call. The disciplines this series walked through (authority, assurance, evidence) turned out not to be advice. They are components. What follows is the architecture they compose, one mechanism per block.

One agent, and barely that

The first architectural decision is where the intelligence is allowed to be. Each step up the agentic ladder buys capability with a new kind of non-determinism — workflow agents have non-deterministic actions, autonomous agents add non-deterministic planning, multi-agent systems add non-deterministic communication — and Archimedes accepts only the first rung. A pipeline rebuilds the dataset weekly, and models help gather it — a contained collection agent hunts disclosures, a second pass verifies headcounts — but every grade and every ranking is computed in code, deterministically, downstream of anything a model wrote. The agent sits on top as one contained tool-caller, answering one question at a time. Read against part 2’s levers the placement was forced anyway: the process is fully enumerable (a question, a tool call, an answer), and the bad run is a wrong number published under my name. Contained in scope, expensive in kind. Both levers point to workflow.

The industry doctrine here is blunt: start with one agent, split only on operational proof. And the proof never arrived. The multi-agent machinery lived somewhere else entirely: in the workshop. Builder and critic loops, parallel subagents, a harness dispatching them; the sophisticated orchestration built the product and stayed out of it.

The grid behind the whole decision plots autonomy against control, and where a system sits on it is the agency question answered in one mark. Archimedes ships high-control and low-autonomy, on purpose. Movement rightward is earned with production data, never granted up front. Mandates for people work the same way.

Evidence is graded before it is used

Every number in the dataset carries a grade from a fixed ladder: audited filings at the top, then named outlets, company statements, specialist shops, analyst estimates, unvetted claims at the bottom. One rule makes the grades load-bearing rather than decorative: numbers of different grades never compete. Private companies rank in two divisions, reported evidence and estimate-grade, always badged, with no comparison across the line. Averaging incommensurables is how indices lie. Both rules live where part 2 said unacceptable-failure rules must live: in code. If an estimate ever reaches the reported division, the build exits with an error and no site ships that week.

Disagreement is machinery too. One company’s revenue arrived twice — its own statement and an unvetted aggregator’s higher figure — so the cell carried the contest, then the ruling for the statement, then the supersession when a newer release from the company moved the number itself. All three chapters stay readable in the cell. An instrument that hides its corrections is an opinion with an interface.

The agent answers from its tools or not at all

The chat holds no knowledge of its own. Each question starts a fresh context; the model reads nothing from any prior conversation, by design. Transcripts are logged for audit, a separate channel the model can never read back, because an analyst that recalls your last session is a liability this experiment does not need. The six tools are read-only and return digests the pipeline computed: aggregation happens in build code, never in the model’s context, which is the reading doctrine from part 5 as architecture: digest first, detail on demand, and counting is a query, not a retrieval.

Every answer lists the tools it checked, so the provenance is on the screen, not in my assurances. And refusal is a designed capability. Ask for a metric the index does not measure — net revenue retention, customer acquisition cost — and the reply names the gap instead of improvising around it. Ask it to compare a reported-division company with an estimate-grade one and it declines the comparison the site itself refuses to draw. The same rules, enforced at every surface, including the MCP endpoint, which closes a six-week circle: in week one my systems learned to call tools, and the capstone ends with the analyst being one, callable by any agent with the same graded answers and the same refusals.

The judge must prove it can fail

The self-audit runs on a schedule, and every run plants a fabricated number in its own inputs. The grounding checks must catch the plant, or the run cannot pass; a sample of source links is re-resolved against the live web at the same time. This is the week-two reflex as a component — my grounding check failed four consecutive ways in development, each bug caught only because the planted fake sailed through — and it is the reason the site can print an audit status without blushing.

PASS from a judge that cannot fail is not assurance.

Assurance also runs forward. The Register locks three falsifiable predictions about the dataset, graded on 1 September 2027, in public. Pre-registration was part 4’s discipline for evaluations; here the instrument applies it to itself. It can be wrong, on a date, where everyone can see.

What the operator owns

What the agent may do: one rung of the ladder, six read-only tools, a deterministic layer owning every number. What counts as evidence: a graded ladder enforced by code that would rather fail the build than blur a division. How it is checked: a judge that must prove it can fail, and predictions that come due on a date. What it may claim: whatever the tools return, and a named refusal for everything else.

None of it required me to become an engineer. All of it required someone to decide what correct means, what evidence counts, and when the system must say no: the decisions this series has argued belong to operators, made here by one, in the margins of a day job. The constraint on this class of system was never engineering capacity. It is judgment, and judgment is an operating resource.

Which also answers week one’s question. The intelligence is the part that is sold: rented by the token, swappable, untouched by anything I did in six weeks. The genuinely hard parts are the ones this post drew: what evidence counts, what correct means, when the system must refuse. None of them can be bought.

The instrument files a shift report with every rebuild; I read it the way I read any delegate’s, starting with what was checked. In a year, the Register will grade my calls in public. That is the arrangement now, and I would not accept a less accountable one from any system, or sign off on one.

COO in energy, previously Twitter and Stripe. I build what I write about.