What agents are built from

Part 3 of a series from six weeks building AI systems by hand: an operator going beneath the abstractions, on purpose. These posts carry more machinery than executive writing usually does. That’s deliberate.

When an executive asks what an AI agent actually is, the honest answer is a bill of materials, and almost nobody shows them one. Here is mine: every layer of the systems I built across the sprint, what each one does, and the decision each one represents. The parts are worth knowing because the stack is now roughly standard. What varies between a serious system and a demo is not which logos appear; it is who owns which layer, and how deliberately.

Nobody sits down and chooses eight tools. Each layer arrived when a problem forced it, so I will give you the stack in the order the sprint did.

The mind, rented by the token

Everything starts with a frontier model reached over an API: my builds ran on OpenAI’s gpt-5-mini, authenticated by a single API key in an environment file. Two operating facts hide in that sentence. The intelligence is rented, priced per token, and swappable: nothing in six weeks of building tied me to one vendor’s model, and a system designed this year should assume the model underneath it will change. And the key is a standing delegation of spend and access: a company credential like any other, except that software exercises it thousands of times a day. A later post is entirely about what keys and memory leak.

The skeleton, which I own

In week one the model needed structure around it, and that structure is the layer I came to care about most: LangGraph, a framework for wiring model calls into an explicit graph. This step, then that one, branch here, stop when this condition holds. The graph is ordinary code. It is where every binding rule from part 2 lives: the budgets, the caps, the stopping conditions, the assembly the model cannot skip. Its sibling LangChain supplies the fittings: prompt templates, tool definitions, structured output that forces the model’s answer into a typed form code can act on. Here is the entire wiring of my week-four research system, verbatim:

builder = StateGraph(ResearchState)
builder.add_node("manager", manager)
builder.add_node("researcher", researcher)
builder.add_node("finalizer", finalizer)
builder.add_edge(START, "manager")
builder.add_conditional_edges(
    "researcher", route, {"research": "researcher", "finalize": "finalizer"}
)
builder.add_edge("finalizer", END)

Nine lines. The researcher looping back on itself is the autonomy; the route function deciding when it stops is the stopping rule from part 2, as code.

The decision this layer represents: the skeleton is not plumbing to delegate away. It is where policy becomes enforceable, which makes it the most governance-dense code in the system.

The instruments, on from day one

Arize came into my stack in week one and never left: an observability platform that records the trace of every run. Each model call, tool invocation, input, output, and token count, in a tree you can open. I wired it in before the first interesting system existed, which is the whole trick: instrumentation added after an incident documents the next incident, not the last one. Traces are the agent’s audit trail, and the next post makes them do forensic work.

The evidence supply

In week three the systems started needing to know things, and two supply lines appeared. Retrieval: a corpus of documents embedded so the system can find passages by meaning rather than keyword; mine ran over 450 documents. And live search: Tavily, a search API built for agents, which returns clean extracted text instead of webpages. The pipes are commodity and rented; the corpus is yours, and so is its quality. What is not commodity is what my evaluation kept finding: the quality of an answer is set less by the model than by what these layers feed it. That finding gets its own post.

The connective standard

By week four the builds were reaching outside my machine, and this is where MCP enters: the Model Context Protocol, an open standard for describing tools so that any agent can call any service without custom glue code for every pairing. It is the reason my memory layer, Mem0, running on someone else’s servers, appeared to my agent as two simple tools: save a memory, search memories. The first time one of my systems called a tool that executed on infrastructure I do not control was a small moment with a large governance shadow. The boundary of the system stopped being the boundary of my machine.

The interface, where the system meets people

Every build wore the same front end: Gradio, a Python library that wraps a working system in a chat page in a dozen lines. The layer matters more than its size suggests. The interface is where a human grants or refuses what an agent proposes, so in the contained shape from part 2 (human approval where the mandate touches the outside world) this is where the approval physically lives. It is also the difference between a pipeline someone must trust and a system anyone can interrogate.

The harness above it all

One more layer sits above the stack and is the reason a COO could assemble the rest: the coding agent itself. I brought the spec and the architecture; a harness running a frontier model wrote and ran the code; working software came out. Its bench partner is the notebook (Jupyter), where a piece gets poked at interactively before it earns a place in the graph. Neither is part of the running system, but they are the operating reality: they are why the cost of an executive going one layer down has collapsed from a career detour to evenings.

Who owns which layer

Read the diagram by the tags and the operating shape appears. Nearly everything is rented: the model, the search, the memory, the observability, the harness. Renting is correct — these are utilities, and building them yourself is how a pilot dies. What cannot be rented is the skeleton and the answer key: the graph that encodes your policy, and the evaluation that defines what correct means for your business. Every vendor will happily sell you the rented layers assembled. None of them can supply the two layers that make the system yours, because those layers are your operating judgment, written down.

That is the real reading of any AI architecture diagram, and it takes about five minutes: ask which boxes are rented, who owns the two that cannot be, and whether anyone owns them at all. In most organisations today, the honest answer to the last question is nobody — the same empty chair the whole series keeps finding, one layer further down.

COO in energy, previously Twitter and Stripe. I build what I write about.