Sixty hours of homework

Six weeks ago I enrolled in an agentic engineering course: lectures, weekly assignments, quizzes, a demo day at the end. I’m a COO. I did the evening and weekend lectures, the homework, the capstone. I sat the quizzes.

The fee was the cheapest part. The real price was sixty hours of evenings, and the willingness to be a novice at something my job title implies I’ve transcended.

For the past year I’ve been building AI into how I work: BI pipelines, automations, agents doing real jobs, and shared services that several of my own tools run on. The meter says forty million tokens of delivered work this year, five billion counting everything the tooling read along the way, as a daily user running my own operating rhythm on it.

Built the disciplined way: specs before code, data models designed up front, QA on everything that shipped. Coding agents met me at that level. I bring the spec and the architecture, the harness writes and runs the code, working software comes out.

Which is exactly the problem. A year of building at that depth never once required me to understand the intelligence itself: why a search misses, or how an agent decides it’s done. The abstractions keep the one layer that matters out of sight, no matter how seriously you build on top of it. And that layer has started carrying money. I sign off on this technology at work, and I underwrite it as an investor. I could build on these systems. I couldn’t yet tell you which parts underneath are genuinely hard, and which parts are just sold that way.

So I went underneath. Six weeks, one small system at a time: a query router, a tool-calling loop written by hand, retrieval over a 450-document corpus, evaluation suites, a multi-agent research system, an agent wired to external services.

Two things from those sixty hours, out of dozens.

The first came in week one. I built a system that sorts incoming questions into four categories and routes each to different instructions, the same pattern that routes support tickets or payment requests and refunds. The engineering took under an hour. Nearly everything else went on a question that turned out not to be technical at all: deciding where a fact ends and a definition begins. Both readings of the same question were defensible. One had to be written down, because every downstream measurement of whether the system “worked” was defined against that choice.

I’d been calling this a technical build. The load-bearing decision in it was policy. Nobody in most org charts owns decisions like that one, and the systems ship anyway.

The second came last week, and it’s the one I’d put in front of a board. I built an agent that evaluates ML experiment runs against a shipping policy: hard constraints, tradeoff weights, the kind of gate a real platform team runs. Before testing it, I did the check by hand: nine candidate runs against four thresholds, worked out on a spreadsheet, so I’d know the right answer before the agent gave me its one. Then I ran it.

The recommendation came back beautifully. Correct format, every claim citing an evidence file, caveats included, confident. And the central numbers in it were invented. The agent had fabricated the latency figures — plausible values, internally consistent, shaped to justify its pick — because my own pipeline had a gap: the summary it read reported values for the runs that failed the gates and a bare “pass” for the runs that mattered. It had rich evidence about the rejects and a void where the winners should be, and it filled the void.

The report looked better than an honest one would have. I only caught it because the hand-checked answers existed first.

I traced the fault, fixed the pipeline, ran it again. The agent then read the real numbers, applied a judgment rule I’d written in plain English, and returned the recommendation my gold labels said was right, caveats and all. The rule was never the problem. What the model was given to read was.

Multiply that finding by every AI quality number currently arriving in board packs and diligence rooms. Confident, fluent output, whether or not it’s right. And the difference does not show on the surface.

Someone has to decide what “correct” means, what evidence a recommendation must carry, which failures are unacceptable, and when a system may act on its own. Those are operating decisions. In most companies right now they’re being made by default, by nobody.

For twenty years, “technical enough” for an operating executive meant reading a dashboard. I think that bar has moved.

I’m not describing a personal quirk. Tobi Lütke wrote to all of Shopify that reflexive AI usage is now a baseline expectation, and that teams asking for headcount must first show why AI can’t do the job. Since Paul Graham put a name on founder mode, going deep has been respectable again — and coding agents have made it cheap, which is the part that changed. What used to cost a leader a career detour now costs evenings. But most of this hands-on renaissance stops at usage. Using these systems daily told me almost nothing about when to trust them. That took the homework.

I don’t mean executives need to become engineers: sixty hours didn’t make me one. What the hours bought is narrower: I can go one level below the demo. I know what an evaluation actually proves, what a trace shows, what questions separate a governed system from a polished one, and what good answers sound like.

The commitment is the point. Watching a lecture about evaluation teaches you the vocabulary. Scoring your own system against answers you worked out by hand, and catching it fabricating — that teaches you the reflex. The reflex is what you carry into the next vendor meeting.

What’s changed: I read traces before I read roadmaps. I ask what the judge’s evaluation looks like, because the judge is a model too. I ask what happens at the exit condition. And when a system hands me something confident and beautifully formatted, I now treat the polish as a reason to check, because I’ve watched polish and fabrication arrive in the same document.

Over the next few weeks I’ll write up the specific decisions this work surfaced: the ones I now think operators should own.

COO in energy, previously Twitter and Stripe. I build what I write about.