9 min read ·
An agent is a prototype of a script
The best demo of AI agents this month shipped a program with no AI in it. Sixteen Claude agents spent two weeks and about $20,000 writing a C compiler, and what came out the other end was a compiler. That's the whole lesson, and most teams building agents right now have it backwards: let the model find the path once, then ship the replay.
On February 5 Nicholas Carlini at Anthropic wrote up the experiment. Nearly 2,000 Claude Code sessions produced about 100,000 lines of Rust that can build a bootable Linux 6.9 on x86, ARM and RISC-V. It still borrows the assembler and linker from GCC, the decades-old standard C compiler, and on x86 it hands GCC one early boot stage. It's real work, and I'm not going to poke holes in it.
I want to point at what the agents were given, and at what they left behind.
They were given a language with a written standard. They were given high-quality compiler test suites that Carlini went out and found. They were given GCC itself as an answer key; the write-up calls it "an online known-good compiler oracle to compare against." And they were given a test harness, the rig that runs every attempt and scores it, which Carlini kept improving with new verifiers and build scripts. In his words, it's important that "the task verifier is nearly perfect, otherwise Claude will solve the wrong problem."
What they left behind is a compiler. When you run it, nothing thinks. It reads C and writes machine code, the same way every time, and it's still the only kind of code generator I fully trust.
No one at Anthropic suggested shipping the sixteen agents as the compiler. Say that out loud and it sounds absurd. Hold on to the feeling, because it's the right reaction to most agents in production today.
The $20,000 was a one-time fee.
Let me define the word, because it gets stretched to cover everything now. An agent is a model in a loop. It looks at where things stand, picks the next action, takes it, and looks again. Every turn of that loop is a decision, and every decision is a model call.
That's exactly what you want when nobody knows the path. It's waste when somebody does.
Say you have an agent that, every Monday morning, logs into an ad dashboard, exports last week's spend, pastes it into a spreadsheet and emails the team. Call it fourteen clicks. Each Monday the agent takes a screenshot, reasons about what it sees, and works out those fourteen clicks again, from scratch, the same way it worked them out last Monday. Fifty-two times a year you pay a model to rediscover a route that hasn't moved.
I call that the reasoning tax. Nobody hires a cartographer for the commute.
The tax is paid in two currencies, and only one of them shows up on an invoice. The first is tokens. The second is variance. A model that picks the next click from a screenshot can pick a different click on a different day, and the report has to come out the same on every Monday. Engineers have a dull word for "same input, same output": deterministic. It's the most valuable property a piece of business software can have, and an agent gives it up on purpose.
We've had tools that turn a walk through a website into a script for years. Playwright, the browser testing library, ships a recorder: run npx playwright codegen with a URL, click around, and it writes the code for what you did. An agent is a far smarter recorder. It can find the path without you sitting there. But a recorder's job is to produce the recording, and most agents throw theirs away at the end of every run.
When I started building browser agents, the kind that drive old web apps with no API the way a person would, the model chose every click. It read the page, decided, clicked, and read the page again. It demoed beautifully. It also paid the tax on every step of every run, and when a step went wrong, my answer was a bigger model thinking harder about the same step.
Now I keep the recording. Most steps are fixed actions in plain code. A few go to a model. Some go to a person. Every step in the procedure carries one field that says which of the three does it, and the fixed steps never wake the model up at all.
The model didn't leave. It got demoted, from driver to exception handler. It handles the parts you can't write down ahead of time: the popup that shows up on a bad day, the free-text field that needs reading, the page that loaded in a state the procedure never mentions. That's real judgment, and it's worth paying for. Clicking "Export" is not. The trick is to decide where judgment is actually needed and keep the model there.
The obvious question is where the fixed steps come from. If you're building a compiler, you get a standard and a test suite. If you're automating ordinary business work, you get neither.
Except you usually do. You just don't call it a spec.
In 2025 tool vendors rediscovered specifications and sold them to programmers as new. AWS put Kiro into preview in July, an editor built around "specs": requirements, a design and a list of tasks written before the code. In September GitHub released Spec Kit, whose announcement says coding agents "still need unambiguous instructions" and calls specs "living, executable artifacts."
Welcome to operations.
Almost every repetitive task a company would hand to an agent already has a standard operating procedure, an SOP: the document that tells a new hire how to do the job, step by step. It's been a living, executable artifact for decades. The executor is a person with a login.
I call ignoring it spec amnesia: building automation as if nobody ever wrote down how the work gets done, then applauding a tool vendor for suggesting somebody should. The motive is mostly vanity, and I've had a mild case of it. An agent that figures it out on its own looks like magic. An agent that follows the SOP looks like a checklist, because it is one, and nobody claps for a checklist.
The SOP isn't complete, of course. It was written for a person, and a person fills its gaps without noticing. "Check the details and submit" hides five decisions a new hire learns by asking the desk next to them. This is where the agent earns its keep. Run it against the procedure and watch where it gets stuck. Every place it stalls is a gap in the spec, and every gap you close is a step you can freeze into code.
That's what a prototype is for. You build it to learn what the real thing has to be.
The standard objection is that scripts are brittle.
Good. Brittle is loud.
When a web app moves a button, a script dies at step nine with "element not found" and a stack trace, and the fix is one line in step nine. An agent that improvises around the change might click the right thing or the thing next to it, report success either way, and give you a confident reason for whichever it picked. I'll take a script that fails at a named step over an agent that quietly finds a way around the problem every time.
None of this is my discovery. In December 2024 Anthropic published "Building effective agents", which splits these systems in two. Workflows are systems "orchestrated through predefined code paths." Agents are systems where models "dynamically direct their own processes and tool usage." Its advice is blunt: "we recommend finding the simplest solution possible, and only increasing complexity when needed. This might mean not building agentic systems at all."
A company that sells tokens for a living told you to buy fewer of them.
Andrej Karpathy made the same point from the other side last October. He called this "the decade of agents," not the year, and described "a very large demo-to-product gap where the demo is very easy, but the product is very hard." His unit for that gap is the nine, as in 90%, then 99%, then 99.9%: a demo that works 90% of the time is "just the first nine," and "every single nine is a constant amount of work."
Freezing steps into code is how you buy nines. A fixed step does on Tuesday exactly what it did on Monday, and if the page changed overnight, it fails loudly at that step. Each step you take out of the model's hands is one less place for the other 10% to hide.
So build the agent. I mean that. Point it at the messy task that's still done by hand, let it wander, watch where it struggles, and learn the path. That's the right job for a model: exploration, judgment, the odd case the SOP left out.
Then do the boring part. Write down the path it found. Turn the steps you understand into code. Leave the model at the few steps that need it, and send the rest to a person when the code can't decide.
An agent that does the same job every day is a prototype nobody finished. Finish yours.