Dear friends,
The key idea of agentic coding loops is to have an agent keep working until it satisfies a condition, such as achieving a product specification. Coming up with the spec, the evals, or test set is one of the hardest tasks for many AI systems, and a key place to inject human knowledge. Once you have a clear spec, the paths to fulfilling it (such as having a coding agent iterate) become clearer. I'd like to share useful practices for building product specs that drive coding loops when quickly building 0-to-1 applications. As I'll explain below, one organizing principle I use is: AI tokens are cheap; human tokens are gold. (Yes, I know humans don't literally tokenize, but I work so much with tokens that I now think of myself as providing tokens to my computers!)
When building a large, complex system, it is worthwhile to spend time architecting it well. This involves deciding, for example, what database do you want to use? How do you break down the workload across different services? What third-party dependencies are you locking yourself into? What are the API boundaries between frontend and backend? Thinking through such questions in advance helps to avoid being stuck with a bad decision that is hard to change.
But when building quick 0-to-1 prototypes or applications in domains I'm unfamiliar with, I often move faster and with less deliberation. I find it convenient to jump quickly into directing a coding agent to start building, so I can examine what it has built in order to make the next set of decisions. No need to spend an hour of human time mulling over a design spec when an AI agent will spend 20 minutes building a simple prototype; I'd rather spend 10 minutes writing an inferior spec, see what the agent has built, examine its assumptions, and then refine the spec and repeat the process. Instead of "spec drive development" becoming a new waterfall process where writing a spec is a gate to further progress, this allows me to more iteratively refine the spec.
For example, the first version of many apps I’ve prototyped had confusing UI designs that I modified the spec to fix, and no amount of my time spent tweaking the initial spec would have led me to realize in advance that these changes would be needed. Coming up with those human tokens on my own would have been too expensive. It was easier (and cheaper) to see what went wrong in order to explain how to do it right.
As a software project matures, coding agents will work longer (maybe hours) to build to more complex specs, But at this early stage, it is fine to throw away the entire codebase and restart from scratch, so nothing is hard to change. All that code was cheap, in human time and token costs. What we learned from testing it was more valuable.
Further, having one implementation to react to (or sometimes I'll ask the agent for a few different designs) gives you an efficient way to get feedback (human tokens) to refine the spec to help drive later iterations. If you're building a 0-to-1 product, I encourage you to put the coding agent to work, maybe even earlier than might feel comfortable, see what it comes up with, and use that as a starting point. Then document your reactions and keep iterating.
Here, too, you can make the agent do most of the work. One common source of frustration among developers is when we tell a coding agent something and later (maybe after memory compaction) it forgets what had said. I hope future coding harness improvements will ameliorate this, but for now, when I make key decisions, I often steer the agent to remember that decision somewhere, say in a SPEC.md file. In the course of building a project, a coding agent may generate millions of tokens — far more than the human developer would type. So if an agent discovers something and later forgets it, there is a cost, since it might burn more tokens to rediscover it — but this is not too bad. But if it forgets something that you told it, that is much more frustrating, since you end up having to tell it again: more golden tokens down the drain.
Coding agents have become less forgetful in the last few months because of model advances, but they still occasionally forget. When I spot something in a prototype that I'm unsatisfied with, often I tell the agent not only to fix it but also to update the spec (and perhaps the test plan), so the agent’s stopping criteria in the future require checking that this problem does not occur again. This is one way to make sure it treats the human tokens as gold.
Keep building!
Andrew