Most AI Agent Problems Aren't AI Problems
For the longest time, I thought building a better AI agent meant choosing a better model.
GPT-5? Claude? Gemini?
After working on AI systems, I've realized that's usually not where the biggest improvements come from.
Recently, I came across "The Agent Loop Decoded" by Richmond Alake, and it perfectly explained something many of us learn the hard way:
Most agent problems aren't model problems. They're harness problems.
The model is only the brain.
Everything around it, including memory, context retrieval, tool calling, retries, stop conditions, logging, caching, and state management, is what makes an AI agent reliable.
Think of it like this: a brilliant employee won't perform well if you never give them the right documents, don't let them use the tools they need, or forget everything they did yesterday.
The same applies to AI agents.
Three Levels of Agent Maturity
One idea that really stood out to me was the three levels of agent maturity.
- Level 1: An LLM with tools. Great for simple, one-off tasks.
- Level 2: Add memory. The agent can remember previous interactions and make better decisions over time.
- Level 3: Build the engineering around the model: context management, caching, semantic retrieval, monitoring, stop conditions, and workflows. This is where production-ready agents are built.
Stopping Is Not Completion
Another takeaway was that "the model stopped" doesn't always mean "the job is done."
A good agent doesn't just wait for the LLM to stop generating text. It checks whether the actual objective has been completed.
That small distinction is the difference between a demo and a production system.
Most agent engineering happens in the harness, not the model.
The more I work on AI products, the more I realize that choosing an LLM is often the easiest decision. Designing everything around it is where the real engineering begins.
If you're building AI agents today, don't just ask:
"Which model should I use?"
Also ask:
"Is my system giving the model everything it needs to succeed?"
Sometimes, that's where the biggest improvements are hiding.
Inspired by "The Agent Loop Decoded" by Richmond Alake on the Oracle Developers Blog.