I was listening to the latest episode of State of Agentic Coding with Armin and Ben, and I was interested in their discussion of how agents are being trained and the optimizations going into reinforcement learning. Working with these agents and seeing what is highlighted in their release notes, it’s obvious that OpenAI and Anthropic are focused on long-running, autonomous agents that can complete tasks independently, like those measured by METR’s task-completion time-horizon benchmark. Last year, the benchmarks that mattered most were software engineering benchmarks such as SWE-bench. This year, it’s METR’s long-task benchmark. It seems like the frontier labs are betting on building a very determined, innately intelligent, newly minted consultant; think Harvard undergrad to Harvard MBA to McKinsey. This would be someone who lacks specific domain knowledge but has the ability to hunt down facts, use them to solve problems from first principles, and try multiple approaches to find a working solution within a set of constraints.
In my own work, I’ve had success asking an agent to figure out what smart approaches others have taken to similar problems, use those approaches to make a plan that we review, and then execute it. Sometimes this is as simple as “Go look at Apple’s official design guidelines,” and sometimes it’s “Find half a dozen open-source projects that deal with storing a large amount of text in a SQLite database in a space-efficient way while maintaining the ability to search it efficiently.” It does feel like working with a consultant.
The latest Sol models are where I feel the biggest difference among Codex, Pi, and Amp. The Amp team has done a good job constraining the Sol models to be more focused and not run off the rails when given an open-ended task. This is something Pi, with its minimal system prompt and emphasis on extensibility, intentionally doesn’t attempt. Meanwhile, the Codex team seems to be actively trying to make the models more helpful to people who are not software engineers. OpenAI now describes Codex as a productivity tool for knowledge workers, not just a coding tool. The Codex harness is really good at tasks outside software engineering—triaging my email inbox, for example—which lean more on the model’s general intelligence. I wonder whether the Pi team will need to take a stronger stance on intentionally directing the model toward coding tasks at the expense of general-purpose work.
I understand why the labs are chasing this scent. It feels like the best shot at reaching some general threshold of “good” across many different tasks as quickly as possible. As someone who uses these models primarily for software engineering, it’s a little frustrating that they are not working toward a more straightforward “write better code” direction. Intuitively, it also feels like the models would be more efficient if they had a better sense of what good code looks like, but I’m not sure that’s true. With Gemini, Google seemed to bet on putting as much knowledge as possible into the model. Gemini 3 Pro stood out to me for factual recall—an area where it led Google’s FACTS benchmark suite—but in my experience, that factual fluency did not translate into equally strong reasoning, and the model was disappointing overall and allegedly so was Gemini 3.5 Pro.
Overall, this really leans into a declaritve style of prompting, where you give the agent a goal and constraints and let it optimize within that space.