Saurabh Kumar

Mario Zechner and Armin Ronacher on agent code that doesn't suck

Matt Pocock talks with Mario Zechner and Armin Ronacher, the people behind Pi, about writing agent code that doesn’t suck. Their line: trust an agent where something can check its work, and keep design and understanding for yourself.

  • Trust follows the oracle. Pi gets 40 to 50 issues a day. A triage bot reproduces each one, writes a failing test and a fix, and sets a priority. A reproduced P0 with a failing test and a passing fix usually just gets merged. Feature work that touches the user: “results are hilarious”.
  • Verification is oversold. Agents still game the lint rules and tests they write. Armin wrote a Lean proof for parts of Pi and found it was one more file for the agent to read, not a constraint. A counterweight to Ron Minsky’s talk, which treats agent-written proofs as a bet worth making.
  • System design hasn’t moved. Mario saw no real jump from Opus 4.8 to 5.5 in architecture or API design; models “still suck” at it. His method: have the agent write ten versions and pick by feel. Matt asked both for design terms to put in prompts. Neither had any to give.
  • Agent code only grows. Agents never say no, add features nobody asked for, and guard every input. Armin: software’s default is “towards entropy, towards having everything.”
  • Don’t run away from understanding. They’ve seen incident calls where nobody knows what’s going on and everyone is prompting Claude. Armin’s rule: build so doing nothing during an incident doesn’t make it worse, with switches that buy breathing room. No company’s disaster plan should be a backup coding agent running on generators.

The detail I’ll remember: Armin’s agent kept notes saying a table had 2,000 records. It wrote them when the database was empty; the table now has millions. Row counts and rates in agent notes go stale without any diff, so I’d store the command that measures them, not the number.

Markdown View