September 4, 2026
Testing an agent when there is no right answer
Unit tests cannot tell you whether an agent did a good job. So we built a harness that scores the real agent against a fake canvas, and learned to distrust our own easy cases.
"Add three scenes and score them" has no single correct output. That makes an agent nearly impossible to test the ordinary way, and very easy to fool yourself about.
The harness
The evaluation runs the real agent, with the real prompts, the real tool definitions and the real loop, against a fake canvas that records everything done to it. Scoring then asks structural questions that do have answers: did the nodes get created, were they wired up, was the order right, did it stay inside the budget it was given.
Two things it taught us straight away
- Easy suites rubber-stamp everything. A set of scenarios the agent passes every time measures nothing at all. It felt like coverage and it was closer to decoration.
- One run is not a result. Repeat the same scenario and the pass rate moves around. Without repeats you can't tell a real regression from a coin flip, and you will confidently ship the coin flip.
The uncomfortable implication: any single-run agent benchmark, ours or anyone else's, is reporting one sample from a distribution. Worth remembering next time you read one.
What it's actually for
Not a grade. A tripwire. It catches the changes that make the agent structurally worse: a prompt edit that stops it wiring nodes, a tool description that makes it skip a step. That's exactly the class of regression no unit test would notice and no human would spot until a user did.