
Check out the video below for highlights from the full talk
At Seattle Systems on September 30, Momento's Khawaja Shams and Daniela Miao gave a talk called Evals Is the New Black (Box). In covering the challenges you need to overcome to trust your coding-agent evals, they highlight that blindly focusing on cost will lead you astray.
They gave three real Valkey bugs to two setups, GLM-5.3-Flash running in the Strands harness and GPT-6 Astra running in Codex. Both finished all the tasks with the GLM+Strands option costing $1.32 and GPT-6+Codex costing $9.28. If cost was the only consideration you might think you're done.
That comparison changed two things at once, the model and the harness. Nothing in it tells you which change drove the cost difference, or whether the gap would hold on a different repo. What you've learned is that in this one case one model+harness combination is cheaper than another.
Change one thing at a time
An AI coding toolchain has more moving parts than most teams account for. There's the agent that kicks off the work, the skills it can call, the harness that turns a request into prompts and tool calls, the model itself, and the provider serving that model. Every tool in your toolchain can change the overall cost, speed and accuracy. If you are changing two variables at the same time, it is impossible to trace the result back to either.
The fix is the same discipline you'd apply to any scientific or performance test. Hold everything else fixed, change one element, and run it again and again until you have statistically meaningful results.
The harness alone moved the bill by 25%
The team ran exactly that kind of test with the model held constant. Claude Opus 4.8 worked through ten evaluation tasks based on the Gin repo, a Go web framework, in both Claude Code and Strands. Both harnesses were able to solve all ten tasks. Using the Claude Code harness cost $4.97 and using the Strands harness cost $3.72, about 25% less.

The gap comes down to how each harness manages context from turn to turn, which we'll dig into in an upcoming post. For platform teams the takeaway is simpler. Your harness choice can move the bill as much as your model choice, and a model-only comparison will never show it.
The provider and the clock matter too
Even with the same model and the same harness, results can still shift. The team ran GLM 5.2 on three different inference providers over the course of 24 hours. The variance between providers turned out to be smaller than the variance within any single provider throughout the day. The same tests may run up to two to three times faster at night, when provider fleets are less busy, and some of the worst scores turned out to be requests that got throttled.

That makes time of day a variable as well. If you run a new configuration at 2 p.m. and compare it to a baseline from 2 a.m., you may be measuring the provider's load instead of your toolchain. Repeating runs at different times is the only way to separate the two.
Keep testing the toolchain you actually run
Khawaja's closing advice was to stay skeptical. The best set of tools for a task today can be the worst one tomorrow, because models, harnesses and providers keep changing underneath you. The only way to keep up is to keep testing, one change at a time.
The full recording is worth an hour of your time, especially the stories about agents finding shortcuts to the answer. The approach from the talk is also what powers mo eval, Momento's MCP server for evaluating coding agents. Try it on our sample repos, or reach out to us to run it against your own code and toolchain.