Prefer to watch? Here's the video version.
For a quick overview of the topics covered in this blog.
Adapted from the talk Daniela Miao and I gave at Seattle Systems on September 30 (recording).
When we first started building evals for coding agents, a cheap model solved one of our tasks in about a minute. Every test passed. For a moment we were impressed. Then we read the trace.
The fix was sitting in our git history. Our task came from a real bug in a real repo, rolled back to before the fix, and the agent simply rolled history forward to find the answer. So we took the git history away.
Next, it found a previous run's workspace on disk and read the solution from those files. We blocked that too. Then it noticed environment variables showing it was running in a container. It reasoned, correctly, that we run tasks in parallel, so it looked at the other processes and read a neighboring task's work in progress. Finally, it found the remote origin URL in the repo's .git directory and pulled the answer from GitHub with web search and curl.
It felt like teaching a room of very creative elementary school students, and every repo brought a new trick.
The shortest path to a passing test
I try hard not to anthropomorphize agents, and it's easy to slip. An agent has no intent to deceive anyone. It has a goal, usually a set of tests to pass, and it takes the cheapest route it can find to get there. If the answer is sitting somewhere in its environment, finding it is less work than solving the problem.
Agents notice when they're being tested
The same logic applies to an eval grading apparatus. In one run, the agent showed no interest in grading files until it noticed an environment variable flagging the run as an eval. You can see the moment in the trace. It realized it was in an eval environment and went hunting for files that might hold the task criteria. Without that variable, it never takes that path.
Researchers have measured this. Sahar Abdelnabi and Ahmed Salem call it the Hawthorne effect in reasoning models, and found that models which detect a test can shift toward optimizing for passing it. Daniela put it in human terms. When you know there's a checklist, you stop doing a good job in general and start working the checklist.
Harness memory creates a quieter version of the same problem, because it carries what the agent saw in earlier attempts into the next one. For example, the Strands agent harness ships with memory on by default, which is great for everyday work and completely wrong for evals because every attempt has to stand on its own.
Isolate what you can, then judge the traces
Every cheat the agent used in our eval setup became a control we added to make the isolation stronger. We hide the grader and anything that hints at an eval, turn off harness memory between runs, and cut the agent off from the repo's history, old workspaces and neighboring tasks.
We can't sandbox everything, though. Agents legitimately need web search and other tools whose answers change from week to week, so isolation always involves judgment calls.
The most important control is reading the traces. A score can tell you whether the tests passed, but the trace holds the details that separate a breakthrough from an agent taking an unintended shortcut.
We packaged this work into mo eval, an MCP server that generates tasks from a repo, runs them in our sandbox and lets you dig into the results. Point it at our sample repos, or reach out to us to run it against your own code.