Model and harness selection.
Choosing the best quality-per-dollar option for a specific workload, not just the one that felt fastest in a demo.
mo eval
Every toolchain change is a bet.
mo eval is how you know if you won.
Every model, harness, or skill change is a bet that things got better, faster, cheaper, or more reliable. A quick spot check on a couple of runs, or trusting that public benchmarks actually reflect your team's workflow, can make the bet look like a good decision.
mo eval replaces the guess with evidence, basing tooling decisions on quality-per-dollar, catching regressions before they ship, and keeping a record of why each decision was made that can be reproduced months later.
A run from the shared catalog · momentohq/gin · Go
Solved the same tasks for 23% less
cost per solved task · shorter is cheaper
Free demo
Run a graded task from our shared catalog inside Claude Code. The model tokens are on us.
claude mcp add --transport http -s user mo-eval https://mcp.evals.gomomento.ai/mcpThen run claude mcp login mo-eval and follow the quickstart to choose a task and start your eval. Quickstart & demo guide →
Why it is built this way
Rigorous evaluation is what turns “it seems better” into decisions you can defend.
Regression detection.
Catching quality, cost, latency, and behavioral regressions before they ship.
Improvement diagnosis.
Understanding why a task succeeded, failed, stalled, or spent more than expected.
Defensible decisions.
Avoiding gut decisions by using explicit questions, reviewable evidence, and claims based on the supporting evidence.
Reusable evidence.
Comparing multiple eval runs without spending new tokens.
Scope
Tested as a system, not just a model
Most evaluation stops at the model. mo eval tests the entire toolchain your developers actually run, the model, the harness executing it, the MCP tools it calls out to, and the skills and context it's given. A model that scores well in isolation can still fail in production if the harness mishandles a tool call or the wrong context gets passed along, testing the pieces separately doesn't tell you whether the whole chain gets the job done together.
The free demo runs a shared catalog of tasks mined from open repositories, so you can see the whole loop working before committing anything. The evaluation that decides your toolchain uses your own repositories instead: code changes, bug fixes, generated tests, refactors, and internal tooling, with your team setting what counts as success, including the quality, cost, and latency targets an eval has to stay inside.
Your company tasks
Built on your processes and repositories.
- Code changesNew features, refactors, or bug patches across services.
- Bug fixesFix regressions in code, processing, or pipelines.
- Generate testsAdd unit, end-to-end, and discovery test automation.
- RefactorsImprove or re-architect services and workflows.
- ToolingBuild internal CLIs, scripts, and utilities.
Acceptable results
Your success is verifiable.
- Passes testsAll tests are clean.
- Meets quality barCode quality, style, and standards.
- No regressionsExisting functionality behaves as expected.
- Within cost and latency targetsStays within budgeted cost and latency.
Judging
A judge that isn't grading its own work
Every task is graded first by the repository's own tests, which are not negotiable and not written by the model under test. On top of that, mo eval scores runs across further dimensions, such as quality and completeness, with a judge kept separate from the models being evaluated. Separating the two is what stops a model reporting its own homework, and human spot checks keep the grading held to your team's quality bar.
Repeated runs
One passing run doesn't tell the whole story
Complex prompts, run against the same toolchain, won't produce identical results every time. A model can pass a task twice and quietly regress on the third attempt, on cost, latency, or behavior, not just correctness. Making a tool change on a single run risks mistaking noise for signal, which is exactly why the free demo checks the workflow rather than settling a decision.
A comparison you can act on runs each task several times, across a set of tasks big enough to mean something, and reads the spread of outcomes rather than the best one. That is what design-run is for: it holds you to one variable per comparison, so a difference in the result can only be attributed to the thing you changed.
Infrastructure
Evidence that keeps its value
Running enough tests for a statistically meaningful answer adds up fast, and the cost compounds every time a new model ships. mo eval re-tests what is actually new and reuses prior baseline results rather than rerunning them, so a new model is measured against evidence you already paid for.
That works because results are kept as a permanent, queryable record. Every run's configuration, inputs, and outcome are preserved, so a comparison from three months ago can be pulled back up and re-checked — and every attempt's patch, logs and trajectory stay readable next to the number they produced. A script on someone's laptop loses all of that, usually about the time its author moves to another project.
Get started
Put your own repositories to the test
The quick start shows you the machine working on shared tasks. The decision you actually need — which model and harness to standardize on — comes from your code. Send us a few real repositories and see the benchmark data, cost comparison, and gateway configuration before you commit to anything.
Request a workload audit →Further reading
The full argument
For additional details, the statistical reasoning behind repeated runs and the infrastructure challenges most teams don't see coming until they've already built around it, dive deeper.