Eval Suite Engineering

We build regression suites that answer, on every change, "did the agent get worse?". Not a dashboard of vibes: a dataset, a set of graders, thresholds, and a CI job that blocks a merge when they are crossed. A typical build takes four to six weeks of senior engineering time and lands entirely in your repository.

What we do

• Dataset and definitions. Curate 300-1,000 cases from production traces, synthetic edge cases and product requirements, every case with a documented origin. Write the correctness definitions down - for an agent that is usually three layers: the final outcome, the trajectory (tools called, arguments, order, termination), and side effects (what changed in connected systems). Build fixtures or recorded responses for external tools so cases replay deterministically.
• Graders. Deterministic graders wherever possible: schema checks, argument equality, state assertions, ordering constraints - cheap, fast and drift-free. Calibrated LLM-as-judge graders where judgment is genuinely required, with agreement measured against 100+ human-labeled cases and the number published; a judge below roughly 85% agreement gets its rubric rewritten or is replaced with a deterministic check. Cost and latency budgets per case class, so a regression in spend shows up alongside a regression in quality.
• CI, dashboards and handover. The suite runs in your CI on every prompt, model, tool-schema or retrieval change, with thresholds agreed and enforced. Results land in the tooling you already use - Langfuse, Braintrust, Promptfoo, or OpenTelemetry into your observability stack. A written runbook covers adding cases, re-calibrating judges and reading a failing run, with a handover session for the team that will own it.

Who this is for

• Teams whose agent changes weekly - prompts, models, tools - and who find regressions from customers rather than from CI.
• Teams that finished a readiness assessment and now need the full harness the starter set pointed at.
• Platform teams standardizing evals across several agents who want the first suite built well enough to copy.

How we work with you

• Hourly time and materials, scoped in writing through an initial conversation. Larger tool surfaces or heavy fixture work extend the estimate, and we say so before work starts, never during it.
• Weekly demos of the working suite, so nothing is a surprise at handover.
• What it is not. Not a rewrite of your agent - we file issues and pair on fixes for failures the suite finds, but product feature work stays with your team. Not a managed service - the harness lives in your repo under your CI; ongoing reliability engineering exists if you want us to keep operating it.

Next step

Tell us what the agent does, roughly how many tools it calls, and what tracing is already in place. Contact us.