A harness that feels better is not evidence. A benchmark that went up is not evidence either, unless you know what it measured, on which tasks, with which graders and how much noise. Evals are how harness engineering becomes a science instead of a sequence of opinions.

Two questions, two evals

Every agent run answers two different questions:

• Outcome: Did the task reach the desired end state? Tests pass, JSON validates, the ticket is correct, the patch applies.
• Trajectory: How did it get there? Which tools, how many steps, what context, which approvals, which retries, which risks?

You need both. A correct answer from a dangerous path is a latent incident. A failed answer with a great trajectory is still a failure. Optimizing only the outcome rewards luck; optimizing only the trajectory rewards process theater.

What a span should record

A span is a timed operation with a parent. For agent systems, the minimum useful set is:

Do not log raw prompts and outputs by default. They contain secrets, personal data and customer content. Use redaction at write time, field-level access controls and retention limits. A trace store is a production database with a privacy budget.

Building an eval set

Start with real tasks, not synthetic puzzles. A good eval set has:

• Representative distribution. The same mix of simple, ambiguous and long tasks your users actually bring.
• A defined environment. Repo snapshot, fixtures, tool versions, model versions, seeds.
• Acceptance criteria. A deterministic check where possible; a rubric where not.
• Held-out cases. Tasks not used while tuning prompts or tools.
• Failure cases. Regressions from real incidents, with the exact trace.
• A cost and latency profile. A harness that is correct but 10× slower may be unusable.

For coding tasks, the outcome check is often a test suite plus a diff review. For research tasks, it may be citation coverage and factual checks. For operational tasks, it is the final system state.

A small, well-chosen set of 30–100 tasks is more useful than a giant noisy benchmark. You will run it constantly, inspect failures and trust the result.

Graders

Three families:

Deterministic. Tests, schema validation, diff checks, exit codes, exact match. Cheap, reproducible, and the default whenever possible.

Rubric / human. A structured checklist scored by a person or a carefully prompted judge. Useful for style, completeness and judgment. Expensive and subjective.

LLM judge. A model scores an output against criteria. Convenient, but it has biases: verbosity, position, self-preference and agreement with confident prose. Treat it as a measurement instrument that needs calibration.

If you use a judge:

• Give it the rubric, the evidence and the reference answer where available.
• Ask for a structured verdict with a reason, not a 1–10 vibe score.
• Randomize answer order for pairwise comparisons.
• Calibrate against human labels on a sample.
• Track judge–human agreement, not just judge scores.
• Re-run the judge when the judge model changes.

A judge that agrees with humans 70% of the time is a noisy signal. That can be useful for triage; it is not a finish line.

Statistics without a statistics degree

Eval results are samples. If 18 of 30 tasks pass, the point estimate is 60%, but the uncertainty is wide. A rough 95% confidence interval for a proportion is

For $\hat{p}=0.6$ and $n=30$, that is about $0.6 \pm 0.175$ — roughly 43% to 78%. A “5-point improvement” on 30 tasks is noise. Either use more tasks, pair the comparison on the same tasks, or report the interval and stop over-claiming.

For paired comparisons, look at the discordant pairs: tasks where A passed and B failed, versus B passed and A failed. A simple sign test or bootstrap on the difference is more informative than comparing two independent percentages.

Also watch for contamination. If the eval tasks, solutions or tests appeared in training data, the score may measure memorization. Keep a private holdout and rotate it.

Trajectory metrics

Outcome alone hides the cost of success. Useful trajectory metrics:

• Steps to completion and steps wasted on retries.
• Tool error rate by tool and error class.
• Context utilization and compactions per task.
• Approvals per task and time waiting for humans.
• Risk events: unapproved writes, network calls, secret access attempts.
• Cost per completed task, including retries and failed runs.
• Trace completeness: percentage of spans with required fields.

A harness change might improve success from 70% to 75% while doubling cost and approvals. Is that better? Only if the product says so.

Reading a trace

A practical debugging sequence:

1. Find the first divergence. Where did the run stop matching the intended path? Not the last error — the first wrong turn.
2. Check the context at that moment. Was the relevant constraint present? Had compaction removed it?
3. Check the tool result. Did the tool return what the agent thought it did?
4. Check the policy. Was the action allowed, denied or approved? By whom?
5. Check the model step. Was the output malformed, or did the harness misparse it?
6. Check the budget. Did a timeout or step limit truncate the work?
7. Compare with a passing run. What was different in context, tools or state?

The trace should let you answer these without re-running the task. If it cannot, the instrumentation is incomplete.

Offline, online and canary evals

• Offline evals run on a fixed dataset before deployment. They catch regressions and compare harness versions.
• Online evals sample real traffic and score outcomes or judge quality. They catch distribution shift.
• Canary evals route a small percentage of traffic to a new harness and compare against a control.
• Shadow evals run the new harness on copies of real requests without serving the result. Useful but expensive.

No single layer is enough. Offline evals are reproducible but stale. Online evals are real but noisy and privacy-sensitive. Canaries are real and controlled but slow. Combine them.

A minimal eval harness

interface EvalCase {
  id: string;
  task: string;
  fixture: string;
  acceptance: (world: World) => Promise<Check>;
}

interface Check {
  pass: boolean;
  evidence: string;
  details?: unknown;
}

async function runCase(c: EvalCase, harness: Harness) {
  const run = await harness.run({ task: c.task, fixture: c.fixture });
  const check = await c.acceptance(run.world);
  return {
    id: c.id,
    pass: check.pass,
    evidence: check.evidence,
    trace: run.traceId,
    steps: run.steps,
    costUsd: run.costUsd,
    durationMs: run.durationMs,
    riskEvents: run.riskEvents,
  };
}

Run it on every harness change. Store the trace IDs with the results. When a case fails, the first question is “what changed in the trace?”, not “which prompt should we tweak?”

Privacy and retention

Traces are useful because they are detailed, which makes them sensitive. Decide:

• What is redacted at write time (secrets, tokens, PII).
• Who can read raw prompts versus aggregates.
• How long traces are retained.
• Whether traces can leave the region.
• How a user can request deletion.
• Whether eval datasets can be derived from customer data.

“We log everything for debugging” is a policy decision with legal and security consequences. Write it down.

Next: cost and routing — why the cheapest token may be the most expensive completed task.