You do not need a framework to build a harness. You need a loop, a way to call a model, a few typed tools, a policy function, and a log. Everything else is an optimization or a product surface.

Scope it down

The fastest way to fail is to build a general agent platform. Start with a single, valuable task:

“Given a failing test in a local repository, find the cause, make the smallest fix, run the suite, and produce a diff.”

One user. One repo. One language. No cloud. No multi-tenant. No plugin marketplace. This scope lets you finish the loop and learn where the real problems are.

The architecture:

Five modules:

1. Loop — builds context, calls the model, parses actions, enforces budgets, stops.
2. Model adapter — one interface over a provider, with token accounting.
3. Tool registry — typed tools with schemas, execution and errors.
4. Policy — deterministic authorization for each tool call.
5. Trace — append-only events for debugging, evals and cost.

The types

Start with a small, explicit vocabulary. Resist the urge to make it generic.

type ToolName = 'read_file' | 'search' | 'edit_file' | 'run_tests' | 'finish';

interface ToolCall {
  id: string;
  name: ToolName;
  args: Record<string, unknown>;
}

interface ToolResult {
  callId: string;
  ok: boolean;
  output: string;
  meta?: { path?: string; exitCode?: number; bytes?: number };
  error?: { class: 'invalid_args' | 'denied' | 'timeout' | 'failed'; message: string };
}

interface Step {
  index: number;
  thought?: string;
  call?: ToolCall;
  result?: ToolResult;
}

interface RunState {
  taskId: string;
  goal: string;
  acceptance: string[];
  steps: Step[];
  filesChanged: string[];
  tests: 'not_run' | 'passed' | 'failed';
  budget: { stepsLeft: number; tokensLeft: number; wallClockMsLeft: number };
}

The finish tool is deliberate. It forces the model to declare completion through the same typed interface as every other action, so the harness can intercept it and run verification.

The model adapter

Wrap the provider behind one function. The loop should not care which model or SDK you use.

interface ModelRequest {
  system: string;
  messages: Message[];
  tools: ToolSchema[];
  maxTokens: number;
  temperature: number;
}

interface ModelResponse {
  text: string;
  toolCalls: ToolCall[];
  usage: { input: number; cached: number; output: number };
  stopReason: 'tool_use' | 'end_turn' | 'max_tokens';
}

async function callModel(req: ModelRequest): Promise<ModelResponse> {
  // Provider-specific call. Normalize tool calls and usage.
}

Log the model, version, prompt hash and token usage on every call. A harness without token accounting cannot answer “why did this cost so much?”

Tools are contracts

Each tool has a schema, a handler and a risk class. Keep the handler small and the errors specific.

interface Tool {
  name: ToolName;
  description: string;
  parameters: JSONSchema;
  risk: 'read' | 'write' | 'exec';
  execute(args: unknown, ctx: ToolContext): Promise<ToolResult>;
}

A useful starting set:

• read_file(path, offset?, limit?) — read inside the workspace, truncate with a pointer.
• search(query, glob?) — literal or regex search, return paths and line numbers.
• edit_file(path, oldString, newString) — exact replacement, fail if the match is ambiguous.
• run_tests(suite?) — execute a fixed command in a sandbox, return a compact result.
• finish(summary, evidence) — request completion; the harness verifies independently.

Do not start with a raw shell. It is the most powerful and least describable tool. Add it later, behind a policy and a sandbox, when the narrow tools genuinely cannot do the job.

The policy function

Policy is a pure function from action and grant to a decision. It should be easy to unit test.

type Decision =
  | { kind: 'allow'; sandbox: 'workspace' | 'none'; timeoutMs: number }
  | { kind: 'deny'; reason: string }
  | { kind: 'ask'; reason: string };

function authorize(call: ToolCall, grant: Grant): Decision {
  const tool = registry[call.name];
  if (!tool) return { kind: 'deny', reason: `unknown tool: ${call.name}` };
  if (tool.risk === 'read' && grant.canRead) return allow(5_000);
  if (tool.risk === 'write' && grant.canWrite) return allow(10_000);
  if (tool.risk === 'exec' && grant.canExec) return allow(30_000);
  return { kind: 'deny', reason: `${tool.risk} not granted` };
}

Then validate the arguments separately. A policy that says “writes are allowed” but does not canonicalize the path is not a policy; it is a wish.

The loop

The loop is where the product lives. It should be boring and explicit.

async function run(task: Task, grant: Grant): Promise<RunResult> {
  const state: RunState = initialState(task);
  const trace = new Trace(task.id);

  for (let i = 0; i < task.maxSteps; i++) {
    if (state.budget.tokensLeft <= 0) return stop('token_budget');
    if (state.budget.wallClockMsLeft <= 0) return stop('time_budget');

    const context = buildContext(task, state);
    const response = await callModel({ ...context, tools: registry.schemas() });
    trace.model(response, context);
    state.budget.tokensLeft -= response.usage.input + response.usage.output;

    if (response.toolCalls.length === 0) {
      state.steps.push({ index: i, thought: response.text });
      continue;
    }

    for (const call of response.toolCalls) {
      const decision = authorize(call, grant);
      trace.policy(call, decision);
      if (decision.kind !== 'allow') {
        state.steps.push({ index: i, call, result: denied(call, decision) });
        continue;
      }

      if (call.name === 'finish') {
        const check = await verify(task, state);
        trace.verification(check);
        if (check.pass) return success(state, check);
        state.steps.push({ index: i, call, result: failure(check.evidence) });
        continue;
      }

      const result = await executeWithTimeout(call, decision.timeoutMs);
      trace.tool(call, result);
      applyResult(state, call, result);
    }
  }

  return stop('step_budget');
}

Notice what is missing: no “agent framework,” no dynamic tool discovery, no autonomous planning layer. The loop is a for statement with budgets and a verification gate.

Context construction

buildContext is the most important function you will tune. A reasonable first version:

system: role, constraints, tools, definition of done
state: goal, acceptance criteria, current plan, files changed, test status
recent: last N tool calls with results (truncated)
workspace: the specific files referenced by the task or recent calls
output: reserve tokens for the next response

Keep the ordering stable for prompt caching. Pin the acceptance criteria. Summarize old exploration. Never let the context builder silently drop an unresolved failure.

Verification

The finish path must not trust the model. For the failing-test harness:

async function verify(task: Task, state: RunState): Promise<Check> {
  const tests = await runCommand('npm test', { timeoutMs: 120_000 });
  const diff = await runCommand('git diff --stat', { timeoutMs: 5_000 });
  const changed = parseDiffPaths(diff.stdout);
  const inScope = changed.every((p) => task.allowedPaths.some((g) => match(g, p)));
  return {
    pass: tests.exitCode === 0 && state.tests === 'passed' && inScope,
    evidence: `tests=${tests.exitCode} changed=${changed.join(',')}`,
  };
}

If verification fails, feed the evidence back into the loop. The agent gets another chance, but the harness owns the verdict.

Tracing and replay

Emit an event for every model call, policy decision, tool result and verification. Store the arguments and outputs, redacted. A trace that cannot be replayed is a log; a trace that can is a test case.

For replay, stub the model and tools with recorded responses keyed by step. Then run the loop again and assert that it reaches the same decision. This catches regressions in parsing, policy and context construction without spending tokens.

Testing the harness

Test the pieces, then the loop:

• Tool tests: schema validation, path traversal, ambiguous edits, output truncation, timeout.
• Policy tests: every risk class, deny by default, symlink escape, approval path.
• Loop tests: step budget, token budget, cancellation, verification failure, finish without evidence.
• Replay tests: a recorded failure reproduces the same first divergence.
• Golden tasks: five to ten real tasks with deterministic acceptance checks.

A harness with no tests is a prompt with delusions of grandeur.

Build milestones

1. Read-only explorer. Model can read and search; harness prints a trace. No writes.
2. Patch loop. Add edit_file and a diff view. Require approval. Run no commands.
3. Verification. Add a fixed test command in a sandbox. finish requires a passing check.
4. Budgets and replay. Add step/token/time limits and recorded traces.
5. Policy hardening. Add path canonicalization, command allowlist, secret scrubbing.
6. Evals. Add a small task suite and a regression dashboard.
7. Only now: a shell, subagents, MCP, multiple models.

Each milestone is usable. Each one teaches you something the next depends on.

What not to build first

• A plugin marketplace.
• A general workflow engine.
• A multi-agent swarm.
• A custom vector database.
• A UI with fourteen panels.
• A model router trained on your own traffic before you have traffic.

These are multipliers on a loop that already works. Before that, they are distractions.

Next: the harness capstone — design a multi-tenant release engineer with real permissions, budgets and evals.