A strong model at $6 per million tokens sounds expensive. A cheap model at $0.15 that fails twice, runs a repair loop and then escalates can cost more — and take five times as long. Cost engineering starts with the denominator: completed, verified tasks.

The only cost metric that matters

Every term matters:

• Model cost includes input, output, cached input, retries and failed attempts.
• Tool cost includes API calls, search, sandbox compute and storage.
• Infra cost includes orchestration, tracing, vector databases and idle capacity.
• Human time includes approvals, reviews and incident response.
• The denominator is verified completions, not attempts. A 60% success rate means the cost of a success is 1.67× the cost per attempt.

A team that compares models on price per million tokens is comparing the numerator of one term. A team that compares cost per completed task is comparing the system.

Where the tokens actually go

In an agent, the input side usually dominates. Each turn resends the system prompt, tool schemas, conversation history, retrieved files and tool outputs. A 12-step task can send the same 8K-token prefix twelve times. Output tokens may be a small fraction of the bill.

Levers, in rough order of impact:

1. Fewer steps. Better tools, better context and a clear stopping condition reduce the number of model calls.
2. Prompt caching. A stable prefix can be charged at a much lower cached-input rate and reduces prefill latency. Keep the prefix stable: system instructions, tools, reference material, then changing turns.
3. Context discipline. Retrieve less, summarize exploration, drop stale tool output. Every unnecessary token is paid on every subsequent call.
4. Model routing. Use a cheaper model for steps that pass its eval, escalate only when needed.
5. Output limits. Cap max_tokens; a runaway generation is a cost incident.
6. Batch and off-peak. Non-interactive work can use cheaper capacity.
7. Retry discipline. A retry is a full additional model call plus tools.
8. Caching tool results. Deterministic reads and searches can be reused within a task.

Prompt caching deserves special attention. It changes the economics of long system prompts and tool definitions, but only if the prefix is byte-stable. Reordering tools, injecting a timestamp at the top, or changing a single whitespace character can invalidate the cache. Design the prompt like a cache key.

Routing by step, not by personality

A common mistake is to route whole requests: “simple users get the small model, complex users get the big one.” Real tasks are mixed. Reading a file, classifying an error and formatting a diff are easy. Diagnosing a race condition or designing a migration is hard. Route at the step level.

A router can use:

• Static rules. Tool type, file size, error class, task category.
• Task features. Number of files, ambiguity score, retrieval confidence, prior failures.
• Model confidence. Token logprobs, self-consistency across samples, schema validity.
• Learned router. A small classifier trained on which model succeeded historically.
• Cascade. Try cheap, verify, escalate on failure.

The cascade is often the most practical:

for tier in [fast, balanced, strong]:
    result = tier.run(step, context)
    if verifier.accept(result):
        return result
    context += verifier.feedback(result)
return escalate_to_human(step)

The verifier is the key. Without a cheap, reliable check, a cascade is just optimistic retrying. Tests, schema validation and deterministic linters make cascades work for code and structured output. Subjective writing is harder to verify, so escalation criteria must be explicit.

The cheap-model trap

Suppose a fast model costs $0.15/M output tokens and a strong model costs $6/M. The fast model is 40× cheaper per token. Now suppose the task needs 4,000 output tokens and the fast model fails 35% of the time, requiring a retry plus a repair pass:

• Fast success path: 4,000 tokens × $0.15/M = $0.0006.
• Fast failure path: 4,000 × 2.2 attempts × $0.15/M ≈ $0.0013, plus tool time and latency.
• Strong success path: 4,000 × $6/M = $0.024.

Even with a retry, the fast model is far cheaper per token. But if the failure produces a bad patch that passes a weak test and reaches a human reviewer, the human time dwarfs both. And if the task fails 60% of the time, the expected cost per success rises sharply.

The lesson is not “always use the strong model.” It is: measure success rate, retry rate and human intervention, then compute cost per verified completion. A cheap model with a strong verifier and a bounded retry can win. A cheap model with no verifier is a gamble.

Routing failure modes

• Distribution shift. The router was trained on last quarter’s tasks. New task types route to the wrong tier.
• Proxy metrics. The router optimizes “short response” and sends complex reasoning to the cheap model, which answers briefly and wrongly.
• Escalation loops. Cheap fails, strong fails, cheap retries — with no budget cap.
• Cache misses. Routing to different replicas destroys prefix-cache locality.
• Evaluation leakage. The router is tuned on the same tasks used to report savings.
• Latency inversion. Two sequential cheap calls can be slower than one strong call.
• Quality cliffs. A model that is 95% as good on average may be much worse on the 5% of tasks that matter most.

Monitor routing decisions alongside outcomes. A router that saves 30% of tokens while increasing escalations by 40% may be a net loss.

Budgets and attribution

Cost control is a harness feature, not a finance report:

• Per-task budget. Stop or escalate when the task exceeds a dollar or token cap.
• Per-tenant budget. Prevent one customer from consuming shared capacity.
• Per-tool budget. Expensive search or sandbox compute gets its own cap.
• Per-step output cap. Prevents a single response from consuming the task budget.
• Alerts. Notify when cost per completed task drifts, not just when total spend rises.
• Attribution. Tag every model call with tenant, task, route and harness version.

A budget without attribution is a bill. A budget with attribution is a product signal.

What to measure

For each route and model tier:

• attempts per completed task,
• success rate by task category,
• tokens per task (input, cached, output),
• cost per completed task,
• p50 and p95 latency,
• escalation rate and reason,
• human review time,
• cache hit rate.

Then run a controlled comparison: same tasks, same tools, same verifier, different routing policy. Report the difference with uncertainty. A routing change that looks good on ten tasks is an anecdote.

A routing policy template

route(step):
  if step.kind in {classify, extract, format, read_summary}:
      return fast
  if step.kind in {diagnose, design, refactor, security_review}:
      return strong
  if step.attempts == 0 and step.verifier == "tests":
      return balanced
  if step.failed_verification:
      return strong
  return balanced

This is a starting point, not a universal answer. The important properties are: explicit categories, a verifier, a bounded escalation path and a budget. The router should be legible enough that a human can predict its decision.

Next: harness teardowns — what Claude Code, Codex, OpenCode, Aider, Cline and OpenHands actually teach us about building the loop.