A demo agent runs on a fast laptop with a reliable network and a cooperative filesystem. A production agent runs on a slow API with rate limits, a flaky test suite and a tool that occasionally returns the wrong answer. Reliability is the engineering that closes that gap.

Classify the failure before you retry

Retries help some failures and make others worse. The first job is classification:

The most dangerous case is ambiguous. A create-payment call times out. Did it succeed? Blind retry can charge twice. The harness needs an idempotency key and a way to query the external state. If it cannot reconcile, it must stop and ask a human.

Retry budgets and the math

Retries multiply load. If each request has independent failure probability $p$, the probability of at least one success after $n$ attempts is

For $p = 0.3$ and $n = 3$, that is $1 - 0.7^3 = 65.7\%$. The expected number of attempts per successful request is

which is 1.43 for $p=0.3$. If the whole fleet retries three times, a 30% failure rate becomes a 43% increase in load on an already struggling service. Retries can turn a partial outage into a total one.

Rules that keep retries sane:

• Cap attempts per operation, not per loop iteration. A retry storm inside a loop is still a storm.
• Use exponential backoff with jitter. A common form is $$t_i = \min(t_{\max}, t_0 \cdot 2^i) \cdot U(0.5, 1.5).$$
• Respect Retry-After. The server knows more than your backoff formula.
• Retry only idempotent operations unless you have a key.
• Count retries in the trace. A success that required six attempts is not a healthy success.
• Stop retrying on 4xx unless the status explicitly says otherwise.

Timeouts and cancellation

Every tool call needs a deadline. Without one, a stuck request holds a worker, a context and a budget. Timeouts should be:

• End-to-end, covering queueing, execution and response parsing.
• Shorter than the task budget, so one tool cannot consume the entire run.
• Propagated, so cancelling the task cancels the tool and its children.
• Recorded, with the phase that timed out.

Cancellation is harder than it looks. A fetch can be aborted, but a shell command needs process-group termination. A subagent needs a stop signal that reaches its own tools. A database transaction needs a rollback. If “stop” only stops the model loop, the side effects keep happening.

Idempotency and checkpoints

An operation is idempotent if repeating it has the same effect as running it once. Reads are naturally idempotent. Writes are not, unless you design them to be.

Techniques:

• Idempotency keys. The client generates a key per logical operation; the server stores the result and returns it on retry.
• Natural keys. “Create issue with ID TASK-42” is safer than “create issue.” Check before create.
• Upserts. Replace “insert” with “insert or update” when the semantics allow it.
• State machines. Move a task from pending to sent only after confirmation; make transitions one-way and logged.
• Checkpoints. Record what happened before attempting the next step, so a resume can skip completed work.
• Compensation. If a multi-step operation fails halfway, undo the earlier steps or mark the task for reconciliation.

A checkpoint without side-effect tracking is not enough. The log may say “step 3 started,” but the question is whether the email left the building.

Loop detection

Agents get stuck in recognizable ways:

• The same tool call with the same arguments, repeated.
• Alternating between two actions without progress.
• Re-reading a file that has not changed.
• Running a failing test with no edit in between.
• Summarizing the same plan in different words.

A simple detector hashes (tool, normalizedArgs) and tracks recent state. If the same call appears three times with identical results, inject a warning or stop. More robust detectors compare state, not text: the workspace hash, test status, open files, task-state version.

When a loop is detected, the harness has options:

1. Nudge. Add a message: “You have already run this exact command; choose a different approach.”
2. Widen. Force a different tool or a subagent with fresh context.
3. Escalate. Ask the user for a decision.
4. Stop. Return a partial result with an explanation.

Do not let a loop run until the global timeout. It burns money and hides the real problem.

Verification and reconciliation

Reliability is not just “the call succeeded.” It is “the work is correct.” Two levels:

Tool-level verification. Did the command exit 0? Did the response parse? Did the file write? Did the schema validate?

Task-level verification. Do the tests pass? Is the diff in scope? Does the output satisfy the acceptance criteria? Is the external system in the expected state?

When a step fails, the harness should reconcile before retrying: read the current state, compare it with the intended state, and choose an action. This is what makes resumption safe after a crash or a timeout.

A useful pattern is verify-then-report: the agent may say “done,” but the harness runs the check and produces the evidence. The final answer includes the evidence, not just the claim.

Graceful degradation

Not every failure should stop the task. A good harness defines fallbacks:

• Use a smaller model for a simple subtask.
• Return partial results with a clear “unverified” marker.
• Switch from live network to cached data.
• Reduce scope and ask the user to confirm.
• Queue the work for later instead of failing the request.

Degradation must be visible. A silent fallback that produces lower-quality output is worse than an honest error. Label the mode in the response and the trace.

A reliability budget

Give each task an explicit budget:

max_steps:        20
max_tool_retries: 2 per operation
max_task_retries: 1 (full re-plan)
tool_timeout:     30 s
task_timeout:     15 min
max_cost:         $0.40
max_idle_turns:   2

When a budget is exhausted, the harness stops with a structured status: what was attempted, what evidence exists, what remains. That is more useful than a generic “timeout.”

Testing reliability

You cannot test reliability only on the happy path. Inject failures:

• Add latency to a tool.
• Return 429 for the first two calls.
• Return a malformed response.
• Kill the process mid-task.
• Fill the disk.
• Make a write succeed but the response time out.
• Have a subagent return an invalid schema.
• Run the same task twice and check for duplicate side effects.

These tests are cheap compared with production incidents. A chaos test that runs in CI is worth more than a retry policy that has never been exercised.

Next: evals and traces — how to turn a run into evidence, and how to know whether a change made the harness better or just different.