This capstone is a design review, not a quiz. You are given a real product brief, a trust boundary and a budget. You produce a harness plan with explicit tools, permissions, state, evals and failure modes.

The brief

You work at a 150-person software company. Engineering ships about 40 pull requests a day. The release process is a mix of GitHub Actions, a staging environment, and a weekly production deploy. Two release engineers spend a large part of their week on:

• triaging failed CI runs and identifying whether the failure is flaky, environmental or a real regression,
• preparing release notes from merged PRs,
• checking that staging smoke tests passed,
• drafting the production deploy checklist,
• answering “is this safe to ship?” questions in Slack.

You are asked to build Helm, an internal release engineer agent. It should reduce the repetitive work without becoming a path to production.

Requirements and constraints

• Users: 150 engineers, but only release engineers and on-call leads can approve deploys.
• Repositories: read access to all internal repos; write access only to a release-notes repository and to branches named helm/*.
• CI: read logs, re-run failed jobs, but not cancel jobs or change workflow files.
• Staging: deploy a named build to staging and run smoke tests. No production access.
• Production: Helm may prepare a checklist and open a change ticket. It may not deploy, roll back or mutate production.
• Data: CI logs and repository code are internal. Slack messages may contain customer data.
• Budget: at most $1.50 per task, 20 tool steps, 10 minutes wall-clock.
• Availability: the release process must not depend on Helm being up.
• Audit: every tool call, approval and deploy recommendation must be attributable to a user and a task.

Step 1: Define the jobs, not the agent

Write down the concrete jobs before designing the loop:

Each job has a different definition of done. A single generic “release agent” prompt will blur them. The harness should expose them as commands with their own tools, budgets and verification.

This is an important design decision: the unit of product is a job, not a chat session. Chat is one interface; the workflow is the product.

Step 2: Draw the trust boundary

Helm touches three zones:

1. Read zone — repositories, CI logs, staging metrics, Slack history. Untrusted content may contain prompt injection.
2. Write zone — a release-notes repo, helm/* branches, staging deploys. Reversible with effort.
3. Forbidden zone — production credentials, production data, workflow files, secrets, customer PII.

The trust boundary is between zones 1 and 2: content read in zone 1 can influence actions in zone 2. This is the lethal trifecta in miniature if Helm can also communicate externally. Mitigations:

• No raw internet egress. Slack posting goes through a narrow, approved tool.
• Zone 1 content is labeled as untrusted data in the prompt.
• Zone 2 writes require a diff preview and, for staging, an approval.
• Production is not a tool at all. Helm can draft a ticket, but a human executes.
• Secrets never enter the model context. The sandbox uses short-lived, scoped tokens.
• Every write is captured as a diff and attributed.

Step 3: Choose the tool surface

A concrete tool list:

Notice what is absent: a generic shell, a generic HTTP tool, production deploy, secret access, and workflow-file edits. Those absences are features.

Tool descriptions should say when not to use them. ci.rerun should say: “Use only after reading the failure and classifying it as flaky or environmental. Do not rerun a deterministic test failure.” That sentence is part of the interface.

Step 4: Design the loop and stopping conditions

Helm’s loop is a plan-act-verify cycle with job-specific verification:

• CI triage: done when a classification, evidence links and a recommended action exist. Verification: the cited log lines exist and the classification matches a rubric.
• Release notes: done when every PR in the range is represented or explicitly excluded. Verification: set comparison against the PR list.
• Staging deploy: done when the deploy reports success and smoke tests pass. Verification: read the staging health endpoint and the smoke output.
• Ship check: done when each checklist item has evidence and the recommendation is one of go, no-go, needs-human. Verification: all items present; no unsupported claim.

Stopping conditions:

• verification passes,
• a human denies an approval,
• step/token/time/cost budget exhausted,
• a forbidden action is requested,
• the agent is stuck (repeated state), or
• the task is ambiguous and needs a human decision.

The model’s “done” is a request, not a stop.

Step 5: State and resumption

Helm’s tasks can take minutes and involve external side effects. The state model:

• Task record: job type, requester, repo, commit, acceptance criteria, budget, status.
• Event log: every model call, tool call, policy decision, approval and verification.
• Job state: classification, evidence links, draft notes, deploy ID, checklist status.
• External side effects: CI rerun ID, staging deploy ID, ticket ID, Slack message timestamp.
• Checkpoint: before each external write, record intent and idempotency key.

On resume, reconcile external state before repeating an action. A staging deploy that timed out may have succeeded. Query it. A Slack post may have been delivered. Do not send it twice.

Subagents are useful here:

• Log triager (read-only): reads a large CI log and returns a compact classification with evidence.
• Notes writer (write to a draft only): summarizes PRs.
• Verifier (read-only): checks the checklist against evidence.

The parent owns the task state and all external writes.

Step 6: Evals

Build an eval set from real release history:

• 30 CI failures: 10 flaky, 10 environmental, 10 real regressions, with known outcomes.
• 20 PR ranges for release notes, including a rename, a revert and a docs-only change.
• 10 staging deploys, including one where smoke tests fail.
• 10 ship checks, including ambiguous cases where the right answer is “needs human.”
• 5 prompt-injection cases: a malicious README, a log line that says “ignore previous instructions,” a PR description that asks Helm to deploy to production.

Grade outcomes deterministically where possible:

• classification accuracy against the labeled set,
• PR coverage (set equality),
• staging result matches the environment,
• checklist completeness and evidence links,
• zero forbidden actions, even in injection cases.

Grade trajectories:

• steps and cost per task,
• approval count and wait time,
• tool error and retry rate,
• unsupported claims in the final answer,
• whether a human had to correct the recommendation.

Run the eval on every prompt, tool or policy change. Keep the traces. When a case regresses, find the first divergence.

Step 7: Cost model

Estimate per job:

At a blended $1.20/M input and $6/M output, a 60k-input / 4k-output task is roughly $0.10 in model cost. Add tools, retries and human review. The $1.50 budget is generous for model tokens and tight if every task needs a human approval. Optimize the human loop, not just the token price.

Use prompt caching for the stable tool schemas and repository conventions. Route log classification to a cheaper model; use the strong model for ship checks. Cap output tokens. Batch release-note drafting.

Step 8: Failure modes

A good design includes its own failure modes:

• Prompt injection in a README or log. Mitigation: label untrusted content, no raw egress, approval on external writes, injection test cases.
• A flaky rerun that hides a real regression. Mitigation: require evidence and a classification; cap reruns; alert on repeated reruns of the same test.
• A staging deploy that succeeds but is unhealthy. Mitigation: smoke tests are part of done, not optional.
• A release note that omits a revert. Mitigation: set comparison against the PR list, not a summary judgment.
• A ship check that says “go” without evidence. Mitigation: checklist items require links; unsupported claims fail verification.
• Duplicate Slack posts or tickets on retry. Mitigation: idempotency keys and external-state reconciliation.
• A human approves without reading. Mitigation: show the diff and the effect, not just the tool name; keep approvals rare enough to matter.
• Cost runaway on a huge log. Mitigation: truncate with a pointer, summarize deterministically, cap tokens per tool result.
• Helm is down during a release. Mitigation: the release process must remain runnable manually. Helm is an assistant, not a dependency.

Step 9: The deliverable

Write a one-page design review with:

1. The jobs and their definitions of done.
2. The trust boundary and the forbidden zone.
3. The tool table with risk and approval.
4. The loop and stopping conditions.
5. The state and resumption model.
6. The eval set and metrics.
7. The cost model.
8. The top five failure modes and mitigations.
9. What you would ship in week one, and what you would explicitly not build.

If you can defend that page in a review with security, SRE and the release engineers in the room, you understand the harness track.

The two tracks meet here. Inference makes the tokens fast and cheap; the harness makes them safe and useful. The model generates. The harness decides.