This capstone is a design review, not a quiz. You get a product brief, a model, a workload and a budget. You produce a serving plan with numbers, assumptions and a measurement plan.

The brief

You are building an internal assistant for a software company of 4,000 engineers. It answers questions about the codebase, summarizes pull requests, and drafts small patches. The product team has given you these requirements:

• Traffic: 12 requests per second at peak, 3 requests per second average.
• Prompt: 6,000 tokens p50, 24,000 tokens p95 (retrieved code and docs).
• Output: 250 tokens p50, 900 tokens p95.
• SLO: p95 TTFT under 1.2 s, p95 TPOT under 50 ms.
• Availability: 99.9% during business hours.
• Budget: minimize cost per completed request, but do not sacrifice the SLO.
• Model: a 70B dense model is the quality baseline. An 8B model is available as a fallback. Both support GQA and a 128K context.
• Hardware: H100 80 GB and H200 141 GB nodes are available. Assume NVLink within a node.

Your job: choose a deployment, estimate capacity and cost, and write down what you would measure before committing.

Step 1: Start with the workload, not the GPU

The p95 prompt is 24K tokens. That is the number that will hurt. At 24K context, a 70B model’s KV cache is roughly:

At p50 (6K), it is about 2 GiB. With 70B BF16 weights at ~141 GB, a single H200 (141 GB) cannot hold the weights plus a meaningful cache. An H100 (80 GB) cannot hold the weights at all. You need at least two H200s or four H100s for the weights, before cache.

If you quantize to FP8, the weights are ~71 GB. Two H100s (160 GB) can hold them, leaving ~89 GB for cache and workspace. Two H200s leave ~211 GB. FP8 also halves the KV cache if you quantize it, which changes the concurrency math substantially.

The first decision is not “which GPU.” It is “what quality bar does the 70B model actually buy us?” If the 8B fallback passes the eval on the actual tasks, the entire capacity problem changes. Run that eval before buying hardware.

Step 2: Estimate concurrency from the SLO

Little’s Law gives a sanity check. At 12 requests/second and an end-to-end latency of, say, 8 seconds, you need about 96 requests in flight. At p95 context (24K) and 7.9 GiB of BF16 cache per request, that is ~758 GiB of KV cache. Impossible on a handful of GPUs. Even at 2 GiB average, it is ~192 GiB.

This is the key insight: the p95 prompt length, not the average, sets the memory requirement. You have four levers:

1. Reduce context — retrieve less, summarize, or route long-context requests to a separate pool.
2. Quantize the cache — FP8 halves it; INT8 is similar; MLA-style compression may help if the model supports it.
3. Increase memory — more GPUs, H200s instead of H100s.
4. Reduce concurrency — queue requests, which raises latency unless you also increase throughput.

A realistic plan might put long-context requests on a dedicated pool with a higher TTFT SLO, while short requests use a smaller, cheaper pool. Product segmentation is a capacity technique.

Step 3: Estimate throughput per GPU

Use the roofline and a measured reference. For a 70B BF16 model on an H200:

• Weights: ~141 GB.
• Bandwidth: ~4.8 TB/s.
• Ideal decode step at batch 1: ~29 ms for the weights alone.
• KV read at 24K context: ~7.9 GiB, another ~1.6 ms.
• Real overhead and kernel inefficiency: assume 50–65% of peak bandwidth.

At batch 1, that is roughly 35–50 tokens/s per stream, before batching. At batch 32, the weights are amortized, but the KV reads scale with the batch. The aggregate token rate rises, while per-stream TPOT may worsen. The SLO of 50 ms TPOT is roughly 20 tokens/s per stream, which is achievable at moderate batch if the cache fits.

Do not trust these estimates for a purchase. Use them to choose the shape of the experiment. Then benchmark the actual engine on the actual model and prompt distribution.

Step 4: Choose a topology

A defensible starting point:

Why TP=2 rather than TP=4? Fewer GPUs per replica means less communication and more replicas for the same fleet, which helps availability. But each replica has less memory, so concurrency per replica is lower. The right answer depends on the measured TP scaling and the KV budget. Benchmark TP=2 and TP=4 on the actual model.

Why FP8? It roughly halves weight and cache memory, and H100/H200 tensor cores support it. Validate quality on code, retrieval and patch generation before committing. If FP8 quality is not acceptable, BF16 on H200s may be the better path.

Why a separate long-context tier? A 24K prompt can block a shared batch and hurt everyone’s ITL. Isolating it protects the p50 experience and lets you tune the long-context pool independently.

Step 5: Estimate cost

Suppose the standard tier needs 4 replicas of 2× H100 to meet peak with headroom, and the long-context tier needs 2 replicas of 2× H200. At rough on-demand prices:

• 8 H100s × $3/hour = $24/hour.
• 4 H200s × $4.50/hour = $18/hour.
• Total: ~$42/hour, or ~$30,000/month at 24/7.

At 12 requests/second peak but 3 average, the average utilization is low. Reserved capacity, spot instances for batch, or a smaller always-on pool with burst capacity can cut the bill. The cost per completed request depends on the traffic profile, not just the hourly rate.

The capstone lesson is not the exact dollar figure. It is that cost is a function of utilization and workload shape, and the p95 prompt is a budget line item.

Step 6: Define the measurement plan

Before you commit, measure:

1. TTFT and TPOT at p50/p95/p99 for the real prompt and output distributions, at 1×, 2× and 3× expected peak.
2. Goodput at the SLO, not raw tokens/s.
3. KV occupancy and preemption rate at p95 context.
4. Prefix-cache hit rate for the codebase assistant’s repeated system prompt and retrieved files.
5. FP8 quality on a held-out task set, compared with BF16.
6. TP=2 vs TP=4 scaling on the actual model and batch sizes.
7. Cold-start time for a new replica, including weight load.
8. Failure behavior: one replica down, cache full, a 100K-token request, a rolling deploy.
9. Cost per completed request at average and peak traffic.

Write the results down with versions, dates and hardware. A capacity plan without a measurement plan is a guess with a spreadsheet.

Step 7: List what could invalidate the plan

A good design review includes its own failure modes:

• The p95 prompt is actually 40K, not 24K.
• The model’s real quality requires BF16, not FP8.
• Prefix caching hits less often than expected because prompts are unique.
• Traffic is burstier than the average suggests.
• The 8B fallback fails the eval on code tasks.
• The engine’s TP=2 scaling is worse than the vendor chart.
• Cold starts take 20 minutes, so autoscaling cannot save the peak.
• The product adds tool calls, turning one request into five sequential model calls.

For each, write the mitigation and the trigger that would cause you to revisit the design.

The harness track starts with a different question: given a model that can generate tokens, what does it take to make it do reliable work?