A 70B model in BF16 needs about 141 GB. In FP8, about 71 GB. In INT4, about 35 GB. That arithmetic is simple. The hard part is keeping the model’s behavior intact and finding kernels that turn the smaller bytes into real speed.

What quantization changes

Most quantization methods reduce the number of bits used to store weights. Some also quantize activations or the KV cache.

“INT4” is not a single algorithm. GPTQ, AWQ, GGUF variants, HQQ and others differ in how they choose scales, handle outliers, and group weights. Two models both labeled “4-bit” can have different quality and different kernel requirements.

The two big families

Weight-only quantization stores weights at low precision but computes in a higher precision (often FP16/BF16) after dequantizing on the fly. It reduces memory and weight bandwidth, which is exactly what decode needs. It does not necessarily reduce compute, and the dequantization step adds overhead.

Weight-and-activation quantization (W8A8, W4A8, etc.) quantizes both weights and activations, enabling faster low-precision matrix multiplies on hardware that supports them. It can improve prefill throughput and reduce activation memory. It is harder to do well because activations have outliers and dynamic ranges.

For decode-bound serving, weight-only 4-bit is often attractive: the model fits, the weight read shrinks, and the math was not the bottleneck anyway. For prefill-heavy workloads, activation quantization and FP8 GEMMs may matter more.

How the math works

A simple affine quantization maps a real value $x$ to an integer $q$:

where $s$ is a scale and $z$ is a zero point. Dequantization is

For symmetric weight quantization, $z = 0$ and $s = \max|x| / (2^{b-1}-1)$. For a group of weights, you can use a per-channel or per-group scale. Smaller groups adapt better to outliers but store more metadata.

The error is not uniform. Weights with large magnitudes tend to matter more; activation outliers can dominate the error in a layer. Methods like AWQ protect salient channels based on activation statistics. GPTQ uses approximate second-order information to choose quantized values that minimize layer-wise output error. SmoothQuant migrates quantization difficulty from activations to weights.

None of these methods is free. They require calibration data, and their quality depends on the model, the task and the evaluation. A perplexity delta of 0.1 can hide a large regression on code, math or structured output.

What gets faster — and what does not

The common claim is “4-bit is 4× faster.” The reality is more nuanced.

• Memory capacity: roughly proportional to bytes per parameter (plus overhead). This is the most reliable win.
• Decode bandwidth: reading 4-bit weights instead of 16-bit can approach a 4× reduction in weight bytes, but dequantization, scale loads and kernel inefficiency eat into it. A 1.5–2.5× decode speedup is plausible; 4× is not typical.
• Prefill compute: weight-only quantization may not speed up math at all, because it dequantizes to BF16 before the GEMM. W8A8/FP8 with tensor-core support can help.
• Quality: task-dependent. Always evaluate.
• Kernel support: the limiting factor. A format without a fast kernel is just a smaller file.

Also watch memory overhead: scales, zero points and group metadata add bytes. A 4-bit model is not exactly one quarter the size of BF16. For small group sizes, overhead can be several percent.

FP8 is a different bargain

FP8 formats (E4M3 and E5M2) keep a floating-point exponent, so they handle range better than INT8. On H100 and newer GPUs, FP8 tensor cores are fast. FP8 is often used for both weights and activations, with per-tensor or per-block scaling.

FP8 is attractive because:

• It halves memory versus BF16 with less quality risk than INT4.
• It maps well to modern hardware.
• It can accelerate both prefill and decode.

It is not universal: older GPUs may not support it, and some models are sensitive to FP8 in certain layers. Calibration and layer-wise fallback are common.

KV-cache quantization

The KV cache is often the largest consumer of memory at long context. Quantizing it to FP8 or INT8 can double or quadruple concurrency, or allow longer contexts. But attention is sensitive: keys and values participate in softmax and value mixing, and errors can accumulate over long sequences.

Practical notes:

• Quantize K and V separately; their distributions differ.
• Use per-head or per-channel scales where possible.
• Evaluate long-context tasks, not just short prompts.
• Check kernel support; a quantized cache with a slow attention kernel can be a net loss.

A decision procedure

1. Establish a BF16 baseline on your actual tasks, with your actual prompts. Record quality, TTFT, TPOT, throughput and memory.
2. Ask what is binding. If memory is the constraint, weight quantization helps immediately. If decode bandwidth is the constraint, weight-only quantization may help. If prefill compute is the constraint, FP8/W8A8 may help more.
3. Pick the least aggressive format that meets the constraint. FP8 before INT4, unless memory demands otherwise.
4. Evaluate quality on a held-out set that includes edge cases: code, math, JSON, multilingual, long context, safety.
5. Benchmark the full engine, not a standalone kernel. Include dequantization, sampling and scheduling.
6. Monitor drift. A quantized model can behave differently after a prompt template change or a new task mix.

A worked example

An 8B model in BF16 on an 80 GB H100:

• Weights: 16 GB.
• Usable memory after workspace: ~60 GB.
• KV cache: 128 KiB/token.
• At 8K context: ~49 concurrent requests (optimistic).

In FP8:

• Weights: 8 GB.
• Usable memory after workspace: ~68 GB.
• KV cache at FP8: 64 KiB/token.
• At 8K context: ~130 concurrent requests (optimistic).

The concurrency roughly doubles from weights and doubles again from the cache. If the workload is memory-bound, that is a large capacity win. If decode bandwidth is the limit, the token rate may improve too. But if the FP8 attention kernel is slower or quality drops, the real gain is smaller.

Next: decoding and speculation — how sampling works, and how a small draft model can turn one target-model pass into several accepted tokens.