# Quantized generation

Rollout generation can serve a **quantized** model while the trainer stays in
bf16, so the rollout fleet runs faster and lighter without quantizing the
training math. The wrinkle is RL refit: fresh weights stream from the trainer to
the inference engine every step, so the quantization must be **re-applied each
refit**, not once at load. Two paths exist — block-wise FP8 and ModelOpt NVFP4.

## Block-wise FP8

vLLM serves a block-wise-FP8 model (`models/generation/vllm/quantization/fp8.py`);
the bf16 weights from the trainer are quantized to FP8 on each refit, including
the per-expert MoE path. Training numerics are untouched — only the served copy
is FP8.

## ModelOpt NVFP4 real-quant

dockyard's ModelOpt path defaults to **fakequant**: a hook that simulates NVFP4's
numerics on a full-precision kernel, useful for studying the accuracy impact
without the real kernel. **Real-quant** (`cfg.real_quant=true`) instead serves a
true NVFP4 W4A16 model through vLLM's native FP4 (Marlin) kernel, setting
`quantization="modelopt"` and the `VLLM_MODELOPT_REAL_QUANT` env so vLLM loads the
FP4 method.

The challenge real-quant solves is refit-repeatability. vLLM's
`process_weights_after_loading` converts the loaded HF-named params (`weight` /
`weight_scale` / `weight_scale_2`) into the kernel layout **once** — deleting and
renaming params and dropping `weight_loader` references along the way. That is
fine for a static deployment but breaks RL, where new NVFP4 weights must be
re-loadable every step. The patches in
`modelopt/models/generation/vllm_modelopt_patch.py` make the conversion
repeatable:

- before converting, the loaded param shape/dtype/loader are captured so a
  subsequent refit can restore loadable parameters
  (`prepare_modelopt_for_weight_reload`),
- a weight-only (W4A16) checkpoint is tagged and routed to the Marlin FP4 GEMM,
- after a refit streams new weights, the conversion is re-run
  (`modelopt_process_weights_after_loading`).

**Scope.** This is the *inference-side* real-quant path: it serves an NVFP4
checkpoint and keeps it loadable across refits. A trainer-side DTensor NVFP4
**producer** (quantizing the policy's weights to NVFP4 before sync) is a
documented integration seam rather than a built component — the only existing
upstream producer is tied to a model-parallel training stack outside dockyard's
DTensor/FSDP2 design. Until that seam is wired, real-quant serves an
externally-produced NVFP4 model.

## Validation

The config switch, the patch wiring, and the W4A16 detection are CPU unit-tested
with the engine mocked. Live NVFP4 serving and the refit re-quantization round
trip are GPU-only and tracked in the hardware-deferred-validation ledger.