Configuration

Configuration is YAML, resolved by OmegaConf with Hydra-style overrides, then validated into a Pydantic MasterConfig (algorithms/grpo.py). A run is fully described by one config file plus any CLI overrides.

The override flow

YAML file ──▶ OmegaConf ──▶ apply CLI overrides ──▶ resolve ${...} ──▶ MasterConfig(**dict)

MasterConfig is declared extra="allow", so configs can carry keys beyond the documented schema (subsystems read their own sub-trees), but the listed top-level sections are required.

CLI overrides

Everything after --config is a Hydra dot-notation override:

python3 examples/run_grpo_swe.py --config examples/configs/grpo_swe.yaml \
    policy.model_name=Qwen/Qwen2.5-14B-Instruct \
    grpo.num_prompts_per_step=32 \
    loss_fn.reference_policy_kl_penalty=0.02

Interpolation and resolvers

Configs use OmegaConf interpolation (${policy.max_total_sequence_length}) to keep derived values in sync, plus custom resolvers registered by register_omegaconf_resolvers(). For example mul: multiplies, used to derive micro-batch token budgets:

train_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.train_micro_batch_size}}

Top-level sections

MasterConfig has these sections:

Section	Type	Controls
policy	dict	The trained model: name, precision, batch sizes, sequence length, the parallelism backend (dtensor_cfg / megatron_cfg / JAX), optimizer, scheduler, generation engine, LoRA (Low-Rank Adaptation), sequence packing.
loss_fn	ClippedPGLossConfig	The clipped policy-gradient loss: clip ranges, reference-KL (Kullback–Leibler) penalty and type, importance-sampling correction, token- vs. sequence-level reduction.
grpo	GRPOConfig	The RL loop: prompts/generations per step, rollout turns, advantage estimator, reward shaping/scaling, invalid-action penalty, structured tool use, async-GRPO settings, validation cadence.
env	dict	Per-environment settings (for SWE: scoring workers, reward mode, integrity check, sandbox URLs, task timeout).
data	dict	Datasets and dataloaders: train/validation dataset names, splits, prompt-length caps, the default processor and environment.
cluster	ClusterConfig	Cluster shape: gpus_per_node, num_nodes.
logger	dict	Logging backends (W&B, TensorBoard, MLflow, SwanLab), GPU monitoring, sample printing.
checkpointing	dict	Save cadence, top-k retention by metric, save format (safetensors), optimizer-state saving.
data_plane	DataPlaneConfig or None	Optional external experience-transfer plane for non-colocated fleets. None uses the in-memory path.

Key knobs

A few settings disproportionately shape a run:

• RL group size — grpo.num_prompts_per_step × grpo.num_generations_per_prompt. This is the number of rollouts per step and should match policy.train_global_batch_size.

• Sequence budget — policy.max_total_sequence_length bounds prompt + response and feeds the packing/dynamic-batching token budgets.

• Async GRPO — grpo.async_grpo.enabled switches loops. When on, several features are unsupported (dynamic sampling, reward scaling/shaping, multiple dataloaders) and IS correction is mandatory; these are validated at launch.

• Parallelism — policy.dtensor_cfg selects tensor/context/expert parallel degrees for the trainer; policy.generation.vllm_cfg selects the independent inference-side parallelism.

• Precision and FP8 — policy.precision is the training/master precision; policy.generation.vllm_cfg.precision=fp8 serves the inference fleet in FP8 while training stays in bf16 (weights are quantized on each refit).

• Generation engine — policy.generation configures sampling and the inference backend (vLLM or SGLang). Two knobs beyond the usual sampling fields: ignore_eos lets a rollout keep generating past the end-of-sequence token (off by default; useful when a reward needs a fixed-length completion), and policy.generation.vllm_cfg.env_vars passes a per-recipe map of environment variables through to the vLLM workers — for example to select a fused-MoE backend for a particular model without baking it into the image. temperature and top_p are validated for finiteness at worker startup — a NaN/Inf value is rejected rather than forwarded to the engine. Per-sample image inputs are gated by allow_multimodal_inputs (default false): a VLM/CUA recipe must set it to true to forward images to vLLM, and when enabled each image is normalized before the engine sees it — EXIF orientation applied, transparency flattened to RGB, and multi-frame inputs pinned to their first frame (GHSA-8jr5-v98p-w75m). max_image_pixels caps a single image’s decoded size (decompression-bomb guard) and max_images_per_sample caps how many images one sample may carry; both reject oversized inputs at the worker boundary. The SGLang backend applies the same gate but does not yet forward images — image inputs are surfaced as an explicit error rather than silently dropped. The text generation path is unaffected by the flag.

• Logging backends — logger enables any of W&B, TensorBoard, MLflow, and SwanLab together. W&B receives full per-step series (including the per-worker generation timeline) as-is. Scalar-only backends (MLflow) cannot hold a list metric, so a list-valued metric is summarized to <name>/{mean,p50,p90,max} and the per-worker generation timeline is merged across workers before that reduction — one bounded set of scalar keys per metric instead of one key per element.

The config family

examples/configs/ holds one config per target environment; they share the GRPO core and differ in dataset, environment, and rollout shape:

Config	Target
grpo_swe.yaml	SWE-bench coding agent (single-shot patch).
grpo_swe_pro.yaml	SWE-bench Pro variant.
grpo_swe_sglang.yaml	SWE-bench on the SGLang generation backend.
grpo_swe_jax.yaml	SWE-bench with the JAX (Flax NNX) trainer backend.
grpo_program_bench.yaml	Program-synthesis benchmark.
grpo_terminal_bench.yaml	Terminal-bench agentic tasks.
grpo_osworld.yaml	OSWorld computer-use agent.
grpo_gdpval.yaml / grpo_gdpval_agentic.yaml	GDPval file-producing tasks.
grpo_hle.yaml	Humanity’s Last Exam.
v4_consolidation.yaml	Multi-teacher on-policy distillation — stage 2 of a V4-style recipe.

Each config’s header comment documents the topology and target hardware it was written for.

The V4-style two-stage recipe

v4_consolidation.yaml is the one config that composes several of the others rather than targeting an environment. DeepSeek-V4’s post-training trains a separate expert per domain and only then distils them into one student, so each domain reaches its own optimum instead of every domain compromising against the others in a single joint run. Run it in three passes:

1. Per-domain SFT — sft.yaml, once per domain, keeping each checkpoint.

2. Per-domain GRPO — the environment config matching that domain (grpo_swe.yaml for code, grpo_hle.yaml or gpqa.yaml for reasoning, grpo_terminal_bench.yaml for agentic work), each starting from its stage-1 checkpoint.

3. Consolidation — v4_consolidation.yaml, with the stage-2 checkpoints as teachers. grpo.adv_estimator.name: "opd" replaces the environment reward with  = sg[log π_teacher − log π_student], and on_policy_distillation.teacher_model_by_agent_name selects which teacher scores which rollout from the agent name — V4’s per-context teacher selection.

Three constraints are enforced in code and the config satisfies all three: the OPD collector lives on the async path, so grpo.async_grpo.enabled must stay true; OPD on the data plane raises NotImplementedError, so data_plane stays undeclared; and the advantage subtracts prev_logprobs, so a config that zeroes them is refused at startup.

One difference from V4 worth stating plainly: this is the token-level estimator, not KL over the full vocabulary. The teacher-side full-vocab logit exporter does not exist yet — see HV-49 in the hardware-deferred validation ledger — so full-vocab KL is a build, not a config change.

Exporting a trained model to GGUF

After training, utils/native_checkpoint.py::convert_dcp_to_hf converts a DCP checkpoint to a regular Hugging Face build. utils/gguf_export.py then produces a switchable GGUF build alongside it so the released model also runs under llama.cpp / ollama and other native GGUF runtimes:

• convert_hf_to_gguf(hf_ckpt_path, out_path, quant_type=...) emits one GGUF; f16/bf16/f32 are written directly, quantized types (Q4_K_M, Q8_0, …) go through llama-quantize. export_gguf_variants(...) produces several at once.

• Conversion shells out to llama.cpp’s convert_hf_to_gguf.py and llama-quantize (a build-time dependency located via DOCKYARD_LLAMA_CPP_DIR or the llama_cpp_dir argument — not a Python package), run with sys.executable.

• Each output is validated with the gguf package (validate_gguf): architecture and tensor presence plus float-tensor finiteness.

Security: the GGUF dequant advisory (GHSA-5jv2-g5wq-cmr4) lives in vLLM’s dequant kernel, and is patched upstream in vLLM 0.24.0. This export+validation path never invokes vLLM, so producing and checking a GGUF does not exercise that kernel at all. dockyard has no GGUF-in-vLLM serve mode, so the advisory is unreachable here on either path. If such a serve mode is added, re-check the advisory against the pinned vLLM version at that point rather than assuming this note still applies.