dockyard_rl.algorithms.async_utils.interfaces

Async RL interfaces.

Module Contents

Classes

ReplayBufferProtocol

Interface for the replay buffer used in async RL training.

API

class dockyard_rl.algorithms.async_utils.interfaces.ReplayBufferProtocol

Bases: typing.Protocol

Interface for the replay buffer used in async RL training.

add(trajectory: dict[str, Any], weight_version: int, target_weight_version: int) → str

Add a per-prompt trajectory group with metadata.

Args: trajectory: Data dict for the trajectory group. weight_version: Version of model weights used for generation. target_weight_version: Version of model weights this trajectory is intended to train on.

Returns: “success” if added, “full” if the buffer is at capacity.

sample(num_prompt_groups: int, current_weight_version: int, max_age_steps: int) → Optional[dict[str, Any]]

Sample per-prompt trajectory groups for the current training step.

Only returns trajectories with target_weight_version == current_weight_version. Returns None if insufficient trajectories are available, stalling training until the remaining trajectories are generated. This ensures no trajectory loses its last chance to be used for its intended training step.

Returns: Dict with ‘trajectories’ and ‘avg_trajectory_age’ keys, or None if insufficient data.

evict() → None

Evict old trajectories.

size() → int

Return current buffer size.

clear() → None

Clear the buffer.