dockyard_rl.experience.rollouts

Module Contents

Functions

generate_responses	Generate responses from policy using synchronous generation.
generate_responses_async	Async version of generate_responses that properly calls generate_async.
calculate_rewards	Calculate rewards for generated responses and get environment feedback.
run_multi_turn_rollout	Runs a multi-turn rollout loop, interacting with the environment.
async_generate_response_for_sample_turn	Generate a response for a single sample’s turn using async generation.
run_sample_multi_turn_rollout	Run a multi-turn rollout for a single sample.
run_async_multi_turn_rollout	Run multi-turn rollouts with sample-level processing.

Data

logger
TokenizerType

API

dockyard_rl.experience.rollouts.logger

‘getLogger(…)’

dockyard_rl.experience.rollouts.TokenizerType

None

dockyard_rl.experience.rollouts.generate_responses(policy_generation: dockyard_rl.models.generation.interfaces.GenerationInterface, generation_input_data: dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.models.generation.interfaces.GenerationDatumSpec], batch: dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.data.interfaces.DatumSpec], tokenizer: dockyard_rl.experience.rollouts.TokenizerType, input_lengths: torch.Tensor, include_logprobs: bool = True, greedy: bool = False, structured_cfg: Optional[dockyard_rl.tool_protocol.protocol.StructuredToolUseConfig] = None) → tuple[dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.data.interfaces.DatumSpec], list[torch.Tensor], dict[str, Any]]

Generate responses from policy using synchronous generation.

structured_cfg: When set with decode_skip_special_tokens=False, the assistant message content is decoded with skip_special_tokens=False so structured tool-call tags (<tool_call>/</tool_call>) survive for the env parser. The generated token_ids are returned verbatim regardless; only the decoded text differs. Default/None leaves the decode as-is.

async dockyard_rl.experience.rollouts.generate_responses_async(policy_generation: dockyard_rl.models.generation.interfaces.GenerationInterface, generation_input_data: dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.models.generation.interfaces.GenerationDatumSpec], batch: dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.data.interfaces.DatumSpec], tokenizer: dockyard_rl.experience.rollouts.TokenizerType, input_lengths: torch.Tensor, include_logprobs: bool = True, greedy: bool = False, structured_cfg: Optional[dockyard_rl.tool_protocol.protocol.StructuredToolUseConfig] = None) → tuple[dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.data.interfaces.DatumSpec], list[torch.Tensor], dict[str, Any]]

Async version of generate_responses that properly calls generate_async.

structured_cfg: see generate_responses; controls only the decode of the assistant content (token_ids returned verbatim).

dockyard_rl.experience.rollouts.calculate_rewards(batch: dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.data.interfaces.DatumSpec], task_to_env: dict[str, dockyard_rl.environments.interfaces.EnvironmentInterface]) → dockyard_rl.environments.interfaces.EnvironmentReturn

Calculate rewards for generated responses and get environment feedback.

Args: batch: Batch containing message_log (LLMMessageLogType) with generated responses task_to_env: Dictionary mapping task names to their corresponding environments

Returns: EnvironmentReturn namedtuple containing: - observations: List of observations from the environment for the next turn. - metadata: List of extracted metadata from the environment. - next_stop_strings: List of stop strings for the next generation step. - rewards: Tensor of rewards for the last turn. - terminateds: Tensor of booleans indicating if an episode ended naturally.

dockyard_rl.experience.rollouts.run_multi_turn_rollout(policy_generation: dockyard_rl.models.generation.interfaces.GenerationInterface, input_batch: dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.data.interfaces.DatumSpec], tokenizer: dockyard_rl.experience.rollouts.TokenizerType, task_to_env: dict[str, dockyard_rl.environments.interfaces.EnvironmentInterface], max_seq_len: int, max_rollout_turns: int = 999999, greedy: bool = False, invalid_action_cfg: Optional[dockyard_rl.rewards.invalid_action.InvalidActionPenaltyConfig] = None, structured_cfg: Optional[dockyard_rl.tool_protocol.protocol.StructuredToolUseConfig] = None) → tuple[dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.data.interfaces.DatumSpec], dict[str, Any]]

Runs a multi-turn rollout loop, interacting with the environment.

Args: policy_generation: The generation interface (policy). input_batch: The starting batch containing initial message logs. tokenizer: The tokenizer. task_to_env: Dictionary mapping task names to environment instances. max_rollout_turns: Maximum number of agent-environment interaction turns. max_seq_len: Maximum sequence length allowed. greedy: Whether to use greedy decoding. invalid_action_cfg: When enabled, per-turn invalid-action / malformed-thinking verdicts are collected and emitted as per-sample count fields (invalid_action_count, malformed_thinking_count) for the reward penalty. structured_cfg: When set, the structured (Hermes) tool-use protocol is active: env observations are wrapped in the Qwen turn envelope before tokenizing (fork 3), the assistant decode may keep special tokens (fork “special-token decode risk”), and the thinking style is threaded into the malformed-thinking detector. None = fenced-text path, byte-identical to before.

Returns: Tuple containing: - BatchedDataDict with the full interaction history and accumulated rewards - Dictionary of rollout metrics

async dockyard_rl.experience.rollouts.async_generate_response_for_sample_turn(policy_generation: dockyard_rl.models.generation.interfaces.GenerationInterface, sample_message_log: list[dict], sample_stop_strings: list[str] | None, tokenizer: dockyard_rl.experience.rollouts.TokenizerType, max_seq_len: int, greedy: bool = False, structured_cfg: Optional[dockyard_rl.tool_protocol.protocol.StructuredToolUseConfig] = None) → tuple[list[dict], torch.Tensor, torch.Tensor, dict[str, Any]]

Generate a response for a single sample’s turn using async generation.

Args: policy_generation: The generation interface to use sample_message_log: Message log for a single sample sample_stop_strings: Stop strings for this sample tokenizer: Tokenizer to use max_seq_len: Maximum sequence length greedy: Whether to use greedy decoding

Returns: Tuple of (updated_message_log, generated_tokens, input_lengths, generation_metrics)

async dockyard_rl.experience.rollouts.run_sample_multi_turn_rollout(sample_idx: int, initial_sample_state: dict, policy_generation: dockyard_rl.models.generation.interfaces.GenerationInterface, tokenizer: dockyard_rl.experience.rollouts.TokenizerType, task_to_env: dict[str, dockyard_rl.environments.interfaces.EnvironmentInterface], max_seq_len: int, max_rollout_turns: int = 999999, greedy: bool = False, invalid_action_cfg: Optional[dockyard_rl.rewards.invalid_action.InvalidActionPenaltyConfig] = None, structured_cfg: Optional[dockyard_rl.tool_protocol.protocol.StructuredToolUseConfig] = None) → tuple[dict, dict[str, Any]]

Run a multi-turn rollout for a single sample.

This function manages the complete lifecycle of one sample’s interaction. Async generation is used internally when available.

Args: sample_idx: Index of this sample in the original batch initial_sample_state: Initial state containing message_log, extra_env_info, etc. policy_generation: The generation interface tokenizer: Tokenizer to use task_to_env: Environment mapping max_seq_len: Maximum sequence length max_rollout_turns: Maximum number of turns greedy: Whether to use greedy decoding

Returns: Tuple of (final_sample_state, sample_metrics)

dockyard_rl.experience.rollouts.run_async_multi_turn_rollout(policy_generation: dockyard_rl.models.generation.interfaces.GenerationInterface, input_batch: dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.data.interfaces.DatumSpec], tokenizer: dockyard_rl.experience.rollouts.TokenizerType, task_to_env: dict[str, dockyard_rl.environments.interfaces.EnvironmentInterface], max_seq_len: int, max_rollout_turns: int = 999999, greedy: bool = False, invalid_action_cfg: Optional[dockyard_rl.rewards.invalid_action.InvalidActionPenaltyConfig] = None, structured_cfg: Optional[dockyard_rl.tool_protocol.protocol.StructuredToolUseConfig] = None) → tuple[dockyard_rl.distributed.batched_data_dict.BatchedDataDict[dockyard_rl.data.interfaces.DatumSpec], dict[str, Any]]

Run multi-turn rollouts with sample-level processing.

Each sample in the batch proceeds through its interaction independently. Async generation is used internally when available but the function is synchronous.

Args: policy_generation: The generation interface (policy) input_batch: The starting batch containing initial message logs tokenizer: The tokenizer task_to_env: Dictionary mapping task names to environment instances max_seq_len: Maximum sequence length allowed max_rollout_turns: Maximum number of agent-environment interaction turns greedy: Whether to use greedy decoding

Returns: Tuple containing: - BatchedDataDict with the full interaction history and accumulated rewards - Dictionary of rollout metrics