dockyard_rl.rewards.program_bench

Module Contents

Classes

ProgramBenchReward

Score a ProgramBench cleanroom submission by clean-rebuild + behavioural tests.

API

class dockyard_rl.rewards.program_bench.ProgramBenchReward(reward_mode: str = 'test_pass_rate', api_token: Optional[str] = None, block_network: bool = True, pull_timeout: int = 1800)

Bases: dockyard_rl.rewards.interfaces.RewardFunction

Score a ProgramBench cleanroom submission by clean-rebuild + behavioural tests.

The agent’s exported workspace (submission_tar) is rebuilt from scratch in the per-instance image (image): wipe → extract → compile.sh → ./executable. Each selected test branch’s suite (branches[*].tar) is then run against that rebuilt executable, and the passed pytest node IDs are matched against the expected list (branches[*].expected) from tests.json.

Score = resolved / total over the union of selected branches, gated on a successful compile (a build failure yields reward 0).

Reads from the trajectory: sandbox_url (str) image (str) — programbench/:task_cleanroom submission_tar(str, base64) — gzip tar of the agent /workspace branches (dict) — {branch: {“tar_b64”: str, “expected”: [node_id]}} timeout (int, optional) — grading container wall-clock cap block_network (bool, optional) — default True (cleanroom: offline)

Args: reward_mode: “binary” → 1.0 iff every expected test passes; “test_pass_rate” → resolved/total, clamped [0, 1].

Initialization