OSWorld — the computer-use agent¶

OSWorld is the computer-use (CUA) path: the agent operates a real desktop GUI — taking screenshots, moving the mouse, typing — to complete tasks. It is the largest environment, with its own subsystem under experience/cua/, and is multimodal: the policy tokenizer loads an AutoProcessor (policy.tokenizer.use_processor=true) so per-turn screenshots are encoded through it.

The episode¶

A turn is: observe (a screenshot, optionally an accessibility tree), the model emits an action, the action is actuated on the guest, repeat — up to max_turns. Actions are parsed (experience/cua/actions/, e.g. pyautogui-style) and actuated through a backend.

Backends¶

env.osworld.backend selects the actuation/observation stack:

Backend	Guest
`official`	`desktop_env.DesktopEnv` (Docker KVM provider) — the offline-validated reference default.
`control_center`	In-house Linux stack (The-Eye + control-center + an OSWorld guest server for setup/grading).
`windows` / `macos`	Windows / macOS guests via control-center actuation. No built-in grading — supply a custom `env.osworld.grader` for non-Ubuntu task sets.

The official backend’s provider is docker (KVM-in-container, slow reset) or aws (snapshot-revert, fast reset at scale). Each concurrent episode needs one live container/VM, so actor concurrency is bounded by the provisioned host pool (experience/cua/provisioner.py); exceeding it is intended backpressure.

Grading¶

Ubuntu-authored OSWorld tasks grade through the guest server; custom Windows/macOS tasks require a supplied grader. Grading material and helpers live under experience/cua/grading/ and experience/cua/tasks/.

Config: examples/configs/grpo_osworld.yaml (its env.osworld block documents every backend, provider, and provisioning knob).