Skip to main content
The Cosmos gateway accepts RoboLab’s OpenPI-compatible WebSocket requests, publishes camera tracks and robot state to Reactor, and returns joint targets to RoboLab. RoboLab runs the controller and scores the episode. Model inference runs on Reactor.

​
Prerequisites

Complete the first-action setup to obtain the public client environment and an API key. Separately install NVIDIA RoboLab, including its assets and an Isaac Sim-compatible NVIDIA GPU. This page references RoboLab revision ad45d4f974725d020f82c2b0d77d78533aeba2b3; keep its simulator environment separate from the gateway. Use RoboLab’s documented hardware requirements. Check out the documented simulator revision before following its installation steps:

​
Start the gateway

In a separate terminal on the simulator host, start at the root of the reactor-cookbook checkout:
check_wiring.py checks the observation split, payload encoding, and request/reply handoff without GPU inference. The gateway listens once its Reactor connection is ready.

​
Run one episode

In a second terminal, from the RoboLab checkout with its simulator Python environment activated:
Use one environment per gateway. The example owns one Reactor session and does not isolate parallel simulator clients. If the simulator runs in a container, use host networking on Linux so 127.0.0.1:8000 reaches the host gateway. Reactor replaces the policy-server step in NVIDIA’s Cosmos3 instructions. The simulator client composes wrist/left/right images; the gateway splits them into the three model tracks. It executes 32 targets at 15 Hz, then requests another chunk. Simulation waits for inference, so its success score does not measure behavior under uninterrupted physical time or observation delay. RoboLab writes scored episodes and recordings beneath output/; inspect those results rather than treating receipt of an action chunk as task success. For missing replies or uncertain timeouts, see troubleshooting.