Skip to main content
Your simulator executes robot actions and renders depth. Cosmos3 Nano Sim Transfer turns that depth into camera-like RGB for your policy. Physics, simulation time, and task scoring remain in your simulator.

​
Try the delivery contract first

Run the incremental depth cookbook example to submit one frame followed by four-frame batches, match episode/step identifiers, verify RGB hashes, and save every PNG. It replays prepared depth; it is not a simulator or policy evaluation. Replace its fixture source with your simulator’s depth renderer when building the loop below.

​
Run the observation loop

  1. Render the initial simulator depth and call initialize. Wait for its one-frame RGB result.
  2. Let the policy consume that observation and choose actions.
  3. Advance the simulator for four render frames, recording the depth at each frame.
  4. Submit those four depth frames with advance. Pause simulation time while rendering completes.
  5. Match the result’s episode and step identifiers, consume its ordered RGB frames, and repeat.
At the documented 30 FPS simulation cadence, four frames represent about 133 ms of simulated time. The policy cannot use the intermediate rendered observations to revise actions inside that chunk. A policy needing a new rendered observation before every simulator frame is not supported by this contract. Simulator physics can use smaller internal timesteps; render cadence and policy cadence are separate choices. Keep one render step outstanding. Register result listeners before sending commands. A command acknowledgement means accepted, not rendered. Use step_result as the observation source; encoded video previews can drop or repeat frames. On timeout, inspect episode state and retrieve or retry the same step rather than advancing simulation again. This tests policy behavior with generated visual observations. Compare against simulator RGB and real-camera evaluation where available. Measure task outcomes from simulator state, and check whether generated appearance preserves task-relevant geometry and objects. Reduced visual mismatch is a hypothesis to evaluate, not an established reduction in the full sim-to-real gap.

​
Send depth and receive frames

Use a normal Reactor session with Python SDK 1.10.0 or JavaScript SDK 3.2.0. Register a step_result listener before submitting commands. These are session commands, not new REST endpoints. step_accepted confirms acceptance. step_result contains episode_id, step_index, first_frame, ordered frames_png_b64, corresponding sha256 hashes of decoded RGB uint8 pixels in row-major order, and complete. Match both identifiers and consume each result once; correlated replies can also reach the SDK message callback. step_failed means the step cannot complete; correct the input and initialize a fresh episode. Use prepared RGB 8-bit PNG depth visualizations, base64 encoded without a data-URL prefix. Metric float arrays, 16-bit depth, and ordinary RGB footage are not accepted substitutes. Each PNG must already match the selected output size: 832×480 for 16,9, 736×544 for 4,3, 640×640 for 1,1, 544×736 for 3,4, or 480×832 for 9,16; episode mode does not resize or crop. Each encoded input PNG is limited to 4 MiB. Only depth is supported in this mode. total_frames is a finite 1 + 4N, from 1 through 597 (default 121). Prompt, framing, seed, and preview selection remain fixed until reset. Set preview: false for reliable PNG results only; with preview enabled, the same generated frames also appear on main_video. The encoded preview is for viewing and is not the authoritative observation source. Large results use the existing SDK data-channel chunking; older clients with a plain 256 KiB limit are unsupported. Keep one step outstanding. An identical retry of the most recent completed advance returns its cached result; changed input is rejected. Older results are not retained. On timeout, query state, retrieve the latest completed result or retry the same input. Never increment step_index merely because a timeout occurred. Session loss requires a new episode; there is no checkpoint/resume. Reset before switching between episode and uploaded-video modes.