Build an AgiBot simulation

Use Cosmos3 Nano Sim Bimanual with an image of the supported AgiBot robot and scene, a task prompt, and calibrated controller poses. This guide covers the session commands and a mouse or trackpad control scheme.

Run the Python examples

Start with the scripted AgiBot rollout to connect to the hosted model, initialize your scene, change an arm pose and gripper value, and reset. It saves three preview snapshots so you can inspect the received video. Then try the interactive terminal controls to select either arm, adjust XYZ positions, open or close a gripper, and save snapshots. These are Python clients with saved images, not live graphical viewers. Both include pinned dependencies and setup instructions; bring a compatible AgiBot starting image and a Reactor API key with model access. The examples were verified against hosted model version 0.2.0 with Python SDK 1.10.0. Preview snapshots are not synchronized action-to-observation results for policy evaluation.

Session contract

Connect to reactor/cosmos3-nano-sim-bimanual. Use the SDK’s commands and uploads after the session is ready. These operations are session commands, not separate HTTP endpoints.

Command	Input	Effect
`set_image`	`image`: uploaded PNG or JPEG; each side at most 4,096 pixels	Sets the scene and restarts the world
`set_prompt`	`prompt`: task text, at most 800 characters	Changing the prompt restarts from the starting image
`set_teleop`	`teleop`: left/right poses and gripper values	Updates the held controller sample
`set_seed`	`seed`: integer from 0 to 2,147,483,647	Applies when the next world starts
`set_paused`	`paused`: boolean	Holds or resumes generation
`reset`	No parameters	Restarts from the current image and controller anchor

Generation starts once an image, nonempty prompt, and valid controller sample are present. Each new world anchors itself to the current controller positions; resetting does not move controllers to zero. Set the seed before starting or resetting a world. The main_video track carries generated RGB frames. Per-frame metadata contains step and frame. The simulation_status message reports input readiness, pause state, frame/step counters, and any configured rollout limit. A command acknowledgement confirms handling, not that its visible effect has reached the screen.

Controller payload

This is an example payload for set_teleop. Positions are meters in your calibrated tracking space; orientations are unit quaternions in [x, y, z, w] order. Align the controller axes and initial poses to the supported robot before applying movement. These numbers illustrate the shape, not a calibration.

{
  "teleop": {
    "left": { "position": [-0.2, 1.0, -0.4], "orientation": [0, 0, 0, 1] },
    "right": { "position": [0.2, 1.0, -0.4], "orientation": [0, 0, 0, 1] },
    "left_grip": 0.0,
    "right_grip": 0.0
  }
}

Grippers range from 0 (open) to 1 (closed). An optional head pose uses the same position and orientation fields; omitting it holds the head camera still.

Build a mouse or trackpad playground

Choose a compatible starting scene, upload it, set a task prompt, and send the initial controller poses. Display main_video with an explicit Left / Right arm selector, pause, and reset controls. Map each gesture to a bounded change in the selected controller’s calibrated pose:

Gesture	Client behavior
Click without dragging	Toggle the selected gripper between open and closed
Click and drag	Move the selected arm across the tabletop
Two-finger scroll or mouse wheel	Raise or lower the selected arm
Release the drag	Hold the current controller pose
Left / Right selector	Choose which controller receives subsequent changes

The model receives set_teleop pose samples, not mouse events. Use a small drag threshold so releasing a drag does not also toggle the gripper. Clear pending movement on focus loss or pointer cancellation. Holding a controller pose does not pause world generation; use set_paused for that. Keep the latest desired pose in the browser and bound pending commands so a slow connection does not replay old movement. Multiple clients share session state, so give one person control and make other clients viewers. The model API does not provide a controller lease. Start with a small number of tested scenes. Arbitrary image support does not establish reliable behavior for unseen robots, objects, or camera layouts. For a custom-policy experiment, first define an embodiment-compatible action mapping and evaluate the generated dynamics; the current controller API does not provide a causal action-to-observation stepping contract or task-success scores.

Get started

Policy inference

Simulation

Model families

Build an AgiBot simulation

Run the Python examples

Session contract

Controller payload

Build a mouse or trackpad playground

​Run the Python examples

​Session contract

​Controller payload

​Build a mouse or trackpad playground

Run the Python examples

Session contract

Controller payload

Build a mouse or trackpad playground