Skip to main content
Your Python client connects to a model on Reactor through the SDK. It sends camera images, robot state, and a task instruction. Then the model runs inference on Reactor’s GPUs and returns predicted actions. Your client runs alongside the robot or simulator and passes those actions to its controller.
Your robot or simulator sends images and state through your Python client to a model on Reactor. Predicted actions return to the client for validation and scheduling, then movement commands go to your controller.

Connection and data

Sessions

A session is a persistent connection between your client and one model. An API key authenticates access, and a model identifier such as reactor/cosmos-nano-policy-droid selects the model. The same session carries inputs and predictions across requests, allowing the model to retain history. Session setup includes allocating model capacity and establishing the network connection, so it can take time. The SDK’s READY status indicates that the connection is ready for inputs.

Input streams and commands

Camera images travel over tracks, named streams of RGB images. For example, wrist_view carries frames from the wrist camera. Commands carry structured JSON data: task instructions, robot state, and model controls. Each model’s reference defines the required tracks, command names, and payloads. Video and commands arrive separately: their arrival times do not establish when the images and state were captured. Each model’s prediction protocol defines how those inputs are used and what triggers inference.

Prediction messages

Predictions arrive as messages containing an action chunk, a sequence of control steps. Each row contains one step’s joint, end-effector, or gripper values; dimensions and units depend on the model. Your client validates the chunk and schedules execution through its controller. New observations feed subsequent predictions as actions execute. Depending on the model, the next prediction may require an explicit request or feedback about executed actions. Capture, network, inference, and execution all contribute to the delay between an observation and robot movement. Each model’s integration guide covers prediction timing and recovery.

Session lifecycle

  1. Select a model, provide your API key, and register message handlers before connecting.
  2. Connect and wait for READY before publishing tracks or sending commands.
  3. Publish the required camera tracks and send the task and measured robot state, following the model’s prediction protocol.
  4. Validate each action chunk and schedule it through your controller. Keep the session open and supply new observations and any required requests or execution feedback for subsequent predictions.
  5. Close the session when finished.

Python SDK and Robotics Cookbook

The quickstarts use the Python SDK and the Robotics Cookbook for camera publishing and message queues. Installation is included in each quickstart; the helpers are installed from the cookbook checkout separately from the SDK. The SDK’s on_message callback receives a dictionary with type and data. This abbreviated Cosmos Nano reply shows the first row of a 32-row action chunk:
The full Cosmos Nano action array has 32 rows. Each row contains seven joint targets in radians and one gripper value; step counts predictions. Other models return different action formats, described in their references. The cookbook’s ReactorSession.next_message(...) returns the inner data dictionary. The examples convert action values to NumPy arrays. See the helper reference for helper return values and the Python SDK reference for callbacks, commands, and errors. Each model’s quickstart provides a runnable example, and its troubleshooting page covers connection and prediction errors. The model overview links to these guides.