Skip to main content
Connect to reactor/cosmos-nano-policy-droid through the SDK and keep the session open while exchanging observations and predictions. A camera track is a named stream of successive images from one camera; publishing it attaches that stream to the session.

API at a glance

The first chunk arrives once valid observations and a task are available. After executing it, send updated observations and report which prediction you executed to receive the next chunk. Your application validates the output and executes it through its own controller. New to Reactor? How the API works explains sessions, tracks, commands, and the difference between an SDK message and the data returned by the example helper. For a complete script, use Get your first actions.

Camera tracks

Publish all three tracks after READY. Supply RGB uint8 arrays shaped (H, W, 3). These are the reference simulator client’s dimensions, not schema-enforced resolutions. The first-action fixture uses (180, 320, 3) for all views. The model composes and resizes the images internally; preserve camera identity and field of view when adapting another rig. The example tracks publish at 15 fps. The endpoint retains the newest frame from each view. Video delivery and state commands are asynchronous; replies contain no frame timestamp or observation identifier. A settling delay reduces stale-frame risk but does not guarantee synchronized observations.

Commands

proprio_json contains row lists, even for a single observation:
Send measured positions, not the last requested targets. Joint columns follow DROID’s seven arm joints (panda_joint1 through panda_joint7 in RoboLab), in radians. Gripper state is normalized closure: 0 open, 1 closed. For history, use matching, nonempty (N, 7) and (N, 1) lists with the current sample last. Validate finite values locally; the endpoint does not enforce every physical constraint.

Action reply

The action_prediction message’s data object contains: Each row is [q1, q2, q3, q4, q5, q6, q7, gripper]. Joint values are absolute position targets in radians, not velocities or increments. Gripper values use the same closure convention as the input. The raw prediction is continuous and is not guaranteed to stay within [0, 1]. RoboLab thresholds gripper output at > 0.5 to close; <= 0.5 opens. The endpoint does not apply this threshold for you. The reference simulation executes rows at 15 Hz: one chunk represents about 2.13 seconds of motion. This is a control cadence, not a promise of 15 inference replies per second or an inference latency bound. No generated video is returned.

Advance the loop

  1. Send the task, publish all camera views, then send measured robot state. The first prediction requires no acknowledgement and returns step: 0.
  2. Check the action array’s shape, finite values, and expected step before executing it through your controller. Publish new camera images and send updated measured robot state after execution.
  3. Send set_executed_step_json with a JSON string containing the received step and the rows actually executed: {"step": 0, "action": [[...], ...]}. The next reply has step: 1.
  4. Repeat for each chunk, checking that step increases by one in each reply. Keep only one chunk outstanding at a time.
After the first prediction, the model waits for you to report an executed step greater than the last one it accepted. Reporting the same or an older step does not request another prediction or resend a previous reply. This report tells Reactor that your client has made progress. Reactor does not verify that the robot moved or compare the reported action rows with the prediction. If a request times out and you cannot tell whether it completed, start a fresh session with current observations. Do not send a made-up step to request another prediction.

Task changes and reset

A changed task applies to the next prediction. After the first chunk, changing the task alone does not request another prediction: you still need to report the executed step. reset restarts the prediction counter at 0 and clears stored camera frames. It does not clear the stored robot state or executed-step reports. Those values can remain from before the reset. Use a fresh session when starting a new episode or recovering from an uncertain timeout if you need to distinguish new replies from earlier ones. Resetting the model never resets or stops the physical robot; your controller must handle that separately.