reactor/cosmos-nano-policy-droid through the SDK and keep the session open while
exchanging observations and predictions. A camera track is a named stream of successive images
from one camera; publishing it attaches that stream to the session.
API at a glance
The first chunk arrives once valid observations and a task are available. After executing it, send
updated observations and report which prediction you executed to receive the next chunk. Your
application validates the output and executes it through its own controller.
New to Reactor? How the API works explains sessions, tracks,
commands, and the difference between an SDK message and the data returned by the example helper. For
a complete script, use Get your first actions.
Camera tracks
Publish all three tracks afterREADY. Supply RGB uint8 arrays shaped (H, W, 3).
These are the
reference simulator client’s
dimensions, not schema-enforced resolutions. The first-action fixture uses
(180, 320, 3) for all
views. The model composes and resizes the images internally; preserve camera identity and field of
view when adapting another rig. The example tracks publish at 15 fps.
The endpoint retains the newest frame from each view. Video delivery and state commands are
asynchronous; replies contain no frame timestamp or observation identifier. A settling delay reduces
stale-frame risk but does not guarantee synchronized observations.
Commands
proprio_json contains row lists, even for a single observation:
panda_joint1 through panda_joint7 in RoboLab), in radians. Gripper state is normalized
closure: 0 open, 1 closed. For history, use matching, nonempty (N, 7) and (N, 1) lists with
the current sample last. Validate finite values locally; the endpoint does not enforce every
physical constraint.
Action reply
Theaction_prediction message’s data object contains:
Each row is
[q1, q2, q3, q4, q5, q6, q7, gripper]. Joint values are absolute position targets in
radians, not velocities or increments. Gripper values use the same closure convention as the
input. The raw prediction is continuous and is not guaranteed to stay within [0, 1].
RoboLab
thresholds gripper output at > 0.5 to close; <= 0.5 opens. The endpoint does not apply this
threshold for you.
The reference simulation executes rows at 15 Hz: one chunk represents about 2.13 seconds of motion.
This is a control cadence, not a promise of 15 inference replies per second or an inference latency
bound. No generated video is returned.
Advance the loop
- Send the task, publish all camera views, then send measured robot state. The first prediction
requires no acknowledgement and returns
step: 0. - Check the action array’s shape, finite values, and expected
stepbefore executing it through your controller. Publish new camera images and send updated measured robot state after execution. - Send
set_executed_step_jsonwith a JSON string containing the receivedstepand the rows actually executed:{"step": 0, "action": [[...], ...]}. The next reply hasstep: 1. - Repeat for each chunk, checking that
stepincreases by one in each reply. Keep only one chunk outstanding at a time.
step greater than the
last one it accepted. Reporting the same or an older step does not request another prediction or
resend a previous reply.
This report tells Reactor that your client has made progress. Reactor does not verify that the robot
moved or compare the reported action rows with the prediction. If a request times out and you cannot
tell whether it completed, start a fresh session with current observations. Do not send a made-up
step to request another prediction.
Task changes and reset
A changed task applies to the next prediction. After the first chunk, changing the task alone does not request another prediction: you still need to report the executedstep.
reset restarts the prediction counter at 0 and clears stored camera frames. It does not clear
the stored robot state or executed-step reports. Those values can remain from before the reset.
Use a fresh session when starting a new episode or recovering from an uncertain timeout if you need
to distinguish new replies from earlier ones. Resetting the model never resets or stops the physical
robot; your controller must handle that separately.