Skip to main content
Connect to reactor/dreamzero through the SDK and keep the session open while exchanging observations and predictions. A camera track is a named stream of successive images from one camera; publishing it attaches that stream to the session.

API at a glance

Chunks stream as observations arrive; there is no separate predict command or execution acknowledgement. Your application validates the output and executes it through its own controller. New to Reactor? How the API works explains sessions, tracks, commands, and the difference between an SDK message and the data returned by the example helper. For a complete script, use Get your first actions.

Camera inputs

Publish RGB uint8 arrays shaped (H, W, 3) on all three named tracks after READY. The model resizes input images to (180, 320). RoboLab’s default second exterior image is black; keep that mapping explicit. Reversing the exterior tracks can leave the primary view black without producing a protocol error.

Commands

Call session.send(command, payload) with these payloads: Joint order is the DROID arm order, corresponding to panda_joint1 through panda_joint7 in RoboLab. Send finite measured values, not the last targets sent to the controller. State and images travel separately; sending state first does not guarantee that the model receives it together with the corresponding images.

Action output

The action_chunk envelope contains this data payload: When measured state is supplied, joint outputs are absolute target angles in radians. A value of 0.5 means move that joint to 0.5 radians; do not add it to the current angle or treat it as a speed. If joint state is missing, the model uses zeros as its reference instead of the robot’s measured position. Always supply measured state before executing predictions. Gripper output uses a closed fraction, with 0 open and 1 closed. Check its range before converting it to your controller’s command format; predictions can fall outside physical limits. The reference simulator uses 15 Hz control: 24 rows represent 1.6 seconds of targets if all are executed. That control rate is independent of inference chunk arrival rate. predicted_video is a composite video output. It carries black frames while preview is disabled; enabling set_video_preview adds decoding work. Preview frames are predictions, not sensor input.

Streaming and episode state

The first chunk uses one frame per camera. Subsequent chunks use a rolling window of up to four frames per view, padding short histories with the oldest frame. Inference waits until every camera has delivered a new frame since the preceding chunk. Missing one view can stall the whole loop. The model assigns a counter as it receives frames. obs_seq is the highest of those counters among the frames used for a chunk. It does not identify a client request or tell you when the images were captured. A prediction already being computed can arrive after you send newer observations. Use the counters to reject duplicate or older replies; they do not prove that a prediction used a particular image you just sent. set_prompt returns prompt_accepted. Changing the prompt starts a new context for predictions without resetting the episode counters. reset returns episode_reset and clears the prompt and measured robot state. Episode counters restart when inference resumes. The first chunk has chunk_index: 0; its obs_seq can already be greater than zero because the model has received frames. After reset, discard pending actions and send fresh state, images, and a prompt. Resetting the model does not stop the robot; your local controller must handle that separately.