Skip to main content
Connect to reactor/praxis-1-droid through the SDK. Keep the session open while you send observations and receive predictions. A camera track is a named stream of images from one camera. When you publish a track, the SDK attaches that stream to the session.

​
API at a glance

The first chunk arrives when the session has a task, enough camera frames, and enough robot state. After your robot executes a chunk, echo its step to receive the next chunk. Your application validates the output and executes it with its own controller. New to Reactor? How the API works explains sessions, tracks, commands, and messages.

​
Camera tracks

Publish both tracks after READY at 15 fps. Supply RGB uint8 frames no larger than 512 × 288 (width × height). If a frame is larger, scale it down and keep its aspect ratio before you publish it. The session does not use a frame that is larger than 512 × 288. It also starts that camera’s window again, so it waits for 25 new frames that fit. Each call uses the last 25 frames of every camera (1.6 seconds at 15 fps). When a camera changes resolution, its window starts again. Video and state travel separately, and replies contain no frame timestamp. Keep the cameras streaming while the robot moves.

​
Commands

A changed task applies to the next action_prediction. Write the task in plain words, as a sentence. A task can be within 300 characters and still be too long. The model accepts at most 64 tokens. A longer task fails with task_too_long.

​
Robot state

robot_state_json is a JSON string with the shape {"ee_pose": [[x, y, z, qw, qx, qy, qz], ...], "grip": [g, ...]}. Send one row for each control step at 15 Hz. Put the oldest row first and the current state last. Send at least the last 25 rows. To keep it short, the example shows one row.
Positions are in meters. Quaternions are scalar first (qw, qx, qy, qz), and they cannot be zero. Gripper values are absolute, and 0 is open. Send measured values, not the last requested targets. Send the state before set_executed_step. If a row contains a non-finite value, the session cannot use the state.

​
Pose frame

Select the frame that your robot uses to report its end-effector pose. The model converts your poses to the frame of its training data. It converts the returned targets back to your frame, so the targets use the frame that you sent.

​
Action reply

The data object of the action_prediction message contains: A chunk has 21 rows when it starts an episode and 24 rows (1.6 seconds at 15 Hz) after that. This is the control cadence. It is not a reply rate or a limit on inference latency. The reply does not include the predicted video. Pose targets are absolute, in the frame that pose_frame selects. They are not joint angles, velocities, or increments. Use your controller to calculate the joint motion. control_json contains the impedance gains of the DROID demonstrations: the joint and Cartesian stiffness and damping. Its format is {"type": "impedance", "kq": [7 values], "kqd": [7 values], "kx": [6 values], "kxd": [6 values]}. The gains are the same for all rows of a chunk. If your controller supports impedance control, apply them. If not, use them as a reference to tune your controller.

​
Other messages

prediction_failed.code is one of: After a failure, the session tries again about half a second later. That chunk starts a new episode. The session stays open.

​
Advance the loop

  1. Send the task, publish all camera tracks, and then send the measured robot state. The first prediction does not need an echo. It returns step: 0.
  2. Before you execute the chunk, check the rows, the values, and the step. All values must be finite. Execute the chunk with your controller.
  3. While the robot moves, keep publishing camera frames and send the measured state.
  4. Send set_executed_step with the received step. The session then calculates the next chunk from the frames and state of the executed chunk. The next reply has the next step.
  5. Repeat for each chunk. Make sure that step increases by one. Keep only one chunk outstanding at a time.
Until you echo the last step, the session waits. get_status shows this in waiting_for and awaiting_step. An echo tells Reactor that your client made progress. Reactor does not check that the robot moved.

​
Episodes and reset

reset_context starts a new episode. The next chunk does not use the chunks that the robot executed before. It does not wait for set_executed_step, and it has 21 rows. The step counter does not restart. context_reset.next_step gives the next value. A new session starts at step: 0. A reset of the model does not reset or stop the physical robot. Use your controller to do that.