Skip to main content
Connect to reactor/praxis-1-yam through the SDK. Keep the session open while you send observations and receive predictions. A camera track is a named stream of images from one camera. When you publish a track, the SDK attaches that stream to the session.

​
API at a glance

The first chunk arrives when the session has a task, enough camera frames, and enough robot state. After your robot executes a chunk, echo its step to receive the next chunk. Your application validates the output and executes it with its own controller. New to Reactor? How the API works explains sessions, tracks, commands, and messages.

​
Camera tracks

Publish all three tracks after READY at 15 fps. Supply RGB uint8 frames in 16:10 aspect, at most 416 pixels on the longer side. Scale each view down and keep its aspect ratio, for example from 1920 × 1200 to 416 × 260. The model was trained on 16:10 views, and it refuses other aspect ratios. The session does not use a frame that is longer than 416 pixels on its longer side. It also starts that camera’s window again, so it waits for 25 new frames that fit. Each call uses the last 25 frames of every camera (1.6 seconds at 15 fps). When a camera changes resolution, its window starts again. Video and state travel separately, and replies contain no frame timestamp. Keep the cameras streaming while the robot moves.

​
Commands

A changed task applies to the next action_prediction. Write the task in the style of the training captions: plain clauses joined with then, with no closing period. A task can be within 300 characters and still be too long. The model accepts at most 64 tokens. A longer task fails with task_too_long.

​
Robot state

robot_state_json is a JSON string with the shape {"left": {"ee_pose": [[x, y, z, qw, qx, qy, qz], ...], "grip": [g, ...]}, "right": {...}}. Send one row for each control step at 15 Hz. Put the oldest row first and the current state last. Send at least the last 25 rows of each arm. To keep it short, the example shows one row.
Positions are in meters. Quaternions are scalar first (qw, qx, qy, qz), and they cannot be zero. Gripper values are absolute, and 0 is open. Send measured values, not the last requested targets. Send the state before set_executed_step. If a row contains a non-finite value, the session cannot use the state.

​
Pose frame

Report the poses of both arms in the world frame of the rig. Both arms were recorded in this frame during training. The model does not convert frames. The returned targets use the same world frame.

​
Action reply

The data object of the action_prediction message contains: A chunk has 21 rows when it starts an episode and 24 rows (1.6 seconds at 15 Hz) after that. This is the control cadence. It is not a reply rate or a limit on inference latency. The reply does not include the predicted video. Pose targets are absolute, in the world frame of the rig. They are not joint angles, velocities, or increments. Use the controller of each arm to calculate its joint motion. The rows of the two arms with the same index have the same timestamp. Execute them together. There is no control-law field. Run the targets with the controller of each arm.

​
Other messages

prediction_failed.code is one of: After a failure, the session tries again about half a second later. That chunk starts a new episode. The session stays open.

​
Advance the loop

  1. Send the task, publish all camera tracks, and then send the measured robot state. The first prediction does not need an echo. It returns step: 0.
  2. Before you execute the chunk, check the rows, the values, and the step. All values must be finite. Execute the chunk with your controller.
  3. While the robot moves, keep publishing camera frames and send the measured state.
  4. Send set_executed_step with the received step. The session then calculates the next chunk from the frames and state of the executed chunk. The next reply has the next step.
  5. Repeat for each chunk. Make sure that step increases by one. Keep only one chunk outstanding at a time.
Until you echo the last step, the session waits. get_status shows this in waiting_for and awaiting_step. An echo tells Reactor that your client made progress. Reactor does not check that the robot moved.

​
Episodes and reset

reset_context starts a new episode. The next chunk does not use the chunks that the robot executed before. It does not wait for set_executed_step, and it has 21 rows. The step counter does not restart. context_reset.next_step gives the next value. A new session starts at step: 0. A reset of the model does not reset or stop the physical robot. Use your controller to do that.