Skip to main content
Reactor receives camera images, measured robot state, and a task instruction. It returns predicted end-effector pose and gripper targets. Your application sends the observations and checks the returned targets. Then it tells the controller of your robot when to execute them. There is no reference client for a physical robot with this model yet. Connect your own camera and robot-control software, and test the complete setup.

​
Check embodiment compatibility

The model expects the cameras and state conventions of the YAM rig. Your application translates between that format and the software of your robot.

​
Implement the control loop

  1. Subscribe to action_prediction and prediction_failed. Connect, and wait for READY. Start to publish all three camera tracks at 15 fps.
  2. Send the task. Keep sending the measured robot state, at least the last 25 rows. The first reply has step: 0 and 21 rows.
  3. Check that all values are finite. Then check that the targets stay in the position, speed, acceleration, and workspace limits of your robot.
  4. Give the targets to your controller. The rows are 1/15th of a second apart. Use timestamps to schedule them.
  5. After execution, send the measured state. Then send set_executed_step with the received step. This tells the model that it can calculate the next chunk.
  6. Keep only one chunk outstanding at a time. Expect step to increase by one with each reply.
  7. If you receive prediction_failed or the connection stops, hold or stop the robot with your controller. Follow the safety policy of your application. After a failure, the next chunk starts a new episode. After a lost connection, start a new session.
A full chunk takes 1.6 seconds to execute. The model calculates the next prediction after your echo, so network transfer and inference add a wait between chunks. Make sure that your controller can hold the robot during that wait.

​
Validate on your rig

Before you connect your integration to hardware, test it with the simulator adapter or with recorded observations. On your robot, first disable movement. Capture observations, inspect the predictions, and check these points:
  • Each camera sends to the correct track.
  • The poses use the correct frame and units.
  • The open and closed gripper values are not reversed.
Then test controlled tasks. Keep the limits of your controller and the emergency stop active. Measure task success. Also measure the time from the capture of an observation to the execution of its prediction, and how accurately the robot follows the requested poses. Your local controller is responsible for motion timing and collision avoidance.

​
Observation timing

Camera images and robot state travel separately. They can arrive at the model at different times, even when you capture them together. Record the capture times in your application. Decide how old an observation can be before you discard it. The step counter of a reply does not show when its input images were captured.