Skip to main content
Reactor takes camera images, measured robot state, and a task instruction, then returns predicted joint and gripper targets. Your application sends those observations, checks the returned targets, and tells your robot’s controller when to execute them. The cookbook example connects Reactor to RoboLab. To use a physical robot, you need to connect your own camera and robot-control software and test the complete setup. The simulator example does not provide that hardware connection.

Check embodiment compatibility

The model expects DROID’s camera setup and interpretation of joint and gripper values. Your application must translate between that format and the format your robot’s software uses. For example, a joint target of 0.5 means a target angle of 0.5 radians. It does not mean adding 0.5 radians to the joint’s current angle. Check the DROID robot definition and Cosmos3 simulator client for the reference setup. Matching the number of input and output values does not establish that a different Franka configuration or gripper is compatible with this model.

Implement the control loop

  1. Register a handler for action replies before connecting. Wait for connect() to reach READY, then start publishing all three camera tracks.
  2. Send the task instruction and current measured robot state. The first reply should have step: 0 and an action array with 32 rows of eight values: seven joint targets and one gripper target per row.
  3. Check that the values are finite and that executing the targets would stay within your robot’s position, speed, acceleration, and workspace limits. Pass the targets to your controller. The reference simulator executes one row at a time at 15 Hz.
  4. After execution, capture new images and measure the robot state again. Publish the images and send the updated state, then send set_executed_step_json with the received step and the action rows you actually executed. This tells the model it can return the next prediction.
  5. Keep one action chunk outstanding at a time. Expect step: 1 after step: 0, then step: 2, and so on. Repeat the checks and execution for each chunk.
  6. If a request times out or the connection drops, stop requesting or scheduling further actions and have your local controller hold or stop the robot according to your application’s policy. Start a fresh session with current observations before resuming.
At 15 Hz, executing all 32 rows takes about 2.13 seconds. In this loop, the next prediction is requested after execution, so network transfer and inference add a wait between chunks. Your controller must handle that wait. If you request predictions earlier or execute fewer rows, test how those changes affect the age of the observations and the model’s behavior.

Validate on your rig

Start with the simulation guide, or replay recorded observations through your integration, before connecting it to hardware. On your robot, first capture observations and inspect predictions with movement disabled. Check that the camera views are assigned correctly, joint angles use the expected units and order, and open and closed gripper values are not reversed. Check that images are recent and replies arrive in the expected order. Then test controlled tasks with your controller’s limits and emergency stop in place. Alongside task success, measure how long it takes from capturing an observation to executing its prediction, and how closely the robot’s measured joint positions follow the requested targets. Reporting an executed step tells Reactor that your client is ready for another prediction; Reactor cannot verify that the robot moved. Your local controller remains responsible for precise motor timing and collision avoidance.

Observation timing

Camera images and robot state travel separately. They can arrive at the model at different times, even when captured together. Record capture times in your application and decide how old an image or state reading can be before you discard it. Also limit the time difference between the three camera views. The reply’s step counter does not tell you when its input images were captured. Send fresh observations as the robot moves. Repeatedly publishing an old image does not make it a new observation. Track how old the observations are when their predicted actions execute, and how many actions are waiting in your controller’s queue.