Check embodiment compatibility
The model expects DROID’s camera setup and interpretation of joint and gripper values. Your application must translate between that format and the format your robot’s software uses.
For example, a joint target of
0.5 means a target angle of 0.5 radians. It does not mean adding
0.5 radians to the joint’s current angle.
Check the
DROID robot definition
and
Cosmos3 simulator client
for the reference setup. Matching the number of input and output values does not establish that a
different Franka configuration or gripper is compatible with this model.
Implement the control loop
- Register a handler for action replies before connecting. Wait for
connect()to reachREADY, then start publishing all three camera tracks. - Send the task instruction and current measured robot state. The first reply should have
step: 0and an action array with 32 rows of eight values: seven joint targets and one gripper target per row. - Check that the values are finite and that executing the targets would stay within your robot’s position, speed, acceleration, and workspace limits. Pass the targets to your controller. The reference simulator executes one row at a time at 15 Hz.
- After execution, capture new images and measure the robot state again. Publish the images and
send the updated state, then send
set_executed_step_jsonwith the receivedstepand the action rows you actually executed. This tells the model it can return the next prediction. - Keep one action chunk outstanding at a time. Expect
step: 1afterstep: 0, thenstep: 2, and so on. Repeat the checks and execution for each chunk. - If a request times out or the connection drops, stop requesting or scheduling further actions and have your local controller hold or stop the robot according to your application’s policy. Start a fresh session with current observations before resuming.
Validate on your rig
Start with the simulation guide, or replay recorded observations through your integration, before connecting it to hardware. On your robot, first capture observations and inspect predictions with movement disabled. Check that the camera views are assigned correctly, joint angles use the expected units and order, and open and closed gripper values are not reversed. Check that images are recent and replies arrive in the expected order. Then test controlled tasks with your controller’s limits and emergency stop in place. Alongside task success, measure how long it takes from capturing an observation to executing its prediction, and how closely the robot’s measured joint positions follow the requested targets. Reporting an executedstep tells Reactor that your client is ready for another prediction; Reactor
cannot verify that the robot moved. Your local controller remains responsible for precise motor
timing and collision avoidance.
Observation timing
Camera images and robot state travel separately. They can arrive at the model at different times, even when captured together. Record capture times in your application and decide how old an image or state reading can be before you discard it. Also limit the time difference between the three camera views. The reply’sstep counter does not tell you when its input images were captured.
Send fresh observations as the robot moves. Repeatedly publishing an old image does not make it a
new observation. Track how old the observations are when their predicted actions execute, and how
many actions are waiting in your controller’s queue.