Observation and action loop
- Use one session for each robot’s episode. Register message handlers before connecting, and wait
for
READYbefore publishing camera tracks. The SDK keeps the connection alive. - Send measured joint and gripper positions, start all three camera streams, then set the task prompt.
- Check each
action_chunkfor the expected(24, 8)shape and finite values. Check that it belongs to the current episode, and usechunk_indexandobs_seqto reject replies already handled. - Check that executing the targets would stay within your robot’s position, speed, acceleration, and workspace limits. Decide which pending actions to replace, then pass the accepted targets to your controller.
- Keep sending fresh images and measured state. The model produces more chunks without waiting for you to report which actions were executed.
panda_joint1 through panda_joint7.
Convert the measured gripper position to a closed fraction: 0 open, 1 closed. Returned joint
values are target angles to move to, rather than amounts to add to the current angles. Check and
convert the returned gripper value to your controller’s command format separately.
RoboLab executes all 24 rows at 15 Hz, taking 1.6 seconds per chunk. For a physical robot, test how
many rows to execute and when to replace the remaining targets with a newer prediction.
Each chunk is a new plan based on the model’s observations. Adding every chunk to the end of a queue
can leave the robot executing targets based on old images. Before switching to a newer plan, check
the difference between its targets and the robot’s current position; your controller must reject or
handle any large jump.
Freshness and temporal context
Send new camera images as you capture them. Repeating a still image can fill the model’s four-frame history with copies, removing the motion information it would otherwise receive. Images and state travel separately.obs_seq counts frames received by the model; it does not tell
you exactly when they were captured or prove that they match the state reading. Record capture and
reply times, model counters, and which action rows you executed. Limit how far apart in time the
camera views can be, and measure how old the observations are when actions execute. Also track how
many actions are waiting in your controller’s queue.
For simulation, use an adapter that manages image delivery and episode state, such as the
RoboLab gateway. Keep image publishing rate, robot control
rate, and inference duration separate when measuring the loop.
Set limits on how old observations and actions can be. If a camera stops updating, inference times
out, the connection drops, or targets are invalid or outside your limits, have your controller hold
or stop the robot. Do not keep executing the last chunk indefinitely.
Episode transitions
Have your controller stop or hold the robot before starting another episode. Changing the prompt starts a new context for predictions but does not reset episode counters. For a new episode, sendreset and wait for episode_reset. Discard pending actions, send fresh
measured state and images, then set the next prompt. The reset reply confirms that the model reset;
it does not confirm that the robot stopped. If an interruption leaves you unsure which episode a
reply belongs to, start a fresh session.
See the reference for exact command names and
troubleshooting for symptoms.