Skip to main content
Connect to reactor/lingbot-va through the SDK and keep the session open while exchanging observations and predictions. A camera track is a named stream of successive images from one camera; publishing it attaches that stream to the session.

​
API at a glance

The first chunk arrives without execution feedback and includes four conditioning rows. Report the rows you executed to receive subsequent predictions. Your application validates the output and executes it through its own controller. New to Reactor? How the API works explains sessions, tracks, commands, and the difference between an SDK message and the data returned by the example helper. For a complete script, use Get your first actions.

​
Camera tracks

Publish in this order after READY, with RGB uint8 arrays of shape (128, 128, 3): The model resizes other resolutions internally. Preserve camera identity and field of view. The reference wrapper vertically flips MuJoCo’s bottom-up frames using image[::-1]; this is not a 180-degree rotation. Do not apply this transform to a camera that already supplies upright images. Each view retains its newest frame. A later chunk commits the video observed during execution: 12 snapshots for the first commit, then 16. Longer windows retain the newest snapshots; shorter windows are resampled. Video and executed-action messages have no shared observation identifier, so delivery order approximates their pairing. A delay alone cannot guarantee synchronization.

​
Commands

The echo is a JSON-encoded nested row list, without a step wrapper. Encode finite numbers with json.dumps(rows, allow_nan=False, separators=(",", ":")). Preserve precision and validate row counts locally. Reset clears the policy memory, retained frames, and prediction counter, but leaves the input state fields intact: clear executed_action_json before resetting. A changed nonempty task also restarts policy memory and the counter. Use a fresh session when changing tasks so a reset cannot seed from the previous task. For an uncertain timeout or stale replies, close the connection and start a fresh session. Reset does not stop or reset your robot.

​
Action reply

The action_prediction message’s data contains action, a (16, 7) nested array, and step, the zero-based prediction counter. Rows are:
These are raw LIBERO controller inputs, already unnormalized through the checkpoint’s training quantiles. Do not unnormalize them again or treat them as metres, joint targets, or absolute poses. The rotation columns are axis-angle delta components, not Euler roll/pitch/yaw. The reference robosuite 1.4.0 controller configuration clips arm inputs to [-1, 1], then scales translation by 0.05 metres and rotation by 0.5 radians. Its goal calculation adds translation in the world frame and left-multiplies the current orientation by the axis-angle rotation. These are simulator controller semantics, not a calibrated physical-robot mapping. The Panda gripper uses the command sign: negative opens, positive closes, and zero preserves the internal target. The scalar is not a jaw width. Raw model values are continuous and are not guaranteed to lie within [-1, 1]; the controller applies its own handling.

​
Advance the loop

  1. In a fresh session, publish both views, clear the executed-action field, reset, and send the task. The first chunk needs no echo and returns step: 0.
  2. Skip rows 0:4 of that first chunk and execute rows 4:16. The skipped values are conditioning slots; normalized zero becomes nonzero motion after unnormalization.
  3. Publish video while executing. Send the 12 executed rows in executed_action_json when done. The first commit uses the cached seed prediction for conditioning; the echo opens the gate.
  4. For subsequent chunks, execute all 16 rows and echo exactly those 16 rows. Keep one chunk outstanding and check that the reply counter increments by one.
Progress requires a different nonempty echo string. An identical echo is not a retry and produces no new chunk. The first echo must parse as a nonempty list; subsequent echoes must reshape to (16, 7). Invalid echoes can leave the client waiting without a reply. Do not change action values just to force progress, or assume arbitrary partial execution is supported. If execution is interrupted, stop the episode and reset with fresh observations. The server cannot verify physical execution. The first-commit behavior also means modifying the seed actions does not update that commit’s cached action conditioning; validate this limitation before adapting the policy to a controller that clips or modifies commands.