A camera track is a named stream of RGB images published through the SDK. Video carries the
images; SDK commands carry the task and state. How the API works
introduces sessions, tracks, and the
{type, data} message envelope.
Camera tracks
Publish both tracks after the session reachesREADY. Supply nonempty RGB uint8 arrays shaped
(H, W, 3) in raw LIBERO renderer orientation.
The service rotates each image by 180°, resizes and center-crops it to 224 × 224, and combines the
exterior view before the wrist view. Do not pre-rotate, concatenate, or normalize the frames in the
client. The upstream evaluator uses 256 × 256 renders; 224 × 224 is the model’s processed size, not
a required transport resolution.
A request waits for each track to deliver a frame with or after that request as observed by the
server. This does not establish equal capture times or atomic pairing with state. Keep both tracks
streaming and capture their images and state together in your client.
Commands
Prediction request
Booleans, numeric strings, and fractional request IDs are invalid. Additional JSON fields are
ignored. Set the task before requesting a prediction.
Convert
robot0_eef_quat from its xyzw quaternion representation to axis-angle. Send physical
state values without applying training-statistic normalization.
Action reply
The SDK receives{"type": "action_prediction", "data": {...}}. The example helper’s next_message
returns the inner data object.
The first six values use LIBERO controller scaling. They are neither absolute joint targets nor
unscaled metre/radian increments. The service already denormalizes actions and converts the gripper
convention; do not apply those conversions again. All 32 rows are predictions, with no conditioning
prefix to skip. The API supplies no per-row timestamps or mandatory execution frequency.
The reference
robosuite 1.4.0 OSC pose controller
clips the six motion inputs to
[-1, 1], then scales translation by 0.05 metres and rotation by
0.5 radians. Its
goal calculation
adds translation in the world frame and left-multiplies orientation by the axis-angle rotation.
These are simulator controller semantics, not a calibrated physical-robot mapping.
Requests, errors, and reset
Use one outstanding request. Matchstep before accepting a reply and consume each request once. A
new state value can replace a pending request. Byte-identical state_json does not trigger another
prediction. For a retry, keep chunk_id and seed and change a retry counter; newly received
frames can still change the result.
An empty state_json requests nothing. A nonempty malformed request emits command_error with
command and reason, and is dropped while the session remains open. Correct it and send changed
JSON. SDK command rejection is separate and can raise from send. An unexpected inference failure
can end the session.
After reset, send changed request bytes and fresh frames. Keep IDs increasing so a late reply from
the previous episode cannot match a new request. Reset acknowledges model state changes; it does not
stop or reset your robot.