reactor/flux3-action-droid through the SDK and keep the session open while exchanging
observations and predictions. A camera track is a named stream of successive images from one
camera; publishing it attaches that stream to the session.
API at a glance
Send a new state request and match its
chunk_id to the returned step. Select a checkpoint before
the first request. Your application validates the output and executes it through its own controller.
New to Reactor? How the API works explains sessions, tracks,
commands, and the difference between an SDK message and the data returned by the example helper. For
a complete script, use Get your first actions.
Camera tracks
Publish RGB
uint8 images in height × width × 3 order. The reference client uses 360 × 640 per
track at 15 fps. The policy composes a 360 × 640 wrist view above two 180 × 320 exterior views;
different aspect ratios are resized, so preserve the reference camera geometry.
Every request waits for a newly delivered frame on each track after the request is accepted. Track
arrival is not proof of simultaneous capture with the supplied state. Keep capture timestamps in
your client.
Commands
The inner
state_json object is:
Each distinct request string is answered once. A byte-identical resend produces no second reply.
Unknown inner keys are ignored, so adding
"retry": 1 requests another prediction with the same
identifier and seed. New frames can change its actions; retries are not cached replays. Keep one
request outstanding and consume a request identifier only once in the controller. Invalid requests
can be dropped without a prediction.
Messages
The SDK receives{"type": "action_prediction", "data": {...}}. Its data contains:
Row
k targets (k + 1) / 15 seconds after the observation: a full chunk spans about 2.13 seconds.
Validate finite values and your controller limits before use. The API does not execute targets or
compensate for network delay.
checkpoint_selected reports checkpoint, locked, and available. command_error reports
command and a human-readable reason; the session stays open. Register handlers before requesting
a selection or prediction.
Episode boundary
Afterreset, the existing state_json value is not answered again. Submit a new request with
current observations. Clear local reply/action queues and do not execute a late reply from the
previous episode. Use a new session when the boundary is uncertain or when changing checkpoints.