Skip to main content
Connect to reactor/flux3-action-droid through the SDK and keep the session open while exchanging observations and predictions. A camera track is a named stream of successive images from one camera; publishing it attaches that stream to the session.

API at a glance

Send a new state request and match its chunk_id to the returned step. Select a checkpoint before the first request. Your application validates the output and executes it through its own controller. New to Reactor? How the API works explains sessions, tracks, commands, and the difference between an SDK message and the data returned by the example helper. For a complete script, use Get your first actions.

Camera tracks

Publish RGB uint8 images in height × width × 3 order. The reference client uses 360 × 640 per track at 15 fps. The policy composes a 360 × 640 wrist view above two 180 × 320 exterior views; different aspect ratios are resized, so preserve the reference camera geometry. After accepting a request, the model waits for a new frame on each track. These frames may have been captured at different times from the supplied robot state. Keep capture timestamps in your client so you can check how closely the observations line up.

Commands

The inner state_json object is:
The model answers each distinct state_json string once. Sending exactly the same string again produces no second reply. Unknown fields are ignored when interpreting the request, but adding "retry": 1 changes the string and requests a new prediction with the same identifier and seed. That prediction uses new camera frames, so its actions can differ from the first reply. Keep one request outstanding at a time. If you retry, ensure your controller executes at most one reply for that request identifier. An invalid request may receive no prediction.

Messages

The SDK receives {"type": "action_prediction", "data": {...}}. Its data contains: Action rows are spaced at 15 Hz. Row k targets (k + 1) / 15 seconds after the observation, with k starting at 0; a full chunk spans about 2.13 seconds. Validate finite values and your controller limits before use. The API does not execute targets or compensate for network delay. checkpoint_selected reports checkpoint, locked, and available. command_error reports command and a human-readable reason; the session stays open. Register handlers before requesting a selection or prediction.

Episode boundary

reset clears the model’s episode memory and any unanswered request. It keeps the task instruction, camera tracks, and selected checkpoint. The previous state_json value does not trigger another prediction after reset; submit a new request with current observations. Discard pending replies and actions so a late reply from the previous episode cannot move the robot. Start a new session if you are unsure which episode a reply belongs to, or if you want to change checkpoints. Resetting the model does not stop the physical robot.