> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reactor.inc/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To build and serve your own model, start at /deploy/development/quickstart and /deploy/development/overview. Deploying is the default path: reactor init scaffolds a workspace, reactor auth login authenticates, and reactor model deploy registers the model, publishes the release with the weights/ folder, and activates it on Reactor's GPUs, in one command from that workspace. Docker must be running, because the publish step builds the image locally. Bump model.version in reactor.yaml before redeploying a change, because a release that already has an image is reactivated as it is. Deployment access is granted per account, so contact team@reactor.inc if a deploy is refused. Every key in reactor.yaml is documented at /deploy/platform/reactor-yaml. Model code imports reactor_runtime; Python client code imports reactor_sdk. The runtime overview explains the model interface. Running the model on your own machine with reactor run is optional and needs a GPU you attach with --gpus; /deploy/development/local-testing covers that loop and pairs a complete brightness model with a Python client test in a separate brightness-test workspace.
> Reactor hosts multiple models, each with its own connect slug (modelName) and command/event schema. The video model catalog — slug, typed SDK package, and links to its schema — is at /model-api-reference/overview. Robotics policy documentation starts at /robotics/overview; X-WAM observations, actions, and client integration are under /robotics/xwam/; Cosmos3 Nano Policy DROID is under /robotics/cosmos/nano-policy-droid/. Some models expose one slug per experience (e.g. HappyOyster); always take the slug from the model's own pages, never guess it.
> Fastest path to a working app: `npx create-reactor-app my-app --model=<slug>` scaffolds a complete app with secure auth wired up. Typed TypeScript SDKs are published as @reactor-models/<model>; Python uses the base reactor-sdk package.
> Auth: exchange an API key (rk_...) for a JWT via POST https://api.reactor.inc/tokens from your server. Never put the API key in client-side code.
> Append .md to any docs URL for clean Markdown. Search these docs via the MCP server at https://docs.reactor.inc/mcp.

# Observations and actions

> X-WAM camera tracks, state layout, commands, and action interpretation.

Connect to `reactor/xwam` through the SDK and keep the session open while exchanging observations
and predictions. A **camera track** is a named stream of successive images from one camera;
publishing it attaches that stream to the session.

## API at a glance

| Direction | What crosses the connection |
| - | - |
| Your client → model | Three camera streams, a task instruction, and a request containing measured end-effector/gripper state. |
| Model → your client | `action_prediction` messages; `data.actions` contains 32 × 14 end-effector/gripper deltas. |

Send a new request and match its `chunk_id` to the returned `step`. Your application validates the
output and executes it through its own controller.

New to Reactor? [How the API works](/robotics/how-the-api-works) explains sessions, tracks,
commands, and the difference between an SDK message and the data returned by the example helper. For
a complete script, use [Get your first actions](/robotics/xwam/robotwin/quickstart).

## Camera tracks

Publish all three named video tracks. The reference client sends RGB `uint8` arrays with shape
`(240, 320, 3)` at 15 frames/s. This is the camera publishing rate, not the robot control rate.

| Track | View |
| - | - |
| `head_view` | Head camera |
| `left_wrist_view` | Left wrist camera |
| `right_wrist_view` | Right wrist camera |

Keep each track publishing while awaiting a reply, repeating the current observation when needed.
The model waits for a frame from every view received at least as recently as the request. Frame
arrival does not prove that the cameras captured simultaneously; your client owns capture alignment.
Swapping cameras under the correct track names is not detected by the API.

## Commands

Send model commands with `await reactor.send_command(name, payload)` after the session is `READY`.

| Command | Payload | Effect |
| - | - | - |
| `set_task_description` | `{"task_description": "place the empty cup onto the coaster"}` | Sets the episode instruction; nonempty, at most 300 characters |
| `set_state_json` | `{"state_json": json.dumps(request)}` | Requests one prediction; serialized JSON is at most 2,000 characters |
| `reset` | `{}` | Clears request bookkeeping for a new episode |

### Prediction request

`state_json` contains a JSON **string**, not a nested object.

| Field inside the string | Type | Meaning |
| - | - | - |
| `proprio` | 16 finite numbers | Current end-effector and gripper states, in the layout below |
| `chunk_id` | Integer | Client request identifier; increment for each new observation |
| `cfg` | Finite number, optional | Guidance scale; defaults to `0.0` |
| `env_rank` | Integer, optional | Evaluation seed component; defaults to `0` |
| `rollout_id` | Integer, optional | Evaluation seed component; defaults to `0` |
| `step_id` | Integer, optional | Evaluation seed component; defaults to `chunk_id` |
| `retry` | Client convention, optional | Increment to change the serialized request when retrying the same `chunk_id` |

Omit the seed fields for a new integration. Set them when replaying a recorded request whose seed
components are known. The server normalizes state and denormalizes outputs; send physical state in
the checkpoint's conventions, without applying training-statistic normalization yourself.

### State layout

Indices are zero-based; slice end indices are exclusive.

| Slice | Quantity | Convention |
| - | - | - |
| `0:3` | Left end-effector position | Metres, checkpoint global frame |
| `3:7` | Left orientation | Unit quaternion, `wxyz` |
| `7` | Left gripper | RoboTwin opening fraction: `0` closed, `1` open |
| `8:11` | Right end-effector position | Metres, checkpoint global frame |
| `11:15` | Right orientation | Unit quaternion, `wxyz` |
| `15` | Right gripper | RoboTwin opening fraction: `0` closed, `1` open |

The checkpoint frame differs from RoboTwin's raw end-pose frame. Apply the
[coordinate mapping](/robotics/xwam/robotwin/integration#coordinate-mapping) before sending state.

## Action reply

The SDK message envelope is `{"type": "action_prediction", "data": {...}}`.

| Field in `data` | Shape/type | Meaning |
| - | - | - |
| `actions` | `(32, 14)` numbers | Denormalized deltas, one row per control step |
| `proprios` | `(9, 16)` numbers | Predicted future states in the input layout; diagnostic output, not controller targets |
| `step` | Integer | Echo of the request's `chunk_id` |

Each action row has this layout:

| Slice | Quantity | Convention |
| - | - | - |
| `0:3` | Left translation delta | Metres, checkpoint global frame |
| `3:6` | Left rotation delta | Rotation vector: axis multiplied by angle in radians |
| `6` | Left gripper delta | Change in opening fraction |
| `7:10` | Right translation delta | Metres, checkpoint global frame |
| `10:13` | Right rotation delta | Rotation vector in radians |
| `13` | Right gripper delta | Change in opening fraction |

Each row is relative to the **previous step**, starting from the request's state. Accumulate
translation and gripper deltas; compose rotations as `R_next = R_delta @ R_previous`. These are
end-effector commands, not joint angles or motor commands. See
[action execution](/robotics/xwam/robotwin/integration#action-execution).

## Ordering and retries

Allow only one request at a time. Match `step` to the outstanding `chunk_id`; discard other replies
and consume a matching reply only once. A second state update can replace an unanswered request.

An unchanged `state_json` string is deduplicated. To retry, preserve the observation, `chunk_id`,
and seed fields and increment `retry`. This requests another prediction with the same sampling seed;
lossy video transport means numerical equality is not guaranteed. A retry does not authorize
executing a chunk twice.

At an episode boundary, stop execution, discard queued replies, send `reset`, and set the new task
and observations. Keep request IDs increasing within the session. A reply already in transit can
still arrive after reset.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.