> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reactor.inc/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To build and serve your own model, start at /deploy/development/quickstart and /deploy/development/overview. Deploying is the default path: reactor init scaffolds a workspace, reactor auth login authenticates, and reactor model deploy registers the model, publishes the release with the weights/ folder, and activates it on Reactor's GPUs, in one command from that workspace. Docker must be running, because the publish step builds the image locally. Bump model.version in reactor.yaml before redeploying a change, because a release that already has an image is reactivated as it is. Deployment access is granted per account, so contact team@reactor.inc if a deploy is refused. Every key in reactor.yaml is documented at /deploy/platform/reactor-yaml. Model code imports reactor_runtime; Python client code imports reactor_sdk. The runtime overview explains the model interface. Running the model on your own machine with reactor run is optional and needs a GPU you attach with --gpus; /deploy/development/local-testing covers that loop and pairs a complete brightness model with a Python client test in a separate brightness-test workspace.
> Reactor hosts multiple models, each with its own connect slug (modelName) and command/event schema. The video model catalog — slug, typed SDK package, and links to its schema — is at /model-api-reference/overview. Robotics policy documentation starts at /robotics/overview; X-WAM observations, actions, and client integration are under /robotics/xwam/; Cosmos3 Nano Policy DROID is under /robotics/cosmos/nano-policy-droid/. Some models expose one slug per experience (e.g. HappyOyster); always take the slug from the model's own pages, never guess it.
> Fastest path to a working app: `npx create-reactor-app my-app --model=<slug>` scaffolds a complete app with secure auth wired up. Typed TypeScript SDKs are published as @reactor-models/<model>; Python uses the base reactor-sdk package.
> Auth: exchange an API key (rk_...) for a JWT via POST https://api.reactor.inc/tokens from your server. Never put the API key in client-side code.
> Append .md to any docs URL for clean Markdown. Search these docs via the MCP server at https://docs.reactor.inc/mcp.

# Observations and actions

> XR-1 RoboCasa365 state history, camera alignment, and 12 active action channels.

Connect to `reactor/xr1-robocasa365` through the SDK and keep the session open while exchanging
observations and predictions. A **camera track** is a named stream of successive images from one
camera; publishing it attaches that stream to the session.

## API at a glance

| Direction | What crosses the connection |
| - | - |
| Your client → model | Three camera histories, four sampled state rows, a task instruction, and execution progress. |
| Model → your client | `action_prediction` messages; `data.action` contains 16 × 60 values, with 12 active simulator columns. |

Every prediction, including the first, requires a new execution-progress message and enough complete
camera sets. Your application validates the output and executes it through its own controller.

New to Reactor? [How the API works](/robotics/how-the-api-works) explains sessions, tracks,
commands, and the difference between an SDK message and the data returned by the example helper. For
a complete script, use [Get your first actions](/robotics/xr1/robocasa365/quickstart).

## Camera tracks

| Reactor track | RoboCasa observation key |
| - | - |
| `left_agentview` | `video.robot0_agentview_left` |
| `right_agentview` | `video.robot0_agentview_right` |
| `wrist_view` | `video.robot0_eye_in_hand` |

Use RGB `uint8` images; the upstream benchmark renders `(256, 256, 3)`. The server applies its
center crop and model image preprocessing, so do not apply the checkpoint's crop a second time.
Publish each captured three-view observation as a set. The server pairs the nth arrival on each
track; it does not match camera capture timestamps. Dropping or duplicating frames on one track can
silently misalign the history.

## Commands

| Command | Payload | Meaning |
| - | - | - |
| `set_task_description` | `{"task_description": "close the blender lid"}` | Task, at most 300 characters |
| `set_state_history_json` | `{"state_history_json": json.dumps({"state_history": rows})}` | Four oldest-first state rows; string at most 24,000 characters |
| `set_executed_step_json` | `{"executed_step_json": json.dumps({"step": executed_total})}` | Increasing execution progress; string at most 8,000 characters |
| `reset` | `{}` | Clear video history, prediction counter, and execution echo |

JSON-bearing fields are strings. The parser accepts exactly four equal-width rows of 1–60 finite
numbers, but checkpoint-compatible inputs use **14 columns**. Parser acceptance alone does not
establish a valid robot state.

### State layout

The
[upstream evaluator](https://github.com/XiaomiRobotics/Xiaomi-Robotics-1/blob/7c20088/eval_robocasa365/entry.py)
constructs each row as follows. Slice ends are exclusive.

| Columns | Observation | Convention |
| - | - | - |
| `0:3` | `state.end_effector_position_relative` | EE position relative to the robot base, metres |
| `3:6` | `state.end_effector_rotation_relative` | Convert `xyzw` quaternion to a three-value axis-angle vector, radians |
| `6:8` | `state.gripper_qpos` | Both gripper joint positions from the simulator, metres |
| `8:11` | `state.base_position` | Base position in the simulator world frame, metres |
| `11:14` | `state.base_rotation` | Convert `xyzw` quaternion to axis-angle, radians, in the world frame |

The server pads columns `14:60` and applies checkpoint state normalization. Send the simulator's
measurements without training-statistic normalization. Do not collapse the two gripper joint values
into a single open/closed action value.

<Note>
  The release `0.2.2` schema's state description labels the 14 values as left/right arm poses. That
  prose does not match this single-arm checkpoint's upstream evaluator. Use the EE/gripper/base
  layout above.
</Note>

### History and readiness

For the reference history settings, retain seven consecutive observations and select indices `t-6`,
`t-4`, `t-2`, and `t`, oldest first. At startup, clamp missing indices to the first observation. Use
the same sampling for state and all camera views.

The serving implementation maintains its own camera buffer and samples four frames at interval two
by default. It accepts state history already sampled by the client. Do not publish four pre-sampled
frames and assume the server will preserve their spacing: that can sample the history twice.

Every prediction, **including the first**, needs a fresh execution echo and enough complete camera
sets received. The cumulative minimum is four sets per accepted echo: 4 for the first prediction, 8
for the second, and so on. The echo's integer may count executed actions (`0`, `16`, `32`, …); it is
not used as a frame count. A compatible simulator bridge must align this receipt gate with history
sampling. See [simulation](/robotics/xr1/robocasa365/simulation).

## Action reply

The message envelope is `{"type": "action_prediction", "data": {...}}`.

| Field | Meaning |
| - | - |
| `action` | `(16, 60)` decoded action chunk; singular field name |
| `step` | Prediction counter starting at zero, not the client's echoed execution count |

Only `action[:, :12]` is used by the benchmark. Pass each selected row through RoboCasa's
`convert_action` to map the controller commands to the simulator action dictionary.

| Columns | RoboCasa action key | Meaning |
| - | - | - |
| `0:3` | `action.end_effector_position` | Controller translation command |
| `3:6` | `action.end_effector_rotation` | Controller rotation command |
| `6:7` | `action.gripper_close` | Gripper closing command |
| `7:11` | `action.base_motion` | Base motion (three values) and torso command (one value) |
| `11:12` | `action.control_mode` | Mobile-base controller mode selection |
| `12:60` | None | Unused packed columns; do not send to the simulator |

These are the simulator controller's scaled commands, not absolute joint targets or raw Cartesian
metres/radians. Checkpoint action denormalization has already run. RoboCasa's
[conversion and controller mapping](https://github.com/robocasa/robocasa/blob/456174f62b89b8fca99eaaf33949c29fec9cfc2a/robocasa/wrappers/gym_wrapper.py)
own the physical scaling and mode interpretation.

## Replanning

Execute the chosen window (upstream uses all 16 rows), collect aligned observations, update
`state_history_json`, and then send an increasing execution echo. Allow one outstanding request.
Changing state or task alone does not trigger another prediction. This variant has no prefix
conditioning or `prefix_rows` output. Track execution separately from response `step`.

At an episode boundary, stop execution, discard queued responses, and reset local history. Reconnect
for strict isolation: `reset` clears the server's history and counters but does not replace all
stored input fields or guarantee that old media has left the transport.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.