> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reactor.inc/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To build and serve your own model, start at /deploy/development/quickstart and /deploy/development/overview. Deploying is the default path: reactor init scaffolds a workspace, reactor auth login authenticates, and reactor model deploy registers the model, publishes the release with the weights/ folder, and activates it on Reactor's GPUs, in one command from that workspace. Docker must be running, because the publish step builds the image locally. Bump model.version in reactor.yaml before redeploying a change, because a release that already has an image is reactivated as it is. Deployment access is granted per account, so contact team@reactor.inc if a deploy is refused. Every key in reactor.yaml is documented at /deploy/platform/reactor-yaml. Model code imports reactor_runtime; Python client code imports reactor_sdk. The runtime overview explains the model interface. Running the model on your own machine with reactor run is optional and needs a GPU you attach with --gpus; /deploy/development/local-testing covers that loop and pairs a complete brightness model with a Python client test in a separate brightness-test workspace.
> Reactor hosts multiple models, each with its own connect slug (modelName) and command/event schema. The video model catalog — slug, typed SDK package, and links to its schema — is at /model-api-reference/overview. Robotics policy documentation starts at /robotics/overview; X-WAM observations, actions, and client integration are under /robotics/xwam/; Cosmos3 Nano Policy DROID is under /robotics/cosmos/nano-policy-droid/. Some models expose one slug per experience (e.g. HappyOyster); always take the slug from the model's own pages, never guess it.
> Fastest path to a working app: `npx create-reactor-app my-app --model=<slug>` scaffolds a complete app with secure auth wired up. Typed TypeScript SDKs are published as @reactor-models/<model>; Python uses the base reactor-sdk package.
> Auth: exchange an API key (rk_...) for a JWT via POST https://api.reactor.inc/tokens from your server. Never put the API key in client-side code.
> Append .md to any docs URL for clean Markdown. Search these docs via the MCP server at https://docs.reactor.inc/mcp.

# Cosmos3 Nano Policy reference

> Camera tracks, joint state, action layout, and executed-step flow control.

Connect to `reactor/cosmos-nano-policy-droid` through the SDK and keep the session open while
exchanging observations and predictions. A **camera track** is a named stream of successive images
from one camera; publishing it attaches that stream to the session.

## API at a glance

| Direction | What crosses the connection |
| - | - |
| Your client → model | Three camera streams, a task instruction, and measured arm/gripper state. |
| Model → your client | `action_prediction` messages; `data.action` contains 32 × 8 absolute joint/gripper targets. |

The first chunk arrives once valid observations and a task are available. After executing it, send
updated observations and report which prediction you executed to receive the next chunk. Your
application validates the output and executes it through its own controller.

New to Reactor? [How the API works](/robotics/how-the-api-works) explains sessions, tracks,
commands, and the difference between an SDK message and the data returned by the example helper. For
a complete script, use [Get your first actions](/robotics/cosmos/nano-policy-droid/quickstart).

## Camera tracks

Publish all three tracks after `READY`. Supply RGB `uint8` arrays shaped `(H, W, 3)`.

| Track | Camera | RoboLab gateway frame shape |
| - | - | - |
| `wrist_view` | Wrist camera | `(360, 640, 3)` |
| `exterior_view_1` | Left exterior camera | `(180, 320, 3)` |
| `exterior_view_2` | Right exterior camera | `(180, 320, 3)` |

These are the
[reference simulator client's](https://github.com/NVlabs/RoboLab/blob/ad45d4f974725d020f82c2b0d77d78533aeba2b3/policies/cosmos3/client.py)
dimensions, not schema-enforced resolutions. The first-action fixture uses `(180, 320, 3)` for all
views. The model composes and resizes the images internally; preserve camera identity and field of
view when adapting another rig. The example tracks publish at 15 fps.

The endpoint retains the newest frame from each view. Video delivery and state commands are
asynchronous; replies contain no frame timestamp or observation identifier. A settling delay reduces
stale-frame risk but does not guarantee synchronized observations.

## Commands

| SDK command | Payload | Constraint |
| - | - | - |
| `set_task_description` | `{"task_description": "put the banana in the bowl"}` | String, at most 300 characters |
| `set_proprio_json` | `{"proprio_json": "<JSON string>"}` | String, at most 8,000 characters |
| `set_executed_step_json` | `{"executed_step_json": "<JSON string>"}` | String, at most 8,000 characters |
| `reset` | `{}` | Restarts session flow control and the output counter |

`proprio_json` contains row lists, even for a single observation:

```json theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
{
  "joint_position": [[0.0, -0.6283, 0.0, -2.5133, 0.0, 1.885, 0.0]],
  "gripper_position": [[0.0]]
}
```

Send measured positions, not the last requested targets. Joint columns follow DROID's seven arm
joints (`panda_joint1` through `panda_joint7` in RoboLab), in radians. Gripper state is normalized
closure: `0` open, `1` closed. For history, use matching, nonempty `(N, 7)` and `(N, 1)` lists with
the current sample last. Validate finite values locally; the endpoint does not enforce every
physical constraint.

## Action reply

The `action_prediction` message's `data` object contains:

| Field | Meaning |
| - | - |
| `action` | `(32, 8)` nested array; singular field name |
| `step` | Server prediction counter: `0`, `1`, `2`, … within the session/reset interval |

Each row is `[q1, q2, q3, q4, q5, q6, q7, gripper]`. Joint values are **absolute position targets in
radians**, not velocities or increments. Gripper values use the same closure convention as the
input. The raw prediction is continuous and is not guaranteed to stay within `[0, 1]`.
[RoboLab](https://github.com/NVlabs/RoboLab/blob/ad45d4f974725d020f82c2b0d77d78533aeba2b3/policies/cosmos3/client.py)
thresholds gripper output at `> 0.5` to close; `<= 0.5` opens. The endpoint does not apply this
threshold for you.

The reference simulation executes rows at 15 Hz: one chunk represents about 2.13 seconds of motion.
This is a control cadence, not a promise of 15 inference replies per second or an inference latency
bound. No generated video is returned.

## Advance the loop

1. Send the task, publish all camera views, then send valid proprioception. The first prediction
   requires no acknowledgement and returns `step: 0`.
2. Execute the chunk in your application. Publish the next observations and send updated
   proprioception.
3. Send `set_executed_step_json` with a JSON string containing the received `step` and the rows
   actually executed: `{"step": 0, "action": [[...], ...]}`. The next reply has `step: 1`.
4. Keep one chunk outstanding. Validate shape, finite values, and the expected next counter.

The server opens the gate when the acknowledged counter exceeds the last acknowledged counter; it
does not verify execution or compare the echoed rows with its prediction. Repeating the same counter
does not request a retransmission. Do not invent a larger counter to recover a timeout.

## Task changes and reset

A changed task applies to the next prediction; it does not bypass the acknowledgement gate. `reset`
restarts the counter and clears retained camera frames, but does **not** clear the input state
fields. Old proprioception and acknowledgements can therefore still be present. Use a fresh session
after an uncertain timeout or at an episode boundary when you need unambiguous pairing. A model
reset never resets or stops the physical robot.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.