> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reactor.inc/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To build and serve your own model, start at /deploy/development/quickstart and /deploy/development/overview. Deploying is the default path: reactor init scaffolds a workspace, reactor auth login authenticates, and reactor model deploy registers the model, publishes the release with the weights/ folder, and activates it on Reactor's GPUs, in one command from that workspace. Docker must be running, because the publish step builds the image locally. Bump model.version in reactor.yaml before redeploying a change, because a release that already has an image is reactivated as it is. Deployment access is granted per account, so contact team@reactor.inc if a deploy is refused. Every key in reactor.yaml is documented at /deploy/platform/reactor-yaml. Model code imports reactor_runtime; Python client code imports reactor_sdk. The runtime overview explains the model interface. Running the model on your own machine with reactor run is optional and needs a GPU you attach with --gpus; /deploy/development/local-testing covers that loop and pairs a complete brightness model with a Python client test in a separate brightness-test workspace.
> Reactor hosts multiple models, each with its own connect slug (modelName) and command/event schema. The video model catalog — slug, typed SDK package, and links to its schema — is at /model-api-reference/overview. Robotics policy documentation starts at /robotics/overview; The launch policies are Cosmos Nano DROID (/robotics/cosmos/nano-policy-droid/), DreamZero DROID (/robotics/dreamzero/droid/), and FLUX Action DROID (/robotics/flux-action/droid/), organized by model family and embodiment. Some models expose one slug per experience (e.g. HappyOyster); always take the slug from the model's own pages, never guess it.
> Fastest path to a working app: `npx create-reactor-app my-app --model=<slug>` scaffolds a complete app with secure auth wired up. Typed TypeScript SDKs are published as @reactor-models/<model>; Python uses the base reactor-sdk package.
> Auth: exchange an API key (rk_...) for a JWT via POST https://api.reactor.inc/tokens from your server. Never put the API key in client-side code.
> Append .md to any docs URL for clean Markdown. Search these docs via the MCP server at https://docs.reactor.inc/mcp.

# Stepped Sessions

> Run a model one step per request and get each step's frames back as arrays, instead of a live stream

## What is a stepped session?

A stepped session runs the model one step at a time, only when your app asks for a step, and returns
each step's output as raw arrays. It is one of the two
[session types](https://docs.reactor.inc/concepts/sessions#session-types). The other, the default, is a
[streamed session](https://docs.reactor.inc/concepts/sessions/streamed.md).

Every model runs as a loop of **steps**. Each step reads the model's current state and inputs and
produces the next piece of output: a frame, a chunk of frames, or a stretch of audio. In a streamed
session the model runs that loop on its own and plays the output to you in real time. In a stepped
session the model waits. Your app calls `step()`, the model runs exactly one step, and the result
comes back to the call that asked for it.

> [!NOTE]
>
> Every current SDK release supports stepped sessions. The model must run on `reactor-runtime` 3.9.0
> or later. [Step data](#step-results), [encoded step frames](#encoded-step-frames) in every codec,
> and [pushed frames](#input-frames) sent as you push them need `reactor-runtime` 3.11.8 and Python
> SDK 1.15.0, JavaScript SDK 3.5.0, C++ SDK 2.8.0, Swift SDK 1.8.0, Java SDK 1.5.0, or Android SDK
> 1.3.0.

## When to use it

Use a stepped session when you need to know exactly which input produced which output:

* **Agents and evaluation.** Set a state, run one step, inspect the frames, decide the next action.
* **Offline or batch generation.** Produce frames as fast as the GPU allows, with no playback clock.
* **Frame-exact control.** Change the prompt or controls between two specific steps, never in the
  middle of one.
* **Processing output as data.** Get frames as arrays you can save, score, or feed to another model,
  without decoding a video stream. A model can also return data with each step, such as an action
  chunk or a score.

Stay with a [streamed session](https://docs.reactor.inc/concepts/sessions/streamed.md) for anything a person watches live. A
stepped session has no video or audio stream to show.

## How a step works

A step request can carry three things. All of them are optional:

1. **`state`**: values for the model's state fields. Each key runs as the model's `set_<key>`
   command, in ascending key order, as if you had sent it yourself.
2. **`commands`**: a list of `{ command, data }` entries, run in order after the state.
3. **`media`**: input frames for this step, by input track name. Frames you push on their own reach
   the model's input tracks as you push them, and the step reads them there.

A step request with nothing in it is valid. `step()` with no arguments runs a step on the model's
current state.

The runtime first checks the whole request: every upload a call references has arrived, every value
fits the model's contract, every frame belongs to an input track the model takes and decodes. A
request that fails a check changes nothing. Only then does the runtime apply it, as separate calls:
the state setters, then the commands, then the `media` frames. Then the model's `process_input()`
decides whether to take the step, and the model generates.

```mermaid theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
sequenceDiagram
    participant App as Your app (SDK)
    participant Model as Model (runtime)
    App->>Model: pushFrame() sends each input frame at once
    App->>Model: step(state, commands, media)
    alt The session cannot run a step yet
        Model-->>App: step_not_ready (nothing applied)
    else The request fails a check
        Model-->>App: step_refused (nothing applied)
    else A command handler fails
        Model->>Model: State setters, then commands, up to the one that fails
        Model-->>App: step_command_failed (earlier calls stay applied)
    else process_input() refuses the step
        Model->>Model: State setters, commands, frames, then process_input()
        Model-->>App: step_refused (the calls and frames stay applied)
    else The step runs
        Model->>Model: State setters, commands, frames, process_input(), generate()
        Model-->>App: Messages the model sends during the step
        Model-->>App: Step n result: frames per output track, data, files, error
    end
```

The step is **not ready** if the session cannot run a step yet, for example because it has not
started. Nothing in the request is applied, and `step()` fails with a step not ready error.

The step is **refused** if the request fails a check (an unknown command or track, a value the model
rejects, an upload that never arrives, a frame that does not decode, more frames than a track holds)
and nothing is applied. It is also refused when the model's `process_input()` turns the step down.
By then the calls and frames are applied, and they stay applied. `step()` fails with a step refused
error.

A **command fails** when one of the model's handlers fails on a call. The calls before it stay
applied, the calls after it and the frames are skipped, and the step does not run. `step()` fails
with a step command failed error that names the entry at fault and the code the handler raised.

A step that is not ready, refused, or failed on a command did not run. It gets no number, it does
not count as one of the session's steps, and the model does not save it. The session carries on.

Otherwise the step runs, and its **result** is the answer to the request. Steps that run are
numbered from 1. The result says what the step produced on each output track, as a list of frames.
[Messages](https://docs.reactor.inc/concepts/commands-and-messages.md) the model sends while it runs a step arrive before that
step's result.

A step can run and still fail: the model can report that it could not produce this step. That step
does not fail `step()`. It returns a result with its `error` set, and the session carries on.

If the session ends while a step is pending, for example because the model crashed, `step()` fails
with a step session ended error. You cannot tell whether that step ran.

### Commands between steps

Commands you send on their own, with `sendCommand()` (`send_command()` in Python), keep working in a
stepped session. They change the model's state, and the next step uses that state. So you can set a
value once and then call `step()` with no arguments as many times as you need.

The runtime applies one connection's commands and step requests in the order they arrive. If several
clients are connected to one session, Reactor does not coordinate them. A command from one client
can change the state between two steps that another client requests.

## Choosing the mode

Ask for a stepped session when you create the client. The SDK requests the mode when it creates the
session, and checks that the model's runtime really started the session that way.

```typescript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
import { Reactor } from "@reactor-team/js-sdk";

const reactor = new Reactor({ modelName: "your-model-name", generation: "stepped" });
await reactor.connect(jwt);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
from reactor_sdk import Reactor

reactor = Reactor("your-model-name", api_key=api_key, generation="stepped")
await reactor.connect()
```

The mode belongs to the session, not to the connection. A client that
[adopts an existing session](https://docs.reactor.inc/concepts/sessions#adopting-an-existing-session) must ask for the same
mode the session was created with. If the modes differ, `connect()` fails with a generation mode
mismatch error.

Not every model can run a stepped session:

* A model on a runtime older than 3.9.0 ignores the request and starts a streamed session. The SDK
  detects this and fails `connect()` with the same mismatch error, so your app never waits for steps
  that will not come.
* A model that runs its own step loop cannot be stepped. Its runtime refuses to start the session,
  and `connect()` fails.

### Creating a stepped session from your backend

If your backend creates sessions with [`POST /sessions`](https://docs.reactor.inc/platform-api/sessions/create-a-session)
and hands the session id to a client, set `generation` in the request's `extra_configs`:

```json theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
{
  "model": { "name": "your-model-name" },
  "extra_configs": { "generation": "stepped" }
}
```

`generation` takes `"streamed"` (the default when the key is missing) or `"stepped"`. Any other
value, including a different spelling such as `"Stepped"`, returns `400 Bad Request` and creates no
session.

A `201 Created` response does not prove that the session is stepped: a model on an older runtime
still starts a streamed session. Connect the client with `generation: "stepped"` so the SDK checks
the mode for you. Your backend can also read `generation` from `GET /sessions/{session_id}` once the
session is waiting for a client.

## Running steps

Push any input frames first, then call `step()`. The call resolves when the step's result arrives.

```typescript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
const camera = await reactor.publishTrack("camera");
await camera.pushFrame(imageData);

const result = await reactor.step({
  state: { prompt: "a red fox" },
  commands: [{ command: "advance", data: { frames: 4 } }],
});
if (result.error) console.warn(`step ${result.step} failed:`, result.error);

const video = result.tracks.find((t) => t.track === "main_video");
console.log(result.step, result.elapsed, video?.frames.length);

// Data the model returned with the step, if any. A dict arrives as JSON.
if (result.data) console.log(JSON.parse(new TextDecoder().decode(result.data)));

// Nothing new: run the next step on the current state.
const next = await reactor.step();
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
import json

from reactor_sdk import StepCommand

camera = await reactor.track("camera").publish()
camera.push_frame(frame)

result = await reactor.step(
    state={"prompt": "a red fox"},
    commands=[StepCommand("advance", {"frames": 4})],
)
if result.error is not None:
    print(f"step {result.step} failed: {result.error}")

frames = result.track("main_video").frames
print(result.step, result.elapsed, len(frames))

# Data the model returned with the step, if any. A dict arrives as JSON.
if result.data is not None:
    print(json.loads(result.data))

# Nothing new: run the next step on the current state.
next_result = await reactor.step()
```

The model keeps its state between steps, as it does in a streamed session. Send a state value again
only when you want to change it.

[Using the SDK › Stepped generation](https://docs.reactor.inc/sdk-reference/using-the-sdk.md#stepped-generation) covers input
frames, reading results, and errors for each SDK.

### Input frames

A step takes input frames in two ways, and you can mix them:

* **Pushed frames.** `pushFrame()` (`push_frame()` in Python) on a published input track sends the
  frame to the model at once, on the same ordered channel as commands and steps. It goes into the
  input track's buffer, and the next step reads it there. So a push, then a command, then a step
  reach the model in that order. A full input buffer keeps its newest frames, and the model drops
  frames that arrive while no stepped session runs.
* **The request's `media`.** Pass a step's own frames by input track name. They go on the track
  after the frames you already pushed. A bad track or frame anywhere in `media` fails the step
  before any of it is applied.

The SDK checks each frame as you push it, and rejects one that does not match its track. A frame
pushed before the session is ready waits in the SDK and goes out first, ahead of the next push or
step. A push too large for one data-channel message is dropped, and the SDK reports it as an error
event rather than failing a step.

A model on a runtime older than 3.11.8 takes no frames between steps. With such a model, the SDK
buffers pushed frames and sends them with the next `step()`, and the buffer drops the oldest frames
when you push faster than steps run.

#### Encoded input frames

You can also push video you already have encoded, such as packets from a video file or a WebCodecs
`VideoEncoder`, as one access unit per frame in H.264, VP8, VP9, or AV1. The runtime decodes a
track's frames as one stream, and the model reads RGB frames as it does for a raw push. Give the
frames in decode order, and start with a keyframe. A step whose frames do not decode is refused.

```typescript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
const result = await reactor.step({
  media: {
    camera: [{ codec: "h264", data: accessUnit, width: 1280, height: 720 }],
  },
});
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
from reactor_sdk import EncodedFrame

frames = [EncodedFrame(bytes(packet), width=1280, height=720) for packet in window]
result = await reactor.step(media={"camera": frames})
```

`pushEncodedFrame()` (`push_encoded_frame()` in Python) pushes one encoded frame on a track instead.
H.264 needs a model on `reactor-runtime` 3.10.0 or later, and VP8, VP9, and AV1 need 3.11.8 or
later.

### Encoded step frames

You do not need to encode anything yourself to keep steps small. When the client and the model share
a video codec, each video track's step frames travel encoded in both directions: the SDK encodes the
frames you push as raw pixels, and decodes the result frames before `step()` returns. You still push
and receive RGB. Each track picks the first codec both ends support, from H.264, VP9, AV1, and VP8
by default. A track with no shared codec stays raw. A lossy codec means a result pixel can differ
slightly from what a raw frame would carry.

`setCodecPreference()` (`set_codec_preference()` in Python) changes the order, for the whole session
or one track. An empty list for the whole session turns encoding off, so every frame travels raw,
and `null` (`None`) goes back to the default. Call it before `connect()` or during the session. It
takes effect on the next connection or reconnect.

```typescript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
reactor.setCodecPreference(["vp9", "h264"]); // VP9 when both ends have it
reactor.setCodecPreference(["av1"], { track: "main_video" });
reactor.setCodecPreference([]); // every frame raw

await reactor.connect(jwt);
console.log(reactor.getStepFrameEncodings()); // { camera: "vp9", main_video: "av1" }
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
reactor.set_codec_preference(["vp9", "h264"])  # VP9 when both ends have it
reactor.set_codec_preference(["av1"], track="main_video")
reactor.set_codec_preference([])  # every frame raw

await reactor.connect()
print(reactor.step_frame_encodings)  # {"camera": "vp9", "main_video": "av1"}
```

The step frame encodings map each track to what the connection negotiated: `raw`, `h264`, `vp8`,
`vp9`, or `av1`. Each track is one stream for the whole connection, so it starts at a keyframe. If
the client can no longer decode an output track's stream, the SDK pauses and resumes that track, and
the model starts it again at a keyframe. Encoded step frames need `reactor-runtime` 3.11.8 or later.
With an older model, every frame travels raw.

### Passing a file into a step

A file goes into a step the same way it goes into a command. First
[upload it](https://docs.reactor.inc/concepts/file-uploads.md) to get a `FileRef`. Then put the `FileRef` in a command's
`data`, or use it as the `state` value of a field that takes an upload.

```typescript JavaScript theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
const ref = await reactor.uploadFile(file);

// As the state value of a field that takes an upload
await reactor.step({ state: { reference_image: ref } });
// Or in a command's data
await reactor.step({ commands: [{ command: "set_image", data: { image: ref } }] });
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
ref = await reactor.upload_file("photo.jpg")

# As the state value of a field that takes an upload
await reactor.step(state={"reference_image": ref})
# Or in a command's data
await reactor.step(commands=[StepCommand("set_image", {"image": ref})])
```

The runtime waits for an upload a step references before it applies the request. One that never
arrives refuses the step.

## Step results

A step result has these fields:

| Field | Description |
| - | - |
| `step` | The step's number in the session, counted from 1. |
| `tracks` | For each output track that produced output: the track's name, its kind, its `rate`, and its frames in production order. A track can produce no frames. |
| `elapsed` | How long the model's `generate()` took for this step, in seconds, or the time the model reported for the step. |
| `files` | The extra files the model kept with the step: each file's name, content type, and size in bytes. Empty when the model kept none or the step was not saved. |
| `error` | Why the step failed, when it ran and the model reported a failure. Not set when the step worked. |
| `data` | What the model returned with the step besides its media, as bytes. A dict the model returned arrives encoded as JSON. Not set when the model returned none. |

A track's `rate` is its frame rate for video, as the model set or measured it for this step, and its
sample rate for audio.

`data` arrives in the same message as the step's frames, so a model whose answer is data alone, such
as an action chunk or a score, needs no output track. The SDK does not read it: decode the JSON or
the bytes yourself. The data counts toward the result's [message size](#message-size), and it is not
kept with the step's files.

Each frame carries:

| Field | Description |
| - | - |
| `encoding` | How the frame travelled: `raw`, or the track's codec. `data` is decoded either way. |
| `dtype` | The element type, in NumPy notation: `uint8` for video, `int16` for audio. |
| `shape` | `(height, width, 3)` RGB for video. `(channels, samples)` planar PCM for audio. |
| `data` | The pixels or samples, row-major. A typed array in JavaScript, `bytes` in Python. |
| `metadata` | The bytes the model attached to the frame, if any. |

In Python, `frame.array` gives the same data as a read-only NumPy array with the frame's `shape` and
`dtype`.

A frame in an encoding the SDK cannot decode fails the step with an unsupported frame encoding
error.

### Reading a step's files

A model can keep extra files with a step, such as a depth map or a JSON summary. The step result
lists them, but does not carry their bytes. The model keeps files only when it saves its steps, so
`files` is empty for a model that does not.

To download the files, call
[`GET /sessions/{session_id}/steps/{step}`](https://docs.reactor.inc/platform-api/sessions/get-the-files-of-a-saved-step)
with the session id and the step's number. Authenticate with your API key or a token bound to the
session.

```json theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
{
  "step": 3,
  "files": [
    { "name": "summary.json", "content_type": "application/json", "url": "https://..." },
    { "name": "result.json", "content_type": "application/json", "url": "https://..." }
  ],
  "expires_at": "2026-11-15T12:05:00Z"
}
```

The list ends with `result.json`, the step's own result as the model saved it. A step that failed
lists only `result.json`, which holds its `error`.

* **`202 Accepted`** means the step is not stored yet. Wait the seconds in the `Retry-After` header
  and ask again.
* **`404 Not Found`** means the step will not arrive: the model does not save its steps, or the
  session ended without saving this one.

The download URLs stop working at `expires_at`, a few minutes after the response. Call the endpoint
again for fresh URLs, during or after the session. Saved steps are kept for a limited time, so
download what you want to keep soon after the step.

### Step results in each SDK

Every SDK returns the same fields, in its own style:

| SDK | Result type | `step` | `elapsed` | `files` | `error` | `data` |
| - | - | - | - | - | - | - |
| JavaScript | `StepResult` | `number` | `number`, seconds | `StepFile[]` | `string \| undefined` | `Uint8Array \| undefined` |
| Python | `StepResult` | `int` | `float`, seconds | `tuple[StepFile, ...]` | `str \| None` | `bytes \| None` |
| C++ | `StepResult` | integer | `double`, seconds | `std::vector<StepFile>` | `std::optional<std::string>` | `std::optional<std::vector<std::uint8_t>>` |
| Swift | `StepResult` | integer | `Duration` | `[StepFile]` | `String?` | `Data?` |
| Java | `StepResult` | integer | `double`, seconds | `List<StepFile>` | `Optional<String>` | `byte[]`, `null` when none |
| Kotlin/Android | `StepResult` | `Long` | `Double`, seconds | `List<StepFile>` | `String?` | `ByteArray?` |

A `StepFile` has `name`, `size`, and the content type, spelled `contentType` in JavaScript, Swift,
Java, and Kotlin, and `content_type` in Python and C++.

## What changes in a stepped session

A stepped session has no live media. In particular:

* **Output tracks stay silent.** The model's video and audio come back only in step results. A
  `<ReactorView>` or an output track handler receives nothing.
* **Input frames never travel as live media.** A pushed frame goes to the model over the data
  channel, not as a video or audio stream. In JavaScript, `publishTrack()` takes no
  `MediaStreamTrack` in a stepped session: it ignores one with a console warning, and you push
  frames through the track it returns.
* **Nothing is recorded.** A stepped session has no recording, and clip requests fail.
* **Frame rate does not pace anything.** Steps run as fast as your app asks for them and the model
  can produce them. Each step still reports its tracks' `rate`.
* **Commands still work.** A command you send on its own changes the state the next step uses. See
  [Commands between steps](#commands-between-steps).

Step requests and results are large messages, so a stepped session needs a connection that
negotiated data-channel chunking. Every SDK that supports stepped sessions does. Without it,
`connect()` fails with a data channel chunking required error.

### Message size

A step's result travels back in one data-channel message. By default a message carries up to 64 MiB,
and a connection uses the smaller of the client's and the model's limits. A model whose steps return
more raises its limit in its `reactor.yaml`. Raise the client's to match with `maxMessageSize` in
JavaScript or `max_message_size` in Python, from 256 KiB to 1 GiB.

## Errors

| Error | When | What changed |
| - | - | - |
| Step not ready | The session cannot run a step yet, for example because it has not started. | Nothing was applied. The step did not run. The session continues. |
| Step refused | The request failed a check, or the model's `process_input()` turned the step down. | A failed check applied nothing. A turned-down step keeps its calls and frames applied. |
| Step command failed | A model handler failed on one of the request's calls. | Calls before it stay applied, the rest are skipped. The step did not run. |
| Step session ended | The session ended while the step was pending. | Unknown whether the step ran. The session is over. |
| Step result too large | The step ran, but its result does not fit in one message on this connection. | The step ran. The session continues. |
| Message too large | The request itself, with its `media` frames, is too large to send. | Nothing was sent. The step's frames are dropped: send them again before you retry. |
| Generation mode mismatch | On `connect()`, when the session's mode is not the mode the client asked for. | The client does not connect. |
| Data channel chunking required | On `connect()`, when the connection cannot carry step requests and results. | The client does not connect. |
| Unsupported frame encoding | A result frame uses an encoding the SDK cannot decode. | The step ran. |
| Decode error | The SDK lost an output track's encoded stream and cannot decode this result. | The step ran. The SDK restarts the track's stream at a keyframe for the next step. |

Step errors name the part of the request at fault, such as `state.prompt` or `commands[1]`. A step
command failed error also carries the code the failing handler raised.

To keep results small, have the model produce fewer or smaller frames per step, or raise the
[message size](#message-size) on both sides. To keep requests small, push frames on their own
instead of in `media`, and keep [encoded step frames](#encoded-step-frames) on.

A step that fails before it runs (message too large, or a refused request) has already taken its
`media` frames, and with a model older than 3.11.8 the frames buffered for it too. The SDK does not
keep them for the next step, so send the input again before you retry. Otherwise the retry runs on
the model's existing input, or fails for missing input.

A step that ran but that the model reported as failed is not an error: `step()` returns its result
with `error` set.

## For model authors

A model needs no changes to run stepped sessions. The runtime calls the same hooks with the same
return values. It only decides when a step runs. A model reports a failed step, keeps extra files
with a step, or returns data with it through `StepCompleted`. See
[The Step Loop › Stepped sessions](https://docs.reactor.inc/deploy/development/reactor-app/step-loop.md#stepped-sessions).

## Next steps

> [card] [Using the SDK](https://docs.reactor.inc/sdk-reference/using-the-sdk.md#stepped-generation)
>
> Push input frames, read step results, and handle step errors.

> [card] [Sessions](https://docs.reactor.inc/concepts/sessions)
>
> How sessions are created, adopted, and ended.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.