Skip to main content

What is a stepped session?

A stepped session runs the model one step at a time, only when your app asks for a step, and returns each step’s output as raw arrays. It is one of the two session types. The other, the default, is a streamed session. Every model runs as a loop of steps. Each step reads the model’s current state and inputs and produces the next piece of output: a frame, a chunk of frames, or a stretch of audio. In a streamed session the model runs that loop on its own and plays the output to you in real time. In a stepped session the model waits. Your app calls step(), the model runs exactly one step, and the result comes back to the call that asked for it.
Every current SDK release supports stepped sessions. The model must run on reactor-runtime 3.9.0 or later. Step data, encoded step frames in every codec, and pushed frames sent as you push them need reactor-runtime 3.11.8 and Python SDK 1.15.0, JavaScript SDK 3.5.0, C++ SDK 2.8.0, Swift SDK 1.8.0, Java SDK 1.5.0, or Android SDK 1.3.0.

When to use it

Use a stepped session when you need to know exactly which input produced which output:
  • Agents and evaluation. Set a state, run one step, inspect the frames, decide the next action.
  • Offline or batch generation. Produce frames as fast as the GPU allows, with no playback clock.
  • Frame-exact control. Change the prompt or controls between two specific steps, never in the middle of one.
  • Processing output as data. Get frames as arrays you can save, score, or feed to another model, without decoding a video stream. A model can also return data with each step, such as an action chunk or a score.
Stay with a streamed session for anything a person watches live. A stepped session has no video or audio stream to show.

How a step works

A step request can carry three things. All of them are optional:
  1. state: values for the model’s state fields. Each key runs as the model’s set_<key> command, in ascending key order, as if you had sent it yourself.
  2. commands: a list of { command, data } entries, run in order after the state.
  3. media: input frames for this step, by input track name. Frames you push on their own reach the model’s input tracks as you push them, and the step reads them there.
A step request with nothing in it is valid. step() with no arguments runs a step on the model’s current state. The runtime first checks the whole request: every upload a call references has arrived, every value fits the model’s contract, every frame belongs to an input track the model takes and decodes. A request that fails a check changes nothing. Only then does the runtime apply it, as separate calls: the state setters, then the commands, then the media frames. Then the model’s process_input() decides whether to take the step, and the model generates. The step is not ready if the session cannot run a step yet, for example because it has not started. Nothing in the request is applied, and step() fails with a step not ready error. The step is refused if the request fails a check (an unknown command or track, a value the model rejects, an upload that never arrives, a frame that does not decode, more frames than a track holds) and nothing is applied. It is also refused when the model’s process_input() turns the step down. By then the calls and frames are applied, and they stay applied. step() fails with a step refused error. A command fails when one of the model’s handlers fails on a call. The calls before it stay applied, the calls after it and the frames are skipped, and the step does not run. step() fails with a step command failed error that names the entry at fault and the code the handler raised. A step that is not ready, refused, or failed on a command did not run. It gets no number, it does not count as one of the session’s steps, and the model does not save it. The session carries on. Otherwise the step runs, and its result is the answer to the request. Steps that run are numbered from 1. The result says what the step produced on each output track, as a list of frames. Messages the model sends while it runs a step arrive before that step’s result. A step can run and still fail: the model can report that it could not produce this step. That step does not fail step(). It returns a result with its error set, and the session carries on. If the session ends while a step is pending, for example because the model crashed, step() fails with a step session ended error. You cannot tell whether that step ran.

Commands between steps

Commands you send on their own, with sendCommand() (send_command() in Python), keep working in a stepped session. They change the model’s state, and the next step uses that state. So you can set a value once and then call step() with no arguments as many times as you need. The runtime applies one connection’s commands and step requests in the order they arrive. If several clients are connected to one session, Reactor does not coordinate them. A command from one client can change the state between two steps that another client requests.

Choosing the mode

Ask for a stepped session when you create the client. The SDK requests the mode when it creates the session, and checks that the model’s runtime really started the session that way.
The mode belongs to the session, not to the connection. A client that adopts an existing session must ask for the same mode the session was created with. If the modes differ, connect() fails with a generation mode mismatch error. Not every model can run a stepped session:
  • A model on a runtime older than 3.9.0 ignores the request and starts a streamed session. The SDK detects this and fails connect() with the same mismatch error, so your app never waits for steps that will not come.
  • A model that runs its own step loop cannot be stepped. Its runtime refuses to start the session, and connect() fails.

Creating a stepped session from your backend

If your backend creates sessions with POST /sessions and hands the session id to a client, set generation in the request’s extra_configs:
generation takes "streamed" (the default when the key is missing) or "stepped". Any other value, including a different spelling such as "Stepped", returns 400 Bad Request and creates no session. A 201 Created response does not prove that the session is stepped: a model on an older runtime still starts a streamed session. Connect the client with generation: "stepped" so the SDK checks the mode for you. Your backend can also read generation from GET /sessions/{session_id} once the session is waiting for a client.

Running steps

Push any input frames first, then call step(). The call resolves when the step’s result arrives.
The model keeps its state between steps, as it does in a streamed session. Send a state value again only when you want to change it. Using the SDK › Stepped generation covers input frames, reading results, and errors for each SDK.

Input frames

A step takes input frames in two ways, and you can mix them:
  • Pushed frames. pushFrame() (push_frame() in Python) on a published input track sends the frame to the model at once, on the same ordered channel as commands and steps. It goes into the input track’s buffer, and the next step reads it there. So a push, then a command, then a step reach the model in that order. A full input buffer keeps its newest frames, and the model drops frames that arrive while no stepped session runs.
  • The request’s media. Pass a step’s own frames by input track name. They go on the track after the frames you already pushed. A bad track or frame anywhere in media fails the step before any of it is applied.
The SDK checks each frame as you push it, and rejects one that does not match its track. A frame pushed before the session is ready waits in the SDK and goes out first, ahead of the next push or step. A push too large for one data-channel message is dropped, and the SDK reports it as an error event rather than failing a step. A model on a runtime older than 3.11.8 takes no frames between steps. With such a model, the SDK buffers pushed frames and sends them with the next step(), and the buffer drops the oldest frames when you push faster than steps run.

Encoded input frames

You can also push video you already have encoded, such as packets from a video file or a WebCodecs VideoEncoder, as one access unit per frame in H.264, VP8, VP9, or AV1. The runtime decodes a track’s frames as one stream, and the model reads RGB frames as it does for a raw push. Give the frames in decode order, and start with a keyframe. A step whose frames do not decode is refused.
pushEncodedFrame() (push_encoded_frame() in Python) pushes one encoded frame on a track instead. H.264 needs a model on reactor-runtime 3.10.0 or later, and VP8, VP9, and AV1 need 3.11.8 or later.

Encoded step frames

You do not need to encode anything yourself to keep steps small. When the client and the model share a video codec, each video track’s step frames travel encoded in both directions: the SDK encodes the frames you push as raw pixels, and decodes the result frames before step() returns. You still push and receive RGB. Each track picks the first codec both ends support, from H.264, VP9, AV1, and VP8 by default. A track with no shared codec stays raw. A lossy codec means a result pixel can differ slightly from what a raw frame would carry. setCodecPreference() (set_codec_preference() in Python) changes the order, for the whole session or one track. An empty list for the whole session turns encoding off, so every frame travels raw, and null (None) goes back to the default. Call it before connect() or during the session. It takes effect on the next connection or reconnect.
The step frame encodings map each track to what the connection negotiated: raw, h264, vp8, vp9, or av1. Each track is one stream for the whole connection, so it starts at a keyframe. If the client can no longer decode an output track’s stream, the SDK pauses and resumes that track, and the model starts it again at a keyframe. Encoded step frames need reactor-runtime 3.11.8 or later. With an older model, every frame travels raw.

Passing a file into a step

A file goes into a step the same way it goes into a command. First upload it to get a FileRef. Then put the FileRef in a command’s data, or use it as the state value of a field that takes an upload.
The runtime waits for an upload a step references before it applies the request. One that never arrives refuses the step.

Step results

A step result has these fields: A track’s rate is its frame rate for video, as the model set or measured it for this step, and its sample rate for audio. data arrives in the same message as the step’s frames, so a model whose answer is data alone, such as an action chunk or a score, needs no output track. The SDK does not read it: decode the JSON or the bytes yourself. The data counts toward the result’s message size, and it is not kept with the step’s files. Each frame carries: In Python, frame.array gives the same data as a read-only NumPy array with the frame’s shape and dtype. A frame in an encoding the SDK cannot decode fails the step with an unsupported frame encoding error.

Reading a step’s files

A model can keep extra files with a step, such as a depth map or a JSON summary. The step result lists them, but does not carry their bytes. The model keeps files only when it saves its steps, so files is empty for a model that does not. To download the files, call GET /sessions/{session_id}/steps/{step} with the session id and the step’s number. Authenticate with your API key or a token bound to the session.
The list ends with result.json, the step’s own result as the model saved it. A step that failed lists only result.json, which holds its error.
  • 202 Accepted means the step is not stored yet. Wait the seconds in the Retry-After header and ask again.
  • 404 Not Found means the step will not arrive: the model does not save its steps, or the session ended without saving this one.
The download URLs stop working at expires_at, a few minutes after the response. Call the endpoint again for fresh URLs, during or after the session. Saved steps are kept for a limited time, so download what you want to keep soon after the step.

Step results in each SDK

Every SDK returns the same fields, in its own style: A StepFile has name, size, and the content type, spelled contentType in JavaScript, Swift, Java, and Kotlin, and content_type in Python and C++.

What changes in a stepped session

A stepped session has no live media. In particular:
  • Output tracks stay silent. The model’s video and audio come back only in step results. A <ReactorView> or an output track handler receives nothing.
  • Input frames never travel as live media. A pushed frame goes to the model over the data channel, not as a video or audio stream. In JavaScript, publishTrack() takes no MediaStreamTrack in a stepped session: it ignores one with a console warning, and you push frames through the track it returns.
  • Nothing is recorded. A stepped session has no recording, and clip requests fail.
  • Frame rate does not pace anything. Steps run as fast as your app asks for them and the model can produce them. Each step still reports its tracks’ rate.
  • Commands still work. A command you send on its own changes the state the next step uses. See Commands between steps.
Step requests and results are large messages, so a stepped session needs a connection that negotiated data-channel chunking. Every SDK that supports stepped sessions does. Without it, connect() fails with a data channel chunking required error.

Message size

A step’s result travels back in one data-channel message. By default a message carries up to 64 MiB, and a connection uses the smaller of the client’s and the model’s limits. A model whose steps return more raises its limit in its reactor.yaml. Raise the client’s to match with maxMessageSize in JavaScript or max_message_size in Python, from 256 KiB to 1 GiB.

Errors

Step errors name the part of the request at fault, such as state.prompt or commands[1]. A step command failed error also carries the code the failing handler raised. To keep results small, have the model produce fewer or smaller frames per step, or raise the message size on both sides. To keep requests small, push frames on their own instead of in media, and keep encoded step frames on. A step that fails before it runs (message too large, or a refused request) has already taken its media frames, and with a model older than 3.11.8 the frames buffered for it too. The SDK does not keep them for the next step, so send the input again before you retry. Otherwise the retry runs on the model’s existing input, or fails for missing input. A step that ran but that the model reported as failed is not an error: step() returns its result with error set.

For model authors

A model needs no changes to run stepped sessions. The runtime calls the same hooks with the same return values. It only decides when a step runs. A model reports a failed step, keeps extra files with a step, or returns data with it through StepCompleted. See The Step Loop › Stepped sessions.

Next steps

Using the SDK

Push input frames, read step results, and handle step errors.

Sessions

How sessions are created, adopted, and ended.