What is a stepped session?
A stepped session runs the model one step at a time, only when your app asks for a step, and returns each step’s output as raw arrays. It is one of the two session types. The other, the default, is a streamed session. Every model runs as a loop of steps. Each step reads the model’s current state and inputs and produces the next piece of output: a frame, a chunk of frames, or a stretch of audio. In a streamed session the model runs that loop on its own and plays the output to you in real time. In a stepped session the model waits. Your app callsstep(), the model runs exactly one step, and the result
comes back to the call that asked for it.
Every current SDK release supports stepped sessions. The model must run on
reactor-runtime 3.9.0
or later. Step data, encoded step frames in every codec,
and pushed frames sent as you push them need reactor-runtime 3.11.8 and Python
SDK 1.15.0, JavaScript SDK 3.5.0, C++ SDK 2.8.0, Swift SDK 1.8.0, Java SDK 1.5.0, or Android SDK
1.3.0.When to use it
Use a stepped session when you need to know exactly which input produced which output:- Agents and evaluation. Set a state, run one step, inspect the frames, decide the next action.
- Offline or batch generation. Produce frames as fast as the GPU allows, with no playback clock.
- Frame-exact control. Change the prompt or controls between two specific steps, never in the middle of one.
- Processing output as data. Get frames as arrays you can save, score, or feed to another model, without decoding a video stream. A model can also return data with each step, such as an action chunk or a score.
How a step works
A step request can carry three things. All of them are optional:state: values for the model’s state fields. Each key runs as the model’sset_<key>command, in ascending key order, as if you had sent it yourself.commands: a list of{ command, data }entries, run in order after the state.media: input frames for this step, by input track name. Frames you push on their own reach the model’s input tracks as you push them, and the step reads them there.
step() with no arguments runs a step on the model’s
current state.
The runtime first checks the whole request: every upload a call references has arrived, every value
fits the model’s contract, every frame belongs to an input track the model takes and decodes. A
request that fails a check changes nothing. Only then does the runtime apply it, as separate calls:
the state setters, then the commands, then the media frames. Then the model’s process_input()
decides whether to take the step, and the model generates.
The step is not ready if the session cannot run a step yet, for example because it has not
started. Nothing in the request is applied, and step() fails with a step not ready error.
The step is refused if the request fails a check (an unknown command or track, a value the model
rejects, an upload that never arrives, a frame that does not decode, more frames than a track holds)
and nothing is applied. It is also refused when the model’s process_input() turns the step down.
By then the calls and frames are applied, and they stay applied. step() fails with a step refused
error.
A command fails when one of the model’s handlers fails on a call. The calls before it stay
applied, the calls after it and the frames are skipped, and the step does not run. step() fails
with a step command failed error that names the entry at fault and the code the handler raised.
A step that is not ready, refused, or failed on a command did not run. It gets no number, it does
not count as one of the session’s steps, and the model does not save it. The session carries on.
Otherwise the step runs, and its result is the answer to the request. Steps that run are
numbered from 1. The result says what the step produced on each output track, as a list of frames.
Messages the model sends while it runs a step arrive before that
step’s result.
A step can run and still fail: the model can report that it could not produce this step. That step
does not fail step(). It returns a result with its error set, and the session carries on.
If the session ends while a step is pending, for example because the model crashed, step() fails
with a step session ended error. You cannot tell whether that step ran.
Commands between steps
Commands you send on their own, withsendCommand() (send_command() in Python), keep working in a
stepped session. They change the model’s state, and the next step uses that state. So you can set a
value once and then call step() with no arguments as many times as you need.
The runtime applies one connection’s commands and step requests in the order they arrive. If several
clients are connected to one session, Reactor does not coordinate them. A command from one client
can change the state between two steps that another client requests.
Choosing the mode
Ask for a stepped session when you create the client. The SDK requests the mode when it creates the session, and checks that the model’s runtime really started the session that way.connect() fails with a generation mode
mismatch error.
Not every model can run a stepped session:
- A model on a runtime older than 3.9.0 ignores the request and starts a streamed session. The SDK
detects this and fails
connect()with the same mismatch error, so your app never waits for steps that will not come. - A model that runs its own step loop cannot be stepped. Its runtime refuses to start the session,
and
connect()fails.
Creating a stepped session from your backend
If your backend creates sessions withPOST /sessions
and hands the session id to a client, set generation in the request’s extra_configs:
generation takes "streamed" (the default when the key is missing) or "stepped". Any other
value, including a different spelling such as "Stepped", returns 400 Bad Request and creates no
session.
A 201 Created response does not prove that the session is stepped: a model on an older runtime
still starts a streamed session. Connect the client with generation: "stepped" so the SDK checks
the mode for you. Your backend can also read generation from GET /sessions/{session_id} once the
session is waiting for a client.
Running steps
Push any input frames first, then callstep(). The call resolves when the step’s result arrives.
Input frames
A step takes input frames in two ways, and you can mix them:- Pushed frames.
pushFrame()(push_frame()in Python) on a published input track sends the frame to the model at once, on the same ordered channel as commands and steps. It goes into the input track’s buffer, and the next step reads it there. So a push, then a command, then a step reach the model in that order. A full input buffer keeps its newest frames, and the model drops frames that arrive while no stepped session runs. - The request’s
media. Pass a step’s own frames by input track name. They go on the track after the frames you already pushed. A bad track or frame anywhere inmediafails the step before any of it is applied.
step(), and the buffer drops the oldest frames
when you push faster than steps run.
Encoded input frames
You can also push video you already have encoded, such as packets from a video file or a WebCodecsVideoEncoder, as one access unit per frame in H.264, VP8, VP9, or AV1. The runtime decodes a
track’s frames as one stream, and the model reads RGB frames as it does for a raw push. Give the
frames in decode order, and start with a keyframe. A step whose frames do not decode is refused.
pushEncodedFrame() (push_encoded_frame() in Python) pushes one encoded frame on a track instead.
H.264 needs a model on reactor-runtime 3.10.0 or later, and VP8, VP9, and AV1 need 3.11.8 or
later.
Encoded step frames
You do not need to encode anything yourself to keep steps small. When the client and the model share a video codec, each video track’s step frames travel encoded in both directions: the SDK encodes the frames you push as raw pixels, and decodes the result frames beforestep() returns. You still push
and receive RGB. Each track picks the first codec both ends support, from H.264, VP9, AV1, and VP8
by default. A track with no shared codec stays raw. A lossy codec means a result pixel can differ
slightly from what a raw frame would carry.
setCodecPreference() (set_codec_preference() in Python) changes the order, for the whole session
or one track. An empty list for the whole session turns encoding off, so every frame travels raw,
and null (None) goes back to the default. Call it before connect() or during the session. It
takes effect on the next connection or reconnect.
raw, h264, vp8,
vp9, or av1. Each track is one stream for the whole connection, so it starts at a keyframe. If
the client can no longer decode an output track’s stream, the SDK pauses and resumes that track, and
the model starts it again at a keyframe. Encoded step frames need reactor-runtime 3.11.8 or later.
With an older model, every frame travels raw.
Passing a file into a step
A file goes into a step the same way it goes into a command. First upload it to get aFileRef. Then put the FileRef in a command’s
data, or use it as the state value of a field that takes an upload.
Step results
A step result has these fields:
A track’s
rate is its frame rate for video, as the model set or measured it for this step, and its
sample rate for audio.
data arrives in the same message as the step’s frames, so a model whose answer is data alone, such
as an action chunk or a score, needs no output track. The SDK does not read it: decode the JSON or
the bytes yourself. The data counts toward the result’s message size, and it is not
kept with the step’s files.
Each frame carries:
In Python,
frame.array gives the same data as a read-only NumPy array with the frame’s shape and
dtype.
A frame in an encoding the SDK cannot decode fails the step with an unsupported frame encoding
error.
Reading a step’s files
A model can keep extra files with a step, such as a depth map or a JSON summary. The step result lists them, but does not carry their bytes. The model keeps files only when it saves its steps, sofiles is empty for a model that does not.
To download the files, call
GET /sessions/{session_id}/steps/{step}
with the session id and the step’s number. Authenticate with your API key or a token bound to the
session.
result.json, the step’s own result as the model saved it. A step that failed
lists only result.json, which holds its error.
202 Acceptedmeans the step is not stored yet. Wait the seconds in theRetry-Afterheader and ask again.404 Not Foundmeans the step will not arrive: the model does not save its steps, or the session ended without saving this one.
expires_at, a few minutes after the response. Call the endpoint
again for fresh URLs, during or after the session. Saved steps are kept for a limited time, so
download what you want to keep soon after the step.
Step results in each SDK
Every SDK returns the same fields, in its own style:
A
StepFile has name, size, and the content type, spelled contentType in JavaScript, Swift,
Java, and Kotlin, and content_type in Python and C++.
What changes in a stepped session
A stepped session has no live media. In particular:- Output tracks stay silent. The model’s video and audio come back only in step results. A
<ReactorView>or an output track handler receives nothing. - Input frames never travel as live media. A pushed frame goes to the model over the data
channel, not as a video or audio stream. In JavaScript,
publishTrack()takes noMediaStreamTrackin a stepped session: it ignores one with a console warning, and you push frames through the track it returns. - Nothing is recorded. A stepped session has no recording, and clip requests fail.
- Frame rate does not pace anything. Steps run as fast as your app asks for them and the model
can produce them. Each step still reports its tracks’
rate. - Commands still work. A command you send on its own changes the state the next step uses. See Commands between steps.
connect() fails with a data channel chunking required error.
Message size
A step’s result travels back in one data-channel message. By default a message carries up to 64 MiB, and a connection uses the smaller of the client’s and the model’s limits. A model whose steps return more raises its limit in itsreactor.yaml. Raise the client’s to match with maxMessageSize in
JavaScript or max_message_size in Python, from 256 KiB to 1 GiB.
Errors
Step errors name the part of the request at fault, such as
state.prompt or commands[1]. A step
command failed error also carries the code the failing handler raised.
To keep results small, have the model produce fewer or smaller frames per step, or raise the
message size on both sides. To keep requests small, push frames on their own
instead of in media, and keep encoded step frames on.
A step that fails before it runs (message too large, or a refused request) has already taken its
media frames, and with a model older than 3.11.8 the frames buffered for it too. The SDK does not
keep them for the next step, so send the input again before you retry. Otherwise the retry runs on
the model’s existing input, or fails for missing input.
A step that ran but that the model reported as failed is not an error: step() returns its result
with error set.
For model authors
A model needs no changes to run stepped sessions. The runtime calls the same hooks with the same return values. It only decides when a step runs. A model reports a failed step, keeps extra files with a step, or returns data with it throughStepCompleted. See
The Step Loop › Stepped sessions.
Next steps
Using the SDK
Push input frames, read step results, and handle step errors.
Sessions
How sessions are created, adopted, and ended.