Tracks
There are no inbound tracks: the model reads no camera and no microphone. The video track keeps
one size for the length of a queue —
set_canvas chooses it and is only accepted while the queue is
empty and nothing plays, since queued clips are already built at the size in force. The audio track
is generated jointly with the video on every clip, so the two never drift.
Session lifecycle
Once the connection reaches ready, the session begins idle against a black output. Adding clips doesn’t start any playback: a built clip only reaches the tracks when aplay (or, with
set_autoplay on, the playout queue’s front-most ready clip) asks for it.
By default the stream flushes to black the moment a clip ends or is stopped and holds there
until the next play. Two things change that boundary behavior, and they’re separate:
set_flush_on_clip_endwithenabled: falsemakes the stream hold the last frame across a normal end or stop instead of going black.- A clip enqueued with
continue_from_clip_idchains off its source clip — when autoplay moves from the source into it, the hand-off is seamless regardless of the flush setting: the model builds the sequel to open on the source’s last frame, so there is no boundary to see.
enqueue or a stop from any client affects everyone, and
every client receives every state_update and queue_update. Generation is gated on having an
audience — with nobody connected no new build starts, though a build already underway finishes into
the queue.
state_update.valid_commands names exactly what the session would accept at that moment, so a
frontend enables and greys out controls from the snapshot instead of re-deriving these rules. The
diagram above shortens the valid lists to illustrative commands for that reason — the snapshot, not
this page, is the authoritative listing at runtime; see the
state_update payload table for what it carries.
Commands
Send commands withreactor.sendCommand() on the base SDK. Commands that have a reply resolve their
awaited call with it (enqueue → clip_queued, move → clip_moved, and so on); commands without
one take effect silently, the change visible in the next broadcast state_update / queue_update.
A refused command has no effect and fails its own call with a stable code and a reason; see
Refusals for every code and how to read one.
enqueue
Enter a prompt into the generation queue. Each request snapshots the session’s conditions
(set_clip_seconds, set_seed, set_canvas) as they stand, receives a UUID, and joins the queue —
at the back, or at position if given (0 = next build). The reply, clip_queued, carries the full
ClipInfo. The generation queue is bounded (default 20); full, the command is
refused with queue_full.
A plain enqueue opens a clip from text alone. Two optional fields open it from a picture instead —
at most one of them may be set on any enqueue:
starting_frameopens the clip from an uploaded still (image-to-video). The frame is fitted to the session canvas and animates forward.continue_from_clip_idopens the clip from the last frame of another clip, chaining the two into a continuing scene. The source can sit in either queue or inqueue_update.history, including a clip that hasn’t built yet — the sequel then waits for its anchor and builds out of order relative to strict queue position, without blocking the builds behind it. A chain keeps the exposure of the clip it opened from, so a long chain does not drift brighter or darker link by link.
ending_framecloses the clip on an uploaded still, fitted to the session canvas the same way asstarting_frame. Pair it withstarting_frameorcontinue_from_clip_idfor first-and-last generation, or send it alone to fix only the ending.ending_from_clip_idcloses the clip on another clip’s clean retained last frame — the same source rules ascontinue_from_clip_id(either queue orqueue_update.history, including a clip that hasn’t built yet). It may name the same clip ascontinue_from_clip_idto create a hold that opens and closes on one generated frame.
starting_frame is a FileRef — the object
uploadFile() returns, with upload_id, name, mime_type,
size, and optional width / height. Upload first, then embed the reference:
empty_prompt when the prompt is empty; prompt_too_long when it is over the
1024-token budget (the reason gives the token count, and the model trims nothing for you);
queue_full when the generation queue is full; invalid_request when both opener fields come in
the same request (“a clip opens from one picture”) or both ending fields do (“a clip closes on one
picture”); unknown_clip when continue_from_clip_id or ending_from_clip_id names a clip that is
in neither queue nor queue_update.history (the id doesn’t exist anywhere the model can
reference); or invalid_image when an uploaded frame is not a decodable image.
metadata is for the frontend, not the model. The model stores it and echoes it back on every
message that references the clip; it never parses it. Carry whatever your application needs to track
— which request produced the clip, who asked for it, text to show while it plays. JSON fits if you
want structure.
move
Reposition a clip within whichever queue holds it. position 0 is the front; a value past the back
is clamped. The reply, clip_moved, names the queue and the landing position.
Fails with
unknown_clip when the clip_id is unknown.
play
Stream a built clip. With no clip_id, play takes the playout queue’s front; a clip_id plays
that specific clip. Playing consumes the entry — the clip leaves the queue when it starts.
clip_started fires as the first frames reach the tracks. Fails with already_playing if another
clip is playing, unknown_clip if the id is unknown, clip_not_ready if the named clip is still
generating, or no_ready_clip if no clip_id is given and the playout queue is empty. play has no
reply, so it resolves the same way whether it starts a clip or not: read a refusal from the
error event.
pop
Remove a clip from whichever queue holds it, freeing its slot; a build in flight for it is discarded
when it finishes (the GPUs cannot abandon mid-build). Replies clip_popped.
Fails with unknown_clip when the id is unknown, or with clip_referenced when a queued (unbuilt)
clip is named by another queued clip’s continue_from_clip_id or ending_from_clip_id. Popping the
source would leave the
dependent scene no frame to open from or close on, so the model refuses until you’ve popped the
dependents first (back-to-front for a bulk clear). Built clips don’t hit this: they persist in queue_update.history after being popped and
can still anchor continuations.
stop
Cut the playing clip. Both queues are untouched. With autoplay on, stop acts as a skip — the
next ready clip starts on its own. Emits clip_stopped. By default the picture goes black within a
fraction of a second, and the rest of the clip is discarded. With set_flush_on_clip_end off, the
transport first drains what it already holds, up to about two seconds. The picture then freezes on
the last frame. Fails with not_playing when nothing is playing.
get_queue
Replies with both queues plus the retained build history — the same payload as queue_update. Front
first, each entry a full ClipInfo.
set_autoplay
When enabled is true, the playout queue’s front clip starts on its own whenever nothing is
playing; stop then acts as a skip. When false (the default), playback waits for an explicit
play. Replies autoplay_accepted.
set_clip_seconds
Default clip length for enqueues that carry no seconds. The value snaps to what the model can
produce; the reply, clip_length_accepted, carries the effective value — read it back before
assuming a duration.
set_seed
Default seed for enqueues that carry none. Each such enqueue advances the default by one, so seeds
are per-clip, not per-session. enqueue with an explicit seed leaves the default untouched.
Replies seed_accepted.
Re-enqueueing the same prompt with the same seed, length, and canvas reproduces the same clip
(approximately — compiled kernels can reorder floating-point operations).
set_canvas
Video size for the session, by aspect (16:9, 1:1, 9:16, 4:3). Only accepted while the queue
is empty and nothing is playing, since queued clips were built at the size in force. Replies
canvas_accepted carrying the exact pixel width and height. Fails with session_not_empty
otherwise, or with invalid_aspect on an unsupported aspect.
set_flush_on_clip_end
Whether the stream cuts to black the moment a clip ends, is stopped, or is followed by a
non-continuing clip — enabled: true is the default. With enabled: false, those transitions
hold the clip’s last frame instead.
This toggle has no say in a genuine continuation: when autoplay moves from a clip into one enqueued
with continue_from_clip_id pointing at it, the sequel opens on the source’s last frame by
construction and the boundary has no cut either way. set_flush_on_clip_end is what you set for
every boundary that isn’t a hand-off. Replies flush_accepted.
reset
Drop the generation queue, the playout queue, and the retained build history; cut any playing clip;
and restore every session default (canvas, seed, clip length, autoplay, flush). Replies
session_reset, carrying how many clips were dropped and whether a clip was playing.
get_state
Replies with the full state_update snapshot.
Refusals
A refused command changes nothing and fails its own call. Only the client that sent it is told. The failure carries a stablecode to branch on and a reason written for your user:
Any command can also fail with
invalid_command when its payload breaks the schema (a negative
seed, say), and a command carrying an upload with unresolved_upload when the file cannot be
fetched.
On the JavaScript SDK the awaited call resolves undefined, and the error event fires first with
operation: "sendCommand" and the code. Every command except play and stop declares a reply, so
for those undefined means refused. play and stop resolve undefined on success too, so only
the error event tells. On the Python SDK, send_command() raises a ReactorError with the code.
command_error message ({ command, reason }) to every connected client
when it refuses a command. Read the refusal off the call instead: the broadcast reaches clients that
did not send the command, and a future major release removes it. Releases before 1.9.0 refuse
only with that broadcast, and their call resolves as if it succeeded. Treat a reply-declaring command
that resolves undefined (None in Python) with no error as refused, as the example’s fallback
reason does. See Replies and refusals for the
SDK side.
Messages
Every message arrives as JSON{ "type": "<name>", "data": { … } }. The Reaches column says
whether a message is delivered to every connected client (everyone) or only to the client whose
command produced it (the caller).
When a build fails, every queued clip that references it fails too. Each of those clips gets its own
clip_failed naming the source it lost.
A refused command fails its own call; see Refusals. FastH3 also broadcasts
command_error ({ command: string, reason: string }) to every connected client when it refuses a
command, until a future major release removes it.
queue_update payload
Carries everything a client needs to render the queue: generation (clips waiting to build, front
first), playout (built clips waiting to play, front first), and history (built clips no longer
queued — played, stopped, or popped — whose last frame is still retained, oldest first). history
entries are not playable again (play refuses them), but each is a legal continue_from_clip_id
source. The oldest history entries are evicted as new builds finish.
state_update payload
A full snapshot of everything observable except the queue’s contents, so a client can render UI from
this alone. Used by a scheduling client to steer; also carries every deployment-published live bound
a writer must not hardcode.
ClipInfo payload
Every message that references a clip embeds the full ClipInfo, so a client never has to join a
UUID against an earlier message.
Example handler (broadcasts only):
Complete example
Enqueue a base scene, then chain a continuation off it so the two play as one shot. Autoplay drives both; thecontinue_from_clip_id hand-off is seamless.
The continuation prompt re-describes the setting from scratch (“the same foothills… rain”) and
opens on a described hard cut rather than inheriting the take — continuity carries the frame,
not the description or the camera; the discipline is the whole prompt
guide.