Tracks
There are no inbound tracks: the model reads no camera and no microphone. The video track keeps
one size for the length of a queue —
set_canvas chooses it and is only accepted while the queue is
empty and nothing plays, since queued clips are already built at the size in force. The audio track
is generated jointly with the video on every clip, so the two never drift.
Session lifecycle
Once the connection reaches ready, the session begins idle against a black output. Adding clips doesn’t start any playback: a built clip only reaches the tracks when aplay (or, with
set_autoplay on, the playout queue’s front-most ready clip) asks for it.
By default the stream flushes to black the moment a clip ends or is stopped and holds there
until the next play. Two things change that boundary behavior, and they’re separate:
set_flush_on_clip_endwithenabled: falsemakes the stream hold the last frame across a normal end or stop instead of going black.- A clip enqueued with
continue_from_clip_idchains off its source clip — when autoplay moves from the source into it, the hand-off is seamless regardless of the flush setting: the model builds the sequel to open on the source’s last frame, so there is no boundary to see.
enqueue or a stop from any client affects everyone, and
every client receives every state_update and queue_update. Generation is gated on having an
audience — with nobody connected no new build starts, though a build already underway finishes into
the queue.
state_update.valid_commands names exactly what the session would accept at that moment, so a
frontend enables and greys out controls from the snapshot instead of re-deriving these rules. The
diagram above shortens the valid lists to illustrative commands for that reason — the snapshot, not
this page, is the authoritative listing at runtime; see the
state_update payload table for what it carries.
Commands
Send commands withreactor.sendCommand() on the base SDK. Commands that have a reply resolve their
awaited call with it (enqueue → clip_queued, move → clip_moved, and so on); commands without
one take effect silently, the change visible in the next broadcast state_update / queue_update.
Any rejected command has no effect and is answered by a broadcast command_error
naming the command and the reason.
enqueue
Enter a prompt into the generation queue. Each request snapshots the session’s conditions
(set_clip_seconds, set_seed, set_canvas) as they stand, receives a UUID, and joins the queue —
at the back, or at position if given (0 = next build). The reply, clip_queued, carries the full
ClipInfo. The generation queue is bounded (default 20); full, the command is
rejected.
A plain enqueue opens a clip from text alone. Two optional fields open it from a picture instead —
at most one of them may be set on any enqueue:
starting_frameopens the clip from an uploaded still (image-to-video). The frame is fitted to the session canvas and animates forward.continue_from_clip_idopens the clip from the last frame of another clip, chaining the two into a continuing scene. The source can sit in either queue or inqueue_update.history, including a clip that hasn’t built yet — the sequel then waits for its anchor and builds out of order relative to strict queue position, without blocking the builds behind it.
starting_frame is a FileRef — the object
uploadFile() returns, with upload_id, name, mime_type,
size, and optional width / height. Upload first, then embed the reference:
command_error when: the prompt is empty or over 800 characters; the generation
queue is full; both picture fields come in the same request (“a clip opens from one picture”); or
continue_from_clip_id names a clip that is in neither queue nor queue_update.history (the id
doesn’t exist anywhere the model can reference).
metadata is for the frontend, not the model. The model stores it and echoes it back on every
message that references the clip; it never parses it. Carry whatever your application needs to track
— which request produced the clip, who asked for it, text to show while it plays. JSON fits if you
want structure.
move
Reposition a clip within whichever queue holds it. position 0 is the front; a value past the back
is clamped. The reply, clip_moved, names the queue and the landing position.
Rejected with
command_error when the clip_id is unknown.
play
Stream a built clip. With no clip_id, play takes the playout queue’s front; a clip_id plays
that specific clip. Playing consumes the entry — the clip leaves the queue when it starts.
clip_started fires as the first frames reach the tracks. Rejected with command_error if another
clip is playing, the id is unknown, or the named clip is still generating.
pop
Remove a clip from whichever queue holds it, freeing its slot; a build in flight for it is discarded
when it finishes (the GPUs cannot abandon mid-build). Replies clip_popped.
Rejected when the id is unknown — or when a queued (unbuilt) clip is the source of another queued
clip’s continue_from_clip_id chain. Popping the source would leave the dependent scene nothing to
open from, so the model refuses until you’ve popped the dependents first (back-to-front for a bulk
clear). Built clips don’t hit this: they persist in queue_update.history after being popped and
can still anchor continuations.
stop
Cut the playing clip. Both queues are untouched; whatever the transport held is discarded. With
autoplay on, stop acts as a skip — the next ready clip starts on its own. Emits clip_stopped.
What the stream shows next depends on set_flush_on_clip_end: black by default, the stopped clip’s
last frame if flush is off. Rejected when nothing is playing.
get_queue
Replies with both queues plus the retained build history — the same payload as queue_update. Front
first, each entry a full ClipInfo.
set_autoplay
When enabled is true, the playout queue’s front clip starts on its own whenever nothing is
playing; stop then acts as a skip. When false (the default), playback waits for an explicit
play. Replies autoplay_accepted.
set_clip_seconds
Default clip length for enqueues that carry no seconds. The value snaps to what the model can
produce; the reply, clip_length_accepted, carries the effective value — read it back before
assuming a duration.
set_seed
Default seed for enqueues that carry none. Each such enqueue advances the default by one, so seeds
are per-clip, not per-session. enqueue with an explicit seed leaves the default untouched.
Replies seed_accepted.
Re-enqueueing the same prompt with the same seed, length, and canvas reproduces the same clip
(approximately — compiled kernels can reorder floating-point operations).
set_canvas
Video size for the session, by aspect (16:9, 1:1, 9:16, 4:3). Only accepted while the queue
is empty and nothing is playing, since queued clips were built at the size in force. Replies
canvas_accepted carrying the exact pixel width and height. Rejected with command_error
otherwise, or on an unsupported aspect.
set_flush_on_clip_end
Whether the stream cuts to black the moment a clip ends, is stopped, or is followed by a
non-continuing clip — enabled: true is the default. With enabled: false, those transitions
hold the clip’s last frame instead.
This toggle has no say in a genuine continuation: when autoplay moves from a clip into one enqueued
with continue_from_clip_id pointing at it, the sequel opens on the source’s last frame by
construction and the boundary has no cut either way. set_flush_on_clip_end is what you set for
every boundary that isn’t a hand-off. Replies flush_accepted.
reset
Drop the generation queue, the playout queue, and the retained build history; cut any playing clip;
and restore every session default (canvas, seed, clip length, autoplay, flush). Replies
session_reset, carrying how many clips were dropped and whether a clip was playing.
get_state
Replies with the full state_update snapshot.
Messages
Every message arrives as JSON{ "type": "<name>", "data": { … } }. The Reaches column says
whether a message is delivered to every connected client (everyone) or only to the client whose
command produced it (the caller).
A rejected command has no effect and is answered by a broadcast
command_error
({ command: string, reason: string }) naming the command and the reason — it reaches every
connected client, not just the caller.
queue_update payload
Carries everything a client needs to render the queue: generation (clips waiting to build, front
first), playout (built clips waiting to play, front first), and history (built clips no longer
queued — played, stopped, or popped — whose last frame is still retained, oldest first). history
entries are not playable again (play refuses them), but each is a legal continue_from_clip_id
source. The oldest history entries are evicted as new builds finish.
state_update payload
A full snapshot of everything observable except the queue’s contents, so a client can render UI from
this alone. Used by a scheduling client to steer; also carries every deployment-published live bound
a writer must not hardcode.
ClipInfo payload
Every message that references a clip embeds the full ClipInfo, so a client never has to join a
UUID against an earlier message.
Example handler (broadcasts only):
Complete example
Enqueue a base scene, then chain a continuation off it so the two play as one shot. Autoplay drives both; thecontinue_from_clip_id hand-off is seamless.
The continuation prompt re-describes the setting from scratch (“the same foothills… rain”) and
opens on a described hard cut rather than inheriting the take — continuity carries the frame,
not the description or the camera; the discipline is the whole prompt
guide.