main_video.
It has no audio track. When the session reaches ready, send commands with
reactor.sendCommand().
Tracks
There are no input tracks. The model gets all its input from commands and uploads.
The track sends nothing before the session’s first
play. After that, it always carries a picture:
the playing clip, or the last frame sent.
Session lifecycle
A session holds one world and at most one open run at a time:- The world is the place where the model makes clips.
create_worldmakes one from a description, reference images, or both.attach_worldmakes a saved one current. - A run is a sequence of clips that continue one another. The first
enqueueon a world starts a run. The run ends withend_run, afteridle_secondswith nothing to build, atmax_seconds(1,800 seconds), or when the session ends. The nextenqueuestarts a new run on the same world. The first clip of the new run does not continue the picture of the last run.
- The generation queue holds clips that are
queuedorgenerating. They build in order, and each one continues from the last frame of the one before. - The playout queue holds built clips that are
readyand wait forplay.
play sends the oldest ready clip, or a named one, on main_video. stop stops it. set_autoplay
plays ready clips on their own, oldest first. Built clips stay ready after their run ends, so you
can play them at any time.
session_state.phase says where the session is:
Several clients can connect to one session. Each client gets
session_state when it connects. All
clients see the same snapshot and the same tracks, and each one can send play or stop. A reply
goes only to the client that sent the command. The other messages go to every client.
Commands
The typed SDK,
@reactor-models/pixverse-r2-director, has one method for each command on
PixverseR2DirectorModel and on the usePixverseR2Director() hook. createWorld, attachWorld,
play, and stop resolve undefined. The other methods resolve their reply message. If the model
refuses the command, they resolve undefined. Parameter fields and message fields keep their
snake_case names. For each uploaded image, pass the FileRef that uploadFile returns.
create_world
Create the world where the model makes clips, and make it current. Valid in idle, world_ready,
ended, and failed. The phase moves to creating_world. After 4 to 12 seconds, it moves to
world_ready, with world_id and reference_images set. Clips that you queue in the meantime wait
for the world.
If the model cannot create the world, the previous world stays current. The model sets last_error
and fails the clips that waited for the world.
Upload each image with
uploadFile(). Then pass the FileRef that
it returns.
attach_world
Make a world from an earlier session current, without creating it again. Valid in idle,
world_ready, ended, and failed. The phase moves to world_ready at once. reference_images
is null, because the model does not know how many images the world has.
When the first clip of the next run starts, the model checks the id. If the id is unknown, that run
fails with last_error code world_not_found, and the model clears world_id.
enqueue
Queue one clip of the current world. Valid in every phase except idle and ending. The model
replies with clip_queued.
When the clip is ready, the model sends clip_generated. If the model cannot build the clip, it
sends clip_failed. A built clip waits for play. When autoplay is on, the clip plays on its own.
set_autoplay turns autoplay on.
From world_ready, ended, or failed, enqueue starts a new run. During creating_world, the
clip waits for the world.
Reference images
Each reference image of a world has an index from 0, in the order thatcreate_world received the
images. Image 0 is the opening image of the world. The first clip of every run opens from image 0,
and that clip takes no reference options. From the second clip of a run on, enqueue can tell the
model how a clip uses the images:
The reference options follow three rules:
- A nonempty
reference_indexescannot go with a frame reference. - Each index is below the
reference_imagescount of the world. - Each index appears once.
enqueue refuses it with invalid_reference. For an attached world,
the model does not know the image count. So an index out of range fails that clip instead.
Not every generation worker supports last_frame_ref. If the worker cannot build the clip, the clip
fails with unsupported_reference.
play
Play a ready clip on main_video. The command plays the clip that clip_id names. If clip_id
is blank, it plays the oldest ready clip. Valid in any phase, even after the run of the clip ends.
When the first frame goes out, the model sends clip_started. After the model sends the whole clip,
it sends clip_finished. If stop cuts the clip, the model sends clip_stopped instead.
stop
Stop playing the current clip, or the clip that play just asked for. The picture holds on the last
frame sent. The model sends clip_stopped, and the status of the clip becomes stopped.
set_autoplay
Choose whether ready clips play on their own, oldest first, when no clip plays. Off by default.
Valid in any phase. The model replies with autoplay_accepted and sends session_state. This
command never fails.
To play without gaps, wait until a few clips are ready, then turn autoplay on.
pop
Remove a clip that is queued or ready. Valid in any phase. The model replies with clip_popped
and sends session_state. The clip becomes cancelled. The model does not build a queued clip that
you remove. To remove the playing clip, send stop instead.
end_run
End the open run. Valid in any phase. The model replies with session_state. Clips that are
queued or generating become cancelled. Built clips stay ready, and you can still play them.
The next enqueue starts a new run on the same world.
get_state
Read the session state. Valid in any phase. The model replies with session_state. You rarely need
it, because the model sends session_state on connect and on every change.
Refusals
When the model refuses a command, nothing changes, and the call fails. Only the client that sent the command gets the error. The error carries a stablecode and a reason that you can show to your
user. The model never refuses set_autoplay, end_run, or get_state.
invalid_input: a blank prompt, a world with no description and no image, more than 6 images, or a malformedworld_id.invalid_image: an image that does not decode, or that is over 20 MiB or 40 megapixels.invalid_reference: reference options that the world or the clip cannot take. See Reference images.busy:create_worldorattach_worldwhile the model creates a world or while a run is open, orenqueuewhile a run closes.no_world:enqueuebefore a world is set.queue_full:enqueuewhenmax_pending_clipsclips are building, or when the ready and building clips together reachmax_ready_clips.already_playing:playwhile a clip plays. Sendstopfirst.no_ready_clip:playwhen no ready clip matches.not_playing:stopwhen nothing plays.clip_not_found:popwith an id that is not in this session.clip_playing:popof the playing clip. Usestop.clip_not_pending:popafter the clip isgeneratingor done, or at the moment that the model sends it for generation. The last case is retryable.
undefined, and the SDK emits an error event with
operation: "sendCommand" and the refusal’s code. A command that has a reply resolves undefined
only when the model refuses it. create_world, attach_world, play, and stop have no reply, so
they resolve undefined on success too. For these four commands, only the error event shows a
refusal. In Python, send_command() raises a ReactorError with the code.
Failures
A failure that no command caused does not refuse a call. Instead, it setslast_error on
session_state. For a clip, it sets error_code on the clip, and the clip arrives in
clip_failed. last_error has these fields:
A failed clip leaves the generation queue, and the clips after it still build.
Messages
clip payload
Every clip message carries the whole clip under clip, in the same shape as the entries of
session_state.clips:
cancelled means that pop removed the clip, or that its run ended before the model built it.
session_state payload
ended_by_client, idle, and max_duration come with phase ended. Every other end_reason
comes with phase failed and a last_error.
Step results
The model keeps every clip as one step of the session, under/sessions/{id}/steps. The step of a
built clip holds three files. output.mp4 is the clip as the track plays it, in H.264.
last_frame.png is its last frame. clip.json holds the clip_id, prompt, metadata,
seconds, and reference options of the clip.
A clip that fails is a failed step. So is a clip that pop or the end of its run cancels. The error
of the step says why: <code>: <reason> for a failed clip, cancelled: removed with pop, or
cancelled: its run ended (<end_reason>).