FastH3-Omni schema

This page lists FastH3-Omni’s messages, tracks, and commands. Wait for the session to be ready before sending commands.

Tracks

Track	Direction	Type
`main_video`	Output	Video, 24 fps
`main_audio`	Output	Audio

Session lifecycle

When you connect to FastH3-Omni, the session starts with empty generation and playout queues, autoplay off, and no clip playing. The initial state_update shows the session settings, queue capacities, and available commands. Check its valid_commands field to see which commands you can send in the current state. The initial queue_update shows the clips waiting to generate or play; both lists start empty. Send enqueue to add your first clip to the generation queue. When generation finishes, the model sends clip_generated and moves the clip to the playout queue. Call play to send it to main_video and main_audio, or turn on autoplay to play ready clips whenever the player is idle. You can generate another clip while one plays. Call reset to stop playback, clear both queues and any retained clips you could continue from, clear the output tracks, and restore the session defaults.

Commands

Command	Description	Reply or event
`enqueue`	Queue a clip from a prompt, with optional references or continuation.	`clip_queued`
`play`	Consume and play a ready clip.	`clip_started`
`pop`	Remove a clip from either queue.	`clip_popped`
`move`	Reposition a queued clip.	`clip_moved`
`stop`	Cut the playing clip.	`clip_stopped`
`get_queue`	Read generation and playout queues.	`queue_update`
`get_state`	Read defaults, capacities, progress, and valid commands.	`state_update`
`set_clip_seconds`	Change the default duration for later enqueues.	`clip_length_accepted`
`set_seed`	Change the advancing default seed.	`seed_accepted`
`set_autoplay`	Play ready clips when idle.	`autoplay_accepted`
`set_flush_on_clip_end`	Choose black or a held last frame at boundaries.	`flush_accepted`
`set_canvas`	Choose an output aspect while the session is empty.	`canvas_accepted`
`reset`	Clear queues and playback and restore defaults.	`session_reset`

`enqueue`

Queue a clip from a prompt, with optional references or continuation. Every clip needs a nonempty prompt. Image references guide characters, scenes, and appearance. Separate first_frame_image and last_frame_image inputs pin the opening and closing frames; each needs a reference image in the same request. Audio references guide the soundtrack and require an image reference or a retained clip to continue. The model keeps each request’s references with its clip. It replies with clip_queued when it accepts the request, then sends clip_generated or clip_failed after generation.

Parameter	Type	Default	Description
`prompt`	string	`""`	Required, nonempty text describing video and sound. Refused if it exceeds the model’s text budget; never truncated.
`metadata`	string	`""`	Client string, up to 2,000 characters, echoed in its `ClipInfo`.
`reference_image`	upload reference or null	`null`	One JPEG, PNG, or WebP image. Use this or `reference_images`.
`reference_images`	upload reference array or null	`null`	Up to nine ordered images; use no more than three for real-time generation. Name them `<Picture 1>`, `<Picture 2>`, and so on in the prompt.
`first_frame_image`	upload reference or null	`null`	Image pinned to the clip’s first frame. Needs at least one reference image; refused with `continue_from_clip_id`.
`last_frame_image`	upload reference or null	`null`	Image pinned to the clip’s last frame. Needs at least one reference image; with continuation, requires `continue_mode: "frame"`.
`reference_audio`	upload reference or null	`null`	One audio clip, 2 to 15 seconds long. Use this or `reference_audios`.
`reference_audios`	upload reference array or null	`null`	Up to three ordered audio clips, each 2 to 15 seconds long; at most two with a continuation.
`continue_from_clip_id`	string	`""`	UUID of an already generated clip retained in this session and at this canvas. An unknown or stale ID is refused.
`continue_mode`	`"latent"` or `"frame"`, or null	`null`	With `continue_from_clip_id`, null defaults to `latent`, carrying motion, camera, and audio. `frame` pins the source clip’s final frame as this clip’s first and uses its soundtrack as audio guidance. Refused without a continuation ID.
`seed`	nonnegative integer or null	`null`	Null uses and advances the session seed. An explicit seed leaves it unchanged.
`seconds`	number or null	`null`	One of 5.875, 8, 10.125, 12.25, or 14.375. Null uses the session default. Other values are refused, not rounded.
`position`	nonnegative integer or null	`null`	Position in the generation queue; 0 is next, and omitted or past the end appends. The running build is unaffected.

Do not send reference_image with reference_images, or reference_audio with reference_audios in the same request.

A continued clip’s soundtrack uses one of the three audio labels. Give every uploaded audio clip a role in the prompt, such as a speaker’s voice, background sound, or music; without that sentence the model has no instruction for how to use it. Keyframes are separate from ordered reference images and have no <Picture N> label. The model checks file types, counts, prompt budget, and queue capacity before accepting a request. It sends command_error if a check fails. Each image must be JPEG, PNG, or WebP, no larger than 25 MiB and no more than 25 million pixels, with an aspect ratio between 1:4 and 4:1. Audio can be WAV, MP3, AAC/M4A, OGG/Opus, FLAC, or WebM. Each file can be up to 25 MiB and 2 to 15 seconds long, in mono or stereo. The model checks the prompt and image references against the deployment’s text and presentation budgets. Handle command_error if either limit is exceeded; the model refuses an over-budget request rather than truncating it. With the base JavaScript SDK 3.1.0 or later, pass FileRef values directly, including inside reference_images and reference_audios arrays. The SDK serializes them for you. The Sandbox converts browser-readable images, including HEIC, before upload and keeps only the first 15 seconds of longer audio. Direct API callers must use an accepted format and keep audio between 2 and 15 seconds.

Keyframes and continuation

reference_image and reference_images guide appearance. To pin a frame, upload a separate image as first_frame_image or last_frame_image. Every request with a keyframe must also include at least one ordinary reference image. Keyframe images are not numbered <Picture N> references. For an independent clip, you can set either or both keyframes. With continue_from_clip_id, the default latent mode carries motion, camera, and audio and accepts no keyframes. Set continue_mode: "frame" to use the earlier clip’s final frame as the new first frame. Do not pass first_frame_image with a continuation. A last_frame_image with continuation requires frame mode and a reference image. After the SDK reaches ready, upload the files and enqueue an independent clip with a pinned start and end:

const scene = await reactor.uploadFile(sceneFile);
const opening = await reactor.uploadFile(firstFrameFile);
const ending = await reactor.uploadFile(lastFrameFile);

await reactor.sendCommand("enqueue", {
  prompt: "<Picture 1> is a ceramic artist in a sunlit studio. She lifts a blue bowl as the wheel hums. No music.",
  reference_image: scene,
  first_frame_image: opening,
  last_frame_image: ending,
  seconds: 8,
});

Wait for that clip’s clip_generated message and save message.data.clip.clip_id as generatedClipId. To begin the next clip on its final frame, pass that ID with frame mode and an ordinary reference image:

await reactor.sendCommand("enqueue", {
  prompt: "Continue with the same artist. She sets the bowl on a shelf; the wheel slows and the room stays quiet.",
  reference_image: scene,
  continue_from_clip_id: generatedClipId,
  continue_mode: "frame",
  seconds: 8,
});

clip_queued confirms acceptance, clip_generated reports readiness, and clip_failed reports a generation failure. Each message’s clip includes the keyframe and continuation fields listed below. Invalid combinations produce command_error with a reason.

`play`, `pop`, `move`, and `stop`

Command	Parameters	Behavior
`play`	`clip_id?: string`	Blank selects the front ready clip. Refuses if playback is active or no ready clip matches.
`pop`	`clip_id: string`	Removes a clip from either queue; an in-flight result is discarded. Use `stop` for the playing clip.
`move`	`clip_id: string`, `position: integer`	Moves within the clip’s current queue. Zero is front; past the end appends. Does not interrupt an active build.
`stop`	none	Cuts playback without clearing queues. Under autoplay, the next ready clip may start.

A clip cannot be replayed after it has played, been stopped, or been removed with pop.

State and defaults

Command	Parameters	Behavior
`get_queue`	none	Replies with both complete queues at any time.
`get_state`	none	Replies with a current state snapshot at any time.
`set_clip_seconds`	`seconds: 5.875 \| 8 \| 10.125 \| 12.25 \| 14.375`	Changes the default for future clips; queued clips keep their duration.
`set_seed`	`seed: nonnegative integer`	Changes the next automatic seed; explicit clip seeds do not advance it.
`set_autoplay`	`enabled: boolean`	Starts ready clips whenever idle. An empty queue waits; no prompts are generated automatically.
`set_flush_on_clip_end`	`enabled: boolean`	True flushes to black. False holds the last frame between clips and may leave silent gaps if generation falls behind.
`set_canvas`	`aspect: "16:9" \| "1:1" \| "9:16" \| "4:3"`	Requires empty queues and no playback. Dimensions are 1344 × 768, 768 × 768, 768 × 1344, or 1024 × 768.
`reset`	none	Clears queues, retained continuations and tracks, cuts playback, and restores defaults.

The defaults are 14.375 seconds, seed 1000, aspect ratio 16:9, autoplay off, and flush on. Read state_update for the current session values.

Messages

Message	When it arrives	Payload
`state_update`	Connect and state changes; reply to `get_state`.	Full state snapshot.
`queue_update`	Connect and queue changes; reply to `get_queue`.	`generation`, `playout`, `history`.
`clip_queued`	Correlated `enqueue` reply.	`clip`.
`clip_generated`	A build enters the ready queue.	`clip`.
`clip_moved`	Correlated `move` reply.	`clip`, `queue`, `position`.
`clip_started`	Playback starts.	`clip`.
`clip_finished`	Video and audio have been sent.	`clip`, `seconds_sent`.
`clip_stopped`	Playback is cut.	`clip`, `seconds_sent`.
`clip_popped`	Correlated `pop` reply.	`clip`.
`clip_failed`	A build fails and leaves the queue.	`clip`, `reason`.
`clip_length_accepted`	Correlated `set_clip_seconds` reply.	`clip_seconds`, `frames`.
`seed_accepted`	Correlated `set_seed` reply.	`seed`.
`autoplay_accepted`	Correlated `set_autoplay` reply.	`enabled`.
`flush_accepted`	Correlated `set_flush_on_clip_end` reply.	`enabled`.
`canvas_accepted`	Correlated `set_canvas` reply.	`aspect`, `width`, `height`.
`session_reset`	Correlated `reset` reply.	`cleared_clips`, `was_playing`.
`command_error`	A command is refused.	`command`, `reason`.

Command replies, including clip_queued, are correlated to the caller. The model broadcasts state, queue, and playback changes to all session clients. New clients receive state_update and queue_update when they connect.

`ClipInfo`

Messages about a clip include this ClipInfo record:

Field	Type	Meaning
`clip_id`	string	Assigned UUID.
`prompt`	string	Prompt as submitted.
`metadata`	string	Client string as submitted.
`frames`	integer	Output frame count.
`seconds`	number	Output duration.
`seed`	integer	Effective seed.
`ready`	boolean	True once built and ready to play.
`has_reference_image`	boolean	Whether an image reference was supplied.
`reference_image_count`	integer	Number of image references.
`has_reference_audio`	boolean	Whether uploaded audio was supplied.
`reference_audio_count`	integer	Number of uploaded audio references.
`has_first_frame`, `has_last_frame`	boolean	Whether this clip has a pinned opening or closing frame.
`continue_mode`	string or null	Continuation mode used for this clip, if any.
`prompt_prefix`	string	Text the model prepended to the prompt for frame continuation.
`build_seconds`	number or null	Build time excluding queue wait; null before build.

`state_update`

This message gives you the current session settings, playback status, queue counts, and available commands. It arrives when you connect or session state changes, and in reply to get_state. Use queue_update to see the clips in each queue.

Field	Meaning
`clip_seconds`	Default duration for clips enqueued without `seconds`.
`clip_seconds_choices`	Allowed durations: 5.875, 8, 10.125, 12.25, and 14.375 seconds.
`seed`	Seed that will be assigned to the next clip enqueued without an explicit seed.
`autoplay`	Whether a ready clip starts automatically when playback is idle.
`flush_on_clip_end`	Whether the video output goes black at a clip boundary. If false, it holds the last frame.
`aspect`, `width`, `height`	Current output aspect ratio and video dimensions in pixels.
`playing`, `playing_clip_id`	Whether a clip is starting or playing, and its UUID. The ID is null when idle.
`generation_queued`, `generation_capacity`	Number of clips waiting or generating, and the maximum generation queue size.
`playout_queued`, `playout_capacity`	Number of ready clips waiting to play, and the maximum playout queue size. Generation pauses when the playout queue is full.
`clips_played`, `seconds_sent`	Clips finished or stopped, and total seconds sent on the output tracks in this session.
`valid_commands`	Commands available in the current queue and playback state.

`queue_update`

This message lists the clips waiting to generate or play. It arrives when you connect or either queue changes, and in reply to get_queue. Each listed clip is a ClipInfo record.

Field	Meaning
`generation`	Clips waiting or generating, front first.
`playout`	Generated clips ready to play, front first.
`history`	Always an empty array. This field is not a playback log.

With autoplay off, a new clip appears in generation after enqueue. When clip_generated arrives, it moves to playout. Calling play removes it from playout; its ID then appears in state_update.playing_clip_id. Save a generated clip’s clip_id if you want to continue from it while the session retains it.

Complete example

This base SDK example queues a clip from a prompt and plays it when ready. See the tutorial for uploads and continuation.

import { Reactor } from "@reactor-team/js-sdk";

const reactor = new Reactor({ modelName: "reactor/fast-h3-omni" });

reactor.on("message", async (message) => {
  if (message.type === "clip_generated") {
    await reactor.sendCommand("play", { clip_id: message.data.clip.clip_id });
  }
});

reactor.on("statusChanged", async (status) => {
  if (status !== "ready") return;
  await reactor.sendCommand("enqueue", {
    prompt: "A blue bowl turns on a pottery wheel. The camera moves toward the artist's hands. The wheel hums and rain taps the window.",
    seconds: 5.875,
  });
});

await reactor.connect(await getToken()); // Your server issues the token.

Overview

FastH3-Omni

Vidu S2-Avatar

Vidu S2-Editing

H3 Reference Turbo Realtime

FastH3

Visko Orbis Stable

Visko Orbis Dynamic

HappyOyster

LingBot World 2

LingBot

X2

Helios

LongLive-2.0

SANA-Streaming

LTX

FastH3-Omni schema

Tracks

Session lifecycle

Commands

`enqueue`

Keyframes and continuation

`play`, `pop`, `move`, and `stop`

State and defaults

Messages

`ClipInfo`

`state_update`

`queue_update`

Complete example

​Tracks

​Session lifecycle

​Commands

​enqueue

​Keyframes and continuation

​play, pop, move, and stop

​State and defaults

​Messages

​ClipInfo

​state_update

​queue_update

​Complete example

Tracks

Session lifecycle

Commands

`enqueue`

Keyframes and continuation

`play`, `pop`, `move`, and `stop`

State and defaults

Messages

`ClipInfo`

`state_update`

`queue_update`

Complete example