Vidu S2-Editing schema

Use these commands to start editing your camera, change the reference image or edit type, and end the edit.

Tracks

Track	Direction	Type	Description
`camera`	Input	Video	Your camera, the whole input. Reaches the model while an edit is `live`.
`main_video`	Output	Video	The edited video. A neutral dark frame appears before and between edits.

There is no audio in either direction. Both tracks exist from connect, so you can attach a player and publish your camera before start_edit. See Tracks for how to publish camera from each SDK.

Keep your camera track flowing. The model never repeats a frame to fill a gap, so a stalled camera means a stalled edit.

Session lifecycle

One session holds at most one active edit. An edit that ends does not end the session, and start_edit works again. session_state.phase is one of these values:

Phase	Meaning	Next
`idle`	No edit has run yet. Every new session starts here.	`start_edit`
`starting`	The edit is being created.	`warming_up`, or `failed`
`warming_up`	The edit is getting ready.	`live`, or `failed`
`live`	The edit is running. `switch_reference` is accepted.	`ending`, or `failed`
`ending`	The edit is being released.	`ended`
`ended`	The edit is over. `end_reason` says why.	`start_edit` again
`failed`	The edit could not start, or broke. `last_error` says why.	`start_edit` again

The first edited frame can arrive after the phase becomes live. Draw your UI from camera_forwarding and video_receiving, not from phase: "live" alone. Each edit has a time limit, edit_max_seconds, reported on the snapshot once the edit starts. Reaching it ends the edit with end_reason: "max_duration".

Commands

Commands report results in two ways. sendCommand() returns a direct reply when the command has one. The model also sends session_state updates and command_error messages to connected clients. start_edit has no direct reply, so watch session_state to see when the edit becomes live. If a command is refused, it has no effect and sends command_error instead of a direct reply.

Command	Description	Valid in	Reply
`start_edit`	Start editing your camera with a reference image and an edit type.	`idle`, `ended`, `failed`	None. See `session_state`.
`switch_reference`	Change the reference image, the edit type, or both.	`live`	`reference_switched`
`end_edit`	End the edit and keep the session.	`starting` through `live`	`edit_ended`
`get_state`	Read the current snapshot.	Any time	`session_state`

Omit an optional field you do not set. Do not send it as null. A command that carries an explicit null is dropped whole, without a command_error, so send { editing_type: "virtual_tryon" }, not { editing_type: "virtual_tryon", reference_image_url: null }.

Edit types

Choose editing_type based on what you want the image to change:

`editing_type`	Effect
`style_transfer`	The look of the whole scene. Default.
`virtual_tryon`	A garment the person on camera wears.
`subject_replacement`	A character that replaces the person.
`background_replacement`	The scene behind the person.

`start_edit`

Start editing your camera track with one reference image and an edit type. Give exactly one of reference_image or reference_image_url.

Parameter	Type	Default	Description
`reference_image`	Upload reference or null	`null`	The reference image, uploaded with the SDK’s file upload. PNG, JPG, or WebP, under 10 MB.
`reference_image_url`	string or null	`null`	A public http(s) URL of the image, up to 2,048 characters. The model fetches it when the edit starts.
`editing_type`	string or null	`"style_transfer"`	The edit type. See Edit types. Change it later with `switch_reference`.

Rotate phone photos before uploading them. The model uses an uploaded image as is and does not apply its EXIF orientation. start_edit begins setup. It does not wait for the camera or the first edited frame. Publish camera and watch session_state move through starting, warming_up, and live. If an edit is already active, the model sends BUSY. If the image is missing or invalid, or you send both image fields, it sends INVALID_INPUT. If setup fails after the command is accepted, phase becomes failed and last_error explains why. See command_error for error codes.

const reference_image = await reactor.uploadFile(file);
await reactor.sendCommand("start_edit", {
  reference_image,
  editing_type: "background_replacement",
});

ref = await reactor.upload_file("beach.jpg")
await reactor.send_command("start_edit", {
    "reference_image": ref,
    "editing_type": "background_replacement",
})

`switch_reference`

Change the reference image, the edit type, or both, without restarting the edit. What you omit stays as it is. Give at least one field.

Parameter	Type	Default	Description
`reference_image`	Upload reference or null	`null`	A new reference image, with the same checks as in `start_edit`. Omit to keep the current image.
`reference_image_url`	string or null	`null`	A public http(s) URL of a new image, instead of `reference_image`.
`editing_type`	string or null	`null`	A new edit type. Omit to keep the current one.

The reference_switched reply confirms that the model sent the switch. It does not mean the new look is visible yet. Its editing_type echoes the requested edit type, and reference_changed is true if you supplied a new image. If the switch fails later, you receive command_error for switch_reference. The previous image and edit type stay in effect, and the edit keeps running. session_state.editing_type returns to its previous value. The error code explains the failure. See command_error.

Each call switches exactly once. Do not resend a switch just because the new look has not appeared yet.

Outside a live edit, the command returns NOT_LIVE. If you omit every field or send an invalid image, it returns INVALID_INPUT. The image checks are the same as in start_edit.

const reply = await reactor.sendCommand("switch_reference", {
  reference_image_url: "https://example.com/jacket.png",
  editing_type: "virtual_tryon",
});
console.log(reply?.data); // { editing_type: "virtual_tryon", reference_changed: true }

// Edit type only: the image stays.
await reactor.sendCommand("switch_reference", { editing_type: "style_transfer" });

reply = await reactor.send_command("switch_reference", {
    "reference_image_url": "https://example.com/jacket.png",
    "editing_type": "virtual_tryon",
})
print(reply["data"])  # {"editing_type": "virtual_tryon", "reference_changed": True}

`end_edit`

Send {} to end the edit. The phase moves through ending to ended. The command replies edit_ended once the edit is released. You can then call start_edit again. If no edit is active, the command returns NOT_LIVE. The reply’s end_reason matches the value in session_state. duration_seconds is the time the edit was live, or 0 if it never became live.

const ended = await reactor.sendCommand("end_edit", {});
console.log(ended?.data.end_reason, ended?.data.duration_seconds);

ended = await reactor.send_command("end_edit", {})
print(ended["data"]["end_reason"], ended["data"]["duration_seconds"])

`get_state`

Send {} at any time. Replies session_state, the same snapshot the model broadcasts on every change. It never fails.

const state = await reactor.sendCommand("get_state", {});
console.log(state?.data.phase, state?.data.editing_type);

state = await reactor.send_command("get_state", {})
print(state["data"]["phase"], state["data"]["editing_type"])

Messages

Every message is broadcast to all connected clients, except command replies, which go only to the client that sent the command.

Message	When	Delivery	Payload
`session_state`	On connect, on every change, and in reply to `get_state`.	Event or command reply	See `session_state`.
`reference_switched`	Reply to `switch_reference`, once the switch is sent.	Command reply	`editing_type`, `reference_changed`
`edit_ended`	Reply to `end_edit`, once the edit is released.	Command reply	`end_reason`, `duration_seconds`
`command_error`	A command was refused, a switch failed, or the edit failed.	Error	See `command_error`.

Register your message listener before connecting so it receives the first session_state. JavaScript uses sendCommand() and Python uses send_command() for the same operation. When a command has a direct reply, both methods return it. Handle state updates and errors with a message listener:

reactor.on("message", (message) => {
  if (message.type === "session_state") {
    console.log("Phase:", message.data.phase);
  } else if (message.type === "command_error") {
    console.error(message.data.code, message.data.reason);
  }
});

def on_message(message):
    data = message["data"]
    if message["type"] == "session_state":
        print("Phase:", data["phase"])
    elif message["type"] == "command_error":
        print(data["code"], data["reason"])

reactor.on("message", on_message)

`session_state`

This is the authoritative snapshot. Drive your UI from it. A client that reconnects during an edit gets the full snapshot on connect. phase is always present. Other fields are null or false when they do not apply.

Field	Type	Description
`phase`	string	The session phase. See Session lifecycle.
`editing_type`	string or null	The edit type in effect, or of the last edit. Null before the first `start_edit`.
`reference_set`	boolean	A reference image is in effect for the current edit.
`edit_started_at`	number or null	Unix time, in seconds, at which the edit became `live`.
`edit_max_seconds`	integer or null	The longest the edit may run before it ends on its own.
`edit_elapsed_seconds`	integer or null	Seconds since the edit became `live`. Null outside an active edit.
`control_ready`	boolean	`switch_reference` is accepted.
`camera_forwarding`	boolean	Your `camera` reached the model in the last 2 seconds. Nothing is edited until it does.
`video_receiving`	boolean	Edited video arrived on `main_video` in the last second.
`last_frame_age_ms`	integer or null	Milliseconds since the last edited frame. Grows when video stalls.
`end_reason`	string or null	Why the last edit ended. Null while an edit runs. See below.
`last_error`	object or null	The most recent `command_error` payload. Cleared when `start_edit` is accepted.

end_reason is one of:

Value	Meaning
`ended_by_client`	A client sent `end_edit`.
`max_duration`	The edit reached `edit_max_seconds`.
`content_policy`	The edit was stopped for its content.
`upstream_quota`	The account’s usage quota ran out. Not the user’s fault.
`upstream_interrupted`	The edit was interrupted. Start a new edit.
`media_lost`	Edited video stopped arriving. Start a new edit.
`bridge_failed`	The edit failed on Reactor’s side. Start a new edit.

content_policy, upstream_quota, media_lost, and a failed start also leave a last_error that explains the reason.

`command_error`

Emitted when a command is refused, a switch fails, or the edit fails. A refused command has no effect. If a switch fails later, the previous look stays in effect. The payload is broadcast and stored as session_state.last_error.

Field	Type	Description
`command`	string	The command that failed, or `session` when nothing the client sent caused it.
`origin`	string	Where the error came from. See below.
`code`	string	A stable code. Branch on this.
`reason`	string	A readable sentence.
`retryable`	boolean	The same command may succeed if you send it again later. Defaults to `false`.
`upstream_status`	integer or null	HTTP status from the generation service, when `origin` is `upstream`.
`upstream_code`	string or null	Error code from the generation service, when it gave one.
`trace_id`	string or null	Quote this when you report the failure.

request means the input was invalid. state means this phase does not allow the command. upstream means the generation service refused or failed. platform means Reactor failed.

Code	Meaning
`INVALID_INPUT`	A parameter is missing, malformed, or out of range.
`BUSY`	An edit is already active.
`NOT_LIVE`	The command needs a live edit.
`WARMUP_TIMEOUT`	The edit did not become live in time.
`UPSTREAM_REJECTED`	The request was refused, for example an image that cannot be used.
`UPSTREAM_CAPACITY`	No capacity is free right now. Retryable.
`UPSTREAM_QUOTA`	The account’s usage quota ran out.
`CONTENT_POLICY`	The edit was stopped for its content.
`UPSTREAM_AUTH`, `UPSTREAM_UNAVAILABLE`, `UPSTREAM_DISCONNECTED`	The generation service refused or failed.
`MEDIA_JOIN_FAILED`, `MEDIA_LOST`, `BRIDGE_CRASHED`	Media could not start or stopped. Start a new edit.

The generation service can also return codes not listed here. These pass through unchanged. Log code and trace_id when reporting an unknown error.

Upload reference

reference_image takes a reference returned by the Reactor upload protocol. All fields are required.

Field	Type
`upload_id`	UUID string
`name`	string
`mime_type`	string
`size`	integer

Multiple clients

Several clients can join the same session. session_state and command_error go to all of them. A command reply goes only to the client that sent the command. A session has one edit, and it edits the session’s camera track. A client that disconnects does not end the edit. Every new session starts in idle.

Overview

Vidu S2-Avatar

Vidu S2-Editing

H3 Reference Turbo Realtime

FastH3

Visko Orbis Stable

Visko Orbis Dynamic

HappyOyster

LingBot World 2

LingBot

X2

Helios

LongLive-2.0

SANA-Streaming

LTX

Vidu S2-Editing schema

Tracks

Session lifecycle

Commands

Edit types

`start_edit`

`switch_reference`

`end_edit`

`get_state`

Messages

`session_state`

`command_error`

Upload reference

Multiple clients

​Tracks

​Session lifecycle

​Commands

​Edit types

​start_edit

​switch_reference

​end_edit

​get_state

​Messages

​session_state

​command_error

​Upload reference

​Multiple clients

Tracks

Session lifecycle

Commands

Edit types

`start_edit`

`switch_reference`

`end_edit`

`get_state`

Messages

`session_state`

`command_error`

Upload reference

Multiple clients