Vidu S2-Avatar schema

Use these commands to create or reuse an avatar, start a call, and change the character during the conversation.

Tracks

Track	Direction	Type	Description
`mic`	Input	Audio	Caller speech during a live call.
`webcam`	Input	Video	Caller camera video during a video call.
`main_video`	Output	Video	Generated character video.
`main_audio`	Output	Audio	Generated character speech.

All four tracks are available when the session connects, so you can set up a player before a call. Character video and speech begin when the call is live. See Tracks for how to publish mic and webcam.

Session lifecycle

A session has one current avatar. You can start, end, and restart calls with it without reconnecting. Ending a call leaves the session open. Reactor sends a session_state message when you connect and whenever the avatar or call changes. Its phase field reports the current step:

Phase	Meaning
`idle`	No avatar has been created or attached in this session.
`preparing_avatar`	The model is creating an avatar from your image.
`avatar_ready`	The avatar is ready for a call.
`starting`	The call is being created.
`warming_up`	The call is getting ready.
`live`	The call is active. You can speak, type, or steer it.
`ending`	The call is closing.
`ended`	The call is over. You can start another with this avatar.
`failed`	The call failed. Check `last_error` before trying again.

If avatar creation fails, the phase returns to idle. If a call fails, the phase becomes failed. Two limits can end a call without end_call. call_max_seconds is the maximum call length; its timer begins when the call starts. Reaching it sets end_reason to max_duration. Two hours without input sets end_reason to idle_timeout.

Commands

Send commands with reactor.sendCommand() in JavaScript or reactor.send_command() in Python. A command with a named reply returns that message. Other commands return no value; watch session_state for their result. A refused command sends command_error without changing the session. See Command errors for the codes.

Command	Description	Valid in	Reply
`create_avatar`	Create an avatar from one image.	No call active, no avatar preparing	None; see `session_state`
`attach_avatar`	Use an avatar saved from an earlier session.	No call active	None; see `session_state`
`list_voices`	List the voices a call can use.	Any time	`voices`
`start_call`	Start a call with the current avatar.	`avatar_ready`, `ended`, `failed`	None; see `session_state`
`say`	Send text for the character to answer.	`live`	None
`interrupt`	Stop the character mid-sentence.	`live`	None
`update_call`	Change voice, persona, turn-taking, or reply settings.	`live`	`call_updated`
`set_reference_images`	Give the character an object, an outfit, or a background.	`live`	`reference_images_applied`
`clear_reference_images`	Remove active reference images.	`live`	`reference_images_applied`
`end_call`	End the call and keep the avatar.	`starting` through `live`	`call_ended`
`get_state`	Get the current session state.	Any time	`session_state`

Omit an optional field you do not set. Do not send it as null. A command that carries an explicit null is dropped whole, without a command_error, so send { persona }, not { persona, voice: null }.

`create_avatar`

Create an avatar from one image of a person. The new avatar becomes the current avatar for this session. Give exactly one of image or image_url.

Parameter	Type	Default	Description
`image`	Upload reference or null	`null`	The character’s image, uploaded with the SDK’s file upload. Use this or `image_url`.
`image_url`	string or null	`null`	A publicly fetchable URL of the image, instead of an upload.
`name`	string or null	`null`	Display name, up to 100 characters. Reported back as `session_state.avatar_name`.

An uploaded image can be PNG, JPG, WebP, or HEIC, under 20 MB and at most 50 megapixels. The model applies the image’s EXIF orientation, so a phone photo arrives upright. While the image is processed, session_state.phase is preparing_avatar. When the avatar is ready, the phase becomes avatar_ready and session_state includes its avatar_id. Save the ID to use the avatar in another session.

const image = await reactor.uploadFile(file);
await reactor.sendCommand("create_avatar", { image, name: "Tina" });

ref = await reactor.upload_file("tina.png")
await reactor.send_command("create_avatar", {"image": ref, "name": "Tina"})

`attach_avatar`

Use an avatar you created earlier by passing its avatar_id. Avatars are kept for 90 days.

Parameter	Type	Description
`avatar_id`	string	Required. An `avatar_id` from an earlier `session_state`.

The model checks the ID before using the avatar. On success, session_state.phase becomes avatar_ready.

`list_voices`

Send {} at any time. The voices reply lists built-in voices and names the default voice. Each system entry has voice, description, and accent. Pass an entry’s voice to start_call or update_call.

const reply = await reactor.sendCommand("list_voices", {});
const voice = reply?.data.system[0].voice;

reply = await reactor.send_command("list_voices", {})
voice = reply["data"]["system"][0]["voice"]

`start_call`

Start a live call with the current avatar. Create or attach an avatar first, then wait for session_state.phase to become avatar_ready. Only persona is required.

string

required

Who the character is and how it behaves, up to 50,000 characters.

string

A voice from list_voices. Omit it to use the model’s default voice.

string

What the character says or does first, before you speak, up to 200 characters.

string

default:"English"

Language the character speaks and replies in, up to 40 characters.

string

default:"audio"

audio forwards your mic track. video also forwards your webcam track. This setting is fixed for the call.

boolean

default:"true"

Send a transcript message for what the caller and character say.

boolean

default:"false"

Expand a short persona into a fuller description before the call starts. Use this when you have only a brief character description.

object

Turn-taking settings. You can change them later with update_call.

object

Reply generation settings. You can change them later with update_call.

call_mode controls whether your camera reaches the character. The character’s generated video still arrives on main_video in both modes.

Turn-taking: `vad`

VAD means voice activity detection. These settings control when caller speech starts or ends a turn. The default server mode filters brief acknowledgments and background noise. Use semantic when the caller should be able to interrupt the character as soon as they begin speaking.

string

default:"server"

server filters back-channel sounds and background noise. semantic lets the caller interrupt the character as soon as they start speaking.

number

default:"0.5"

With server, controls how much noise is filtered, from 0 to 1. Raise it in a noisy room; lower it if quiet speech is missed.

integer

default:"400"

With server, how long the caller must be silent before the character answers, from 200 to 6000 milliseconds. Raise it to allow longer pauses; lower it for faster replies.

See the turn-taking examples for common settings.

Reply generation: `llm`

These settings shape each reply. A token is a piece of text. You can change these settings during a call with update_call; the new values apply on the next turn.

integer

default:"50"

The maximum length of one reply in tokens. Raise it for longer answers; lower it for brief exchanges.

number

Controls variation, from 0 up to but not including 2. Lower values make replies more consistent; higher values make them more varied.

number

Nucleus sampling, greater than 0 and at most 1. Lower values narrow the next-token choices, making replies less varied.

integer

Top-k sampling, from 0 to 100. A lower positive value limits how many next-token choices the model considers.

number

Discourages repeated words and phrases. Raise it when replies repeat themselves; the value must be greater than 0.

number

Encourages new topics instead of returning to ones already covered. Raise it for more topic variety, from 0 to 2.

integer

Controls randomness. Use a fixed value to make a demo more repeatable; -1 picks a random seed.

The phase moves through starting and warming_up to live. Once live, main_video and main_audio carry the character, and your mic reaches it. If the call fails, the phase becomes failed; inspect command_error or session_state.last_error for the reason.

await reactor.sendCommand("start_call", {
  persona: "You are Tina, a warm museum guide. Answer in two short sentences.",
  greeting: "Say hello and wave.",
  call_mode: "audio",
  vad: { type: "server", silence_duration_ms: 800 },
  llm: { max_tokens: 80 },
});

await reactor.send_command("start_call", {
    "persona": "You are Tina, a warm museum guide. Answer in two short sentences.",
    "greeting": "Say hello and wave.",
    "call_mode": "audio",
    "vad": {"type": "server", "silence_duration_ms": 800},
    "llm": {"max_tokens": 80},
})

`say`

Send text for the character to answer, as if you had spoken it.

Parameter	Type	Description
`text`	string	Required. What to tell the character, 1 to 2,000 characters.

There is no reply. The answer arrives on main_video and main_audio, and as a transcript when transcripts are on.

await reactor.sendCommand("say", { text: "Tell me about the sea." });

await reactor.send_command("say", {"text": "Tell me about the sea."})

`interrupt`

Send {} to stop the character mid-sentence, as speaking over it would. There is no reply. To stop it and give a new instruction, send interrupt followed by say; interrupt takes no text.

await reactor.sendCommand("interrupt", {});
await reactor.sendCommand("say", {
  text: "Let's talk about the sea instead.",
});

await reactor.send_command("interrupt", {})
await reactor.send_command("say", {"text": "Let's talk about the sea instead."})

`update_call`

Change the live call without restarting it. Give at least one field.

Parameter	Type	Description
`voice`	string	A `voice` from `list_voices`. Takes effect after the current sentence.
`persona`	string	A new persona, up to 50,000 characters. Replaces the current one after the current sentence.
`vad`	object	Turn-taking settings, as in `start_call`. Applied on the next turn.
`llm`	object	Reply settings, as in `start_call`. Applied on the next turn.

The model applies all supplied fields in one update. If it rejects any field, none of the changes take effect. The call_updated reply lists the settings that changed.

const reply = await reactor.sendCommand("update_call", { voice: "Ethan" });
console.log(reply?.data.applied); // ["voice"]

reply = await reactor.send_command("update_call", {"voice": "Ethan"})
print(reply["data"]["applied"])  # ["voice"]

`set_reference_images`

Give the character one to three reference images to pick up live: an object to hold, an outfit to wear, or a background.

Parameter	Type	Description
`images`	array	Required. One to three reference image entries.

Each entry has these fields:

string

required

A publicly fetchable image. Reference images are URL-only; uploads are not accepted here.

string

required

Your own stable id for the image, 1 to 128 characters, unique in the call. Sending an id that is already in effect replaces that image.

string

object, garment, or background. Inferred when omitted.

string

One sentence, up to 200 characters, that describes the change.

The change does not interrupt speech. The reference_images_applied reply lists every image ID now in effect.

const reply = await reactor.sendCommand("set_reference_images", {
  images: [
    {
      image_url: "https://example.com/jacket.png",
      image_id: "jacket-1",
      kind: "garment",
      text: "She puts on the denim jacket.",
    },
  ],
});

reply = await reactor.send_command("set_reference_images", {
    "images": [{
        "image_url": "https://example.com/jacket.png",
        "image_id": "jacket-1",
        "kind": "garment",
        "text": "She puts on the denim jacket.",
    }],
})

`clear_reference_images`

Remove reference images from the live call. Pass the image_id values you supplied with set_reference_images. Omit image_ids to undo the most recent set_reference_images command.

Parameter	Type	Default	Description
`image_ids`	string array or null	`null`	Up to three `image_id` values to clear. Omit to undo the most recent `set_reference_images`.

The reference_images_applied reply lists the IDs still in effect. This example removes the jacket from the preceding set_reference_images example:

const reply = await reactor.sendCommand("clear_reference_images", {
  image_ids: ["jacket-1"],
});
console.log(reply?.data.image_ids);

reply = await reactor.send_command("clear_reference_images", {
    "image_ids": ["jacket-1"],
})
print(reply["data"]["image_ids"])

`end_call`

Send {} to end the call. The call_ended reply arrives once the call is released. Its duration_seconds is the time the call was live, or 0 if it never became live. The phase moves through ending to ended. The avatar remains available, so start_call works again.

const ended = await reactor.sendCommand("end_call", {});
console.log(ended?.data.end_reason);

ended = await reactor.send_command("end_call", {})
print(ended["data"]["end_reason"])

`get_state`

Send {} at any time to get the current session_state. Reactor also broadcasts this message when the avatar or call changes.

const state = await reactor.sendCommand("get_state", {});
console.log(state?.data.phase, state?.data.avatar_id);

state = await reactor.send_command("get_state", {})
print(state["data"]["phase"], state["data"]["avatar_id"])

Messages

Every message is broadcast to all connected clients, except command replies, which go only to the client that sent the command.

Message	When	Delivery	Payload
`session_state`	On connect, on every change, and in reply to `get_state`.	Event or command reply	See `session_state`.
`transcript`	You or the character finished a sentence, with transcripts on.	Event	`speaker`, `text`, `final`
`voices`	Reply to `list_voices`.	Command reply	`system`, `default_voice`
`call_updated`	Reply to `update_call`, once the call took the new settings.	Command reply	`applied`
`reference_images_applied`	Reply to `set_reference_images` and `clear_reference_images`.	Command reply	`image_ids`
`call_ended`	Reply to `end_call`, once the call is released.	Command reply	`end_reason`, `duration_seconds`
`command_error`	A command was refused, or a call failed.	Error	See `command_error`.

For transcript, speaker is user or character, and final is true when the text is settled. Listen for messages before connecting so you receive the first session_state:

reactor.on("message", (message) => {
  switch (message.type) {
    case "session_state":
      console.log("Phase:", message.data.phase);
      break;
    case "transcript":
      if (message.data.final) {
        console.log(`${message.data.speaker}: ${message.data.text}`);
      }
      break;
    case "command_error":
      console.error(message.data.code, message.data.reason);
      break;
  }
});

def on_message(message):
    data = message["data"]
    if message["type"] == "session_state":
        print("Phase:", data["phase"])
    elif message["type"] == "transcript" and data["final"]:
        print(f'{data["speaker"]}: {data["text"]}')
    elif message["type"] == "command_error":
        print(data["code"], data["reason"])

reactor.on("message", on_message)

`session_state`

The authoritative snapshot. Drive your UI from this message alone. A client that reconnects mid session gets a full snapshot on connect. Only phase is always present; the other fields are null or false when they do not apply.

Field	Type	Description
`phase`	string	The session phase. See Session lifecycle.
`avatar_id`	string or null	The current avatar’s id. Save it to reuse the avatar with `attach_avatar`.
`avatar_status`	string or null	`processing`, `ready`, or `failed` for the current avatar.
`avatar_name`	string or null	The avatar’s display name.
`voice`	string or null	The voice in effect for the current or next call.
`persona_set`	boolean	A persona has been given for the current call.
`call_mode`	string or null	`audio` or `video` for the current call.
`warmup_attempts`	integer	Readiness checks made while `warming_up`.
`call_started_at`	number or null	Unix time, in seconds, at which the call became `live`.
`call_max_seconds`	integer or null	The longest the call may run before it ends on its own.
`call_elapsed_seconds`	integer or null	Seconds since the call became `live`.
`control_ready`	boolean	Commands such as `say` and `interrupt` are accepted.
`video_receiving`	boolean	Character video is arriving on `main_video`.
`audio_receiving`	boolean	Character speech is arriving on `main_audio`.
`mic_forwarding`	boolean	Your `mic` track has reached the character in this call.
`camera_forwarding`	boolean	Your `webcam` track has reached the character. Video mode only.
`last_frame_age_ms`	integer or null	Milliseconds since the last character video frame. Grows when video stalls.
`reference_images`	array or null	Reference images in effect, `[{ image_id, kind }]`. Null or empty when there are none.
`end_reason`	string or null	Why the last call ended. Null while a call runs. See below.
`last_error`	object or null	The most recent `command_error` payload. Cleared when a call starts.

end_reason is one of:

Value	Meaning
`ended_by_client`	A client sent `end_call`.
`max_duration`	The call reached `call_max_seconds`.
`idle_timeout`	The call had no input for two hours.
`content_policy`	The call was stopped for its content.
`upstream_quota`	The account’s usage quota ran out. Not the user’s fault.
`upstream_interrupted`	The call was interrupted. Start a new call.
`media_lost`	Character media stopped arriving. Start a new call.
`bridge_failed`	The call failed on Reactor’s side. Start a new call.

Any value other than ended_by_client also leaves a last_error that explains it.

`command_error`

Emitted when a command is refused or a call fails. A refused command does not perform the requested action; a call failure moves the phase to failed. The payload is also stored as session_state.last_error.

Field	Type	Description
`command`	string	The command that failed, or `session` when nothing the client sent caused it.
`origin`	string	Whose fault it is. See below.
`code`	string	A stable code. Branch on this.
`reason`	string	A readable sentence.
`retryable`	boolean	The same command may succeed if you send it again later. Defaults to `false`.
`upstream_status`	integer or null	HTTP status from the generation service, when `origin` is `upstream`.
`upstream_code`	string or null	Error code from the generation service, when it gave one.
`trace_id`	string or null	Quote this when you report the failure.

origin is one of request (your input was invalid), state (the command is not allowed in this phase), upstream (the generation service refused or failed), or platform (a failure on Reactor’s side).

Code	Meaning
`INVALID_INPUT`	A parameter is missing, malformed, or out of range.
`BUSY`	A call is active, or an avatar is being prepared.
`NO_AVATAR`	`start_call` was sent with no avatar ready.
`NOT_LIVE`	The command needs a live call.
`AVATAR_FAILED`, `AVATAR_TIMEOUT`	The image could not become an avatar.
`AVATAR_NOT_FOUND`	The `avatar_id` is unknown or has expired.
`WARMUP_TIMEOUT`	The call did not become live in time.
`UPSTREAM_CAPACITY`	No capacity is free right now. Retryable.
`UPSTREAM_QUOTA`	The account’s usage quota ran out.
`CONTENT_POLICY`	The request or the call was stopped for its content.
`UPSTREAM_REJECTED`, `UPSTREAM_AUTH`, `UPSTREAM_UNAVAILABLE`, `UPSTREAM_DISCONNECTED`	The generation service refused or failed.
`MEDIA_JOIN_FAILED`, `MEDIA_LOST`, `BRIDGE_CRASHED`	Media could not start or stopped. Start a new call.

The generation service can also return its own codes, which pass through unchanged. Examples are LIVE_CONN_INIT_FAILED and codes that start with PROMPT_OP_.

Upload reference

create_avatar.image takes a reference returned by the Reactor upload protocol. All fields are required.

Field	Type
`upload_id`	UUID string
`name`	string
`mime_type`	string
`size`	integer

Multiple clients

Several clients can join the same session. session_state and command_error go to all of them; a command reply goes only to the client that sent the command. A session has one call, so every client steers the same character. A client that disconnects does not end the call. Every new session starts in idle, without an avatar.

Overview

Vidu S2-Avatar

H3 Reference Turbo Realtime

FastH3

Visko Orbis Stable

Visko Orbis Dynamic

HappyOyster

LingBot World 2

LingBot

X2

Helios

LongLive-2.0

SANA-Streaming

LTX

Vidu S2-Avatar schema

Tracks

Session lifecycle

Commands

`create_avatar`

`attach_avatar`

`list_voices`

`start_call`

Turn-taking: `vad`

Reply generation: `llm`

`say`

`interrupt`

`update_call`

`set_reference_images`

`clear_reference_images`

`end_call`

`get_state`

Messages

`session_state`

`command_error`

Upload reference

Multiple clients

​Tracks

​Session lifecycle

​Commands

​create_avatar

​attach_avatar

​list_voices

​start_call

​Turn-taking: vad

​Reply generation: llm

​say

​interrupt

​update_call

​set_reference_images

​clear_reference_images

​end_call

​get_state

​Messages

​session_state

​command_error

​Upload reference

​Multiple clients

Tracks

Session lifecycle

Commands

`create_avatar`

`attach_avatar`

`list_voices`

`start_call`

Turn-taking: `vad`

Reply generation: `llm`

`say`

`interrupt`

`update_call`

`set_reference_images`

`clear_reference_images`

`end_call`

`get_state`

Messages

`session_state`

`command_error`

Upload reference

Multiple clients