Vidu S2-Avatar tutorial

This tutorial builds a browser client with the base JavaScript SDK. By the end, the app can:

create an avatar from an image and save its id
start a voice call and play the character
publish your microphone, and send text with say
interrupt the character, and change its voice or outfit mid-call
end the call, and handle a call that ends on its own

If you’re new to Reactor, start with the Quickstart for authentication and connecting a browser client. See the schema for every Avatar command and field.

Install the SDK

npm
pnpm

npm install @reactor-team/js-sdk

pnpm add @reactor-team/js-sdk

Understand the model

Three ideas cover most of the model:

The avatar and the call are separate. Create or attach an avatar, then start and end calls with it. Ending a call keeps the avatar, so the next start_call is fast.
session_state reports the avatar and call state. It arrives on connect and on every change. Use it for the phase and controls; use transcript messages for captions.
Model refusals and call failures arrive as command_error. Branch on its code and use origin to distinguish invalid input from a service failure. Handle SDK connection and upload errors separately.

Connect

Mint a token on your server, as Authentication describes, and pass it to connect(). Add media elements to the page, then subscribe to messages and tracks before you connect so you do not miss the first snapshot. jwtToken is the token your server returns. Connect render, appendTranscript, and showError to your app’s UI.

<video autoplay playsinline></video>
<audio autoplay controls></audio>

import { Reactor } from "@reactor-team/js-sdk";

const reactor = new Reactor({
  modelName: "reactor/vidu-s2-avatar",
  modelTracks: [
    { name: "mic", kind: "audio", direction: "sendonly" },
    { name: "webcam", kind: "video", direction: "sendonly" },
    { name: "main_video", kind: "video", direction: "recvonly" },
    { name: "main_audio", kind: "audio", direction: "recvonly" },
  ],
});

const resumeOutputs = () =>
  Promise.all([
    reactor.resumeTrack("main_video"),
    reactor.resumeTrack("main_audio"),
  ]);

let state = { phase: "idle" };

reactor.on("message", (msg) => {
  switch (msg.type) {
    case "session_state":
      state = msg.data;
      render(state);
      if (state.phase === "live") {
        void resumeOutputs().catch(console.error);
      }
      break;
    case "transcript":
      appendTranscript(msg.data.speaker, msg.data.text);
      break;
    case "command_error":
      showError(msg.data);
      break;
  }
});

const video = document.querySelector("video");
const audio = document.querySelector("audio");
reactor.on("trackReceived", (name, track) => {
  if (name === "main_video") {
    video.srcObject = new MediaStream([track]);
    void video.play().catch(console.error);
  }
  if (name === "main_audio") {
    audio.srcObject = new MediaStream([track]);
    void audio.play().catch(console.error);
  }
});

const initialState = new Promise((resolve) => {
  const onInitialState = (msg) => {
    if (msg.type !== "session_state") return;
    reactor.off("message", onInitialState);
    resolve(msg.data);
  };
  reactor.on("message", onInitialState);
});

await reactor.connect(jwtToken);
await initialState;
await resumeOutputs();

Declare all four model tracks, including webcam even for an audio call. The model sends its first session_state as soon as the session connects. Wait for it before sending a command.

Browsers may block sound until the user interacts with the page. Start the call from a button click. If autoplay is still blocked, the user can press play on the audio element.

Create or attach an avatar

On the first visit, upload an image and create the avatar. The snapshot moves to preparing_avatar, then to avatar_ready. Save the avatar_id it reports.

async function createAvatar(file) {
  const image = await reactor.uploadFile(file);
  await reactor.sendCommand("create_avatar", { image, name: "Tina" });
}

// In render(), once the avatar is ready:
if (state.phase === "avatar_ready" && state.avatar_id) {
  localStorage.setItem("avatarId", state.avatar_id);
}

On a later visit, attach the saved avatar instead. Avatars are kept for 90 days. If the id has expired, command_error reports AVATAR_NOT_FOUND; clear the saved id and create a new avatar.

const savedId = localStorage.getItem("avatarId");
if (savedId) {
  await reactor.sendCommand("attach_avatar", { avatar_id: savedId });
}

Start the call

Publish the microphone before start_call so it reaches the character when the call becomes live. Speech before that phase may not reach the character.

let mic = null;

async function startCall(voice) {
  let stream;
  try {
    stream = await navigator.mediaDevices.getUserMedia({
      audio: { echoCancellation: true, noiseSuppression: true, channelCount: 1 },
    });
    mic = stream.getAudioTracks()[0];
    await reactor.publishTrack("mic", mic);
  } catch {
    stream?.getTracks().forEach((track) => track.stop());
    mic = null; // The call still works with `say`.
  }

  await reactor.sendCommand("start_call", {
    persona: "You are Tina, a warm museum guide. Answer in one or two short sentences.",
    greeting: "Say hello and ask what brought them to the museum.",
    ...(voice && { voice }), // omit voice to use the default
  });
}

To offer a voice picker, get the catalog from the list_voices reply:

const reply = await reactor.sendCommand("list_voices", {});
const voices = reply?.data.system ?? [];

The command takes no settings, so pass {}. Use a voice value from the catalog in start_call. Omit voice to use the default; do not send null. While the phase is starting or warming_up, show a spinner. When the microphone reaches the live call, the snapshot reports mic_forwarding: true. If the user blocks the microphone, start the call anyway: say still reaches the character. To mute, set mic.enabled = false when mic is set. To let the character see the user, start the call with call_mode: "video", and publish the camera to webcam the same way. call_mode is fixed for the call.

Talk and steer

With the microphone published, the user can just speak. The transcript messages give you both sides of the conversation as captions. For typed chat, send say:

async function sendText(text) {
  if (state.phase !== "live" || !state.control_ready) return;
  await reactor.sendCommand("say", { text });
}

await sendText("What is the oldest thing in this room?");

Add controls for the rest of the live commands. Call putOnJacket with a publicly fetchable image URL, and keep the returned ID to remove that image later. image_id is a label you choose, unique within the call; it does not need to be a UUID. Use a different ID for each additional image.

async function interrupt() {
  await reactor.sendCommand("interrupt", {});
}

async function changeVoice() {
  const catalog = (await reactor.sendCommand("list_voices", {}))?.data.system ?? [];
  const nextVoice = catalog.find((item) => item.voice !== state.voice)?.voice;
  if (nextVoice) await reactor.sendCommand("update_call", { voice: nextVoice });
}

async function putOnJacket(jacketUrl) {
  const jacketId = "my_jacket";
  await reactor.sendCommand("set_reference_images", {
    images: [{ image_url: jacketUrl, image_id: jacketId, kind: "garment" }],
  });
  return jacketId;
}

async function removeJacket(jacketId) {
  await reactor.sendCommand("clear_reference_images", { image_ids: [jacketId] });
}

Only send these while state.phase is live and state.control_ready is true. Enable the controls from those two fields together. Outside a live call the commands are refused with NOT_LIVE.

End the call

const ended = await reactor.sendCommand("end_call", {});
console.log(ended?.data.duration_seconds);

The snapshot moves through ending to ended. The avatar stays available for another call. When the call ends, or start_call is refused, stop the microphone with mic?.stop() and call reactor.unpublishTrack("mic"). A “Call again” button should publish the microphone again before sending start_call. A session bills while it is ready, including time between calls. Call reactor.disconnect() when the user is done, or after a few idle minutes with no call. A call can also end on its own. Handle it in render() from end_reason:

if (state.phase === "ended") {
  switch (state.end_reason) {
    case "ended_by_client":
      break;
    case "max_duration":
    case "idle_timeout":
      offerCallAgain();
      break;
    case "content_policy":
      showMessage(state.last_error?.reason);
      break;
    case "upstream_quota":
      showMessage("Calls are unavailable right now. Try again later.");
      break;
    default:
      offerCallAgain(); // upstream_interrupted, media_lost, bridge_failed
  }
}

call_max_seconds and call_elapsed_seconds on the snapshot let you show a countdown before the time limit ends the call.

Surface errors

Every refused command arrives as command_error, and the latest one also sits on state.last_error.

function showError(err) {
  if (err.origin === "request") {
    showMessage(err.reason); // Fix the input and try again.
  } else if (err.retryable) {
    showMessage("Busy right now. Please try again shortly.");
  } else {
    showMessage(`${err.reason} (trace ${err.trace_id ?? "none"})`);
  }
}

If start_call fails with UPSTREAM_CAPACITY, wait a few seconds before trying again. Log trace_id with every error so you can quote it when you report a failure.

What this tutorial leaves out

Multiple clients. Several clients can join one session and steer the same character. See Multiple clients.
Turn-taking and reply tuning. See the prompt guide.

Overview

Vidu S2-Avatar

H3 Reference Turbo Realtime

FastH3

Visko Orbis Stable

Visko Orbis Dynamic

HappyOyster

LingBot World 2

LingBot

X2

Helios

LongLive-2.0

SANA-Streaming

LTX

Vidu S2-Avatar tutorial

Install the SDK

Understand the model

Connect

Create or attach an avatar

Start the call

Talk and steer

End the call

Surface errors

What this tutorial leaves out

​Install the SDK

​Understand the model

​Connect

​Create or attach an avatar

​Start the call

​Talk and steer

​End the call

​Surface errors

​What this tutorial leaves out

Install the SDK

Understand the model

Connect

Create or attach an avatar

Start the call

Talk and steer

End the call

Surface errors

What this tutorial leaves out