Vidu S2-Editing tutorial

This tutorial builds a browser client with the base JavaScript SDK. By the end, the app can:

publish your camera and show the edited video
start an edit from an uploaded reference image
switch the image or edit type while the edit runs
end the edit, and handle an edit that ends on its own or a switch that fails

If this is your first Reactor app, read the Quickstart to learn how a browser client connects. The schema lists every command and field. Add two video elements to your page: <video id="camera"> previews the webcam, and <video id="edited"> shows the model’s output. Add a button with id="switch" if you want to change the look. The snippets use showStatus(), hideStatus(), showMessage(), and offerStartAgain() as placeholders for your own UI. The render() and showError() functions appear later in this tutorial. Get jwtToken from your server as Authentication describes.

Install the SDK

npm
pnpm

npm install @reactor-team/js-sdk

pnpm add @reactor-team/js-sdk

Understand the model

The session works like this:

Your camera is the whole input. The model edits nothing until your camera reaches it. Publish it, and watch camera_forwarding on the snapshot.
session_state shows the current edit. It arrives on connect and on every change. Use it to show progress and decide when controls are available.
Model refusals and edit failures arrive as command_error. Check its code. Use origin to distinguish invalid input from a service failure. Handle SDK connection and upload errors separately. A failed switch can send this error after its reply.

Connect

Mint a token on your server, as Authentication describes, and pass it to connect(). Subscribe to messages and tracks before you connect, so you do not miss the first snapshot.

import { Reactor } from "@reactor-team/js-sdk";

const reactor = new Reactor({
  modelName: "reactor/vidu-s2-editing",
  modelTracks: [
    { name: "camera", kind: "video", direction: "sendonly" },
    { name: "main_video", kind: "video", direction: "recvonly" },
  ],
});

let state = { phase: "idle" };

reactor.on("message", (msg) => {
  if (msg.type === "session_state") {
    const wasLive = state.phase === "live";
    state = msg.data;
    render(state);
    if (state.phase === "live" && !wasLive) {
      void reactor.resumeTrack("main_video").catch((error) => {
        console.error("Could not resume edited video", error);
      });
    }
  }
  if (msg.type === "command_error") showError(msg.data);
});

const output = document.querySelector("video#edited");
const switchButton = document.querySelector("button#switch");
reactor.on("trackReceived", (name, track) => {
  if (name === "main_video") output.srcObject = new MediaStream([track]);
});

const initialState = new Promise((resolve) => {
  const onInitialState = (msg) => {
    if (msg.type !== "session_state") return;
    // Remove this listener after the first state message.
    reactor.off("message", onInitialState);
    resolve(msg.data);
  };
  reactor.on("message", onInitialState);
});

await reactor.connect(jwtToken);
await initialState;
await reactor.resumeTrack("main_video");

The model sends its first session_state when the session connects. Wait for it before you send a command. The wasLive check resumes the output track each time an edit enters live, including later edits in the same session. Style the output <video> with object-fit: contain so the whole edited frame stays visible. The output can differ from the camera in size and orientation.

Publish the camera

Publish the camera as soon as the connection is ready. Show it in a second <video> so the user can see what goes in.

const stream = await navigator.mediaDevices.getUserMedia({
  video: { width: 1280, height: 720 },
});
document.querySelector("video#camera").srcObject = stream;
await reactor.publishTrack("camera", stream.getVideoTracks()[0]);

This example requests 1280 × 720. The browser may choose different dimensions.

Start an edit

Let the user pick a reference image and an edit type, then start the edit.

async function startEdit(file, editingType) {
  const reference_image = await reactor.uploadFile(file);
  await reactor.sendCommand("start_edit", {
    reference_image,
    editing_type: editingType, // "style_transfer", "virtual_tryon", ...
  });
}

The phase moves through starting and warming_up to live. The first edited frame can arrive after the phase becomes live. Drive the loading state from the snapshot:

function render(state) {
  if (state.phase === "starting" || state.phase === "warming_up") {
    showStatus("Starting the edit…");
  } else if (state.phase === "live" && !state.camera_forwarding) {
    showStatus("Waiting for your camera…");
  } else if (state.phase === "live" && !state.video_receiving) {
    showStatus("Editing…");
  } else if (state.phase === "live") {
    hideStatus();
  }
  switchButton.disabled = !state.control_ready;
}

Switch the look

Call switch_reference with what you want to change. What you omit stays as it is.

// A new garment, same edit type.
await reactor.sendCommand("switch_reference", { reference_image_url: jacketUrl });

// The same image, read as a background instead.
await reactor.sendCommand("switch_reference", { editing_type: "background_replacement" });

The reply, reference_switched, means the switch was sent. The picture changes within a few seconds. There is no later success message, so do not wait for one, and do not resend the switch when nothing seems to change. If the switch fails, a command_error for switch_reference can arrive after the reply. The edit keeps running with the previous look. Handle that error in Surface errors.

End the edit

const ended = await reactor.sendCommand("end_edit", {});
console.log(ended?.data.duration_seconds);

The phase moves through ending to ended. This tutorial leaves the camera published. A “Start again” button can send start_edit without another camera step. If your app unpublishes the camera between edits, publish it again before the next edit. A session bills while it is ready, including time between edits. Call reactor.disconnect() when the user is done, or after a few idle minutes with no edit. An edit can also end on its own. Handle it in render() from end_reason:

if (state.phase === "ended") {
  switch (state.end_reason) {
    case "ended_by_client":
      break;
    case "max_duration":
      offerStartAgain();
      break;
    case "content_policy":
      showMessage(state.last_error?.reason);
      break;
    case "upstream_quota":
      showMessage("Editing is unavailable right now. Try again later.");
      break;
    default:
      offerStartAgain(); // upstream_interrupted, media_lost, bridge_failed
  }
}

edit_max_seconds and edit_elapsed_seconds on the snapshot let you show how long the edit has left.

Surface errors

The command_error handler in Connect calls showError(). When a switch fails, tell the user and keep the previous look on screen. Use origin and retryable for other model errors:

function showError(err) {
  if (err.command === "switch_reference") {
    showMessage("That image could not be used. The previous look is still on.");
    return;
  }
  if (err.origin === "request") {
    showMessage(err.reason); // Fix the input and try again.
  } else if (err.retryable) {
    showMessage("Busy right now. Try again in a few seconds.");
  } else {
    showMessage(`${err.reason} (trace ${err.trace_id ?? "none"})`);
  }
}

Handle SDK connection and upload errors where those calls run. They do not arrive as command_error messages.

Overview

Vidu S2-Avatar

Vidu S2-Editing

H3 Reference Turbo Realtime

FastH3

Visko Orbis Stable

Visko Orbis Dynamic

HappyOyster

LingBot World 2

LingBot

X2

Helios

LongLive-2.0

SANA-Streaming

LTX

Vidu S2-Editing tutorial

Install the SDK

Understand the model

Connect

Publish the camera

Start an edit

Switch the look

End the edit

Surface errors

​Install the SDK

​Understand the model

​Connect

​Publish the camera

​Start an edit

​Switch the look

​End the edit

​Surface errors

Install the SDK

Understand the model

Connect

Publish the camera

Start an edit

Switch the look

End the edit

Surface errors