Vidu S2-Editing overview

Vidu S2-Editing edits your live webcam video with one reference image. Choose an edit type to tell the model what to take from the image: its visual style, a garment, a character, or a background. Publish your webcam and start an edit. The model returns edited video in real time. While the edit runs, you can change the image, the edit type, or both, without restarting it. When you end the edit, the session stays open, so you can start another one. The schema lists the model’s commands and messages. The prompt guide helps you choose images for each edit type. The tutorial builds a browser app. The base wire protocol is the same as every other Reactor model: open a session with the Reactor class, send named commands, and receive messages. The model name is reactor/vidu-s2-editing. The examples use the base SDK.

Usage is billed per session-minute while the session is ready, including time between edits. Disconnect when you are done. See Billing for how billing maps to rates.

At a glance

Spec	Value
Model name	`reactor/vidu-s2-editing`
Pricing
Input	Webcam video, one reference image, and an edit type
Output	Video
Output aspect ratio	7:4 (952 × 544) in current edited output. Check the received track’s dimensions at runtime.

Key features

Live camera in, edited video out

Publish your camera and watch the edit in real time. There is nothing to render ahead of time.

Four edit types

Restyle the whole scene, try on a garment, replace yourself with a character, or replace the background, all from one reference image.

Switch mid-edit

Change the reference image, the edit type, or both with switch_reference. The picture changes within a few seconds and the edit keeps running.

Quick start

Connect with the base SDK, publish your camera, and start an edit:

import { Reactor } from "@reactor-team/js-sdk";

async function startEditing(jwtToken, file, video) {
  const reactor = new Reactor({
    modelName: "reactor/vidu-s2-editing",
    modelTracks: [
      { name: "camera", kind: "video", direction: "sendonly" },
      { name: "main_video", kind: "video", direction: "recvonly" },
    ],
  });

  let previousPhase;
  reactor.on("message", (msg) => {
    if (msg.type === "session_state") {
      console.log(msg.data.phase);
      if (msg.data.phase === "live" && previousPhase !== "live") {
        void reactor.resumeTrack("main_video").catch(console.error);
      }
      previousPhase = msg.data.phase;
    }
    if (msg.type === "command_error") console.error(msg.data.code, msg.data.reason);
  });
  reactor.on("trackReceived", (name, track) => {
    if (name === "main_video") video.srcObject = new MediaStream([track]);
  });

  const initialState = new Promise((resolve) => {
    const onInitialState = (msg) => {
      if (msg.type !== "session_state") return;
      // Stop this temporary listener after the first snapshot.
      reactor.off("message", onInitialState);
      resolve(msg.data);
    };
    reactor.on("message", onInitialState);
  });
  await reactor.connect(jwtToken);
  await initialState;
  await reactor.resumeTrack("main_video");

  // Publish the camera first: nothing is edited until it arrives.
  const stream = await navigator.mediaDevices.getUserMedia({ video: true });
  await reactor.publishTrack("camera", stream.getVideoTracks()[0]);

  const reference_image = await reactor.uploadFile(file);
  await reactor.sendCommand("start_edit", {
    reference_image,
    editing_type: "background_replacement",
  });
  return reactor;
}

from reactor_sdk import Reactor

async with Reactor(model_name="reactor/vidu-s2-editing", api_key="rk_...") as reactor:

    @reactor.on_message
    def on_message(msg):
        if msg["type"] == "session_state":
            print(msg["data"]["phase"], msg["data"]["camera_forwarding"])

    await reactor.connect()

    # Publish the camera slot, then push frames into it (see Tracks).
    camera = await reactor.track("camera").publish()

    await reactor.send_command("start_edit", {
        "reference_image_url": "https://example.com/beach.jpg",
        "editing_type": "background_replacement",
    })

See Authentication to connect your app, Tracks to publish a camera from each SDK, and the tutorial for a full browser app.

How it works

Connect. The first session_state message reports phase idle. The camera and main_video tracks exist from the start, so you can attach a player and publish your camera right away. Until an edit is running, main_video shows a neutral dark frame.
Publish your camera. The model edits nothing until your camera reaches it. Publish camera before or right after start_edit.
Start an edit. Call start_edit with a reference image and an edit type. The phase moves through starting and warming_up to live. camera_forwarding turns true once your camera reaches the model. video_receiving turns true when edited frames arrive. Use these fields to show progress. The first edited frame can arrive after the phase becomes live.
Switch. Call switch_reference to change the image, the edit type, or both. The picture changes within a few seconds.
End the edit. Call end_edit, or let the edit end at its time limit. The session stays open, so start_edit works again.

Overview

Vidu S2-Avatar

Vidu S2-Editing

H3 Reference Turbo Realtime

FastH3

Visko Orbis Stable

Visko Orbis Dynamic

HappyOyster

LingBot World 2

LingBot

X2

Helios

LongLive-2.0

SANA-Streaming

LTX

Vidu S2-Editing overview

At a glance

Key features

Live camera in, edited video out

Four edit types

Switch mid-edit

Quick start

How it works

​At a glance

​Key features

Live camera in, edited video out

Four edit types

Switch mid-edit

​Quick start

​How it works

At a glance

Key features

Quick start

How it works