X2 Schema Reference

This page documents the complete X2 wire surface: the media tracks it consumes and produces, the session lifecycle, every command you can send, and the messages the model emits back. For what X2 is and a quick start, see the overview.

Tracks

Direction	Name	Type	Description
Inbound	`source`	Video	The stream to edit, published by the client
Outbound	`main_video`	Video	The re-rendered stream, published by the model

Unlike the generation-only models, X2 takes an inbound track: the client publishes the video to be edited on source, and the model streams the transformed result back on main_video. The model’s native source pacing is 24 fps, and the output plays at 24 fps with no frame interpolation and no upscaling. The model chooses the output resolution (a bucket around 832p) once at the first generation, from the source stream’s aspect ratio. It holds for the whole session and arrives in generation_started and in every state_update.

A still image works as a source too: draw it to a canvas and capture it with canvas.captureStream(24), repainting at the stream rate so the capturer keeps emitting frames. From the model’s side that is indistinguishable from a video of a motionless scene, which is the drag-to-animate setup. The tutorial shows the pattern.

Session lifecycle

There is no start command and no staged or started state machine to drive. Generation begins on its own once two things are true: a non-empty prompt is set, and source frames are arriving. It runs until the session ends, a new reference image forces an automatic restart, or you call reset. Reset stops generation and clears the prompt, reference image, and pointer. See Sessions for the connection-level lifecycle (disconnected → connecting → waiting → ready) the session passes through first. Every control follows block semantics: the model generates one block at a time, a prompt change applies from the next block, it samples the pointer once per block, and the backlog policy switches at the next block.

Commands

Send commands with reactor.sendCommand() on the base SDK, or the typed methods on the generated model client / useX2() hook. Below are all available commands:

Command	Description
`set_prompt`	Set or hot-swap the editing instruction (required for generation)
`set_reference_image`	Set the character/object image that anchors swaps and insertions
`set_pointer`	Update the drag pointer: position and press state together
`set_pointer_x`	Update only the pointer’s horizontal position
`set_pointer_y`	Update only the pointer’s vertical position
`set_pointer_active`	Update only whether a drag is in progress
`set_keep_backlog`	Set the source-frame policy between bounded latency and every frame
`reset`	Stop generation and clear the prompt, reference image, and pointer

`set_prompt`

Set the editing instruction that guides the re-render (for example a character swap or insertion). Generation needs a non-empty prompt before it begins; setting one while generating applies from the next block. Emits prompt_accepted, then state_update. Parameters:

Parameter	Type	Required	Description
`prompt`	string	No	The editing instruction. Up to 1000 characters. Default `""` (empty)

await reactor.sendCommand("set_prompt", {
  prompt: "replace the character in the video with the character from the reference image",
});

const { setPrompt } = useX2();
await setPrompt({
  prompt: "replace the character in the video with the character from the reference image",
});

await reactor.send_command("set_prompt", {
    "prompt": "replace the character in the video with the character from the reference image",
})

`set_reference_image`

Provide a reference image of a character or object to insert or swap into the video. Upload the file first with uploadFile(), then pass the returned FileRef. Emits reference_image_accepted and state_update on success, or command_error if the file cannot be decoded as an image. The reference can be set before generation or replaced while generating. A mid-run replacement restarts the stream on its own (generation_stopped with reason reference_image_changed, then a fresh generation_started) and the new image conditions the edit from its first block. The prompt stays set across the swap, so the edit resumes without a re-arm; no reset is needed. Parameters:

Parameter	Type	Required	Description
`reference_image`	FileRef	Yes	A reference to an uploaded image, returned by `uploadFile()`

const ref = await reactor.uploadFile(imageFile);
await reactor.sendCommand("set_reference_image", { reference_image: ref });

const { uploadFile, setReferenceImage } = useX2();
const ref = await uploadFile(imageFile);
await setReferenceImage({ reference_image: ref });

ref = await reactor.upload_file("character.jpg")
await reactor.send_command("set_reference_image", {"reference_image": ref})

`set_pointer`

Update the drag pointer that steers the edited subject’s motion, setting its position and press state together. Valid at any time; the model samples the pointer once per generated block, and it only has an effect while active is true. Coordinates map to the output frame, so account for any letterboxing your player introduces when you map pointer events. Emits pointer_changed, then state_update. Parameters:

Parameter	Type	Required	Description
`x`	number	No	Horizontal position, `0` (left) to `1` (right). Default `0.5`
`y`	number	No	Vertical position, `0` (top) to `1` (bottom). Default `0.5`
`active`	boolean	No	Whether a drag is in progress; ignored while false. Default `false`

// Drag in progress at the frame's center-right.
await reactor.sendCommand("set_pointer", { x: 0.8, y: 0.5, active: true });

// Release: the pointer stops steering.
await reactor.sendCommand("set_pointer", { x: 0.8, y: 0.5, active: false });

const { setPointer } = useX2();
await setPointer({ x: 0.8, y: 0.5, active: true });
await setPointer({ active: false });

await reactor.send_command("set_pointer", {"x": 0.8, "y": 0.5, "active": True})
await reactor.send_command("set_pointer", {"active": False})

Because the model samples the pointer once per block, sending positions faster than about 30 Hz buys nothing. Throttle pointer-move events, and use a trailing send so the final position of a fast gesture still lands. Always send active: false on release (and on unmount) so the model does not keep steering toward a stale point.

`set_pointer_x`, `set_pointer_y`, `set_pointer_active`

Single-field variants of set_pointer, for updating one pointer field at a time. Same semantics: sampled once per block, effective only while the pointer is active. Prefer set_pointer when you have all three values; the position and press state then change in one step. Parameters (one per command):

Command	Parameter	Type	Description
`set_pointer_x`	`pointer_x`	number	Horizontal position, `0` to `1`. Default `0.5`
`set_pointer_y`	`pointer_y`	number	Vertical position, `0` to `1`. Default `0.5`
`set_pointer_active`	`pointer_active`	boolean	Whether a drag is in progress. Default `false`

await reactor.sendCommand("set_pointer_x", { pointer_x: 0.25 });
await reactor.sendCommand("set_pointer_active", { pointer_active: true });

const { setPointerX, setPointerActive } = useX2();
await setPointerX({ pointer_x: 0.25 });
await setPointerActive({ pointer_active: true });

await reactor.send_command("set_pointer_x", {"pointer_x": 0.25})
await reactor.send_command("set_pointer_active", {"pointer_active": True})

`set_keep_backlog`

Pick the source-frame consumption policy, applied from the next block:

false (default): newest frames. The model always reads the most recent source frames and drops any backlog, keeping latency bounded when inference runs slower than the source. Right for live sources like a webcam, where the edit should track “now”.
true: every frame. The model consumes every source frame in order, for smoother motion at the cost of a growing delay. Right for pre-recorded clips and for drag-to-animate, where smoothness matters more than latency.

Emits state_update. reset does not clear this setting, unlike the prompt, reference image, and pointer: it persists until the session ends, so re-sync any UI toggle to the default on disconnect rather than on reset. Parameters:

Parameter	Type	Required	Description
`keep_backlog`	boolean	No	`true` = every frame in order. Default `false`

await reactor.sendCommand("set_keep_backlog", { keep_backlog: true });

const { setKeepBacklog } = useX2();
await setKeepBacklog({ keep_backlog: true });

await reactor.send_command("set_keep_backlog", {"keep_backlog": True})

`reset`

Stop generation and clear the prompt, reference image, and pointer, returning the session to waiting for new conditions. Valid at any time. Emits generation_stopped with reason reset if a run was active, and state_update. Takes no arguments. Note two things reset leaves alone: the output resolution (chosen once per session, at the first generation) and the keep_backlog policy (persists until the session ends).

await reactor.sendCommand("reset", {});

const { reset } = useX2();
await reset();

await reactor.send_command("reset", {})

Messages

X2 emits the following messages, each arriving as JSON { "type": "<name>", "data": { … } }.

Event	When	Payload
`prompt_accepted`	After `set_prompt`	`{ prompt: string }`
`reference_image_accepted`	After `set_reference_image` decodes	`{ width: int, height: int }`
`pointer_changed`	After `set_pointer`	`{ x: number, y: number, active: bool }`
`generation_started`	When a run begins producing `main_video`	`{ prompt: string, width: int, height: int, has_reference_image: bool }`
`generation_stopped`	When a run ends	`{ reason: "reset" \| "reference_image_changed" }`
`command_error`	When a command is rejected	`{ command: string, reason: string }`
`state_update`	On connect and after every observable change	Full session snapshot (see below)

A generation_stopped with reason reference_image_changed is an automatic restart: a fresh generation_started follows at once, so treat it as a hiccup rather than an end state. Only reason reset means the session is back to waiting for conditions; drive any “clear the drafts” UI off that reason, not off every stop.

`state_update` payload

state_update is the single source of truth for driving UI. It arrives on connect and after every observable state change, always carrying the full state. Subscribe once and reduce it into your app state; the only fact it does not carry is the decoded reference image’s dimensions, which come from the discrete reference_image_accepted ack.

Field	Type	Meaning
`prompt`	`string \| null`	The active editing instruction, or `null` when none is set
`has_reference_image`	`bool`	Whether a reference image is conditioning generation
`pointer_x`	`number`	Current horizontal pointer position, `0` to `1`
`pointer_y`	`number`	Current vertical pointer position, `0` to `1`
`pointer_active`	`bool`	Whether a drag is steering the edited subject
`keep_backlog`	`bool`	Current source-frame policy (see `set_keep_backlog`)
`generating`	`bool`	Whether a run is producing `main_video` frames
`width`	`int \| null`	Output frame width, or `null` before the resolution is chosen
`height`	`int \| null`	Output frame height, or `null` before the resolution is chosen

Example handler:

reactor.on("message", (msg) => {
  switch (msg.type) {
    case "state_update":
      console.log(`generating: ${msg.data.generating} · prompt: ${msg.data.prompt}`);
      break;
    case "generation_started":
      console.log(`output: ${msg.data.width}x${msg.data.height}`);
      break;
    case "command_error":
      console.error(`${msg.data.command} rejected: ${msg.data.reason}`);
      break;
  }
});

import { useX2StateUpdate, useX2CommandError } from "@reactor-models/x2";

function StateReadout() {
  useX2StateUpdate((s) => console.log(`generating: ${s.generating} · prompt: ${s.prompt}`));
  useX2CommandError((e) => console.error(`${e.command} rejected: ${e.reason}`));
  return null;
}

@reactor.on_message
def handle(msg):
    data = msg["data"]
    if msg["type"] == "state_update":
        print(f"generating: {data['generating']} · prompt: {data['prompt']}")
    elif msg["type"] == "command_error":
        print(f"{data['command']} rejected: {data['reason']}")

Complete example

Publish a webcam source, set a reference image and prompt, then steer the subject with a short drag.

import { X2Model } from "@reactor-models/x2";

const x2 = new X2Model();

x2.onGenerationStarted(({ width, height, has_reference_image }) => {
  console.log(`run started at ${width}x${height}, reference: ${has_reference_image}`);
});
x2.onCommandError(({ command, reason }) => {
  console.error(`${command} rejected: ${reason}`);
});
x2.onMainVideo((track, stream) => {
  video.srcObject = stream;
  void video.play();
});

await x2.connect(jwt);

// Publish the stream to edit.
const media = await navigator.mediaDevices.getUserMedia({ video: true });
await x2.publishSource(media.getVideoTracks()[0]);

// Anchor the swap to a reference image, then arm generation with a prompt.
const ref = await x2.uploadFile(characterImageFile);
await x2.setReferenceImage({ reference_image: ref });
await x2.setPrompt({
  prompt: "replace the character in the video with the character from the reference image",
});
// No start command: generation begins now that a prompt is set and frames are flowing.

// Steer the subject with a short drag to the right.
await x2.setPointer({ x: 0.5, y: 0.5, active: true });
await x2.setPointer({ x: 0.8, y: 0.5, active: true });
await x2.setPointer({ x: 0.8, y: 0.5, active: false });

import asyncio, os
from reactor_sdk import Reactor

async def main():
    reactor = Reactor(model_name="x2", api_key=os.environ["REACTOR_API_KEY"])

    @reactor.on_message("generation_started")
    def _(msg):
        d = msg["data"]
        print(f"run started at {d['width']}x{d['height']}")

    await reactor.connect()

    ref = await reactor.upload_file("character.jpg")
    await reactor.send_command("set_reference_image", {"reference_image": ref})
    await reactor.send_command("set_prompt", {
        "prompt": "replace the character in the video with the character from the reference image",
    })

    # Steer with a short drag.
    await reactor.send_command("set_pointer", {"x": 0.5, "y": 0.5, "active": True})
    await reactor.send_command("set_pointer", {"x": 0.8, "y": 0.5, "active": True})
    await reactor.send_command("set_pointer", {"x": 0.8, "y": 0.5, "active": False})
    await asyncio.Event().wait()

asyncio.run(main())

Overview

HappyOyster

LingBot World 2

LingBot

X2

Helios

LongLive-2.0

SANA-Streaming

LTX

X2 Schema Reference

Tracks

Session lifecycle

Commands

`set_prompt`

`set_reference_image`

`set_pointer`

`set_pointer_x`, `set_pointer_y`, `set_pointer_active`

`set_keep_backlog`

`reset`

Messages

`state_update` payload

Complete example

​Tracks

​Session lifecycle

​Commands

​set_prompt

​set_reference_image

​set_pointer

​set_pointer_x, set_pointer_y, set_pointer_active

​set_keep_backlog

​reset

​Messages

​state_update payload

​Complete example

Tracks

Session lifecycle

Commands

`set_prompt`

`set_reference_image`

`set_pointer`

`set_pointer_x`, `set_pointer_y`, `set_pointer_active`

`set_keep_backlog`

`reset`

Messages

`state_update` payload

Complete example