LingBot schema reference

This page documents the complete LingBot wire surface: the media tracks it produces, the session lifecycle, every command you can send, and the messages the model emits back. For what LingBot is and a quick start, see the overview.

Tracks

Direction	Name	Type	Format	Rate
Outbound	`main_video`	Video	`(N, H, W, 3)` uint8 RGB	Adaptive, paced to model throughput

Default resolution is 1664 x 960. There are no inbound tracks; all client → model communication is through commands.

Session lifecycle

Once the connection reaches ready, the session begins in WAITING. start transitions to GENERATING (provided a prompt and seed image are set); pause moves to PAUSED; resume returns to GENERATING; reset clears state and returns to WAITING from any state. The Commands table below lists each command’s preconditions and effects. See Sessions for the separate connection-level lifecycle (disconnected → connecting → waiting → ready) the session passes through before reaching the runtime states above. When all chunks of a run complete and the session is still started, the server automatically kicks off the next run with the same prompt and image. Call reset to stop the loop and re-stage with new conditions.

Commands

Send commands to the model using reactor.sendCommand(). Setter commands (set_*) take effect at the next chunk boundary. Below are all available commands:

Command	Description
`set_prompt`	Set or swap the text prompt (works before and during generation)
`set_image`	Set the seed image that anchors the world (required before `start`)
`set_movement`	Set WASD movement direction
`set_look_horizontal`	Set yaw (look left/right) direction
`set_look_vertical`	Set pitch (look up/down) direction
`set_rotation_speed_deg`	Set how fast the camera rotates when a look axis is active
`set_seed`	Set the RNG seed for the next generation
`start`	Begin generation (requires a prompt and a seed image)
`pause`	Pause after the current chunk
`resume`	Resume from a pause
`reset`	Clear all session state and return to `WAITING`

set_prompt

Stores the prompt and re-applies it on the next chunk boundary. Works before start and mid-generation for live prompt swaps. Emits prompt_accepted then conditions_ready.Parameters:

Parameter	Type	Required	Description
`prompt`	string	Yes	Prompt text, ≤ 1000 chars

await reactor.sendCommand("set_prompt", {
  prompt: "A misty old-growth forest, soft morning light filtering through the canopy.",
});

const { setPrompt } = useLingbot();
await setPrompt({
  prompt: "A misty old-growth forest, soft morning light filtering through the canopy.",
});

await reactor.send_command("set_prompt", {
    "prompt": "A misty old-growth forest, soft morning light filtering through the canopy.",
})

set_image

Decodes the uploaded file and caches it as the seed image. Upload the file first with uploadFile(), then pass the returned FileRef as image. Rejects non-image MIME types with command_error. Emits image_accepted then conditions_ready.Parameters:

Parameter	Type	Required	Description
`image`	FileRef	Yes	A reference to an uploaded image, returned by `uploadFile()`

const ref = await reactor.uploadFile(seedImageFile);
await reactor.sendCommand("set_image", { image: ref });

const { uploadFile, setImage } = useLingbot();
const ref = await uploadFile(seedImageFile);
await setImage({ image: ref });

ref = await reactor.upload_file("seed.png")
await reactor.send_command("set_image", {"image": ref})

set_movement / set_look_horizontal / set_look_vertical

The three drive setters. Movement and look are persistent state, not pulses: a value holds across chunks until you send a new one (including "idle"). To stop, send "idle" explicitly. All apply at the next chunk boundary.Parameters:

Command	Value	Default
`set_movement`	`"idle" \| "forward" \| "back" \| "strafe_left" \| "strafe_right"`	`"idle"`
`set_look_horizontal`	`"idle" \| "left" \| "right"`	`"idle"`
`set_look_vertical`	`"idle" \| "up" \| "down"`	`"idle"`

// Start walking forward and panning left
await reactor.sendCommand("set_movement", { movement: "forward" });
await reactor.sendCommand("set_look_horizontal", { look_horizontal: "left" });

// Stop moving (look continues until you idle it too)
await reactor.sendCommand("set_movement", { movement: "idle" });

const { setMovement, setLookHorizontal } = useLingbot();
await setMovement({ movement: "forward" });
await setLookHorizontal({ look_horizontal: "left" });

await reactor.send_command("set_movement", {"movement": "forward"})
await reactor.send_command("set_look_horizontal", {"look_horizontal": "left"})

set_rotation_speed_deg

Degrees the camera rotates per latent frame when a look axis is active. Setting it to 0 disables rotation entirely even if a look axis is non-idle.Parameters:

Parameter	Type	Required	Description
`rotation_speed_deg`	float	Yes	0.0 – 30.0. Default 5.0

await reactor.sendCommand("set_rotation_speed_deg", { rotation_speed_deg: 10.0 });

const { setRotationSpeedDeg } = useLingbot();
await setRotationSpeedDeg({ rotation_speed_deg: 10.0 });

await reactor.send_command("set_rotation_speed_deg", {"rotation_speed_deg": 10.0})

set_seed

RNG seed for the next generation. Read once when start fires; later changes require reset then a new start.Parameters:

Parameter	Type	Required	Description
`seed`	int	Yes	≥ 0. Same seed → same output. Default 42

await reactor.sendCommand("set_seed", { seed: 42 });

const { setSeed } = useLingbot();
await setSeed({ seed: 42 });

await reactor.send_command("set_seed", {"seed": 42})

start

Begins generation. Requires both a prompt and a seed image to be set; fails with command_error otherwise. Emits generation_started.

pause / resume

pause halts after the current chunk finishes (generation_paused) and is valid only while GENERATING; resume continues from the next chunk (generation_resumed) and is valid only while PAUSED.

reset

Clears the prompt, image, and the started flag, returning to WAITING from any state. Emits generation_reset.

await reactor.sendCommand("start", {});
await reactor.sendCommand("pause", {});
await reactor.sendCommand("resume", {});
await reactor.sendCommand("reset", {});

const { start, pause, resume, reset } = useLingbot();

await reactor.send_command("start", {})
await reactor.send_command("pause", {})
await reactor.send_command("resume", {})
await reactor.send_command("reset", {})

Messages

LingBot emits the following messages. Every message is delivered as JSON { "type": "<name>", "data": { … } }.

Event	When	Payload
`prompt_accepted`	After `set_prompt`	`{ prompt: string }`
`image_accepted`	After `set_image` decodes	`{ width: int, height: int }`
`conditions_ready`	After `set_prompt` or `set_image`	`{ has_prompt: bool, has_image: bool }`
`generation_started`	After `start`	`{ prompt: string, chunk_num: int, frame_num: int }`
`chunk_complete`	After each chunk emits	`{ chunk_index: int, frames_emitted: int, active_prompt: string, active_action: string }`
`generation_paused`	After `pause`	`{ chunk_index: int }`
`generation_resumed`	After `resume`	`{ chunk_index: int }`
`generation_complete`	After all chunks of a run finish	`{ total_chunks: int }`
`generation_reset`	After `reset`	`{ reason: string }`
`command_error`	When a command is rejected	`{ command: string, reason: string }`
`state`	On connect, after every command, and after every `chunk_complete`	Full session snapshot

`state` payload

state is the single source of truth for driving UI. Subscribe once and treat it as the authoritative session snapshot; you generally do not need to track individual set_* and chunk_complete events yourself.

Field	Type	Meaning
`running`	`bool`	`started && !paused`. Equivalent to “frames are actively streaming”
`started`	`bool`	True once `start` has been accepted; remains true through pauses and across auto-restarts. Flips back to false on `reset`
`paused`	`bool`	True while in the `PAUSED` state
`current_chunk`	`int`	Zero-based index of the last completed chunk; `0` before the first chunk and after `reset`
`current_prompt`	`string \| null`	The prompt currently driving generation, or `null` if none set
`has_prompt`	`bool`	Whether a prompt has been set this session
`has_image`	`bool`	Whether a seed image has been set this session
`current_action`	`string`	`+`-joined composite of movement and look. E.g. `"w+left"`, or `"still"` when idle
`movement`	`string`	Current value of the `movement` input field
`look_horizontal`	`string`	Current value of the `look_horizontal` input field
`look_vertical`	`string`	Current value of the `look_vertical` input field
`rotation_speed_deg`	`float`	Current rotation speed (0.0 – 30.0)
`seed`	`int`	Current seed value (effective only on the next `start`)

Complete example

Stage a seed image and prompt, wait for the image to decode, start, then drive the camera.

import { LingbotModel } from "@reactor-models/lingbot";

const lingbot = new LingbotModel();

lingbot.onChunkComplete(({ chunk_index, active_action }) => {
  console.log(`chunk ${chunk_index}: ${active_action}`);
});
lingbot.onCommandError(({ command, reason }) => {
  console.error(`${command} rejected: ${reason}`);
});

await lingbot.connect(jwt);

// Wait for the seed image to decode so the first chunk renders from it.
const imageAccepted = new Promise<void>((resolve) => lingbot.onImageAccepted(() => resolve()));
const ref = await lingbot.uploadFile(seedImageFile);
await lingbot.setImage({ image: ref });
await imageAccepted;

await lingbot.setPrompt({
  prompt: "A misty old-growth forest, soft morning light filtering through the canopy.",
});
await lingbot.start();

// Drive the camera (persistent state, holds until you change it).
await lingbot.setMovement({ movement: "forward" });
await lingbot.setLookHorizontal({ look_horizontal: "left" });

import asyncio, os
from reactor_sdk import Reactor

async def main():
    reactor = Reactor(model_name="lingbot", api_key=os.environ["REACTOR_API_KEY"])

    image_accepted = asyncio.Event()

    @reactor.on_message("image_accepted")
    def _(msg):
        image_accepted.set()

    await reactor.connect()

    ref = await reactor.upload_file("seed.png")
    await reactor.send_command("set_image", {"image": ref})
    await image_accepted.wait()

    await reactor.send_command("set_prompt", {
        "prompt": "A misty old-growth forest, soft morning light filtering through the canopy.",
    })
    await reactor.send_command("start", {})

    # Drive the camera.
    await reactor.send_command("set_movement", {"movement": "forward"})
    await asyncio.Event().wait()

asyncio.run(main())

​Tracks

​Session lifecycle

​Commands

​set_prompt

​set_image

​set_movement / set_look_horizontal / set_look_vertical

​set_rotation_speed_deg

​set_seed

​start

​pause / resume

​reset

​Messages

​state payload

​Complete example